{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "076ddccb",
   "metadata": {},
   "source": [
    "# Data access\n",
    "\n",
    "IceCube has a bunch of public datasets available at [https://icecube.wisc.edu/science/data-releases/](https://icecube.wisc.edu/science/data-releases/). `icecube_tools` provides an easy interface to this repository so that you can download and organise your data through python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "3f498920",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-08T10:34:35.827386Z",
     "iopub.status.busy": "2024-11-08T10:34:35.827176Z",
     "iopub.status.idle": "2024-11-08T10:34:36.665795Z",
     "shell.execute_reply": "2024-11-08T10:34:36.665067Z"
    }
   },
   "outputs": [],
   "source": [
    "from icecube_tools.utils.data import IceCubeData"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ee47025",
   "metadata": {},
   "source": [
    "The `IceCubeData` class provides this functionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "dc32b9fe",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-08T10:34:36.668701Z",
     "iopub.status.busy": "2024-11-08T10:34:36.668066Z",
     "iopub.status.idle": "2024-11-08T10:34:36.680870Z",
     "shell.execute_reply": "2024-11-08T10:34:36.680225Z"
    }
   },
   "outputs": [],
   "source": [
    "my_data = IceCubeData()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0270bbc8",
   "metadata": {},
   "source": [
    "You can use the `find` method to pick out datasets you are interested in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "729ea15a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-08T10:34:36.683339Z",
     "iopub.status.busy": "2024-11-08T10:34:36.682850Z",
     "iopub.status.idle": "2024-11-08T10:34:36.689442Z",
     "shell.execute_reply": "2024-11-08T10:34:36.688824Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['20181018']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "found_dataset = my_data.find(\"20181018\")\n",
    "found_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6225035f",
   "metadata": {},
   "source": [
    "The `my_data` object has been inititalised to store data in the package's default location (\"~/.icecube_data\"). This is where other `icecube_tools` modules will look for stored data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0accfdec",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-08T10:34:36.691929Z",
     "iopub.status.busy": "2024-11-08T10:34:36.691492Z",
     "iopub.status.idle": "2024-11-08T10:34:36.695668Z",
     "shell.execute_reply": "2024-11-08T10:34:36.695017Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/home/runner/.icecube_data'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_data.data_directory"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68c1d7f6",
   "metadata": {},
   "source": [
    "The `fetch` method will download this dataset to this default location for later use by `icecube_tools`. This method takes a list of names, so can also be used to download multiple datasets. `fetch` has a built in delay of a few seconds between HTTP requests to avoid spamming the website. `fetch` will not overwrite files by default, but this can be forced with `overwrite=True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "bbe1aab7",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-08T10:34:36.697627Z",
     "iopub.status.busy": "2024-11-08T10:34:36.697428Z",
     "iopub.status.idle": "2024-11-08T10:34:46.992246Z",
     "shell.execute_reply": "2024-11-08T10:34:46.991536Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "20181018: 0it [00:00, ?it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "20181018: 1048576it [00:00, 10450166.23it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "20181018: 11534336it [00:00, 63288538.29it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "20181018: 11730609it [00:00, 56099319.94it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "my_data.fetch(found_dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42e6700c",
   "metadata": {},
   "source": [
    "You may not want to use `icecube_tools` for other stuff, so you can also fetch to a specificed location with the keyword `write_to`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "3e403092",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-08T10:34:46.994666Z",
     "iopub.status.busy": "2024-11-08T10:34:46.994459Z",
     "iopub.status.idle": "2024-11-08T10:34:56.420429Z",
     "shell.execute_reply": "2024-11-08T10:34:56.419776Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "20181018: 0it [00:00, ?it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "20181018: 1048576it [00:00, 9162310.58it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "20181018: 11534336it [00:00, 60123632.28it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "20181018: 11730609it [00:00, 52666286.58it/s]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "my_data.fetch(found_dataset, write_to=\"data\", overwrite=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d421f4d2",
   "metadata": {},
   "source": [
    "For convenience, there is also the `fetch_all_to` method to download all the available data to a specified location. We comment this here as it can take a while to execute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a4e50ea3",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-11-08T10:34:56.422779Z",
     "iopub.status.busy": "2024-11-08T10:34:56.422371Z",
     "iopub.status.idle": "2024-11-08T10:34:56.425318Z",
     "shell.execute_reply": "2024-11-08T10:34:56.424781Z"
    }
   },
   "outputs": [],
   "source": [
    "# my_data.fetch_all_to(\"data\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "icecube_dev",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.20"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}