intro_pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data analysis and visualization with `scikit` and `matplotlib`\n",
    "-----\n",
    "\n",
    "Created by: @vichawans\n",
    "\n",
    "Adapted from: [Project Pythia](https://foundations.projectpythia.org/core/pandas/pandas.html)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports\n",
    "\n",
    "You will often see the nickname `pd` used as an abbreviation for pandas in the import statement, just like `numpy` is often imported as `np`. We also import the `DATASETS` class from `pythia_datasets`, which allows us to use example datasets created for Pythia."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The pandas [`DataFrame`](https://pandas.pydata.org/docs/user_guide/dsintro.html#dataframe)...\n",
    "...is a **labeled**, two-dimensional columnar structure, similar to a table, spreadsheet, or the R `data.frame`.\n",
    "\n",
    "![dataframe schematic](https://github.com/pandas-dev/pandas/raw/main/doc/source/_static/schemas/01_table_dataframe.svg \"Schematic of a pandas DataFrame\")\n",
    "\n",
    "The `columns` that make up our `DataFrame` can be lists, dictionaries, NumPy arrays, pandas `Series`, or many other data types not mentioned here. Within these `columns`, you can have data values of many different data types used in Python and NumPy, including text, numbers, and dates/times. The first column of a `DataFrame`, shown in the image above in dark gray, is uniquely referred to as an `index`; this column contains information characterizing each row of our `DataFrame`. Similar to any other `column`, the `index` can label rows by text, numbers, datetime objects, and many other data types. Datetime objects are a quite popular way to label rows.\n",
    "\n",
    "For our first example using Pandas DataFrames, we start by reading in some data in comma-separated value (`.csv`) format. We retrieve this dataset from the Pythia DATASETS class (imported at the top of this page); however, the dataset was originally contained within the NCDC teleconnections database. This dataset contains many types of geoscientific data, including El Nino/Southern Oscillation indices. For more information on this dataset, review the description [here](https://www.ncdc.noaa.gov/teleconnections/enso/indicators/sst/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-09-26T01:47:27.588098400Z",
     "start_time": "2023-09-26T01:47:27.096383200Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading: 'https://github.com/ProjectPythia/pythia-datasets/raw/main/data/enso_data.csv' to E:\\PycharmProjects\\ai4er\\jupyter-workshop\\data\\enso_data.csv\n"
     ]
    },
    {
     "data": {
      "text/plain": "[1/1] Requesting enso_data.csv:   0%|          | 0.00B/26.6kB [00:00<?, ?B/s]",
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "93f1990e25584665b4115a2ef8df0e16"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloaded in E:\\PycharmProjects\\ai4er\\jupyter-workshop\\data\\enso_data.csv\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "from utils import guarantee_existence, downloader\n",
    "\n",
    "url = [\"https://github.com/ProjectPythia/pythia-datasets/raw/main/data/enso_data.csv\"]\n",
    "data_dir = guarantee_existence(os.path.abspath(\"./data\"))\n",
    "downloader(url, target_dir=data_dir)\n",
    "\n",
    "filepath = os.path.join(data_dir, \"NARR_19930313_0000.nc\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have a valid path to a data file that Pandas knows how to read, we can open it, as shown in the following example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(filepath)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>datetime</th>\n",
       "      <th>Nino12</th>\n",
       "      <th>Nino12anom</th>\n",
       "      <th>Nino3</th>\n",
       "      <th>Nino3anom</th>\n",
       "      <th>Nino4</th>\n",
       "      <th>Nino4anom</th>\n",
       "      <th>Nino34</th>\n",
       "      <th>Nino34anom</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1982-01-01</td>\n",
       "      <td>24.29</td>\n",
       "      <td>-0.17</td>\n",
       "      <td>25.87</td>\n",
       "      <td>0.24</td>\n",
       "      <td>28.30</td>\n",
       "      <td>0.00</td>\n",
       "      <td>26.72</td>\n",
       "      <td>0.15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1982-02-01</td>\n",
       "      <td>25.49</td>\n",
       "      <td>-0.58</td>\n",
       "      <td>26.38</td>\n",
       "      <td>0.01</td>\n",
       "      <td>28.21</td>\n",
       "      <td>0.11</td>\n",
       "      <td>26.70</td>\n",
       "      <td>-0.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1982-03-01</td>\n",
       "      <td>25.21</td>\n",
       "      <td>-1.31</td>\n",
       "      <td>26.98</td>\n",
       "      <td>-0.16</td>\n",
       "      <td>28.41</td>\n",
       "      <td>0.22</td>\n",
       "      <td>27.20</td>\n",
       "      <td>-0.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1982-04-01</td>\n",
       "      <td>24.50</td>\n",
       "      <td>-0.97</td>\n",
       "      <td>27.68</td>\n",
       "      <td>0.18</td>\n",
       "      <td>28.92</td>\n",
       "      <td>0.42</td>\n",
       "      <td>28.02</td>\n",
       "      <td>0.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1982-05-01</td>\n",
       "      <td>23.97</td>\n",
       "      <td>-0.23</td>\n",
       "      <td>27.79</td>\n",
       "      <td>0.71</td>\n",
       "      <td>29.49</td>\n",
       "      <td>0.70</td>\n",
       "      <td>28.54</td>\n",
       "      <td>0.69</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>467</th>\n",
       "      <td>2020-12-01</td>\n",
       "      <td>22.16</td>\n",
       "      <td>-0.60</td>\n",
       "      <td>24.38</td>\n",
       "      <td>-0.83</td>\n",
       "      <td>27.65</td>\n",
       "      <td>-0.95</td>\n",
       "      <td>25.53</td>\n",
       "      <td>-1.12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>468</th>\n",
       "      <td>2021-01-01</td>\n",
       "      <td>23.89</td>\n",
       "      <td>-0.64</td>\n",
       "      <td>25.06</td>\n",
       "      <td>-0.55</td>\n",
       "      <td>27.10</td>\n",
       "      <td>-1.25</td>\n",
       "      <td>25.58</td>\n",
       "      <td>-0.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>469</th>\n",
       "      <td>2021-02-01</td>\n",
       "      <td>25.55</td>\n",
       "      <td>-0.66</td>\n",
       "      <td>25.80</td>\n",
       "      <td>-0.57</td>\n",
       "      <td>27.20</td>\n",
       "      <td>-1.00</td>\n",
       "      <td>25.81</td>\n",
       "      <td>-0.92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>470</th>\n",
       "      <td>2021-03-01</td>\n",
       "      <td>26.48</td>\n",
       "      <td>-0.26</td>\n",
       "      <td>26.80</td>\n",
       "      <td>-0.39</td>\n",
       "      <td>27.79</td>\n",
       "      <td>-0.55</td>\n",
       "      <td>26.75</td>\n",
       "      <td>-0.51</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>471</th>\n",
       "      <td>2021-04-01</td>\n",
       "      <td>24.89</td>\n",
       "      <td>-0.80</td>\n",
       "      <td>26.96</td>\n",
       "      <td>-0.65</td>\n",
       "      <td>28.47</td>\n",
       "      <td>-0.21</td>\n",
       "      <td>27.40</td>\n",
       "      <td>-0.49</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>472 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       datetime  Nino12  Nino12anom  Nino3  Nino3anom  Nino4  Nino4anom  \\\n",
       "0    1982-01-01   24.29       -0.17  25.87       0.24  28.30       0.00   \n",
       "1    1982-02-01   25.49       -0.58  26.38       0.01  28.21       0.11   \n",
       "2    1982-03-01   25.21       -1.31  26.98      -0.16  28.41       0.22   \n",
       "3    1982-04-01   24.50       -0.97  27.68       0.18  28.92       0.42   \n",
       "4    1982-05-01   23.97       -0.23  27.79       0.71  29.49       0.70   \n",
       "..          ...     ...         ...    ...        ...    ...        ...   \n",
       "467  2020-12-01   22.16       -0.60  24.38      -0.83  27.65      -0.95   \n",
       "468  2021-01-01   23.89       -0.64  25.06      -0.55  27.10      -1.25   \n",
       "469  2021-02-01   25.55       -0.66  25.80      -0.57  27.20      -1.00   \n",
       "470  2021-03-01   26.48       -0.26  26.80      -0.39  27.79      -0.55   \n",
       "471  2021-04-01   24.89       -0.80  26.96      -0.65  28.47      -0.21   \n",
       "\n",
       "     Nino34  Nino34anom  \n",
       "0     26.72        0.15  \n",
       "1     26.70       -0.02  \n",
       "2     27.20       -0.02  \n",
       "3     28.02        0.24  \n",
       "4     28.54        0.69  \n",
       "..      ...         ...  \n",
       "467   25.53       -1.12  \n",
       "468   25.58       -0.99  \n",
       "469   25.81       -0.92  \n",
       "470   26.75       -0.51  \n",
       "471   27.40       -0.49  \n",
       "\n",
       "[472 rows x 9 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the moment, the index column of our `DataFrame` is not very helpful for humans. However, Pandas has clever ways to make index columns more human-readable. The next example demonstrates how to use optional keyword arguments to convert `DataFrame` index IDs to a human-friendly datetime format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Nino12</th>\n",
       "      <th>Nino12anom</th>\n",
       "      <th>Nino3</th>\n",
       "      <th>Nino3anom</th>\n",
       "      <th>Nino4</th>\n",
       "      <th>Nino4anom</th>\n",
       "      <th>Nino34</th>\n",
       "      <th>Nino34anom</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>datetime</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1982-01-01</th>\n",
       "      <td>24.29</td>\n",
       "      <td>-0.17</td>\n",
       "      <td>25.87</td>\n",
       "      <td>0.24</td>\n",
       "      <td>28.30</td>\n",
       "      <td>0.00</td>\n",
       "      <td>26.72</td>\n",
       "      <td>0.15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1982-02-01</th>\n",
       "      <td>25.49</td>\n",
       "      <td>-0.58</td>\n",
       "      <td>26.38</td>\n",
       "      <td>0.01</td>\n",
       "      <td>28.21</td>\n",
       "      <td>0.11</td>\n",
       "      <td>26.70</td>\n",
       "      <td>-0.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1982-03-01</th>\n",
       "      <td>25.21</td>\n",
       "      <td>-1.31</td>\n",
       "      <td>26.98</td>\n",
       "      <td>-0.16</td>\n",
       "      <td>28.41</td>\n",
       "      <td>0.22</td>\n",
       "      <td>27.20</td>\n",
       "      <td>-0.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1982-04-01</th>\n",
       "      <td>24.50</td>\n",
       "      <td>-0.97</td>\n",
       "      <td>27.68</td>\n",
       "      <td>0.18</td>\n",
       "      <td>28.92</td>\n",
       "      <td>0.42</td>\n",
       "      <td>28.02</td>\n",
       "      <td>0.24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1982-05-01</th>\n",
       "      <td>23.97</td>\n",
       "      <td>-0.23</td>\n",
       "      <td>27.79</td>\n",
       "      <td>0.71</td>\n",
       "      <td>29.49</td>\n",
       "      <td>0.70</td>\n",
       "      <td>28.54</td>\n",
       "      <td>0.69</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2020-12-01</th>\n",
       "      <td>22.16</td>\n",
       "      <td>-0.60</td>\n",
       "      <td>24.38</td>\n",
       "      <td>-0.83</td>\n",
       "      <td>27.65</td>\n",
       "      <td>-0.95</td>\n",
       "      <td>25.53</td>\n",
       "      <td>-1.12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-01-01</th>\n",
       "      <td>23.89</td>\n",
       "      <td>-0.64</td>\n",
       "      <td>25.06</td>\n",
       "      <td>-0.55</td>\n",
       "      <td>27.10</td>\n",
       "      <td>-1.25</td>\n",
       "      <td>25.58</td>\n",
       "      <td>-0.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-02-01</th>\n",
       "      <td>25.55</td>\n",
       "      <td>-0.66</td>\n",
       "      <td>25.80</td>\n",
       "      <td>-0.57</td>\n",
       "      <td>27.20</td>\n",
       "      <td>-1.00</td>\n",
       "      <td>25.81</td>\n",
       "      <td>-0.92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-03-01</th>\n",
       "      <td>26.48</td>\n",
       "      <td>-0.26</td>\n",
       "      <td>26.80</td>\n",
       "      <td>-0.39</td>\n",
       "      <td>27.79</td>\n",
       "      <td>-0.55</td>\n",
       "      <td>26.75</td>\n",
       "      <td>-0.51</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-04-01</th>\n",
       "      <td>24.89</td>\n",
       "      <td>-0.80</td>\n",
       "      <td>26.96</td>\n",
       "      <td>-0.65</td>\n",
       "      <td>28.47</td>\n",
       "      <td>-0.21</td>\n",
       "      <td>27.40</td>\n",
       "      <td>-0.49</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>472 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            Nino12  Nino12anom  Nino3  Nino3anom  Nino4  Nino4anom  Nino34  \\\n",
       "datetime                                                                     \n",
       "1982-01-01   24.29       -0.17  25.87       0.24  28.30       0.00   26.72   \n",
       "1982-02-01   25.49       -0.58  26.38       0.01  28.21       0.11   26.70   \n",
       "1982-03-01   25.21       -1.31  26.98      -0.16  28.41       0.22   27.20   \n",
       "1982-04-01   24.50       -0.97  27.68       0.18  28.92       0.42   28.02   \n",
       "1982-05-01   23.97       -0.23  27.79       0.71  29.49       0.70   28.54   \n",
       "...            ...         ...    ...        ...    ...        ...     ...   \n",
       "2020-12-01   22.16       -0.60  24.38      -0.83  27.65      -0.95   25.53   \n",
       "2021-01-01   23.89       -0.64  25.06      -0.55  27.10      -1.25   25.58   \n",
       "2021-02-01   25.55       -0.66  25.80      -0.57  27.20      -1.00   25.81   \n",
       "2021-03-01   26.48       -0.26  26.80      -0.39  27.79      -0.55   26.75   \n",
       "2021-04-01   24.89       -0.80  26.96      -0.65  28.47      -0.21   27.40   \n",
       "\n",
       "            Nino34anom  \n",
       "datetime                \n",
       "1982-01-01        0.15  \n",
       "1982-02-01       -0.02  \n",
       "1982-03-01       -0.02  \n",
       "1982-04-01        0.24  \n",
       "1982-05-01        0.69  \n",
       "...                ...  \n",
       "2020-12-01       -1.12  \n",
       "2021-01-01       -0.99  \n",
       "2021-02-01       -0.92  \n",
       "2021-03-01       -0.51  \n",
       "2021-04-01       -0.49  \n",
       "\n",
       "[472 rows x 8 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(filepath, index_col=0, parse_dates=True)\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatetimeIndex(['1982-01-01', '1982-02-01', '1982-03-01', '1982-04-01',\n",
       "               '1982-05-01', '1982-06-01', '1982-07-01', '1982-08-01',\n",
       "               '1982-09-01', '1982-10-01',\n",
       "               ...\n",
       "               '2020-07-01', '2020-08-01', '2020-09-01', '2020-10-01',\n",
       "               '2020-11-01', '2020-12-01', '2021-01-01', '2021-02-01',\n",
       "               '2021-03-01', '2021-04-01'],\n",
       "              dtype='datetime64[ns]', name='datetime', length=472, freq=None)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each of our data rows is now helpfully labeled by a datetime-object-like index value; this means that we can now easily identify data values not only by named columns, but also by date labels on rows. This is a sneak preview of the `DatetimeIndex` functionality of Pandas; this functionality enables a large portion of Pandas' timeseries-related usage. Don't worry; `DatetimeIndex` will be discussed in full detail later on this page. In the meantime, let's look at the columns of data read in from the `.csv` file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Nino12', 'Nino12anom', 'Nino3', 'Nino3anom', 'Nino4', 'Nino4anom',\n",
       "       'Nino34', 'Nino34anom'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The pandas [`Series`](https://pandas.pydata.org/docs/user_guide/dsintro.html#series)...\n",
    "\n",
    "...is essentially any one of the columns of our `DataFrame`. A `Series` also includes the index column from the source `DataFrame`, in order to provide a label for each value in the `Series`.\n",
    "\n",
    "![pandas Series](https://github.com/pandas-dev/pandas/raw/main/doc/source/_static/schemas/01_table_series.svg \"Schematic of a pandas Series\")\n",
    "\n",
    "The pandas `Series` is a fast and capable 1-dimensional array of nearly any data type we could want, and it can behave very similarly to a NumPy `ndarray` or a Python `dict`. You can take a look at any of the `Series` that make up your `DataFrame`, either by using its column name and the Python `dict` notation, or by using dot-shorthand with the column name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime\n",
       "1982-01-01    26.72\n",
       "1982-02-01    26.70\n",
       "1982-03-01    27.20\n",
       "1982-04-01    28.02\n",
       "1982-05-01    28.54\n",
       "              ...  \n",
       "2020-12-01    25.53\n",
       "2021-01-01    25.58\n",
       "2021-02-01    25.81\n",
       "2021-03-01    26.75\n",
       "2021-04-01    27.40\n",
       "Name: Nino34, Length: 472, dtype: float64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"Nino34\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>Tip:</b> You can also use the dot notation illustrated below to specify a column name, but this syntax is mostly provided for convenience. For the most part, this notation is interchangeable with the dictionary notation; however, if the column name is not a valid Python identifier (e.g., it starts with a number or space), you cannot use dot notation.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime\n",
       "1982-01-01    26.72\n",
       "1982-02-01    26.70\n",
       "1982-03-01    27.20\n",
       "1982-04-01    28.02\n",
       "1982-05-01    28.54\n",
       "              ...  \n",
       "2020-12-01    25.53\n",
       "2021-01-01    25.58\n",
       "2021-02-01    25.81\n",
       "2021-03-01    26.75\n",
       "2021-04-01    27.40\n",
       "Name: Nino34, Length: 472, dtype: float64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.Nino34"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Slicing and Dicing the `DataFrame` and `Series`\n",
    "\n",
    "In this section, we will expand on topics covered in the previous sections on this page. One of the most important concepts to learn about Pandas is that it allows you to _**access anything by its associated label**_, regardless of data organization structure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indexing a `Series`\n",
    "\n",
    "As a review of previous examples, we'll start our next example by pulling a `Series` out of our `DataFrame` using its column label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime\n",
       "1982-01-01    26.72\n",
       "1982-02-01    26.70\n",
       "1982-03-01    27.20\n",
       "1982-04-01    28.02\n",
       "1982-05-01    28.54\n",
       "              ...  \n",
       "2020-12-01    25.53\n",
       "2021-01-01    25.58\n",
       "2021-02-01    25.81\n",
       "2021-03-01    26.75\n",
       "2021-04-01    27.40\n",
       "Name: Nino34, Length: 472, dtype: float64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nino34_series = df[\"Nino34\"]\n",
    "\n",
    "nino34_series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use syntax similar to that of NumPy `ndarrays` to index, select, and subset with Pandas `Series`, as shown in this example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "28.02"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nino34_series[3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also use labels alongside Python dictionary syntax to perform the same operations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "28.02"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nino34_series[\"1982-04-01\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can probably figure out some ways to extend these indexing methods, as shown in the following examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime\n",
       "1982-01-01    26.72\n",
       "1982-02-01    26.70\n",
       "1982-03-01    27.20\n",
       "1982-04-01    28.02\n",
       "1982-05-01    28.54\n",
       "1982-06-01    28.75\n",
       "1982-07-01    28.10\n",
       "1982-08-01    27.93\n",
       "1982-09-01    28.11\n",
       "1982-10-01    28.64\n",
       "1982-11-01    28.81\n",
       "1982-12-01    29.21\n",
       "Name: Nino34, dtype: float64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nino34_series[0:12]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition alert alert-info\">\n",
    "    <p class=\"admonition-title\" style=\"font-weight:bold\">Info</p>\n",
    "    Index-based slices are <b>exclusive</b> of the final value, similar to Python's usual indexing rules.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, there are many more ways to index a `Series`. The following example shows a powerful and useful indexing method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime\n",
       "1982-01-01    26.72\n",
       "1982-02-01    26.70\n",
       "1982-03-01    27.20\n",
       "1982-04-01    28.02\n",
       "1982-05-01    28.54\n",
       "1982-06-01    28.75\n",
       "1982-07-01    28.10\n",
       "1982-08-01    27.93\n",
       "1982-09-01    28.11\n",
       "1982-10-01    28.64\n",
       "1982-11-01    28.81\n",
       "1982-12-01    29.21\n",
       "Name: Nino34, dtype: float64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nino34_series[\"1982-01-01\":\"1982-12-01\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is an example of label-based slicing. With label-based slicing, Pandas will automatically find a range of values based on the labels you specify."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition alert alert-info\">\n",
    "    <p class=\"admonition-title\" style=\"font-weight:bold\">Info</p>\n",
    "    As opposed to index-based slices, label-based slices are <b>inclusive</b> of the final value.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you already have some knowledge of xarray, you will quite likely know how to create `slice` objects by hand. This can also be used in pandas, as shown below.  If you are completely unfamiliar with xarray, it will be covered on a [later Pythia tutorial page](../xarray)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime\n",
       "1982-01-01    26.72\n",
       "1982-02-01    26.70\n",
       "1982-03-01    27.20\n",
       "1982-04-01    28.02\n",
       "1982-05-01    28.54\n",
       "1982-06-01    28.75\n",
       "1982-07-01    28.10\n",
       "1982-08-01    27.93\n",
       "1982-09-01    28.11\n",
       "1982-10-01    28.64\n",
       "1982-11-01    28.81\n",
       "1982-12-01    29.21\n",
       "Name: Nino34, dtype: float64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nino34_series[slice(\"1982-01-01\", \"1982-12-01\")]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using `.iloc` and `.loc` to index\n",
    "\n",
    "In this section, we introduce ways to access data that are preferred by Pandas over the methods listed above. When accessing by label, it is preferred to use the `.loc` method, and when accessing by index, the `.iloc` method is preferred. These methods behave similarly to the notation introduced above, but provide more speed, security, and rigor in your value selection. Using these methods can also help you avoid [chained assignment warnings](https://pandas.pydata.org/docs/user_guide/indexing.html#returning-a-view-versus-a-copy) generated by pandas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "28.02"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nino34_series.iloc[3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime\n",
       "1982-01-01    26.72\n",
       "1982-02-01    26.70\n",
       "1982-03-01    27.20\n",
       "1982-04-01    28.02\n",
       "1982-05-01    28.54\n",
       "1982-06-01    28.75\n",
       "1982-07-01    28.10\n",
       "1982-08-01    27.93\n",
       "1982-09-01    28.11\n",
       "1982-10-01    28.64\n",
       "1982-11-01    28.81\n",
       "1982-12-01    29.21\n",
       "Name: Nino34, dtype: float64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nino34_series.iloc[0:12]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "28.02"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nino34_series.loc[\"1982-04-01\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime\n",
       "1982-01-01    26.72\n",
       "1982-02-01    26.70\n",
       "1982-03-01    27.20\n",
       "1982-04-01    28.02\n",
       "1982-05-01    28.54\n",
       "1982-06-01    28.75\n",
       "1982-07-01    28.10\n",
       "1982-08-01    27.93\n",
       "1982-09-01    28.11\n",
       "1982-10-01    28.64\n",
       "1982-11-01    28.81\n",
       "1982-12-01    29.21\n",
       "Name: Nino34, dtype: float64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nino34_series.loc[\"1982-01-01\":\"1982-12-01\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extending to the `DataFrame`\n",
    "\n",
    "These subsetting capabilities can also be used in a full `DataFrame`; however, if you use the same syntax, there are issues, as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyError",
     "evalue": "'1982-01-01'",
     "output_type": "error",
     "traceback": [
      "\u001B[0;31m---------------------------------------------------------------------------\u001B[0m",
      "\u001B[0;31mKeyError\u001B[0m                                  Traceback (most recent call last)",
      "File \u001B[0;32m~/anaconda3/envs/cliaer-env/lib/python3.11/site-packages/pandas/core/indexes/base.py:3802\u001B[0m, in \u001B[0;36mIndex.get_loc\u001B[0;34m(self, key, method, tolerance)\u001B[0m\n\u001B[1;32m   3801\u001B[0m \u001B[39mtry\u001B[39;00m:\n\u001B[0;32m-> 3802\u001B[0m     \u001B[39mreturn\u001B[39;00m \u001B[39mself\u001B[39m\u001B[39m.\u001B[39m_engine\u001B[39m.\u001B[39mget_loc(casted_key)\n\u001B[1;32m   3803\u001B[0m \u001B[39mexcept\u001B[39;00m \u001B[39mKeyError\u001B[39;00m \u001B[39mas\u001B[39;00m err:\n",
      "File \u001B[0;32m~/anaconda3/envs/cliaer-env/lib/python3.11/site-packages/pandas/_libs/index.pyx:138\u001B[0m, in \u001B[0;36mpandas._libs.index.IndexEngine.get_loc\u001B[0;34m()\u001B[0m\n",
      "File \u001B[0;32m~/anaconda3/envs/cliaer-env/lib/python3.11/site-packages/pandas/_libs/index.pyx:165\u001B[0m, in \u001B[0;36mpandas._libs.index.IndexEngine.get_loc\u001B[0;34m()\u001B[0m\n",
      "File \u001B[0;32mpandas/_libs/hashtable_class_helper.pxi:5745\u001B[0m, in \u001B[0;36mpandas._libs.hashtable.PyObjectHashTable.get_item\u001B[0;34m()\u001B[0m\n",
      "File \u001B[0;32mpandas/_libs/hashtable_class_helper.pxi:5753\u001B[0m, in \u001B[0;36mpandas._libs.hashtable.PyObjectHashTable.get_item\u001B[0;34m()\u001B[0m\n",
      "\u001B[0;31mKeyError\u001B[0m: '1982-01-01'",
      "\nThe above exception was the direct cause of the following exception:\n",
      "\u001B[0;31mKeyError\u001B[0m                                  Traceback (most recent call last)",
      "\u001B[1;32m/media/vichawan/data/induction-jupyter/main_afternoon3_pandas.ipynb Cell 41\u001B[0m line \u001B[0;36m1\n\u001B[0;32m----> <a href='vscode-notebook-cell:/media/vichawan/data/induction-jupyter/main_afternoon3_pandas.ipynb#X64sZmlsZQ%3D%3D?line=0'>1</a>\u001B[0m df[\u001B[39m\"\u001B[39m\u001B[39m1982-01-01\u001B[39m\u001B[39m\"\u001B[39m]\n",
      "File \u001B[0;32m~/anaconda3/envs/cliaer-env/lib/python3.11/site-packages/pandas/core/frame.py:3807\u001B[0m, in \u001B[0;36mDataFrame.__getitem__\u001B[0;34m(self, key)\u001B[0m\n\u001B[1;32m   3805\u001B[0m \u001B[39mif\u001B[39;00m \u001B[39mself\u001B[39m\u001B[39m.\u001B[39mcolumns\u001B[39m.\u001B[39mnlevels \u001B[39m>\u001B[39m \u001B[39m1\u001B[39m:\n\u001B[1;32m   3806\u001B[0m     \u001B[39mreturn\u001B[39;00m \u001B[39mself\u001B[39m\u001B[39m.\u001B[39m_getitem_multilevel(key)\n\u001B[0;32m-> 3807\u001B[0m indexer \u001B[39m=\u001B[39m \u001B[39mself\u001B[39m\u001B[39m.\u001B[39mcolumns\u001B[39m.\u001B[39mget_loc(key)\n\u001B[1;32m   3808\u001B[0m \u001B[39mif\u001B[39;00m is_integer(indexer):\n\u001B[1;32m   3809\u001B[0m     indexer \u001B[39m=\u001B[39m [indexer]\n",
      "File \u001B[0;32m~/anaconda3/envs/cliaer-env/lib/python3.11/site-packages/pandas/core/indexes/base.py:3804\u001B[0m, in \u001B[0;36mIndex.get_loc\u001B[0;34m(self, key, method, tolerance)\u001B[0m\n\u001B[1;32m   3802\u001B[0m     \u001B[39mreturn\u001B[39;00m \u001B[39mself\u001B[39m\u001B[39m.\u001B[39m_engine\u001B[39m.\u001B[39mget_loc(casted_key)\n\u001B[1;32m   3803\u001B[0m \u001B[39mexcept\u001B[39;00m \u001B[39mKeyError\u001B[39;00m \u001B[39mas\u001B[39;00m err:\n\u001B[0;32m-> 3804\u001B[0m     \u001B[39mraise\u001B[39;00m \u001B[39mKeyError\u001B[39;00m(key) \u001B[39mfrom\u001B[39;00m \u001B[39merr\u001B[39;00m\n\u001B[1;32m   3805\u001B[0m \u001B[39mexcept\u001B[39;00m \u001B[39mTypeError\u001B[39;00m:\n\u001B[1;32m   3806\u001B[0m     \u001B[39m# If we have a listlike key, _check_indexing_error will raise\u001B[39;00m\n\u001B[1;32m   3807\u001B[0m     \u001B[39m#  InvalidIndexError. Otherwise we fall through and re-raise\u001B[39;00m\n\u001B[1;32m   3808\u001B[0m     \u001B[39m#  the TypeError.\u001B[39;00m\n\u001B[1;32m   3809\u001B[0m     \u001B[39mself\u001B[39m\u001B[39m.\u001B[39m_check_indexing_error(key)\n",
      "\u001B[0;31mKeyError\u001B[0m: '1982-01-01'"
     ]
    }
   ],
   "source": [
    "df[\"1982-01-01\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition alert alert-danger\">\n",
    "    <p class=\"admonition-title\" style=\"font-weight:bold\">Danger</p>\n",
    "    Attempting to use <code>Series</code> subsetting with a <code>DataFrame</code> can crash your program. A proper way to subset a <code>DataFrame</code> is shown below.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When indexing a `DataFrame`, pandas will not assume as readily the intention of your code. In this case, using a row label by itself will not work; **with `DataFrames`, labels are used for identifying columns**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "datetime\n",
       "1982-01-01    26.72\n",
       "1982-02-01    26.70\n",
       "1982-03-01    27.20\n",
       "1982-04-01    28.02\n",
       "1982-05-01    28.54\n",
       "              ...  \n",
       "2020-12-01    25.53\n",
       "2021-01-01    25.58\n",
       "2021-02-01    25.81\n",
       "2021-03-01    26.75\n",
       "2021-04-01    27.40\n",
       "Name: Nino34, Length: 472, dtype: float64"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"Nino34\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As shown below, you also cannot subset columns in a `DataFrame` using integer indices:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyError",
     "evalue": "0",
     "output_type": "error",
     "traceback": [
      "\u001B[0;31m---------------------------------------------------------------------------\u001B[0m",
      "\u001B[0;31mKeyError\u001B[0m                                  Traceback (most recent call last)",
      "File \u001B[0;32m~/anaconda3/envs/cliaer-env/lib/python3.11/site-packages/pandas/core/indexes/base.py:3802\u001B[0m, in \u001B[0;36mIndex.get_loc\u001B[0;34m(self, key, method, tolerance)\u001B[0m\n\u001B[1;32m   3801\u001B[0m \u001B[39mtry\u001B[39;00m:\n\u001B[0;32m-> 3802\u001B[0m     \u001B[39mreturn\u001B[39;00m \u001B[39mself\u001B[39m\u001B[39m.\u001B[39m_engine\u001B[39m.\u001B[39mget_loc(casted_key)\n\u001B[1;32m   3803\u001B[0m \u001B[39mexcept\u001B[39;00m \u001B[39mKeyError\u001B[39;00m \u001B[39mas\u001B[39;00m err:\n",
      "File \u001B[0;32m~/anaconda3/envs/cliaer-env/lib/python3.11/site-packages/pandas/_libs/index.pyx:138\u001B[0m, in \u001B[0;36mpandas._libs.index.IndexEngine.get_loc\u001B[0;34m()\u001B[0m\n",
      "File \u001B[0;32m~/anaconda3/envs/cliaer-env/lib/python3.11/site-packages/pandas/_libs/index.pyx:165\u001B[0m, in \u001B[0;36mpandas._libs.index.IndexEngine.get_loc\u001B[0;34m()\u001B[0m\n",
      "File \u001B[0;32mpandas/_libs/hashtable_class_helper.pxi:5745\u001B[0m, in \u001B[0;36mpandas._libs.hashtable.PyObjectHashTable.get_item\u001B[0;34m()\u001B[0m\n",
      "File \u001B[0;32mpandas/_libs/hashtable_class_helper.pxi:5753\u001B[0m, in \u001B[0;36mpandas._libs.hashtable.PyObjectHashTable.get_item\u001B[0;34m()\u001B[0m\n",
      "\u001B[0;31mKeyError\u001B[0m: 0",
      "\nThe above exception was the direct cause of the following exception:\n",
      "\u001B[0;31mKeyError\u001B[0m                                  Traceback (most recent call last)",
      "\u001B[1;32m/media/vichawan/data/induction-jupyter/main_afternoon3_pandas.ipynb Cell 46\u001B[0m line \u001B[0;36m1\n\u001B[0;32m----> <a href='vscode-notebook-cell:/media/vichawan/data/induction-jupyter/main_afternoon3_pandas.ipynb#Y102sZmlsZQ%3D%3D?line=0'>1</a>\u001B[0m df[\u001B[39m0\u001B[39m]\n",
      "File \u001B[0;32m~/anaconda3/envs/cliaer-env/lib/python3.11/site-packages/pandas/core/frame.py:3807\u001B[0m, in \u001B[0;36mDataFrame.__getitem__\u001B[0;34m(self, key)\u001B[0m\n\u001B[1;32m   3805\u001B[0m \u001B[39mif\u001B[39;00m \u001B[39mself\u001B[39m\u001B[39m.\u001B[39mcolumns\u001B[39m.\u001B[39mnlevels \u001B[39m>\u001B[39m \u001B[39m1\u001B[39m:\n\u001B[1;32m   3806\u001B[0m     \u001B[39mreturn\u001B[39;00m \u001B[39mself\u001B[39m\u001B[39m.\u001B[39m_getitem_multilevel(key)\n\u001B[0;32m-> 3807\u001B[0m indexer \u001B[39m=\u001B[39m \u001B[39mself\u001B[39m\u001B[39m.\u001B[39mcolumns\u001B[39m.\u001B[39mget_loc(key)\n\u001B[1;32m   3808\u001B[0m \u001B[39mif\u001B[39;00m is_integer(indexer):\n\u001B[1;32m   3809\u001B[0m     indexer \u001B[39m=\u001B[39m [indexer]\n",
      "File \u001B[0;32m~/anaconda3/envs/cliaer-env/lib/python3.11/site-packages/pandas/core/indexes/base.py:3804\u001B[0m, in \u001B[0;36mIndex.get_loc\u001B[0;34m(self, key, method, tolerance)\u001B[0m\n\u001B[1;32m   3802\u001B[0m     \u001B[39mreturn\u001B[39;00m \u001B[39mself\u001B[39m\u001B[39m.\u001B[39m_engine\u001B[39m.\u001B[39mget_loc(casted_key)\n\u001B[1;32m   3803\u001B[0m \u001B[39mexcept\u001B[39;00m \u001B[39mKeyError\u001B[39;00m \u001B[39mas\u001B[39;00m err:\n\u001B[0;32m-> 3804\u001B[0m     \u001B[39mraise\u001B[39;00m \u001B[39mKeyError\u001B[39;00m(key) \u001B[39mfrom\u001B[39;00m \u001B[39merr\u001B[39;00m\n\u001B[1;32m   3805\u001B[0m \u001B[39mexcept\u001B[39;00m \u001B[39mTypeError\u001B[39;00m:\n\u001B[1;32m   3806\u001B[0m     \u001B[39m# If we have a listlike key, _check_indexing_error will raise\u001B[39;00m\n\u001B[1;32m   3807\u001B[0m     \u001B[39m#  InvalidIndexError. Otherwise we fall through and re-raise\u001B[39;00m\n\u001B[1;32m   3808\u001B[0m     \u001B[39m#  the TypeError.\u001B[39;00m\n\u001B[1;32m   3809\u001B[0m     \u001B[39mself\u001B[39m\u001B[39m.\u001B[39m_check_indexing_error(key)\n",
      "\u001B[0;31mKeyError\u001B[0m: 0"
     ]
    }
   ],
   "source": [
    "df[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From earlier examples, we know that we can use an index or label with a `DataFrame` to pull out a column as a `Series`, and we know that we can use an index or label with a `Series` to pull out a single value.  Therefore, by chaining brackets, we can pull any individual data value out of the `DataFrame`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "28.02"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"Nino34\"][\"1982-04-01\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "28.02"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"Nino34\"][3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, subsetting data using this chained-bracket technique is not preferred by Pandas. As described above, Pandas prefers us to use the `.loc` and `.iloc` methods for subsetting.  In addition, these methods provide a clearer, more efficient way to extract specific data from a `DataFrame`, as illustrated below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "28.02"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[\"1982-04-01\", \"Nino34\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition alert alert-info\">\n",
    "    <p class=\"admonition-title\" style=\"font-weight:bold\">Info</p>\n",
    "    When using this syntax to pull individual data values from a DataFrame, make sure to list the row first, and then the column.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `.loc` and `.iloc` methods also allow us to pull entire rows out of a `DataFrame`, as shown in these examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.loc[\"1982-04-01\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.loc[\"1982-01-01\":\"1982-12-01\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.iloc[3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.iloc[0:12]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next example, we illustrate how you can use slices of rows and lists of columns to create a smaller `DataFrame` out of an existing `DataFrame`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.loc[\n",
    "    \"1982-01-01\":\"1982-12-01\",  # slice of rows\n",
    "    [\"Nino12\", \"Nino3\", \"Nino4\", \"Nino34\"],  # list of columns\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition alert alert-info\">\n",
    "    <p class=\"admonition-title\" style=\"font-weight:bold\">Info</p>\n",
    "    There are certain limitations to these subsetting techniques. For more information on these limitations, as well as a comparison of <code>DataFrame</code> and <code>Series</code> indexing methods, see the <a href=\"https://pandas.pydata.org/docs/user_guide/indexing.html\">Pandas indexing documentation.</a>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want a specific statistic for only one column in the `DataFrame`, pull the column out of the `DataFrame` with dot notation, then call the statistic function (in this case, mean) on that column, as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.Nino34.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Subsetting Using the Datetime Column\n",
    "\n",
    "Slicing is a useful technique for subsetting a `DataFrame`, but there are also other options that can be equally useful. In this section, some of these additional techniques are covered.\n",
    "\n",
    "If your `DataFrame` uses datetime values for indices, you can select data from only one month using `df.index.month`. In this example, we specify the number 1, which only selects data from January."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uses the datetime column\n",
    "df[df.index.month == 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This example shows how to create a new column containing the month portion of the datetime index for each data row. The value returned by `df.index.month` is used to obtain the data for this new column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['month'] = df.index.month"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This next example illustrates how to use the new month column to calculate average monthly values over the other data columns. First, we use the `groupby` method to group the other columns by the month.  Second, we take the average (mean) to obtain the monthly averages. Finally, we plot the resulting data as a line plot by simply calling `plot()`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Investigating Extreme Values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you need to search for rows that meet a specific criterion, you can use **conditional indexing**.  In this example, we search for rows where the Nino34 anomaly value (`Nino34anom`) is greater than 2:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[df.Nino34anom > 2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This example shows how to use the `sort_values` method on a `DataFrame`. This method sorts values in a `DataFrame` by the column specified as an argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.sort_values('Nino34anom')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also reverse the ordering of the sort by specifying the `ascending` keyword argument as `False`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.sort_values('Nino34anom', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Resampling\n",
    "In these examples, we illustrate a process known as resampling. Using resampling, you can change the frequency of index data values, reducing so-called 'noise' in a data plot. This is especially useful when working with timeseries data; plots can be equally effective with resampled data in these cases. The resampling performed in these examples converts monthly values to yearly averages. This is performed by passing the value '1Y' to the `resample` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.Nino34.plot();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Applying operations to a DataFrame\n",
    "\n",
    "One of the most commonly used features in Pandas is the performing of calculations to multiple data values in a `DataFrame` simultaneously. Let's first look at a familiar concept: a function that converts single values.  The following example uses such a function to convert temperature values from degrees Celsius to Kelvin."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert_degc_to_kelvin(temperature_degc):\n",
    "    \"\"\"\n",
    "    Converts from degrees celsius to Kelvin\n",
    "    \"\"\"\n",
    "\n",
    "    return temperature_degc + 273.15"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert a single value\n",
    "convert_degc_to_kelvin(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following examples instead illustrate a new concept: using such functions with `DataFrames` and `Series`. For the first example, we start by creating a `Series`; in order to do so, we subset the `DataFrame` by the `Nino34` column. This has already been done earlier in this page; we do not need to create this `Series` again. We are using this particular `Series` for a reason: the data values are in degrees Celsius."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nino34_series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we look at a portion of an existing `DataFrame` column. Notice that this column portion is a Pandas `Series`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(df.Nino12[0:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As shown in the following example, each Pandas `Series` contains a representation of its data in numpy format. Therefore, it is possible to convert a Pandas `Series` into a numpy array; this is done using the `.values` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(df.Nino12.values[0:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This example illustrates how to use the temperature-conversion function defined above on a `Series` object. Just as calling the function with a single value returns a single value, calling the function on a `Series` object returns another `Series` object. The function performs the temperature conversion on each data value in the `Series`, and returns a `Series` with all values converted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "convert_degc_to_kelvin(nino34_series)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we call the `.values` method on the `Series` passed to the function, the `Series` is converted to a numpy array, as described above. The function then converts each value in the numpy array, and returns a new numpy array with all values sorted."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also use lambda function as well"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition alert alert-warning\">\n",
    "    <p class=\"admonition-title\" style=\"font-weight:bold\">Warning</p>\n",
    "    It is recommended to only convert <code>Series</code> to NumPy arrays when necessary; doing so removes the label information that enables much of the Pandas core functionality.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "convert_degc_to_kelvin(nino34_series.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As described above, when our temperature-conversion function accepts a `Series` as an argument, it returns a `Series`. We can directly assign this returned `Series` to a new column in our `DataFrame`, as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['Nino34_degK'] = convert_degc_to_kelvin(nino34_series)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.Nino34_degK"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this final example, we demonstrate the use of the `to_csv` method to save a `DataFrame` as a `.csv` file. This example also demonstrates the `read_csv` method, which reads `.csv` files into Pandas `DataFrames`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv('nino_analyzed_output.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.read_csv('nino_analyzed_output.csv', index_col=0, parse_dates=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, csv comes with a lot of problems if you have large dataset with large/small numbers. Another helpful format is 'pickle'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_pickle('nino_analyzed_output.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (adv) `polars` as a faster alternative to `pandas`\n",
    "\n",
    "[Polars](https://www.pola.rs/) is a lightning-fast DataFrame library for Rust and Python"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "cliaer-env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}