manipulate-summarize-plot-numpy-arrays.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Indexing For Numpy Arrays\n",
    "\n",
    "In the lessons on lists, you learned that `Python` indexing begins with `[0]`, and that you can use indexing to query the value of items within `Python` lists.\n",
    "\n",
    "You can also access elements (i.e. values) in `numpy arrays` using indexing. \n",
    "\n",
    "\n",
    "### One-dimensional Numpy Arrays\n",
    "\n",
    "For one-dimensional `numpy arrays`, you only need to specific one index value to access the elements in the `numpy array` (e.g.  `arrayname[index,]`). \n",
    "\n",
    "The example below is an one-dimensional array that has 3 elements, or values. \n",
    "\n",
    "```python\n",
    "avg_monthly_precip = numpy.array([0.70, 0.75, 1.85])\n",
    "```\n",
    "\n",
    "You can use `avg_monthly_precip[2,]` to get the third element in (`1.85`) from this one-dimensional `numpy array`. \n",
    "\n",
    "Recall that you are using use the index `[2]` for the third place because `Python` indexing begins with `[0]`, not with `[1]`.\n",
    "\n",
    "\n",
    "### Two-dimensional Numpy Arrays\n",
    "\n",
    "With two-dimensional arrays, you need to specify to both a row index and a column index. \n",
    "\n",
    "The example below is an two-dimensional array with 2 rows and 3 columns. \n",
    "\n",
    "```python\n",
    "precip_2002_2013 = numpy.array([[1.07, 0.44, 1.5],\n",
    "                              [0.27, 1.13, 1.72]])\n",
    "```\n",
    "\n",
    "You can use `precip_2002_2013[1, 2]` to get the element in the second row, third column (`1.72`) of this two-dimensional `numpy array`. \n",
    "\n",
    "Just like you saw for the one-dimensional `numpy array`, you use the index `[1,2]` for the second row and third column because `Python` indexing begins with `[0]`, not with `[1]`\n",
    "\n",
    "In this lesson, you will use indexing to select elements within one-dimensional and two-dimensional `numpy arrays`, and you will learn how to manipulate, summarize, and plot these `numpy arrays`.\n",
    "\n",
    "You will use the same datasets from the previous lesson on importing text data:\n",
    "\n",
    "* a .txt file containing the average monthly precipitation data for Boulder, CO\n",
    "* a .csv file containing the monthly precipitation for Boulder, CO for the years 2002 and 2013\n",
    "\n",
    "\n",
    "## Begin Writing Your Code\n",
    "\n",
    "###  Import Packages\n",
    "\n",
    "From the previous lesson, you have already learned how to import the necessary packages to set the working directory and download the needed datasets using the `os` and `urllib` packages.\n",
    "\n",
    "To work with `numpy arrays`, you will also need to import the `numpy` package with the alias `np`, and you will need to import the `matplotlib.pyplot` module with the alias `plt` to plot data. Begin by reviewing these tasks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import necessary Python packages\n",
    "import os\n",
    "import numpy as np\n",
    "import urllib.request\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# print message after packages imported successfully\n",
    "print(\"import of packages successful\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set Working Directory\n",
    "\n",
    "Remember that you can check the current working directory using `os.getcwd()` and set the current working directory using `os.chdir()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set the working directory to the `earth-analytics-bootcamp` directory\n",
    "# replace `jpalomino` with your username here and all paths in this lesson\n",
    "os.chdir(\"/home/jpalomino/earth-analytics-bootcamp/\")\n",
    "\n",
    "# For Windows users, you may have to set the directory using the full path that Windows maintains for your user. For example:\n",
    "# os.chdir(\"C:/Users/Jenny Palomino/earth-analytics-bootcamp/\")\n",
    "\n",
    "# For Mac Users, you may have to set the directory using the full path that Mac maintains for your user. For example:\n",
    "# os.chdir(\"/Users/jpalomino/earth-analytics-bootcamp/\")\n",
    "\n",
    "# print the current working directory\n",
    "os.getcwd()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download Data\n",
    "\n",
    "In the previous lesson, you used the `urllib` package to download data from the Earth Lab `Figshare.com` repository. You will use these same datasets in this lesson."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use `urllib` download files from Earth Lab figshare repository\n",
    "\n",
    "# download .txt containing monthly average precipitation for Boulder, CO\n",
    "urllib.request.urlretrieve(url = \"https://ndownloader.figshare.com/files/12565616\", \n",
    "                           filename = \"data/avg-monthly-precip.txt\")\n",
    "\n",
    "# download .txt containing month names\n",
    "urllib.request.urlretrieve(url = \"https://ndownloader.figshare.com/files/12565619\", \n",
    "                           filename = \"data/months.txt\")\n",
    "\n",
    "# download .csv containing monthly average precipitation for Boulder, CO\n",
    "urllib.request.urlretrieve(url = \"https://ndownloader.figshare.com/files/12707792\", \n",
    "                           filename = \"data/monthly-precip-2002-2013.csv\")\n",
    "\n",
    "\n",
    "# print message that data downloads were successful\n",
    "print(\"datasets downloaded successfully\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import Data Into Numpy Arrays\n",
    "\n",
    "You also already learned how to import data from text files into `numpy arrays`. Be sure to update the paths for the files to your home directory. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the monthly average values from `avg-monthly-precip.txt` as a numpy array\n",
    "avg_monthly_precip = np.loadtxt(fname = \"/home/jpalomino/earth-analytics-bootcamp/data/avg-monthly-precip.txt\")\n",
    "\n",
    "# import the names of the months from month.txt as a numpy array\n",
    "months = np.genfromtxt(\"/home/jpalomino/earth-analytics-bootcamp/data/months.txt\", dtype='str')\n",
    "\n",
    "# import the monthly average values from `monthly-precip-2002-2013.csv` as a numpy array\n",
    "precip_2002_2013 = np.loadtxt(fname= \"/home/jpalomino/earth-analytics-bootcamp/data/monthly-precip-2002-2013.csv\", delimiter = \",\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Describe Contents of Numpy Arrays\n",
    "\n",
    "To begin working with `numpy arrays`, it is helpful to get some more details about the contents of data, such as the number of rows and columns in the data. \n",
    "\n",
    "You can use `.shape` after the variable name of the `numpy array` (e.g. `variablename.shape`) to get its dimensions (i.e. number of rows and columns). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print the dimensions of months\n",
    "print(months.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use `.shape` to compare the dimensions of `avg_monthly_precip` versus `precip_2002_2013`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print the dimensions of avg_monthly_precip\n",
    "print(avg_monthly_precip.shape)\n",
    "\n",
    "# print the dimensions of precip_2002_2013\n",
    "print(precip_2002_2013.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output for `avg_monthly_precip` indicates that it is composed of 12 elements along one-dimension. In fact, this `numpy arrays` is one-dimensional, meaning that all values exist within a single vector or list. \n",
    "\n",
    "The output for `precip_2002_2013` indicates that it is composed of 2 rows and 12 columns. This is two-dimensional `numpy array` that has two observations - one for the year 2002 and another for the year 2013 - and 12 measurements for observation - one for each month of the year. \n",
    "\n",
    "\n",
    "## Use Indexing to Query Numpy Arrays\n",
    "\n",
    "### One-dimensional Numpy Arrays\n",
    "\n",
    "By listing the dimensions of `avg_monthly_precip` using `.shape`, you know that it contains 12 elements along one dimension (e.g. `[12,]`). \n",
    "\n",
    "As this `numpy array` is one-dimensional, you can leave the second parameter blank when use indexing to access elements in this `numpy array` (e.g. `precip[X,]`). \n",
    "\n",
    "For example, because indexing in `Python` begins with `[0]`, you can use the index `[11,]` to query the last element in `avg_monthly_precip`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select the last element in `avg_monthly_precip` using the index [11,]\n",
    "avg_monthly_precip[11,]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check what happens when you query for an index location that does not exist in the array, say the index `[12,]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# change the value below from 11 to 12 to check what happens when you query for an index location that does not exist\n",
    "avg_monthly_precip[11,]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using `.shape`, you also saw that `precip_2002_2013` has row count of 2 with a column count of 12. \n",
    "\n",
    "Because `precip_2002_2013` is a two-dimensional `numpy array`, you need to specify both a row index and a column index to select elements in the `numpy array` \n",
    "\n",
    "For example, because indexing in `Python` begins with `[0]`, you can use the index `[0,0]` to query the first element in `precip_2002_2013` (i.e. first row, first column)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select the element in the first row, first column in the array\n",
    "precip_2002_2013[0,0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or, use the index `[1,11]` to query the last element in `precip_2002_2013` (i.e. last row, last column)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select the element in the last row, last column\n",
    "precip_2002_2013[1,11]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Two-dimensional Numpy Arrays\n",
    "\n",
    "For two-dimensional `numpy arrays`, you can also use a series for the row index and/or column index to select multiple elements using the index structure `[rowindex : rowindex, columnindex : columnindex]`.\n",
    "\n",
    "The index structure is inclusive of the first index, but not the second index. \n",
    "\n",
    "For example, you can use the index `[0:1, 0:3]` to select the first row and the first three columns (again because `Python` indexing begins with `[0]`). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select the first row and the first three columns\n",
    "precip_2002_2013[0:1, 0:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you wanted to include the second row and fourth column, you would need to use the index `[0:2, 0:4]`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select the first two rows and the first four columns\n",
    "precip_2002_2013[0:2, 0:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also store selected data as a new `numpy array`. \n",
    "\n",
    "For example, you can create a new `numpy array` for the precipitation data in 2002 by selecting the first row of values from `precip_2002_2013`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select the first row and all twelve columns of monthly values\n",
    "precip_2002 = precip_2002_2013[0:1, 0:12]\n",
    "\n",
    "# print data in `precip_2002`\n",
    "precip_2002"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can check the `.shape` of the new array to see that it has remained a two-dimensional array, but it only has one row of data, not two like `precip_2002_2013`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print dimensions of `precip_2002`\n",
    "precip_2002.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run Calculations on Numpy Arrays\n",
    "\n",
    "`Numpy arrays` calculations highlight the major differences between `Python` lists and `numpy arrays`.\n",
    "\n",
    "Recall that in lessons on variables and lists, you created separate variables for each monthly average precipitation value to convert it to millimeters (e.g. `jan = 0.70 * 25.4`), and then you created a new list containing all of these converted monthly values. \n",
    "\n",
    "`Numpy arrays` make it easy to run calculations on data as needed, while `Python` lists do not support these kinds of calculations. \n",
    "\n",
    "`Numpy arrays` support mathematical operations on an element-by-element basis, meaning that you can actually run one operation (e.g. `* 25.4`) on the entire array with a single line of code. \n",
    "\n",
    "Review this primary difference betweens lists and `numpy arrays` below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uncomment the code below to run it. Note: this code will result in an error, as you cannot run this operation on a list\n",
    "#preciplist = [0.70, 0.75, 1.85, 2.93, 3.05, 2.02, 1.93, 1.62, 1.84, 1.31, 1.39, 0.84]\n",
    "#preciplist = preciplist * 25.4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print the values in the array `avg_monthly_precip`\n",
    "print(avg_monthly_precip)\n",
    "\n",
    "# multiply each element in the array `avg_monthly_precip` by 25.4\n",
    "# assign the results to a new array also called `avg_monthly_precip`\n",
    "avg_monthly_precip = avg_monthly_precip * 25.4\n",
    "\n",
    "# print the values in the new array `avg_monthly_precip`\n",
    "print(avg_monthly_precip)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See how easy these calculations can be with `numpy arrays`! These arithmetic calculations will work on any `numpy array`, including multi-dimensional `numpy arrays`. \n",
    "\n",
    "Recall the previous lessons on variables and lists. Instead of creating separate variables for each month to run these calculations, you can now create a single `numpy array` imported from `avg-monthly-precip.txt` and run a single multiplication operation on the entire `numpy array` to the convert the values from inches to millimeters. \n",
    "\n",
    "\n",
    "## Summarize Data in Numpy Arrays\n",
    "\n",
    "Another great feature of `numpy arrays` is the ability to run summary statistics (e.g. calculating averages, finding min or max values) across the entire array of values. `Lists` do not support this functionality either.\n",
    "\n",
    "For example, you can use the `mean()` function in `numpy` to calculate the average value across an array (e.g. `np.mean(arrayname)`). You can also store results as a new variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate the mean and store the result as a new variable\n",
    "mean_avg_precip = np.mean(avg_monthly_precip)\n",
    "\n",
    "# you can expand the print statement to include a text string to label the data output\n",
    "print(\"mean of average monthly precipitation:\", mean_avg_precip)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, we can use `min()` and `max()` to find the minimum and maximum values in an array.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# find the min value and store the result as a new variable\n",
    "min_avg_precip = np.min(avg_monthly_precip)\n",
    "\n",
    "# find the max value and store the result as a new variable\n",
    "max_avg_precip = np.max(avg_monthly_precip)\n",
    "\n",
    "# print these values along with a message that labels each result\n",
    "print(\"minimum of average monthly precipitation:\", min_avg_precip)\n",
    "print(\"maximum of average monthly precipitation:\", max_avg_precip)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that in this code, you can only identify the value that is the minimum or maximum but not the month in which the value occurred. This is because `precip` and `months` are not connected in an easy way that would allow you to determine the month that matches the values. \n",
    "\n",
    "You could use indexing to determine the index location of the maximum value in `precip` and then query that same index location in `months`, but rest assured, there is an easier way to do this! \n",
    "\n",
    "In future lessons on `pandas dataframes`, you will learn how to work with data in a tabular structure, so that precip values are linked with their corresponding month names.\n",
    "\n",
    "\n",
    "## Plot Numpy Arrays\n",
    "\n",
    "Since you have now completed an easy calculation to convert the precipitation values using `numpy array` calculations, you can use this `numpy array` to plot the precipitation data, rather than relying on `Python` lists.\n",
    "\n",
    "In order to use multiple `numpy arrays` within the same plot, you need to make sure that the dimensions of the arrays are compatible. \n",
    "\n",
    "You have already done this by checking the `.shape` of `avg_monthly_precip` and `months`, which indicates that both have 12 elements along one dimension (`(12,)`). \n",
    "\n",
    "You can re-use your `matplotlib` code from the lesson on plotting with matplotlib to create the same plot of average monthly precipitation in Boulder, CO using `numpy arrays`. Recall that you can set the `color` in the plot (e.g. `grey`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "caption": "This plot displays a bar plot created from numpy arrays for average monthly precipitation for Boulder, CO.",
    "label": "fig:barplot_numpy"
   },
   "outputs": [],
   "source": [
    "# set plot size for all plots that follow\n",
    "plt.rcParams[\"figure.figsize\"] = (8, 8)\n",
    "\n",
    "# create the plot space upon which to plot the data\n",
    "fig, ax = plt.subplots()\n",
    "\n",
    "# add the x-axis and the y-axis to the plot\n",
    "ax.bar(months, avg_monthly_precip, color=\"grey\")\n",
    "\n",
    "# set plot title\n",
    "ax.set(title=\"Average Monthly Precipitation in Boulder, CO\")\n",
    "\n",
    "# add labels to the axes\n",
    "ax.set(xlabel=\"Month\", ylabel=\"Precipitation (mm)\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that `precip_2002` is still two dimensional array, so you cannot use it to plot data against `months`, which is a one-dimensional array.\n",
    "\n",
    "In future lessons, you will learn how to convert two-dimensional `numpy arrays` to one-dimensional `numpy arrays`. \n",
    "\n",
    "Congratulations! You have learned how to use indexing to select data from one-dimensional and two-dimensional `numpy arrays`, and how to run calculations and summary statistics on these `numpy arrays`. You also learned how to plot data from one-dimensional `numpy arrays`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optional Challenge \n",
    "\n",
    "Test your `Python` skills to:\n",
    "\n",
    "1. Convert the data values in `precip_2002_2013` from inches to millimeters (one inch = 25.4 millimeters). \n",
    "\n",
    "2. Create a new `numpy array` for 2013 by selecting all data values in the last row in `precip_2002_2013` (i.e. data for the year 2013).\n",
    "\n",
    "3. Calculate the minimum, mean, and maximum values for 2013. \n",
    "\n",
    "4. Print these values along with a message that labels each result (e.g. `mean precipitation in 2013:`). \n"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Edit Metadata",
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}