import-text-files-numpy-arrays.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text Files\n",
    "\n",
    "Scientific data can come in a variety of file formats and types. In this course, you will work with data stored in plain text files (.txt) and comma-delimited text files (.csv).\n",
    "\n",
    "### Plain Text Files\n",
    "\n",
    "Plain text files simply list out the values on separate lines without any symbols or delimiters to indicate separate values. For example, data for the average monthly precipitation data for Boulder, CO can be stored as a plain text file (.txt), with a separate line for each month's value. \n",
    "\n",
    "```python\n",
    "0.70\n",
    "0.75\n",
    "1.85\n",
    "2.93\n",
    "3.05\n",
    "2.02\n",
    "1.93\n",
    "1.62\n",
    "1.84\n",
    "1.31\n",
    "1.39\n",
    "0.84\n",
    "```\n",
    "\n",
    "Due to their simplicity, text files (.txt) can be very useful for collecting very large datasets that are all the same type of observation or data type. \n",
    "\n",
    "\n",
    "### CSV Files\n",
    "\n",
    "Unlike plain-text files which simply list out the values on separate lines without any symbols or delimiters, comma delimited (CSV) files use commas (or some other delimiter like tab spaces or semi-colons) to indicate separate values.\n",
    "\n",
    "This means that CSV files can easily support multiple rows and columns of related data. For example, data for the monthly precipitation for Boulder, CO for the years 2002 and 2013 can be stored together in a comma delimited (.csv) file.\n",
    "\n",
    "```python\n",
    "1.07, 0.44, 1.50, 0.20, 3.20, 1.18, 0.09, 1.44, 1.52, 2.44, 0.78, 0.02\n",
    "0.27, 1.13, 1.72, 4.14, 2.66, 0.61, 1.03, 1.40, 18.16, 2.24, 0.29, 0.5\n",
    "```\n",
    "\n",
    "In this lesson and the next lesson, you will use data from:\n",
    "\n",
    "* a .txt file containing the average monthly precipitation data for Boulder, CO\n",
    "* a .csv file containing the monthly precipitation for Boulder, CO for the years 2002 and 2013\n",
    "\n",
    "  \n",
    "## Begin Writing Your Code\n",
    "\n",
    "From previous lessons, you know that you always begin your `Python` code by importing the necessary packages and checking the working directory. \n",
    "\n",
    "### Import Packages\n",
    "\n",
    "In this lesson, you will use the `os` package along with some new packages:\n",
    "\n",
    "1. `numpy` with the alias `np`: to create and work with data as `numpy arrays`\n",
    "2. `urllib`: to download the datasets for this lesson"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import necessary Python packages\n",
    "import os\n",
    "import numpy as np\n",
    "import urllib.request\n",
    "\n",
    "# print message after packages imported successfully\n",
    "print(\"import of packages successful\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set Working Directory\n",
    "\n",
    "Remember that you can check the current working directory using `os.getcwd()`. You can also set the current working directory using another useful function `os.chdir()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set the working directory to the `earth-analytics-bootcamp` directory\n",
    "# replace `jpalomino` with your username here and all paths in this lesson\n",
    "os.chdir(\"/home/jpalomino/earth-analytics-bootcamp/\")\n",
    "\n",
    "# For Windows users, you may have to set the directory using the full path that Windows maintains for your user. For example:\n",
    "# os.chdir(\"C:/Users/Jenny Palomino/earth-analytics-bootcamp/\")\n",
    "\n",
    "# For Mac Users, you may have to set the directory using the full path that Mac maintains for your user. For example:\n",
    "# os.chdir(\"/Users/jpalomino/earth-analytics-bootcamp/\")\n",
    "\n",
    "# print the current working directory\n",
    "os.getcwd()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download Data Using URLs\n",
    "\n",
    "You can use the `urllib` package to download data from online sources such as `Figshare.com`, where the datasets for this lesson are published. \n",
    "\n",
    "To use `urllib`, you provide parameter values for `url` as well as `filename` for the downloaded file. \n",
    "\n",
    "For this lesson, you will download the .txt files for average monthly precipitation for Boulder, CO as well as the month names from the Earth Lab `Figshare.com` repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use `urllib` download files from Earth Lab figshare repository\n",
    "\n",
    "# download .txt containing monthly average precipitation for Boulder, CO\n",
    "urllib.request.urlretrieve(url = \"https://ndownloader.figshare.com/files/12565616\", \n",
    "                           filename = \"data/avg-monthly-precip.txt\")\n",
    "\n",
    "# download .txt containing month names\n",
    "urllib.request.urlretrieve(url = \"https://ndownloader.figshare.com/files/12565619\", \n",
    "                           filename = \"data/months.txt\")\n",
    "\n",
    "# print message that data downloads were successful\n",
    "print(\"datasets downloaded successfully\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that you do not have to provide the full path for filename because it is relative to the current working directory that you set using `os.chdir()`.\n",
    "\n",
    "**Into which directory were your files downloaded?** Open the directory on your computer to see your downloaded files.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import Text Data Into Numpy Arrays\n",
    "\n",
    "### Numeric Data\n",
    "\n",
    "You can create new `numpy arrays` by importing data from files, such as text files. You can import these data using the `loadtxt()` function from `numpy`, which you imported as `np`. \n",
    "\n",
    "For both .txt and .csv files, you need to specify a value for the parameter called `fname` for the file name (e.g. `np.loadtxt(fname = \"filename.txt\")`). Be sure to update the path for the file to your home directory. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the monthly average values from `avg-monthly-precip.txt` as a numpy array\n",
    "avg_monthly_precip = np.loadtxt(fname = \"/home/jpalomino/earth-analytics-bootcamp/data/avg-monthly-precip.txt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall that that you can use the `print()` function to see the values stored in a variable (e.g. `print(variablename)`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print the data in `avg_monthly_precip`\n",
    "print(avg_monthly_precip)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also use the `type()` function to check the type of data structure (e.g. `type(variablename)`) and see that `avg_monthly_precip` is a `numpy array`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print the type for the `avg_monthly_precip` variable\n",
    "print(type(avg_monthly_precip))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text String Data\n",
    "\n",
    "In addition to numeric data, you can also import text strings to `numpy arrays` using the `genfromtxt()` function from `numpy`. You need to specify a parameter value for `filename` as well as for the data type as `dtype='str'`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the names of the months from month.txt as a numpy array\n",
    "months = np.genfromtxt(\"/home/jpalomino/earth-analytics-bootcamp/data/months.txt\", dtype='str')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, you can check the type and the data in your new `numpy array`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print the type for the `months` variable\n",
    "print(type(months))\n",
    "\n",
    "# print the values in `months`\n",
    "print(months)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations! You have now learned how to create `numpy arrays` by importing numeric and text string data from text files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optional Challenge \n",
    "\n",
    "Test your `Python` skills to:\n",
    "\n",
    "1. Download a .csv file containing the monthly precipitation for Boulder, CO for the years 2002 and 2013 (`monthly-precip-2002-2013.csv`)from `https://ndownloader.figshare.com/files/12707792`. \n",
    "    * Be sure to assign a useful variable name that is short but indicative of what it contains (e.g. `precip_2002_2013`).\n",
    "\n",
    "2. Import the data from this .csv file into a `numpy array`. \n",
    "    * Note that for .csv files, you need to specify another parameter in addition to `filename`.  You need to provide a value for `delimiter`, which indicates the symbol that is used to separate the values (e.g. `delimiter = \",\"`). \n",
    "    \n",
    "3. Print the data type of your new `numpy array` as well as its contents.\n",
    "    * Note that you can add a line of code **before print(variablename)** to display the values in the `numpy array` as floats, rather than scientific notation (`np.set_printoptions(suppress=True)`).\n",
    "    \n",
    "4. Print the data contained in `avg_monthly_precip`, and compare it to your new `numpy array`. **Do you notice any differences in the structure of the data between these two `numpy arrays`?**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# download .csv containing monthly average precipitation for Boulder, CO\n",
    "urllib.request.urlretrieve(url = \"https://ndownloader.figshare.com/files/12707792\", \n",
    "                           filename = \"data/monthly-precip-2002-2013.csv\")\n",
    "\n",
    "# import the monthly average values from `monthly-precip-2002-2013.csv` as a numpy array\n",
    "precip_2002_2013 = np.loadtxt(fname= \"/home/jpalomino/earth-analytics-bootcamp/data/monthly-precip-2002-2013.csv\", delimiter = \",\")\n",
    "\n",
    "# print the type for the `months` variable\n",
    "print(type(precip_2002_2013))\n",
    "\n",
    "# display the values in the `numpy array` as floats, rather than scientific notation\n",
    "np.set_printoptions(suppress=True)\n",
    "\n",
    "# print the values in `months`\n",
    "print(precip_2002_2013)\n",
    "\n",
    "# print the data in `avg_monthly_precip`\n",
    "print(avg_monthly_precip)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}