intro_python_exercise.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d245d99e-3dd3-4587-8d4b-38a7394075ee",
   "metadata": {},
   "source": [
    "# Introduction to Python Programming\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6e39c97-02bf-4211-9616-2ca92bfbb54e",
   "metadata": {},
   "source": [
    "By: Dr. Eric Godat and Dr. Rob Kalescky "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57d35198-922a-4af4-8abe-0e73ef030793",
   "metadata": {},
   "source": [
    "# Intro to Python Basics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e8e5fa3-2c3a-462e-8a42-1a6fa3b6585e",
   "metadata": {},
   "source": [
    "*This section is designed to guide you from having never used Python before to feeling comfortable with basic operations.*\n",
    "\n",
    "Python is a programming language capable of just about anything you could want to do. It is designed to be human readable and robust. This particular document is called a Jupyter Notebook. Notebooks are tools for developing Python code and running that code in small steps and see the intermediate results in line."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "084423ad-5f0d-4a1c-8e5b-134bca82652d",
   "metadata": {},
   "source": [
    "## Operating this notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6fdbf44-d99f-4b77-81fa-a4eb219e621b",
   "metadata": {},
   "source": [
    "To operate this notebook, you will need to execute boxes of code, called cells. To do this you can either click the *Run* button on the toolbar or use [Shift]+[Enter].\n",
    "\n",
    "Throughout this notebook you will see cells that are not code (like this one). These cells contain text in a language called Markdown. You can execute these cells to render formatted text. A cheatsheet for Markdown can be found by clicking `Help -> Markdown Reference` or articles [like this](https://medium.com/analytics-vidhya/the-ultimate-markdown-guide-for-jupyter-notebook-d5e5abf728fd). There will also be comments with in the code itself that give additional information about the operation of the code, describe functions or show you places where you might want to make changes. These comments are always preceded by an octothorp (#) and in most cases be a different color from the surrounding text.\n",
    "\n",
    "Notebooks save automatically, however if you have made big changes your code, you can checkpoint it. This allows you to roll back changes to this point (or any previous checkpoints) should something break in the future.\n",
    "\n",
    "There are a few useful [keyboard shortcuts](https://gist.github.com/discdiver/9e00618756d120a8c9fa344ac1c375ac) for modifying the cells in a notebook. To use these, click on the far left side of the cell such that the outline changes color (to blue in most cases), press [Enter]] to return to editing the cell's contents.\n",
    "\n",
    "-Add a cell **a**bove the current cell   [A]\n",
    "\n",
    "-Add a cell **b**elow the current cell   [B]\n",
    "\n",
    "-**D**elete the current cell   [D,D]\n",
    "\n",
    "**Let's try it out!**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bae1bd9a-4a82-4448-9e4c-1a67cfdd0e5c",
   "metadata": {},
   "source": [
    "### Basic Math"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1afbc885-f695-4a95-8336-ae6a57357dc5",
   "metadata": {},
   "source": [
    "Python can do simple mathematical operations just like a calculator. Try executing the following cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0fb9e19c-cee8-446a-bb8c-c82ccb1457a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "1+1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3cca59a9-a37c-4bc8-9edb-2baece702070",
   "metadata": {},
   "outputs": [],
   "source": [
    "2.5-2.0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47fae216-9909-434b-8bdf-951c99ea6816",
   "metadata": {},
   "source": [
    "## Variables and Objects"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ac125ec-8c29-458f-acd8-7fd89695b669",
   "metadata": {},
   "source": [
    "Calculations are nice but we really want to be able to store our calculations in memory so that we can access them later. We do that by assigning variables. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4160024d-240e-4651-9527-dd83e8fde279",
   "metadata": {},
   "outputs": [],
   "source": [
    "a=2\n",
    "b=3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a46aa39e-3302-4f8a-aaaf-4af07076bf95",
   "metadata": {},
   "outputs": [],
   "source": [
    "a+b"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "056177a4-17c9-4ccc-a160-07eb9328f462",
   "metadata": {},
   "source": [
    "Variables can be named just about anything"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c555b471-14c2-42bf-9aea-da0532686b46",
   "metadata": {},
   "outputs": [],
   "source": [
    "Dallas=2\n",
    "SMU=7\n",
    "texas_TX=3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "64fc57ae-716f-4541-8435-494edd026b6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Notice that if we assign a value to a variable, our notebook won't return that value to our screen.\n",
    "#If we want it to show us the value, we can call that variable after it is assigned.\n",
    "ouranswer = SMU*Dallas*texas_TX\n",
    "ouranswer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea38466e-6376-4462-be65-feddce50ba9e",
   "metadata": {},
   "source": [
    "So far we have been doing all of our operations on numbers, integers and floats (decimals), but there are other useful objects we can use in Python.\n",
    "\n",
    "Strings allow us to store and operate on text data. Strings in Python are surrounded by either single quotes '' or double quotes \"\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dff12a9e-f115-4d1c-a5c1-990de6fbf903",
   "metadata": {},
   "outputs": [],
   "source": [
    "ourstring = \"Here is a string in Python!\"\n",
    "ourstring"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f07f4957-b20e-451a-8aea-fd690cf5c1bf",
   "metadata": {},
   "source": [
    "If we later want to see the value stored in a variable we can always use the built in **print** function. **Print** is recognized as a keyword by our notebook and so the color changes to make our lives as programmers easier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "903b836e-fd1a-4c29-89c2-acd158b2c3b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(ourstring)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63feed17-5741-45e0-85cd-aa52b62a406c",
   "metadata": {},
   "source": [
    "What if we want to have multiple strings stored at once but we don't want to assign a bunch of variables? Then we can use a list.\n",
    "\n",
    "Lists are enclosed by brackets [] and separated by commas."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d9dd070-3157-4d68-8b26-c9be901b87f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "ourlist = ['This', 'is', \"a\", \"list\", 'of', \"strings\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12c8c4f1-4cc7-4542-a967-0b80f755e4e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(ourlist)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0871985f-390b-4934-987a-3ed723360daa",
   "metadata": {},
   "source": [
    "To access individual elements in a list you call it by its reference number.\n",
    "\n",
    "*__Note:__ In Python, counting starts with 0. Thus in the list [a,b,c], a is the 0th element and c is the 2nd*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5baadaa8-e155-461e-8d58-d7bb996e1f52",
   "metadata": {},
   "outputs": [],
   "source": [
    "first = ourlist[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "038ccb28-1e62-4235-a8af-9c8c5458005d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Notice we can call the last element by using a negative reference number. This lets us see the end without knowing how long the list is.\n",
    "last = ourlist[-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6149b0ad-e4d1-4a8a-8e96-a7545385a742",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(first,last)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31a44220-f746-4764-bf19-625f980b04c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# If we did want to know the length of our list we can use the len() function\n",
    "len(ourlist)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8957a63-d176-4fa8-861c-c3c300029cbd",
   "metadata": {},
   "source": [
    "### Dictionaries"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88cf796b-c068-4664-b4d9-0ae2e91cf815",
   "metadata": {},
   "source": [
    "Dictionaries are a different way of storing data than lists. They rely on a key and value system as opposed to the order of the entries.\n",
    "\n",
    "Dictionaries are enclosed in curly braces {}, the key and value are separated by a colon : and entries are separated by commas ,\n",
    "\n",
    "Values can then be accessed by referencing the key.\n",
    "\n",
    "*__Note:__ Finding dictionary values does not go both ways, so using the value to find the key will cause an error.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b435ee6-f68b-452d-9371-5432517a4a64",
   "metadata": {},
   "outputs": [],
   "source": [
    "ourdictionary = {\"Dallas\":1345000,\"Taos\":5668,'Houston':2326000}\n",
    "ourdictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55dc09e2-c4f0-48b5-bbf4-442bbd5c1983",
   "metadata": {},
   "outputs": [],
   "source": [
    "ourdictionary[\"Taos\"] \n",
    "#NOTE: ourdictionary[5668] will NOT give you \"Taos\", but an error."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce3bc9e8-375c-4b9e-8bf8-05784a7a1a13",
   "metadata": {},
   "source": [
    "## Loops and Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aacd57b7-09f3-4825-b5ec-47ff9e70df60",
   "metadata": {},
   "source": [
    "Computers are much better at doing simple repeatable tasks than humans are so to leverage this advantage, we will cover two different ways of writing these kind of repeatable instructions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2750bd6b-6e7e-4e12-bfb1-0849962a215a",
   "metadata": {},
   "source": [
    "### Loops"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aac73a36-f696-4ee7-8ed0-b617a79cee64",
   "metadata": {},
   "source": [
    "Loops are a way of having Python complete a task over and over.\n",
    "\n",
    "The most common form if a **for** loop. A **for** loop completes a task a fixed number of times by iterating a variable over the members of a sequence in order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3cf5e24c-8817-4f2f-bfa5-d634b85ed4a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Example for loop\n",
    "for i in [0,1,2]:\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a3e236a-8813-42a0-bbd9-2c7295f174cb",
   "metadata": {},
   "source": [
    "*__Note__: The indention is important, it tells Python that the line *print(i)* belongs inside the loop.*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a2c1256-4329-48f8-b40f-44adc82fcc2c",
   "metadata": {},
   "source": [
    "### Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "823d9bb9-5974-4eae-885c-2be1828e304e",
   "metadata": {},
   "source": [
    "Sometimes it would be nice to be able to save a group of instructions in a single block, that way we won't need to rewrite several lines of code each time we want to do that set of operations.\n",
    "\n",
    "To do this we define a **function**.\n",
    "\n",
    "Functions allow us to define a more complicated set of instructions as a single entity and call that entire block of code directly.\n",
    "\n",
    "Functions are defined (*def*), named (lowercase is a standard practice) and accept arguments (). They then can return a value if needed (*return*).\n",
    "\n",
    "Python has several built in functions (*print()* is a good example) and when you combine external packages and libraries, there are functions for just about everything you could think of."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "883cb9aa-44e7-4941-be9e-f0958a8ef2f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example function\n",
    "def square_me(n):\n",
    "    return n*n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42ca89d4-046b-4fad-87d6-9662294e09e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "square_me(4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a26535e6-f455-4320-9ad6-a7a1165954aa",
   "metadata": {},
   "source": [
    ">**Discussion**: What is something that you do over and over again?\n",
    ">Is that task something you could use a *loop* for or a *function* for?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a46196bc-f330-435e-889d-b1e515b5075b",
   "metadata": {},
   "source": [
    "Putting loops and functions together is where python can become very powerful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2cbf6449-05ba-40c7-81a5-a54cab122a52",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Looping over a function\n",
    "for i in [0,1,2,3]:\n",
    "    print(square_me(i)) # note that this is still tabbed in within the for loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "525e48bd-9f5f-4738-aac0-36f918353e24",
   "metadata": {},
   "outputs": [],
   "source": [
    "# A function with a loop\n",
    "def print_numbers(start,stop):\n",
    "    for i in range(start,stop): #Here the range() function gives us all the values from start up to but excluding the stop value\n",
    "        print(i) # this is indented twice, so Python knows it is within the loop within the function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e32d8a11-7089-4a36-b47d-b8f68b0dd3b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "print_numbers(0,4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f5f1bd5-8c85-4c2f-9369-10a7a0ee8695",
   "metadata": {},
   "source": [
    "# Libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a18f8387-2bb8-41fb-aaf3-314c1bea3f7c",
   "metadata": {},
   "source": [
    "Libraries are pre-written chunks of code designed to be integrated into projects for a specific purpose.  The idea is to keep people from having to reinvent the wheel every time they want to write a program.  When we encounter situations where a libary might be useful, we use the keyword *import* to tell the program that we want to use this pre-existing code.  As an example, in the next section you will use a library called Pandas which helps visualize data and make it easier to digest, which is very useful for us.  Because of this, we will import that code into this program simply by running the cell with the import that goes with that code.  Here is what you will see, only in commented code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9ee2437-e0cf-4619-a4cc-55bbd1c708dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "#import pandas as pd\n",
    "\n",
    "#by using the keyword *as* and giving pandas an alternative name (pd) in this program, it allows us to use the\n",
    "#functions within pandas while only having to type our pd instead of pandas every time."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6b20e15-3222-4247-8b62-5ee732622b47",
   "metadata": {},
   "source": [
    "Most programming languages have libraries that allow you to import code this way, and it will save you a lot of time and allow you to quickly use many new functions that are highly polished."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1577267-7690-48d9-ba84-603e92e0bd59",
   "metadata": {},
   "source": [
    "# DataFrames and Pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d96682b-54cd-45f0-a42f-a7911624de3d",
   "metadata": {},
   "source": [
    "Now it is time to use what we have learned to start doing some data science.\n",
    "\n",
    "Pandas is the name of a library in which there are functions and tools for doing operations common in data analysis and data science. One particularly great feature is the DataFrame structure which allows us to work with an object similar to an Excel spreadsheet but with the flexibility and power of Python behind it.\n",
    "\n",
    "First we will need to import the Pandas library and create a DataFrame. DataFrames are can also be created when you import files (like CSV's) or tables from a database (like SQL)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef27e699-5949-4e8f-8d6c-347f8ff1d6a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6540ba65-2ab7-4695-8c7e-39ed1b52ed19",
   "metadata": {},
   "outputs": [],
   "source": [
    "d = {\"Books\":[\"The Silmarillion\",\"The Hobbit\",\"The Fellowship of the Ring\",\"The Two Towers\",\"The Return of the King\"],\n",
    "     \"Words\":[130115,95506,187726,156147,137037]\n",
    "    }\n",
    "df = pd.DataFrame(d)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24bf4312-2e95-41fa-b0b8-af19873baee6",
   "metadata": {},
   "source": [
    "Now we can operate on entire rows or columns in our DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "429acdef-9240-4521-86d2-2687d9286d3b",
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Books\"] #Selecting the column \"Books\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7bf53a2e-81a6-41a2-8979-a15c694858c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "df[:2] #Selecting the first 2 rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "000a9537-dad7-4e29-bac6-2ac69d18096c",
   "metadata": {},
   "outputs": [],
   "source": [
    "df[2:4] #Selecting rows 2 up to 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9f4de9b-3052-43be-81bd-cecd70ff0633",
   "metadata": {},
   "outputs": [],
   "source": [
    "df[\"Words\"][2:4] #Selecting the second and third row from the \"Words\" column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a022a861-491d-43dc-a1e0-dec08497670d",
   "metadata": {},
   "outputs": [],
   "source": [
    "df[2:4][\"Words\"] #Same result but from the opposite order"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c128778a-50e5-4422-ae33-eb457135afdb",
   "metadata": {},
   "source": [
    "You can also operate on entire columns in your DataFrame. Just be careful because this can give you unexpected results. Rows are generally more consistent for elements within a DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40ab702b-2bf5-485f-8ec5-76c7beeb03ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "total = sum(df[\"Words\"])\n",
    "total"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a300d589-d98c-4d9c-add4-fd62030bdee6",
   "metadata": {},
   "source": [
    "## More Complex Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7dd81c00-bb28-42cd-a003-ff4ef75724bc",
   "metadata": {},
   "source": [
    "To look at slightly more complex data operations, we need more complex data.\n",
    "\n",
    "Below we have all the chapters from the Hobbit and Lord of the Rings books along with their word counts. Notice that each chapter is a list of information and then each of those list is an element in a larger list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "28f9a735-c173-4dec-bb78-bc5a0df8830d",
   "metadata": {},
   "outputs": [],
   "source": [
    "chapters = [[0,1,'An Unexpected Party',8638,0],\n",
    "            [0,2,'Roast Mutton',5257,0],\n",
    "            [0,3,'A Short Rest',2876,0],\n",
    "            [0,4,'Over Hill and Under Hill',4034,0],\n",
    "            [0,5,'Riddles in the Dark',6967,0],\n",
    "            [0,6,'Out of the Frying Pan into the Fire',6703,0],\n",
    "            [0,7,'Queer Lodgings',9027,0],\n",
    "            [0,8,'Flies and Spiders',10223,0],\n",
    "            [0,9,'Barrels Out of Bond',5833,0],\n",
    "            [0,10,'A Warm Welcome',3930,0],\n",
    "            [0,11,'On the Doorstep',3001,0],\n",
    "            [0,12,'Inside Information',7132,0],\n",
    "            [0,13,'Not At Home',3909,0],\n",
    "            [0,14,'Fire and Water',3236,0],\n",
    "            [0,15,'The Gathering of the Clouds',3362,0],\n",
    "            [0,16,'A Thief in the Night',2153,0],\n",
    "            [0,17,'The Clouds Burst',3949,0],\n",
    "            [0,18,'The Return Journey',2815,0],\n",
    "            [0,19,'The Last Stage',2461,0],\n",
    "            [1,-4,'Concerning Hobbits',3406,1],\n",
    "            [1,-3,'Concerning Pipeweed',600,1],\n",
    "            [1,-2,'Of the Ordering of the Shire',2431,1],\n",
    "            [1,-1,'Note on the Shire Records',914,1],\n",
    "            [1,1,'A Long-expected Party',10012,1],\n",
    "            [1,2,'The Shadow of the Past',11311,1],\n",
    "            [1,3,'Three is Company',9763,1],\n",
    "            [1,4,'A Short Cut to Mushrooms',5957,1],\n",
    "            [1,5,'A Conspiracy Unmasked',5196,1],\n",
    "            [1,6,'The Old Forest',6502,1],\n",
    "            [1,7,'In the House of Tom Bombadil',5501,1],\n",
    "            [1,8,'Fog on the Barrow-downs',6694,1],\n",
    "            [1,9,'At the Sign of the Prancing Pony',6251,1],\n",
    "            [1,10,'Strider',5905,1],\n",
    "            [1,11,'A Knife in the Dark',9468,1],\n",
    "            [1,12,'Flight to the Ford',8805,1],\n",
    "            [1,1,'Many Meetings',9085,2],\n",
    "            [1,2,'The Council of Elrond',16360,2],\n",
    "            [1,3,'The Ring goes South',10656,2],\n",
    "            [1,4,'A Journey in the Dark',11501,2],\n",
    "            [1,5,'The Bridge of Khazad-dum',5428,2],\n",
    "            [1,6,'Lothlorien',9387,2],\n",
    "            [1,7,'The Mirror of Gladriel',6896,2],\n",
    "            [1,8,'Farewell to Lorien',6174,2],\n",
    "            [1,9,'The Great River',7218,2],\n",
    "            [1,10,'The Breaking of the Fellowship',6305,2],\n",
    "            [2,1,'The Departure of Boromir',3397,3],\n",
    "            [2,2,'The Riders of Rohan',11133,3],\n",
    "            [2,3,'The Uruk-hai',7854,3],\n",
    "            [2,4,'Treebeard',12876,3],\n",
    "            [2,5,'The White Rider',8856,3],\n",
    "            [2,6,'The King of the Golden Hall',9303,3],\n",
    "            [2,7,\"Helm's Deep\",7575,3],\n",
    "            [2,8,'The Road to Isengard',7899,3],\n",
    "            [2,9,'Flotsam and Jetsam',7789,3],\n",
    "            [2,10,'The Voice of Saruman',5663,3],\n",
    "            [2,11,'The Palantir',6325,3],\n",
    "            [2,1,'The Taming of Smeagol',8375,4],\n",
    "            [2,2,'The Passage of the Marshes',7357,4],\n",
    "            [2,3,'The Black Gate is Closed',5881,4],\n",
    "            [2,4,'Of Herbs and Stewed Rabbit',6975,4],\n",
    "            [2,5,'The Window on the West',10120,4],\n",
    "            [2,6,'The Forbidden Pool',5179,4],\n",
    "            [2,7,'Journey to the Crossroads',4266,4],\n",
    "            [2,8,'The Stairs of Cirith Ungol',6793,4],\n",
    "            [2,9,\"Shelob's Lair\",5209,4],\n",
    "            [2,10,'The Choices of Master Samwise',7322,4],\n",
    "            [3,1,'Minas Tirith',13100,5],\n",
    "            [3,2,'The Passing of the Grey Company',8586,5],\n",
    "            [3,3,'The Muster of Rohan',6951,5],\n",
    "            [3,4,'The Siege of Gondor',11793,5],\n",
    "            [3,5,'The Ride of the Rohirrim',4358,5],\n",
    "            [3,6,'The Battle of the Pelennor Fields',5225,5],\n",
    "            [3,7,'The Pyre of Denethor',3736,5],\n",
    "            [3,8,'The Houses of Healing',6731,5],\n",
    "            [3,9,'The Last Debate',5416,5],\n",
    "            [3,10,'The Black Gate Opens',5204,5],\n",
    "            [3,1,'The Tower of Cirith Ungol',9721,6],\n",
    "            [3,2,'The Land of Shadow',8446,6],\n",
    "            [3,3,'Mount Doom',7777,6],\n",
    "            [3,4,'The Field of Cormallen',4721,6],\n",
    "            [3,5,'The Steward and the King',7639,6],\n",
    "            [3,6,'Many Partings',7440,6],\n",
    "            [3,7,'Homeward Bound',4106,6],\n",
    "            [3,8,'The Scouring of the Shire',11296,6],\n",
    "            [3,9,'The Grey Havens',4791,6]\n",
    "           ]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbaf5010-1b93-49e8-85cc-5833854d86d5",
   "metadata": {},
   "source": [
    "Now we can turn the list of the lists into a dataframe. We have also named our columns. This isn't necessary but it does make things clearer to work with."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3856ec50-f348-4a33-b91d-c8c6b5dbb591",
   "metadata": {},
   "outputs": [],
   "source": [
    "cols = ['CollectionNum','ChapterNum','ChapterName','WordCount','BookNum']\n",
    "data = pd.DataFrame(chapters, columns=cols)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6878626-1a39-4339-a065-835da4ddb3af",
   "metadata": {},
   "outputs": [],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fdd4068-e270-4043-b381-257b74ce5c56",
   "metadata": {},
   "source": [
    "Now we might want to apply the names of each of the traditional \"books\" you might think of when you think about the Lord of the Rings. To do this we need to use a function that is very powerful and flexible: [apply](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.apply.html). This function will allow us to apply an operation or function to each value in a column and save it into a new column. In this example, our \"function\" is using our *CollectionNum* to reference a book name in our list *titles*."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80e27d46-2ad8-4cbb-8eb7-3748f64ad956",
   "metadata": {},
   "source": [
    "The structure of the apply function is: `apply(lambda *variable* : *function*(*variable*) )`\n",
    "Where the variable represents the value in each row and the opertaion of function will return a value that can be stored in our new column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b86d550-6c96-4444-b6bb-247d5b11a06c",
   "metadata": {},
   "outputs": [],
   "source": [
    "titles = ['The Hobbit','The Fellowship of the Ring', 'The Two Towers', 'The Return of the King']\n",
    "data['CollectionName']=data['CollectionNum'].apply(lambda x: titles[x])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13e3c0f6-31c9-4dc6-86a4-ded57616b146",
   "metadata": {},
   "source": [
    "## [GroupBy](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.groupby.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77af8133-1a86-4483-be95-8a2744edb15d",
   "metadata": {},
   "source": [
    "Now that we have data for each of the chapters, we can group them using groupby. This lets us do aggregate operations like \"add all the wordcounts for each book\" or \"count how many chapters there are in each book\".\n",
    "\n",
    "Groupby is a powerful tool but if you don't understand your data, it can quickly introduce errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57960a96-7784-4a9a-99be-8a00842fb692",
   "metadata": {},
   "outputs": [],
   "source": [
    "data.groupby(by='CollectionName').count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0a6ca5c-3ad0-487d-bd4a-aeb2e93cf377",
   "metadata": {},
   "outputs": [],
   "source": [
    "data.groupby(by='CollectionName').sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6ea60e3-b009-4c26-84e4-d4be44118fae",
   "metadata": {},
   "source": [
    "## Selecting (Advanced Slicing)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af981f7a-93bc-4b65-ad17-cc3568706c19",
   "metadata": {},
   "source": [
    "You can select slices of your dataframe using conditional logic as well. Conditional logic means determining if a condition is `True` or `False` and often relies on operations like: equals `==`, not `!`, and `&`, or `|`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4003b743-7163-4787-a39e-702cdd97e231",
   "metadata": {},
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5c1290c-740e-4a53-ab24-68dd6bf8acef",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select values based on exact matches\n",
    "df[df['Books']=='The Hobbit']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d7ce4d9-1a55-4dd2-9c9b-2fc9c18c0843",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select values by negating a match\n",
    "df[df['Books']!='The Hobbit']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2373f825-08eb-4e9a-acd0-26e7c4d4c0b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select values containing substrings\n",
    "df[df['Books'].str.contains('ing')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eff15414-3427-42fb-8ac2-c042844302bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select values on multiple conditions using different columns\n",
    "df[(df['Books'].str.contains('ing')) & (df['Words']>150000)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19745208-6b8b-4e57-8a22-3f6c06c3c559",
   "metadata": {},
   "source": [
    "## Joining"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0434b69-6f8a-45ea-bf55-dddcf709c4a5",
   "metadata": {},
   "source": [
    "You can also merge 2 dataframes if you want. Merging allows you to combine datasets in new ways and is a great tool to have when working with complex datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25016789-e4f5-44ea-8741-5c77d42d9570",
   "metadata": {},
   "source": [
    "First we need a second dataframe to work with. Here, we have made a dataframe with the publication year of each of of our Tolkien books."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da0d151b-5d36-430e-bc4c-66345895806b",
   "metadata": {},
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "64ebf3d4-a59e-4863-b9fa-c44329e26955",
   "metadata": {},
   "outputs": [],
   "source": [
    "dates = pd.DataFrame({'Name':df['Books'],'Year':pd.Series([1977,1934,1954,1954,1954])})\n",
    "dates\n",
    "# NOTE: We named our column 'Name' for instructional purposes, but a better name would be 'Books' so that it matches my other dataframe.\n",
    "#       Typically you want to use a unique identifier in your data and then merge based on that column."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "018b1ef4-2358-437d-981d-551cb47e5816",
   "metadata": {},
   "source": [
    "Now we can merge our new dates dataframe with our existing dataframe.\n",
    "\n",
    "We have done an inner join (only rows that match in both dataframes will appear in our join) where the column in the left dataframe is called \"Books\" and the column in the right dataframe is called \"Name\".\n",
    "\n",
    "More documentation on merging dataframes can be found [here](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.merge.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f4d7723-55c6-4e8b-9b47-aef2052e1fa1",
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.merge(df,dates, left_on=\"Books\", right_on=\"Name\", how = 'inner')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8af0b962-c321-45d9-be01-e7529e24bc56",
   "metadata": {},
   "source": [
    "Notice that our merge duplicated our matching column because they had different names. This would not happen if we had the same name for columns in both dataframes. However, will not always be the case, so you can clean up your dataframe by dropping one of the redundant columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6948b2de-ac20-4f8c-a6c4-2a692a7f9019",
   "metadata": {},
   "outputs": [],
   "source": [
    "merged = pd.merge(df,dates, left_on=\"Books\", right_on=\"Name\", how = 'inner').drop(columns=['Name'])\n",
    "merged"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3078a70-5219-459f-8275-46f78710d17b",
   "metadata": {},
   "source": [
    "# Plotting"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da11394a-4650-4523-8d7e-681480deb721",
   "metadata": {},
   "source": [
    "Visualizing data is incredibly important when trying to convey findings. There are several libraries available for doing data visualization. Pandas has some built in plotting functionality, but we have provided access to MatPlotLib as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e762aea-fe10-434b-bbe8-8da2591cce73",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is only needed in this case because we wanted to show the names of the books. You could make a similar plot in Pandas.\n",
    "import matplotlib as plt\n",
    "%matplotlib inline\n",
    "# This allows the plot to be shown inline in the notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66b329da-88e4-4ff9-88a7-7c09ddcdfb03",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot = df.plot(kind='bar') # we are setting the type of plot to a bar graph\n",
    "plot.set_xticklabels(df['Books']); #This lets us rename the x axis labels"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "312c6a96-b913-48df-98f1-e0f0fd4c01a2",
   "metadata": {},
   "source": [
    "## More Complex Plotting"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1ddacc1-889b-4764-ba7c-98b07c316751",
   "metadata": {},
   "source": [
    "Lastly, we can combine several of the things we have learned into a single plot. Here we can group each book together and then look at the word counts by chapter throughout the story."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8569c24-df4d-4e01-925b-bfd7c18323c3",
   "metadata": {},
   "source": [
    "First, we want to come up with a way to step through the books in a linear way. We have provided a few ways to do this in increasing complexity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f658c35f-9635-4408-a0b4-32976934115f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Simple - add the collection number to the chapter number divided by 10\n",
    "data['BookChapter'] = data['CollectionNum']+0.1*data['ChapterNum']\n",
    "\n",
    "# Much better - add the book number to the chapter number divided by 10\n",
    "#data['BookChapter'] = data['BookNum']+0.1*data['ChapterNum']\n",
    "\n",
    "# Best but complicated - add the book number to the chapter number scaled by 1/n where n is the max number of chapters in that book.\n",
    "#data['BookChapter']=data.apply(lambda x: x['BookNum']+(1/data.groupby(by='BookNum').max()['ChapterNum'][x['BookNum']])*x['ChapterNum'], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52162f3a-2f4c-404c-a904-03ba299418cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "groups = data.groupby(\"CollectionName\")\n",
    "\n",
    "for name, group in groups:\n",
    "\n",
    "    plt.pyplot.plot(group[\"BookChapter\"], group[\"WordCount\"],label=name)\n",
    "\n",
    "plt.pyplot.legend()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "868b410e-6958-4fad-a7d8-32c4908abb88",
   "metadata": {},
   "source": [
    " Notice that the prologue materials in *The Fellowship of the Ring* cause it to overlap with the *Hobbit*. If we wanted to resolve this, we would need to revisit our conventions in our data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21fc6862-a576-474b-8cf4-13446ff5d639",
   "metadata": {},
   "source": [
    "# Wrapping up"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22ddc74b-f299-4698-955f-8fbbfbd9ff46",
   "metadata": {},
   "source": [
    "We covered:\n",
    "- Basic Math\n",
    "- Variables and Objects\n",
    "- Dictionaries\n",
    "- Loops and Functions\n",
    "- DataFrames in Pandas\n",
    "- Plotting"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "formats": "ipynb,md"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}