618_03_Pandas_Advanced.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# SI 618: Data Manipulation and Analysis\n",
    "## 03 - Advanced pandas\n",
    "### Dr. Chris Teplovs, School of Information, University of Michigan\n",
    "<small><a rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-sa/4.0/\"><img alt=\"Creative Commons License\" style=\"border-width:0\" src=\"https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png\" /></a> This work is licensed under a <a rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-sa/4.0/\">Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### IMPORTANT: Replace ```?``` in the following code with your uniqname."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "MY_UNIQNAME = 'xinyexu'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Before we start...\n",
    "## <font color=\"magenta\">Q0: (2 point) Please let us know what you found confusing in the last class. </font>\n",
    "We'll try to take time in the next class to review these concepts next class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loading and manipulating data in pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning Objectives\n",
    "* load CSV files\n",
    "* load JSON files\n",
    "* use pd.read_html to extract tables from web pages\n",
    "* load data from simple APIs \n",
    "* load data from a SQL database\n",
    "* handle missing data (dropna and fillna)\n",
    "* use vectorized string functions\n",
    "* Pandas refresher (or introduction)\n",
    "* explain how pandas operations differ from \"traditional\" python\n",
    "* be able to load a CSV file into a Pandas DataFrame\n",
    "* explain how to extract columns from a DataFrame\n",
    "* sort a DataFrame\n",
    "* assign a column as the index of a DataFrame\n",
    "* filter a DataFrame according to some criteria\n",
    "* explain how boolean masks work in filtering DataFrames"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall the ```pd.read_csv``` function that we used to load data sets in previous classes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Category</th>\n",
       "      <th>Item</th>\n",
       "      <th>Serving Size</th>\n",
       "      <th>Calories</th>\n",
       "      <th>Calories from Fat</th>\n",
       "      <th>Total Fat</th>\n",
       "      <th>Total Fat (% Daily Value)</th>\n",
       "      <th>Saturated Fat</th>\n",
       "      <th>Saturated Fat (% Daily Value)</th>\n",
       "      <th>Trans Fat</th>\n",
       "      <th>...</th>\n",
       "      <th>Carbohydrates</th>\n",
       "      <th>Carbohydrates (% Daily Value)</th>\n",
       "      <th>Dietary Fiber</th>\n",
       "      <th>Dietary Fiber (% Daily Value)</th>\n",
       "      <th>Sugars</th>\n",
       "      <th>Protein</th>\n",
       "      <th>Vitamin A (% Daily Value)</th>\n",
       "      <th>Vitamin C (% Daily Value)</th>\n",
       "      <th>Calcium (% Daily Value)</th>\n",
       "      <th>Iron (% Daily Value)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Breakfast</td>\n",
       "      <td>Egg McMuffin</td>\n",
       "      <td>4.8 oz (136 g)</td>\n",
       "      <td>300</td>\n",
       "      <td>120</td>\n",
       "      <td>13.0</td>\n",
       "      <td>20</td>\n",
       "      <td>5.0</td>\n",
       "      <td>25</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>31</td>\n",
       "      <td>10</td>\n",
       "      <td>4</td>\n",
       "      <td>17</td>\n",
       "      <td>3</td>\n",
       "      <td>17</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>25</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Breakfast</td>\n",
       "      <td>Egg White Delight</td>\n",
       "      <td>4.8 oz (135 g)</td>\n",
       "      <td>250</td>\n",
       "      <td>70</td>\n",
       "      <td>8.0</td>\n",
       "      <td>12</td>\n",
       "      <td>3.0</td>\n",
       "      <td>15</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>30</td>\n",
       "      <td>10</td>\n",
       "      <td>4</td>\n",
       "      <td>17</td>\n",
       "      <td>3</td>\n",
       "      <td>18</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>25</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Breakfast</td>\n",
       "      <td>Sausage McMuffin</td>\n",
       "      <td>3.9 oz (111 g)</td>\n",
       "      <td>370</td>\n",
       "      <td>200</td>\n",
       "      <td>23.0</td>\n",
       "      <td>35</td>\n",
       "      <td>8.0</td>\n",
       "      <td>42</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>29</td>\n",
       "      <td>10</td>\n",
       "      <td>4</td>\n",
       "      <td>17</td>\n",
       "      <td>2</td>\n",
       "      <td>14</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>25</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Breakfast</td>\n",
       "      <td>Sausage McMuffin with Egg</td>\n",
       "      <td>5.7 oz (161 g)</td>\n",
       "      <td>450</td>\n",
       "      <td>250</td>\n",
       "      <td>28.0</td>\n",
       "      <td>43</td>\n",
       "      <td>10.0</td>\n",
       "      <td>52</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>30</td>\n",
       "      <td>10</td>\n",
       "      <td>4</td>\n",
       "      <td>17</td>\n",
       "      <td>2</td>\n",
       "      <td>21</td>\n",
       "      <td>15</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Breakfast</td>\n",
       "      <td>Sausage McMuffin with Egg Whites</td>\n",
       "      <td>5.7 oz (161 g)</td>\n",
       "      <td>400</td>\n",
       "      <td>210</td>\n",
       "      <td>23.0</td>\n",
       "      <td>35</td>\n",
       "      <td>8.0</td>\n",
       "      <td>42</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>30</td>\n",
       "      <td>10</td>\n",
       "      <td>4</td>\n",
       "      <td>17</td>\n",
       "      <td>2</td>\n",
       "      <td>21</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>25</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Breakfast</td>\n",
       "      <td>Steak &amp; Egg McMuffin</td>\n",
       "      <td>6.5 oz (185 g)</td>\n",
       "      <td>430</td>\n",
       "      <td>210</td>\n",
       "      <td>23.0</td>\n",
       "      <td>36</td>\n",
       "      <td>9.0</td>\n",
       "      <td>46</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>31</td>\n",
       "      <td>10</td>\n",
       "      <td>4</td>\n",
       "      <td>18</td>\n",
       "      <td>3</td>\n",
       "      <td>26</td>\n",
       "      <td>15</td>\n",
       "      <td>2</td>\n",
       "      <td>30</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Breakfast</td>\n",
       "      <td>Bacon, Egg &amp; Cheese Biscuit (Regular Biscuit)</td>\n",
       "      <td>5.3 oz (150 g)</td>\n",
       "      <td>460</td>\n",
       "      <td>230</td>\n",
       "      <td>26.0</td>\n",
       "      <td>40</td>\n",
       "      <td>13.0</td>\n",
       "      <td>65</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>38</td>\n",
       "      <td>13</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>3</td>\n",
       "      <td>19</td>\n",
       "      <td>10</td>\n",
       "      <td>8</td>\n",
       "      <td>15</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Breakfast</td>\n",
       "      <td>Bacon, Egg &amp; Cheese Biscuit (Large Biscuit)</td>\n",
       "      <td>5.8 oz (164 g)</td>\n",
       "      <td>520</td>\n",
       "      <td>270</td>\n",
       "      <td>30.0</td>\n",
       "      <td>47</td>\n",
       "      <td>14.0</td>\n",
       "      <td>68</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>43</td>\n",
       "      <td>14</td>\n",
       "      <td>3</td>\n",
       "      <td>12</td>\n",
       "      <td>4</td>\n",
       "      <td>19</td>\n",
       "      <td>15</td>\n",
       "      <td>8</td>\n",
       "      <td>20</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Breakfast</td>\n",
       "      <td>Bacon, Egg &amp; Cheese Biscuit with Egg Whites (R...</td>\n",
       "      <td>5.4 oz (153 g)</td>\n",
       "      <td>410</td>\n",
       "      <td>180</td>\n",
       "      <td>20.0</td>\n",
       "      <td>32</td>\n",
       "      <td>11.0</td>\n",
       "      <td>56</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>36</td>\n",
       "      <td>12</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>3</td>\n",
       "      <td>20</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>15</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Breakfast</td>\n",
       "      <td>Bacon, Egg &amp; Cheese Biscuit with Egg Whites (L...</td>\n",
       "      <td>5.9 oz (167 g)</td>\n",
       "      <td>470</td>\n",
       "      <td>220</td>\n",
       "      <td>25.0</td>\n",
       "      <td>38</td>\n",
       "      <td>12.0</td>\n",
       "      <td>59</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>42</td>\n",
       "      <td>14</td>\n",
       "      <td>3</td>\n",
       "      <td>12</td>\n",
       "      <td>4</td>\n",
       "      <td>20</td>\n",
       "      <td>6</td>\n",
       "      <td>8</td>\n",
       "      <td>15</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Breakfast</td>\n",
       "      <td>Sausage Biscuit (Regular Biscuit)</td>\n",
       "      <td>4.1 oz (117 g)</td>\n",
       "      <td>430</td>\n",
       "      <td>240</td>\n",
       "      <td>27.0</td>\n",
       "      <td>42</td>\n",
       "      <td>12.0</td>\n",
       "      <td>62</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>34</td>\n",
       "      <td>11</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Breakfast</td>\n",
       "      <td>Sausage Biscuit (Large Biscuit)</td>\n",
       "      <td>4.6 oz (131 g)</td>\n",
       "      <td>480</td>\n",
       "      <td>280</td>\n",
       "      <td>31.0</td>\n",
       "      <td>48</td>\n",
       "      <td>13.0</td>\n",
       "      <td>65</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>39</td>\n",
       "      <td>13</td>\n",
       "      <td>3</td>\n",
       "      <td>11</td>\n",
       "      <td>3</td>\n",
       "      <td>11</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Breakfast</td>\n",
       "      <td>Sausage Biscuit with Egg (Regular Biscuit)</td>\n",
       "      <td>5.7 oz (163 g)</td>\n",
       "      <td>510</td>\n",
       "      <td>290</td>\n",
       "      <td>33.0</td>\n",
       "      <td>50</td>\n",
       "      <td>14.0</td>\n",
       "      <td>71</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>36</td>\n",
       "      <td>12</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>18</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Breakfast</td>\n",
       "      <td>Sausage Biscuit with Egg (Large Biscuit)</td>\n",
       "      <td>6.2 oz (177 g)</td>\n",
       "      <td>570</td>\n",
       "      <td>330</td>\n",
       "      <td>37.0</td>\n",
       "      <td>57</td>\n",
       "      <td>15.0</td>\n",
       "      <td>74</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>42</td>\n",
       "      <td>14</td>\n",
       "      <td>3</td>\n",
       "      <td>11</td>\n",
       "      <td>3</td>\n",
       "      <td>18</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Breakfast</td>\n",
       "      <td>Sausage Biscuit with Egg Whites (Regular Biscuit)</td>\n",
       "      <td>5.9 oz (167 g)</td>\n",
       "      <td>460</td>\n",
       "      <td>250</td>\n",
       "      <td>27.0</td>\n",
       "      <td>42</td>\n",
       "      <td>12.0</td>\n",
       "      <td>62</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>34</td>\n",
       "      <td>11</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>18</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Breakfast</td>\n",
       "      <td>Sausage Biscuit with Egg Whites (Large Biscuit)</td>\n",
       "      <td>6.4 oz (181 g)</td>\n",
       "      <td>520</td>\n",
       "      <td>280</td>\n",
       "      <td>32.0</td>\n",
       "      <td>49</td>\n",
       "      <td>13.0</td>\n",
       "      <td>65</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>40</td>\n",
       "      <td>13</td>\n",
       "      <td>3</td>\n",
       "      <td>11</td>\n",
       "      <td>3</td>\n",
       "      <td>18</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Breakfast</td>\n",
       "      <td>Southern Style Chicken Biscuit (Regular Biscuit)</td>\n",
       "      <td>5 oz (143 g)</td>\n",
       "      <td>410</td>\n",
       "      <td>180</td>\n",
       "      <td>20.0</td>\n",
       "      <td>31</td>\n",
       "      <td>8.0</td>\n",
       "      <td>41</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>41</td>\n",
       "      <td>14</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>17</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Breakfast</td>\n",
       "      <td>Southern Style Chicken Biscuit (Large Biscuit)</td>\n",
       "      <td>5.5 oz (157 g)</td>\n",
       "      <td>470</td>\n",
       "      <td>220</td>\n",
       "      <td>24.0</td>\n",
       "      <td>37</td>\n",
       "      <td>9.0</td>\n",
       "      <td>45</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>46</td>\n",
       "      <td>15</td>\n",
       "      <td>3</td>\n",
       "      <td>11</td>\n",
       "      <td>4</td>\n",
       "      <td>17</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Breakfast</td>\n",
       "      <td>Steak &amp; Egg Biscuit (Regular Biscuit)</td>\n",
       "      <td>7.1 oz (201 g)</td>\n",
       "      <td>540</td>\n",
       "      <td>290</td>\n",
       "      <td>32.0</td>\n",
       "      <td>49</td>\n",
       "      <td>16.0</td>\n",
       "      <td>78</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>38</td>\n",
       "      <td>13</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>3</td>\n",
       "      <td>25</td>\n",
       "      <td>10</td>\n",
       "      <td>2</td>\n",
       "      <td>20</td>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Breakfast</td>\n",
       "      <td>Bacon, Egg &amp; Cheese McGriddles</td>\n",
       "      <td>6.1 oz (174 g)</td>\n",
       "      <td>460</td>\n",
       "      <td>190</td>\n",
       "      <td>21.0</td>\n",
       "      <td>32</td>\n",
       "      <td>9.0</td>\n",
       "      <td>44</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>48</td>\n",
       "      <td>16</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>15</td>\n",
       "      <td>19</td>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "      <td>20</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Breakfast</td>\n",
       "      <td>Bacon, Egg &amp; Cheese McGriddles with Egg Whites</td>\n",
       "      <td>6.3 oz (178 g)</td>\n",
       "      <td>400</td>\n",
       "      <td>140</td>\n",
       "      <td>15.0</td>\n",
       "      <td>24</td>\n",
       "      <td>7.0</td>\n",
       "      <td>34</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>47</td>\n",
       "      <td>16</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>16</td>\n",
       "      <td>20</td>\n",
       "      <td>2</td>\n",
       "      <td>10</td>\n",
       "      <td>15</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Breakfast</td>\n",
       "      <td>Sausage McGriddles</td>\n",
       "      <td>5 oz (141 g)</td>\n",
       "      <td>420</td>\n",
       "      <td>200</td>\n",
       "      <td>22.0</td>\n",
       "      <td>34</td>\n",
       "      <td>8.0</td>\n",
       "      <td>40</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>44</td>\n",
       "      <td>15</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>15</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Breakfast</td>\n",
       "      <td>Sausage, Egg &amp; Cheese McGriddles</td>\n",
       "      <td>7.1 oz (201 g)</td>\n",
       "      <td>550</td>\n",
       "      <td>280</td>\n",
       "      <td>31.0</td>\n",
       "      <td>48</td>\n",
       "      <td>12.0</td>\n",
       "      <td>61</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>48</td>\n",
       "      <td>16</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>15</td>\n",
       "      <td>20</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Breakfast</td>\n",
       "      <td>Sausage, Egg &amp; Cheese McGriddles with Egg Whites</td>\n",
       "      <td>7.2 oz (205 g)</td>\n",
       "      <td>500</td>\n",
       "      <td>230</td>\n",
       "      <td>26.0</td>\n",
       "      <td>40</td>\n",
       "      <td>10.0</td>\n",
       "      <td>52</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>46</td>\n",
       "      <td>15</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>15</td>\n",
       "      <td>21</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>20</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Breakfast</td>\n",
       "      <td>Bacon, Egg &amp; Cheese Bagel</td>\n",
       "      <td>6.9 oz (197 g)</td>\n",
       "      <td>620</td>\n",
       "      <td>280</td>\n",
       "      <td>31.0</td>\n",
       "      <td>48</td>\n",
       "      <td>11.0</td>\n",
       "      <td>56</td>\n",
       "      <td>0.5</td>\n",
       "      <td>...</td>\n",
       "      <td>57</td>\n",
       "      <td>19</td>\n",
       "      <td>3</td>\n",
       "      <td>11</td>\n",
       "      <td>7</td>\n",
       "      <td>30</td>\n",
       "      <td>20</td>\n",
       "      <td>15</td>\n",
       "      <td>20</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>Breakfast</td>\n",
       "      <td>Bacon, Egg &amp; Cheese Bagel with Egg Whites</td>\n",
       "      <td>7.1 oz (201 g)</td>\n",
       "      <td>570</td>\n",
       "      <td>230</td>\n",
       "      <td>25.0</td>\n",
       "      <td>39</td>\n",
       "      <td>9.0</td>\n",
       "      <td>45</td>\n",
       "      <td>0.5</td>\n",
       "      <td>...</td>\n",
       "      <td>55</td>\n",
       "      <td>18</td>\n",
       "      <td>3</td>\n",
       "      <td>12</td>\n",
       "      <td>8</td>\n",
       "      <td>30</td>\n",
       "      <td>10</td>\n",
       "      <td>15</td>\n",
       "      <td>20</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>Breakfast</td>\n",
       "      <td>Steak, Egg &amp; Cheese Bagel</td>\n",
       "      <td>8.5 oz (241 g)</td>\n",
       "      <td>670</td>\n",
       "      <td>310</td>\n",
       "      <td>35.0</td>\n",
       "      <td>53</td>\n",
       "      <td>13.0</td>\n",
       "      <td>63</td>\n",
       "      <td>1.5</td>\n",
       "      <td>...</td>\n",
       "      <td>56</td>\n",
       "      <td>19</td>\n",
       "      <td>3</td>\n",
       "      <td>12</td>\n",
       "      <td>7</td>\n",
       "      <td>33</td>\n",
       "      <td>20</td>\n",
       "      <td>4</td>\n",
       "      <td>25</td>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Breakfast</td>\n",
       "      <td>Big Breakfast (Regular Biscuit)</td>\n",
       "      <td>9.5 oz (269 g)</td>\n",
       "      <td>740</td>\n",
       "      <td>430</td>\n",
       "      <td>48.0</td>\n",
       "      <td>73</td>\n",
       "      <td>17.0</td>\n",
       "      <td>87</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>51</td>\n",
       "      <td>17</td>\n",
       "      <td>3</td>\n",
       "      <td>12</td>\n",
       "      <td>3</td>\n",
       "      <td>28</td>\n",
       "      <td>15</td>\n",
       "      <td>2</td>\n",
       "      <td>15</td>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>Breakfast</td>\n",
       "      <td>Big Breakfast (Large Biscuit)</td>\n",
       "      <td>10 oz (283 g)</td>\n",
       "      <td>800</td>\n",
       "      <td>470</td>\n",
       "      <td>52.0</td>\n",
       "      <td>80</td>\n",
       "      <td>18.0</td>\n",
       "      <td>90</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>56</td>\n",
       "      <td>19</td>\n",
       "      <td>4</td>\n",
       "      <td>17</td>\n",
       "      <td>3</td>\n",
       "      <td>28</td>\n",
       "      <td>15</td>\n",
       "      <td>2</td>\n",
       "      <td>15</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>Breakfast</td>\n",
       "      <td>Big Breakfast with Egg Whites (Regular Biscuit)</td>\n",
       "      <td>9.6 oz (272 g)</td>\n",
       "      <td>640</td>\n",
       "      <td>330</td>\n",
       "      <td>37.0</td>\n",
       "      <td>57</td>\n",
       "      <td>14.0</td>\n",
       "      <td>69</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>50</td>\n",
       "      <td>17</td>\n",
       "      <td>3</td>\n",
       "      <td>12</td>\n",
       "      <td>3</td>\n",
       "      <td>26</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>10</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>230</th>\n",
       "      <td>Coffee &amp; Tea</td>\n",
       "      <td>Frappé Chocolate Chip (Medium)</td>\n",
       "      <td>16 fl oz cup</td>\n",
       "      <td>630</td>\n",
       "      <td>240</td>\n",
       "      <td>26.0</td>\n",
       "      <td>41</td>\n",
       "      <td>17.0</td>\n",
       "      <td>85</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>91</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>81</td>\n",
       "      <td>9</td>\n",
       "      <td>15</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>231</th>\n",
       "      <td>Coffee &amp; Tea</td>\n",
       "      <td>Frappé Chocolate Chip (Large)</td>\n",
       "      <td>22 fl oz cup</td>\n",
       "      <td>760</td>\n",
       "      <td>280</td>\n",
       "      <td>31.0</td>\n",
       "      <td>48</td>\n",
       "      <td>20.0</td>\n",
       "      <td>101</td>\n",
       "      <td>1.5</td>\n",
       "      <td>...</td>\n",
       "      <td>111</td>\n",
       "      <td>37</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>99</td>\n",
       "      <td>12</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>35</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>232</th>\n",
       "      <td>Smoothies &amp; Shakes</td>\n",
       "      <td>Blueberry Pomegranate Smoothie (Small)</td>\n",
       "      <td>12 fl oz cup</td>\n",
       "      <td>220</td>\n",
       "      <td>5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>50</td>\n",
       "      <td>17</td>\n",
       "      <td>3</td>\n",
       "      <td>12</td>\n",
       "      <td>44</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>233</th>\n",
       "      <td>Smoothies &amp; Shakes</td>\n",
       "      <td>Blueberry Pomegranate Smoothie (Medium)</td>\n",
       "      <td>16 fl oz cup</td>\n",
       "      <td>260</td>\n",
       "      <td>5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>62</td>\n",
       "      <td>21</td>\n",
       "      <td>4</td>\n",
       "      <td>15</td>\n",
       "      <td>54</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>234</th>\n",
       "      <td>Smoothies &amp; Shakes</td>\n",
       "      <td>Blueberry Pomegranate Smoothie (Large)</td>\n",
       "      <td>22 fl oz cup</td>\n",
       "      <td>340</td>\n",
       "      <td>10</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.5</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>79</td>\n",
       "      <td>26</td>\n",
       "      <td>5</td>\n",
       "      <td>19</td>\n",
       "      <td>70</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>235</th>\n",
       "      <td>Smoothies &amp; Shakes</td>\n",
       "      <td>Strawberry Banana Smoothie (Small)</td>\n",
       "      <td>12 fl oz cup</td>\n",
       "      <td>210</td>\n",
       "      <td>5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>47</td>\n",
       "      <td>16</td>\n",
       "      <td>3</td>\n",
       "      <td>10</td>\n",
       "      <td>44</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>236</th>\n",
       "      <td>Smoothies &amp; Shakes</td>\n",
       "      <td>Strawberry Banana Smoothie (Medium)</td>\n",
       "      <td>16 fl oz cup</td>\n",
       "      <td>250</td>\n",
       "      <td>5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>58</td>\n",
       "      <td>19</td>\n",
       "      <td>3</td>\n",
       "      <td>13</td>\n",
       "      <td>54</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>35</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>237</th>\n",
       "      <td>Smoothies &amp; Shakes</td>\n",
       "      <td>Strawberry Banana Smoothie (Large)</td>\n",
       "      <td>22 fl oz cup</td>\n",
       "      <td>330</td>\n",
       "      <td>10</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.5</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>74</td>\n",
       "      <td>25</td>\n",
       "      <td>4</td>\n",
       "      <td>16</td>\n",
       "      <td>70</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>45</td>\n",
       "      <td>10</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>238</th>\n",
       "      <td>Smoothies &amp; Shakes</td>\n",
       "      <td>Mango Pineapple Smoothie (Small)</td>\n",
       "      <td>12 fl oz cup</td>\n",
       "      <td>210</td>\n",
       "      <td>5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>50</td>\n",
       "      <td>17</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>46</td>\n",
       "      <td>2</td>\n",
       "      <td>30</td>\n",
       "      <td>20</td>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>239</th>\n",
       "      <td>Smoothies &amp; Shakes</td>\n",
       "      <td>Mango Pineapple Smoothie (Medium)</td>\n",
       "      <td>16 fl oz cup</td>\n",
       "      <td>260</td>\n",
       "      <td>10</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>61</td>\n",
       "      <td>20</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>56</td>\n",
       "      <td>3</td>\n",
       "      <td>40</td>\n",
       "      <td>25</td>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>240</th>\n",
       "      <td>Smoothies &amp; Shakes</td>\n",
       "      <td>Mango Pineapple Smoothie (Large)</td>\n",
       "      <td>22 fl oz cup</td>\n",
       "      <td>340</td>\n",
       "      <td>10</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.5</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>78</td>\n",
       "      <td>26</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>72</td>\n",
       "      <td>4</td>\n",
       "      <td>50</td>\n",
       "      <td>30</td>\n",
       "      <td>10</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>241</th>\n",
       "      <td>Smoothies &amp; Shakes</td>\n",
       "      <td>Vanilla Shake (Small)</td>\n",
       "      <td>12 fl oz cup</td>\n",
       "      <td>530</td>\n",
       "      <td>140</td>\n",
       "      <td>15.0</td>\n",
       "      <td>24</td>\n",
       "      <td>10.0</td>\n",
       "      <td>49</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>86</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>63</td>\n",
       "      <td>11</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>242</th>\n",
       "      <td>Smoothies &amp; Shakes</td>\n",
       "      <td>Vanilla Shake (Medium)</td>\n",
       "      <td>16 fl oz cup</td>\n",
       "      <td>660</td>\n",
       "      <td>170</td>\n",
       "      <td>19.0</td>\n",
       "      <td>29</td>\n",
       "      <td>12.0</td>\n",
       "      <td>61</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>109</td>\n",
       "      <td>36</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>81</td>\n",
       "      <td>14</td>\n",
       "      <td>25</td>\n",
       "      <td>0</td>\n",
       "      <td>50</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>243</th>\n",
       "      <td>Smoothies &amp; Shakes</td>\n",
       "      <td>Vanilla Shake (Large)</td>\n",
       "      <td>22 fl oz cup</td>\n",
       "      <td>820</td>\n",
       "      <td>210</td>\n",
       "      <td>23.0</td>\n",
       "      <td>35</td>\n",
       "      <td>15.0</td>\n",
       "      <td>73</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>135</td>\n",
       "      <td>45</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>101</td>\n",
       "      <td>18</td>\n",
       "      <td>30</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>244</th>\n",
       "      <td>Smoothies &amp; Shakes</td>\n",
       "      <td>Strawberry Shake (Small)</td>\n",
       "      <td>12 fl oz cup</td>\n",
       "      <td>550</td>\n",
       "      <td>150</td>\n",
       "      <td>16.0</td>\n",
       "      <td>25</td>\n",
       "      <td>10.0</td>\n",
       "      <td>52</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>90</td>\n",
       "      <td>30</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>79</td>\n",
       "      <td>12</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>245</th>\n",
       "      <td>Smoothies &amp; Shakes</td>\n",
       "      <td>Strawberry Shake (Medium)</td>\n",
       "      <td>16 fl oz cup</td>\n",
       "      <td>690</td>\n",
       "      <td>180</td>\n",
       "      <td>20.0</td>\n",
       "      <td>30</td>\n",
       "      <td>13.0</td>\n",
       "      <td>63</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>114</td>\n",
       "      <td>38</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>15</td>\n",
       "      <td>25</td>\n",
       "      <td>0</td>\n",
       "      <td>50</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>246</th>\n",
       "      <td>Smoothies &amp; Shakes</td>\n",
       "      <td>Strawberry Shake (Large)</td>\n",
       "      <td>22 fl oz cup</td>\n",
       "      <td>850</td>\n",
       "      <td>210</td>\n",
       "      <td>24.0</td>\n",
       "      <td>36</td>\n",
       "      <td>15.0</td>\n",
       "      <td>75</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>140</td>\n",
       "      <td>47</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>123</td>\n",
       "      <td>18</td>\n",
       "      <td>30</td>\n",
       "      <td>0</td>\n",
       "      <td>70</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>247</th>\n",
       "      <td>Smoothies &amp; Shakes</td>\n",
       "      <td>Chocolate Shake (Small)</td>\n",
       "      <td>12 fl oz cup</td>\n",
       "      <td>560</td>\n",
       "      <td>150</td>\n",
       "      <td>16.0</td>\n",
       "      <td>25</td>\n",
       "      <td>10.0</td>\n",
       "      <td>51</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>91</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>77</td>\n",
       "      <td>12</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248</th>\n",
       "      <td>Smoothies &amp; Shakes</td>\n",
       "      <td>Chocolate Shake (Medium)</td>\n",
       "      <td>16 fl oz cup</td>\n",
       "      <td>700</td>\n",
       "      <td>180</td>\n",
       "      <td>20.0</td>\n",
       "      <td>30</td>\n",
       "      <td>12.0</td>\n",
       "      <td>62</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>114</td>\n",
       "      <td>38</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>97</td>\n",
       "      <td>15</td>\n",
       "      <td>25</td>\n",
       "      <td>0</td>\n",
       "      <td>50</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>249</th>\n",
       "      <td>Smoothies &amp; Shakes</td>\n",
       "      <td>Chocolate Shake (Large)</td>\n",
       "      <td>22 fl oz cup</td>\n",
       "      <td>850</td>\n",
       "      <td>210</td>\n",
       "      <td>23.0</td>\n",
       "      <td>36</td>\n",
       "      <td>15.0</td>\n",
       "      <td>74</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>141</td>\n",
       "      <td>47</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>120</td>\n",
       "      <td>19</td>\n",
       "      <td>30</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>250</th>\n",
       "      <td>Smoothies &amp; Shakes</td>\n",
       "      <td>Shamrock Shake (Medium)</td>\n",
       "      <td>16 fl oz cup</td>\n",
       "      <td>660</td>\n",
       "      <td>170</td>\n",
       "      <td>19.0</td>\n",
       "      <td>29</td>\n",
       "      <td>12.0</td>\n",
       "      <td>61</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>109</td>\n",
       "      <td>36</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>93</td>\n",
       "      <td>14</td>\n",
       "      <td>25</td>\n",
       "      <td>0</td>\n",
       "      <td>50</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>251</th>\n",
       "      <td>Smoothies &amp; Shakes</td>\n",
       "      <td>Shamrock Shake (Large)</td>\n",
       "      <td>22 fl oz cup</td>\n",
       "      <td>820</td>\n",
       "      <td>210</td>\n",
       "      <td>23.0</td>\n",
       "      <td>35</td>\n",
       "      <td>15.0</td>\n",
       "      <td>73</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>135</td>\n",
       "      <td>45</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>115</td>\n",
       "      <td>18</td>\n",
       "      <td>30</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>252</th>\n",
       "      <td>Smoothies &amp; Shakes</td>\n",
       "      <td>McFlurry with M&amp;M’s Candies (Small)</td>\n",
       "      <td>10.9 oz (310 g)</td>\n",
       "      <td>650</td>\n",
       "      <td>210</td>\n",
       "      <td>23.0</td>\n",
       "      <td>35</td>\n",
       "      <td>14.0</td>\n",
       "      <td>72</td>\n",
       "      <td>0.5</td>\n",
       "      <td>...</td>\n",
       "      <td>96</td>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>89</td>\n",
       "      <td>13</td>\n",
       "      <td>15</td>\n",
       "      <td>0</td>\n",
       "      <td>45</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>253</th>\n",
       "      <td>Smoothies &amp; Shakes</td>\n",
       "      <td>McFlurry with M&amp;M’s Candies (Medium)</td>\n",
       "      <td>16.2 oz (460 g)</td>\n",
       "      <td>930</td>\n",
       "      <td>290</td>\n",
       "      <td>33.0</td>\n",
       "      <td>50</td>\n",
       "      <td>20.0</td>\n",
       "      <td>102</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>139</td>\n",
       "      <td>46</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>128</td>\n",
       "      <td>20</td>\n",
       "      <td>25</td>\n",
       "      <td>0</td>\n",
       "      <td>70</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>254</th>\n",
       "      <td>Smoothies &amp; Shakes</td>\n",
       "      <td>McFlurry with M&amp;M’s Candies (Snack)</td>\n",
       "      <td>7.3 oz (207 g)</td>\n",
       "      <td>430</td>\n",
       "      <td>140</td>\n",
       "      <td>15.0</td>\n",
       "      <td>24</td>\n",
       "      <td>10.0</td>\n",
       "      <td>48</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>64</td>\n",
       "      <td>21</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>59</td>\n",
       "      <td>9</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>255</th>\n",
       "      <td>Smoothies &amp; Shakes</td>\n",
       "      <td>McFlurry with Oreo Cookies (Small)</td>\n",
       "      <td>10.1 oz (285 g)</td>\n",
       "      <td>510</td>\n",
       "      <td>150</td>\n",
       "      <td>17.0</td>\n",
       "      <td>26</td>\n",
       "      <td>9.0</td>\n",
       "      <td>44</td>\n",
       "      <td>0.5</td>\n",
       "      <td>...</td>\n",
       "      <td>80</td>\n",
       "      <td>27</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>64</td>\n",
       "      <td>12</td>\n",
       "      <td>15</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>256</th>\n",
       "      <td>Smoothies &amp; Shakes</td>\n",
       "      <td>McFlurry with Oreo Cookies (Medium)</td>\n",
       "      <td>13.4 oz (381 g)</td>\n",
       "      <td>690</td>\n",
       "      <td>200</td>\n",
       "      <td>23.0</td>\n",
       "      <td>35</td>\n",
       "      <td>12.0</td>\n",
       "      <td>58</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>106</td>\n",
       "      <td>35</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>85</td>\n",
       "      <td>15</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>50</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>257</th>\n",
       "      <td>Smoothies &amp; Shakes</td>\n",
       "      <td>McFlurry with Oreo Cookies (Snack)</td>\n",
       "      <td>6.7 oz (190 g)</td>\n",
       "      <td>340</td>\n",
       "      <td>100</td>\n",
       "      <td>11.0</td>\n",
       "      <td>17</td>\n",
       "      <td>6.0</td>\n",
       "      <td>29</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>53</td>\n",
       "      <td>18</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>43</td>\n",
       "      <td>8</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>25</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>258</th>\n",
       "      <td>Smoothies &amp; Shakes</td>\n",
       "      <td>McFlurry with Reese's Peanut Butter Cups (Medium)</td>\n",
       "      <td>14.2 oz (403 g)</td>\n",
       "      <td>810</td>\n",
       "      <td>290</td>\n",
       "      <td>32.0</td>\n",
       "      <td>50</td>\n",
       "      <td>15.0</td>\n",
       "      <td>76</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>114</td>\n",
       "      <td>38</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>103</td>\n",
       "      <td>21</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>60</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>259</th>\n",
       "      <td>Smoothies &amp; Shakes</td>\n",
       "      <td>McFlurry with Reese's Peanut Butter Cups (Snack)</td>\n",
       "      <td>7.1 oz (202 g)</td>\n",
       "      <td>410</td>\n",
       "      <td>150</td>\n",
       "      <td>16.0</td>\n",
       "      <td>25</td>\n",
       "      <td>8.0</td>\n",
       "      <td>38</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>57</td>\n",
       "      <td>19</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>51</td>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>30</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>260 rows × 24 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               Category                                               Item  \\\n",
       "0             Breakfast                                       Egg McMuffin   \n",
       "1             Breakfast                                  Egg White Delight   \n",
       "2             Breakfast                                   Sausage McMuffin   \n",
       "3             Breakfast                          Sausage McMuffin with Egg   \n",
       "4             Breakfast                   Sausage McMuffin with Egg Whites   \n",
       "5             Breakfast                               Steak & Egg McMuffin   \n",
       "6             Breakfast      Bacon, Egg & Cheese Biscuit (Regular Biscuit)   \n",
       "7             Breakfast        Bacon, Egg & Cheese Biscuit (Large Biscuit)   \n",
       "8             Breakfast  Bacon, Egg & Cheese Biscuit with Egg Whites (R...   \n",
       "9             Breakfast  Bacon, Egg & Cheese Biscuit with Egg Whites (L...   \n",
       "10            Breakfast                  Sausage Biscuit (Regular Biscuit)   \n",
       "11            Breakfast                    Sausage Biscuit (Large Biscuit)   \n",
       "12            Breakfast         Sausage Biscuit with Egg (Regular Biscuit)   \n",
       "13            Breakfast           Sausage Biscuit with Egg (Large Biscuit)   \n",
       "14            Breakfast  Sausage Biscuit with Egg Whites (Regular Biscuit)   \n",
       "15            Breakfast    Sausage Biscuit with Egg Whites (Large Biscuit)   \n",
       "16            Breakfast   Southern Style Chicken Biscuit (Regular Biscuit)   \n",
       "17            Breakfast     Southern Style Chicken Biscuit (Large Biscuit)   \n",
       "18            Breakfast              Steak & Egg Biscuit (Regular Biscuit)   \n",
       "19            Breakfast                     Bacon, Egg & Cheese McGriddles   \n",
       "20            Breakfast     Bacon, Egg & Cheese McGriddles with Egg Whites   \n",
       "21            Breakfast                                 Sausage McGriddles   \n",
       "22            Breakfast                   Sausage, Egg & Cheese McGriddles   \n",
       "23            Breakfast   Sausage, Egg & Cheese McGriddles with Egg Whites   \n",
       "24            Breakfast                          Bacon, Egg & Cheese Bagel   \n",
       "25            Breakfast          Bacon, Egg & Cheese Bagel with Egg Whites   \n",
       "26            Breakfast                          Steak, Egg & Cheese Bagel   \n",
       "27            Breakfast                    Big Breakfast (Regular Biscuit)   \n",
       "28            Breakfast                      Big Breakfast (Large Biscuit)   \n",
       "29            Breakfast    Big Breakfast with Egg Whites (Regular Biscuit)   \n",
       "..                  ...                                                ...   \n",
       "230        Coffee & Tea                     Frappé Chocolate Chip (Medium)   \n",
       "231        Coffee & Tea                      Frappé Chocolate Chip (Large)   \n",
       "232  Smoothies & Shakes             Blueberry Pomegranate Smoothie (Small)   \n",
       "233  Smoothies & Shakes            Blueberry Pomegranate Smoothie (Medium)   \n",
       "234  Smoothies & Shakes             Blueberry Pomegranate Smoothie (Large)   \n",
       "235  Smoothies & Shakes                 Strawberry Banana Smoothie (Small)   \n",
       "236  Smoothies & Shakes                Strawberry Banana Smoothie (Medium)   \n",
       "237  Smoothies & Shakes                 Strawberry Banana Smoothie (Large)   \n",
       "238  Smoothies & Shakes                   Mango Pineapple Smoothie (Small)   \n",
       "239  Smoothies & Shakes                  Mango Pineapple Smoothie (Medium)   \n",
       "240  Smoothies & Shakes                   Mango Pineapple Smoothie (Large)   \n",
       "241  Smoothies & Shakes                              Vanilla Shake (Small)   \n",
       "242  Smoothies & Shakes                             Vanilla Shake (Medium)   \n",
       "243  Smoothies & Shakes                              Vanilla Shake (Large)   \n",
       "244  Smoothies & Shakes                           Strawberry Shake (Small)   \n",
       "245  Smoothies & Shakes                          Strawberry Shake (Medium)   \n",
       "246  Smoothies & Shakes                           Strawberry Shake (Large)   \n",
       "247  Smoothies & Shakes                            Chocolate Shake (Small)   \n",
       "248  Smoothies & Shakes                           Chocolate Shake (Medium)   \n",
       "249  Smoothies & Shakes                            Chocolate Shake (Large)   \n",
       "250  Smoothies & Shakes                            Shamrock Shake (Medium)   \n",
       "251  Smoothies & Shakes                             Shamrock Shake (Large)   \n",
       "252  Smoothies & Shakes                McFlurry with M&M’s Candies (Small)   \n",
       "253  Smoothies & Shakes               McFlurry with M&M’s Candies (Medium)   \n",
       "254  Smoothies & Shakes                McFlurry with M&M’s Candies (Snack)   \n",
       "255  Smoothies & Shakes                 McFlurry with Oreo Cookies (Small)   \n",
       "256  Smoothies & Shakes                McFlurry with Oreo Cookies (Medium)   \n",
       "257  Smoothies & Shakes                 McFlurry with Oreo Cookies (Snack)   \n",
       "258  Smoothies & Shakes  McFlurry with Reese's Peanut Butter Cups (Medium)   \n",
       "259  Smoothies & Shakes   McFlurry with Reese's Peanut Butter Cups (Snack)   \n",
       "\n",
       "        Serving Size  Calories  Calories from Fat  Total Fat  \\\n",
       "0     4.8 oz (136 g)       300                120       13.0   \n",
       "1     4.8 oz (135 g)       250                 70        8.0   \n",
       "2     3.9 oz (111 g)       370                200       23.0   \n",
       "3     5.7 oz (161 g)       450                250       28.0   \n",
       "4     5.7 oz (161 g)       400                210       23.0   \n",
       "5     6.5 oz (185 g)       430                210       23.0   \n",
       "6     5.3 oz (150 g)       460                230       26.0   \n",
       "7     5.8 oz (164 g)       520                270       30.0   \n",
       "8     5.4 oz (153 g)       410                180       20.0   \n",
       "9     5.9 oz (167 g)       470                220       25.0   \n",
       "10    4.1 oz (117 g)       430                240       27.0   \n",
       "11    4.6 oz (131 g)       480                280       31.0   \n",
       "12    5.7 oz (163 g)       510                290       33.0   \n",
       "13    6.2 oz (177 g)       570                330       37.0   \n",
       "14    5.9 oz (167 g)       460                250       27.0   \n",
       "15    6.4 oz (181 g)       520                280       32.0   \n",
       "16      5 oz (143 g)       410                180       20.0   \n",
       "17    5.5 oz (157 g)       470                220       24.0   \n",
       "18    7.1 oz (201 g)       540                290       32.0   \n",
       "19    6.1 oz (174 g)       460                190       21.0   \n",
       "20    6.3 oz (178 g)       400                140       15.0   \n",
       "21      5 oz (141 g)       420                200       22.0   \n",
       "22    7.1 oz (201 g)       550                280       31.0   \n",
       "23    7.2 oz (205 g)       500                230       26.0   \n",
       "24    6.9 oz (197 g)       620                280       31.0   \n",
       "25    7.1 oz (201 g)       570                230       25.0   \n",
       "26    8.5 oz (241 g)       670                310       35.0   \n",
       "27    9.5 oz (269 g)       740                430       48.0   \n",
       "28     10 oz (283 g)       800                470       52.0   \n",
       "29    9.6 oz (272 g)       640                330       37.0   \n",
       "..               ...       ...                ...        ...   \n",
       "230     16 fl oz cup       630                240       26.0   \n",
       "231     22 fl oz cup       760                280       31.0   \n",
       "232     12 fl oz cup       220                  5        0.5   \n",
       "233     16 fl oz cup       260                  5        1.0   \n",
       "234     22 fl oz cup       340                 10        1.0   \n",
       "235     12 fl oz cup       210                  5        0.5   \n",
       "236     16 fl oz cup       250                  5        1.0   \n",
       "237     22 fl oz cup       330                 10        1.0   \n",
       "238     12 fl oz cup       210                  5        0.5   \n",
       "239     16 fl oz cup       260                 10        1.0   \n",
       "240     22 fl oz cup       340                 10        1.0   \n",
       "241     12 fl oz cup       530                140       15.0   \n",
       "242     16 fl oz cup       660                170       19.0   \n",
       "243     22 fl oz cup       820                210       23.0   \n",
       "244     12 fl oz cup       550                150       16.0   \n",
       "245     16 fl oz cup       690                180       20.0   \n",
       "246     22 fl oz cup       850                210       24.0   \n",
       "247     12 fl oz cup       560                150       16.0   \n",
       "248     16 fl oz cup       700                180       20.0   \n",
       "249     22 fl oz cup       850                210       23.0   \n",
       "250     16 fl oz cup       660                170       19.0   \n",
       "251     22 fl oz cup       820                210       23.0   \n",
       "252  10.9 oz (310 g)       650                210       23.0   \n",
       "253  16.2 oz (460 g)       930                290       33.0   \n",
       "254   7.3 oz (207 g)       430                140       15.0   \n",
       "255  10.1 oz (285 g)       510                150       17.0   \n",
       "256  13.4 oz (381 g)       690                200       23.0   \n",
       "257   6.7 oz (190 g)       340                100       11.0   \n",
       "258  14.2 oz (403 g)       810                290       32.0   \n",
       "259   7.1 oz (202 g)       410                150       16.0   \n",
       "\n",
       "     Total Fat (% Daily Value)  Saturated Fat  Saturated Fat (% Daily Value)  \\\n",
       "0                           20            5.0                             25   \n",
       "1                           12            3.0                             15   \n",
       "2                           35            8.0                             42   \n",
       "3                           43           10.0                             52   \n",
       "4                           35            8.0                             42   \n",
       "5                           36            9.0                             46   \n",
       "6                           40           13.0                             65   \n",
       "7                           47           14.0                             68   \n",
       "8                           32           11.0                             56   \n",
       "9                           38           12.0                             59   \n",
       "10                          42           12.0                             62   \n",
       "11                          48           13.0                             65   \n",
       "12                          50           14.0                             71   \n",
       "13                          57           15.0                             74   \n",
       "14                          42           12.0                             62   \n",
       "15                          49           13.0                             65   \n",
       "16                          31            8.0                             41   \n",
       "17                          37            9.0                             45   \n",
       "18                          49           16.0                             78   \n",
       "19                          32            9.0                             44   \n",
       "20                          24            7.0                             34   \n",
       "21                          34            8.0                             40   \n",
       "22                          48           12.0                             61   \n",
       "23                          40           10.0                             52   \n",
       "24                          48           11.0                             56   \n",
       "25                          39            9.0                             45   \n",
       "26                          53           13.0                             63   \n",
       "27                          73           17.0                             87   \n",
       "28                          80           18.0                             90   \n",
       "29                          57           14.0                             69   \n",
       "..                         ...            ...                            ...   \n",
       "230                         41           17.0                             85   \n",
       "231                         48           20.0                            101   \n",
       "232                          1            0.0                              0   \n",
       "233                          1            0.0                              0   \n",
       "234                          2            0.5                              3   \n",
       "235                          1            0.0                              0   \n",
       "236                          1            0.0                              0   \n",
       "237                          2            0.5                              3   \n",
       "238                          1            0.0                              0   \n",
       "239                          1            0.0                              0   \n",
       "240                          2            0.5                              3   \n",
       "241                         24           10.0                             49   \n",
       "242                         29           12.0                             61   \n",
       "243                         35           15.0                             73   \n",
       "244                         25           10.0                             52   \n",
       "245                         30           13.0                             63   \n",
       "246                         36           15.0                             75   \n",
       "247                         25           10.0                             51   \n",
       "248                         30           12.0                             62   \n",
       "249                         36           15.0                             74   \n",
       "250                         29           12.0                             61   \n",
       "251                         35           15.0                             73   \n",
       "252                         35           14.0                             72   \n",
       "253                         50           20.0                            102   \n",
       "254                         24           10.0                             48   \n",
       "255                         26            9.0                             44   \n",
       "256                         35           12.0                             58   \n",
       "257                         17            6.0                             29   \n",
       "258                         50           15.0                             76   \n",
       "259                         25            8.0                             38   \n",
       "\n",
       "     Trans Fat          ...           Carbohydrates  \\\n",
       "0          0.0          ...                      31   \n",
       "1          0.0          ...                      30   \n",
       "2          0.0          ...                      29   \n",
       "3          0.0          ...                      30   \n",
       "4          0.0          ...                      30   \n",
       "5          1.0          ...                      31   \n",
       "6          0.0          ...                      38   \n",
       "7          0.0          ...                      43   \n",
       "8          0.0          ...                      36   \n",
       "9          0.0          ...                      42   \n",
       "10         0.0          ...                      34   \n",
       "11         0.0          ...                      39   \n",
       "12         0.0          ...                      36   \n",
       "13         0.0          ...                      42   \n",
       "14         0.0          ...                      34   \n",
       "15         0.0          ...                      40   \n",
       "16         0.0          ...                      41   \n",
       "17         0.0          ...                      46   \n",
       "18         1.0          ...                      38   \n",
       "19         0.0          ...                      48   \n",
       "20         0.0          ...                      47   \n",
       "21         0.0          ...                      44   \n",
       "22         0.0          ...                      48   \n",
       "23         0.0          ...                      46   \n",
       "24         0.5          ...                      57   \n",
       "25         0.5          ...                      55   \n",
       "26         1.5          ...                      56   \n",
       "27         0.0          ...                      51   \n",
       "28         0.0          ...                      56   \n",
       "29         0.0          ...                      50   \n",
       "..         ...          ...                     ...   \n",
       "230        1.0          ...                      91   \n",
       "231        1.5          ...                     111   \n",
       "232        0.0          ...                      50   \n",
       "233        0.0          ...                      62   \n",
       "234        0.0          ...                      79   \n",
       "235        0.0          ...                      47   \n",
       "236        0.0          ...                      58   \n",
       "237        0.0          ...                      74   \n",
       "238        0.0          ...                      50   \n",
       "239        0.0          ...                      61   \n",
       "240        0.0          ...                      78   \n",
       "241        1.0          ...                      86   \n",
       "242        1.0          ...                     109   \n",
       "243        1.0          ...                     135   \n",
       "244        1.0          ...                      90   \n",
       "245        1.0          ...                     114   \n",
       "246        1.0          ...                     140   \n",
       "247        1.0          ...                      91   \n",
       "248        1.0          ...                     114   \n",
       "249        1.0          ...                     141   \n",
       "250        1.0          ...                     109   \n",
       "251        1.0          ...                     135   \n",
       "252        0.5          ...                      96   \n",
       "253        1.0          ...                     139   \n",
       "254        0.0          ...                      64   \n",
       "255        0.5          ...                      80   \n",
       "256        1.0          ...                     106   \n",
       "257        0.0          ...                      53   \n",
       "258        1.0          ...                     114   \n",
       "259        0.0          ...                      57   \n",
       "\n",
       "     Carbohydrates (% Daily Value)  Dietary Fiber  \\\n",
       "0                               10              4   \n",
       "1                               10              4   \n",
       "2                               10              4   \n",
       "3                               10              4   \n",
       "4                               10              4   \n",
       "5                               10              4   \n",
       "6                               13              2   \n",
       "7                               14              3   \n",
       "8                               12              2   \n",
       "9                               14              3   \n",
       "10                              11              2   \n",
       "11                              13              3   \n",
       "12                              12              2   \n",
       "13                              14              3   \n",
       "14                              11              2   \n",
       "15                              13              3   \n",
       "16                              14              2   \n",
       "17                              15              3   \n",
       "18                              13              2   \n",
       "19                              16              2   \n",
       "20                              16              2   \n",
       "21                              15              2   \n",
       "22                              16              2   \n",
       "23                              15              2   \n",
       "24                              19              3   \n",
       "25                              18              3   \n",
       "26                              19              3   \n",
       "27                              17              3   \n",
       "28                              19              4   \n",
       "29                              17              3   \n",
       "..                             ...            ...   \n",
       "230                             30              1   \n",
       "231                             37              1   \n",
       "232                             17              3   \n",
       "233                             21              4   \n",
       "234                             26              5   \n",
       "235                             16              3   \n",
       "236                             19              3   \n",
       "237                             25              4   \n",
       "238                             17              1   \n",
       "239                             20              1   \n",
       "240                             26              2   \n",
       "241                             29              0   \n",
       "242                             36              0   \n",
       "243                             45              0   \n",
       "244                             30              0   \n",
       "245                             38              0   \n",
       "246                             47              0   \n",
       "247                             30              1   \n",
       "248                             38              2   \n",
       "249                             47              2   \n",
       "250                             36              0   \n",
       "251                             45              0   \n",
       "252                             32              1   \n",
       "253                             46              2   \n",
       "254                             21              1   \n",
       "255                             27              1   \n",
       "256                             35              1   \n",
       "257                             18              1   \n",
       "258                             38              2   \n",
       "259                             19              1   \n",
       "\n",
       "     Dietary Fiber (% Daily Value)  Sugars  Protein  \\\n",
       "0                               17       3       17   \n",
       "1                               17       3       18   \n",
       "2                               17       2       14   \n",
       "3                               17       2       21   \n",
       "4                               17       2       21   \n",
       "5                               18       3       26   \n",
       "6                                7       3       19   \n",
       "7                               12       4       19   \n",
       "8                                7       3       20   \n",
       "9                               12       4       20   \n",
       "10                               6       2       11   \n",
       "11                              11       3       11   \n",
       "12                               6       2       18   \n",
       "13                              11       3       18   \n",
       "14                               6       3       18   \n",
       "15                              11       3       18   \n",
       "16                               6       3       17   \n",
       "17                              11       4       17   \n",
       "18                               8       3       25   \n",
       "19                               9      15       19   \n",
       "20                               9      16       20   \n",
       "21                               8      15       11   \n",
       "22                               9      15       20   \n",
       "23                               9      15       21   \n",
       "24                              11       7       30   \n",
       "25                              12       8       30   \n",
       "26                              12       7       33   \n",
       "27                              12       3       28   \n",
       "28                              17       3       28   \n",
       "29                              12       3       26   \n",
       "..                             ...     ...      ...   \n",
       "230                              5      81        9   \n",
       "231                              5      99       12   \n",
       "232                             12      44        2   \n",
       "233                             15      54        3   \n",
       "234                             19      70        4   \n",
       "235                             10      44        3   \n",
       "236                             13      54        4   \n",
       "237                             16      70        5   \n",
       "238                              4      46        2   \n",
       "239                              5      56        3   \n",
       "240                              6      72        4   \n",
       "241                              0      63       11   \n",
       "242                              0      81       14   \n",
       "243                              0     101       18   \n",
       "244                              0      79       12   \n",
       "245                              0     100       15   \n",
       "246                              0     123       18   \n",
       "247                              5      77       12   \n",
       "248                              6      97       15   \n",
       "249                              8     120       19   \n",
       "250                              0      93       14   \n",
       "251                              0     115       18   \n",
       "252                              6      89       13   \n",
       "253                              7     128       20   \n",
       "254                              4      59        9   \n",
       "255                              4      64       12   \n",
       "256                              5      85       15   \n",
       "257                              2      43        8   \n",
       "258                              9     103       21   \n",
       "259                              5      51       10   \n",
       "\n",
       "     Vitamin A (% Daily Value)  Vitamin C (% Daily Value)  \\\n",
       "0                           10                          0   \n",
       "1                            6                          0   \n",
       "2                            8                          0   \n",
       "3                           15                          0   \n",
       "4                            6                          0   \n",
       "5                           15                          2   \n",
       "6                           10                          8   \n",
       "7                           15                          8   \n",
       "8                            2                          8   \n",
       "9                            6                          8   \n",
       "10                           0                          0   \n",
       "11                           4                          0   \n",
       "12                           6                          0   \n",
       "13                          10                          0   \n",
       "14                           0                          0   \n",
       "15                           4                          0   \n",
       "16                           0                          2   \n",
       "17                           4                          2   \n",
       "18                          10                          2   \n",
       "19                          10                         10   \n",
       "20                           2                         10   \n",
       "21                           0                          0   \n",
       "22                          10                          0   \n",
       "23                           2                          0   \n",
       "24                          20                         15   \n",
       "25                          10                         15   \n",
       "26                          20                          4   \n",
       "27                          15                          2   \n",
       "28                          15                          2   \n",
       "29                           0                          2   \n",
       "..                         ...                        ...   \n",
       "230                         15                          0   \n",
       "231                         20                          0   \n",
       "232                          0                          2   \n",
       "233                          0                          4   \n",
       "234                          0                          4   \n",
       "235                          0                         30   \n",
       "236                          0                         35   \n",
       "237                          0                         45   \n",
       "238                         30                         20   \n",
       "239                         40                         25   \n",
       "240                         50                         30   \n",
       "241                         20                          0   \n",
       "242                         25                          0   \n",
       "243                         30                          0   \n",
       "244                         20                          0   \n",
       "245                         25                          0   \n",
       "246                         30                          0   \n",
       "247                         20                          0   \n",
       "248                         25                          0   \n",
       "249                         30                          0   \n",
       "250                         25                          0   \n",
       "251                         30                          0   \n",
       "252                         15                          0   \n",
       "253                         25                          0   \n",
       "254                         10                          0   \n",
       "255                         15                          0   \n",
       "256                         20                          0   \n",
       "257                         10                          0   \n",
       "258                         20                          0   \n",
       "259                         10                          0   \n",
       "\n",
       "     Calcium (% Daily Value)  Iron (% Daily Value)  \n",
       "0                         25                    15  \n",
       "1                         25                     8  \n",
       "2                         25                    10  \n",
       "3                         30                    15  \n",
       "4                         25                    10  \n",
       "5                         30                    20  \n",
       "6                         15                    15  \n",
       "7                         20                    20  \n",
       "8                         15                    10  \n",
       "9                         15                    15  \n",
       "10                         6                    15  \n",
       "11                         8                    15  \n",
       "12                        10                    20  \n",
       "13                        10                    20  \n",
       "14                         8                    15  \n",
       "15                         8                    15  \n",
       "16                         6                    15  \n",
       "17                         8                    15  \n",
       "18                        20                    25  \n",
       "19                        20                    15  \n",
       "20                        15                    10  \n",
       "21                         8                    10  \n",
       "22                        20                    15  \n",
       "23                        20                    10  \n",
       "24                        20                    20  \n",
       "25                        20                    15  \n",
       "26                        25                    25  \n",
       "27                        15                    25  \n",
       "28                        15                    30  \n",
       "29                        10                    15  \n",
       "..                       ...                   ...  \n",
       "230                       30                     4  \n",
       "231                       35                     6  \n",
       "232                        6                     2  \n",
       "233                        8                     2  \n",
       "234                       10                     2  \n",
       "235                        8                     2  \n",
       "236                        8                     4  \n",
       "237                       10                     4  \n",
       "238                        8                     2  \n",
       "239                        8                     2  \n",
       "240                       10                     2  \n",
       "241                       40                     0  \n",
       "242                       50                     0  \n",
       "243                       60                     0  \n",
       "244                       40                     0  \n",
       "245                       50                     0  \n",
       "246                       70                     0  \n",
       "247                       40                     8  \n",
       "248                       50                    10  \n",
       "249                       60                    15  \n",
       "250                       50                     0  \n",
       "251                       60                     0  \n",
       "252                       45                     8  \n",
       "253                       70                    10  \n",
       "254                       30                     4  \n",
       "255                       40                     8  \n",
       "256                       50                    10  \n",
       "257                       25                     6  \n",
       "258                       60                     6  \n",
       "259                       30                     4  \n",
       "\n",
       "[260 rows x 24 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "menu = pd.read_csv('data/menu.csv') \n",
    "menu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That works great for well-formatted CSV files, but what happens when you get something that looks like the ```data/avocado_eu.csv``` file.\n",
    "Go ahead and browse that in JupyterLab's CSV browser.\n",
    "\n",
    "You'll notice a new drop-down menu labelled \"Delimiter\".  Go ahead and change that to ```;```.\n",
    "\n",
    "Referring back to your readings and the [read_csv documentation online](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html), complete the following exercise\n",
    "\n",
    "\n",
    "Read the data/avocado_eu.csv file into a pandas DataFrame and show the first 5 rows.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Date</th>\n",
       "      <th>AveragePrice</th>\n",
       "      <th>Total Volume</th>\n",
       "      <th>4046</th>\n",
       "      <th>4225</th>\n",
       "      <th>4770</th>\n",
       "      <th>Total Bags</th>\n",
       "      <th>Small Bags</th>\n",
       "      <th>Large Bags</th>\n",
       "      <th>XLarge Bags</th>\n",
       "      <th>type</th>\n",
       "      <th>year</th>\n",
       "      <th>region</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>2015-12-27</td>\n",
       "      <td>1,33</td>\n",
       "      <td>64236,62</td>\n",
       "      <td>1036,74</td>\n",
       "      <td>54454,85</td>\n",
       "      <td>48,16</td>\n",
       "      <td>8696,87</td>\n",
       "      <td>8603,62</td>\n",
       "      <td>93,25</td>\n",
       "      <td>0,0</td>\n",
       "      <td>conventional</td>\n",
       "      <td>2015</td>\n",
       "      <td>Albany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2015-12-20</td>\n",
       "      <td>1,35</td>\n",
       "      <td>54876,98</td>\n",
       "      <td>674,28</td>\n",
       "      <td>44638,81</td>\n",
       "      <td>58,33</td>\n",
       "      <td>9505,56</td>\n",
       "      <td>9408,07</td>\n",
       "      <td>97,49</td>\n",
       "      <td>0,0</td>\n",
       "      <td>conventional</td>\n",
       "      <td>2015</td>\n",
       "      <td>Albany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>2015-12-13</td>\n",
       "      <td>0,93</td>\n",
       "      <td>118220,22</td>\n",
       "      <td>794,7</td>\n",
       "      <td>109149,67</td>\n",
       "      <td>130,5</td>\n",
       "      <td>8145,35</td>\n",
       "      <td>8042,21</td>\n",
       "      <td>103,14</td>\n",
       "      <td>0,0</td>\n",
       "      <td>conventional</td>\n",
       "      <td>2015</td>\n",
       "      <td>Albany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>2015-12-06</td>\n",
       "      <td>1,08</td>\n",
       "      <td>78992,15</td>\n",
       "      <td>1132,0</td>\n",
       "      <td>71976,41</td>\n",
       "      <td>72,58</td>\n",
       "      <td>5811,16</td>\n",
       "      <td>5677,4</td>\n",
       "      <td>133,76</td>\n",
       "      <td>0,0</td>\n",
       "      <td>conventional</td>\n",
       "      <td>2015</td>\n",
       "      <td>Albany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>2015-11-29</td>\n",
       "      <td>1,28</td>\n",
       "      <td>51039,6</td>\n",
       "      <td>941,48</td>\n",
       "      <td>43838,39</td>\n",
       "      <td>75,78</td>\n",
       "      <td>6183,95</td>\n",
       "      <td>5986,26</td>\n",
       "      <td>197,69</td>\n",
       "      <td>0,0</td>\n",
       "      <td>conventional</td>\n",
       "      <td>2015</td>\n",
       "      <td>Albany</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Unnamed: 0        Date AveragePrice Total Volume     4046       4225   4770  \\\n",
       "0          0  2015-12-27         1,33     64236,62  1036,74   54454,85  48,16   \n",
       "1          1  2015-12-20         1,35     54876,98   674,28   44638,81  58,33   \n",
       "2          2  2015-12-13         0,93    118220,22    794,7  109149,67  130,5   \n",
       "3          3  2015-12-06         1,08     78992,15   1132,0   71976,41  72,58   \n",
       "4          4  2015-11-29         1,28      51039,6   941,48   43838,39  75,78   \n",
       "\n",
       "  Total Bags Small Bags Large Bags XLarge Bags          type  year  region  \n",
       "0    8696,87    8603,62      93,25         0,0  conventional  2015  Albany  \n",
       "1    9505,56    9408,07      97,49         0,0  conventional  2015  Albany  \n",
       "2    8145,35    8042,21     103,14         0,0  conventional  2015  Albany  \n",
       "3    5811,16     5677,4     133,76         0,0  conventional  2015  Albany  \n",
       "4    6183,95    5986,26     197,69         0,0  conventional  2015  Albany  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "avocado = pd.read_csv('data/avocado_eu.csv', delimiter=';')\n",
    "avocado.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll notice that, unless you did something special in the previous read_csv invocation, the decimal points don't look quite right.  Go ahead and find the right option to convert commas to periods when loading a CSV file.\n",
    "\n",
    "## <font color=\"magenta\">Q1 (2 point): \n",
    "Read the data/avocado_eu.csv file using the correct delimiter and decimal character into a dataframe and show the first 5 rows: </font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Date</th>\n",
       "      <th>AveragePrice</th>\n",
       "      <th>Total Volume</th>\n",
       "      <th>4046</th>\n",
       "      <th>4225</th>\n",
       "      <th>4770</th>\n",
       "      <th>Total Bags</th>\n",
       "      <th>Small Bags</th>\n",
       "      <th>Large Bags</th>\n",
       "      <th>XLarge Bags</th>\n",
       "      <th>type</th>\n",
       "      <th>year</th>\n",
       "      <th>region</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>2015-12-27</td>\n",
       "      <td>1.33</td>\n",
       "      <td>64236.62</td>\n",
       "      <td>1036.74</td>\n",
       "      <td>54454.85</td>\n",
       "      <td>48.16</td>\n",
       "      <td>8696.87</td>\n",
       "      <td>8603.62</td>\n",
       "      <td>93.25</td>\n",
       "      <td>0.0</td>\n",
       "      <td>conventional</td>\n",
       "      <td>2015</td>\n",
       "      <td>Albany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2015-12-20</td>\n",
       "      <td>1.35</td>\n",
       "      <td>54876.98</td>\n",
       "      <td>674.28</td>\n",
       "      <td>44638.81</td>\n",
       "      <td>58.33</td>\n",
       "      <td>9505.56</td>\n",
       "      <td>9408.07</td>\n",
       "      <td>97.49</td>\n",
       "      <td>0.0</td>\n",
       "      <td>conventional</td>\n",
       "      <td>2015</td>\n",
       "      <td>Albany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>2015-12-13</td>\n",
       "      <td>0.93</td>\n",
       "      <td>118220.22</td>\n",
       "      <td>794.70</td>\n",
       "      <td>109149.67</td>\n",
       "      <td>130.50</td>\n",
       "      <td>8145.35</td>\n",
       "      <td>8042.21</td>\n",
       "      <td>103.14</td>\n",
       "      <td>0.0</td>\n",
       "      <td>conventional</td>\n",
       "      <td>2015</td>\n",
       "      <td>Albany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>2015-12-06</td>\n",
       "      <td>1.08</td>\n",
       "      <td>78992.15</td>\n",
       "      <td>1132.00</td>\n",
       "      <td>71976.41</td>\n",
       "      <td>72.58</td>\n",
       "      <td>5811.16</td>\n",
       "      <td>5677.40</td>\n",
       "      <td>133.76</td>\n",
       "      <td>0.0</td>\n",
       "      <td>conventional</td>\n",
       "      <td>2015</td>\n",
       "      <td>Albany</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>2015-11-29</td>\n",
       "      <td>1.28</td>\n",
       "      <td>51039.60</td>\n",
       "      <td>941.48</td>\n",
       "      <td>43838.39</td>\n",
       "      <td>75.78</td>\n",
       "      <td>6183.95</td>\n",
       "      <td>5986.26</td>\n",
       "      <td>197.69</td>\n",
       "      <td>0.0</td>\n",
       "      <td>conventional</td>\n",
       "      <td>2015</td>\n",
       "      <td>Albany</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Unnamed: 0        Date  AveragePrice  Total Volume     4046       4225  \\\n",
       "0          0  2015-12-27          1.33      64236.62  1036.74   54454.85   \n",
       "1          1  2015-12-20          1.35      54876.98   674.28   44638.81   \n",
       "2          2  2015-12-13          0.93     118220.22   794.70  109149.67   \n",
       "3          3  2015-12-06          1.08      78992.15  1132.00   71976.41   \n",
       "4          4  2015-11-29          1.28      51039.60   941.48   43838.39   \n",
       "\n",
       "     4770  Total Bags  Small Bags  Large Bags  XLarge Bags          type  \\\n",
       "0   48.16     8696.87     8603.62       93.25          0.0  conventional   \n",
       "1   58.33     9505.56     9408.07       97.49          0.0  conventional   \n",
       "2  130.50     8145.35     8042.21      103.14          0.0  conventional   \n",
       "3   72.58     5811.16     5677.40      133.76          0.0  conventional   \n",
       "4   75.78     6183.95     5986.26      197.69          0.0  conventional   \n",
       "\n",
       "   year  region  \n",
       "0  2015  Albany  \n",
       "1  2015  Albany  \n",
       "2  2015  Albany  \n",
       "3  2015  Albany  \n",
       "4  2015  Albany  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Insert your code here\n",
    "avocado = pd.read_csv('data/avocado_eu.csv', delimiter=';',decimal = ',')\n",
    "avocado.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Counting the number of values\n",
    "\n",
    "Sometimes, you'll want to count the number of times values occur.  For example, we might want to know the number of times each 'type'\n",
    "is reported in our avocado data.  Use the ```value_counts()``` function on a Series to do so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "conventional    9126\n",
       "organic         9123\n",
       "Name: type, dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "avocado['type'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loading JSON data\n",
    "\n",
    "In addition to CSV files, JSON (JavaScript Object Notation) files or data is commonly used.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "nfl_football_players = pd.read_json('data/nfl_football_profiles.json')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birth_date</th>\n",
       "      <th>birth_place</th>\n",
       "      <th>college</th>\n",
       "      <th>current_salary</th>\n",
       "      <th>current_team</th>\n",
       "      <th>death_date</th>\n",
       "      <th>draft_position</th>\n",
       "      <th>draft_round</th>\n",
       "      <th>draft_team</th>\n",
       "      <th>draft_year</th>\n",
       "      <th>height</th>\n",
       "      <th>high_school</th>\n",
       "      <th>hof_induction_year</th>\n",
       "      <th>name</th>\n",
       "      <th>player_id</th>\n",
       "      <th>position</th>\n",
       "      <th>weight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1967-05-12</td>\n",
       "      <td>Bay City, TX</td>\n",
       "      <td>Baylor</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>34.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Seattle Seahawks</td>\n",
       "      <td>1990.0</td>\n",
       "      <td>6-0</td>\n",
       "      <td>Van Vleck, TX</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Robert Blackmon</td>\n",
       "      <td>1809</td>\n",
       "      <td>DB</td>\n",
       "      <td>208.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1970-07-20</td>\n",
       "      <td>Louisville, KY</td>\n",
       "      <td>Kentucky</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>85.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Seattle Seahawks</td>\n",
       "      <td>1993.0</td>\n",
       "      <td>6-3</td>\n",
       "      <td>Holy Cross, KY</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Dean Wells</td>\n",
       "      <td>23586</td>\n",
       "      <td>LB</td>\n",
       "      <td>248.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1990-08-14</td>\n",
       "      <td>Newton, MA</td>\n",
       "      <td>Oregon</td>\n",
       "      <td>1,075,000</td>\n",
       "      <td>Miami Dolphins</td>\n",
       "      <td>None</td>\n",
       "      <td>46.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Buffalo Bills</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>6-3</td>\n",
       "      <td>Los Gatos, CA</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Kiko Alonso</td>\n",
       "      <td>355</td>\n",
       "      <td>ILB</td>\n",
       "      <td>238.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1948-04-22</td>\n",
       "      <td>Dallas, TX</td>\n",
       "      <td>North Texas</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>1999-10-15</td>\n",
       "      <td>126.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>New Orleans Saints</td>\n",
       "      <td>1970.0</td>\n",
       "      <td>6-2</td>\n",
       "      <td>W.W. Samuell, TX</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Steve Ramsey</td>\n",
       "      <td>18182</td>\n",
       "      <td>QB</td>\n",
       "      <td>210.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1988-02-27</td>\n",
       "      <td>Neptune, NJ</td>\n",
       "      <td>Miami (FL)</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6-0</td>\n",
       "      <td>Neptune, NJ</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Cory Nelms</td>\n",
       "      <td>16250</td>\n",
       "      <td>CB</td>\n",
       "      <td>195.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   birth_date     birth_place      college current_salary    current_team  \\\n",
       "0  1967-05-12    Bay City, TX       Baylor           None            None   \n",
       "1  1970-07-20  Louisville, KY     Kentucky           None            None   \n",
       "2  1990-08-14      Newton, MA       Oregon      1,075,000  Miami Dolphins   \n",
       "3  1948-04-22      Dallas, TX  North Texas           None            None   \n",
       "4  1988-02-27     Neptune, NJ   Miami (FL)           None            None   \n",
       "\n",
       "   death_date  draft_position  draft_round          draft_team  draft_year  \\\n",
       "0        None            34.0          2.0    Seattle Seahawks      1990.0   \n",
       "1        None            85.0          4.0    Seattle Seahawks      1993.0   \n",
       "2        None            46.0          2.0       Buffalo Bills      2013.0   \n",
       "3  1999-10-15           126.0          5.0  New Orleans Saints      1970.0   \n",
       "4        None             NaN          NaN                None         NaN   \n",
       "\n",
       "  height       high_school  hof_induction_year             name  player_id  \\\n",
       "0    6-0     Van Vleck, TX                 NaN  Robert Blackmon       1809   \n",
       "1    6-3    Holy Cross, KY                 NaN       Dean Wells      23586   \n",
       "2    6-3     Los Gatos, CA                 NaN      Kiko Alonso        355   \n",
       "3    6-2  W.W. Samuell, TX                 NaN     Steve Ramsey      18182   \n",
       "4    6-0       Neptune, NJ                 NaN       Cory Nelms      16250   \n",
       "\n",
       "  position  weight  \n",
       "0       DB   208.0  \n",
       "1       LB   248.0  \n",
       "2      ILB   238.0  \n",
       "3       QB   210.0  \n",
       "4       CB   195.0  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nfl_football_players.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And, just for fun, show the player with the highest Current Salary from that dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birth_date</th>\n",
       "      <th>birth_place</th>\n",
       "      <th>college</th>\n",
       "      <th>current_salary</th>\n",
       "      <th>current_team</th>\n",
       "      <th>death_date</th>\n",
       "      <th>draft_position</th>\n",
       "      <th>draft_round</th>\n",
       "      <th>draft_team</th>\n",
       "      <th>draft_year</th>\n",
       "      <th>height</th>\n",
       "      <th>high_school</th>\n",
       "      <th>hof_induction_year</th>\n",
       "      <th>name</th>\n",
       "      <th>player_id</th>\n",
       "      <th>position</th>\n",
       "      <th>weight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6454</th>\n",
       "      <td>1993-01-17</td>\n",
       "      <td>Ibadan, Nigeria</td>\n",
       "      <td>Georgia Tech</td>\n",
       "      <td>993,150</td>\n",
       "      <td>Los Angeles Chargers</td>\n",
       "      <td>None</td>\n",
       "      <td>50.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>San Diego Chargers</td>\n",
       "      <td>2014.0</td>\n",
       "      <td>6-3</td>\n",
       "      <td>Archbishop Carroll, DE</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Jeremiah Attaochu</td>\n",
       "      <td>721</td>\n",
       "      <td>OLB</td>\n",
       "      <td>252.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      birth_date      birth_place       college current_salary  \\\n",
       "6454  1993-01-17  Ibadan, Nigeria  Georgia Tech        993,150   \n",
       "\n",
       "              current_team death_date  draft_position  draft_round  \\\n",
       "6454  Los Angeles Chargers       None            50.0          2.0   \n",
       "\n",
       "              draft_team  draft_year height             high_school  \\\n",
       "6454  San Diego Chargers      2014.0    6-3  Archbishop Carroll, DE   \n",
       "\n",
       "      hof_induction_year               name  player_id position  weight  \n",
       "6454                 NaN  Jeremiah Attaochu        721      OLB   252.0  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nfl_football_players.sort_values('current_salary', ascending=False).head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fixing up the data\n",
    "Assuming you did something like sort_values on one of the original columns, you probably got the wrong result.\n",
    "\n",
    "Looking a bit more closely at the results, you'll notice that the current_salary column.  Remembering that we have made the shift from pythonic to pandorable, we can leverage the impressive-sounding \"vectorized string functions\" mentioned in Section 7.3 of the McKinney book.  Specifically, we can use the str.replace(...) method.  Note that had we use read_csv to load the file we could have used the ```thousands=``` option and avoided all this, but sometimes data doesn't come in a convenient format.\n",
    "\n",
    "One way to apply functions is to operate on a column and then assign the results to another column.  For example, if we wanted to eliminate commas, we could replace them with null strings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0           None\n",
       "1           None\n",
       "2        1075000\n",
       "3           None\n",
       "4           None\n",
       "5           None\n",
       "6        1762000\n",
       "7           None\n",
       "8           None\n",
       "9           None\n",
       "10          None\n",
       "11          None\n",
       "12          None\n",
       "13        774294\n",
       "14          None\n",
       "15          None\n",
       "16          None\n",
       "17          None\n",
       "18          None\n",
       "19          None\n",
       "20          None\n",
       "21          None\n",
       "22          None\n",
       "23          None\n",
       "24          None\n",
       "25          None\n",
       "26          None\n",
       "27          None\n",
       "28          None\n",
       "29          None\n",
       "          ...   \n",
       "25013       None\n",
       "25014       None\n",
       "25015       None\n",
       "25016       None\n",
       "25017       None\n",
       "25018       None\n",
       "25019       None\n",
       "25020       None\n",
       "25021       None\n",
       "25022       None\n",
       "25023       None\n",
       "25024       None\n",
       "25025       None\n",
       "25026       None\n",
       "25027       None\n",
       "25028       None\n",
       "25029       None\n",
       "25030       None\n",
       "25031       None\n",
       "25032       None\n",
       "25033       None\n",
       "25034       None\n",
       "25035       None\n",
       "25036       None\n",
       "25037       None\n",
       "25038       None\n",
       "25039       None\n",
       "25040       None\n",
       "25041       None\n",
       "25042       None\n",
       "Name: current_salary, Length: 25043, dtype: object"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nfl_football_players['current_salary'].str.replace(',', '') \n",
    "# str.replace?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And assign the results to a column in the original dataframe (in this case I'm calling the column current_salary_nocommas). We need to assign it to a column to ensure the operation is done in-place."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "nfl_football_players['current_salary_nocommas'] = nfl_football_players['current_salary'].str.replace(',', '')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But you'll notice that the type of the column is string, and we want to convert it to a float so we can sort it numerically.  So we can use the astype() function to convert it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "nfl_football_players['current_salary_cleaned'] = nfl_football_players['current_salary_nocommas'].astype(float)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birth_date</th>\n",
       "      <th>birth_place</th>\n",
       "      <th>college</th>\n",
       "      <th>current_salary</th>\n",
       "      <th>current_team</th>\n",
       "      <th>death_date</th>\n",
       "      <th>draft_position</th>\n",
       "      <th>draft_round</th>\n",
       "      <th>draft_team</th>\n",
       "      <th>draft_year</th>\n",
       "      <th>height</th>\n",
       "      <th>high_school</th>\n",
       "      <th>hof_induction_year</th>\n",
       "      <th>name</th>\n",
       "      <th>player_id</th>\n",
       "      <th>position</th>\n",
       "      <th>weight</th>\n",
       "      <th>current_salary_nocommas</th>\n",
       "      <th>current_salary_cleaned</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1967-05-12</td>\n",
       "      <td>Bay City, TX</td>\n",
       "      <td>Baylor</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>34.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Seattle Seahawks</td>\n",
       "      <td>1990.0</td>\n",
       "      <td>6-0</td>\n",
       "      <td>Van Vleck, TX</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Robert Blackmon</td>\n",
       "      <td>1809</td>\n",
       "      <td>DB</td>\n",
       "      <td>208.0</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1970-07-20</td>\n",
       "      <td>Louisville, KY</td>\n",
       "      <td>Kentucky</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>85.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Seattle Seahawks</td>\n",
       "      <td>1993.0</td>\n",
       "      <td>6-3</td>\n",
       "      <td>Holy Cross, KY</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Dean Wells</td>\n",
       "      <td>23586</td>\n",
       "      <td>LB</td>\n",
       "      <td>248.0</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   birth_date     birth_place   college current_salary current_team  \\\n",
       "0  1967-05-12    Bay City, TX    Baylor           None         None   \n",
       "1  1970-07-20  Louisville, KY  Kentucky           None         None   \n",
       "\n",
       "  death_date  draft_position  draft_round        draft_team  draft_year  \\\n",
       "0       None            34.0          2.0  Seattle Seahawks      1990.0   \n",
       "1       None            85.0          4.0  Seattle Seahawks      1993.0   \n",
       "\n",
       "  height     high_school  hof_induction_year             name  player_id  \\\n",
       "0    6-0   Van Vleck, TX                 NaN  Robert Blackmon       1809   \n",
       "1    6-3  Holy Cross, KY                 NaN       Dean Wells      23586   \n",
       "\n",
       "  position  weight current_salary_nocommas  current_salary_cleaned  \n",
       "0       DB   208.0                    None                     NaN  \n",
       "1       LB   248.0                    None                     NaN  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nfl_football_players.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we can re-run our command to sort by salary and get the correct result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birth_date</th>\n",
       "      <th>birth_place</th>\n",
       "      <th>college</th>\n",
       "      <th>current_salary</th>\n",
       "      <th>current_team</th>\n",
       "      <th>death_date</th>\n",
       "      <th>draft_position</th>\n",
       "      <th>draft_round</th>\n",
       "      <th>draft_team</th>\n",
       "      <th>draft_year</th>\n",
       "      <th>height</th>\n",
       "      <th>high_school</th>\n",
       "      <th>hof_induction_year</th>\n",
       "      <th>name</th>\n",
       "      <th>player_id</th>\n",
       "      <th>position</th>\n",
       "      <th>weight</th>\n",
       "      <th>current_salary_nocommas</th>\n",
       "      <th>current_salary_cleaned</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>17756</th>\n",
       "      <td>1988-08-19</td>\n",
       "      <td>Holland, MI</td>\n",
       "      <td>Michigan St.</td>\n",
       "      <td>23,943,600</td>\n",
       "      <td>Washington Redskins</td>\n",
       "      <td>None</td>\n",
       "      <td>102.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Washington Redskins</td>\n",
       "      <td>2012.0</td>\n",
       "      <td>6-3</td>\n",
       "      <td>Holland Christian, MI</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Kirk Cousins</td>\n",
       "      <td>4644</td>\n",
       "      <td>QB</td>\n",
       "      <td>214.0</td>\n",
       "      <td>23943600</td>\n",
       "      <td>23943600.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       birth_date  birth_place       college current_salary  \\\n",
       "17756  1988-08-19  Holland, MI  Michigan St.     23,943,600   \n",
       "\n",
       "              current_team death_date  draft_position  draft_round  \\\n",
       "17756  Washington Redskins       None           102.0          4.0   \n",
       "\n",
       "                draft_team  draft_year height            high_school  \\\n",
       "17756  Washington Redskins      2012.0    6-3  Holland Christian, MI   \n",
       "\n",
       "       hof_induction_year          name  player_id position  weight  \\\n",
       "17756                 NaN  Kirk Cousins       4644       QB   214.0   \n",
       "\n",
       "      current_salary_nocommas  current_salary_cleaned  \n",
       "17756                23943600              23943600.0  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nfl_football_players.sort_values('current_salary_cleaned', ascending=False).head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropping missing values\n",
    "\n",
    "In addition to the \"all\" or \"any\" functionality described in McKinney section 7.1, it's sometimes useful to drop a row only if a certain column or columns have missing data.  To do this, use the subset= option with dropna().  So, for example, to drop all players for whom we do not have salary information, we could use the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25043\n"
     ]
    }
   ],
   "source": [
    "nfl_football_players.current_salary_cleaned.isna().sum()\n",
    "print(len(nfl_football_players))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "nfl_football_players_salaries = nfl_football_players.dropna(subset=['current_salary_cleaned'])\n",
    "# current_salary_cleaned meaning? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birth_date</th>\n",
       "      <th>birth_place</th>\n",
       "      <th>college</th>\n",
       "      <th>current_salary</th>\n",
       "      <th>current_team</th>\n",
       "      <th>death_date</th>\n",
       "      <th>draft_position</th>\n",
       "      <th>draft_round</th>\n",
       "      <th>draft_team</th>\n",
       "      <th>draft_year</th>\n",
       "      <th>height</th>\n",
       "      <th>high_school</th>\n",
       "      <th>hof_induction_year</th>\n",
       "      <th>name</th>\n",
       "      <th>player_id</th>\n",
       "      <th>position</th>\n",
       "      <th>weight</th>\n",
       "      <th>current_salary_nocommas</th>\n",
       "      <th>current_salary_cleaned</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1990-08-14</td>\n",
       "      <td>Newton, MA</td>\n",
       "      <td>Oregon</td>\n",
       "      <td>1,075,000</td>\n",
       "      <td>Miami Dolphins</td>\n",
       "      <td>None</td>\n",
       "      <td>46.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Buffalo Bills</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>6-3</td>\n",
       "      <td>Los Gatos, CA</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Kiko Alonso</td>\n",
       "      <td>355</td>\n",
       "      <td>ILB</td>\n",
       "      <td>238.0</td>\n",
       "      <td>1075000</td>\n",
       "      <td>1075000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1992-10-27</td>\n",
       "      <td>Cincinnati, OH</td>\n",
       "      <td>Louisville</td>\n",
       "      <td>1,762,000</td>\n",
       "      <td>Buffalo Bills</td>\n",
       "      <td>None</td>\n",
       "      <td>73.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>Buffalo Bills</td>\n",
       "      <td>2014.0</td>\n",
       "      <td>6-1</td>\n",
       "      <td>Northwest, OH</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Preston Brown</td>\n",
       "      <td>2701</td>\n",
       "      <td>ILB</td>\n",
       "      <td>251.0</td>\n",
       "      <td>1762000</td>\n",
       "      <td>1762000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>1993-06-14</td>\n",
       "      <td>Cleveland, OH</td>\n",
       "      <td>Michigan</td>\n",
       "      <td>774,294</td>\n",
       "      <td>Seattle Seahawks</td>\n",
       "      <td>None</td>\n",
       "      <td>63.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Seattle Seahawks</td>\n",
       "      <td>2015.0</td>\n",
       "      <td>6-2</td>\n",
       "      <td>Glenville, OH</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Frank Clark</td>\n",
       "      <td>3966</td>\n",
       "      <td>DE</td>\n",
       "      <td>270.0</td>\n",
       "      <td>774294</td>\n",
       "      <td>774294.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>1987-03-16</td>\n",
       "      <td>Bellville, TX</td>\n",
       "      <td>SMU</td>\n",
       "      <td>6,750,000</td>\n",
       "      <td>Denver Broncos</td>\n",
       "      <td>None</td>\n",
       "      <td>82.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>Pittsburgh Steelers</td>\n",
       "      <td>2010.0</td>\n",
       "      <td>5-11</td>\n",
       "      <td>Bellville, TX</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Emmanuel Sanders</td>\n",
       "      <td>19449</td>\n",
       "      <td>WR</td>\n",
       "      <td>186.0</td>\n",
       "      <td>6750000</td>\n",
       "      <td>6750000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53</th>\n",
       "      <td>1988-10-27</td>\n",
       "      <td>Lakeland, FL</td>\n",
       "      <td>Louisville</td>\n",
       "      <td>3,750,000</td>\n",
       "      <td>New York Jets</td>\n",
       "      <td>None</td>\n",
       "      <td>126.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>New York Jets</td>\n",
       "      <td>2011.0</td>\n",
       "      <td>5-11</td>\n",
       "      <td>Lake Gibson, FL</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Bilal Powell</td>\n",
       "      <td>17858</td>\n",
       "      <td>RB</td>\n",
       "      <td>204.0</td>\n",
       "      <td>3750000</td>\n",
       "      <td>3750000.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    birth_date     birth_place     college current_salary      current_team  \\\n",
       "2   1990-08-14      Newton, MA      Oregon      1,075,000    Miami Dolphins   \n",
       "6   1992-10-27  Cincinnati, OH  Louisville      1,762,000     Buffalo Bills   \n",
       "13  1993-06-14   Cleveland, OH    Michigan        774,294  Seattle Seahawks   \n",
       "37  1987-03-16   Bellville, TX         SMU      6,750,000    Denver Broncos   \n",
       "53  1988-10-27    Lakeland, FL  Louisville      3,750,000     New York Jets   \n",
       "\n",
       "   death_date  draft_position  draft_round           draft_team  draft_year  \\\n",
       "2        None            46.0          2.0        Buffalo Bills      2013.0   \n",
       "6        None            73.0          3.0        Buffalo Bills      2014.0   \n",
       "13       None            63.0          2.0     Seattle Seahawks      2015.0   \n",
       "37       None            82.0          3.0  Pittsburgh Steelers      2010.0   \n",
       "53       None           126.0          4.0        New York Jets      2011.0   \n",
       "\n",
       "   height      high_school  hof_induction_year               name  player_id  \\\n",
       "2     6-3    Los Gatos, CA                 NaN        Kiko Alonso        355   \n",
       "6     6-1    Northwest, OH                 NaN      Preston Brown       2701   \n",
       "13    6-2    Glenville, OH                 NaN        Frank Clark       3966   \n",
       "37   5-11    Bellville, TX                 NaN  Emmanuel Sanders       19449   \n",
       "53   5-11  Lake Gibson, FL                 NaN       Bilal Powell      17858   \n",
       "\n",
       "   position  weight current_salary_nocommas  current_salary_cleaned  \n",
       "2       ILB   238.0                 1075000               1075000.0  \n",
       "6       ILB   251.0                 1762000               1762000.0  \n",
       "13       DE   270.0                  774294                774294.0  \n",
       "37       WR   186.0                 6750000               6750000.0  \n",
       "53       RB   204.0                 3750000               3750000.0  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nfl_football_players_salaries.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating dummy variables\n",
    "\n",
    "We might, on occasion, want to \"bin\" or \"discretize\" a variable.  For example, we might want to take the previous dataframe and add dummy variables that map onto whether the salaries are \"small\" (< \\\\$1M) , \"medium\" (\\\\$1M - \\\\$10M), or \"large\" (> \\\\$10M). A **dummy variable** is a binary indicator variable (often used in regressions) to represent a categorical variable. We could do something like the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "bins = [0,1000000,10000000,1000000000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2      medium\n",
       "6      medium\n",
       "13      small\n",
       "37     medium\n",
       "53     medium\n",
       "57      small\n",
       "90      small\n",
       "106    medium\n",
       "113     small\n",
       "114    medium\n",
       "Name: current_salary_cleaned, dtype: category\n",
       "Categories (3, object): [small < medium < large]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.cut(nfl_football_players_salaries['current_salary_cleaned'],bins,labels=['small','medium','large'])[0:10]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get_dummies create three levels above, \n",
    "dummies = pd.get_dummies(pd.cut(nfl_football_players_salaries['current_salary_cleaned'],bins,labels=['small','medium','large']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>small</th>\n",
       "      <th>medium</th>\n",
       "      <th>large</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    small  medium  large\n",
       "2       0       1      0\n",
       "6       0       1      0\n",
       "13      1       0      0\n",
       "37      0       1      0\n",
       "53      0       1      0"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dummies.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "pd.concat stacks together objects along an axis _(Section 8.2 McKinney)_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "nfl_cats = pd.concat([nfl_football_players_salaries,dummies],axis=1) # axis=1 merge with the columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birth_date</th>\n",
       "      <th>birth_place</th>\n",
       "      <th>college</th>\n",
       "      <th>current_salary</th>\n",
       "      <th>current_team</th>\n",
       "      <th>death_date</th>\n",
       "      <th>draft_position</th>\n",
       "      <th>draft_round</th>\n",
       "      <th>draft_team</th>\n",
       "      <th>draft_year</th>\n",
       "      <th>...</th>\n",
       "      <th>hof_induction_year</th>\n",
       "      <th>name</th>\n",
       "      <th>player_id</th>\n",
       "      <th>position</th>\n",
       "      <th>weight</th>\n",
       "      <th>current_salary_nocommas</th>\n",
       "      <th>current_salary_cleaned</th>\n",
       "      <th>small</th>\n",
       "      <th>medium</th>\n",
       "      <th>large</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>24885</th>\n",
       "      <td>1994-02-12</td>\n",
       "      <td>San Antonio, TX</td>\n",
       "      <td>Memphis</td>\n",
       "      <td>880,741</td>\n",
       "      <td>Denver Broncos</td>\n",
       "      <td>None</td>\n",
       "      <td>26.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Denver Broncos</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Paxton Lynch</td>\n",
       "      <td>13753</td>\n",
       "      <td>QB</td>\n",
       "      <td>244.0</td>\n",
       "      <td>880741</td>\n",
       "      <td>880741.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24917</th>\n",
       "      <td>1991-05-13</td>\n",
       "      <td>Fairfield, CA</td>\n",
       "      <td>TCU</td>\n",
       "      <td>860,000</td>\n",
       "      <td>Los Angeles Chargers</td>\n",
       "      <td>None</td>\n",
       "      <td>25.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>San Diego Chargers</td>\n",
       "      <td>2014.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Jason Verrett</td>\n",
       "      <td>22916</td>\n",
       "      <td>CB</td>\n",
       "      <td>189.0</td>\n",
       "      <td>860000</td>\n",
       "      <td>860000.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24923</th>\n",
       "      <td>1993-01-21</td>\n",
       "      <td>Sacramento, CA</td>\n",
       "      <td>Stanford</td>\n",
       "      <td>772,413</td>\n",
       "      <td>New England Patriots</td>\n",
       "      <td>None</td>\n",
       "      <td>64.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>New England Patriots</td>\n",
       "      <td>2015.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Jordan Richards</td>\n",
       "      <td>18586</td>\n",
       "      <td>SS</td>\n",
       "      <td>211.0</td>\n",
       "      <td>772413</td>\n",
       "      <td>772413.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24967</th>\n",
       "      <td>1989-11-27</td>\n",
       "      <td>St. Paul, MN</td>\n",
       "      <td>Notre Dame</td>\n",
       "      <td>887,058</td>\n",
       "      <td>Minnesota Vikings</td>\n",
       "      <td>None</td>\n",
       "      <td>13.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Arizona Cardinals</td>\n",
       "      <td>2012.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Michael Floyd</td>\n",
       "      <td>7063</td>\n",
       "      <td>WR</td>\n",
       "      <td>220.0</td>\n",
       "      <td>887058</td>\n",
       "      <td>887058.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24984</th>\n",
       "      <td>1988-03-22</td>\n",
       "      <td>Longview, TX</td>\n",
       "      <td>Washington St.</td>\n",
       "      <td>4,500,000</td>\n",
       "      <td>Jacksonville Jaguars</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Chris Ivory</td>\n",
       "      <td>10701</td>\n",
       "      <td>RB</td>\n",
       "      <td>222.0</td>\n",
       "      <td>4500000</td>\n",
       "      <td>4500000.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       birth_date      birth_place         college current_salary  \\\n",
       "24885  1994-02-12  San Antonio, TX         Memphis        880,741   \n",
       "24917  1991-05-13    Fairfield, CA             TCU        860,000   \n",
       "24923  1993-01-21   Sacramento, CA        Stanford        772,413   \n",
       "24967  1989-11-27     St. Paul, MN      Notre Dame        887,058   \n",
       "24984  1988-03-22     Longview, TX  Washington St.      4,500,000   \n",
       "\n",
       "               current_team death_date  draft_position  draft_round  \\\n",
       "24885        Denver Broncos       None            26.0          1.0   \n",
       "24917  Los Angeles Chargers       None            25.0          1.0   \n",
       "24923  New England Patriots       None            64.0          2.0   \n",
       "24967     Minnesota Vikings       None            13.0          1.0   \n",
       "24984  Jacksonville Jaguars       None             NaN          NaN   \n",
       "\n",
       "                 draft_team  draft_year  ...   hof_induction_year  \\\n",
       "24885        Denver Broncos      2016.0  ...                  NaN   \n",
       "24917    San Diego Chargers      2014.0  ...                  NaN   \n",
       "24923  New England Patriots      2015.0  ...                  NaN   \n",
       "24967     Arizona Cardinals      2012.0  ...                  NaN   \n",
       "24984                  None         NaN  ...                  NaN   \n",
       "\n",
       "                  name  player_id position  weight current_salary_nocommas  \\\n",
       "24885    Paxton Lynch       13753       QB   244.0                  880741   \n",
       "24917    Jason Verrett      22916       CB   189.0                  860000   \n",
       "24923  Jordan Richards      18586       SS   211.0                  772413   \n",
       "24967   Michael Floyd        7063       WR   220.0                  887058   \n",
       "24984      Chris Ivory      10701       RB   222.0                 4500000   \n",
       "\n",
       "       current_salary_cleaned small  medium  large  \n",
       "24885                880741.0     1       0      0  \n",
       "24917                860000.0     1       0      0  \n",
       "24923                772413.0     1       0      0  \n",
       "24967                887058.0     1       0      0  \n",
       "24984               4500000.0     0       1      0  \n",
       "\n",
       "[5 rows x 22 columns]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nfl_cats.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scraping Tables from HTML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```pd.read_html``` function returns a list of DataFrames read from an HTML source.  The following line will return a list of DataFrames from https://en.wikipedia.org/wiki/List_of_largest_sports_contracts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "contracts_scraped = pd.read_html('https://en.wikipedia.org/wiki/List_of_largest_sports_contracts',header=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "list"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(contracts_scraped) \n",
    "type(contracts_scraped) # read html is a list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get the first table, you'll need to pull off the 0th element:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Rank</th>\n",
       "      <th>Player</th>\n",
       "      <th>Team/Streaming Service</th>\n",
       "      <th>Sport</th>\n",
       "      <th>Length of contract</th>\n",
       "      <th>Contract value (USD)</th>\n",
       "      <th>Average per year (USD)</th>\n",
       "      <th>Average per game/fight4 (USD)</th>\n",
       "      <th>Ref[1]</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>01</td>\n",
       "      <td>Canelo Álvarez</td>\n",
       "      <td>DAZN*</td>\n",
       "      <td>Boxing</td>\n",
       "      <td>5 years (2018–2023)</td>\n",
       "      <td>$365,000,000</td>\n",
       "      <td>$73,000,000</td>\n",
       "      <td>$33,181,818.18</td>\n",
       "      <td>[2]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>02</td>\n",
       "      <td>Giancarlo Stanton</td>\n",
       "      <td>Miami Marlins*</td>\n",
       "      <td>Baseball</td>\n",
       "      <td>13 years (2014–2027)</td>\n",
       "      <td>$325,000,000</td>\n",
       "      <td>$25,000,000</td>\n",
       "      <td>$154,320.99</td>\n",
       "      <td>[1]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>03</td>\n",
       "      <td>Alex Rodriguez1R</td>\n",
       "      <td>New York Yankees*</td>\n",
       "      <td>Baseball</td>\n",
       "      <td>10 years (2008–2017)</td>\n",
       "      <td>$275,000,000</td>\n",
       "      <td>$27,500,000</td>\n",
       "      <td>$169,753.09</td>\n",
       "      <td>[3]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>04</td>\n",
       "      <td>Alex Rodriguez2R</td>\n",
       "      <td>Texas Rangers*</td>\n",
       "      <td>Baseball</td>\n",
       "      <td>10 years (2001–2010)</td>\n",
       "      <td>$252,000,000</td>\n",
       "      <td>$25,200,000</td>\n",
       "      <td>$155,555.56</td>\n",
       "      <td>[4]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>05</td>\n",
       "      <td>Miguel Cabrera</td>\n",
       "      <td>Detroit Tigers</td>\n",
       "      <td>Baseball</td>\n",
       "      <td>8 years (2016–2023)</td>\n",
       "      <td>$247,000,000</td>\n",
       "      <td>$31,000,000</td>\n",
       "      <td>$191,358.02</td>\n",
       "      <td>[5]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Rank             Player Team/Streaming Service     Sport  \\\n",
       "0   01     Canelo Álvarez                  DAZN*    Boxing   \n",
       "1   02  Giancarlo Stanton         Miami Marlins*  Baseball   \n",
       "2   03   Alex Rodriguez1R      New York Yankees*  Baseball   \n",
       "3   04   Alex Rodriguez2R         Texas Rangers*  Baseball   \n",
       "4   05     Miguel Cabrera         Detroit Tigers  Baseball   \n",
       "\n",
       "     Length of contract Contract value (USD) Average per year (USD)  \\\n",
       "0   5 years (2018–2023)         $365,000,000            $73,000,000   \n",
       "1  13 years (2014–2027)         $325,000,000            $25,000,000   \n",
       "2  10 years (2008–2017)         $275,000,000            $27,500,000   \n",
       "3  10 years (2001–2010)         $252,000,000            $25,200,000   \n",
       "4   8 years (2016–2023)         $247,000,000            $31,000,000   \n",
       "\n",
       "  Average per game/fight4 (USD) Ref[1]  \n",
       "0                $33,181,818.18    [2]  \n",
       "1                   $154,320.99    [1]  \n",
       "2                   $169,753.09    [3]  \n",
       "3                   $155,555.56    [4]  \n",
       "4                   $191,358.02    [5]  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "contracts = contracts_scraped[0]\n",
    "contracts.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color=\"magenta\"> Q2 (2 points): \n",
    "Count the number of players from each sport in the List of Largest Sports Contracts. \n",
    "Hint:  see value_counts() description above\n",
    "<font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Baseball                54\n",
       "Basketball              27\n",
       "American football       15\n",
       "Auto racing              2\n",
       "Boxing                   1\n",
       "Association football     1\n",
       "Hockey                   1\n",
       "Name: Sport, dtype: int64"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Insert your code here\n",
    "contracts['Sport'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Insert English explanation of findings here._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color=\"magenta\">Q3 (12 points): \n",
    "Create a new dataframe that contains all the columns in the nfl_football_players dataframe as well as an additional column that contains each player's height in centimeters. Show the first 5 rows of your result. </font>\n",
    "\n",
    "**Hints:** \n",
    "- 1 inch = 2.54 cm\n",
    "- you can use the vectorized string function str.split() to separate feet and inches from the original dataframe column\n",
    "- remember to cast strings to numeric types if you're going to perform math on them\n",
    "- you might want to create an intermediate (temporary) DataFrame to help you keep things clear instead of attempting to do \n",
    "this in one line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda2/lib/python2.7/site-packages/ipykernel_launcher.py:3: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  This is separate from the ipykernel package so we can avoid doing imports until\n",
      "/anaconda2/lib/python2.7/site-packages/ipykernel_launcher.py:5: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  \"\"\"\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>birth_date</th>\n",
       "      <th>birth_place</th>\n",
       "      <th>college</th>\n",
       "      <th>current_salary</th>\n",
       "      <th>current_team</th>\n",
       "      <th>death_date</th>\n",
       "      <th>draft_position</th>\n",
       "      <th>draft_round</th>\n",
       "      <th>draft_team</th>\n",
       "      <th>draft_year</th>\n",
       "      <th>...</th>\n",
       "      <th>high_school</th>\n",
       "      <th>hof_induction_year</th>\n",
       "      <th>name</th>\n",
       "      <th>player_id</th>\n",
       "      <th>position</th>\n",
       "      <th>weight</th>\n",
       "      <th>current_salary_nocommas</th>\n",
       "      <th>current_salary_cleaned</th>\n",
       "      <th>height_feet</th>\n",
       "      <th>height_cm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1967-05-12</td>\n",
       "      <td>Bay City, TX</td>\n",
       "      <td>Baylor</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>34.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Seattle Seahawks</td>\n",
       "      <td>1990.0</td>\n",
       "      <td>...</td>\n",
       "      <td>Van Vleck, TX</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Robert Blackmon</td>\n",
       "      <td>1809</td>\n",
       "      <td>DB</td>\n",
       "      <td>208.0</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(6, 0)</td>\n",
       "      <td>182.88</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1970-07-20</td>\n",
       "      <td>Louisville, KY</td>\n",
       "      <td>Kentucky</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>85.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Seattle Seahawks</td>\n",
       "      <td>1993.0</td>\n",
       "      <td>...</td>\n",
       "      <td>Holy Cross, KY</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Dean Wells</td>\n",
       "      <td>23586</td>\n",
       "      <td>LB</td>\n",
       "      <td>248.0</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(6, 3)</td>\n",
       "      <td>190.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1990-08-14</td>\n",
       "      <td>Newton, MA</td>\n",
       "      <td>Oregon</td>\n",
       "      <td>1,075,000</td>\n",
       "      <td>Miami Dolphins</td>\n",
       "      <td>None</td>\n",
       "      <td>46.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Buffalo Bills</td>\n",
       "      <td>2013.0</td>\n",
       "      <td>...</td>\n",
       "      <td>Los Gatos, CA</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Kiko Alonso</td>\n",
       "      <td>355</td>\n",
       "      <td>ILB</td>\n",
       "      <td>238.0</td>\n",
       "      <td>1075000</td>\n",
       "      <td>1075000.0</td>\n",
       "      <td>(6, 3)</td>\n",
       "      <td>190.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1948-04-22</td>\n",
       "      <td>Dallas, TX</td>\n",
       "      <td>North Texas</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>1999-10-15</td>\n",
       "      <td>126.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>New Orleans Saints</td>\n",
       "      <td>1970.0</td>\n",
       "      <td>...</td>\n",
       "      <td>W.W. Samuell, TX</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Steve Ramsey</td>\n",
       "      <td>18182</td>\n",
       "      <td>QB</td>\n",
       "      <td>210.0</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(6, 2)</td>\n",
       "      <td>187.96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1988-02-27</td>\n",
       "      <td>Neptune, NJ</td>\n",
       "      <td>Miami (FL)</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>Neptune, NJ</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Cory Nelms</td>\n",
       "      <td>16250</td>\n",
       "      <td>CB</td>\n",
       "      <td>195.0</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>(6, 0)</td>\n",
       "      <td>182.88</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   birth_date     birth_place      college current_salary    current_team  \\\n",
       "0  1967-05-12    Bay City, TX       Baylor           None            None   \n",
       "1  1970-07-20  Louisville, KY     Kentucky           None            None   \n",
       "2  1990-08-14      Newton, MA       Oregon      1,075,000  Miami Dolphins   \n",
       "3  1948-04-22      Dallas, TX  North Texas           None            None   \n",
       "4  1988-02-27     Neptune, NJ   Miami (FL)           None            None   \n",
       "\n",
       "   death_date  draft_position  draft_round          draft_team  draft_year  \\\n",
       "0        None            34.0          2.0    Seattle Seahawks      1990.0   \n",
       "1        None            85.0          4.0    Seattle Seahawks      1993.0   \n",
       "2        None            46.0          2.0       Buffalo Bills      2013.0   \n",
       "3  1999-10-15           126.0          5.0  New Orleans Saints      1970.0   \n",
       "4        None             NaN          NaN                None         NaN   \n",
       "\n",
       "     ...          high_school hof_induction_year             name player_id  \\\n",
       "0    ...        Van Vleck, TX                NaN  Robert Blackmon      1809   \n",
       "1    ...       Holy Cross, KY                NaN       Dean Wells     23586   \n",
       "2    ...        Los Gatos, CA                NaN      Kiko Alonso       355   \n",
       "3    ...     W.W. Samuell, TX                NaN     Steve Ramsey     18182   \n",
       "4    ...          Neptune, NJ                NaN       Cory Nelms     16250   \n",
       "\n",
       "   position weight  current_salary_nocommas current_salary_cleaned  \\\n",
       "0        DB  208.0                     None                    NaN   \n",
       "1        LB  248.0                     None                    NaN   \n",
       "2       ILB  238.0                  1075000              1075000.0   \n",
       "3        QB  210.0                     None                    NaN   \n",
       "4        CB  195.0                     None                    NaN   \n",
       "\n",
       "   height_feet height_cm  \n",
       "0       (6, 0)    182.88  \n",
       "1       (6, 3)    190.50  \n",
       "2       (6, 3)    190.50  \n",
       "3       (6, 2)    187.96  \n",
       "4       (6, 0)    182.88  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Insert Code Here. You should use multiple code boxes. You can add them below.\n",
    "nfl_football_players_heigh = nfl_football_players.dropna(subset=['height'])\n",
    "nfl_football_players_heigh['height_feet'] = [(it.split('-')[0],it.split('-')[1]) for it in nfl_football_players_heigh['height']]# .astype(float)\n",
    "# nfl_football_players_heigh['height_cm'] = nfl_football_players_heigh['height_cm'].astype(float)\n",
    "nfl_football_players_heigh['height_cm'] = [30.48*float(it[0]) + 2.54*float(it[1]) for it in nfl_football_players_heigh['height_feet']]\n",
    "nfl_football_players_heigh.head()\n",
    "\n",
    "# can create a new dataframe: pd.dataframe(columns= ['raw','temp'])\n",
    "# nfl_football_players_heigh['height_feet'] = (30.48* nfl_height['feet']) + (nfl_height['inch'] * 2.54)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'6'"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'6-0'.split('-')[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Joining, Combining, and Reshaping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning Objectives\n",
    "* use pd.read_html's parameters to extract specific tables from web pages\n",
    "* create dataframes from lists and dictionaries\n",
    "* use Pandas' apply function to run a function on each row of a dataframe\n",
    "* view and set the indexes of a dataframe, including hierarchical indexes\n",
    "* use loc to explore a dataframe with hierarchical indexes\n",
    "* use stack and unstack to reshape dataframes\n",
    "* concatenate two DataFrames by columns\n",
    "* rename a dataframe's columns with a dictionary\n",
    "* use Pandas' merge functionn to join dataframes in a SQL-like way"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Frodo Baggins</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Tolkien character</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Information</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Aliases</td>\n",
       "      <td>Mr. Underhill,Maura Labingi</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Race</td>\n",
       "      <td>Hobbit</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Book(s)</td>\n",
       "      <td>The Fellowship of the RingThe Two TowersThe Re...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   0                                                  1\n",
       "0      Frodo Baggins                                                NaN\n",
       "1  Tolkien character                                                NaN\n",
       "2        Information                                                NaN\n",
       "3            Aliases                        Mr. Underhill,Maura Labingi\n",
       "4               Race                                             Hobbit\n",
       "5            Book(s)  The Fellowship of the RingThe Two TowersThe Re..."
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "frodo_url = 'https://en.wikipedia.org/wiki/Frodo_Baggins'\n",
    "frodo_tables = pd.read_html(frodo_url)\n",
    "frodo_tables[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's load the page for [Legolas](https://en.wikipedia.org/wiki/Legolas):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>NaN</td>\n",
       "      <td>This article relies too much on references to ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    0                                                  1\n",
       "0 NaN  This article relies too much on references to ..."
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "legolas_url = 'https://en.wikipedia.org/wiki/Legolas'\n",
    "legolas_tables = pd.read_html(legolas_url)\n",
    "legolas_tables[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hmmmm.  That doesn't look quite right.\n",
    "\n",
    "Let's take a look at some URLs and figure out what's going on:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "INSTALLED VERSIONS\n",
      "------------------\n",
      "commit: None\n",
      "python: 2.7.14.final.0\n",
      "python-bits: 64\n",
      "OS: Darwin\n",
      "OS-release: 17.2.0\n",
      "machine: x86_64\n",
      "processor: i386\n",
      "byteorder: little\n",
      "LC_ALL: en_US.UTF-8\n",
      "LANG: en_US.UTF-8\n",
      "LOCALE: None.None\n",
      "\n",
      "pandas: 0.22.0\n",
      "pytest: 3.3.2\n",
      "pip: 18.1\n",
      "setuptools: 38.4.0\n",
      "Cython: 0.27.3\n",
      "numpy: 1.14.0\n",
      "scipy: 1.0.0\n",
      "pyarrow: None\n",
      "xarray: None\n",
      "IPython: 5.4.1\n",
      "sphinx: 1.6.6\n",
      "patsy: 0.5.0\n",
      "dateutil: 2.6.1\n",
      "pytz: 2017.3\n",
      "blosc: None\n",
      "bottleneck: 1.2.1\n",
      "tables: 3.4.2\n",
      "numexpr: 2.6.4\n",
      "feather: None\n",
      "matplotlib: 2.1.2\n",
      "openpyxl: 2.4.10\n",
      "xlrd: 1.1.0\n",
      "xlwt: 1.2.0\n",
      "xlsxwriter: 1.0.2\n",
      "lxml: 4.1.1\n",
      "bs4: 4.6.0\n",
      "html5lib: 1.0.1\n",
      "sqlalchemy: 1.2.1\n",
      "pymysql: None\n",
      "psycopg2: None\n",
      "jinja2: 2.10\n",
      "s3fs: None\n",
      "fastparquet: None\n",
      "pandas_gbq: None\n",
      "pandas_datareader: None\n"
     ]
    }
   ],
   "source": [
    "pd.show_versions()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Inspect the Frodo and Legolas pages and see if you can figure out some _attributes_ of the table we're interested in.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "knowledge_building": {
     "cell_groupid": "c04-q2",
     "cell_type": "reflection_answer"
    }
   },
   "source": [
    "Describe what you found."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll notice that there are some characteristics that the \"Information\" box share across pages.  We can leverage that \n",
    "information by using the ```attrs``` attribute of ```read_html```.  For example, if we wanted to extract  the element(s) that had\n",
    "an ```id``` of ```info```, we could use\n",
    "\n",
    "```pd.read_html(url,{'id':'info'})```\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color=\"magenta\">Q4: (2 point) Fill in the following code block to extract only the \"Information\" table for the Legolas page:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "knowledge_building": {
     "cell_groupid": "c04-q3",
     "cell_type": "code_answer"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[                   0                                                  1\n",
       " 0            Legolas                                                NaN\n",
       " 1  Tolkien character                                                NaN\n",
       " 2        Information                                                NaN\n",
       " 3            Aliases        Greenleaf, (Legolas translatedinto English)\n",
       " 4               Race                                         Sindar Elf\n",
       " 5             Gender                                               Male\n",
       " 6            Book(s)  The Fellowship of the Ring The Two Towers The ...]"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = {'class':'infobox'} # create an appropriate dictionary\n",
    "pd.read_html(legolas_url, attrs=a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's define a function that, given a Wikipedia URL, will extract the contents of the Aliases component of the infobox table:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_aliases(url):\n",
    "    tables = pd.read_html(url, attrs={'class':'infobox'}) # extract only tables with class=infobox\n",
    "    print(url,len(tables))   # sanity check: we should have just 1 table\n",
    "    infotable = tables[0]    # pull the first table into a DataFrame\n",
    "    ret = ''                 # initialize an empty string for our return value\n",
    "    try:                     # in case the next line throws an exception\n",
    "        x = infotable.set_index(0).loc['Aliases'] # setting the index on column 0 will allow us to use .loc to look up the value of 'Aliases'\n",
    "        ret = x.values[0]\n",
    "    except:\n",
    "        ret = 'None'\n",
    "    return ret"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And let's try it out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('https://en.wikipedia.org/wiki/Legolas', 1)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "u'Greenleaf, (Legolas translatedinto English)'"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_aliases(legolas_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far, so good.  It seems to work.  Now let's set up a DataFrame with a bunch of URLs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "urls = ['https://en.wikipedia.org/wiki/Gimli_(Middle-earth)',\n",
    "        'https://en.wikipedia.org/wiki/Frodo_Baggins',\n",
    "        'https://en.wikipedia.org/wiki/Legolas',\n",
    "        'https://en.wikipedia.org/wiki/Bilbo_Baggins',\n",
    "        'https://en.wikipedia.org/wiki/Samwise_Gamgee',\n",
    "        'https://en.wikipedia.org/wiki/Peregrin_Took',\n",
    "        'https://en.wikipedia.org/wiki/Boromir',\n",
    "        'https://en.wikipedia.org/wiki/Galadriel',\n",
    "        'https://en.wikipedia.org/wiki/Meriadoc_Brandybuck']\n",
    "names = ['Gimli',\n",
    "         'Frodo',\n",
    "         'Legolas',\n",
    "         'Bilbo',\n",
    "         'Sam',\n",
    "         'Pippin',\n",
    "         'Boromir',\n",
    "         'Galadriel',\n",
    "         'Meriadoc']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "udf = pd.DataFrame()\n",
    "udf['name'] = names\n",
    "udf['url'] = urls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Gimli</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Gimli_(Middle-ea...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Frodo</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Frodo_Baggins</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Legolas</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Legolas</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Bilbo</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Bilbo_Baggins</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Sam</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Samwise_Gamgee</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Pippin</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Peregrin_Took</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Boromir</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Boromir</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Galadriel</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Galadriel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Meriadoc</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Meriadoc_Brandybuck</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        name                                                url\n",
       "0      Gimli  https://en.wikipedia.org/wiki/Gimli_(Middle-ea...\n",
       "1      Frodo        https://en.wikipedia.org/wiki/Frodo_Baggins\n",
       "2    Legolas              https://en.wikipedia.org/wiki/Legolas\n",
       "3      Bilbo        https://en.wikipedia.org/wiki/Bilbo_Baggins\n",
       "4        Sam       https://en.wikipedia.org/wiki/Samwise_Gamgee\n",
       "5     Pippin        https://en.wikipedia.org/wiki/Peregrin_Took\n",
       "6    Boromir              https://en.wikipedia.org/wiki/Boromir\n",
       "7  Galadriel            https://en.wikipedia.org/wiki/Galadriel\n",
       "8   Meriadoc  https://en.wikipedia.org/wiki/Meriadoc_Brandybuck"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "udf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The pythonic way of iterating through each of those rows would involve the use of some sort of ```for``` loop.  In pandas,\n",
    "however, as can use the ```apply``` function to process an entire column!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('https://en.wikipedia.org/wiki/Gimli_(Middle-earth)', 1)\n",
      "('https://en.wikipedia.org/wiki/Frodo_Baggins', 1)\n",
      "('https://en.wikipedia.org/wiki/Legolas', 1)\n",
      "('https://en.wikipedia.org/wiki/Bilbo_Baggins', 1)\n",
      "('https://en.wikipedia.org/wiki/Samwise_Gamgee', 1)\n",
      "('https://en.wikipedia.org/wiki/Peregrin_Took', 1)\n",
      "('https://en.wikipedia.org/wiki/Boromir', 1)\n",
      "('https://en.wikipedia.org/wiki/Galadriel', 1)\n",
      "('https://en.wikipedia.org/wiki/Meriadoc_Brandybuck', 1)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0    Elf-friend Lockbearer Lord of the Glittering C...\n",
       "1                          Mr. Underhill,Maura Labingi\n",
       "2          Greenleaf, (Legolas translatedinto English)\n",
       "3                                        Bilba Labingi\n",
       "4    Samwise Gardner, Sam, Samwise the Brave,Banazî...\n",
       "5    Pippin, Pip,\"Ernil i Pheriannath\"Thain Peregri...\n",
       "6    Captain of the White Tower,High Warden of the ...\n",
       "7                       AlatárielAltárielArtanisNerwen\n",
       "8    Merry,Kalimac Brandagamba,Meriadoc the Magnifi...\n",
       "Name: url, dtype: object"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "udf['url'].apply(get_aliases) # row of url as the input of get_aliases"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can take the resulting Series and assign it to a new column in our DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('https://en.wikipedia.org/wiki/Gimli_(Middle-earth)', 1)\n",
      "('https://en.wikipedia.org/wiki/Frodo_Baggins', 1)\n",
      "('https://en.wikipedia.org/wiki/Legolas', 1)\n",
      "('https://en.wikipedia.org/wiki/Bilbo_Baggins', 1)\n",
      "('https://en.wikipedia.org/wiki/Samwise_Gamgee', 1)\n",
      "('https://en.wikipedia.org/wiki/Peregrin_Took', 1)\n",
      "('https://en.wikipedia.org/wiki/Boromir', 1)\n",
      "('https://en.wikipedia.org/wiki/Galadriel', 1)\n",
      "('https://en.wikipedia.org/wiki/Meriadoc_Brandybuck', 1)\n"
     ]
    }
   ],
   "source": [
    "udf['aliases'] = udf['url'].apply(get_aliases)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>url</th>\n",
       "      <th>aliases</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Gimli</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Gimli_(Middle-ea...</td>\n",
       "      <td>Elf-friend Lockbearer Lord of the Glittering C...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Frodo</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Frodo_Baggins</td>\n",
       "      <td>Mr. Underhill,Maura Labingi</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Legolas</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Legolas</td>\n",
       "      <td>Greenleaf, (Legolas translatedinto English)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Bilbo</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Bilbo_Baggins</td>\n",
       "      <td>Bilba Labingi</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Sam</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Samwise_Gamgee</td>\n",
       "      <td>Samwise Gardner, Sam, Samwise the Brave,Banazî...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Pippin</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Peregrin_Took</td>\n",
       "      <td>Pippin, Pip,\"Ernil i Pheriannath\"Thain Peregri...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Boromir</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Boromir</td>\n",
       "      <td>Captain of the White Tower,High Warden of the ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Galadriel</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Galadriel</td>\n",
       "      <td>AlatárielAltárielArtanisNerwen</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Meriadoc</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Meriadoc_Brandybuck</td>\n",
       "      <td>Merry,Kalimac Brandagamba,Meriadoc the Magnifi...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        name                                                url  \\\n",
       "0      Gimli  https://en.wikipedia.org/wiki/Gimli_(Middle-ea...   \n",
       "1      Frodo        https://en.wikipedia.org/wiki/Frodo_Baggins   \n",
       "2    Legolas              https://en.wikipedia.org/wiki/Legolas   \n",
       "3      Bilbo        https://en.wikipedia.org/wiki/Bilbo_Baggins   \n",
       "4        Sam       https://en.wikipedia.org/wiki/Samwise_Gamgee   \n",
       "5     Pippin        https://en.wikipedia.org/wiki/Peregrin_Took   \n",
       "6    Boromir              https://en.wikipedia.org/wiki/Boromir   \n",
       "7  Galadriel            https://en.wikipedia.org/wiki/Galadriel   \n",
       "8   Meriadoc  https://en.wikipedia.org/wiki/Meriadoc_Brandybuck   \n",
       "\n",
       "                                             aliases  \n",
       "0  Elf-friend Lockbearer Lord of the Glittering C...  \n",
       "1                        Mr. Underhill,Maura Labingi  \n",
       "2        Greenleaf, (Legolas translatedinto English)  \n",
       "3                                      Bilba Labingi  \n",
       "4  Samwise Gardner, Sam, Samwise the Brave,Banazî...  \n",
       "5  Pippin, Pip,\"Ernil i Pheriannath\"Thain Peregri...  \n",
       "6  Captain of the White Tower,High Warden of the ...  \n",
       "7                     AlatárielAltárielArtanisNerwen  \n",
       "8  Merry,Kalimac Brandagamba,Meriadoc the Magnifi...  "
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "udf\n",
    "# udf.to_csv('udf.csv', index = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's just put the ```udf``` DataFrame aside for now.  We'll return to it later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating DataFrames and Exploring Indexes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load the usual libraries..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create some lists of data that we can use to construct a DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "names = ['Gandalf',\n",
    "         'Gimli',\n",
    "         'Frodo',\n",
    "         'Legolas',\n",
    "         'Bilbo',\n",
    "         'Sam',\n",
    "         'Pippin',\n",
    "         'Boromir',\n",
    "         'Aragorn',\n",
    "         'Galadriel',\n",
    "         'Meriadoc',\n",
    "        'Lily']\n",
    "races = ['Maia',\n",
    "         'Dwarf',\n",
    "         'Hobbit',\n",
    "         'Elf',\n",
    "         'Hobbit',\n",
    "         'Hobbit',\n",
    "         'Hobbit',\n",
    "         'Man',\n",
    "         'Man',\n",
    "         'Elf',\n",
    "         'Hobbit',\n",
    "        'Hobbit']\n",
    "magic = [10, 1, 4, 6, 4, 2, 0, 0, 2, 9, 0, np.NaN]\n",
    "aggression = [7, 10, 2, 5, 1, 6, 3, 8, 7, 2, 4, np.NaN ]\n",
    "stealth = [8, 2, 5, 10, 5, 4 ,5, 3, 9, 10, 6, np.NaN]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a few different ways to construct a DataFrame.  We can either use an empty constructor and assign Series:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color=\"magenta\"> Q5: (2 points) Construct a dataframe with 5 columns (names, races, magic, aggression, and stealth) using the lists above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {
    "knowledge_building": {
     "cell_groupid": "c04-q4",
     "cell_type": "rcode_answer"
    }
   },
   "outputs": [],
   "source": [
    "df = pd.DataFrame(np.column_stack([names, races, magic, aggression, stealth]), columns=['names', 'races', 'magic', 'aggression','stealth'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>names</th>\n",
       "      <th>races</th>\n",
       "      <th>magic</th>\n",
       "      <th>aggression</th>\n",
       "      <th>stealth</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Gandalf</td>\n",
       "      <td>Maia</td>\n",
       "      <td>10.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Gimli</td>\n",
       "      <td>Dwarf</td>\n",
       "      <td>1.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Frodo</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Legolas</td>\n",
       "      <td>Elf</td>\n",
       "      <td>6.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Bilbo</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Sam</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>2.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Pippin</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Boromir</td>\n",
       "      <td>Man</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Aragorn</td>\n",
       "      <td>Man</td>\n",
       "      <td>2.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Galadriel</td>\n",
       "      <td>Elf</td>\n",
       "      <td>9.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Meriadoc</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Lily</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "      <td>nan</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        names   races magic aggression stealth\n",
       "0     Gandalf    Maia  10.0        7.0     8.0\n",
       "1       Gimli   Dwarf   1.0       10.0     2.0\n",
       "2       Frodo  Hobbit   4.0        2.0     5.0\n",
       "3     Legolas     Elf   6.0        5.0    10.0\n",
       "4       Bilbo  Hobbit   4.0        1.0     5.0\n",
       "5         Sam  Hobbit   2.0        6.0     4.0\n",
       "6      Pippin  Hobbit   0.0        3.0     5.0\n",
       "7     Boromir     Man   0.0        8.0     3.0\n",
       "8     Aragorn     Man   2.0        7.0     9.0\n",
       "9   Galadriel     Elf   9.0        2.0    10.0\n",
       "10   Meriadoc  Hobbit   0.0        4.0     6.0\n",
       "11       Lily  Hobbit   nan        nan     nan"
      ]
     },
     "execution_count": 182,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, we could have set things up with a dict:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame({'name': names,'race':races,'magic':magic,'aggression': aggression,'stealth':stealth})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>aggression</th>\n",
       "      <th>magic</th>\n",
       "      <th>name</th>\n",
       "      <th>race</th>\n",
       "      <th>stealth</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>Gandalf</td>\n",
       "      <td>Maia</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Gimli</td>\n",
       "      <td>Dwarf</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Frodo</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Legolas</td>\n",
       "      <td>Elf</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Bilbo</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   aggression  magic     name    race  stealth\n",
       "0         7.0   10.0  Gandalf    Maia      8.0\n",
       "1        10.0    1.0    Gimli   Dwarf      2.0\n",
       "2         2.0    4.0    Frodo  Hobbit      5.0\n",
       "3         5.0    6.0  Legolas     Elf     10.0\n",
       "4         1.0    4.0    Bilbo  Hobbit      5.0"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the index on the resulting DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RangeIndex(start=0, stop=12, step=1)"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can set the index to something more useful than the default RangeIndex:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>aggression</th>\n",
       "      <th>magic</th>\n",
       "      <th>race</th>\n",
       "      <th>stealth</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>name</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Gandalf</th>\n",
       "      <td>7.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>Maia</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Gimli</th>\n",
       "      <td>10.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Dwarf</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Frodo</th>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Legolas</th>\n",
       "      <td>5.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Elf</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Bilbo</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sam</th>\n",
       "      <td>6.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pippin</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Boromir</th>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Aragorn</th>\n",
       "      <td>7.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Galadriel</th>\n",
       "      <td>2.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>Elf</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Meriadoc</th>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lily</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           aggression  magic    race  stealth\n",
       "name                                         \n",
       "Gandalf           7.0   10.0    Maia      8.0\n",
       "Gimli            10.0    1.0   Dwarf      2.0\n",
       "Frodo             2.0    4.0  Hobbit      5.0\n",
       "Legolas           5.0    6.0     Elf     10.0\n",
       "Bilbo             1.0    4.0  Hobbit      5.0\n",
       "Sam               6.0    2.0  Hobbit      4.0\n",
       "Pippin            3.0    0.0  Hobbit      5.0\n",
       "Boromir           8.0    0.0     Man      3.0\n",
       "Aragorn           7.0    2.0     Man      9.0\n",
       "Galadriel         2.0    9.0     Elf     10.0\n",
       "Meriadoc          4.0    0.0  Hobbit      6.0\n",
       "Lily              NaN    NaN  Hobbit      NaN"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_nameindexed = df.set_index('name')\n",
    "df_nameindexed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And if we take a look at the results, we see that we have a pandas Index instead of a RangeIndex:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index([u'Gandalf', u'Gimli', u'Frodo', u'Legolas', u'Bilbo', u'Sam', u'Pippin',\n",
       "       u'Boromir', u'Aragorn', u'Galadriel', u'Meriadoc', u'Lily'],\n",
       "      dtype='object', name=u'name')"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_nameindexed.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>aggression</th>\n",
       "      <th>magic</th>\n",
       "      <th>race</th>\n",
       "      <th>stealth</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>name</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Gandalf</th>\n",
       "      <td>7.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>Maia</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Gimli</th>\n",
       "      <td>10.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Dwarf</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Frodo</th>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Legolas</th>\n",
       "      <td>5.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Elf</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Bilbo</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sam</th>\n",
       "      <td>6.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pippin</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Boromir</th>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Aragorn</th>\n",
       "      <td>7.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Galadriel</th>\n",
       "      <td>2.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>Elf</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Meriadoc</th>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lily</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           aggression  magic    race  stealth\n",
       "name                                         \n",
       "Gandalf           7.0   10.0    Maia      8.0\n",
       "Gimli            10.0    1.0   Dwarf      2.0\n",
       "Frodo             2.0    4.0  Hobbit      5.0\n",
       "Legolas           5.0    6.0     Elf     10.0\n",
       "Bilbo             1.0    4.0  Hobbit      5.0\n",
       "Sam               6.0    2.0  Hobbit      4.0\n",
       "Pippin            3.0    0.0  Hobbit      5.0\n",
       "Boromir           8.0    0.0     Man      3.0\n",
       "Aragorn           7.0    2.0     Man      9.0\n",
       "Galadriel         2.0    9.0     Elf     10.0\n",
       "Meriadoc          4.0    0.0  Hobbit      6.0\n",
       "Lily              NaN    NaN  Hobbit      NaN"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_nameindexed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setting the name Series as the index allows us to do things like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "aggression      7\n",
       "magic           2\n",
       "race          Man\n",
       "stealth         9\n",
       "Name: Aragorn, dtype: object"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_nameindexed.loc['Aragorn']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now recall the Hierarchical indexing from the readings. _(Section 8.1 McKinney)_ We can pass a list of column names to set_index to create a Hierarchical Index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>aggression</th>\n",
       "      <th>magic</th>\n",
       "      <th>stealth</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>race</th>\n",
       "      <th>name</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Maia</th>\n",
       "      <th>Gandalf</th>\n",
       "      <td>7.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Dwarf</th>\n",
       "      <th>Gimli</th>\n",
       "      <td>10.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Hobbit</th>\n",
       "      <th>Frodo</th>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Elf</th>\n",
       "      <th>Legolas</th>\n",
       "      <td>5.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">Hobbit</th>\n",
       "      <th>Bilbo</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sam</th>\n",
       "      <td>6.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pippin</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Man</th>\n",
       "      <th>Boromir</th>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Aragorn</th>\n",
       "      <td>7.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Elf</th>\n",
       "      <th>Galadriel</th>\n",
       "      <td>2.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">Hobbit</th>\n",
       "      <th>Meriadoc</th>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lily</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  aggression  magic  stealth\n",
       "race   name                                 \n",
       "Maia   Gandalf           7.0   10.0      8.0\n",
       "Dwarf  Gimli            10.0    1.0      2.0\n",
       "Hobbit Frodo             2.0    4.0      5.0\n",
       "Elf    Legolas           5.0    6.0     10.0\n",
       "Hobbit Bilbo             1.0    4.0      5.0\n",
       "       Sam               6.0    2.0      4.0\n",
       "       Pippin            3.0    0.0      5.0\n",
       "Man    Boromir           8.0    0.0      3.0\n",
       "       Aragorn           7.0    2.0      9.0\n",
       "Elf    Galadriel         2.0    9.0     10.0\n",
       "Hobbit Meriadoc          4.0    0.0      6.0\n",
       "       Lily              NaN    NaN      NaN"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_racename_indexed = df.set_index(['race','name'])\n",
    "df_racename_indexed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MultiIndex(levels=[[u'Dwarf', u'Elf', u'Hobbit', u'Maia', u'Man'], [u'Aragorn', u'Bilbo', u'Boromir', u'Frodo', u'Galadriel', u'Gandalf', u'Gimli', u'Legolas', u'Lily', u'Meriadoc', u'Pippin', u'Sam']],\n",
       "           labels=[[3, 0, 2, 1, 2, 2, 2, 4, 4, 1, 2, 2], [5, 6, 3, 7, 1, 11, 10, 2, 0, 4, 9, 8]],\n",
       "           names=[u'race', u'name'])"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_racename_indexed.index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will allow us to get a DataFrame that matches a value on the outer index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>aggression</th>\n",
       "      <th>magic</th>\n",
       "      <th>stealth</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>name</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Frodo</th>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Bilbo</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sam</th>\n",
       "      <td>6.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pippin</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Meriadoc</th>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lily</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          aggression  magic  stealth\n",
       "name                                \n",
       "Frodo            2.0    4.0      5.0\n",
       "Bilbo            1.0    4.0      5.0\n",
       "Sam              6.0    2.0      4.0\n",
       "Pippin           3.0    0.0      5.0\n",
       "Meriadoc         4.0    0.0      6.0\n",
       "Lily             NaN    NaN      NaN"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_racename_indexed.loc['Hobbit'] # return all the names under hobbit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use the index on a Series to match the outer index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "name\n",
       "Frodo       4.0\n",
       "Bilbo       4.0\n",
       "Sam         2.0\n",
       "Pippin      0.0\n",
       "Meriadoc    0.0\n",
       "Lily        NaN\n",
       "Name: magic, dtype: float64"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_racename_indexed['magic'].loc['Hobbit']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or both indexes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.0"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_racename_indexed['magic'].loc['Hobbit','Frodo']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or just the inner index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "race\n",
       "Hobbit    4.0\n",
       "Name: magic, dtype: float64"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_racename_indexed['magic'].loc[:,'Frodo']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color=\"magenta\"> Q6: (2 point) Using .loc find how much aggression Legolas, an Elf, has."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {
    "knowledge_building": {
     "cell_groupid": "c04-q4",
     "cell_type": "rcode_answer"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5.0"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Insert your code here\n",
    "df_racename_indexed['aggression'].loc['Elf', 'Legolas']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stacking and Unstacking"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stacking takes \"wide\" data and makes it \"taller\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "race              \n",
       "Maia    aggression            7\n",
       "        magic                10\n",
       "        name            Gandalf\n",
       "        stealth               8\n",
       "Dwarf   aggression           10\n",
       "        magic                 1\n",
       "        name              Gimli\n",
       "        stealth               2\n",
       "Hobbit  aggression            2\n",
       "        magic                 4\n",
       "        name              Frodo\n",
       "        stealth               5\n",
       "Elf     aggression            5\n",
       "        magic                 6\n",
       "        name            Legolas\n",
       "        stealth              10\n",
       "Hobbit  aggression            1\n",
       "        magic                 4\n",
       "        name              Bilbo\n",
       "        stealth               5\n",
       "        aggression            6\n",
       "        magic                 2\n",
       "        name                Sam\n",
       "        stealth               4\n",
       "        aggression            3\n",
       "        magic                 0\n",
       "        name             Pippin\n",
       "        stealth               5\n",
       "Man     aggression            8\n",
       "        magic                 0\n",
       "        name            Boromir\n",
       "        stealth               3\n",
       "        aggression            7\n",
       "        magic                 2\n",
       "        name            Aragorn\n",
       "        stealth               9\n",
       "Elf     aggression            2\n",
       "        magic                 9\n",
       "        name          Galadriel\n",
       "        stealth              10\n",
       "Hobbit  aggression            4\n",
       "        magic                 0\n",
       "        name           Meriadoc\n",
       "        stealth               6\n",
       "        name               Lily\n",
       "dtype: object"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.set_index(['race']).stack()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we call reset_index on the resulting Series, we get the following DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>race</th>\n",
       "      <th>level_1</th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Maia</td>\n",
       "      <td>aggression</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Maia</td>\n",
       "      <td>magic</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Maia</td>\n",
       "      <td>name</td>\n",
       "      <td>Gandalf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Maia</td>\n",
       "      <td>stealth</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Dwarf</td>\n",
       "      <td>aggression</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Dwarf</td>\n",
       "      <td>magic</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Dwarf</td>\n",
       "      <td>name</td>\n",
       "      <td>Gimli</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Dwarf</td>\n",
       "      <td>stealth</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>aggression</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>magic</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>name</td>\n",
       "      <td>Frodo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>stealth</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Elf</td>\n",
       "      <td>aggression</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Elf</td>\n",
       "      <td>magic</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Elf</td>\n",
       "      <td>name</td>\n",
       "      <td>Legolas</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Elf</td>\n",
       "      <td>stealth</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>aggression</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>magic</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>name</td>\n",
       "      <td>Bilbo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>stealth</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>aggression</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>magic</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>name</td>\n",
       "      <td>Sam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>stealth</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>aggression</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>magic</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>name</td>\n",
       "      <td>Pippin</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>stealth</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>Man</td>\n",
       "      <td>aggression</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>Man</td>\n",
       "      <td>magic</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>Man</td>\n",
       "      <td>name</td>\n",
       "      <td>Boromir</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>Man</td>\n",
       "      <td>stealth</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>Man</td>\n",
       "      <td>aggression</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>Man</td>\n",
       "      <td>magic</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>Man</td>\n",
       "      <td>name</td>\n",
       "      <td>Aragorn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>Man</td>\n",
       "      <td>stealth</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>Elf</td>\n",
       "      <td>aggression</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>Elf</td>\n",
       "      <td>magic</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>Elf</td>\n",
       "      <td>name</td>\n",
       "      <td>Galadriel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>Elf</td>\n",
       "      <td>stealth</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>aggression</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>magic</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>name</td>\n",
       "      <td>Meriadoc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>stealth</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>name</td>\n",
       "      <td>Lily</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      race     level_1          0\n",
       "0     Maia  aggression          7\n",
       "1     Maia       magic         10\n",
       "2     Maia        name    Gandalf\n",
       "3     Maia     stealth          8\n",
       "4    Dwarf  aggression         10\n",
       "5    Dwarf       magic          1\n",
       "6    Dwarf        name      Gimli\n",
       "7    Dwarf     stealth          2\n",
       "8   Hobbit  aggression          2\n",
       "9   Hobbit       magic          4\n",
       "10  Hobbit        name      Frodo\n",
       "11  Hobbit     stealth          5\n",
       "12     Elf  aggression          5\n",
       "13     Elf       magic          6\n",
       "14     Elf        name    Legolas\n",
       "15     Elf     stealth         10\n",
       "16  Hobbit  aggression          1\n",
       "17  Hobbit       magic          4\n",
       "18  Hobbit        name      Bilbo\n",
       "19  Hobbit     stealth          5\n",
       "20  Hobbit  aggression          6\n",
       "21  Hobbit       magic          2\n",
       "22  Hobbit        name        Sam\n",
       "23  Hobbit     stealth          4\n",
       "24  Hobbit  aggression          3\n",
       "25  Hobbit       magic          0\n",
       "26  Hobbit        name     Pippin\n",
       "27  Hobbit     stealth          5\n",
       "28     Man  aggression          8\n",
       "29     Man       magic          0\n",
       "30     Man        name    Boromir\n",
       "31     Man     stealth          3\n",
       "32     Man  aggression          7\n",
       "33     Man       magic          2\n",
       "34     Man        name    Aragorn\n",
       "35     Man     stealth          9\n",
       "36     Elf  aggression          2\n",
       "37     Elf       magic          9\n",
       "38     Elf        name  Galadriel\n",
       "39     Elf     stealth         10\n",
       "40  Hobbit  aggression          4\n",
       "41  Hobbit       magic          0\n",
       "42  Hobbit        name   Meriadoc\n",
       "43  Hobbit     stealth          6\n",
       "44  Hobbit        name       Lily"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.set_index(['race']).stack().reset_index()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The column names in the above DataFrame aren't particularly helpful, so we can rename them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>race</th>\n",
       "      <th>variable</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Maia</td>\n",
       "      <td>aggression</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Maia</td>\n",
       "      <td>magic</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Maia</td>\n",
       "      <td>name</td>\n",
       "      <td>Gandalf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Maia</td>\n",
       "      <td>stealth</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Dwarf</td>\n",
       "      <td>aggression</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Dwarf</td>\n",
       "      <td>magic</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Dwarf</td>\n",
       "      <td>name</td>\n",
       "      <td>Gimli</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Dwarf</td>\n",
       "      <td>stealth</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>aggression</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>magic</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>name</td>\n",
       "      <td>Frodo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>stealth</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Elf</td>\n",
       "      <td>aggression</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Elf</td>\n",
       "      <td>magic</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Elf</td>\n",
       "      <td>name</td>\n",
       "      <td>Legolas</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Elf</td>\n",
       "      <td>stealth</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>aggression</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>magic</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>name</td>\n",
       "      <td>Bilbo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>stealth</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>aggression</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>magic</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>name</td>\n",
       "      <td>Sam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>stealth</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>aggression</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>magic</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>name</td>\n",
       "      <td>Pippin</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>stealth</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>Man</td>\n",
       "      <td>aggression</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>Man</td>\n",
       "      <td>magic</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>Man</td>\n",
       "      <td>name</td>\n",
       "      <td>Boromir</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>Man</td>\n",
       "      <td>stealth</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>Man</td>\n",
       "      <td>aggression</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>Man</td>\n",
       "      <td>magic</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>Man</td>\n",
       "      <td>name</td>\n",
       "      <td>Aragorn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>Man</td>\n",
       "      <td>stealth</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>Elf</td>\n",
       "      <td>aggression</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>Elf</td>\n",
       "      <td>magic</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>Elf</td>\n",
       "      <td>name</td>\n",
       "      <td>Galadriel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>Elf</td>\n",
       "      <td>stealth</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>aggression</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>magic</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>name</td>\n",
       "      <td>Meriadoc</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>stealth</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>Hobbit</td>\n",
       "      <td>name</td>\n",
       "      <td>Lily</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      race    variable      value\n",
       "0     Maia  aggression          7\n",
       "1     Maia       magic         10\n",
       "2     Maia        name    Gandalf\n",
       "3     Maia     stealth          8\n",
       "4    Dwarf  aggression         10\n",
       "5    Dwarf       magic          1\n",
       "6    Dwarf        name      Gimli\n",
       "7    Dwarf     stealth          2\n",
       "8   Hobbit  aggression          2\n",
       "9   Hobbit       magic          4\n",
       "10  Hobbit        name      Frodo\n",
       "11  Hobbit     stealth          5\n",
       "12     Elf  aggression          5\n",
       "13     Elf       magic          6\n",
       "14     Elf        name    Legolas\n",
       "15     Elf     stealth         10\n",
       "16  Hobbit  aggression          1\n",
       "17  Hobbit       magic          4\n",
       "18  Hobbit        name      Bilbo\n",
       "19  Hobbit     stealth          5\n",
       "20  Hobbit  aggression          6\n",
       "21  Hobbit       magic          2\n",
       "22  Hobbit        name        Sam\n",
       "23  Hobbit     stealth          4\n",
       "24  Hobbit  aggression          3\n",
       "25  Hobbit       magic          0\n",
       "26  Hobbit        name     Pippin\n",
       "27  Hobbit     stealth          5\n",
       "28     Man  aggression          8\n",
       "29     Man       magic          0\n",
       "30     Man        name    Boromir\n",
       "31     Man     stealth          3\n",
       "32     Man  aggression          7\n",
       "33     Man       magic          2\n",
       "34     Man        name    Aragorn\n",
       "35     Man     stealth          9\n",
       "36     Elf  aggression          2\n",
       "37     Elf       magic          9\n",
       "38     Elf        name  Galadriel\n",
       "39     Elf     stealth         10\n",
       "40  Hobbit  aggression          4\n",
       "41  Hobbit       magic          0\n",
       "42  Hobbit        name   Meriadoc\n",
       "43  Hobbit     stealth          6\n",
       "44  Hobbit        name       Lily"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.set_index(['race']).stack().reset_index().rename(columns = {'level_0':'ID','level_1':'variable',0:'value'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can do the opposite of stacking by using the ```unstack``` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_stacked = df.stack()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0   aggression            7\n",
       "    magic                10\n",
       "    name            Gandalf\n",
       "    race               Maia\n",
       "    stealth               8\n",
       "1   aggression           10\n",
       "    magic                 1\n",
       "    name              Gimli\n",
       "    race              Dwarf\n",
       "    stealth               2\n",
       "2   aggression            2\n",
       "    magic                 4\n",
       "    name              Frodo\n",
       "    race             Hobbit\n",
       "    stealth               5\n",
       "3   aggression            5\n",
       "    magic                 6\n",
       "    name            Legolas\n",
       "    race                Elf\n",
       "    stealth              10\n",
       "4   aggression            1\n",
       "    magic                 4\n",
       "    name              Bilbo\n",
       "    race             Hobbit\n",
       "    stealth               5\n",
       "5   aggression            6\n",
       "    magic                 2\n",
       "    name                Sam\n",
       "    race             Hobbit\n",
       "    stealth               4\n",
       "6   aggression            3\n",
       "    magic                 0\n",
       "    name             Pippin\n",
       "    race             Hobbit\n",
       "    stealth               5\n",
       "7   aggression            8\n",
       "    magic                 0\n",
       "    name            Boromir\n",
       "    race                Man\n",
       "    stealth               3\n",
       "8   aggression            7\n",
       "    magic                 2\n",
       "    name            Aragorn\n",
       "    race                Man\n",
       "    stealth               9\n",
       "9   aggression            2\n",
       "    magic                 9\n",
       "    name          Galadriel\n",
       "    race                Elf\n",
       "    stealth              10\n",
       "10  aggression            4\n",
       "    magic                 0\n",
       "    name           Meriadoc\n",
       "    race             Hobbit\n",
       "    stealth               6\n",
       "11  name               Lily\n",
       "    race             Hobbit\n",
       "dtype: object"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_stacked"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>aggression</th>\n",
       "      <th>magic</th>\n",
       "      <th>name</th>\n",
       "      <th>race</th>\n",
       "      <th>stealth</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7</td>\n",
       "      <td>10</td>\n",
       "      <td>Gandalf</td>\n",
       "      <td>Maia</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10</td>\n",
       "      <td>1</td>\n",
       "      <td>Gimli</td>\n",
       "      <td>Dwarf</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>Frodo</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>Legolas</td>\n",
       "      <td>Elf</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>Bilbo</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>Sam</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>Pippin</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>Boromir</td>\n",
       "      <td>Man</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>7</td>\n",
       "      <td>2</td>\n",
       "      <td>Aragorn</td>\n",
       "      <td>Man</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>Galadriel</td>\n",
       "      <td>Elf</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>Meriadoc</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>Lily</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   aggression magic       name    race stealth\n",
       "0           7    10    Gandalf    Maia       8\n",
       "1          10     1      Gimli   Dwarf       2\n",
       "2           2     4      Frodo  Hobbit       5\n",
       "3           5     6    Legolas     Elf      10\n",
       "4           1     4      Bilbo  Hobbit       5\n",
       "5           6     2        Sam  Hobbit       4\n",
       "6           3     0     Pippin  Hobbit       5\n",
       "7           8     0    Boromir     Man       3\n",
       "8           7     2    Aragorn     Man       9\n",
       "9           2     9  Galadriel     Elf      10\n",
       "10          4     0   Meriadoc  Hobbit       6\n",
       "11       None  None       Lily  Hobbit    None"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_stacked.unstack()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why would we want to stack or unstack?  It depends on what sorts of analyses we want to do \"downstream\".  It's also the basis for pivoting, melting, and pivot tables, which we'll cover in the next class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Joining Data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Let's say we have another CSV file that contains URLs to Wikipedia pages for some of the LOTR characters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>name</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>Gandalf</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Gandalf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Gimli</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Gimli_(Middle-ea...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>Frodo</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Frodo_Baggins</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>Legolas</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Legolas</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>Bilbo</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Bilbo_Baggins</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>5</td>\n",
       "      <td>Sam</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Samwise_Gamgee</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>6</td>\n",
       "      <td>Pippin</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Peregrin_Took</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>7</td>\n",
       "      <td>Boromir</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Boromir</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>8</td>\n",
       "      <td>Aragorn</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Aragorn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>9</td>\n",
       "      <td>Galadriel</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Galadriel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>10</td>\n",
       "      <td>Meriadoc</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Meriadoc_Brandybuck</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>11</td>\n",
       "      <td>Lily</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Unnamed: 0       name                                                url\n",
       "0            0    Gandalf              https://en.wikipedia.org/wiki/Gandalf\n",
       "1            1      Gimli  https://en.wikipedia.org/wiki/Gimli_(Middle-ea...\n",
       "2            2      Frodo        https://en.wikipedia.org/wiki/Frodo_Baggins\n",
       "3            3    Legolas              https://en.wikipedia.org/wiki/Legolas\n",
       "4            4      Bilbo        https://en.wikipedia.org/wiki/Bilbo_Baggins\n",
       "5            5        Sam       https://en.wikipedia.org/wiki/Samwise_Gamgee\n",
       "6            6     Pippin        https://en.wikipedia.org/wiki/Peregrin_Took\n",
       "7            7    Boromir              https://en.wikipedia.org/wiki/Boromir\n",
       "8            8    Aragorn              https://en.wikipedia.org/wiki/Aragorn\n",
       "9            9  Galadriel            https://en.wikipedia.org/wiki/Galadriel\n",
       "10          10   Meriadoc  https://en.wikipedia.org/wiki/Meriadoc_Brandybuck\n",
       "11          11       Lily                                                NaN"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "urls = pd.read_csv('data/lotr_wikipedia.csv')\n",
    "urls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [],
   "source": [
    "urls = pd.read_csv('data/lotr_wikipedia.csv', index_col=0) # index_col: making o column to be index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Gandalf</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Gandalf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Gimli</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Gimli_(Middle-ea...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Frodo</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Frodo_Baggins</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Legolas</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Legolas</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Bilbo</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Bilbo_Baggins</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Sam</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Samwise_Gamgee</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Pippin</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Peregrin_Took</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Boromir</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Boromir</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Aragorn</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Aragorn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Galadriel</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Galadriel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Meriadoc</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Meriadoc_Brandybuck</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Lily</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         name                                                url\n",
       "0     Gandalf              https://en.wikipedia.org/wiki/Gandalf\n",
       "1       Gimli  https://en.wikipedia.org/wiki/Gimli_(Middle-ea...\n",
       "2       Frodo        https://en.wikipedia.org/wiki/Frodo_Baggins\n",
       "3     Legolas              https://en.wikipedia.org/wiki/Legolas\n",
       "4       Bilbo        https://en.wikipedia.org/wiki/Bilbo_Baggins\n",
       "5         Sam       https://en.wikipedia.org/wiki/Samwise_Gamgee\n",
       "6      Pippin        https://en.wikipedia.org/wiki/Peregrin_Took\n",
       "7     Boromir              https://en.wikipedia.org/wiki/Boromir\n",
       "8     Aragorn              https://en.wikipedia.org/wiki/Aragorn\n",
       "9   Galadriel            https://en.wikipedia.org/wiki/Galadriel\n",
       "10   Meriadoc  https://en.wikipedia.org/wiki/Meriadoc_Brandybuck\n",
       "11       Lily                                                NaN"
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "urls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the original DataFrame:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like the rows are \"aligned\", so we can use the ```concat``` function to concatenate the two DataFrames.\n",
    "Note that we specify the axis to be the columns.  The default is to concatenate by rows, which isn't what we want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>aggression</th>\n",
       "      <th>magic</th>\n",
       "      <th>name</th>\n",
       "      <th>race</th>\n",
       "      <th>stealth</th>\n",
       "      <th>name</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>Gandalf</td>\n",
       "      <td>Maia</td>\n",
       "      <td>8.0</td>\n",
       "      <td>Gandalf</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Gandalf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Gimli</td>\n",
       "      <td>Dwarf</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Gimli</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Gimli_(Middle-ea...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Frodo</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Frodo</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Frodo_Baggins</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Legolas</td>\n",
       "      <td>Elf</td>\n",
       "      <td>10.0</td>\n",
       "      <td>Legolas</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Legolas</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Bilbo</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Bilbo</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Bilbo_Baggins</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Sam</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Sam</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Samwise_Gamgee</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Pippin</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "      <td>Pippin</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Peregrin_Took</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Boromir</td>\n",
       "      <td>Man</td>\n",
       "      <td>3.0</td>\n",
       "      <td>Boromir</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Boromir</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>7.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Aragorn</td>\n",
       "      <td>Man</td>\n",
       "      <td>9.0</td>\n",
       "      <td>Aragorn</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Aragorn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>Galadriel</td>\n",
       "      <td>Elf</td>\n",
       "      <td>10.0</td>\n",
       "      <td>Galadriel</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Galadriel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Meriadoc</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Meriadoc</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Meriadoc_Brandybuck</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Lily</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Lily</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    aggression  magic       name    race  stealth       name  \\\n",
       "0          7.0   10.0    Gandalf    Maia      8.0    Gandalf   \n",
       "1         10.0    1.0      Gimli   Dwarf      2.0      Gimli   \n",
       "2          2.0    4.0      Frodo  Hobbit      5.0      Frodo   \n",
       "3          5.0    6.0    Legolas     Elf     10.0    Legolas   \n",
       "4          1.0    4.0      Bilbo  Hobbit      5.0      Bilbo   \n",
       "5          6.0    2.0        Sam  Hobbit      4.0        Sam   \n",
       "6          3.0    0.0     Pippin  Hobbit      5.0     Pippin   \n",
       "7          8.0    0.0    Boromir     Man      3.0    Boromir   \n",
       "8          7.0    2.0    Aragorn     Man      9.0    Aragorn   \n",
       "9          2.0    9.0  Galadriel     Elf     10.0  Galadriel   \n",
       "10         4.0    0.0   Meriadoc  Hobbit      6.0   Meriadoc   \n",
       "11         NaN    NaN       Lily  Hobbit      NaN       Lily   \n",
       "\n",
       "                                                  url  \n",
       "0               https://en.wikipedia.org/wiki/Gandalf  \n",
       "1   https://en.wikipedia.org/wiki/Gimli_(Middle-ea...  \n",
       "2         https://en.wikipedia.org/wiki/Frodo_Baggins  \n",
       "3               https://en.wikipedia.org/wiki/Legolas  \n",
       "4         https://en.wikipedia.org/wiki/Bilbo_Baggins  \n",
       "5        https://en.wikipedia.org/wiki/Samwise_Gamgee  \n",
       "6         https://en.wikipedia.org/wiki/Peregrin_Took  \n",
       "7               https://en.wikipedia.org/wiki/Boromir  \n",
       "8               https://en.wikipedia.org/wiki/Aragorn  \n",
       "9             https://en.wikipedia.org/wiki/Galadriel  \n",
       "10  https://en.wikipedia.org/wiki/Meriadoc_Brandybuck  \n",
       "11                                                NaN  "
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.concat([df,urls],axis=\"columns\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's great, and it's consistent with what we've used in previous classes.  But what happens if the \n",
    "rows in the two DataFrames don't match up?  Let's load another file that has a slightly different\n",
    "sequence of rows:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color=\"magenta\"> Q7: (2 point) Construct a dataframe with lotr_wikipedia_wrong_order.csv which is in the data folder and concat it with df."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Boromir</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Boromir</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Aragorn</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Aragorn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Gandalf</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Gandalf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Meriadoc</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Meriadoc_Brandybuck</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Gimli</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Gimli_(Middle-ea...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Legolas</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Legolas</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Bilbo</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Bilbo_Baggins</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Pippin</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Peregrin_Took</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Galadriel</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Galadriel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Lily</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Frodo</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Frodo_Baggins</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Sam</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Samwise_Gamgee</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         name                                                url\n",
       "0     Boromir              https://en.wikipedia.org/wiki/Boromir\n",
       "1     Aragorn              https://en.wikipedia.org/wiki/Aragorn\n",
       "2     Gandalf              https://en.wikipedia.org/wiki/Gandalf\n",
       "3    Meriadoc  https://en.wikipedia.org/wiki/Meriadoc_Brandybuck\n",
       "4       Gimli  https://en.wikipedia.org/wiki/Gimli_(Middle-ea...\n",
       "5     Legolas              https://en.wikipedia.org/wiki/Legolas\n",
       "6       Bilbo        https://en.wikipedia.org/wiki/Bilbo_Baggins\n",
       "7      Pippin        https://en.wikipedia.org/wiki/Peregrin_Took\n",
       "8   Galadriel            https://en.wikipedia.org/wiki/Galadriel\n",
       "9        Lily                                                NaN\n",
       "10      Frodo        https://en.wikipedia.org/wiki/Frodo_Baggins\n",
       "11        Sam       https://en.wikipedia.org/wiki/Samwise_Gamgee"
      ]
     },
     "execution_count": 150,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Insert your code here\n",
    "urls_wrong_order = pd.read_csv('data/lotr_wikipedia_wrong_order.csv') # , index_col= 'name'\n",
    "urls_wrong_order\n",
    "# df.set_index(['name']) # should use : df_names = df.set_index('name') \n",
    "# new = pd.concat([df,url_wrong],axis=\"name\")\n",
    "# new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a closer look at the name and url columns.  Something's not quite right.\n",
    "\n",
    "We can work around that by using the appropriate indexing and then using the SQL-like ```merge``` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>aggression</th>\n",
       "      <th>magic</th>\n",
       "      <th>name</th>\n",
       "      <th>race</th>\n",
       "      <th>stealth</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>Gandalf</td>\n",
       "      <td>Maia</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Gimli</td>\n",
       "      <td>Dwarf</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Frodo</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Legolas</td>\n",
       "      <td>Elf</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Bilbo</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Sam</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Pippin</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Boromir</td>\n",
       "      <td>Man</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>7.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Aragorn</td>\n",
       "      <td>Man</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>Galadriel</td>\n",
       "      <td>Elf</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Meriadoc</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Lily</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    aggression  magic       name    race  stealth\n",
       "0          7.0   10.0    Gandalf    Maia      8.0\n",
       "1         10.0    1.0      Gimli   Dwarf      2.0\n",
       "2          2.0    4.0      Frodo  Hobbit      5.0\n",
       "3          5.0    6.0    Legolas     Elf     10.0\n",
       "4          1.0    4.0      Bilbo  Hobbit      5.0\n",
       "5          6.0    2.0        Sam  Hobbit      4.0\n",
       "6          3.0    0.0     Pippin  Hobbit      5.0\n",
       "7          8.0    0.0    Boromir     Man      3.0\n",
       "8          7.0    2.0    Aragorn     Man      9.0\n",
       "9          2.0    9.0  Galadriel     Elf     10.0\n",
       "10         4.0    0.0   Meriadoc  Hobbit      6.0\n",
       "11         NaN    NaN       Lily  Hobbit      NaN"
      ]
     },
     "execution_count": 143,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_names = df.set_index('name')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>aggression</th>\n",
       "      <th>magic</th>\n",
       "      <th>race</th>\n",
       "      <th>stealth</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>name</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Gandalf</th>\n",
       "      <td>7.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>Maia</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Gimli</th>\n",
       "      <td>10.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Dwarf</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Frodo</th>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Legolas</th>\n",
       "      <td>5.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Elf</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Bilbo</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sam</th>\n",
       "      <td>6.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pippin</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Boromir</th>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Aragorn</th>\n",
       "      <td>7.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Galadriel</th>\n",
       "      <td>2.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>Elf</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Meriadoc</th>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lily</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           aggression  magic    race  stealth\n",
       "name                                         \n",
       "Gandalf           7.0   10.0    Maia      8.0\n",
       "Gimli            10.0    1.0   Dwarf      2.0\n",
       "Frodo             2.0    4.0  Hobbit      5.0\n",
       "Legolas           5.0    6.0     Elf     10.0\n",
       "Bilbo             1.0    4.0  Hobbit      5.0\n",
       "Sam               6.0    2.0  Hobbit      4.0\n",
       "Pippin            3.0    0.0  Hobbit      5.0\n",
       "Boromir           8.0    0.0     Man      3.0\n",
       "Aragorn           7.0    2.0     Man      9.0\n",
       "Galadriel         2.0    9.0     Elf     10.0\n",
       "Meriadoc          4.0    0.0  Hobbit      6.0\n",
       "Lily              NaN    NaN  Hobbit      NaN"
      ]
     },
     "execution_count": 145,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [],
   "source": [
    "urls_wrong_order_names = urls_wrong_order.set_index('name')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>aggression</th>\n",
       "      <th>magic</th>\n",
       "      <th>race</th>\n",
       "      <th>stealth</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>name</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Gandalf</th>\n",
       "      <td>7.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>Maia</td>\n",
       "      <td>8.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Gandalf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Gimli</th>\n",
       "      <td>10.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Dwarf</td>\n",
       "      <td>2.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Gimli_(Middle-ea...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Frodo</th>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Frodo_Baggins</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Legolas</th>\n",
       "      <td>5.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Elf</td>\n",
       "      <td>10.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Legolas</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Bilbo</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Bilbo_Baggins</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Sam</th>\n",
       "      <td>6.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>4.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Samwise_Gamgee</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pippin</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Peregrin_Took</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Boromir</th>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>3.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Boromir</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Aragorn</th>\n",
       "      <td>7.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>9.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Aragorn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Galadriel</th>\n",
       "      <td>2.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>Elf</td>\n",
       "      <td>10.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Galadriel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Meriadoc</th>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>6.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Meriadoc_Brandybuck</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lily</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           aggression  magic    race  stealth  \\\n",
       "name                                            \n",
       "Gandalf           7.0   10.0    Maia      8.0   \n",
       "Gimli            10.0    1.0   Dwarf      2.0   \n",
       "Frodo             2.0    4.0  Hobbit      5.0   \n",
       "Legolas           5.0    6.0     Elf     10.0   \n",
       "Bilbo             1.0    4.0  Hobbit      5.0   \n",
       "Sam               6.0    2.0  Hobbit      4.0   \n",
       "Pippin            3.0    0.0  Hobbit      5.0   \n",
       "Boromir           8.0    0.0     Man      3.0   \n",
       "Aragorn           7.0    2.0     Man      9.0   \n",
       "Galadriel         2.0    9.0     Elf     10.0   \n",
       "Meriadoc          4.0    0.0  Hobbit      6.0   \n",
       "Lily              NaN    NaN  Hobbit      NaN   \n",
       "\n",
       "                                                         url  \n",
       "name                                                          \n",
       "Gandalf                https://en.wikipedia.org/wiki/Gandalf  \n",
       "Gimli      https://en.wikipedia.org/wiki/Gimli_(Middle-ea...  \n",
       "Frodo            https://en.wikipedia.org/wiki/Frodo_Baggins  \n",
       "Legolas                https://en.wikipedia.org/wiki/Legolas  \n",
       "Bilbo            https://en.wikipedia.org/wiki/Bilbo_Baggins  \n",
       "Sam             https://en.wikipedia.org/wiki/Samwise_Gamgee  \n",
       "Pippin           https://en.wikipedia.org/wiki/Peregrin_Took  \n",
       "Boromir                https://en.wikipedia.org/wiki/Boromir  \n",
       "Aragorn                https://en.wikipedia.org/wiki/Aragorn  \n",
       "Galadriel            https://en.wikipedia.org/wiki/Galadriel  \n",
       "Meriadoc   https://en.wikipedia.org/wiki/Meriadoc_Brandybuck  \n",
       "Lily                                                     NaN  "
      ]
     },
     "execution_count": 152,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_names.join(urls_wrong_order_names) # join instead of concate because the indexs are the same"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>aggression</th>\n",
       "      <th>magic</th>\n",
       "      <th>name</th>\n",
       "      <th>race</th>\n",
       "      <th>stealth</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>Gandalf</td>\n",
       "      <td>Maia</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Gimli</td>\n",
       "      <td>Dwarf</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Frodo</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Legolas</td>\n",
       "      <td>Elf</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Bilbo</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   aggression  magic     name    race  stealth\n",
       "0         7.0   10.0  Gandalf    Maia      8.0\n",
       "1        10.0    1.0    Gimli   Dwarf      2.0\n",
       "2         2.0    4.0    Frodo  Hobbit      5.0\n",
       "3         5.0    6.0  Legolas     Elf     10.0\n",
       "4         1.0    4.0    Bilbo  Hobbit      5.0"
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Boromir</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Boromir</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Aragorn</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Aragorn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Gandalf</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Gandalf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Meriadoc</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Meriadoc_Brandybuck</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Gimli</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Gimli_(Middle-ea...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       name                                                url\n",
       "0   Boromir              https://en.wikipedia.org/wiki/Boromir\n",
       "1   Aragorn              https://en.wikipedia.org/wiki/Aragorn\n",
       "2   Gandalf              https://en.wikipedia.org/wiki/Gandalf\n",
       "3  Meriadoc  https://en.wikipedia.org/wiki/Meriadoc_Brandybuck\n",
       "4     Gimli  https://en.wikipedia.org/wiki/Gimli_(Middle-ea..."
      ]
     },
     "execution_count": 154,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "urls_wrong_order.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       Boromir\n",
       "1       Aragorn\n",
       "2       Gandalf\n",
       "3      Meriadoc\n",
       "4         Gimli\n",
       "5       Legolas\n",
       "6         Bilbo\n",
       "7        Pippin\n",
       "8     Galadriel\n",
       "9          Lily\n",
       "10        Frodo\n",
       "11          Sam\n",
       "Name: name, dtype: object"
      ]
     },
     "execution_count": 155,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "urls_wrong_order['name']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0       Gandalf\n",
       "1         Gimli\n",
       "2         Frodo\n",
       "3       Legolas\n",
       "4         Bilbo\n",
       "5           Sam\n",
       "6        Pippin\n",
       "7       Boromir\n",
       "8       Aragorn\n",
       "9     Galadriel\n",
       "10     Meriadoc\n",
       "11         Lily\n",
       "Name: name, dtype: object"
      ]
     },
     "execution_count": 156,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['name']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>aggression</th>\n",
       "      <th>magic</th>\n",
       "      <th>name</th>\n",
       "      <th>race</th>\n",
       "      <th>stealth</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>Gandalf</td>\n",
       "      <td>Maia</td>\n",
       "      <td>8.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Gandalf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Gimli</td>\n",
       "      <td>Dwarf</td>\n",
       "      <td>2.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Gimli_(Middle-ea...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Frodo</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Frodo_Baggins</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Legolas</td>\n",
       "      <td>Elf</td>\n",
       "      <td>10.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Legolas</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Bilbo</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Bilbo_Baggins</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Sam</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>4.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Samwise_Gamgee</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Pippin</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Peregrin_Took</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Boromir</td>\n",
       "      <td>Man</td>\n",
       "      <td>3.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Boromir</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>7.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Aragorn</td>\n",
       "      <td>Man</td>\n",
       "      <td>9.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Aragorn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>Galadriel</td>\n",
       "      <td>Elf</td>\n",
       "      <td>10.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Galadriel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Meriadoc</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>6.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Meriadoc_Brandybuck</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Lily</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    aggression  magic       name    race  stealth  \\\n",
       "0          7.0   10.0    Gandalf    Maia      8.0   \n",
       "1         10.0    1.0      Gimli   Dwarf      2.0   \n",
       "2          2.0    4.0      Frodo  Hobbit      5.0   \n",
       "3          5.0    6.0    Legolas     Elf     10.0   \n",
       "4          1.0    4.0      Bilbo  Hobbit      5.0   \n",
       "5          6.0    2.0        Sam  Hobbit      4.0   \n",
       "6          3.0    0.0     Pippin  Hobbit      5.0   \n",
       "7          8.0    0.0    Boromir     Man      3.0   \n",
       "8          7.0    2.0    Aragorn     Man      9.0   \n",
       "9          2.0    9.0  Galadriel     Elf     10.0   \n",
       "10         4.0    0.0   Meriadoc  Hobbit      6.0   \n",
       "11         NaN    NaN       Lily  Hobbit      NaN   \n",
       "\n",
       "                                                  url  \n",
       "0               https://en.wikipedia.org/wiki/Gandalf  \n",
       "1   https://en.wikipedia.org/wiki/Gimli_(Middle-ea...  \n",
       "2         https://en.wikipedia.org/wiki/Frodo_Baggins  \n",
       "3               https://en.wikipedia.org/wiki/Legolas  \n",
       "4         https://en.wikipedia.org/wiki/Bilbo_Baggins  \n",
       "5        https://en.wikipedia.org/wiki/Samwise_Gamgee  \n",
       "6         https://en.wikipedia.org/wiki/Peregrin_Took  \n",
       "7               https://en.wikipedia.org/wiki/Boromir  \n",
       "8               https://en.wikipedia.org/wiki/Aragorn  \n",
       "9             https://en.wikipedia.org/wiki/Galadriel  \n",
       "10  https://en.wikipedia.org/wiki/Meriadoc_Brandybuck  \n",
       "11                                                NaN  "
      ]
     },
     "execution_count": 157,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.merge(urls_wrong_order,on='name') # can use the column named \"name\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's add a few additional URLs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {},
   "outputs": [],
   "source": [
    "# urls_extras = pd.read_csv(\"data/lotr_wikipedia_extras.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "urls_extras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now let's use concat to add the new entries to the DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "urls_complete = pd.concat([urls,urls_extras])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "urls_complete"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've got a complete (for our purposes) list of URLs, let's use that DataFrame and our original\n",
    "one to demonstrate the different types of ```join```s.\n",
    "\n",
    "By default, ```join``` uses a left join, which means the all the values from the \"left\"\n",
    "side are used, whether or not there's a corresponding entry from the \"right\" side.  In the example \n",
    "below, note that the url value for \"Lily\" is \"NaN\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>aggression</th>\n",
       "      <th>magic</th>\n",
       "      <th>name</th>\n",
       "      <th>race</th>\n",
       "      <th>stealth</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>Gandalf</td>\n",
       "      <td>Maia</td>\n",
       "      <td>8.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Gandalf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Gimli</td>\n",
       "      <td>Dwarf</td>\n",
       "      <td>2.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Gimli_(Middle-ea...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Frodo</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Frodo_Baggins</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Legolas</td>\n",
       "      <td>Elf</td>\n",
       "      <td>10.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Legolas</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Bilbo</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Bilbo_Baggins</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Sam</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>4.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Samwise_Gamgee</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Pippin</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Peregrin_Took</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Boromir</td>\n",
       "      <td>Man</td>\n",
       "      <td>3.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Boromir</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>7.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Aragorn</td>\n",
       "      <td>Man</td>\n",
       "      <td>9.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Aragorn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>Galadriel</td>\n",
       "      <td>Elf</td>\n",
       "      <td>10.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Galadriel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Meriadoc</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>6.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Meriadoc_Brandybuck</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Lily</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    aggression  magic       name    race  stealth  \\\n",
       "0          7.0   10.0    Gandalf    Maia      8.0   \n",
       "1         10.0    1.0      Gimli   Dwarf      2.0   \n",
       "2          2.0    4.0      Frodo  Hobbit      5.0   \n",
       "3          5.0    6.0    Legolas     Elf     10.0   \n",
       "4          1.0    4.0      Bilbo  Hobbit      5.0   \n",
       "5          6.0    2.0        Sam  Hobbit      4.0   \n",
       "6          3.0    0.0     Pippin  Hobbit      5.0   \n",
       "7          8.0    0.0    Boromir     Man      3.0   \n",
       "8          7.0    2.0    Aragorn     Man      9.0   \n",
       "9          2.0    9.0  Galadriel     Elf     10.0   \n",
       "10         4.0    0.0   Meriadoc  Hobbit      6.0   \n",
       "11         NaN    NaN       Lily  Hobbit      NaN   \n",
       "\n",
       "                                                  url  \n",
       "0               https://en.wikipedia.org/wiki/Gandalf  \n",
       "1   https://en.wikipedia.org/wiki/Gimli_(Middle-ea...  \n",
       "2         https://en.wikipedia.org/wiki/Frodo_Baggins  \n",
       "3               https://en.wikipedia.org/wiki/Legolas  \n",
       "4         https://en.wikipedia.org/wiki/Bilbo_Baggins  \n",
       "5        https://en.wikipedia.org/wiki/Samwise_Gamgee  \n",
       "6         https://en.wikipedia.org/wiki/Peregrin_Took  \n",
       "7               https://en.wikipedia.org/wiki/Boromir  \n",
       "8               https://en.wikipedia.org/wiki/Aragorn  \n",
       "9             https://en.wikipedia.org/wiki/Galadriel  \n",
       "10  https://en.wikipedia.org/wiki/Meriadoc_Brandybuck  \n",
       "11                                                NaN  "
      ]
     },
     "execution_count": 160,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.merge(urls_wrong_order,on='name',how='left') # keep all values in the df dataframe: so only 12 rows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The \"opposite\" of a left join is, perhaps unsurprisingly, a \"right\" join, in which\n",
    "all the values from the \"right\" side are used, whether or not a corresponding\n",
    "value from the \"left\" side exists. Note in the following example that \"Lily\" has\n",
    "disappeared, and Treebeard and Elrond lack information about \"race\", \"magic\", \"aggression\", and \"stealth\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>aggression</th>\n",
       "      <th>magic</th>\n",
       "      <th>name</th>\n",
       "      <th>race</th>\n",
       "      <th>stealth</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>Gandalf</td>\n",
       "      <td>Maia</td>\n",
       "      <td>8.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Gandalf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Gimli</td>\n",
       "      <td>Dwarf</td>\n",
       "      <td>2.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Gimli_(Middle-ea...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Frodo</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Frodo_Baggins</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Legolas</td>\n",
       "      <td>Elf</td>\n",
       "      <td>10.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Legolas</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Bilbo</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Bilbo_Baggins</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Sam</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>4.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Samwise_Gamgee</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Pippin</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Peregrin_Took</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Boromir</td>\n",
       "      <td>Man</td>\n",
       "      <td>3.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Boromir</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>7.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Aragorn</td>\n",
       "      <td>Man</td>\n",
       "      <td>9.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Aragorn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>Galadriel</td>\n",
       "      <td>Elf</td>\n",
       "      <td>10.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Galadriel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Meriadoc</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>6.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Meriadoc_Brandybuck</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Lily</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    aggression  magic       name    race  stealth  \\\n",
       "0          7.0   10.0    Gandalf    Maia      8.0   \n",
       "1         10.0    1.0      Gimli   Dwarf      2.0   \n",
       "2          2.0    4.0      Frodo  Hobbit      5.0   \n",
       "3          5.0    6.0    Legolas     Elf     10.0   \n",
       "4          1.0    4.0      Bilbo  Hobbit      5.0   \n",
       "5          6.0    2.0        Sam  Hobbit      4.0   \n",
       "6          3.0    0.0     Pippin  Hobbit      5.0   \n",
       "7          8.0    0.0    Boromir     Man      3.0   \n",
       "8          7.0    2.0    Aragorn     Man      9.0   \n",
       "9          2.0    9.0  Galadriel     Elf     10.0   \n",
       "10         4.0    0.0   Meriadoc  Hobbit      6.0   \n",
       "11         NaN    NaN       Lily  Hobbit      NaN   \n",
       "\n",
       "                                                  url  \n",
       "0               https://en.wikipedia.org/wiki/Gandalf  \n",
       "1   https://en.wikipedia.org/wiki/Gimli_(Middle-ea...  \n",
       "2         https://en.wikipedia.org/wiki/Frodo_Baggins  \n",
       "3               https://en.wikipedia.org/wiki/Legolas  \n",
       "4         https://en.wikipedia.org/wiki/Bilbo_Baggins  \n",
       "5        https://en.wikipedia.org/wiki/Samwise_Gamgee  \n",
       "6         https://en.wikipedia.org/wiki/Peregrin_Took  \n",
       "7               https://en.wikipedia.org/wiki/Boromir  \n",
       "8               https://en.wikipedia.org/wiki/Aragorn  \n",
       "9             https://en.wikipedia.org/wiki/Galadriel  \n",
       "10  https://en.wikipedia.org/wiki/Meriadoc_Brandybuck  \n",
       "11                                                NaN  "
      ]
     },
     "execution_count": 161,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.merge(urls_wrong_order,on='name',how='right')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to \"left\" and \"right\" joins, we have \"outer\" joins, which include\n",
    "values from both the \"left\" and \"right\" DataFrames, regardless of whether\n",
    "there are corresponding values in the other DataFrame.  Note that all of \n",
    "\"Lily\", \"Treebeard\" and \"Elrond\" are present in the following DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>aggression</th>\n",
       "      <th>magic</th>\n",
       "      <th>name</th>\n",
       "      <th>race</th>\n",
       "      <th>stealth</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>Gandalf</td>\n",
       "      <td>Maia</td>\n",
       "      <td>8.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Gandalf</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Gimli</td>\n",
       "      <td>Dwarf</td>\n",
       "      <td>2.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Gimli_(Middle-ea...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Frodo</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Frodo_Baggins</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Legolas</td>\n",
       "      <td>Elf</td>\n",
       "      <td>10.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Legolas</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Bilbo</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Bilbo_Baggins</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Sam</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>4.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Samwise_Gamgee</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Pippin</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Peregrin_Took</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Boromir</td>\n",
       "      <td>Man</td>\n",
       "      <td>3.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Boromir</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>7.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Aragorn</td>\n",
       "      <td>Man</td>\n",
       "      <td>9.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Aragorn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>Galadriel</td>\n",
       "      <td>Elf</td>\n",
       "      <td>10.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Galadriel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Meriadoc</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>6.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Meriadoc_Brandybuck</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Lily</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    aggression  magic       name    race  stealth  \\\n",
       "0          7.0   10.0    Gandalf    Maia      8.0   \n",
       "1         10.0    1.0      Gimli   Dwarf      2.0   \n",
       "2          2.0    4.0      Frodo  Hobbit      5.0   \n",
       "3          5.0    6.0    Legolas     Elf     10.0   \n",
       "4          1.0    4.0      Bilbo  Hobbit      5.0   \n",
       "5          6.0    2.0        Sam  Hobbit      4.0   \n",
       "6          3.0    0.0     Pippin  Hobbit      5.0   \n",
       "7          8.0    0.0    Boromir     Man      3.0   \n",
       "8          7.0    2.0    Aragorn     Man      9.0   \n",
       "9          2.0    9.0  Galadriel     Elf     10.0   \n",
       "10         4.0    0.0   Meriadoc  Hobbit      6.0   \n",
       "11         NaN    NaN       Lily  Hobbit      NaN   \n",
       "\n",
       "                                                  url  \n",
       "0               https://en.wikipedia.org/wiki/Gandalf  \n",
       "1   https://en.wikipedia.org/wiki/Gimli_(Middle-ea...  \n",
       "2         https://en.wikipedia.org/wiki/Frodo_Baggins  \n",
       "3               https://en.wikipedia.org/wiki/Legolas  \n",
       "4         https://en.wikipedia.org/wiki/Bilbo_Baggins  \n",
       "5        https://en.wikipedia.org/wiki/Samwise_Gamgee  \n",
       "6         https://en.wikipedia.org/wiki/Peregrin_Took  \n",
       "7               https://en.wikipedia.org/wiki/Boromir  \n",
       "8               https://en.wikipedia.org/wiki/Aragorn  \n",
       "9             https://en.wikipedia.org/wiki/Galadriel  \n",
       "10  https://en.wikipedia.org/wiki/Meriadoc_Brandybuck  \n",
       "11                                                NaN  "
      ]
     },
     "execution_count": 162,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.merge(urls_wrong_order,on='name',how='outer')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, there are \"inner\" joins, which include only those values that exist in both the \"left\" and \"right\" DataFrames:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.merge(urls_wrong_order,on='name',how='inner')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes it's nice to know how a particular row got added to the resulting DataFrame.  Using ```indicator=True```\n",
    "allows us to examine this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.merge(urls_wrong_order,how='outer',indicator=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inner join is the default.\n",
    "\n",
    "You'll note that we used the ```merge``` function from the DataFrame and passed in the other DataFrame as an argument.\n",
    "You can also call the ```merge``` function from pandas directly and pass it the two DataFrames you are merging:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![pivot 1](http://www.datasciencemadesimple.com/wp-content/uploads/2017/09/join-or-merge-in-python-pandas-1.png \"pivots\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>aggression</th>\n",
       "      <th>magic</th>\n",
       "      <th>name</th>\n",
       "      <th>race</th>\n",
       "      <th>stealth</th>\n",
       "      <th>url</th>\n",
       "      <th>_merge</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>Gandalf</td>\n",
       "      <td>Maia</td>\n",
       "      <td>8.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Gandalf</td>\n",
       "      <td>both</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Gimli</td>\n",
       "      <td>Dwarf</td>\n",
       "      <td>2.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Gimli_(Middle-ea...</td>\n",
       "      <td>both</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Frodo</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Frodo_Baggins</td>\n",
       "      <td>both</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Legolas</td>\n",
       "      <td>Elf</td>\n",
       "      <td>10.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Legolas</td>\n",
       "      <td>both</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Bilbo</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Bilbo_Baggins</td>\n",
       "      <td>both</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Sam</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>4.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Samwise_Gamgee</td>\n",
       "      <td>both</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Pippin</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Peregrin_Took</td>\n",
       "      <td>both</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Boromir</td>\n",
       "      <td>Man</td>\n",
       "      <td>3.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Boromir</td>\n",
       "      <td>both</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>7.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Aragorn</td>\n",
       "      <td>Man</td>\n",
       "      <td>9.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Aragorn</td>\n",
       "      <td>both</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>Galadriel</td>\n",
       "      <td>Elf</td>\n",
       "      <td>10.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Galadriel</td>\n",
       "      <td>both</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Meriadoc</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>6.0</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Meriadoc_Brandybuck</td>\n",
       "      <td>both</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Lily</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>both</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    aggression  magic       name    race  stealth  \\\n",
       "0          7.0   10.0    Gandalf    Maia      8.0   \n",
       "1         10.0    1.0      Gimli   Dwarf      2.0   \n",
       "2          2.0    4.0      Frodo  Hobbit      5.0   \n",
       "3          5.0    6.0    Legolas     Elf     10.0   \n",
       "4          1.0    4.0      Bilbo  Hobbit      5.0   \n",
       "5          6.0    2.0        Sam  Hobbit      4.0   \n",
       "6          3.0    0.0     Pippin  Hobbit      5.0   \n",
       "7          8.0    0.0    Boromir     Man      3.0   \n",
       "8          7.0    2.0    Aragorn     Man      9.0   \n",
       "9          2.0    9.0  Galadriel     Elf     10.0   \n",
       "10         4.0    0.0   Meriadoc  Hobbit      6.0   \n",
       "11         NaN    NaN       Lily  Hobbit      NaN   \n",
       "\n",
       "                                                  url _merge  \n",
       "0               https://en.wikipedia.org/wiki/Gandalf   both  \n",
       "1   https://en.wikipedia.org/wiki/Gimli_(Middle-ea...   both  \n",
       "2         https://en.wikipedia.org/wiki/Frodo_Baggins   both  \n",
       "3               https://en.wikipedia.org/wiki/Legolas   both  \n",
       "4         https://en.wikipedia.org/wiki/Bilbo_Baggins   both  \n",
       "5        https://en.wikipedia.org/wiki/Samwise_Gamgee   both  \n",
       "6         https://en.wikipedia.org/wiki/Peregrin_Took   both  \n",
       "7               https://en.wikipedia.org/wiki/Boromir   both  \n",
       "8               https://en.wikipedia.org/wiki/Aragorn   both  \n",
       "9             https://en.wikipedia.org/wiki/Galadriel   both  \n",
       "10  https://en.wikipedia.org/wiki/Meriadoc_Brandybuck   both  \n",
       "11                                                NaN   both  "
      ]
     },
     "execution_count": 158,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.merge(df,urls_wrong_order,how='outer',indicator=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color=\"magenta\">Q8: (2 points) Join the ```udf``` DataFrame (that contains aliases) to the ```df``` DataFrame using an appropriate merge"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>url</th>\n",
       "      <th>aliases</th>\n",
       "      <th>aggression</th>\n",
       "      <th>magic</th>\n",
       "      <th>race</th>\n",
       "      <th>stealth</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Gimli</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Gimli_(Middle-ea...</td>\n",
       "      <td>Elf-friend Lockbearer Lord of the Glittering C...</td>\n",
       "      <td>10.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Dwarf</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Frodo</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Frodo_Baggins</td>\n",
       "      <td>Mr. Underhill,Maura Labingi</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Legolas</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Legolas</td>\n",
       "      <td>Greenleaf, (Legolas translatedinto English)</td>\n",
       "      <td>5.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Elf</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Bilbo</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Bilbo_Baggins</td>\n",
       "      <td>Bilba Labingi</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Sam</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Samwise_Gamgee</td>\n",
       "      <td>Samwise Gardner, Sam, Samwise the Brave,Banazî...</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Pippin</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Peregrin_Took</td>\n",
       "      <td>Pippin, Pip,\"Ernil i Pheriannath\"Thain Peregri...</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Boromir</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Boromir</td>\n",
       "      <td>Captain of the White Tower,High Warden of the ...</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Galadriel</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Galadriel</td>\n",
       "      <td>AlatárielAltárielArtanisNerwen</td>\n",
       "      <td>2.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>Elf</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Meriadoc</td>\n",
       "      <td>https://en.wikipedia.org/wiki/Meriadoc_Brandybuck</td>\n",
       "      <td>Merry,Kalimac Brandagamba,Meriadoc the Magnifi...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Gandalf</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>Maia</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Aragorn</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Man</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Lily</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         name                                                url  \\\n",
       "0       Gimli  https://en.wikipedia.org/wiki/Gimli_(Middle-ea...   \n",
       "1       Frodo        https://en.wikipedia.org/wiki/Frodo_Baggins   \n",
       "2     Legolas              https://en.wikipedia.org/wiki/Legolas   \n",
       "3       Bilbo        https://en.wikipedia.org/wiki/Bilbo_Baggins   \n",
       "4         Sam       https://en.wikipedia.org/wiki/Samwise_Gamgee   \n",
       "5      Pippin        https://en.wikipedia.org/wiki/Peregrin_Took   \n",
       "6     Boromir              https://en.wikipedia.org/wiki/Boromir   \n",
       "7   Galadriel            https://en.wikipedia.org/wiki/Galadriel   \n",
       "8    Meriadoc  https://en.wikipedia.org/wiki/Meriadoc_Brandybuck   \n",
       "9     Gandalf                                                NaN   \n",
       "10    Aragorn                                                NaN   \n",
       "11       Lily                                                NaN   \n",
       "\n",
       "                                              aliases  aggression  magic  \\\n",
       "0   Elf-friend Lockbearer Lord of the Glittering C...        10.0    1.0   \n",
       "1                         Mr. Underhill,Maura Labingi         2.0    4.0   \n",
       "2         Greenleaf, (Legolas translatedinto English)         5.0    6.0   \n",
       "3                                       Bilba Labingi         1.0    4.0   \n",
       "4   Samwise Gardner, Sam, Samwise the Brave,Banazî...         6.0    2.0   \n",
       "5   Pippin, Pip,\"Ernil i Pheriannath\"Thain Peregri...         3.0    0.0   \n",
       "6   Captain of the White Tower,High Warden of the ...         8.0    0.0   \n",
       "7                      AlatárielAltárielArtanisNerwen         2.0    9.0   \n",
       "8   Merry,Kalimac Brandagamba,Meriadoc the Magnifi...         4.0    0.0   \n",
       "9                                                 NaN         7.0   10.0   \n",
       "10                                                NaN         7.0    2.0   \n",
       "11                                                NaN         NaN    NaN   \n",
       "\n",
       "      race  stealth  \n",
       "0    Dwarf      2.0  \n",
       "1   Hobbit      5.0  \n",
       "2      Elf     10.0  \n",
       "3   Hobbit      5.0  \n",
       "4   Hobbit      4.0  \n",
       "5   Hobbit      5.0  \n",
       "6      Man      3.0  \n",
       "7      Elf     10.0  \n",
       "8   Hobbit      6.0  \n",
       "9     Maia      8.0  \n",
       "10     Man      9.0  \n",
       "11  Hobbit      NaN  "
      ]
     },
     "execution_count": 173,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Insert your code here\n",
    "udf # len = 8, len(df) = 12\n",
    "pd.merge(udf, df, on = 'name', how = 'right') \n",
    "# in this case same as pd.merge(udf, df, on = 'name', how = 'outer', indicator=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>aggression</th>\n",
       "      <th>magic</th>\n",
       "      <th>name</th>\n",
       "      <th>race</th>\n",
       "      <th>stealth</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>Gandalf</td>\n",
       "      <td>Maia</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>10.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Gimli</td>\n",
       "      <td>Dwarf</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Frodo</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Legolas</td>\n",
       "      <td>Elf</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>Bilbo</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>6.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Sam</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Pippin</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Boromir</td>\n",
       "      <td>Man</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>7.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>Aragorn</td>\n",
       "      <td>Man</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>Galadriel</td>\n",
       "      <td>Elf</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Meriadoc</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Lily</td>\n",
       "      <td>Hobbit</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    aggression  magic       name    race  stealth\n",
       "0          7.0   10.0    Gandalf    Maia      8.0\n",
       "1         10.0    1.0      Gimli   Dwarf      2.0\n",
       "2          2.0    4.0      Frodo  Hobbit      5.0\n",
       "3          5.0    6.0    Legolas     Elf     10.0\n",
       "4          1.0    4.0      Bilbo  Hobbit      5.0\n",
       "5          6.0    2.0        Sam  Hobbit      4.0\n",
       "6          3.0    0.0     Pippin  Hobbit      5.0\n",
       "7          8.0    0.0    Boromir     Man      3.0\n",
       "8          7.0    2.0    Aragorn     Man      9.0\n",
       "9          2.0    9.0  Galadriel     Elf     10.0\n",
       "10         4.0    0.0   Meriadoc  Hobbit      6.0\n",
       "11         NaN    NaN       Lily  Hobbit      NaN"
      ]
     },
     "execution_count": 166,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Aggregation and Grouping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning Objectives\n",
    "* use StringIO to create a DataFrame\n",
    "* use the .describe() function\n",
    "* understand .groupby()\n",
    "* know how to use pivot and pivot_table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Some more basic pandas functionality\n",
    "One of the nice things about pandas is that it simplifies many common operations on datasets.  Let's load yet another LOTR dataset, \n",
    "this time using StringIO.  StringIO allows us to create a string that's then available as a file!\n",
    "\n",
    "Why would you want to do this?  Sometimes it's easier to just paste data right into a Jupyter notebook (or python script)\n",
    "than it is to create another CSV file.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "metadata": {},
   "outputs": [],
   "source": [
    "# cannot run in python 2\n",
    "# from io import StringIO\n",
    "\n",
    "# LOTRDATA=StringIO(\"\"\"name,race,gender,magic,aggression,stealth\n",
    "# Gandalf,Maia,Male,10.0,7.0,8.0\n",
    "# Gimli,Dwarf,None,1.0,10.0,2.0\n",
    "# Frodo,Hobbit,Male,4.0,2.0,5.0\n",
    "# Legolas,Elf,Male,6.0,5.0,10.0\n",
    "# Bilbo,Hobbit,Male,4.0,1.0,5.0\n",
    "# Sam,Hobbit,Male,2.0,6.0,4.0\n",
    "# Pippin,Hobbit,Male,0.0,3.0,5.0\n",
    "# Boromir,Human,Male,0.0,8.0,3.0\n",
    "# Aragorn,Human,Male,2.0,7.0,9.0\n",
    "# Galadriel,Elf,Female,9.0,2.0,10.0\n",
    "# Lily,Hobbit,Female,,,\n",
    "# Meriadoc,Hobbit,Male,,4.0,6.0\n",
    "# Melian,Maia,Female,10.0,5.0,9.0\n",
    "# Idril,Elf,Female,8.0,,8.0\n",
    "# \"\"\")\n",
    "\n",
    "# lotr = pd.read_csv(LOTRDATA, index_col=None)\n",
    "# lotr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a DataFrame, we can get some basic statistics about it using the ```describe()``` function.\n",
    "Note that ```describe()``` only returns values for numeric columns.  Note too that it returns another DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {},
   "outputs": [],
   "source": [
    "# lotr.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {},
   "outputs": [],
   "source": [
    "# type(lotr.describe())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Pivots/Stack/Unstack Intro\n",
    "The following cells are based on: \n",
    "http://nikgrozev.com/2015/07/01/reshaping-in-pandas-pivot-pivot-table-stack-and-unstack-explained-with-pictures/ , which is\n",
    "one of the best guides to pivots, pivot tables, stacking and unstacking that I've encountered."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For demonstration purposes, let's create the same DataFrame that Nikolay Grozev uses in his tutorial:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Item</th>\n",
       "      <th>CType</th>\n",
       "      <th>USD</th>\n",
       "      <th>EU</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Item0</td>\n",
       "      <td>Gold</td>\n",
       "      <td>1$</td>\n",
       "      <td>1€</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Item0</td>\n",
       "      <td>Bronze</td>\n",
       "      <td>2$</td>\n",
       "      <td>2€</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Item1</td>\n",
       "      <td>Gold</td>\n",
       "      <td>3$</td>\n",
       "      <td>3€</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Item1</td>\n",
       "      <td>Silver</td>\n",
       "      <td>4$</td>\n",
       "      <td>4€</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Item   CType USD  EU\n",
       "0  Item0    Gold  1$  1€\n",
       "1  Item0  Bronze  2$  2€\n",
       "2  Item1    Gold  3$  3€\n",
       "3  Item1  Silver  4$  4€"
      ]
     },
     "execution_count": 194,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import OrderedDict\n",
    "from pandas import DataFrame\n",
    "\n",
    "table = OrderedDict((\n",
    "    (\"Item\", ['Item0', 'Item0', 'Item1', 'Item1']),\n",
    "    ('CType',['Gold', 'Bronze', 'Gold', 'Silver']),\n",
    "    ('USD',  ['1$', '2$', '3$', '4$']),\n",
    "    ('EU',   ['1€', '2€', '3€', '4€'])\n",
    "))\n",
    "metal = DataFrame(table)\n",
    "metal\n",
    "\n",
    "# create the table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>CType</th>\n",
       "      <th>Bronze</th>\n",
       "      <th>Gold</th>\n",
       "      <th>Silver</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Item</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Item0</th>\n",
       "      <td>2$</td>\n",
       "      <td>1$</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Item1</th>\n",
       "      <td>None</td>\n",
       "      <td>3$</td>\n",
       "      <td>4$</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "CType Bronze Gold Silver\n",
       "Item                    \n",
       "Item0     2$   1$   None\n",
       "Item1   None   3$     4$"
      ]
     },
     "execution_count": 196,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# make a table of items (rows) and costs (USD) \n",
    "# for each in gold and bronze\n",
    "metal.pivot(index='Item',columns='CType',values='USD') # columns should be unque values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See the image below... we're telling Pandas to take the table above, create a row for every item. \n",
    "This is done by setting index to Item (the column in the original table that contains item names)\n",
    "We then are telling pandas we want to create a column for every unique element in the\n",
    "original CType column.  And finally, we want the value in the cells to be the value from the USD\n",
    "column in the original table."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![pivot 1](http://nikgrozev.com/images/blog/Reshaping%20in%20Pandas%20-%20Pivot%20Pivot-Table%20Stack%20and%20Unstack%20explained%20with%20Pictures/pivoting_simple1.png \"pivots\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color=\"magenta\">Q9: (2 points) Using pivot and the dataframe below make a table with each person as the index, their purchased items as columns, and the price they paid as the values.</font>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Item Purchased</th>\n",
       "      <th>Apple</th>\n",
       "      <th>Ferrero Rocher</th>\n",
       "      <th>Kit-Kat</th>\n",
       "      <th>M&amp;Ms</th>\n",
       "      <th>Mango</th>\n",
       "      <th>Pear</th>\n",
       "      <th>Pineapple</th>\n",
       "      <th>Snickers</th>\n",
       "      <th>Twix</th>\n",
       "      <th>Watermelon</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Person</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Julie</th>\n",
       "      <td>$1</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>$2</td>\n",
       "      <td>$2</td>\n",
       "      <td>$2</td>\n",
       "      <td>$2</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Scott</th>\n",
       "      <td>None</td>\n",
       "      <td>$1.5</td>\n",
       "      <td>$1</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>$1.5</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Shiyan</th>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>$1.5</td>\n",
       "      <td>$1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Item Purchased Apple Ferrero Rocher Kit-Kat  M&Ms Mango  Pear Pineapple  \\\n",
       "Person                                                                    \n",
       "Julie             $1           None    None    $2    $2    $2        $2   \n",
       "Scott           None           $1.5      $1  None  None  None      None   \n",
       "Shiyan          None           None    None  None  None  None      None   \n",
       "\n",
       "Item Purchased Snickers  Twix Watermelon  \n",
       "Person                                    \n",
       "Julie              None  None       None  \n",
       "Scott              $1.5  None       None  \n",
       "Shiyan             None  $1.5         $1  "
      ]
     },
     "execution_count": 201,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "table1 = OrderedDict((\n",
    "    (\"Person\", ['Scott', 'Julie', 'Shiyan', 'Julie','Scott','Julie', 'Shiyan', 'Julie','Scott', 'Julie']),\n",
    "    ('Item Purchased',['Kit-Kat','Mango', 'Twix', 'M&Ms', 'Ferrero Rocher','Apple', 'Watermelon', 'Pineapple', 'Snickers','Pear']),\n",
    "    ('Price Paid',  ['$1','$2', '$1.5', '$2', '$1.5','$1','$1','$2', '$1.5', '$2']),\n",
    "    ))\n",
    "fav = DataFrame(table1)\n",
    "fav\n",
    "fav.pivot(index='Person',columns='Item Purchased',values='Price Paid') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Item</th>\n",
       "      <th>CType</th>\n",
       "      <th>USD</th>\n",
       "      <th>EU</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Item0</td>\n",
       "      <td>Gold</td>\n",
       "      <td>1$</td>\n",
       "      <td>1€</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Item0</td>\n",
       "      <td>Bronze</td>\n",
       "      <td>2$</td>\n",
       "      <td>2€</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Item1</td>\n",
       "      <td>Gold</td>\n",
       "      <td>3$</td>\n",
       "      <td>3€</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Item1</td>\n",
       "      <td>Silver</td>\n",
       "      <td>4$</td>\n",
       "      <td>4€</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Item   CType USD  EU\n",
       "0  Item0    Gold  1$  1€\n",
       "1  Item0  Bronze  2$  2€\n",
       "2  Item1    Gold  3$  3€\n",
       "3  Item1  Silver  4$  4€"
      ]
     },
     "execution_count": 206,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Insert Code Here\n",
    "metal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# end of exercise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "p = metal.pivot(index='Item',columns='CType',values='USD')  # same as above, I'm just renaming the pivot as \"p\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Access the USD cost of Item0 for Gold customers...\n",
    "\n",
    "First we find the row for Item0/Gold and then we select the USD column and pull out the value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "result = metal[((metal.Item == 'Item0') & (metal.CType == 'Gold'))].USD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.series.Series"
      ]
     },
     "execution_count": 204,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1$\n",
       "Name: USD, dtype: object"
      ]
     },
     "execution_count": 205,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do the same thing on pivoted table. Here we pull out the row for Item0, grab the Gold column and print the value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 207,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['1$'], dtype=object)"
      ]
     },
     "execution_count": 207,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p[p.index == 'Item0'].Gold.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Now pivot by multiple columns, I want USD and EU prices. It returns a hierarchical index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"3\" halign=\"left\">USD</th>\n",
       "      <th colspan=\"3\" halign=\"left\">EU</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CType</th>\n",
       "      <th>Bronze</th>\n",
       "      <th>Gold</th>\n",
       "      <th>Silver</th>\n",
       "      <th>Bronze</th>\n",
       "      <th>Gold</th>\n",
       "      <th>Silver</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Item</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Item0</th>\n",
       "      <td>2$</td>\n",
       "      <td>1$</td>\n",
       "      <td>None</td>\n",
       "      <td>2€</td>\n",
       "      <td>1€</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Item1</th>\n",
       "      <td>None</td>\n",
       "      <td>3$</td>\n",
       "      <td>4$</td>\n",
       "      <td>None</td>\n",
       "      <td>3€</td>\n",
       "      <td>4€</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         USD                 EU            \n",
       "CType Bronze Gold Silver Bronze Gold Silver\n",
       "Item                                       \n",
       "Item0     2$   1$   None     2€   1€   None\n",
       "Item1   None   3$     4$   None   3€     4€"
      ]
     },
     "execution_count": 208,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metal.pivot(index='Item',columns='CType')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Access the USD cost of Item0 for Gold customers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['1$'], dtype=object)"
      ]
     },
     "execution_count": 209,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = metal.pivot(index='Item',columns='CType')\n",
    "p.USD[p.USD.index == 'Item0'].Gold.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['1$'], dtype=object)"
      ]
     },
     "execution_count": 211,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p.USD[p.USD.index == 'Item0'].Gold.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# What happens if there is a collision? \n",
    "See the problem?  There are two Item0/Golds:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![pivot 2](http://nikgrozev.com/images/blog/Reshaping%20in%20Pandas%20-%20Pivot%20Pivot-Table%20Stack%20and%20Unstack%20explained%20with%20Pictures/pivoting_simple_error.png \"pivots\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's set up another DataFrame to demonstrate this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "table = OrderedDict((\n",
    "    (\"Item\", ['Item0', 'Item0', 'Item0', 'Item1']),\n",
    "    ('CType',['Gold', 'Bronze', 'Gold', 'Silver']),\n",
    "    ('USD',  ['1$', '2$', '3$', '4$']),\n",
    "    ('EU',   ['1€', '2€', '3€', '4€'])\n",
    "))\n",
    "metal = DataFrame(table)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Item</th>\n",
       "      <th>CType</th>\n",
       "      <th>USD</th>\n",
       "      <th>EU</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Item0</td>\n",
       "      <td>Gold</td>\n",
       "      <td>1$</td>\n",
       "      <td>1€</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Item0</td>\n",
       "      <td>Bronze</td>\n",
       "      <td>2$</td>\n",
       "      <td>2€</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Item0</td>\n",
       "      <td>Gold</td>\n",
       "      <td>3$</td>\n",
       "      <td>3€</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Item1</td>\n",
       "      <td>Silver</td>\n",
       "      <td>4$</td>\n",
       "      <td>4€</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Item   CType USD  EU\n",
       "0  Item0    Gold  1$  1€\n",
       "1  Item0  Bronze  2$  2€\n",
       "2  Item0    Gold  3$  3€\n",
       "3  Item1  Silver  4$  4€"
      ]
     },
     "execution_count": 213,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metal.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next cell will generate an error:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "p = metal.pivot(index='Item', columns='CType', values='USD')\n",
    "# will return an error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## pivot_tables is your friend\n",
    "![pivot 1](http://nikgrozev.com/images/blog/Reshaping%20in%20Pandas%20-%20Pivot%20Pivot-Table%20Stack%20and%20Unstack%20explained%20with%20Pictures/pivoting_table_simple1.png \"pivots\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create yet another DataFrame to play with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Item</th>\n",
       "      <th>CType</th>\n",
       "      <th>USD</th>\n",
       "      <th>EU</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Item0</td>\n",
       "      <td>Gold</td>\n",
       "      <td>1</td>\n",
       "      <td>1.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Item0</td>\n",
       "      <td>Bronze</td>\n",
       "      <td>2</td>\n",
       "      <td>2.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Item0</td>\n",
       "      <td>Gold</td>\n",
       "      <td>3</td>\n",
       "      <td>3.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Item1</td>\n",
       "      <td>Silver</td>\n",
       "      <td>4</td>\n",
       "      <td>4.4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Item   CType  USD   EU\n",
       "0  Item0    Gold    1  1.1\n",
       "1  Item0  Bronze    2  2.2\n",
       "2  Item0    Gold    3  3.3\n",
       "3  Item1  Silver    4  4.4"
      ]
     },
     "execution_count": 214,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "table = OrderedDict((\n",
    "    (\"Item\", ['Item0', 'Item0', 'Item0', 'Item1']),\n",
    "    ('CType',['Gold', 'Bronze', 'Gold', 'Silver']),\n",
    "    ('USD',  [1, 2, 3, 4]),\n",
    "    ('EU',   [1.1, 2.2, 3.3, 4.4])\n",
    "))\n",
    "metal = DataFrame(table)\n",
    "metal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "pivot_table is  a bit different than pivot... It's the same with the first part\n",
    "index, columns, values remain the same as before BUT we added a rule (aggfunc)\n",
    "that says: whey you hit a conflict, the way to resolve it is X (in this case\n",
    "x is the \"mean\"... so find the mean of the two numbers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 215,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>CType</th>\n",
       "      <th>Bronze</th>\n",
       "      <th>Gold</th>\n",
       "      <th>Silver</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Item</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Item0</th>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Item1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "CType  Bronze  Gold  Silver\n",
       "Item                       \n",
       "Item0     2.0   2.0     NaN\n",
       "Item1     NaN   NaN     4.0"
      ]
     },
     "execution_count": 215,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = metal.pivot_table(index='Item',columns='CType',values='USD',aggfunc=np.mean) # aggfunc: aggregation function\n",
    "p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You could have also resolved the conflict in other ways.  Here we tell it to take the \"min\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 226,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>CType</th>\n",
       "      <th>Bronze</th>\n",
       "      <th>Gold</th>\n",
       "      <th>Silver</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Item</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Item0</th>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Item1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "CType  Bronze  Gold  Silver\n",
       "Item                       \n",
       "Item0     2.0   1.0     NaN\n",
       "Item1     NaN   NaN     4.0"
      ]
     },
     "execution_count": 226,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = metal.pivot_table(index='Item',columns='CType',values='USD',aggfunc=np.min) # np.min\n",
    "p "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color=\"magenta\">Q10: (2 points) Using pivot_table and the dataframe below make a table with each person as the index, their purchased items as columns, and the sum of the price they paid as the values (sometimes people bought more than one of the same item).</font>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Item Purchased</th>\n",
       "      <th>Apple</th>\n",
       "      <th>Ferrero Rocher</th>\n",
       "      <th>Kit-Kat</th>\n",
       "      <th>M&amp;Ms</th>\n",
       "      <th>Mango</th>\n",
       "      <th>Pear</th>\n",
       "      <th>Pineapple</th>\n",
       "      <th>Snickers</th>\n",
       "      <th>Twix</th>\n",
       "      <th>Watermelon</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Person</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Julie</th>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Scott</th>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Shiyan</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Item Purchased  Apple  Ferrero Rocher  Kit-Kat  M&Ms  Mango  Pear  Pineapple  \\\n",
       "Person                                                                         \n",
       "Julie             2.0             NaN      NaN   4.0    4.0   2.0        2.0   \n",
       "Scott             NaN             3.0      2.0   NaN    NaN   NaN        NaN   \n",
       "Shiyan            NaN             NaN      NaN   NaN    NaN   NaN        NaN   \n",
       "\n",
       "Item Purchased  Snickers  Twix  Watermelon  \n",
       "Person                                      \n",
       "Julie                NaN   NaN         NaN  \n",
       "Scott                1.5   NaN         NaN  \n",
       "Shiyan               NaN   3.0         1.0  "
      ]
     },
     "execution_count": 230,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "table1 = OrderedDict((\n",
    "    (\"Person\", ['Scott', 'Julie', 'Shiyan', 'Julie',\n",
    "                'Scott','Julie', 'Shiyan', 'Julie','Scott', \n",
    "                'Julie','Scott', 'Julie', 'Shiyan', 'Julie',\n",
    "                'Scott','Julie',]),\n",
    "    ('Item Purchased',['Kit-Kat','Mango', 'Twix', 'M&Ms', 'Ferrero Rocher',\n",
    "                       'Apple', 'Watermelon', 'Pineapple', 'Snickers','Pear','Kit-Kat','Mango', 'Twix', 'M&Ms', 'Ferrero Rocher',\n",
    "                       'Apple']),\n",
    "    ('Price Paid (USD)',  [1,2, 1.5, 2, 1.5,1,1,2, 1.5, 2,\n",
    "                    1,2, 1.5, 2, 1.5,1]),\n",
    "    ))\n",
    "fav = DataFrame(table1)\n",
    "fav\n",
    "fav.pivot_table(index='Person',columns='Item Purchased',values='Price Paid (USD)',aggfunc=np.sum)  \n",
    "# only pivot_table has aggfunc, pivot dodn't have aggfunc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Insert Code Here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Pivots are a specific form of stack/unstack (remember those?)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![pivot 1](http://nikgrozev.com/images/blog/Reshaping%20in%20Pandas%20-%20Pivot%20Pivot-Table%20Stack%20and%20Unstack%20explained%20with%20Pictures/stack-unstack1.png \"pivots\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## A worked example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# to start let's make a fake dataset: sales of fruit across US states.\n",
    "# Don't worry about the details here, but basically we'll pretend\n",
    "# this string is a CSV file and use the standard loading ops\n",
    "from io import StringIO\n",
    "\n",
    "TESTDATA=StringIO(\"\"\"State,Retailer,Fruit,Sales\n",
    "MI,Walmart,Apple,100\n",
    "MI,Wholefoods,Apple,150\n",
    "MI,Kroger,Orange,180\n",
    "CA,Walmart,Apple,220\n",
    "CA,Wholefoods,Apple,180\n",
    "CA,Safeway,Apple,220\n",
    "CA,Safeway,Orange,110\n",
    "NY,Walmart,Apple,90\n",
    "NY,Walmart,Orange,80\n",
    "NY,Wholefoods,Orange,120\n",
    "\"\"\")\n",
    "\n",
    "fruit = pd.read_csv(TESTDATA, index_col=None)\n",
    "fruit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## (a) What is the total sales for each state?\n",
    "This requires us to group by state, and aggregate sales by taking the sum.\n",
    "\n",
    "The easiest way of doing this if to use `groupby`\n",
    "\n",
    "If you execute groupby on the dataframe what you'll get back is an object called DataFrameGroupBy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "fruit.groupby('State')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On its own it's a bit useless... it just keeps track of which rows should go into each \"pile\" (where pile here means a unique group for each state)\n",
    "\n",
    "If we ask this object to describe itself, you can see what is inside notice that it threw away all the other columns because they were not numerical.  Only \"Sales\" which is a number, was kept"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "fruit.groupby('State').describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Now, if we had another numerical column, let's call it \"Sales2,\" that column would also be kept.  Let's make a fruit2 DataFrame so you can see that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "from io import StringIO\n",
    "\n",
    "TESTDATA=StringIO(\"\"\"State,Retailer,Fruit,Sales,Sales2\n",
    "MI,Walmart,Apple,100,10\n",
    "MI,Wholefoods,Apple,150,20\n",
    "MI,Kroger,Orange,180,30\n",
    "CA,Walmart,Apple,220,20\n",
    "CA,Wholefoods,Apple,180,40\n",
    "CA,Safeway,Apple,220,30\n",
    "CA,Safeway,Orange,110,20\n",
    "NY,Walmart,Apple,90,40\n",
    "NY,Walmart,Orange,80,20\n",
    "NY,Wholefoods,Orange,120,60\n",
    "\"\"\")\n",
    "\n",
    "fruit2 = pd.read_csv(TESTDATA, index_col=None)\n",
    "fruit2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "fruit2.groupby(\"State\").describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "To actually make use of the groupby, we need to tell pandas what to use to measure what's in each group. In other words, I've created a pile for California, a pile for Michigan, and a pile for New York.  I want a number to what's *inside* each pile.  I could ask for the \"size\" (so how many rows are in each pile), or I could calculate some mathematical function.  For example, if I wanted to know the total sales, I would call \"sum.\"  What happens is pandas goes through every pile, looks at every \"row\" inside that pile and, for all numerical properties, calculated something.  In this case it's sum... it adds up everything.  So in our original table we had three items for Michigan (Walmart, Kroger, Wholefoods).  This is our Michigan pile.  We then look at numerical properties for Walmart, Kroger, and Wholefoods. In this case Sales.  Because we are using sum() that means add the sales of each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# What are the total sales for each state?\n",
    "fruit.groupby('State').sum()  # instead of size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What just happend? A couple of things:\n",
    "- `groupby()` got first executed on `df`, returning an `DataFrameGroupBy` object. This object itself is useless unless coupled with an aggregation function, such as `sum()`, `mean()`, `max()`, `apply()`. We will talk about `apply()` more in the next week.\n",
    "- Then, `sum()` got executed on the `DataFrameGroupBy` object, generating the `DataFrame` object you see above. Notice how the table looks different than the original DataFrame `df`? Here are the differences:\n",
    "  - The `State` column now becomes the index of the DataFrame. The string \"State\" is the name of the index. Notice how the index name is displayed on a lower level than column names.\n",
    "  - Since we performed a `groupby` operation by `State`, so only the unique values of `State` are kept as index.\n",
    "  - Among the other columns, Retailer, Fruit, and Sales, only Sales is kept in the result table. This is because the aggregation function `sum()` only knows how to aggregate numerical values. And only Sales is a numerical column. The other columns are hence dropped."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color=\"magenta\">Q11: (2 points) Using groupby, which Retailer had the highest total Sales and how much was that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Insert code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Insert written Answer here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## (b) What is the total sales for each state for each fruit?\n",
    "This requires us to perform `groupby` on two columns. So, we provide a list of column names to the `groupby` function.\n",
    "\n",
    "Don't forget that an aggregation function needs to follow the `groupby` function in order to generate results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# What is the total sales for each state for each fruit?\n",
    "fruit.groupby(['State','Fruit']).sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "How is this DataFrame different from the previous one?\n",
    "\n",
    "The biggest different is that this DataFrame has what is called a `MultiIndex` (or hierarchical index), as opposed to a simple index. In this table, the left two \"columns\" are not columns but actually part of the `MultiIndex`, and the `Sales` is the single real \"column\" in the DataFrame. (Running out of terminologies here...)\n",
    "\n",
    "The hierarchical index can be organized in an alternative way if we swapped the order of State and Fruit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "fruit.groupby(['Fruit','State']).sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## (c) Which state has the maximum total sales?\n",
    "This question is not asking about the maximum value, but rather which state holds that maximum. There are multiple ways to do it. A principled way is to use `idxmax`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Which state has the maximum total sales?\n",
    "fruitSalesByState = fruit.groupby('State').sum()\n",
    "print(fruitSalesByState)\n",
    "max_state = fruitSalesByState['Sales'].idxmax()\n",
    "print(\"The state with the maximum sales is: \",max_state)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "\n",
    "What if I want to display the maximum value alongside the state? Well, we can use that returned label to _select_ the corresponding row from the original DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# the deets\n",
    "fruitSalesByState.loc['CA']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A less efficient but more intuitive way of doing the same thing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fruitSalesByState.sort_values('Sales',ascending=False).iloc[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Which state has the maximum total sales for apples?</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Which state has the maximum total sales for apples?\n",
    "# give me apple sellers\n",
    "apples = fruit[fruit.Fruit == 'Apple']\n",
    "apples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# aggr. by state\n",
    "applesByState = apples.groupby('State').sum()\n",
    "applesByState"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "applesByState.Sales"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "applesByState.Sales.idxmax()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "applesByState.loc[applesByState.Sales.idxmax()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above command, `.loc[]` looks up the index label and returns that row.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Applying what we learned to the LOTR data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What are the average values for magic, aggression, and stealth for each race?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from io import StringIO\n",
    "\n",
    "LOTRDATA=StringIO(\"\"\"name,race,gender,magic,aggression,stealth\n",
    "Gandalf,Maia,Male,10.0,7.0,8.0\n",
    "Gimli,Dwarf,None,1.0,10.0,2.0\n",
    "Frodo,Hobbit,Male,4.0,2.0,5.0\n",
    "Legolas,Elf,Male,6.0,5.0,10.0\n",
    "Bilbo,Hobbit,Male,4.0,1.0,5.0\n",
    "Sam,Hobbit,Male,2.0,6.0,4.0\n",
    "Pippin,Hobbit,Male,0.0,3.0,5.0\n",
    "Boromir,Human,Male,0.0,8.0,3.0\n",
    "Aragorn,Human,Male,2.0,7.0,9.0\n",
    "Galadriel,Elf,Female,9.0,2.0,10.0\n",
    "Lily,Hobbit,Female,,,\n",
    "Meriadoc,Hobbit,Male,,4.0,6.0\n",
    "Melian,Maia,Female,10.0,5.0,9.0\n",
    "Idril,Elf,Female,8.0,,8.0\n",
    "\"\"\")\n",
    "\n",
    "lotr = pd.read_csv(LOTRDATA, index_col=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lotr.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <font color=\"magenta\">Q12: (2 points) Create a pivot table showing the maximum values of magic using gender for columns and race for rows.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Insert Code Here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lotr.groupby('race').mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can make things a bit more complex by creating a dictionary of aggregations.  \n",
    "The keys of the dictionary refer to the originating column we want to transform;\n",
    "the values of the dictionary contain lists of the functions we want to apply to those columns.\n",
    "\n",
    "Note that we can use built-in functions (see Table 10-1 in py4da) or define our own, possibly with a lambda:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aggregations = {\n",
    "    'magic': [min, max, sum, len],\n",
    "    'aggression': [min, max, sum, len],\n",
    "    'stealth': [min, max, lambda x: max(x) - min(x)]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read more about DataFrame.agg(func, axis=0, *args, **kwargs) here: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.agg.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grouped = lotr.groupby('race').agg(aggregations)\n",
    "grouped"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grouped.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grouped.columns = [\"_\".join(x) for x in grouped.columns.ravel()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grouped.columns.ravel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grouped"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Which race has the lowest average magic value?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lotr.groupby('race')['magic'].mean().idxmin()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <font color=\"magenta\">END OF NOTEBOOK</font>\n",
    "## Remember to submit this file in HTML and IPYNB formats via Canvas."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}