01-The-Neighborhood-fa2019.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the code cell below with the ▶| button above to set up this notebook, or type `SHIFT-ENTER`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datascience import *\n",
    "import pandas as pd\n",
    "import geojson\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import folium\n",
    "from IPython.display import HTML, display, IFrame\n",
    "from folium import plugins\n",
    "%matplotlib inline\n",
    "from scipy import stats\n",
    "import ipywidgets as widgets\n",
    "from soc_module import *\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "from datetime import datetime\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sociology 130 Module: \"The Neighborhood Project\"\n",
    "\n",
    "Welcome to the data science part of your project! You have gathered data and entered it [here](https://goo.gl/forms/eY1mephilS6VqAT83) from your assigned census tracts.  Now it's time to explore our class data and quantify our observations using Python, a popular programming language used in data science. \n",
    "\n",
    "You won't need any prior programming knowledge to do this! The purpose of this module is not to teach you programming, but rather to show you the power of these tools and give you the intuition for how they work. It also allows us to quickly produce summarizations of our data!\n",
    "\n",
    "## Table of Contents\n",
    "\n",
    "0. [Python and Jupyter Notebooks](#jupyter)\n",
    "1. [Class Data](#yourdata)\n",
    "2. [Our Metrics](#ourmetrics)\n",
    "3. [Census Data](#census)\n",
    "4. [Correlation](#correlation)\n",
    "5. [Regression](#regression)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Completing the Notebooks\n",
    "\n",
    "\n",
    "<div class=\"alert alert-info\"> \n",
    "\n",
    "**QUESTION** cells are in blue and ask you to answers questions or fill in code cells. To receive full credit for your assignment, you must complete all **QUESTION** cells.\n",
    "\n",
    "\n",
    "</div>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 0: Introduction to Python and Jupyter Notebooks: <a id='jupyter'></a>\n",
    "\n",
    "## 1. Cells and Code\n",
    "In a notebook, each rectangle containing text or code is called a *cell*.\n",
    "\n",
    "Cells (like this one) can be edited by double-clicking on them. This cell is a text cell, written in a simple format called [Markdown](http://daringfireball.net/projects/markdown/syntax) to add formatting and section headings.  You don't need to worry about Markdown today, but it's a pretty fun+easy tool to learn.\n",
    "\n",
    "After you edit a cell, click the \"run cell\" button at the top that looks like ▶| to confirm any changes. (Try not to delete the instructions.) You can also press `SHIFT-ENTER` to run any cell or progress from one cell to the next.\n",
    "\n",
    "Other cells contain code in the Python programming language.  Running a code cell will execute all of the code it contains.\n",
    "\n",
    "Try running this cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Hello, World!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you need to run an entire Jupyter notebook, click on the \"Cell\" option at the top and then select \"Run All\". This should execute each cell of the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# <-- Hash symbol marks a \"comment\" in Python that would not be executed or printed out. \n",
    "# Comments are often used to explain the actual code. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Technical Issues\n",
    "If you run a cell and see a `NameError` message below the cell, you probably forgot to run one of the cells above it. Make sure you run all the cells in order each time you open the notebook. You can do this easily by going to the \"Cell\" menu at the top and clicking \"Run all\".\n",
    "\n",
    "If at the top right side of the notebook you get a flag \"Cannot Connect to Kernel\", this may have happened due to poor internet connectivity. Try clicking \"Kernel\" at the top left and selecting either \"Restart\" or \"Restart & Run All\".\n",
    "\n",
    "Still having issues? **Come to Data Peer Consulting Office Hours at 3rd Floor Moffitt!** Find a Peer Consultant with the expertise you need and get your questions answered: [Office Hours Schedule is linked here]( https://data.berkeley.edu/education/peer-consulting)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Save and Access the Notebook\n",
    "To save your progress in your Jupyter notebook, click on the floppy disk icon at the top left corner or go to the File menu and select \"Save and Checkpoint\". If you want to come back to this notebook later, use the link provided by your professor or go to https://datahub.berkeley.edu and sign in with your bCourses account."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Basic Python Functionality\n",
    "\n",
    "[Optional] Click on the [link](http://datahub.berkeley.edu/user-redirect/interact?account=ds-modules&repo=SOC-130AC&branch=master&path=Part-0-Intro-to-Python.ipynb) which will take you to another Jupyter notebook where you can further explore Python. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1: Class Data<a id='yourdata'></a>\n",
    "\n",
    "We can read in the data you submitted through the form by asking Google for the form information and turning it into a table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# The following two lines read two spreadsheets.\n",
    "image_data = Table.read_table(\"data/image_data.csv\")\n",
    "class_data = Table.read_table(\"data/class_data.csv\")\n",
    "\n",
    "# The following block cleans entries in the \"Census Tract\" column.\n",
    "image_data[\"Census Tract\"] = [str(i) for i in image_data[\"Census Tract\"]]\n",
    "class_data[\"Census Tract\"] = [str(i) for i in class_data[\"Census Tract\"]]\n",
    "sub = lambda s: re.sub(r\"\\.[0-9]*\", \"\", s) # Defines a function that removes decimals.\n",
    "\n",
    "# Applies the function defined above to the \"Census Tract\" column in both tables.\n",
    "image_data[\"Census Tract\"] = image_data.apply(sub, \"Census Tract\")\n",
    "class_data[\"Census Tract\"] = class_data.apply(sub, \"Census Tract\")\n",
    "\n",
    "image_data.show(5)\n",
    "class_data.show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's a lot of columns! Now that our data is inside the `class_data` variable, we can ask that varible for some information. We can get a list of the column names with the `.labels` attribute of the table:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class_data.labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get some summary statistics and do some plotting.\n",
    "\n",
    "How many of you reported on which census tracts?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class_data.group(\"Census Tract\").show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "We can use the `.plot.barh()` method to this to visualize the counts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class_data.group(\"Census Tract\").barh(\"Census Tract\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the key principles of coding is to recycle the code we have written and reduce repetition.\n",
    "\n",
    "We can write a short function, `bar_chart_column`, to plot the counts for any of our columns in the table. All we have to do is select the column label in the dropdown."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This function takes one argument: COL.\n",
    "# It then draws a bar chart for this column.\n",
    "def bar_chart_column(col):\n",
    "    bar_chart_data.group(col).barh(col)\n",
    "\n",
    "bar_chart_data = class_data.drop(\"What kinds of establishments are there on the block face? Select all that apply.\")\n",
    "\n",
    "# Interative widget. Feel free to choose any column\n",
    "# in the dropdown menu.\n",
    "dropdown = widgets.Dropdown(options=bar_chart_data.labels[1:], description=\"Column\")\n",
    "display(widgets.interactive(bar_chart_column, col=dropdown))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then ask for these columns and plot their means too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(class_data\n",
    "    .to_df()\n",
    "    .iloc[:,2:] # Keeps all the rows, but keeps only columns that have indices >= 2\n",
    "    .mean()     # Takes the mean of each column\n",
    "    .plot       # Draws a bar chart.\n",
    "    .barh());"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "One of the questions had checkbox answers that listed all of the establishments that were observed by each student in their assigned census tract. Let's create a seperate column for each of the possible options. A value of `1` in the column indicates that the estalishment was observed. A value of `0` indicates that the establishment was not observed.\n",
    "\n",
    "This is called **one-hot encoding**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extracts responses about establishments.\n",
    "raw_establishment_responses = class_data.to_df().iloc[:, 12]\n",
    "\n",
    "# For each response, split it into a list of options selected.\n",
    "split_establishment_responses = [response.split(\", \") for response in raw_establishment_responses\n",
    "                                 if not pd.isnull(response)]\n",
    "\n",
    "# The variable above is a list of lists.\n",
    "# We want to flatten the list.\n",
    "establishments = pd.Series([item for sublist in split_establishment_responses for item in sublist])\n",
    "\n",
    "\n",
    "# Now let's do one-hot encoding!\n",
    "ests_table = Table.empty().with_column(\"Types of Establishments\", class_data[\"Types of Establishments\"])\n",
    "\n",
    "for establishment in establishments.unique():\n",
    "    establishment_data = []\n",
    "\n",
    "    for row in class_data.rows:\n",
    "        ests = row.item('Types of Establishments')\n",
    "       \n",
    "        if not pd.isnull(ests):\n",
    "            row_establishments = ests.split(', ')\n",
    "        \n",
    "        if establishment in row_establishments:\n",
    "            establishment_data.append(1)\n",
    "        else: \n",
    "            establishment_data.append(0)\n",
    "    \n",
    "    ests_table[establishment] = establishment_data\n",
    "    \n",
    "ests_table.show(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find the sum of each of the establishments.\n",
    "col_sums = []\n",
    "for col in ests_table.drop(0).labels:\n",
    "    col_sums.append(sum(ests_table[col]))\n",
    "\n",
    "# Put them into a series : index ~ type of establishment, value ~ count\n",
    "establishment_counts = pd.Series(col_sums, index = ests_table.drop(0).labels)\n",
    "establishment_counts = establishment_counts.drop(\"N/A\")\n",
    "\n",
    "establishment_counts.plot.barh()\n",
    "plt.title(ests_table.labels[0])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "---\n",
    "\n",
    "## Mapping\n",
    "\n",
    "We can also visualize how your responses mapped out over the census tracts. We'll use a library called `folium` to map your observations onto a map of the census tracts, and include popups with your comments and photos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "alameda = geojson.load(open(\"data/alameda-2010.geojson\"))\n",
    "myMap = folium.Map(location=(37.8044, -122.2711), zoom_start=11.4)\n",
    "\n",
    "map_data(myMap, alameda, image_data).save(\"maps/map1.html\") # remove null row\n",
    "IFrame('maps/map1.html', width=700, height=400)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Click around census tracts near yours to see if the other students' observations are similar and see if you can eyeball any trends. Check out other areas on the map and see if there are trends for tracts in specific areas."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "\n",
    "**QUESTION:** Do specific characteristics cluster in different areas? Which ones? Which characteristsics seem to cluster together? What types of data do you think will correlate with socioeconomic characteristics like median income, poverty rate, education?  Why?\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Type your answer here, replacing this text._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Part 2: Our Metrics<a id='ourmetrics'></a>\n",
    "\n",
    "Now that you have made some predictions, we can compare our data with socioeconomic data from the U.S. Census for the different tracts we visited and see if we can find evidence to support them. From your data, we can create some point scales that measure different aspects of a neighborhood.\n",
    "\n",
    "For example, we can make a scale called “social disorder” for the first part of your responses. Let's first subeset our data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#class data is not changed after selection\n",
    "social_disorder = class_data.select(range(1, 12))\n",
    "social_disorder.show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll need to scale the values because all responses were not on the same scale. But for this part, the higher the value the more negative the social disorder was:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "social_disorder = scale_values(social_disorder, np.arange(1, 11))\n",
    "social_disorder.show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that our values are scaled, we can take the mean across all observation for a given census tract for a given column, and then take the mean across columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#extracting means across columns\n",
    "means = social_disorder.group(\"Census Tract\", np.mean).drop(\"Census Tract\").values.mean(axis=1)\n",
    "\n",
    "#assigning Census Tract to their respective means\n",
    "social_disorder = Table().with_columns(\n",
    "    \"Census Tract\", np.unique(social_disorder.column(\"Census Tract\")),\n",
    "    \"Social Disorder\", means\n",
    ")\n",
    "social_disorder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember, the higher the value the more negative we perceived the census tract to be.\n",
    "\n",
    "We can do the same for our amenities part:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#adding 2 new columns to Establishments table to create a new table for amenities\n",
    "#note: the columns are added at the end of the table\n",
    "amenities = ests_table.with_columns(\n",
    "    \"Census Tract\", class_data.column(\"Census Tract\"),\n",
    "    \"Trees\", class_data.column(\"Amount of Trees Linked the Block Fence (1 (Few) to 3 (Most) scale)\")\n",
    ")\n",
    "\n",
    "#converting Trees values to binary: 0 if \"1 (Few)\" and 1 if more\n",
    "amenities[\"Trees\"] = [(0, 1)[value > 1] for value in amenities[\"Trees\"]]\n",
    "amenities.show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Certain amenities are positive and indicate desirable conditions in a neighborhood. These characteristics include things like School or Daycares, and supermarkets. Let's create a table containing only positive amenities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#selecting all the necessary columns from amenities table to be included in positive_amenities\n",
    "positive_amenities = amenities.select(\n",
    "    'Census Tract',\n",
    "    'Banks or credit unions',\n",
    "    'Chain retail stores',\n",
    "    'Community center',\n",
    "    'Eating places/restaurants',\n",
    "    'Fire station',\n",
    "    'Parks',\n",
    "    'Playgrounds',\n",
    "     # 'Public library',\n",
    "    'Post office',\n",
    "    'Professional offices (doctor dentist lawyer accountant real estate)',\n",
    "    'Schools or daycare centers',\n",
    "    'Supermarkets/grocery stores',\n",
    "    'Trees'\n",
    ")\n",
    "positive_amenities.show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make positive amenities comparable between census tracts, we can find the mean of positive amenities for each census tract. A higher value indicates a more positive census tract"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#extracting means across columns\n",
    "means = positive_amenities.group(\"Census Tract\", np.mean).drop(\"Census Tract\").values.mean(axis=1)\n",
    "\n",
    "#assigning Census Tract to the respective means of positive amenities\n",
    "positive_amenities = Table().with_columns(\n",
    "    \"Census Tract\", np.unique(positive_amenities.column(\"Census Tract\")),\n",
    "    \"Positive Amenities\", means\n",
    ")\n",
    "positive_amenities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Certain amenities are negative and indicate undesirable conditions in a neighborhood. These characteristics include things like Bars or Fast Food Restaurants. Let's create a Data Frame with only negative amenities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#selecting all the necessary columns from amenities table to be included in negative_amenities\n",
    "negative_amenities = amenities.select(\n",
    "    'Census Tract',\n",
    "    'Auto repair/auto body shop',\n",
    "    'Bars and alcoholic beverage services',\n",
    "    'Bodega deli corner-store convenience store',\n",
    "    'Fast food or take-out places',\n",
    "    'Gas station',\n",
    "    'Liquor stores or Marijuana Dispensaries',\n",
    "    'Manufacturing' ,\n",
    "    'Payday lenders check cashers or pawn shops',\n",
    "    'Warehouses'\n",
    ")\n",
    "negative_amenities.show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To make negative amenities comparable between census tracts, we can find the mean of negative amenities for each census tract. A higher value indicates a more negative census tract"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#extracting means across columns\n",
    "means = negative_amenities.group(\"Census Tract\", np.mean).drop(\"Census Tract\").values.mean(axis=1)\n",
    "\n",
    "#assigning Census Tract to the respective means of  negative amenities\n",
    "negative_amenities = Table().with_columns(\n",
    "    \"Census Tract\", np.unique(negative_amenities.column(\"Census Tract\")),\n",
    "    \"Negative Amenities\", means\n",
    ")\n",
    "negative_amenities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Part 3: Census Data<a id='census'></a>\n",
    "\n",
    "Let's read in some data for census tracts from the [American FactFinder](https://factfinder.census.gov/faces/nav/jsf/pages/index.xhtml):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "official_data = Table.read_table(\"data/merged-census.csv\")\n",
    "official_data[\"Census Tract\"] = [str(i) for i in official_data[\"Census Tract\"]]\n",
    "\n",
    "# sub is the function that removes decimals. We defined this function in Part 1.\n",
    "official_data[\"Census Tract\"] = official_data.apply(sub, \"Census Tract\")\n",
    "official_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can add our columns to this table to put it all in one place:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "joined_data = (official_data\n",
    "               .join(\"Census Tract\", social_disorder)\n",
    "               .join(\"Census Tract\", positive_amenities)\n",
    "               .join(\"Census Tract\", negative_amenities)\n",
    "              )\n",
    "joined_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Add zeros for all features in tracts for which we did not collect data\n",
    "unobserved_tracts = [str(alameda[\"features\"][x]['properties']['name10']) for x in range(len(alameda[\"features\"])) \\\n",
    "                     if alameda[\"features\"][x]['properties']['name10'] not in list(joined_data[\"Census Tract\"])]\n",
    "\n",
    "unobserved_data = Table().with_column(\"Census Tract\", unobserved_tracts)\n",
    "for col in joined_data.labels[1:]:\n",
    "    unobserved_data[col] = np.zeros(len(unobserved_tracts))\n",
    "\n",
    "    \n",
    "# Append these rows filled with 0 to the end of the table.\n",
    "joined_data = joined_data.append(unobserved_data)\n",
    "joined_data.show(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Mapping Exploration\n",
    "\n",
    "Before we quantify the relationship between the census data and our own metrics, let's do some exploratory mapping. We can now add our social disorder and amenities metrics to the popup too!\n",
    "\n",
    "First we'll map a choropleth of unemployment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "map2 = folium.Map(location=(37.8044, -122.2711), zoom_start=11.4)\n",
    "folium.features.Choropleth(geo_data=alameda,\n",
    "             name='unemployment', \n",
    "             data=joined_data.to_df(),\n",
    "             columns=['Census Tract', 'Unemployment %'],\n",
    "             key_on='feature.properties.name10',  \n",
    "             fill_color='BuPu', \n",
    "             fill_opacity=0.7, \n",
    "             line_opacity=0.2,\n",
    "             legend_name='Unemployment Rate (%)'\n",
    "            ).add_to(map2)\n",
    "folium.LayerControl().add_to(map2)\n",
    "map2.save(\"maps/map2.html\")\n",
    "IFrame('maps/map2.html', width=700, height=400)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Household Median Income:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "map3 = folium.Map(location=(37.8044, -122.2711), zoom_start=11.4)\n",
    "folium.Choropleth(geo_data=alameda, \n",
    "             name='household median income', \n",
    "             data=joined_data.to_df(),\n",
    "             columns=['Census Tract', 'Household Median Income'],\n",
    "             key_on='feature.properties.name10',  \n",
    "             fill_color='BuPu', \n",
    "             fill_opacity=0.7, \n",
    "             line_opacity=0.2,\n",
    "             legend_name='Household Median Income'\n",
    "            ).add_to(map3)\n",
    "map3.save(\"maps/map3.html\")\n",
    "IFrame('maps/map3.html', width=700, height=400)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bachelor's Degree or higher %:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "map4 = folium.Map(location=(37.8044, -122.2711), zoom_start=11.4)\n",
    "folium.Choropleth(geo_data=alameda, \n",
    "             name=\">= bachelor's %\", \n",
    "             data=joined_data.to_df(),\n",
    "             columns=['Census Tract', \"Bachelor's Degree or higher %\"],\n",
    "             key_on='feature.properties.name10',  \n",
    "             fill_color='BuPu', \n",
    "             fill_opacity=0.7, \n",
    "             line_opacity=0.2,\n",
    "             legend_name=\">= Bachelors Degree\"\n",
    "            ).add_to(map4)\n",
    "map4.save(\"maps/map4.html\")\n",
    "IFrame('maps/map4.html', width=700, height=400)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now our \"social disorder\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = joined_data.to_df()\n",
    "df[\"Social Disorder\"].min()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "map5 = folium.Map(location=(37.8044, -122.2711), zoom_start=11.4)\n",
    "folium.Choropleth(geo_data=alameda, \n",
    "             name='social disorder', \n",
    "             data=joined_data.to_df(),\n",
    "             columns=['Census Tract', \"Social Disorder\"],\n",
    "             key_on='feature.properties.name10',  \n",
    "             fill_color='BuPu', \n",
    "             fill_opacity=0.7, \n",
    "             line_opacity=0.2,\n",
    "             legend_name=\"Social Disorder\"\n",
    "            ).add_to(map5)\n",
    "map5.save(\"maps/map5.html\")\n",
    "IFrame('maps/map5.html', width=700, height=400)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now \"Positive Amenities\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "map6 = folium.Map(location=(37.8044, -122.2711), zoom_start=11.4)\n",
    "folium.Choropleth(geo_data=alameda, \n",
    "             name='positive amenities', \n",
    "             data=joined_data.to_df(),\n",
    "             columns=['Census Tract', \"Positive Amenities\"],\n",
    "             key_on='feature.properties.name10',  \n",
    "             fill_color='BuPu', \n",
    "             fill_opacity=0.7, \n",
    "             line_opacity=0.2,\n",
    "             legend_name=\"Positive Amenities\"\n",
    "            ).add_to(map6)\n",
    "map6.save(\"maps/map6.html\")\n",
    "IFrame('maps/map6.html', width=700, height=400)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, \"Negative Amenities\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "map7 = folium.Map(location=(37.8044, -122.2711), zoom_start=11.4)\n",
    "folium.Choropleth(geo_data=alameda, \n",
    "             name='negative amenities', \n",
    "             data=joined_data.to_df(),\n",
    "             # change the second element with any column you like!\n",
    "             columns=['Census Tract', \"Negative Amenities\"], \n",
    "             key_on='feature.properties.name10',  \n",
    "             fill_color='BuPu', \n",
    "             fill_opacity=0.7, \n",
    "             line_opacity=0.2,\n",
    "             legend_name=\"Negative Amenities\" # also remember to change the legend name\n",
    "            ).add_to(map7)\n",
    "map7.save(\"maps/map7.html\")\n",
    "IFrame('maps/map7.html', width=700, height=400)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "    \n",
    "**QUESTION:** What do you notice?\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Type your answer here, replacing this text._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "\n",
    "**QUESTION:** Try copying and pasting one of the mapping cells above and change the `column_name` variable to a different variable (column in our data) you'd like to map, then run the cell!\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Hint:** Go to the cell above (map7 cell) and read the comments in the code. You will find how to change the variables plotted!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Variable Distributions\n",
    "\n",
    "We can also visualize the distributions of these variables according to census tract with [histograms](https://en.wikipedia.org/wiki/Histogram). A histogram will create bins, or ranges, within a variable and then count up the frequency for that bin. If we look at household median income, we may have bins of $10,000, and then we'd count how many tracts fall within that bin:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# only take rows where we have data\n",
    "valid_data = joined_data.where(joined_data['Unemployment %'] != 0)\n",
    "\n",
    "# some interactive function\n",
    "def viz_dist(var_name, tract):\n",
    "    x = valid_data.where(var_name, lambda x: not pd.isna(x))[var_name]\n",
    "    \n",
    "    plt.hist(x)\n",
    "    plt.axvline(x=valid_data.where(\"Census Tract\", tract)[var_name], color = \"RED\")\n",
    "    plt.xlabel(var_name, fontsize=18)\n",
    "    plt.show()\n",
    "\n",
    "display(widgets.interactive(viz_dist, var_name=list(valid_data.labels[1:]), \n",
    "                            tract=list(valid_data[\"Census Tract\"])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "\n",
    "**QUESTION:** What do these distributions tell you?\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Type your answer here, replacing this text._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "\n",
    "# Part 4: Correlation<a id='correlation'></a>\n",
    "\n",
    "Let's first analyze income levels. We have sorted the data according to income level. Compare the income levels to the level of social disorder and amenities. Is there a correlation you can spot(as one increases or decreases, does the other do the same)?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(valid_data\n",
    " .sort(\"Household Median Income\", descending=True)\n",
    " .select(\"Household Median Income\", \"Social Disorder\", \"Positive Amenities\", \"Negative Amenities\")\n",
    ").show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "\n",
    "**QUESTION:** Did you look at the whole table? A common mistake is to assume that since the top 10 results follow or do not follow a pattern, the rest don't. Real life data is often messy and not clean. Does the correlation continue throughout the whole table (a.k.a. as income decreases the points decrease) or is there no pattern? What does this mean about the data?\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Type your answer here, replacing this text._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "Eyeballing patterns is not the same as a statisical measure of a correlation; you must quantify it with numbers and statistics to prove your thoughts. Looking at the tables is not a very statistical measure of how much a variable correlates to the results. What does it mean for a variable \"income\" to match 7 out of the top 15 social disorder points? Does this correlate to the rest of the results? How well does it correlate? \n",
    "\n",
    "### The correlation coefficient - *r*\n",
    "\n",
    "> The correlation coefficient ranges from −1 to 1. A value of 1 implies that a linear equation describes the relationship between X and Y perfectly, with all data points lying on a line for which Y increases as X increases. A value of −1 implies that all data points lie on a line for which Y decreases as X increases. A value of 0 implies that there is no linear correlation between the variables. ~Wikipedia\n",
    "\n",
    "*r* = 1: the scatter diagram is a perfect straight line sloping upwards\n",
    "\n",
    "*r* = -1: the scatter diagram is a perfect straight line sloping downwards.\n",
    "\n",
    "Let's calculate the correlation coefficient between acceleration and price. We can use the `corr` method of a DataFrame to generate a correlation matrix of our `joined_data`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_data.to_df().corr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll notice that the matrix is mirrored with a `1.000000` going down the diagonal. This matrix yields the correlation coefficient for each variable to every other variable in our data.\n",
    "\n",
    "For example, if we look at the `Social Disorder`, we see that there is a `0.911115` correlation, implying that there is a strong positive relationship between our constructed social disorder variable and the unemployment rate (i.e., as one goes up the other goes up too)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\">\n",
    "\n",
    "**QUESTION:** What else do you notice about the correlation values above?\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Type your answere here, replacing this text._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 5: Regression<a id='regression'></a>\n",
    "\n",
    "We will now use a method called linear regression to make a graph that will show the best fit line that correlates to the data. The slope of the line will show whether it is positively correlated or negatively correlated. The code that we've created so far has helped us establish a relationship between our two variables. Once a relationship has been established, it's time to create a model of the data. To do this we'll find the equation of the **regression line**!\n",
    "\n",
    "The regression line is the **best fit** line for our data. It’s like an average of where all the points line up. In linear regression, the regression line is a perfectly straight line! Below is a picture showing the best fit line.\n",
    "\n",
    "![image](http://onlinestatbook.com/2/regression/graphics/gpa.jpg)\n",
    "\n",
    "As you can infer from the picture, once we find the **slope** and the **y-intercept** we can start predicting values! The equation for the above regression to predict university GPA based on high school GPA would look like this:\n",
    "\n",
    "$\\text{UNIGPA}_i= \\alpha + \\beta \\cdot \\text{HSGPA} + \\epsilon_i$\n",
    "\n",
    "The variable we want to predict (or model) is the left side `y` variable, the variable which we think has an influence on our left side variable is on the right side. The $\\alpha$ term is the y-intercept and the $\\epsilon_i$ describes the randomness.\n",
    "\n",
    "We can set up a visualization to choose which variables we want as `x` and `y` and then plot the line of best fit:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def f(x_variable, y_variable):\n",
    "    \n",
    "    if \"median house value\" in [x_variable, y_variable]:\n",
    "         # if not all census tracts have values, we drop N/A values and create new table\n",
    "        drop_na = joined_data.to_df().dropna() \n",
    "        #selecting columns from the new table\n",
    "        x = drop_na[x_variable]\n",
    "        y = drop_na[y_variable]\n",
    "        \n",
    "    else:\n",
    "        x = joined_data[x_variable]\n",
    "        y = joined_data[y_variable]\n",
    "        \n",
    "    #plotting the graph\n",
    "    plt.scatter(x, y)\n",
    "    plt.xlabel(x_variable, fontsize=18)\n",
    "    plt.ylabel(y_variable, fontsize=18)\n",
    "    plt.plot(np.unique(x), np.poly1d(np.polyfit(x, y, 1))(np.unique(x)), color=\"r\") #calculate line of best fit\n",
    "    plt.show()\n",
    "    slope, intercept, r_value, p_value, std_err = stats.linregress(x, y) #gets the r_value\n",
    "    print(\"R-squared: \", r_value**2)\n",
    "    \n",
    "display(widgets.interactive(f, x_variable=list(valid_data)[1:], y_variable=list(valid_data)[1:]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Note:*** The `R-squared` tells us how much of the variation in the data can be explained by our model.\n",
    "\n",
    "Why is this a better method than just sorting tables? First of all, we are now comparing all of the data in the graph to the variable, rather than comparing what our eyes glance quickly over. It shows a more complete picture than just saying \"There are some similar results in the top half of the sorted data\". Second of all, the graph gives a more intuitive sense to see if your variable does match the data. You can quickly see if the data points match up with the regression line. Lastly, the r-squared value will give you a way to quantify how good the variable is to explain the data.\n",
    "\n",
    "One of the beautiful things about computer science and statistics is that you do not need to reinvent the wheel. You don't need to know how to calculate the `R-squared` value, or draw the regression line; someone has already implemented it! You simply need to tell the computer to calculate it. However, if you are interested in these mathematical models, take a data science or statistics course!\n",
    "\n",
    "\n",
    "---\n",
    "## Peer Consulting Office Hours\n",
    "\n",
    "Not quite understand everything covered in this notebook? Curious about concepts covered in this lab at a deeper level? Looking for more data enabled courses with modules like this? **Come to Peer Consulting Office Hours at 3rd Floor Moffitt!** Find a Peer Consultant with the expertise you need and get your questions answered: [Office Hours Schedule is linked here]( https://data.berkeley.edu/education/peer-consulting)!\n",
    "\n",
    "\n",
    "## We Want Your Feedback!\n",
    "\n",
    "Help us make your module experience better in future courses: ***Please fill out our short [feedback form](https://docs.google.com/forms/d/e/1FAIpQLSfahkYSKqlEEfC6WMKlaqeIxRVj0r7T4N5lgBf9bRVwRG58wQ/viewform)!***\n",
    "\n",
    "---\n",
    "\n",
    "Fall 2019 Notebook revised by: Yana Mykhailovska, Timlan Wong, Xiantao Wang\n",
    "\n",
    "Fall 2018 Notebook developed by: Keeley Takimoto, Anna Nguyen, Taj Shaik, Keiko Kamei\n",
    "\n",
    "Adapted from Spring 2018 and Fall 2017 materials by: Anna Nguyen, Sujude Dalieh, Michaela Palmer, Gavin Poe, Theodore Tran \n",
    "\n",
    "Data Science Modules: http://data.berkeley.edu/education/modules"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " "
   ]
  }
 ],
 "metadata": {
  "@webio": {
   "lastCommId": null,
   "lastKernelId": null
  },
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}