R-Data-Visualization.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# (Draft) Data Science - Professional Cerificate Program - Havard University"
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Data Science: Visualization"
      ],
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Brief preface\n",
        "\n",
        "**A picture is worth a thousand words**\n",
        "\n",
        "Data visualization provides a powerful way to communicate a data driven finding. In some cases, the visualization is so convincing that no follow up analysis is required.\n",
        "\n",
        "An example: [Battling Infectious Diseases in the 20th Century: The Impact of Vaccines](http://graphics.wsj.com/infectious-diseases-and-vaccines/)\n",
        "\n",
        "> **The greatest value of a picture\n",
        "is when it forces us to notice what we never expected to see - John Tukey**\n",
        "\n",
        "Exploratory data analysis is perhaps the most important part of data analysis,\n",
        "yet it is often overlooked. Data visualization is also now pervasive and philanthropic\n",
        "in educational organizations.\n",
        "\n",
        "One example comes from GAPminder and the talks, \"New Insights on Poverty\"\n",
        "and \"The Best Stats You&#39;ve Ever Seen\", Hans Roslings forced us to notice the unexpected with a series of plots\n",
        "related to world health and economics. In his videos, he used animated graphs to show us how the world was changing,\n",
        "and how old narratives are no longer true.\n",
        "\n",
        "[New Insights on Poverty](https://www.ted.com/talks/hans_rosling_reveals_new_insights_on_poverty)\n",
        "\n",
        "[The Best Stats You&#39;ve Ever Seen](https://www.ted.com/talks/hans_rosling_shows_the_best_stats_you_ve_ever_seen)\n",
        "\n",
        "[Let my dataset change your mindset](https://www.gapminder.org/videos/ted-us-state-department/)\n",
        "\n",
        "It is also important to note that mistakes, biases, systematic errors,\n",
        "and other unexpected problems often lead to data\n",
        "that should be handled with care. Failure to discover these problems often leads to flawed analyzes\n",
        "and false discoveries.\n",
        "\n",
        "The fact that it can be hard or impossible to notice an error just\n",
        "from the reporter results makes data visualization particularly important."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Section 1 - Overview\n",
        "\n",
        "- understand the importance of data visualization for communicating data-driven findings.\n",
        "\n",
        "- be able to use distributions to summarize data.\n",
        "\n",
        "- be able to use the average and the standard deviation to understand the normal distribution.\n",
        "\n",
        "- be able to assess how well a normal distribution fits the data using a quantile-quantile plot.\n",
        "\n- be able to interpret data from a boxplot."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Section 1.1 Introduction to Data Visualization"
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Distributions\n",
        "\n",
        "A distribution is a basic summary of data values. An average score number and occasionally the standard deviation can form a distribution.\n",
        "\n",
        "So, for example, you might read a report stating that scores at this high school\n",
        "were 680 plus or minus 50.\n",
        "\n",
        "**680 ± 50** (The first is the average score, the last the standard deviation)\n",
        "\n",
        "It turns out that in some cases, these two numbers\n",
        "are pretty much all we need to understand the data. Data visualization techniques will help us determine when this two-number summary is appropriate. These same techniques will serve as alternatives for when these two numbers are not enough.\n",
        "\n",
        "The most basic statistical summary of a list of objects or numbers\n",
        "is its **distribution**.\n",
        "\n",
        "Once a vector has been summarized as a distribution,\n",
        "there are several data visualization techniques to effectively relay\n",
        "this information."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "install.packages(\"dslabs\")"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "code",
      "source": [
        "library(dslabs)"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Data Types\n",
        "\n",
        "Categorial\n",
        "(Variables defined by a small number of groups: Sex(male, female), Regions(North, South)\n",
        "- Ordinals (Can be ordered in a semantic way: Spiciness(mild, medium, hot))\n",
        "- Non-Ordinals\n",
        "\n",
        "Numericals\n",
        "- Discrete (Have to be round numbers)\n",
        "- Continuous (Can have any value)"
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Data Type Semantics"
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "The heights$height vector could be considered a **categorial** with categories like small, average, tall,\n",
        "\n",
        "but,\n",
        "\nit is more effective to consider the heights$height vector to be **numerical** given the number of unique values we observe and the fact that if we keep collecting data even more will be observed."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "data(heights)\n",
        "str(heights)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "'data.frame':\t1050 obs. of  2 variables:\n",
            " $ sex   : Factor w/ 2 levels \"Female\",\"Male\": 2 2 2 2 2 1 1 1 1 2 ...\n",
            " $ height: num  75 70 68 74 61 65 66 62 66 67 ...\n"
          ]
        }
      ],
      "execution_count": 6,
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tab <- table(heights$height)\n",
        "sum(tab == 1) # unique values we observe"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "63"
            ],
            "text/markdown": [
              "63"
            ],
            "text/latex": [
              "63"
            ],
            "text/plain": [
              "[1] 63"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 7,
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Section 1.2 - Introduction to Distributions"
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Describe Heights to ET**\n",
        "\n",
        "A first artificial problem:\n",
        "\n",
        "Pretend that we have to describe the heights of our classmates\n",
        "to ET, an extraterrestrial that has never seen humans.\n",
        "\n",
        "As a first step, we need to collect data.\n",
        "To do this, we asked students to report their heights in inches.\n",
        "We asked them to provide sex information because we\n",
        "know there are two different groups of heights, males and females.\n",
        "\n",
        "One way to convey the heights to ET is to simply\n",
        "send him the list of 924 heights. But there are much more effective ways to convey this information, and understanding the concepts of distributions will help.\n",
        "\n",
        "Again: The most basic statistical summary of a list of objects or numbers\n",
        "is its distribution.\n",
        "\n\n"
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "**The simplest way to think of a distribution\n",
        "is as a compact description of a list with many elements.** For example: With categorical data, the distribution simply describes the proportions of each unique category. The sex represented in the heights data set can be summarized by the proportions of each of the two categories, female and male. This two category frequency table is the simplest form\n",
        "of a distribution we can form.\n"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "one_percent = length(heights$sex)\n",
        "male_percentage = length(which(heights$sex == 'Male')) / length(heights$sex) * 100\n",
        "female_percantage = length(which(heights$sex == 'Female')) / length(heights$sex) * 100\n",
        "print('Distribution: Male/Female %')\n",
        "round(male_percentage)\n",
        "round(female_percantage)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[1] \"Distribution: Male/Female %\"\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "77"
            ],
            "text/markdown": [
              "77"
            ],
            "text/latex": [
              "77"
            ],
            "text/plain": [
              "[1] 77"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "23"
            ],
            "text/markdown": [
              "23"
            ],
            "text/latex": [
              "23"
            ],
            "text/plain": [
              "[1] 23"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 38,
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "When the data is numerical, the task is much more challenging.\n",
        "\n",
        "Why ? In general, when data is not categorical, reporting the frequency of each unique entry is not an effective summary since most entries are unique.\n",
        "\n",
        "For example, while several students reported a height of 68 inches,\n",
        "only one student reported a height of 68.503937007874 inches.\n",
        "And only one student reported a height of 68.8976377952756 inches.\n",
        "We assume that they converted from 174 centimeters and 175 centimeters\n",
        "to inches respectively.\n",
        "\n",
        "Statistics textbooks teach us that a more useful way\n",
        "to define a distribution for numerical data\n",
        "is to define a function that reports the proportion of the data\n",
        "below a value A for all possible values of A. **This function is called a cumulative distribution function or CDF.**\n",
        "\n",
        "In fact, we can report the proportion of values between any two heights,\n",
        "say a and b, by computing f of b, and then subtracting f of a.\n",
        "This means that if we send this plot to ET,\n",
        "he will have all the information needed to construct the entire list.\n",
        "\n",
        "<small>Final note: Because CDFs can be determined mathematically,\n",
        "as opposed to using data as we do here, the word empirical\n",
        "is added to distinguish, and we use the term empirical CDF or ECDF.</small>"
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "(ADD EXAMPLE CDF PLOT)"
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "Although the CDF provides all the information we need and it is widely discussed in statistics textbooks, the plot is actually not very popular in practice.\n",
        "\n",
        "The main reason is that it does not easily convey characteristics of interest, such as,\n",
        "at what value is the distribution centered, is the distribution symmetric, what range contains 95% of the data.\n",
        "\n",
        "We can decipher all this from the plot, but it's not that easy.\n",
        "\n",
        "Histograms are much preferred because they greatly\n",
        "facilitate answering such questions. Histograms sacrifice just a bit of information to produce plots that are much easier to interpret.\n",
        "\nThe simplest way to make a histogram is to divide a span of our data into non-overlapping bins of the same size. Then for each bin, we count the number of values that fall in that interval. The histogram plots these counts as bars with the base of the bar the interval."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "head(heights)"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table>\n",
              "<thead><tr><th scope=col>sex</th><th scope=col>height</th></tr></thead>\n",
              "<tbody>\n",
              "\t<tr><td>Male  </td><td>75    </td></tr>\n",
              "\t<tr><td>Male  </td><td>70    </td></tr>\n",
              "\t<tr><td>Male  </td><td>68    </td></tr>\n",
              "\t<tr><td>Male  </td><td>74    </td></tr>\n",
              "\t<tr><td>Male  </td><td>61    </td></tr>\n",
              "\t<tr><td>Female</td><td>65    </td></tr>\n",
              "</tbody>\n",
              "</table>\n"
            ],
            "text/markdown": [
              "\n",
              "sex | height | \n",
              "|---|---|---|---|---|---|\n",
              "| Male   | 75     | \n",
              "| Male   | 70     | \n",
              "| Male   | 68     | \n",
              "| Male   | 74     | \n",
              "| Male   | 61     | \n",
              "| Female | 65     | \n",
              "\n\n"
            ],
            "text/latex": [
              "\\begin{tabular}{r|ll}\n",
              " sex & height\\\\\n",
              "\\hline\n",
              "\t Male   & 75    \\\\\n",
              "\t Male   & 70    \\\\\n",
              "\t Male   & 68    \\\\\n",
              "\t Male   & 74    \\\\\n",
              "\t Male   & 61    \\\\\n",
              "\t Female & 65    \\\\\n",
              "\\end{tabular}\n"
            ],
            "text/plain": [
              "  sex    height\n",
              "1 Male   75    \n",
              "2 Male   70    \n",
              "3 Male   68    \n",
              "4 Male   74    \n",
              "5 Male   61    \n",
              "6 Female 65    "
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 8,
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Smooth Density Plots**\n",
        "\n",
        "Smooth density plots are similar to histograms,\n",
        "but are aesthetically more appealing.\n",
        "\n",
        "Heuristic explanation\n",
        "\n",
        "The main new concept to understand\n",
        "is that we assume that our list of observed values of observed heights\n",
        "comes from a much, much larger list of unobserved values.\n",
        "\n",
        "So in the case of male heights, you can imagine\n",
        "our lists of students heights comes from a hypothetical list containing\n",
        "all the heights of all the male students in all the world measured\n",
        "very precisely.\n",
        "\n",
        "Because we're assuming that we have (supposed) a million values measured very precisely,\n",
        "we can make a histogram with very, very small bins.\n",
        "This is going to help us understand what smooth densities are.\n",
        "The assumption is that if we do this, consecutive bins will be similar.\n",
        "This is what we mean by smooth. To make the curve not depend on the hypothetical size\n",
        "of the hypothetical list, we compute the curve on the frequency scale\n",
        "rather than the count scale.\n",
        "\n",
        "Now back to reality.\n",
        "\n",
        "With our example, we don't have millions of measurements.\n",
        "Instead, we have 708 and we can't make a histogram with very, very small bins.\n",
        "So how can we estimate this hypothetical smooth curve\n",
        "that we would see if we would see all the measurements?"
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "So what we're going to do is we're going to start\n",
        "by making a histogram with our data, compute frequencies rather than counts,\n",
        "and using the bin size appropriate for our data,\n",
        "so the histogram looks like this."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "(ADD EXAMPLE)"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "hist(heights$height) # unfinished example"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Plot with title \"Histogram of heights$height\""
            ],
            "image/png": [
              "iVBORw0KGgoAAAANSUhEUgAAA0gAAANICAMAAADKOT/pAAAAMFBMVEUAAABNTU1oaGh8fHyM\n",
              "jIyampqnp6eysrK9vb3Hx8fQ0NDZ2dnh4eHp6enw8PD////QFLu4AAAACXBIWXMAABJ0AAAS\n",
              "dAHeZh94AAAgAElEQVR4nO3djXqqvBJA4YCI/3j/d1sBf0CwCs6Qybje55yv7u42wZhVq7Lb\n",
              "cAbwtRD7AAAPCAkQQEiAAEICBBASIICQAAGEBAggJEAAIQECCAkQQEiAAEICBBASIICQAAGE\n",
              "BAggJEAAIQECCAkQQEiAAEICBBASIICQAAGEBAggJEAAIQECCAkQQEiAAEICBBASIICQAAGE\n",
              "BAggJEAAIQECCAkQQEiAAEICBBASIICQAAGEBAggJEAAIaEjsB9mYuFeCfdNdb0URnfZepGD\n",
              "2WQhPGYaP5L//m7w3tHDrmfJt6pTuEVIr3wU0iFbZAE3l7klQxo/7FVolIpT+PVb13aKj0L6\n",
              "b78JykM4fjjrR7t89IO24eqoNoVjv3VtpxiE9P8HLXQsM2f9ZIBLrbvmf2u1KRz7rWs7xat7\n",
              "pGpTfwtU7Nr3hNuH7df1d1/766ecLn9abTufecqbb5l2xeVyXp5u423zkB8udwZZWB360/fG\n",
              "G9mkh8tRrE/tn6syC1k76P1DB0fw+IzHYXeuS/sxzf8yySl+xS9d12lehHTKrptk1dsu18cX\n",
              "oWg+43D9kMdn5s0n3D4qHNr3tn8+lff33XXHG+zK+oFM866s2dm3Qzp0DnZ4BI/PuI/XvS7n\n",
              "c9aGJDvFz/il6zrNi5AuX4UvX8Cry/bZdrZLcSukLSm7//H2maH+tMujkFV1Ppe9Pi47r9tg\n",
              "qzfeWEhX6+5sWedgR4/g+hn393evS/Ondfc6i0zxM37puk4Tuq7vaP9bf42uLncy9/ed95e3\n",
              "2+ryndLl7eW7scvjjKx+kz0+sw6ofhRy6o10eW/9ED8/Nm8ecz+NN/KtXXZogqw3dptn1VZx\n",
              "/dCRI+h8Rvdbzsd1ObXh7kWn+Bm/dW2neBFSvW/uD4Vu22Xdfk1vttG6uT9pPmL3+Mz909Dt\n",
              "fw+9N48PeBpvJKR6vCrc7g2r9r3F/UNfHEHVuypP16V+xrq2kpziZ/zWtZ3iRUib67cv9y/c\n",
              "1zfNRmu+qrePNkb/+vIBu3IV7iGdB2/un9cd79UzYve/vMru7xs5gpE/9q/LuY2ifSFJaoqf\n",
              "8VvXdorHTuhvjfK2pU6Dv7pdCi/22HmXd8r8P6TepQ9DCsNP+H+X965L+xfH3ves30/xK37r\n",
              "2k7xKqRztWufUlv1/up+D5K9/mJdf8HP19vjpHuk7Pkvn/ds1vvLaXcXneuS1XOG8JTJ91P8\n",
              "iN+6tlO8DKnWvMrzeF/x9jFS87f59f1vQyrePkbqvC16j8D673s+grHJrtfl8t/Nq5C+neIH\n",
              "/Na1neJFSPn9cfftrqJ6+axdGN9j7++R3j9r13m7a58u23XvI98cwfWwe9elnvPy/1Pvrvbr\n",
              "KX4GIb3yIqTLXludmsfp9WPy+nmu+u39ldb+yy5Pe2zVfPA+exvS83j/hvSY7TDyvtFdfj3s\n",
              "/nW5v3a1E5ziZxDSKy9Cuj9Ab54lXt8u3HZ+e4r2/vohT3vsei5A/Qrs4f+Qnsb7P6TrbL3T\n",
              "tl8cwfXt7bB71+U250p0il9BSK+8Cql9TLFqH8PUX8av7ayzzhPJx/o0tP0gkvrd2fp4up2w\n",
              "MDL6TW+8/0M6V2XefSX1nyO4vb0ddu+61Gf+Pf97pO+n+BGEpKi6vsqfzhHMeIYg/pU0gZAU\n",
              "XB9nHFf9E+gSOIIJIcW/kqYQkoLHUwX9U4NcHUH8K2kKISm4//OEaM9bLXAE8a+kKYSkodrU\n",
              "zyVn63hfqhc4gvhX0hJCAgQQEiCAkAABhAQIICRAACEBAggJEEBIgABCAgQQEiCAkAABhAQI\n",
              "ICRAACEBAggJEEBIgABCAgQQEiCAkAABhAQIICRAACEBAggJEEBIgABCAgQQEiCAkAABhAQI\n",
              "ICRAACEBAggJEEBIgABCAgQQEiCAkAABhAQIICRAACEBAggJEEBIgABCAgQQEiCAkAABhAQI\n",
              "ICRAACEBAggJEEBIgABCAgQQEiCAkAABhAQIICRAACEBAggJEEBIgABCAgQQkivhM7EP0yHW\n",
              "1JXPbk5udHmsqSuEFAtr6gohxcKaukJIsbCmrhBSLKypK4QUC2vqCiHFwpq6QkixsKauEFIs\n",
              "rKkrhBQLa+oKIcXCmrpCSLGwpq4QUiysqSuEFAtr6gohxcKaukJIsbCmrhBSLKypK4QUC2vq\n",
              "CiHFwpq6QkixsKauEFIsrKkrhBQLa+oKIcXCmrpCSLGwpq4QUiysqSuEFAtr6gohxcKaukJI\n",
              "sbCmrhBSLKypK4QUC2vqCiHFwpq6QkixsKauEFIsrKkrhBQLa5oKwV9qyY0ujzVNxUe3FCHF\n",
              "wpqmgpBMY01TQUimsaapICTTWNNUEJJprGkqCMk01jQVhGQaa5oKQjKNNU0FIZnGmqaCkExj\n",
              "TVNBSKaxpqkgJNNY01QQkmmsaSoIyTTWNBWEZNr8NT1siuZfwBTlQfB48AohmTZ3Tau886/J\n",
              "VqKHhFGEZNrcNS1Dtjs2l077LJRyB4QXCMm0uWuaheP98jFkMgeDfxCSaXPXtPfTAT78UQH4\n",
              "BiGZxj1SKgjJtC8eI+1PzSUeIy2DkEybvaarzrN2eSV5SBhFSKZ98TpS2byOlBUbXkdaAiGZ\n",
              "xpqmgpBMY01TQUimcYpQKgjJNE4RSgUhmcYpQqkgJNN4QTYVhGSa0ilC03/VCN4gJNMWuEfi\n",
              "ZhNBSKYtcIoQN5sIQjJtgVOEuNlEEJJpC5wixM0mgpBMW2BNudlEEJJphJQKQjJt9ppW6xBW\n",
              "++sg/47CzSaCkEybfYpQ1p5o1w5CSPoIybT5T39vLzVts+Y0O0JaACGZNv8F2ebNKctPhLQI\n",
              "QjLt21OEqtWKkBZBSKbNXdM83F6EzVeEtARCMm3umm7D+nrpFFaEtABCMm32mpb3evZvTvDm\n",
              "ZhNBSKbNX9Njcbt0WhOSPkIyjTMbUkFIphFSKgjJNEJKBSGZRkipICTTCCkVhGQaIaWCkEwj\n",
              "pFQQkmmElApCMo2QUkFIphFSKgjJNEJKBSGZRkipICTTCCkVhGQaIaWCkEwjpFQQkmmElApC\n",
              "Mo2QUkFIphFSKgjJNEJKBSGZRkipICTTCCkVhGQaIaWCkEwjpFQQkmmElApCMo2QUkFIphFS\n",
              "KgjJNEJKBSGZRkipICTTCCkVhGQaIaWCkEwjpFQQkmmElApCMo2QUkFIphFSKgjJNEJKBSGZ\n",
              "RkipkAzpM18d7q8hpFQsfo/E7TYFIaWCkEwjpFQQkmmElApCMo2QUkFIphFSKgjJNEJKBSGZ\n",
              "RkipICTTCCkVhGQaIaWCkEwjpFQQkmmElApCMo2QUkFIphFSKgjJNEJKBSGZRkipICTTCCkV\n",
              "hGQaIaWCkEwjpFQQkmmElApCMo2QUkFIphFSKgjJNEJKBSGZRkipICTTCCkVhGQaIaWCkEwj\n",
              "pFQQkmmElApCMo2QUkFIphFSKgjJNEJKBSGZRkipICTTCCkVhGQaIaWCkEwjpFQQkmmElApC\n",
              "Mo2QUkFIphFSKgjJNEJKBSGZRkipICTTCCkVhGQaIaWCkEwjpFQQkmmElApCMo2QUkFIphFS\n",
              "KgjJNEJKBSGZRkipICTTCCkVhGQaIaWCkEwjpFQQkmmElApCMo2QUkFIphFSKgjJNEJKBSGZ\n",
              "RkipICTTCCkVhGQaIaWCkEwjpFQQkmmElApCMm3+ah02RagV5UFrCnQQkmlzV6vKw8NKZQr0\n",
              "EJJpc1erDNnu2Fw67bNQakyBHkIybe5qZeF4v3wMmcYU6CEk0+auVgiv/iA2BXoIyTTukVJB\n",
              "SKZ98Rhpf2ou8RhpGYRk2uzVWnWetcsrlSnQRUimffE6Utm8jpQVG15HWgIhmcaZDakgJNMI\n",
              "KRWEZNqXq7XNQyj2qlOgRUimffc60vUZh3+ftOMGkUFIpn0VUhnK6nw+lWGrMQV6CMm0r0LK\n",
              "QvO8dxVyjSnQQ0imfRXS7dSg4SlCoWv+0eGBkEz7KqT1LSROEdJHSKbND6nYbPdhd7lYlZwi\n",
              "tABCMm1+SPdv20LIOEVIHyGZNnu1jsfttiiapxzKfzviBpFBSKZxZkMqCMk0QkoFIZlGSKkg\n",
              "JNMIKRWEZBohpYKQTPv26e8PTl7gBhFBSKbNXa0tIS2MkEyb/zpS9v/PVxWYAl2EZNr81Tq+\n",
              "+WdIAlOgg5BM+2K1tp0fbac0BR4IyTSetUsFIZlGSKkgJNMIKRWEZBohpYKQTCOkVBCSaYSU\n",
              "CkIyjZBSQUimEVIqCMk0QkoFIZlGSKkgJNMIKb7wmY+G+mxCwY9Ci5DiE9zXhBQLIcVHSA4Q\n",
              "UnyE5AAhxUdIDhBSfITkACHFR0gOEFJ8hOQAIcVHSA4QUnyE5AAhxUdIDhBSfITkACHFR0gO\n",
              "EFJ8hOQAIcVHSA4QUnyE5AAhxUdIDhBSfITkACHFR0gOEFJ8hOQAIcVHSA50VyvfnLSnwAhC\n",
              "cqC7WiEEjZa4Qd4gJAe6q1Xt1hotcYO8QUgOPK/WYZNLt8QN8gYhOTCyWsfscr+0VZ0CXYTk\n",
              "wHC19qvm53quFKdADyE58LRa1eZyd5Tvq0tNhdIUeEZIDvRW61A/2VAe278QW0dukDcIyYHe\n",
              "60iXO6NtdfuLTGMKjCAkB3qvIxV77SkwgpAc6L2OpD8FRhCSA73Vqsr6+7mslC2KG+QNQnKg\n",
              "u1qnrHmGIYRM9NwGbpA3CMmB7mqtwrq+L6pKuae+n6fACEJyoH/S6vMF8SkwgpAc6K5WFtoH\n",
              "RxUhLYqQHOiuVhlWh8ubwyqUWlNgBCE50Fut1fX3Z8udZzeYAkOE5EB/tXZFnZHgmd/DKTBA\n",
              "SA4ssFrcIG8QkgOEFB8hOUBI8RGSA73Vqv+ZeUttCgwRkgPd1dqEQEgREJID/RdkhZ+vG06B\n",
              "EYTkwOgpQnpTYAQhOdBdrSKo/IskbpA3CMmB/j+jaE4R0pwCIwjJgf63djzZEAMhOUBI8RGS\n",
              "A7wgGx8hOUBI8RGSA/3V2hf1d3WF7K+j4AZ5g5AcGP57pMv7+OEniyIkB7qrtQ2r5l+Zb8Na\n",
              "awqMICQH+qcIVefrD+TSmgIjCMmB51OECGl5hORAd7Xy6z3SMeRaU2AEITkw8hhpL3wWODfI\n",
              "G4TkQG+1Cn6KUAyE5MDwdaRQ7DSnwAAhOcCZDfERkgOEFB8hOUBI8RGSA/wzivgIyQFCio+Q\n",
              "HBhZrcNK9PeMcYO8Q0gOjK1WxUmriyIkB0ZXi2/tFkVIDoyt1jZk2lOgg5AcGH+yYaM1BUYQ\n",
              "kgNjIeWyP7mYG+QNQnKAF2TjIyQHCCk+QnLgxQuyki/KcoO8QUgOEFJ8hORAb7U22f7y30PG\n",
              "P+xbFCE50F2tTTg2b49B9BwhbpA3CMmB/rd2zxfEp8AIQnKgu1rZ/R6JnyK0JEJyoLtaZWge\n",
              "I/FThBZGSA70Vmt1fb6u1JsCQ4TkQH+1ds1PEdprToEBQnKAMxviIyQHCCk+QnKgv1pTftHY\n",
              "YdP+YNaifPOr0LlB3iAkB4ZPNpw/+kVjVd45mej/MyG4Qd4gJAe6qzXlF42VIdu1rzqd9tn/\n",
              "T/Nxg7xBSA70X5D9/BeN3V68rR3//6fp3CBvEJIDz6cIfRpSeP7Ez6bACEJyoLtaU37RGPdI\n",
              "cgjJgZHHSB+dIlSfTtQ+JcFjpG8RkgO91Zryi8ZWnWft8urjKTBESA4MX0f69BeNHcqmu6zY\n",
              "8DrSdwjJAc5siI+QHOiuViF71vfYFBhBSA58/iz2M04RkkJIDjw//f0pThGSQ0gOdFerKlZv\n",
              "7lweOEVIDiE50P/W7vOfaccLsnIIyYG5Ib05RUjlJ016RUgOzF0t7pHkEJIDc1eLU4TkEJID\n",
              "t9Wa/A0YpwiJISQH+iFNyYlThKQQkgPzQ5o8BV4gJAcIKT5CcoCQ4iMkBwgpPkJygJDiIyQH\n",
              "HiFNOxlhwsdzg7xBSA7MDWlLSGIIyYHZq3X8+DfNcoO8QUgOzF+t46e/Rokb5A1CcuCL1dp2\n",
              "zltVmuI3EJIDC6wWN8gbhOQAIcVHSA4QUnyE5AAhxUdIDhBSfITkACHFR0gOEFJ8hOQAIcVH\n",
              "SA4QUnyE5AAhxUdIDhBSfITkACHFR0gOEFJ8hOQAIcVHSA4QUnyE5AAhxUdIDhBSfITkACHF\n",
              "R0gOEFJ8hOQAIcVHSA4QUnyE5AAhxUdIDhBSfITkACHFR0gOEJKm8JnPxhL7IEJSQEiaFt/X\n",
              "hBQLIWkipJ9BSJoI6WcQkiZC+hmEpImQfgYhaSKkn0FImgjpZxCSJkL6GYSkiZB+BiFpIqSf\n",
              "QUiaCOlnEJKmtEOSO0/wBxCSprRDkhvqBxCSJqP7mpDkEZImo/uakOQRkiaj+5qQ5BGSJqP7\n",
              "mpDkEZImo/uakOQRkiaj+5qQ5BGSJqP7mpDkEZImo/uakOQRkiaj+5qQ5BGSJqP7mpDkEZIm\n",
              "o/uakOQRkiaj+5qQ5BGSJqP7mpDkEZImo/uakOQRkiaj+5qQ5BGSJqP7mpDkEZImo/uakOQR\n",
              "kiaj+5qQ5BGSJqP7mpDkEZImo/uakOQRkiaj+5qQ5BGSJqP7mpDkEZImo/uakOQRkiaj+5qQ\n",
              "5BGSJqP7mpDkEZImo/uakOQRkiaj+5qQ5BGSJqP7mpDkEZImo/uakOQRkiaj+5qQ5BGSJqP7\n",
              "mpDkEZImo/uakOQRkiaj+5qQ5BGSJqP7mpDkEZImo/uakOQRkiaj+5qQ5BGSJqP7mpDkEZIm\n",
              "o/uakOQRkiaj+5qQ5BGSJqP7mpDkEZImo/uakOQRkiaj+5qQ5BGSJqP7mpDkEZImo/uakOQR\n",
              "kiaj+5qQ5BGSJqP7mpDkEZImo/uakOQRkiaj+5qQ5BGSJqP7mpDkEZImo/uakOQRkiaj+5qQ\n",
              "5BGSJqP7mpDkEZImo/uakOQRkiaj+5qQ5BGSJqP7mpDkEZImo/uakOQRkiaj+5qQ5BGSJqP7\n",
              "mpDkEZImo/uakOQRkiaj+5qQ5BGSJqP7mpDkEZImo/uakOQRkiaj+5qQ5BGSJqP7mpDkzV+I\n",
              "w6YItaI8aE2RPKP7mpDkzV2IKg8PK5UpHDC6rwlJ3tyFKEO2OzaXTvsslBpTOGB0XxOSvLkL\n",
              "kYXj/fIxZBpTOGB0XxOSvLkLEcKrP4hN4YDRfU1I8rhH0mR0XxOSvC8eI+1PzSUeI71mdF8T\n",
              "krzZC7HqPGuXVypTpM/oviYkeV+8jlQ2ryNlxYbXkV4xuq8JSR5nNmgyuq8JSZ7SQoQunSlS\n",
              "YHRfE5K8Lxdim4dQ7FWnSJnRfU1I8r57Hen6jMO/T9r98lob3deEJO+rkMpQVufzqQxbjSkc\n",
              "MLqvCUneVyFloXneuwq5xhQOGN3XhCTvq5BuzyNwitALRvc1Icn7KqT1LSROERpndF8Tkrz5\n",
              "IRWb7T7sLherklOEXjC6rwlJ3vyQ7q8RhZBxitA4o/uakOTNXojjcbstiuYph/Lfjn55rY3u\n",
              "a0KSxylCmozua0KSR0iajO5rQpJHSJqM7mtCkkdImozua0KSR0iajO5rQpJHSJqM7mtCkkdI\n",
              "mozua0KSR0iajO5rQpJHSJqM7mtCkkdImozua0KSR0iajO5rQpJHSJqM7mtCkkdImozua0KS\n",
              "R0iajO5rQpJHSJqM7mtCkkdImozua0KSR0iajO5rQpJHSJqM7mtCkkdImozua0KSR0iajO5r\n",
              "QpJHSJqM7mtCkkdImozua0KSR0iajO5rQpJHSJqM7mvJCT/z0YRJIyRNPxCS4IRJIyRNSe9r\n",
              "QpqCkDQlva8JaQpC0pT0viakKQhJU9L7mpCmICRNSe9rQpqCkDQlva8JaQpCmknw9ROj+5qQ\n",
              "piCkmfzva0KagpBm8r+vCWkKQprJ/74mpCkIaSb/+5qQpiCkmfzva0KagpBm8r+vCWkKQprJ\n",
              "/74mpCkIaSb/+5qQpiCkmfzva0KagpBm8r+vCWkKQprJ/74mpCkIaSb/+5qQpiCkmfzva0Ka\n",
              "gpBm8r+vCWkKQprJ/74mpCkIaSb/+5qQpiCkmfzva0KagpAGBH94aNL7mpCmIKQBo9vM/4RJ\n",
              "I6QBo9vM/4RJI6QBo9vM/4RJI6QBo9vM/4RJI6QBo9vM/4RJI6QBo9vM/4RJI6QBo9vM/4RJ\n",
              "I6QBo9vM/4RJI6QBo9vM/4RJI6QBo9vM/4RJI6QBo9vM/4RJI6QBo9vM/4RJI6QBo9vM/4RJ\n",
              "I6QBo9vM/4RJI6QBo9vM/4RJI6QBo9vM/4RJI6QBo9vM/4RJI6QBo9vM/4RJI6QBo9vM/4RJ\n",
              "I6QBo9vM/4RJI6QBo9vM/4RJI6QBo9vM/4RJI6QBo9vM/4RJI6QBo9vM/4RJI6QBo9vM/4RJ\n",
              "I6QBo9vM/4RJI6QBo9vM/4RJI6QBo9vM/4RJI6QBo9vM/4RJI6QBo9vM/4RJI6QBo9vM/4RJ\n",
              "I6QBo9vM/4RJI6QBo9vM/4RJI6QBo9vM/4RJI6QBo9vM/4RJI6QBo9vM/4RJI6QBo9vM/4RJ\n",
              "I6QBo9vM/4RJI6QBo9vM/4RJI6QBo9vM/4RJI6QBo9vM/4RJI6QBo9vM/4RJI6QBo9ss6Qk/\n",
              "89FYRhHSwA/s64QntIqQBlLeZv4ntIqQBlLeZv4ntIqQBlLeZv4ntIqQBlLeZv4ntIqQBlLe\n",
              "Zv4ntIqQBlLeZmlPmPJz5IQ0YHWbMeHnQy2PkAb8bzP/Ey6PkAb8bzP/Ey6PkAb8bzP/Ey4v\n",
              "ZkhGz8Dyv838T7i8qCF99+la/G8z/xMuj5BmTpjyNvM/4fIIaeaEKW8z/xMuj5BmTpjyNvM/\n",
              "4fIIaeaEKW8z/xMuL4GQln5uz/828z/h8hIISfCjlp7Q6DbzP+HyCElzQqPbzP+Ey5t/XIdN\n",
              "0XxPVZSHmVMQEhMqTbi8ucdV5Z3HJ6t5UxASEypNuLy5x1WGbHdsLp32WShnTSF6IwiSOyyj\n",
              "2yzpCZe+oT80d7AsHO+XjyGbNcUP3OpMaHhCEyH1ch62/VH4kl9cgMlm7v3xzTzz8ybcIwH+\n",
              "ffEYaX9qLr19jAT4N/vubdW5i8wryUMC0vPF60hl8zpSVmzevI4E+Gf1aXkgKYQECCAkQAAh\n",
              "AQIICRBASIAAQgIEEBIggJAAAYQECCAkQAAhAQIICRBASIAAQgIEEBIggJAAAYQECIgZUqQf\n",
              "wgS8NH8zC4ZhbG7d4Rmd0WU+83uExOhuRickRmd0gdEJidEZXWB0QmJ0RhcYnZAYndEFRick\n",
              "Rmd0gdEJidEZXWB0QmJ0RhcYnZAYndEFRickRmd0gdEJidEZXWB0zv4GBBASIICQAAGEBAgg\n",
              "JEAAIQECCAkQQEiAAEICBBASIICQAAGEBAggJEAAIQECCAkQQEiAgAghdX9ceZmFrKy0hv/2\n",
              "B6OPOa5DWJ+aiwoH/xhd/th7Pyhe/Ni7oyuse9U5YPl174w+99iXD+nYOdRVcylXGv6ocIPu\n",
              "mxGzes0VDv4xusKx33Z6dtY49s7oCsd+ytrB668x8sfeGX32sccIqbhdPITseD5m4aAzfOei\n",
              "mOxyxFURSp2Df4yuceyNfX3AGsf+GF3h2Nf1mpzLsFY59s7os499+ZC2YXO7WIb95b+7xztk\n",
              "h9+KDtzYNUte1V93FQ6+M7rCsTeqrN4oGgv/GF3h2K93EfUbhWPvjD772GOEtL1dLEJ9Vy37\n",
              "BawzfOeilHU43i4qHHxndIVjbxShOuss/GN0hWPPrls9Uzn2zuizj335kIqwX18e2TWTP74S\n",
              "aAzfuSglD+dNFtb1dlE4+M7oCsdeOwa1hX+MrnDsm+s3XxuVY++MPvvYY4TUWJ21QroP37ko\n",
              "JYTi9nhd4eA7oysc+/l8u8vQCuk6usaxb+vnA7L63kLj2B+jzz725UMKYVc/31jfhWqsSW/4\n",
              "+0W50eunA9ZKXxl7o4sf+7m+y1hfJ+q+kR9d/tg3zf7enHWOvTv6zGOP9YJsVT9/qfSF8Tb8\n",
              "4OL3QvMo5qR08J3RW6LHfr49UNda+NvoLdFj39bffF2+xOh89e2M3ppx7NHObKgXIlMLqTum\n",
              "7Fa/v1E4+OEeEV6a6zErLXzWH0720WP9TWOzvxWOvTP61fTRo4bUPgFz0njJRGkzFo9bUeHg\n",
              "C+WQ7k91qSz88xNpSl9iFI5d4gvY8iFlTf7NQmya7wX2QfIJns7wnYtS2iM+1Q9GFQ6+M7rC\n",
              "sXee29VY+MfoCsfe3g01r7ApHHtn9NnHvnxIZfMNafP9tMYL7J3hOxelXB6/VPV30zuVg++M\n",
              "rnDs9Rfz6+tUKmc23EdXOPbLkNV1YIVj74w++9iXD6lqT2xqvqDk8s+TdobvziRl8zhihYN/\n",
              "jK5x7NfHAu0l+SfX76NrHPtKdd1X3697hMdI9am2+fZ+UfpVx6fhc+FX2fer2xFrHHx/dOlj\n",
              "f3znr3Hs/dGlj/1xwBrH3h99zrHz75EAAYQECCAkQAAhAQIICRBASIAAQgIEEBIggJAAAYQE\n",
              "CCAkQAAhAQIICRBASIAAQgIEEBIggJAAAYQECCAkQAAhAQIICRBASIAAQgIEEBIggJAAAYQE\n",
              "CCAkQAAhAQIICRBASIAAQgIEEBIggJDSxu1nBDdETKO/PPvpnZ0/Pv9i09O6/kVz1fcD4WuE\n",
              "FNO0/Z8/ffSx+XWn9e/i/nIgfI8ljWl0/7/8mOePXoWyCtWq/s3B3w2E77GkMX23/+s/h3NV\n",
              "3yURUmwsaUyXHV2GbNNc3uYh217feW5+z3bZpnL7mOb7uHPzm8/Dqn6Uk4XqdvvNGgiCWNCY\n",
              "QijqTV1v++ZCWJ2v+39V/2nd7v/rx1z3/7Z9ZHT5nDLk+zB/IEhiQWO6bPjqUkZ+uZupL10e\n",
              "7+zb/b8P2fF8zNr9f/uYdvdn4Xg+7+o/n9d1I4e5A0ESSxpTCIdzu6+LUD+LXYXi9sf6e7d9\n",
              "u/9vH9Pu/xAeT14fy0tKxcyBIIkljemWxvVxS/stV2erd/O5P+a5pHM83kfY5/U3dHMGgiSW\n",
              "NKZZ+3+T1S8ena4jXO58ckKKjyWNaXRvv9//+zJvHiOdm9vvZSSEtCCWNKbH3i4ej3wGD21u\n",
              "7+zu//py+/R353WkOQNBBksa02Nv7+pn187b23MEvSfbbh8TQv39XB5212ft1qHon9kwbfnA\n",
              "iT0AAADiSURBVCBIIqSYHnu7fb2neejT+WN//+fNeXW79i8O53OVPZ1rN20gSCKkmDr7vz4h\n",
              "IaxP9z+WWVgd+vv/kDf7vzmzoXn16FT2z/6eOBAEEZJlzQkK/3+E1ED4DiGZFOoHQlVRP/x5\n",
              "84FSA+E7hGTS5vFPjYwMhP8Rkk3bywOhXOJuRGwg/IuQAAGEBAggJEAAIQECCAkQQEiAAEIC\n",
              "BBASIICQAAGEBAggJEAAIQECCAkQQEiAAEICBBASIICQAAGEBAggJEAAIQECCAkQQEiAAEIC\n",
              "BBASIICQAAGEBAggJEDAH/P6AxQNPbISAAAAAElFTkSuQmCC"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 11,
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false,
        "outputExpanded": true
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "To smooth the histogram what we're going to do\n",
        "is we're going to start by keeping the heights of the histogram bar.\n",
        "And now we draw a smooth curve that goes through the top\n",
        "of these histograms bars."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can actually control the smoothness of the curve that\n",
        "defines a smooth density through an option in the function, the GG plot\n",
        "option that computes the smooth density."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Understanding the units of the y-axis**\n",
        "\n",
        "We point out that interpreting the y-axis of a smooth density plot is not straightforward.\n",
        "\n",
        "It is scaled so that the area under the density curve adds up to 1.\n",
        "So if you imagine, you form a bin with a base that is 1 unit long.\n",
        "The y-axis value tells us the proportion of values in that bin.\n",
        "\n",
        "But this is only true if the bin is of size 1.\n",
        "\n",
        "For other size intervals, the best way to determine the proportion of data\n",
        "in that interval is by computing the proportion of the total area contained\n",
        "in that interval."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "(ADD SMOOTH DENSITY PLOT EXAMPLE)"
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "We should select a degree of smoothness that we\n",
        "can defend as being representative of the underlying data.\n",
        "In the case of height, we really do have reason\n",
        "to believe that there are a proportion of people\n",
        "with similar heights should be about the same."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "Always keep in mind that while the histogram is an assumption\n",
        "free summary, the smooth density is based on assumptions and choices\n",
        "that you make as a data analyst."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Normal Distribution\n",
        "(Also known as 'the bell curve' and 'Gaussian distribution')\n",
        "\n",
        "It is one of the most famous mathematical concepts in history. A reason for this is that approximately normal distributions occur in many situations.\n",
        "\nExamples include, gambling winnings, heights, weights, blood pressure, standardized test scores, and experimental measurement error."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "(ADD FURTHER EXPLANATION)"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "index <- heights$sex == \"Male\"\n",
        "x <- heights$height[index]\n",
        "\n",
        "#average <- sum(x) / length(x)\n",
        "average <- mean(x)\n",
        "#standard_deviation <- sqrt(sum(x-average)^2 / length(x))\n",
        "standard_deviation <- sd(x)\n",
        "\nc(Average = average, SD=standard_deviation)"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<dl class=dl-horizontal>\n",
              "\t<dt>Average</dt>\n",
              "\t\t<dd>69.3147549414356</dd>\n",
              "\t<dt>SD</dt>\n",
              "\t\t<dd>3.6110236746673</dd>\n",
              "</dl>\n"
            ],
            "text/markdown": [
              "Average\n",
              ":   69.3147549414356SD\n",
              ":   3.6110236746673\n",
              "\n"
            ],
            "text/latex": [
              "\\begin{description*}\n",
              "\\item[Average] 69.3147549414356\n",
              "\\item[SD] 3.6110236746673\n",
              "\\end{description*}\n"
            ],
            "text/plain": [
              "  Average        SD \n",
              "69.314755  3.611024 "
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 43,
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can think of this as the average distance\n",
        "between the values and their average."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Standard Units\n",
        "\n",
        "For data that is approximately normal, it\n",
        "is convenient to think in terms of standard units.\n",
        "The standard unit of value tells us how many standard deviations away\n",
        "from the average this value is.\n",
        "\n",
        "z = (x-average)/standard_deviation\n",
        "\n",
        "To understand why standard units are useful,\n",
        "notice that if we convert normally distributed data into standard units, we can quickly know if for example, a person is about average height, that would mean z equals 0.\n",
        "\n",
        "A person that is tall would be z equals 2.\n",
        "\n",
        "A person that is short would be z equals negative 2.\n",
        "\n",
        "And extremely rare occurrences, say a 7 footer or something like that, would have a z bigger than 3.\n",
        "\n",
        "Note that it does not matter what the original units are.\n",
        "These rules apply to any data that is approximately normal."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "In R, we can quickly obtain standard units using the function _scale_."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "head(scale(x))"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table>\n",
              "<tbody>\n",
              "\t<tr><td> 1.5744137</td></tr>\n",
              "\t<tr><td> 0.1897648</td></tr>\n",
              "\t<tr><td>-0.3640948</td></tr>\n",
              "\t<tr><td> 1.2974839</td></tr>\n",
              "\t<tr><td>-2.3026033</td></tr>\n",
              "\t<tr><td>-0.6410246</td></tr>\n",
              "</tbody>\n",
              "</table>\n"
            ],
            "text/markdown": [
              "\n",
              "|  1.5744137 | \n",
              "|  0.1897648 | \n",
              "| -0.3640948 | \n",
              "|  1.2974839 | \n",
              "| -2.3026033 | \n",
              "| -0.6410246 | \n",
              "\n\n"
            ],
            "text/latex": [
              "\\begin{tabular}{l}\n",
              "\t  1.5744137\\\\\n",
              "\t  0.1897648\\\\\n",
              "\t -0.3640948\\\\\n",
              "\t  1.2974839\\\\\n",
              "\t -2.3026033\\\\\n",
              "\t -0.6410246\\\\\n",
              "\\end{tabular}\n"
            ],
            "text/plain": [
              "     [,1]      \n",
              "[1,]  1.5744137\n",
              "[2,]  0.1897648\n",
              "[3,] -0.3640948\n",
              "[4,]  1.2974839\n",
              "[5,] -2.3026033\n",
              "[6,] -0.6410246"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 45,
      "metadata": {
        "collapsed": false,
        "outputHidden": false,
        "inputHidden": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "To see how many men are within two standard deviations from the average, now that we were already converted to standard units, **all we have to do is count the number of z's that are less than 2 and bigger than negative 2, and then divide by the total**."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "z = (x-average)/standard_deviation\n",
        "mean(abs(z) < 2)"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "0.949507389162562"
            ],
            "text/markdown": [
              "0.949507389162562"
            ],
            "text/latex": [
              "0.949507389162562"
            ],
            "text/plain": [
              "[1] 0.9495074"
            ]
          },
          "metadata": {}
        }
      ],
      "execution_count": 47,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "If we can assume that the data is approximately normal,\n",
        "at least for this interval, we can predict the proportion\n",
        "without actually looking at the data.\n",
        "We simply know that 95% of the data for normally\n",
        "distributed data is between negative 2 and 2."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now to further confirm that in fact the approximation is a good one, we need to look at other intervals.\n",
        "And for this, we will use quantile plots."
      ],
      "metadata": {}
    }
  ],
  "metadata": {
    "kernel_info": {
      "name": "ir"
    },
    "language_info": {
      "name": "R",
      "codemirror_mode": "r",
      "pygments_lexer": "r",
      "mimetype": "text/x-r-source",
      "file_extension": ".r",
      "version": "3.5.1"
    },
    "kernelspec": {
      "name": "ir",
      "language": "R",
      "display_name": "R"
    },
    "nteract": {
      "version": "0.11.9"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}