Analyze_ab_test_results_notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyze A/B Test Results\n",
    "\n",
    "\n",
    "## Table of Contents\n",
    "- [Introduction](#intro)\n",
    "- [Part I - Probability](#probability)\n",
    "- [Part II - A/B Test](#ab_test)\n",
    "- [Part III - Regression](#regression)\n",
    "- [Part IV - Resources](#resources)\n",
    "\n",
    "\n",
    "<a id='intro'></a>\n",
    "### Introduction\n",
    "\n",
    "A/B tests are very commonly performed by data analysts and data scientists. For this project, I will be working to understand the results of an A/B test run by an e-commerce website.  This project is aimed at helping the company understand if they should implement the new page, keep the old page, or perhaps run the experiment longer to make their decision. The datasets for this project, were provided by Udacity as part of the Data Analsis Nanodegree Program.\n",
    "\n",
    "\n",
    "<a id='probability'></a>\n",
    "#### Part I - Probability\n",
    "\n",
    "To get started, let's import our libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "Bad key savefig.frameon in file C:\\Users\\zariped\\Anaconda3\\lib\\site-packages\\matplotlib\\mpl-data\\stylelib\\_classic_test.mplstyle, line 421 ('savefig.frameon : True')\n",
      "You probably need to get an updated matplotlibrc file from\n",
      "https://github.com/matplotlib/matplotlib/blob/v3.3.1/matplotlibrc.template\n",
      "or from the matplotlib source distribution\n",
      "\n",
      "Bad key verbose.level in file C:\\Users\\zariped\\Anaconda3\\lib\\site-packages\\matplotlib\\mpl-data\\stylelib\\_classic_test.mplstyle, line 472 ('verbose.level  : silent      # one of silent, helpful, debug, debug-annoying')\n",
      "You probably need to get an updated matplotlibrc file from\n",
      "https://github.com/matplotlib/matplotlib/blob/v3.3.1/matplotlibrc.template\n",
      "or from the matplotlib source distribution\n",
      "\n",
      "Bad key verbose.fileo in file C:\\Users\\zariped\\Anaconda3\\lib\\site-packages\\matplotlib\\mpl-data\\stylelib\\_classic_test.mplstyle, line 473 ('verbose.fileo  : sys.stdout  # a log filename, sys.stdout or sys.stderr')\n",
      "You probably need to get an updated matplotlibrc file from\n",
      "https://github.com/matplotlib/matplotlib/blob/v3.3.1/matplotlibrc.template\n",
      "or from the matplotlib source distribution\n",
      "In C:\\Users\\zariped\\Anaconda3\\lib\\site-packages\\matplotlib\\mpl-data\\stylelib\\_classic_test.mplstyle: \n",
      "The text.latex.preview rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.\n",
      "In C:\\Users\\zariped\\Anaconda3\\lib\\site-packages\\matplotlib\\mpl-data\\stylelib\\_classic_test.mplstyle: \n",
      "The mathtext.fallback_to_cm rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.\n",
      "In C:\\Users\\zariped\\Anaconda3\\lib\\site-packages\\matplotlib\\mpl-data\\stylelib\\_classic_test.mplstyle: Support for setting the 'mathtext.fallback_to_cm' rcParam is deprecated since 3.3 and will be removed two minor releases later; use 'mathtext.fallback : 'cm' instead.\n",
      "In C:\\Users\\zariped\\Anaconda3\\lib\\site-packages\\matplotlib\\mpl-data\\stylelib\\_classic_test.mplstyle: \n",
      "The validate_bool_maybe_none function was deprecated in Matplotlib 3.3 and will be removed two minor releases later.\n",
      "In C:\\Users\\zariped\\Anaconda3\\lib\\site-packages\\matplotlib\\mpl-data\\stylelib\\_classic_test.mplstyle: \n",
      "The savefig.jpeg_quality rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.\n",
      "In C:\\Users\\zariped\\Anaconda3\\lib\\site-packages\\matplotlib\\mpl-data\\stylelib\\_classic_test.mplstyle: \n",
      "The keymap.all_axes rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.\n",
      "In C:\\Users\\zariped\\Anaconda3\\lib\\site-packages\\matplotlib\\mpl-data\\stylelib\\_classic_test.mplstyle: \n",
      "The animation.avconv_path rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.\n",
      "In C:\\Users\\zariped\\Anaconda3\\lib\\site-packages\\matplotlib\\mpl-data\\stylelib\\_classic_test.mplstyle: \n",
      "The animation.avconv_args rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import random\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "%matplotlib inline\n",
    "#Setting the seed to assure you get the same answers.\n",
    "random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, I will read in the `ab_data.csv` data, then store it in `df` and will take a look at the top few rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0\n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0\n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('ab_data.csv')\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The number of rows in the dataset are shown in the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(294478, 5)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape\n",
    "# there are 294478 rows and 5 columns in the dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The number of unique users in the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "290584"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['user_id'].nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "The proportion of users converted:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11965919355605512"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['converted'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "The number of times the `new_page` and `treatment` don't match:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3893"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[((df['landing_page'] == 'new_page') == (df['group'] == 'treatment')) == False].shape[0]\n",
    "# There are 3893 times where `new_page` and `treatment` don't match."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see if any of the rows have missing values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isnull().any(axis=1).sum()\n",
    "# No missing values observed in any rows."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the rows where **treatment** does not match with **new_page** or **control** does not match with **old_page**, we cannot be sure if this row truly received the new or old page. Therefore, I removed the rows where the landing_page and group columns don't match bacause I was not sure about the accuracy of the data and created a new dataframe that meets the specifications stated above. The new dataset is stored in df2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "df2 = df[((df['landing_page'] == 'new_page') == (df['group'] == 'treatment')) == True]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's double Check and see if all of the correct rows were removed - this should be 0\n",
    "df2[((df2['landing_page'] == 'new_page') == (df2['group'] == 'treatment')) == False].shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "a. How many unique **user_id**s are in **df2**?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "290584"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2['user_id'].nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "There is one **user_id** repeated in **df2**.  Let's look at the row information for the repeat **user_id**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1899</th>\n",
       "      <td>773192</td>\n",
       "      <td>2017-01-09 05:37:58.781806</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2893</th>\n",
       "      <td>773192</td>\n",
       "      <td>2017-01-14 02:55:59.590927</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      user_id                   timestamp      group landing_page  converted\n",
       "1899   773192  2017-01-09 05:37:58.781806  treatment     new_page          0\n",
       "2893   773192  2017-01-14 02:55:59.590927  treatment     new_page          0"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "duplicate = df2[df2.duplicated(subset=['user_id'], keep=False)]\n",
    "duplicate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I removed the second duplicate user_id to avoid counting the same user twice. It looks like this user visited the new_page twice in different dates.then kept the dataframe as **df2**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\zariped\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:1: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n"
     ]
    }
   ],
   "source": [
    "df2.drop_duplicates(subset=['user_id'], inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Checking to make sure there are no more duplicates in df2 dataset.\n",
    "df2.duplicated().any()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The probability of an individual converting regardless of the page they receive:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11959708724499628"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2['converted'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given that an individual was in the `control` group, let's find out the probability they converted:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.1203863045004612"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.query('group == \"control\"')['converted'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given that an individual was in the `treatment` group, let's find out the probability they converted:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11880806551510564"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.query('group == \"treatment\"')['converted'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The probability that an individual received the new page:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.5000619442226688"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(df2['landing_page'] == 'new_page').mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### The probability of receiving the new page is 0.500062."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Base on the results from the above analysis, the indivuals in the treatment group had a conversion rate of 11.88% and the individuals in the control group had a conversion rate of 12.04%. Therefore, there is not sufficient evidencce to conclude that new treatment page leads to more conversions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='ab_test'></a>\n",
    "### Part II - A/B Test\n",
    "\n",
    "Because of the time stamp associated with each event, we could technically run a hypothesis test continuously as each observation was observed.  \n",
    "\n",
    "However, then the hard question is do we stop as soon as one page is considered significantly better than another or does it need to happen consistently for a certain amount of time?  How long do we run to render a decision that neither page is better than another?  \n",
    "\n",
    "These questions are the difficult parts associated with A/B tests in general.  \n",
    "\n",
    "\n",
    "For now, let's consider to make the decision just based on all the data provided.  If we want to assume that the old page is better unless the new page proves to be definitely better at a Type I error rate of 5%, we can define our null and alternative hypotheses as follows."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$H_0:$ $P_{old}$ >= $P_{new}$ \n",
    "\n",
    "$H_1:$ $P_{old}$ < $P_{new}$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We assume under the null hypothesis, $p_{new}$ and $p_{old}$ both have \"true\" success rates equal to the **converted** success rate regardless of page - that is $p_{new}$ and $p_{old}$ are equal. Furthermore,we assume they are equal to the **converted** rate in **ab_data.csv** regardless of the page. <br><br>\n",
    "\n",
    "The sample size used for each page is equal to the ones in **ab_data.csv**.  <br><br>\n",
    "\n",
    "The sampling distribution is performed for the difference in **converted** between the two pages over 10,000 iterations of calculating an estimate from the null.  <br><br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>ab_page1</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>day</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>936923</td>\n",
       "      <td>2017-01-10 15:20:49.083499</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>679687</td>\n",
       "      <td>2017-01-19 03:26:46.940749</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>719014</td>\n",
       "      <td>2017-01-17 01:48:29.539573</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>817355</td>\n",
       "      <td>2017-01-04 17:58:08.979471</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>839785</td>\n",
       "      <td>2017-01-15 18:11:06.610965</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>929503</td>\n",
       "      <td>2017-01-18 05:37:11.527370</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>834487</td>\n",
       "      <td>2017-01-21 22:37:47.774891</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>803683</td>\n",
       "      <td>2017-01-09 06:05:16.222706</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>944475</td>\n",
       "      <td>2017-01-22 01:31:09.573836</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>718956</td>\n",
       "      <td>2017-01-22 11:45:11.327945</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>644214</td>\n",
       "      <td>2017-01-22 02:05:21.719434</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>847721</td>\n",
       "      <td>2017-01-17 14:01:00.090575</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>888545</td>\n",
       "      <td>2017-01-08 06:37:26.332945</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>650559</td>\n",
       "      <td>2017-01-24 11:55:51.084801</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>935734</td>\n",
       "      <td>2017-01-17 20:33:37.428378</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    user_id                  timestamp      group landing_page  converted  \\\n",
       "0    851104 2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1    804228 2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2    661590 2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3    853541 2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "4    864975 2017-01-21 01:52:26.210827    control     old_page          1   \n",
       "5    936923 2017-01-10 15:20:49.083499    control     old_page          0   \n",
       "6    679687 2017-01-19 03:26:46.940749  treatment     new_page          1   \n",
       "7    719014 2017-01-17 01:48:29.539573    control     old_page          0   \n",
       "8    817355 2017-01-04 17:58:08.979471  treatment     new_page          1   \n",
       "9    839785 2017-01-15 18:11:06.610965  treatment     new_page          1   \n",
       "10   929503 2017-01-18 05:37:11.527370  treatment     new_page          0   \n",
       "11   834487 2017-01-21 22:37:47.774891  treatment     new_page          0   \n",
       "12   803683 2017-01-09 06:05:16.222706  treatment     new_page          0   \n",
       "13   944475 2017-01-22 01:31:09.573836  treatment     new_page          0   \n",
       "14   718956 2017-01-22 11:45:11.327945  treatment     new_page          0   \n",
       "15   644214 2017-01-22 02:05:21.719434    control     old_page          1   \n",
       "16   847721 2017-01-17 14:01:00.090575    control     old_page          0   \n",
       "17   888545 2017-01-08 06:37:26.332945  treatment     new_page          1   \n",
       "18   650559 2017-01-24 11:55:51.084801    control     old_page          0   \n",
       "19   935734 2017-01-17 20:33:37.428378    control     old_page          0   \n",
       "\n",
       "    intercept  ab_page1  ab_page  day  \n",
       "0           1         1        0   21  \n",
       "1           1         1        0   12  \n",
       "2           1         0        1   11  \n",
       "3           1         0        1    8  \n",
       "4           1         1        0   21  \n",
       "5           1         1        0   10  \n",
       "6           1         0        1   19  \n",
       "7           1         1        0   17  \n",
       "8           1         0        1    4  \n",
       "9           1         0        1   15  \n",
       "10          1         0        1   18  \n",
       "11          1         0        1   21  \n",
       "12          1         0        1    9  \n",
       "13          1         0        1   22  \n",
       "14          1         0        1   22  \n",
       "15          1         1        0   22  \n",
       "16          1         1        0   17  \n",
       "17          1         0        1    8  \n",
       "18          1         1        0   24  \n",
       "19          1         1        0   17  "
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2.head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **conversion rate** for $p_{new}$ under the null is shown below: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11959708724499628"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p_new = df2['converted'].mean()\n",
    "p_new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **conversion rate** for $p_{old}$ under the null is: <br><br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.11959708724499628"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Since we're assuming that there is no difference in conversion based on the page under the null, the conversions for each page are the same.\n",
    "p_old = df2['converted'].mean()\n",
    "p_old"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The number of individuals in the treatment group, $n_{new}$, is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "145310"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_new = df2.query('group == \"treatment\"').count()[0]\n",
    "n_new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The number of individuals in the control group, $n_{old}$, is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "145274"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_old = df2.query('group == \"control\"').count()[0]\n",
    "n_old"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We simulate $n_{new}$ transactions with a conversion rate of $p_{new}$ under the null and store these $n_{new}$ 1's and 0's in **new_page_converted**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "17532"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# new_page_converted is the number of 1's, simulating n_new transactions from a binomial distribution. \n",
    "new_page_converted = np.random.binomial(n_new,p_new)\n",
    "\n",
    "new_page_converted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We simulate $n_{old}$ transactions with a conversion rate of $p_{old}$ under the null and store these $n_{old}$ 1's and 0's in **old_page_converted**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "17387"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# old_page_converted is the number of 1's, simulating n_old transactions from a binomial distribution. \n",
    "old_page_converted = np.random.binomial(n_old,p_old)\n",
    "\n",
    "old_page_converted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we find $p_{new}$ - $p_{old}$ for our simulated values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0009682153286923317"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(new_page_converted/n_new) - (old_page_converted/n_old)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then create 10,000 $p_{new}$ - $p_{old}$ values using the same simulation process we used above. Next we stored all 10,000 values in a NumPy array called **p_diffs**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "p_diffs = []\n",
    "\n",
    "for _ in range(10000):\n",
    "    new_converted_simulation = np.random.binomial(n_new,p_new)/n_new\n",
    "    old_converted_simulation = np.random.binomial(n_old,p_old)/n_old\n",
    "    diff = new_converted_simulation - old_converted_simulation \n",
    "    p_diffs.append(diff)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-0.0015782389853555567"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p_treatment_act = df2.query('group == \"treatment\"')['converted'].mean()\n",
    "p_control_act = df2.query('group == \"control\"')['converted'].mean()\n",
    "actual_difference = p_treatment_act - p_control_act\n",
    "\n",
    "actual_difference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plotting a histogram of the **p_diffs** is shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.lines.Line2D at 0x681a9440b8>"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAmmElEQVR4nO3de5wcVZ338c+XcAvXiIExECCIEReIRgmXfdR1vCHiKrAiBBGIoFkVVt2Njwb1UVw3r8ULsCLKGlflsgoGEUUE5aIDygIxQCAERLIkQEwEQRGCCEz4PX+c01Lp9HT1TKa6e2a+79erX119uk6dU6er69d1TnWVIgIzM7NmNup0BczMrPs5WJiZWSkHCzMzK+VgYWZmpRwszMyslIOFmZmVcrAwACQdLenKipZ9jqR/24D8ayS9cDjrNNwkzZL0y07XA0DSKZL+O09PkRSSNh5g3hWS3tDeGjasx1/rPIzL3CVvO+OGc7ljlYPFGCLpVZL+R9KfJP1B0vWS9gWIiG9HxIFdUMc+Se8ppkXEVhFxb6fqZMNLUq+klVWXExH3521nbdVljQUNf23Y6CNpG+Ay4P3AAmBT4NXAU52sl4Gkcd6hWbfzkcXY8WKAiLggItZGxJMRcWVE3A7rd6PkrosPSLpH0uOSPitpd0k3SHpM0gJJmzbKW8j/ovpKSHqepMsk/V7SH/P05PzePFIAOyt3H5xVvyxJ20o6L+e/T9InJW1UrIekL+ZlL5f05oEapL6Oxe6y2q9fSXMkPSRptaR3F+Z9vqRLc1ssBHavW/ZLJF2Vj+DulnREXTlnS7pc0hPAaxvUrS+3+fW5/a+UNLFYt7r5N6Q7abqk2/MR53clbZ6XeYektxbK2ETSw5KmF7q3ZktaldtnTmHezST9R35vVZ7eTNKWwBXAjvkzXiNpx5xt0/zZPi5pqaQZheXtKOni/Lkvl/TBwnv7SVqUP4sHJZ2e09fpgsvbx715+cslHT3E9hqTHCzGjt8AayWdK+nNkp7XQp6DgH2AA4CPAvOBo4Gdgb2Bo4ZQj42AbwG7ArsATwJnAUTEJ4BfACfl7oOTGuT/MrAt8ELgNcCxwLsL7+8P3A1MBD4PfEOShlBPgBfksnYCTgC+Umi3rwB/ASYBx+cHAHmHeBXwHWAHUjt9VdJehWW/E5gHbA0MNNbxzrxuO5COBD8yxPUocwTps94NeCkwK6efB7yrMN/BwOqIWFxIey0wFTgQmFsIWJ8gbTfTgZcB+wGfjIgngDcDq/JnvFVErMp53gZcCEwALiVvF/nHwI+A20ifxeuBD0t6U873JeBLEbENKWgvqF/B/JmcCbw5IrYG/g+wuH4+G5iDxRgREY8BrwIC+Drw+/zLuKdJts9FxGMRsRS4A7gyIu6NiD+Rfh2+fAj1eCQiLo6IP0fE46Qd5mtayas0UHkkcHJEPB4RK4DTgGMKs90XEV/P3TrnknbmzdaxmWeAf42IZyLicmANsEeux9uBT0XEExFxRy6r5u+BFRHxrYjoj4hbgIuBwwvz/DAiro+IZyPiLwOU/62I+E1EPEnaAU4f4nqUOTMiVkXEH0g75Vo5/w0crNSFCamdz6/L+5ncBktIPwJqPyCOJrXdQxHxe+AzrPs5NfLLiLg8f3bnk4IMwL7A9hHxrxHxdB6/+jowM7//DPAiSRMjYk1E3DjA8p8F9pY0PiJW5+3aWuRgMYZExF0RMSsiJpOODHYE/qNJlgcL0082eL3VYOsgaQtJX8tdSI8B1wET1NoZKxNJv7DvK6TdR/q1WfO72kRE/DlPDrqe2SMR0V94/ee8rO1J430P1NWjZldgf0mP1h6knecLCvMU8w7kd4XpWtlVaFhO/sV/PfB2SRNIRwTfrstb3wa1LqUdWf9z2pHm6uuxee5C2pXUbVVsz4/z3I+AE0jdrL+W9CtJf1+/4HxEcyTwPmC1pB9LeklJfazAwWKMiohfA+eQgsaGegLYovZC0guazDsH2APYP3cb/F0tW61qTfI+TPoVuWshbRfgt4OtcPZnCvVm3Z15M78H+kndccV61DwAXBsREwqPrSLi/YV5NuRyz/XtPY4UwKpwLqkr6h3ADRFR39b1bVDrUlrF+p9T7b3BrvsDwPK69tw6Ig4GiIh7IuIoUnfd54Dv5W6ndUTETyPijaSjzV+Tjk6sRQ4WY0QecJ2j5waTdyZ1GQx0yD4YtwF75YHPzYFTmsy7Nemo5FFJ2wGfrnv/QdJ4xHpy98QCYJ6krSXtCvwLqbtkKBYD75Q0TtJBtNgdluvxfeCUfKS0J3BcYZbLgBdLOiYPCm8iaV9JfzPEetb7DelX91skbQJ8EthsmJZd7wfAK4APkcYw6v2/3AZ7kcZXvpvTLwA+KWn7PDD/KZ77nB4Eni9p2xbrsBB4TNLHJI3Pn9feyqd9S3qXpO0j4lng0ZxnnbPLJPVIelsOIk+RuhR9BtogOFiMHY+TBn9vUjoD50bSOMScprlaEBG/Af4VuBq4h4EHbCF1e40nHSXcCPyk7v0vAYcrnc10ZoP8/0T6ZX1vLuc7wDeHWPUPAW8l7WCOJu0YW3USqbvmd6QjtG/V3shjMQeS+tRX5Xk+xzDt0POY0QeA/yIdVT0BVPK/hTxecjFp8Pv7DWa5FlgGXAN8MSJqf+z8N2ARcDuwBLglp9WOai8A7s3dSk27p3JwfitpLGU5adv5L9LJB5AG55dKWkPafmY2GAfaiLStrwL+QPph8IHyFrAa+eZHZtaMpE8BL46IdxXSppB23JvUjevYKOU/5ZnZgHJX4QmUn8lko5y7ocysIUnvJQ0uXxER13W6PtZZ7oYyM7NSPrIwM7NSo3bMYuLEiTFlypS2lvnEE0+w5Zbrnd495gypHe6+Oz3vscfwV6hDvD0kboeR1QY333zzwxGx3v92KgsW+Xz760inC24MfC8iPp0HzL4LTAFWAEdExB9znpNJg2lrgQ9GxE9z+j6k0xPHA5cDH4qS/rMpU6awaNGi4V+xJvr6+ujt7W1rmd1oSO1Qm7+vb5hr0zneHhK3w8hqA0n3NUqvshvqKeB1EfEy0vnRB0k6AJgLXBMRU0nnZs/NFdyTdF76XqTzpr9auATE2cBs0gXLpub3zcysTSoLFpGsyS83yY8ADuG5i66dCxyapw8BLoyIpyJiOemPPvtJmgRsExE35KOJ8wp5zMysDSods8hHBjcDLwK+EhE3SeqJiNUAEbFa0g559p1Y99ITK3PaM6z779RaeqPyZpOOQOjp6aGvzV0aa9asaXuZ3Wgo7TD90UcBWDyK2s/bQ+J2GB1tUGmwyH/Tn56vWHmJpGYXrWt0z4Fokt6ovPmkey4wY8aMaHcf4Ujql6zSkNphwgSAUdV+3h4St8PoaIO2nDobEY8CfaSxhgdz1xL5+aE820rWvYLlZNJ1XFbm6fp0MzNrk8qCRb7a5IQ8PR54A+mywJfy3BU6jwN+mKcvBWYq3XpxN9JA9sLcZfW4pAPyHc+OLeQxM7M2qLIbahJwbh632AhYEBGXSboBWCDpBOB+0nXyiYilkhYAd5LuFXBi4Sb27+e5U2evyA8zM2uTyoJFRNxOg9tuRsQjpHvoNsozj3Sbzfr0RQzPTXrMzGwIfLkPMzMrNWov92FWZsrcH1e6/DnT+pnVoIwVp76l0nLNquAjCzMzK+VgYWZmpRwszMyslIOFmZmVcrAwM7NSDhZmZlbKwcLMzEo5WJiZWSkHCzMzK+VgYWZmpRwszMyslIOFmZmVcrAwM7NSDhZmZlbKwcLMzEo5WJiZWSkHCzMzK+VgYWZmpRwszMyslIOFmZmVcrAwM7NSDhZmZlZq405XwGysmTL3xx0re8Wpb+lY2Tay+cjCzMxKOViYmVmpyoKFpJ0l/VzSXZKWSvpQTj9F0m8lLc6Pgwt5Tpa0TNLdkt5USN9H0pL83pmSVFW9zcxsfVWOWfQDcyLiFklbAzdLuiq/d0ZEfLE4s6Q9gZnAXsCOwNWSXhwRa4GzgdnAjcDlwEHAFRXW3czMCio7soiI1RFxS55+HLgL2KlJlkOACyPiqYhYDiwD9pM0CdgmIm6IiADOAw6tqt5mZra+tpwNJWkK8HLgJuCVwEmSjgUWkY4+/kgKJDcWsq3Mac/k6fr0RuXMJh2B0NPTQ19f37CuR5k1a9a0vcxuNJR2mP7oowAsbmP7zZnWX+nye8ZXX8ZgdWL79PdidLRB5cFC0lbAxcCHI+IxSWcDnwUiP58GHA80GoeIJunrJ0bMB+YDzJgxI3p7eze4/oPR19dHu8vsRkNqhwkTANrafrMqPoV1zrR+TlvSXWenrzi6t+1l+nsxOtqg0rOhJG1CChTfjojvA0TEgxGxNiKeBb4O7JdnXwnsXMg+GViV0yc3SDczszap8mwoAd8A7oqI0wvpkwqzHQbckacvBWZK2kzSbsBUYGFErAYel3RAXuaxwA+rqreZma2vymPkVwLHAEskLc5pHweOkjSd1JW0AvhHgIhYKmkBcCfpTKoT85lQAO8HzgHGk86C8plQZmZtVFmwiIhf0ni84fImeeYB8xqkLwL2Hr7amZnZYPgf3GZmVsrBwszMSjlYmJlZKQcLMzMr5WBhZmalHCzMzKyUg4WZmZVysDAzs1IOFmZmVsrBwszMSjlYmJlZKQcLMzMr5WBhZmalHCzMzKyUg4WZmZVysDAzs1IOFmZmVsrBwszMSjlYmJlZKQcLMzMr5WBhZmalHCzMzKyUg4WZmZVysDAzs1IOFmZmVsrBwszMSlUWLCTtLOnnku6StFTSh3L6dpKuknRPfn5eIc/JkpZJulvSmwrp+0hakt87U5KqqreZma2vyiOLfmBORPwNcABwoqQ9gbnANRExFbgmvya/NxPYCzgI+KqkcXlZZwOzgan5cVCF9TYzszqVBYuIWB0Rt+Tpx4G7gJ2AQ4Bz82znAofm6UOACyPiqYhYDiwD9pM0CdgmIm6IiADOK+QxM7M2aMuYhaQpwMuBm4CeiFgNKaAAO+TZdgIeKGRbmdN2ytP16WZm1iYbV12ApK2Ai4EPR8RjTYYbGr0RTdIblTWb1F1FT08PfX19g67vhlizZk3by+xGQ2mH6Y8+CsDiNrbfnGn9lS6/Z3z1ZQxWJ7ZPfy9GRxtUGiwkbUIKFN+OiO/n5AclTYqI1bmL6aGcvhLYuZB9MrAqp09ukL6eiJgPzAeYMWNG9Pb2DteqtKSvr492l9mNhtQOEyYAtLX9Zs39caXLnzOtn9OWVP57bFBWHN3b9jL9vRgdbVDl2VACvgHcFRGnF966FDguTx8H/LCQPlPSZpJ2Iw1kL8xdVY9LOiAv89hCHjMza4Mqf/a8EjgGWCJpcU77OHAqsEDSCcD9wDsAImKppAXAnaQzqU6MiLU53/uBc4DxwBX5YWZmbVJZsIiIX9J4vAHg9QPkmQfMa5C+CNh7+GpnZmaD4X9wm5lZKQcLMzMr5WBhZmalHCzMzKyUg4WZmZVqKVhI8plIZmZjWKtHFv8paaGkD0iaUGWFzMys+7T0P4uIeJWkqcDxwCJJC4FvRcRVldbORr0p+ZIbF977CAAzK74Eh5kNTctjFhFxD/BJ4GPAa4AzJf1a0j9UVTkzM+sOrY5ZvFTSGaR7UrwOeGu+qdHrgDMqrJ+ZmXWBVi/3cRbwdeDjEfFkLTEiVkn6ZCU1MzOzrtFqsDgYeLJ2YT9JGwGbR8SfI+L8ympnZmZdodUxi6tJV3yt2SKnmZnZGNBqsNg8ItbUXuTpLaqpkpmZdZtWg8UTkl5ReyFpH+DJJvObmdko0uqYxYeBiyTVbmc6CTiykhqZmVnXafVPeb+S9BJgD9INjX4dEc9UWjMzM+sag7lT3r7AlJzn5ZKIiPMqqZWZmXWVloKFpPOB3YHFQO2+2AE4WJiZjQGtHlnMAPaMiKiyMmZm1p1aPRvqDuAFVVbEzMy6V6tHFhOBO/PVZp+qJUbE2yqplZmZdZVWg8UpVVbCzMy6W6unzl4raVdgakRcLWkLYFy1VTMzs27R6iXK3wt8D/haTtoJ+EFFdTIzsy7T6gD3icArgcfgrzdC2qGqSpmZWXdpNVg8FRFP115I2pj0PwszMxsDWg0W10r6ODBe0huBi4AfNcsg6ZuSHpJ0RyHtFEm/lbQ4Pw4uvHeypGWS7pb0pkL6PpKW5PfOlKTBraKZmW2oVoPFXOD3wBLgH4HLSffjbuYc4KAG6WdExPT8uBxA0p7ATGCvnOerkmoD6GcDs4Gp+dFomWZmVqFWz4Z6lnRb1a+3uuCIuE7SlBZnPwS4MCKeApZLWgbsJ2kFsE1E3AAg6TzgUOCKVuthZmYbrtVrQy2nwRhFRLxwCGWeJOlYYBEwJyL+SDq76sbCPCtz2jN5uj59oHrOJh2F0NPTQ19f3xCqN3Rr1qxpe5ndaDDtMGdaPwCTt4x1Xo8GPeO7b306sX36ezE62mAw14aq2Rx4B7DdEMo7G/gsKfB8FjgNOJ502fN60SS9oYiYD8wHmDFjRvT29g6hikPX19dHu8vsRoNph1lzfwzAvk+kj/q0JYO5EHJ3mzOtv+vWZ8XRvW0v09+L0dEGLY1ZRMQjhcdvI+I/gNcNtrCIeDAi1ha6tfbLb60Edi7MOhlYldMnN0g3M7M2arUb6hWFlxuRjjS2HmxhkiZFxOr88jDSBQoBLgW+I+l0YEfSQPbCiFgr6XFJBwA3AccCXx5suWZmtmFaPUY+rTDdD6wAjmiWQdIFQC8wUdJK4NNAr6TppK6kFaQzq4iIpZIWAHfm5Z8YEbX7ZryfdGbVeNLAtge3zczarNWzoV472AVHxFENkr/RZP55wLwG6YuAvQdbvpmZDZ9Wu6H+pdn7EXH68FTHzMy60WDOhtqXNLYA8FbgOuCBKiplZtWYks8+a6c50/rpbXupNtwGc/OjV0TE45Au2wFcFBHvqapiZmbWPVq93McuwNOF108DU4a9NmZm1pVaPbI4H1go6RLSmUyHAedVViszM+sqrZ4NNU/SFcCrc9K7I+LW6qplZmbdpNVuKIAtgMci4kvASkm7VVQnMzPrMq3eVvXTwMeAk3PSJsB/V1UpMzPrLq0eWRwGvA14AiAiVjGEy32YmdnI1GqweDoignzFV0lbVlclMzPrNq0GiwWSvgZMkPRe4GoGcSMkMzMb2UrPhsr3vP4u8BLgMWAP4FMRcVXFdTMzsy5RGiwiIiT9ICL2ARwgzMzGoFa7oW6UtG+lNTEzs67V6j+4Xwu8T9IK0hlRIh10vLSqipmZWfdoGiwk7RIR9wNvblN9zMysC5UdWfyAdLXZ+yRdHBFvb0OdzMysy5SNWagw/cIqK2JmZt2rLFjEANNmZjaGlHVDvUzSY6QjjPF5Gp4b4N6m0tqZmVlXaBosImJcuypiZmbdazCXKDczszHKwcLMzEo5WJiZWSkHCzMzK+VgYWZmpRwszMysVGXBQtI3JT0k6Y5C2naSrpJ0T35+XuG9kyUtk3S3pDcV0veRtCS/d2a+v4aZmbVRlUcW5wAH1aXNBa6JiKnANfk1kvYEZgJ75TxflVT7j8fZwGxgan7UL9PMzCpWWbCIiOuAP9QlHwKcm6fPBQ4tpF8YEU9FxHJgGbCfpEnANhFxQ74H+HmFPGZm1iat3s9iuPRExGqAiFgtaYecvhNwY2G+lTntmTxdn96QpNmkoxB6enro6+sbvpq3YM2aNW0vsxsNph3mTOsHYPKWsc7r0aBn/Ohan6HqGc+Y/16Mhn1Du4PFQBqNQ0ST9IYiYj4wH2DGjBnR29s7LJVrVV9fH+0usxsNph1mzf0xAPs+kT7q05Z0yya54eZM6x9V6zNUc6b1c8QY/16Mhn1Du8+GejB3LZGfH8rpK4GdC/NNBlbl9MkN0s3MrI3aHSwuBY7L08cBPyykz5S0maTdSAPZC3OX1eOSDshnQR1byGNmZm1S2TGypAuAXmCipJXAp4FTgQWSTgDuB94BEBFLJS0A7gT6gRMjYm1e1PtJZ1aNB67IDzMza6PKgkVEHDXAW68fYP55wLwG6YuAvYexamZmNkj+B7eZmZVysDAzs1IOFmZmVsrBwszMSjlYmJlZKQcLMzMr5WBhZmalHCzMzKyUg4WZmZVysDAzs1IOFmZmVsrBwszMSjlYmJlZKQcLMzMr5WBhZmalHCzMzKyUg4WZmZVysDAzs1IOFmZmVsrBwszMSm3c6QpYd5gy98fDtqw50/qZNYzLM7PO85GFmZmVcrAwM7NSDhZmZlbKwcLMzEo5WJiZWamOBAtJKyQtkbRY0qKctp2kqyTdk5+fV5j/ZEnLJN0t6U2dqLOZ2VjWySOL10bE9IiYkV/PBa6JiKnANfk1kvYEZgJ7AQcBX5U0rhMVNjMbq7qpG+oQ4Nw8fS5waCH9woh4KiKWA8uA/dpfPTOzsatTf8oL4EpJAXwtIuYDPRGxGiAiVkvaIc+7E3BjIe/KnLYeSbOB2QA9PT309fVVVP3G1qxZ0/Yyh8ucaf3Dtqye8YNf3uQtY9jr0WlDaYfRqGc8I/Z7MVxG8r6hplPB4pURsSoHhKsk/brJvGqQFo1mzEFnPsCMGTOit7d3gys6GH19fbS7zOEynP+4njOtn9OWDG7T2veJ9DEPNl83G0o7jEZzpvVzxAj9XgyXkbxvqOlIN1RErMrPDwGXkLqVHpQ0CSA/P5RnXwnsXMg+GVjVvtqamVnbg4WkLSVtXZsGDgTuAC4FjsuzHQf8ME9fCsyUtJmk3YCpwML21trMbGzrxDFyD3CJpFr534mIn0j6FbBA0gnA/cA7ACJiqaQFwJ1AP3BiRKztQL3NzMastgeLiLgXeFmD9EeA1w+QZx4wr+KqmZnZALrp1FkzM+tSPlXDzCo3nPdLGYwVp76lI+WORj6yMDOzUg4WZmZWysHCzMxKOViYmVkpBwszMyvlYGFmZqUcLMzMrJSDhZmZlXKwMDOzUg4WZmZWysHCzMxKOViYmVkpBwszMyvlYGFmZqUcLMzMrJSDhZmZlXKwMDOzUr5TXhfp1N3EzMzK+MjCzMxKOViYmVkpBwszMyvlYGFmZqUcLMzMrJTPhjKzUauTZxiuOPUtHSu7Cj6yMDOzUiMmWEg6SNLdkpZJmtvp+piZjSUjIlhIGgd8BXgzsCdwlKQ9O1srM7OxY6SMWewHLIuIewEkXQgcAtxZRWFD7eecM62fWf4Xtpmx7n6knfuGqsZKFBGVLHg4STocOCgi3pNfHwPsHxEn1c03G5idX+4B3N3WisJE4OE2l9mN3A6J2yFxO4ysNtg1IravTxwpRxZqkLZelIuI+cD86qvTmKRFETGjU+V3C7dD4nZI3A6jow1GxJgFsBLYufB6MrCqQ3UxMxtzRkqw+BUwVdJukjYFZgKXdrhOZmZjxojohoqIfkknAT8FxgHfjIilHa5WIx3rAusybofE7ZC4HUZBG4yIAW4zM+uskdINZWZmHeRgYWZmpRwsWiBpO0lXSbonPz9vgPkaXpKkLL+kXSStkfSRqtdlQ1TVDpLeKOlmSUvy8+vatU6tKrvcjJIz8/u3S3pFWd5W27ObVNQOX5D06zz/JZImtGl1hqyKdii8/xFJIWli1esxKBHhR8kD+DwwN0/PBT7XYJ5xwP8CLwQ2BW4D9mwlP3AxcBHwkU6vayfaAXg5sGOe3hv4bafXtdV1KsxzMHAF6T9BBwA3beh20W2PCtvhQGDjPP25sdoO+f2dSSfy3AdM7PS6Fh8+smjNIcC5efpc4NAG8/z1kiQR8TRQuyRJ0/ySDgXuBbrx7K56lbRDRNwaEbX/zSwFNpe02bDXfuiarVPNIcB5kdwITJA0qSRvK+3ZTSpph4i4MiL6c/4bSf+j6mZVbQ8AZwAfpcGfjjvNwaI1PRGxGiA/79Bgnp2ABwqvV+a0AfNL2hL4GPCZiuo93CpphzpvB26NiKeGrdYbrtk6lc2zoe3RTapqh6LjSb/Iu1kl7SDpbaSj6tuGu8LDYUT8z6IdJF0NvKDBW59odREN0sp+HXwGOCMi1kiNsrdfh9qhVvZepG6IA1ssq11aWaeB5hlye3ShSttB0ieAfuDbQ6pd+wx7O0jagvQd67Zt/68cLLKIeMNA70l6UNKkiFidDyUfajBbs0uSDJR/f+BwSZ8HJgDPSvpLRJy1oeszVB1qByRNBi4Bjo2I/93gFRlerVxuZqB5Nm2St5X27CZVtQOSjgP+Hnh95M77LlZFO+wO7Abcln84TgZukbRfRPxuWGs/VJ0eNBkJD+ALrDsQ+fkG82xMGnvYjecGrvYaRP5T6P4B7kragRQobwPe3ul1HGC9B1ynwjxvYd0BzYXDsV1006PCdjiIdLuB7Tu9jp1sh7r8K+iyAe6OV2AkPIDnA9cA9+Tn7XL6jsDlhfkOBn5DOtvhE2X568oYCcGiknYAPgk8ASwuPHbo9PrWrft66wS8D3hfnhbpBl3/CywBZgzHdtFtj4raYRmpH7/22f9np9ezE+1Qt/wVdFmw8OU+zMyslM+GMjOzUg4WZmZWysHCzMxKOViYmVkpBwszMyvlYDFIktZKWixpqaTbJP2LpI3yezMknZmnN5N0dZ73SEmvznkWSxrf2bVoTNLlI+GKnwCSeiVdNsg8H87/lB3xJE2R9M4h5Jsladj/9DnQcvPn9H+GsZxhXV6nyxlJHCwG78mImB4RewFvJJ0z/WmAiFgUER/M870c2CTP+13gaOCL+fWTZYXkSxy39fOJiIMj4tF2ltlmHwZGRbAApgANg4WkbroyQy/QcKc7xHoOuLxhtkHldOL7W7lO/9FjpD2ANXWvXwg8QvoTTi9wGemCcMuAP5H+ZPSPwB+A5cC3c77/C/wKuB34TE6bAtwFfBW4Fdi1ZL6vk67SeiUwPr/3IuBq0j9DbwF2H6i8Buu2ApjYbPl1858DnA38nPSv1NcA38x5zynMdyBwQ67PRcBWOf1TuU53kO5RXPvfTx/pGlELSX9eenWDsnuB60iXCLkT+E9go4HKAz4IPE36g9TPgSOA0/P8HwLuzdO7A7/M0/sA1wI3ky4bPakwz09y+i+AlxTa40zgf3J7HD5AOx+bP4fbgPNz2q6kP+bdnp93abZM0tVZa9vXPwOz8rr+CPgZsB3wg7y8G4GX5nyzgLMa1Gm/XMat+XmPwvzfz+t7D4V/mQPvzp/PtaRt5ay6ZU4Bfgf8Ntfz1Xl9Ts+fwWlN2vKtwE25PlcDPU2WtyHb4ArSNdpuIW0bL2lUTt16bQ9clfN8jXw5cRp/f79A2r6XAEcWtt3LCss7C5hVqE9t218IvCinvyMv5zbguo7s+zq98x1pD+qCRU77Y96Y/7oRNNggzuG5L/qB5J0j6ejuMuDv8sb2LHBAC/P1A9PzfAuAd+Xpm4DD8vTmpF/SDZfTYD1WFDb6hsuvm/8c0iWWRbrM8mPAtFzGzcD0vLzrgC1zno8Bn8rT2xWWdT7w1jzdB5yWpw8Grm5Qdi/wF1KwHkf68h5eUt4K8r9iSRdL/FWe/h4paO0EHAf8O7AJaae5fZ7nSOCbefoaYGqe3h/4WaE9LsrrvyfpUtT19d4LuLtQj9q/2H8EHJenjwd+0GyZrL99zSJdj6i2vC8Dn87TrwMWF+ZrFCy24bl7SrwBuLgw/73AtqTt6T7StY0mAfeTdpybAtcPsNxTKFyZIK/PZcC4krZ8Hs/9eHhPYXtotLwN2QZXAP+Upz8A/FejcurW6Szg5Dx9EOkCgbXvTfH7+3bSdjmOtH+4P7db/WdXHyxq/wg/luf2J0uAnfL0hE7s+7rpcHUkG+wlYw/Mj1vz662AqaSN6b5I178vm295RCzO6TcDUyRtTdqgLgGIiL8ASBpoOdc1qeN6yx9gvh9FREhaAjwYEUtymUtznsmkndz1+QJpm5J+4QG8VtJHSQFtO9JRzI/ye99voeyFEXFvLu8C4FWkADJQeX8VEb+TtFVus52B75AC8atz2XuQbsR0VV7OOGC1pK1I3RMXFa4UXLz3xg8i4lngTkk9Der8OuB7EfFwrscfcvrfAv+Qp88n3Rip1WXWXFVY3qtIOysi4meSni9p2yZ5twXOlTSVtPPbpPDeNRHxJwBJd5J+MU8E+iLi9zn9u8CLmyy/6KKIWFvSlpOB7+YLLG5KOiofyIZsg7DutvYPlHsVcBhARPxE0h8L7xW/v68CLoiItaSLRl4L7EsKaM1cUHg+I09fD5wjaUGhvm3lYLGBJL0QWEu6YujftJoN+PeI+FrdsqaQrpHUynzF+z2sBcYzcNBquJwSjZbfbL5n6/I8S9q+1pJ2YketUyFpc9Lh+oyIeEDSKaRfrvXLXcvA22k0eK1G5Q3gBlJXyt2kLpDjSTvtOcAuwNKI+Nu6em8DPBoR0wdYZrENGn0ealDvRorzlC2zpn7babbMep8Ffh4Rh+Xtq2+A8oufRyvr0UitnhsxcFt+mdRNeKmkXtIv/YEMaRtskL/Ztla0IZ8BpKP24njG5nXvR/10RLxP0v6kCxQuljQ9Ih5poa7DZnQNwLSZpO1JfeVnRT4+bNFPgePzLysk7SSp0Y1vWp0PgIh4DFipdPe92hlZWwx2OcPsRuCVkl6Uy95C0ot57gvycK7X4UNY9n6SdssDiUcCv2xSHsDjwNaF/NcBH8nPtwKvBZ7Kv6LvBraX9Ld5OZtI2iu38XJJ78jpkvSyQdT5GuAISc/P+bfL6f8DzMzTR+d1aaZ+Xepdl5dD3tk+nOs+kG1JffSQup7K3AT05iOWTUh96oOqZ0lbFutzXCvLa6LZNjGQZuX8kjTmVTtqH+je6dcBR0oal/cVf0cah7gP2DN/P7cFXl+X78jC8w25nN0j4qaI+BTwMOte5rwtHCwGb3w+/XUpaeDtSgZ5p7uIuJLU7XFDPnT+Hg02zFbnq3MM8EFJt5N2QC8Y4nKGRe6mmAVckOt0I2kQ81HSoOgS0kDsr4aw+BuAU0kDf8uBSwYqL88/H7hC0s/z61+QvnTX5a6CB8g76Ui3vDwc+Jyk20gDnbWzY44GTsjpS1n/lpoDioilwDzg2pz/9PzWB4F35zofQxp0b+Z2oF/p9O1/bvD+KcCMvLxTWXeH28jngX+XdD2py61sPVbnMm4gfQ9uGWDWHwGH5e/Mqxu8P1BbnkLqnvoFaefY6vIa1bXZNjGQZuV8BjhQ0i3Am4HVpOBS7xKeO5HhZ8BHI+J3EfEAaRzwdtKNnm6ty7eZpJtI20Dts/2CpCWS7iAFobbfTc9XnTUzGwSl+8OvjYj+fOR5dpNuycEuewWpa/bhsnnbzWMWZmaDswuwIHd/Pg28t8P1aQsfWZiZWSmPWZiZWSkHCzMzK+VgYWZmpRwszMyslIOFmZmV+v9Nrko/SU3AbgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plotting sampling distribution\n",
    "plt.hist(p_diffs)\n",
    "plt.title('Simulation under null hypothesis')\n",
    "plt.xlabel('Difference in mean between control and treatment groups')\n",
    "plt.ylabel('Frequency')\n",
    "plt.grid()\n",
    "plt.axvline(actual_difference, color='r', label='null mean')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's compute proportion of the **p_diffs** that are greater than the actual difference observed in **ab_data.csv**?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9075"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Converting p_diffs to array\n",
    "p_diffs = np.array(p_diffs)\n",
    "\n",
    "(p_diffs > actual_difference).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We just computed the p-value in the cell above. The p-value is the probability of observing our statistic (or one more extreme in favor of the alternative) assuming the null hypothesis is true. Based on the significance level of $\\alpha$ = 0.05, and the p-value= 0.907, we conclude that we fail to reject the null hypothesis as our p-value is much greater than $\\alpha$. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could also use a built-in to achieve similar results.  Though using the built-in might be easier to code, the above portions are a walkthrough of the ideas that are critical to correctly thinking about statistical significance. We calculate the number of conversions for each page, as well as the number of individuals who received each page.`n_old` and `n_new` refer the the number of rows associated with the old page and new pages, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "import statsmodels.api as sm\n",
    "\n",
    "convert_old = df2[df2.group == 'control'].converted.sum()\n",
    "convert_new = df2[df2.group == 'treatment'].converted.sum()\n",
    "n_old = df2[df2.group == 'control'].converted.count()\n",
    "n_new = df2[df2.group == 'treatment'].converted.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we use `stats.proportions_ztest` to compute our test statistic and p-value. [Here](https://docs.w3cub.com/statsmodels/generated/statsmodels.stats.proportion.proportions_ztest/) is a helpful link on using the built in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1.3109241984234394, 0.9050583127590245)"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from statsmodels.stats.proportion import proportions_ztest\n",
    "stat, p_value = proportions_ztest([convert_old, convert_new], [n_old, n_new], alternative='smaller')\n",
    "stat, p_value "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The stat (z_score), measured in standard deviation, represents how far from the mean our data point is. In this case, our data point is 1.311 standard deviation from the mean. The p-value of 0.905 is quite close to the p-value we computed in the previous section, 0.907 and based on both p-values, we can confidently say that we fail to reject the null hypothesis and that there is not enough evidence to conclude that the new page leads to more conversions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='regression'></a>\n",
    "### Part III - A regression approach\n",
    "\n",
    "In this final part, we will see that the result achieved in the A/B test in Part II above can also be achieved by performing regression.<br><br> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Since each row is either a conversion or no conversion, we can use the logistic regression approach in this case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal in this section is to use **statsmodels** to fit the regression model that we have specified to see if there is a significant difference in conversion based on which page a customer receives. However, we first need to create in df2 a column for the intercept, and create a dummy variable column for which page each user received. We also need to add an **intercept** column, as well as an **ab_page** column, which is 1 when an individual receives the **treatment** and 0 if **control**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\zariped\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:1: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n"
     ]
    }
   ],
   "source": [
    "df2['intercept'] = 1\n",
    "df2[['old_page', 'ab_page']] = pd.get_dummies(df2['group'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use **statsmodels** to instantiate our regression model on the two columns created above, then we fit the model to predict whether or not an individual converts. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366118\n",
      "         Iterations 6\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Logit Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>       <td>converted</td>    <th>  No. Observations:  </th>   <td>290584</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>                 <td>Logit</td>      <th>  Df Residuals:      </th>   <td>290582</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>                 <td>MLE</td>       <th>  Df Model:          </th>   <td>     1</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>            <td>Tue, 08 Dec 2020</td> <th>  Pseudo R-squ.:     </th>  <td>8.077e-06</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>                <td>10:43:32</td>     <th>  Log-Likelihood:    </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>converged:</th>             <td>True</td>       <th>  LL-Null:           </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Covariance Type:</th>     <td>nonrobust</td>    <th>  LLR p-value:       </th>   <td>0.1899</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>         <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th> <td>   -1.9888</td> <td>    0.008</td> <td> -246.669</td> <td> 0.000</td> <td>   -2.005</td> <td>   -1.973</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page</th>   <td>   -0.0150</td> <td>    0.011</td> <td>   -1.311</td> <td> 0.190</td> <td>   -0.037</td> <td>    0.007</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                           Logit Regression Results                           \n",
       "==============================================================================\n",
       "Dep. Variable:              converted   No. Observations:               290584\n",
       "Model:                          Logit   Df Residuals:                   290582\n",
       "Method:                           MLE   Df Model:                            1\n",
       "Date:                Tue, 08 Dec 2020   Pseudo R-squ.:               8.077e-06\n",
       "Time:                        10:43:32   Log-Likelihood:            -1.0639e+05\n",
       "converged:                       True   LL-Null:                   -1.0639e+05\n",
       "Covariance Type:            nonrobust   LLR p-value:                    0.1899\n",
       "==============================================================================\n",
       "                 coef    std err          z      P>|z|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "intercept     -1.9888      0.008   -246.669      0.000      -2.005      -1.973\n",
       "ab_page       -0.0150      0.011     -1.311      0.190      -0.037       0.007\n",
       "==============================================================================\n",
       "\"\"\""
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log_mod = sm.Logit(df2['converted'], df2[['intercept', 'ab_page']]) \n",
    "results = log_mod.fit()\n",
    "results.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The summary of my model is shown in the cell above. Now let's analyze the results of our model summary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0151020136964721"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1/np.exp(results.params[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We can see that for every 1 point increase in ab_page, conversions are 1.015 times more likely, holding all other variables constant. The p-value of 0.190 for the ab_page also confirms that the landing page (ab_page) is not statistically significant."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We can see that our p-value from the logistic regression is different from the p-value that we calculated in part II. The reason being that the logistic regression is a two- tailed test and we are exploring only two possible outcomes.\n",
    "\n",
    "#### The null and alternative hypotheses for our regression model are:\n",
    "$H_0:$ $P_{old}$ = $P_{new}$ \n",
    "\n",
    "$H_1:$ $P_{old}$ $\\neq$ $P_{new}$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's consider other things that might influence whether or not an individual converts.  There are two main advantages to analyzing data using a multiple regression model. The first is the ability to determine the relative influence of one or more predictor variables to the criterion value. The second advantage is the ability to identify outliers, or anomalies. Accordingly, multiple regression model is used to obtain findings with greater accuracy.\n",
    "Any disadvantage of using a multiple regression model usually comes down to the data being used. Two examples of this are using incomplete data and falsely concluding that a correlation is a causation.\n",
    "More details on this is provided [Here](https://sciencing.com/advantages-disadvantages-multiple-regression-model-12070171.html) ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### One of the most important things to consider in our regression model is the duration of the experiment. The duration of our experiment is as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\zariped\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:2: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  \n"
     ]
    }
   ],
   "source": [
    "# First we need to Change column type — from Object to DateTime.\n",
    "df2['timestamp'] = pd.to_datetime(df2['timestamp'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\zariped\\Anaconda3\\lib\\site-packages\\ipykernel_launcher.py:2: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "user_id                  int64\n",
       "timestamp       datetime64[ns]\n",
       "group                   object\n",
       "landing_page            object\n",
       "converted                int64\n",
       "intercept                int64\n",
       "ab_page1                 uint8\n",
       "ab_page                  uint8\n",
       "day                      int64\n",
       "dtype: object"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from datetime import datetime as dt\n",
    "df2['day'] = df2['timestamp'].dt.day\n",
    "df2.dtypes\n",
    "# We made a day column by calculating the delta between the timestamps in days."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "22"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now we can calculate the duration of the experiment in days.\n",
    "duration = np.array(df2['day'].sort_values(ascending=True))\n",
    "td = duration[-1] - duration[0]\n",
    "td"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### As shown in the cell above, the duration of our experiment is 22 days which is quite short. It is recommended to increase the duration of this experiment to 1-2 months depending on the computational costs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Another factor to consider in our regression model would be the country a user lives in. We will need to read in the **countries.csv** dataset and merge together our datasets on the appropriate rows.  [Here](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.join.html) are the docs for joining tables. Next we will investigate to see whether country had an impact on conversion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>country</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>834778</td>\n",
       "      <td>UK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>928468</td>\n",
       "      <td>US</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>822059</td>\n",
       "      <td>UK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>711597</td>\n",
       "      <td>UK</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>710616</td>\n",
       "      <td>UK</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id country\n",
       "0   834778      UK\n",
       "1   928468      US\n",
       "2   822059      UK\n",
       "3   711597      UK\n",
       "4   710616      UK"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "country_df = pd.read_csv('countries.csv')\n",
    "country_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['US', 'CA', 'UK'], dtype=object)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We will join the 2 dataframes based on the user_id column that exists in both and will then get a list of the unique countries in our new dataframe.\n",
    "country_df2 = df2.join(country_df.set_index('user_id'), on='user_id')\n",
    "country_df2.country.unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Now we need to create dummy variables for these country columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>ab_page1</th>\n",
       "      <th>ab_page</th>\n",
       "      <th>day</th>\n",
       "      <th>country</th>\n",
       "      <th>CA</th>\n",
       "      <th>UK</th>\n",
       "      <th>US</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>21</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>21</td>\n",
       "      <td>US</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                  timestamp      group landing_page  converted  \\\n",
       "0   851104 2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1   804228 2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2   661590 2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3   853541 2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "4   864975 2017-01-21 01:52:26.210827    control     old_page          1   \n",
       "\n",
       "   intercept  ab_page1  ab_page  day country  CA  UK  US  \n",
       "0          1         1        0   21      US   0   0   1  \n",
       "1          1         1        0   12      US   0   0   1  \n",
       "2          1         0        1   11      US   0   0   1  \n",
       "3          1         0        1    8      US   0   0   1  \n",
       "4          1         1        0   21      US   0   0   1  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "country_df2[['CA','UK', 'US']] = pd.get_dummies(country_df2['country'])\n",
    "country_df2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "US    203619\n",
       "UK     72466\n",
       "CA     14499\n",
       "Name: country, dtype: int64"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "country_df2['country'].value_counts()\n",
    "# The most common country is the US."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366113\n",
      "         Iterations 6\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Logit Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>       <td>converted</td>    <th>  No. Observations:  </th>   <td>290584</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>                 <td>Logit</td>      <th>  Df Residuals:      </th>   <td>290580</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>                 <td>MLE</td>       <th>  Df Model:          </th>   <td>     3</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>            <td>Wed, 09 Dec 2020</td> <th>  Pseudo R-squ.:     </th>  <td>2.323e-05</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>                <td>09:53:55</td>     <th>  Log-Likelihood:    </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>converged:</th>             <td>True</td>       <th>  LL-Null:           </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Covariance Type:</th>     <td>nonrobust</td>    <th>  LLR p-value:       </th>   <td>0.1760</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>         <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th> <td>   -2.0300</td> <td>    0.027</td> <td>  -76.249</td> <td> 0.000</td> <td>   -2.082</td> <td>   -1.978</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page</th>   <td>   -0.0149</td> <td>    0.011</td> <td>   -1.307</td> <td> 0.191</td> <td>   -0.037</td> <td>    0.007</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>UK</th>        <td>    0.0506</td> <td>    0.028</td> <td>    1.784</td> <td> 0.074</td> <td>   -0.005</td> <td>    0.106</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>US</th>        <td>    0.0408</td> <td>    0.027</td> <td>    1.516</td> <td> 0.130</td> <td>   -0.012</td> <td>    0.093</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                           Logit Regression Results                           \n",
       "==============================================================================\n",
       "Dep. Variable:              converted   No. Observations:               290584\n",
       "Model:                          Logit   Df Residuals:                   290580\n",
       "Method:                           MLE   Df Model:                            3\n",
       "Date:                Wed, 09 Dec 2020   Pseudo R-squ.:               2.323e-05\n",
       "Time:                        09:53:55   Log-Likelihood:            -1.0639e+05\n",
       "converged:                       True   LL-Null:                   -1.0639e+05\n",
       "Covariance Type:            nonrobust   LLR p-value:                    0.1760\n",
       "==============================================================================\n",
       "                 coef    std err          z      P>|z|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "intercept     -2.0300      0.027    -76.249      0.000      -2.082      -1.978\n",
       "ab_page       -0.0149      0.011     -1.307      0.191      -0.037       0.007\n",
       "UK             0.0506      0.028      1.784      0.074      -0.005       0.106\n",
       "US             0.0408      0.027      1.516      0.130      -0.012       0.093\n",
       "==============================================================================\n",
       "\"\"\""
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "country_df2['intercept'] = 1\n",
    "\n",
    "log_mod = sm.Logit(country_df2['converted'], country_df2[['intercept', 'ab_page', 'UK', 'US']]) \n",
    "results = log_mod.fit()\n",
    "results.summary()\n",
    "# We chose Canada as the baseline country."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1.015055597022635, 1.051944407708369, 1.0415989517670823)"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1/np.exp(results.params[1]), np.exp(results.params[2]), np.exp(results.params[3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Based on the above results, the users from the UK are 1.05 times more likely to convert as compared to the users from Canada. Similary, the users from the US are 1.04 times more likely to convert as compared to the users from Canada. The p-values for UK and US are 0.074 and 0.13 and not statistically significant. We can see that for every 1 point increase in ab_page, conversions are 1.015 times more likely, holding all other variables constant. The p-value of 0.190 for the ab_page also confirms that the landing page (ab_page) is not statistically significant. Therefore, it appears that country does not have an impact on conversion."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Though we have now looked at the individual factors of country and page on conversion, we would now like to look at an interaction between page and country to see if there are significant effects on conversion.  To investigate that, we will create the necessary additional columns, and fit the new model.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "country_df2['US_ab_page'] = country_df2['US'] * country_df2['ab_page']\n",
    "country_df2['UK_ab_page'] = country_df2['UK'] * country_df2['ab_page']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366109\n",
      "         Iterations 6\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<caption>Logit Regression Results</caption>\n",
       "<tr>\n",
       "  <th>Dep. Variable:</th>       <td>converted</td>    <th>  No. Observations:  </th>   <td>290584</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Model:</th>                 <td>Logit</td>      <th>  Df Residuals:      </th>   <td>290578</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Method:</th>                 <td>MLE</td>       <th>  Df Model:          </th>   <td>     5</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Date:</th>            <td>Tue, 08 Dec 2020</td> <th>  Pseudo R-squ.:     </th>  <td>3.482e-05</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Time:</th>                <td>10:43:48</td>     <th>  Log-Likelihood:    </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>converged:</th>             <td>True</td>       <th>  LL-Null:           </th> <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>Covariance Type:</th>     <td>nonrobust</td>    <th>  LLR p-value:       </th>   <td>0.1920</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "       <td></td>         <th>coef</th>     <th>std err</th>      <th>z</th>      <th>P>|z|</th>  <th>[0.025</th>    <th>0.975]</th>  \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th>  <td>   -2.0040</td> <td>    0.036</td> <td>  -55.008</td> <td> 0.000</td> <td>   -2.075</td> <td>   -1.933</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page</th>    <td>   -0.0674</td> <td>    0.052</td> <td>   -1.297</td> <td> 0.195</td> <td>   -0.169</td> <td>    0.034</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>US</th>         <td>    0.0175</td> <td>    0.038</td> <td>    0.465</td> <td> 0.642</td> <td>   -0.056</td> <td>    0.091</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>UK</th>         <td>    0.0118</td> <td>    0.040</td> <td>    0.296</td> <td> 0.767</td> <td>   -0.066</td> <td>    0.090</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>US_ab_page</th> <td>    0.0469</td> <td>    0.054</td> <td>    0.872</td> <td> 0.383</td> <td>   -0.059</td> <td>    0.152</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>UK_ab_page</th> <td>    0.0783</td> <td>    0.057</td> <td>    1.378</td> <td> 0.168</td> <td>   -0.033</td> <td>    0.190</td>\n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary.Summary'>\n",
       "\"\"\"\n",
       "                           Logit Regression Results                           \n",
       "==============================================================================\n",
       "Dep. Variable:              converted   No. Observations:               290584\n",
       "Model:                          Logit   Df Residuals:                   290578\n",
       "Method:                           MLE   Df Model:                            5\n",
       "Date:                Tue, 08 Dec 2020   Pseudo R-squ.:               3.482e-05\n",
       "Time:                        10:43:48   Log-Likelihood:            -1.0639e+05\n",
       "converged:                       True   LL-Null:                   -1.0639e+05\n",
       "Covariance Type:            nonrobust   LLR p-value:                    0.1920\n",
       "==============================================================================\n",
       "                 coef    std err          z      P>|z|      [0.025      0.975]\n",
       "------------------------------------------------------------------------------\n",
       "intercept     -2.0040      0.036    -55.008      0.000      -2.075      -1.933\n",
       "ab_page       -0.0674      0.052     -1.297      0.195      -0.169       0.034\n",
       "US             0.0175      0.038      0.465      0.642      -0.056       0.091\n",
       "UK             0.0118      0.040      0.296      0.767      -0.066       0.090\n",
       "US_ab_page     0.0469      0.054      0.872      0.383      -0.059       0.152\n",
       "UK_ab_page     0.0783      0.057      1.378      0.168      -0.033       0.190\n",
       "==============================================================================\n",
       "\"\"\""
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "country_df2['intercept'] = 1\n",
    "\n",
    "log_mod = sm.Logit(country_df2['converted'], country_df2[['intercept', 'ab_page', 'US', 'UK', 'US_ab_page', 'UK_ab_page']]) \n",
    "results = log_mod.fit()\n",
    "results.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1.0697232819209153,\n",
       " 1.0176540221507617,\n",
       " 1.011869894648401,\n",
       " 1.048017202119183,\n",
       " 1.0814470441230692)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1/np.exp(-0.0674), np.exp( 0.0175), np.exp(0.0118), np.exp(0.0469), np.exp(0.0783)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results in the cell above show that for every unit of change in conversion of users from Canada, there will be a 1.048 change in conversion for the users from the US and a 1.081 change in the conversion of the users from the UK which show that there doesn't seem to be interaction between the conversion and country the user is from. Moreover, it can be seen that none of the variables have significant p-values. Therefore, we fail to reject the null hypothesis and we can confidently conclude that there is not enough evidence to suggest that there is an interaction between the page received or the country the user is from on the conversion rate. \n",
    "\n",
    "#### Based on the data available and the results of the analyses in this project, we conclude that there is not sufficient evidence to predict that the new page results in more conversions than the old page."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='resources'></a>\n",
    "### Part IV - Resources\n",
    "\n",
    "#### Mastering Dates and Timestamps in Pandas (and Python in general)\n",
    "https://towardsdatascience.com/mastering-dates-and-timestamps-in-pandas-and-python-in-general-5b8c6edcc50c\n",
    "#### Extracting days from a numpy.timedelta64 value and calculating the duration\n",
    "https://stackoverflow.com/questions/18215317/extracting-days-from-a-numpy-timedelta64-value\n",
    "#### Join columns of another DataFrame\n",
    "https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.join.html\n",
    "#### The Advantages & Disadvantages of a Multiple Regression Model\n",
    "https://sciencing.com/advantages-disadvantages-multiple-regression-model-12070171.html"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}