Analyze_ab_test_results_FINAL.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analyze A/B Test Results \n",
    "\n",
    "This project will assure you have mastered the subjects covered in the statistics lessons. We have organized the current notebook into the following sections: \n",
    "\n",
    "- [Introduction](#intro)\n",
    "- [Part I - Probability](#probability)\n",
    "- [Part II - A/B Test](#ab_test)\n",
    "- [Part III - Regression](#regression)\n",
    "- [Final Check](#finalcheck)\n",
    "- [Submission](#submission)\n",
    "\n",
    "Specific programming tasks are marked with a **ToDo** tag. \n",
    "\n",
    "<a id='intro'></a>\n",
    "## Introduction\n",
    "\n",
    "A/B tests are very commonly performed by data analysts and data scientists. For this project, you will be working to understand the results of an A/B test run by an e-commerce website.  Your goal is to work through this notebook to help the company understand if they should:\n",
    "- Implement the new webpage, \n",
    "- Keep the old webpage, or \n",
    "- Perhaps run the experiment longer to make their decision.\n",
    "\n",
    "Each **ToDo** task below has an associated quiz present in the classroom.  Though the classroom quizzes are **not necessary** to complete the project, they help ensure you are on the right track as you work through the project, and you can feel more confident in your final submission meeting the [rubric](https://review.udacity.com/#!/rubrics/1214/view) specification. \n",
    "\n",
    "\n",
    "\n",
    "<a id='probability'></a>\n",
    "## Part I - Probability\n",
    "\n",
    "To get started, let's import our libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import random\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "#We are setting the seed to assure you get the same answers on quizzes as we set up\n",
    "random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ToDo 1.1\n",
    "Now, read in the `ab_data.csv` data. Store it in `df`. Below is the description of the data, there are a total of 5 columns:\n",
    "\n",
    "<center>\n",
    "\n",
    "|Data columns|Purpose|Valid values|\n",
    "| ------------- |:-------------| -----:|\n",
    "|user_id|Unique ID|Int64 values|\n",
    "|timestamp|Time stamp when the user visited the webpage|-|\n",
    "|group|In the current A/B experiment, the users are categorized into two broad groups. <br>The `control` group users are expected to be served with `old_page`; and `treatment` group users are matched with the `new_page`. <br>However, **some inaccurate rows** are present in the initial data, such as a `control` group user is matched with a `new_page`. |`['control', 'treatment']`|\n",
    "|landing_page|It denotes whether the user visited the old or new webpage.|`['old_page', 'new_page']`|\n",
    "|converted|It denotes whether the user decided to pay for the company's product. Here, `1` means yes, the user bought the product.|`[0, 1]`|\n",
    "</center>\n",
    "Use your dataframe to answer the questions in Quiz 1 of the classroom.\n",
    "\n",
    "\n",
    "\n",
    "**a.** Read in the dataset from the `ab_data.csv` file and take a look at the top few rows here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0\n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0\n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0\n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0\n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('ab_data.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b.** Use the cell below to find the number of rows in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The number of rows in the dataset = 294478\n"
     ]
    }
   ],
   "source": [
    "print('The number of rows in the dataset =', df.shape[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c.** The number of unique users in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The number of users in the dataset =  290584\n"
     ]
    }
   ],
   "source": [
    "print('The number of users in the dataset = ', df['user_id'].nunique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**d.** The proportion of users converted."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The proportion of users converted = 11.9659 %\n"
     ]
    }
   ],
   "source": [
    "proportion = (len(df.query('converted == 1')) / len(df.index)) * 100\n",
    "print('The proportion of users converted =',\"{:.4f}\".format(proportion),'%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**e.** The number of times when the \"group\" is `treatment` but \"landing_page\" is not a `new_page`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The number of times when the group is not inline with the landing page = 3893\n"
     ]
    }
   ],
   "source": [
    "group_page_not_inline = len(df.query(\"group == 'treatment' & landing_page != 'new_page'\")) + len(df.query(\"group == 'control' & landing_page != 'old_page'\"))\n",
    "print('The number of times when the group is not inline with the landing page =', group_page_not_inline)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**f.** Do any of the rows have missing values?\n",
    "No."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 294478 entries, 0 to 294477\n",
      "Data columns (total 5 columns):\n",
      "user_id         294478 non-null int64\n",
      "timestamp       294478 non-null object\n",
      "group           294478 non-null object\n",
      "landing_page    294478 non-null object\n",
      "converted       294478 non-null int64\n",
      "dtypes: int64(2), object(3)\n",
      "memory usage: 11.2+ MB\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "user_id         0\n",
       "timestamp       0\n",
       "group           0\n",
       "landing_page    0\n",
       "converted       0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 141,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.info()\n",
    "df.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ToDo 1.2  \n",
    "In a particular row, the **group** and **landing_page** columns should have either of the following acceptable values:\n",
    "\n",
    "|user_id| timestamp|group|landing_page|converted|\n",
    "|---|---|---|---|---|\n",
    "|XXXX|XXXX|`control`| `old_page`|X |\n",
    "|XXXX|XXXX|`treatment`|`new_page`|X |\n",
    "\n",
    "\n",
    "It means, the `control` group users should match with `old_page`; and `treatment` group users should matched with the `new_page`. \n",
    "\n",
    "However, for the rows where `treatment` does not match with `new_page` or `control` does not match with `old_page`, we cannot be sure if such rows truly received the new or old wepage.  \n",
    "\n",
    "\n",
    "Use **Quiz 2** in the classroom to figure out how should we handle the rows where the group and landing_page columns don't match?\n",
    "\n",
    "**a.** Now use the answer to the quiz to create a new dataset that meets the specifications from the quiz.  Store your new dataframe in **df2**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove the inaccurate rows, and store the result in a new dataframe \n",
    "df2 = df[(df['group'] == 'control') & (df['landing_page'] == 'old_page') | (df['group'] == 'treatment') & (df['landing_page'] == 'new_page')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 143,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Double Check all of the incorrect rows were removed from df2 \n",
    "# Output of the statement below should be 0\n",
    "df2[((df2['group'] == 'treatment') == (df2['landing_page'] == 'new_page')) == False].shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ToDo 1.3  \n",
    "Use **df2** and the cells below to answer questions for **Quiz 3** in the classroom."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**a.** How many unique **user_id**s are in **df2**?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The number of unique user_ids = 290584\n"
     ]
    }
   ],
   "source": [
    "print('The number of unique user_ids =', df2['user_id'].nunique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**b.** There is one **user_id** repeated in **df2**.  What is it?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The user_id repeated is [773192]\n"
     ]
    }
   ],
   "source": [
    "duplicated_id = df2[df2.duplicated(['user_id'])]\n",
    "print('The user_id repeated is', duplicated_id['user_id'].unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c.** Display the rows for the duplicate **user_id**? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1899</th>\n",
       "      <td>773192</td>\n",
       "      <td>2017-01-09 05:37:58.781806</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2893</th>\n",
       "      <td>773192</td>\n",
       "      <td>2017-01-14 02:55:59.590927</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      user_id                   timestamp      group landing_page  converted\n",
       "1899   773192  2017-01-09 05:37:58.781806  treatment     new_page          0\n",
       "2893   773192  2017-01-14 02:55:59.590927  treatment     new_page          0"
      ]
     },
     "execution_count": 146,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2[df2.duplicated(['user_id'], keep= False)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**d.** Remove **one** of the rows with a duplicate **user_id**, from the **df2** dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/pandas/core/frame.py:3697: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  errors=errors)\n"
     ]
    }
   ],
   "source": [
    "# Remove one of the rows with a duplicate user_id..\n",
    "# Hint: The dataframe.drop_duplicates() may not work in this case because the rows with duplicate user_id are not entirely identical. \n",
    "df2.drop(labels = 1899, axis = 0, inplace = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2893</th>\n",
       "      <td>773192</td>\n",
       "      <td>2017-01-14 02:55:59.590927</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      user_id                   timestamp      group landing_page  converted\n",
       "2893   773192  2017-01-14 02:55:59.590927  treatment     new_page          0"
      ]
     },
     "execution_count": 148,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check again if the row with a duplicate user_id is deleted or not\n",
    "df2.loc[df2['user_id'] == 773192]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ToDo 1.4  \n",
    "Use **df2** in the cells below to answer the quiz questions related to **Quiz 4** in the classroom.\n",
    "\n",
    "**a.** What is the probability of an individual converting regardless of the page they receive?<br><br>\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The probability of an individual converting regardless of page received = 11.9597 %\n"
     ]
    }
   ],
   "source": [
    "p_population = df2['converted'].mean() \n",
    "p_population_percentage = p_population * 100\n",
    "print('The probability of an individual converting regardless of page received =', \"{:.4f}\".format(p_population_percentage),'%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b.** Given that an individual was in the `control` group, what is the probability they converted?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The probabilty of individuals in the control group who converted 12.0386 %\n"
     ]
    }
   ],
   "source": [
    "control_converted = (df2.query(\"group == 'control'\")['converted']).mean() \n",
    "control_converted_percentage = control_converted * 100\n",
    "print('The probabilty of individuals in the control group who converted', \"{:.4f}\".format(control_converted_percentage),'%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c.** Given that an individual was in the `treatment` group, what is the probability they converted?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The probability of individuals in the treatment group who converted 11.8808 %\n"
     ]
    }
   ],
   "source": [
    "treatment_converted = (df2.query(\"group == 'treatment'\")['converted']).mean() \n",
    "treatment_converted_percentage = treatment_converted * 100\n",
    "print('The probability of individuals in the treatment group who converted', \"{:.4f}\".format(treatment_converted_percentage),'%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">**Tip**: The probabilities you've computed in the points (b). and (c). above can also be treated as conversion rate. \n",
    "Calculate the actual difference  (`obs_diff`) between the conversion rates for the two groups. You will need that later.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The actual difference between conversion rates for the two groups = -0.0016 %\n"
     ]
    }
   ],
   "source": [
    "# Calculate the actual difference (obs_diff) between the conversion rates for the two groups.\n",
    "obs_diff = treatment_converted - control_converted\n",
    "print('The actual difference between conversion rates for the two groups =',\"{:.4f}\".format(obs_diff),'%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**d.** What is the probability that an individual received the new page?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The probability than an individual received the new page is 49.3449 %\n"
     ]
    }
   ],
   "source": [
    "p_new_page = df2.query(\"landing_page == 'new_page'\").count()[0] / df.shape[0]\n",
    "p_new_page_percentage = p_new_page * 100\n",
    "print('The probability than an individual received the new page is', \"{:.4f}\".format(p_new_page_percentage),'%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**e.** Consider your results from parts (a) through (d) above, and explain below whether the new `treatment` group users lead to more conversions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "#### 1. Based on the above results, the treatment group has conversion rate of 11.88% and the control group has a conversion rate of 12.04%; this suggests that the new treatment group users don't lead to more conversions.\n",
    "\n",
    "\n",
    "#### 2. Although the results show that the conversion rate of control group is greater than that of the treatment group, it is not statistically sufficient to conclude that the old page drives more conversions because the difference of -0.1578% is not significant.\n",
    "\n",
    "#### 3. Aside from the difference being small, there are some factors that also need to be considered such as change aversion, novelty effect, practical signifiance, etc.\n",
    "\n",
    "#### 4. So at this point of analysis, we can not confidently say that the old page control group users lead to more conversions. A hypothesis test is recommended to help achieve an informed decision.\n",
    "\n",
    "#### ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='ab_test'></a>\n",
    "## Part II - A/B Test\n",
    "\n",
    "Since a timestamp is associated with each event, you could run a hypothesis test continuously as long as you observe the events. \n",
    "\n",
    "However, then the hard questions would be: \n",
    "- Do you stop as soon as one page is considered significantly better than another or does it need to happen consistently for a certain amount of time?  \n",
    "- How long do you run to render a decision that neither page is better than another?  \n",
    "\n",
    "These questions are the difficult parts associated with A/B tests in general.  \n",
    "\n",
    "\n",
    "### ToDo 2.1\n",
    "For now, consider you need to make the decision just based on all the data provided.  \n",
    "\n",
    "> Recall that you just calculated that the \"converted\" probability (or rate) for the old page is *slightly* higher than that of the new page (ToDo 1.4.c). \n",
    "\n",
    "If you want to assume that the old page is better unless the new page proves to be definitely better at a Type I error rate of 5%, what should be your null and alternative hypotheses (**$H_0$** and **$H_1$**)?  \n",
    "\n",
    "You can state your hypothesis in terms of words or in terms of **$p_{old}$** and **$p_{new}$**, which are the \"converted\" probability (or rate) for the old and new pages respectively."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **$H_0$: $p_{new}$ ≤ $p_{old}$**\n",
    "#### **$H_1$: $p_{new}$ > $p_{old}$**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ToDo 2.2 - Null Hypothesis $H_0$ Testing\n",
    "Under the null hypothesis $H_0$, assume that $p_{new}$ and $p_{old}$ are equal. Furthermore, assume that $p_{new}$ and $p_{old}$ both are equal to the **converted** success rate in the `df2` data regardless of the page. So, our assumption is: <br><br>\n",
    "<center>\n",
    "$p_{new}$ = $p_{old}$ = $p_{population}$\n",
    "</center>\n",
    "\n",
    "In this section, you will: \n",
    "\n",
    "- Simulate (bootstrap) sample data set for both groups, and compute the  \"converted\" probability $p$ for those samples. \n",
    "\n",
    "\n",
    "- Use a sample size for each group equal to the ones in the `df2` data.\n",
    "\n",
    "\n",
    "- Compute the difference in the \"converted\" probability for the two samples above. \n",
    "\n",
    "\n",
    "- Perform the sampling distribution for the \"difference in the converted probability\" between the two simulated-samples over 10,000 iterations; and calculate an estimate. \n",
    "\n",
    "\n",
    "\n",
    "Use the cells below to provide the necessary parts of this simulation.  You can use **Quiz 5** in the classroom to make sure you are on the right track."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**a.** What is the **conversion rate** for $p_{new}$ under the null hypothesis? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The conversion rate for p_new under the null hypothesis is 11.9597 %\n"
     ]
    }
   ],
   "source": [
    "p_new = df2['converted'].mean() \n",
    "p_new_percentage = p_new * 100\n",
    "print('The conversion rate for p_new under the null hypothesis is', \"{:.4f}\".format(p_new_percentage),'%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b.** What is the **conversion rate** for $p_{old}$ under the null hypothesis? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The conversion rate for p_old under the null hypothesis is 11.9597 %\n"
     ]
    }
   ],
   "source": [
    "p_old = df2['converted'].mean()\n",
    "p_old_percentage = p_old * 100\n",
    "print('The conversion rate for p_old under the null hypothesis is', \"{:.4f}\".format(p_old_percentage),'%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c.** What is $n_{new}$, the number of individuals in the treatment group? <br><br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The number of individuals in the treatment group = 145310\n"
     ]
    }
   ],
   "source": [
    "n_new = df2['landing_page'].value_counts()[0]\n",
    "print('The number of individuals in the treatment group =', n_new)\n",
    "# 145310"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**d.** What is $n_{old}$, the number of individuals in the control group?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The number of individuals in the control group = 145274\n"
     ]
    }
   ],
   "source": [
    "n_old = df2['landing_page'].value_counts()[1]\n",
    "print('The number of individuals in the control group =', n_old)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**e. Simulate Sample for the `treatment` Group**<br> \n",
    "Simulate $n_{new}$ transactions with a conversion rate of $p_{new}$ under the null hypothesis.  <br><br>\n",
    "*Hint*: Use `numpy.random.choice()` method to randomly generate $n_{new}$ number of values. <br>\n",
    "Store these $n_{new}$ 1's and 0's in the `new_page_converted` numpy array.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Simulate a Sample for the treatment Group\n",
    "new_page_converted = np.random.choice([0, 1], size = n_new, p = [1-p_new, p_new])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**f. Simulate Sample for the `control` Group** <br>\n",
    "Simulate $n_{old}$ transactions with a conversion rate of $p_{old}$ under the null hypothesis. <br> Store these $n_{old}$ 1's and 0's in the `old_page_converted` numpy array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Simulate a Sample for the control Group\n",
    "old_page_converted = np.random.choice([0, 1], size = n_old, p = [1-p_old, p_old])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**g.** Find the difference in the \"converted\" probability $(p{'}_{new}$ - $p{'}_{old})$ for your simulated samples from the parts (e) and (f) above. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The difference in the converted probability for simulated samples = 0.1058 %\n"
     ]
    }
   ],
   "source": [
    "diff_mean = (new_page_converted.mean() - old_page_converted.mean())\n",
    "diff_mean_percentage = diff_mean * 100\n",
    "print('The difference in the converted probability for simulated samples =', \"{:.4f}\".format(diff_mean_percentage),'%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "**h. Sampling distribution** <br>\n",
    "Re-create `new_page_converted` and `old_page_converted` and find the $(p{'}_{new}$ - $p{'}_{old})$ value 10,000 times using the same simulation process you used in parts (a) through (g) above. \n",
    "\n",
    "<br>\n",
    "Store all  $(p{'}_{new}$ - $p{'}_{old})$  values in a NumPy array called `p_diffs`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sampling distribution \n",
    "p_diffs = []\n",
    "for _ in range (10000):\n",
    "    new_page_converted = np.random.choice([0, 1], size = n_new, p = [1-p_new, p_new])\n",
    "    old_page_converted = np.random.choice([0, 1], size = n_old, p = [1-p_old, p_old])\n",
    "    p_diffs.append(new_page_converted.mean() - old_page_converted.mean())\n",
    "p_diffs = np.array(p_diffs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**i. Histogram**<br> \n",
    "Plot a histogram of the **p_diffs**.  Does this plot look like what you expected?  Use the matching problem in the classroom to assure you fully understand what was computed here.<br><br>\n",
    "\n",
    "Also, use `plt.axvline()` method to mark the actual difference observed  in the `df2` data (recall `obs_diff`), in the chart.  \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0,0.5,'Simulations')"
      ]
     },
     "execution_count": 163,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEWCAYAAABxMXBSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAHBFJREFUeJzt3XuUHWWZ7/HvjxBA5ZJAAoYk2BGDCh4N2EA4cFxBIITLEBwVwzgaEVdcZ4CDZxxmBcHhohlBEZAj4okSDIgJjOiQwSiEcFFcQNJgSAghQwvBNMlJGhPCRWAmzHP+qLdJpbN7966kq/fu7t9nrVq79lNvVT27VuiHqrfqLUUEZmZmtdqp3gmYmVnf4sJhZmaFuHCYmVkhLhxmZlaIC4eZmRXiwmFmZoW4cNiAIukBSV/qpX39T0nrJL0qaZ+S9jFBUlvu+3JJE9K8JN0kaaOkRb2Vk/V/O9c7AbOeJmkVsB/wFvAaMB84LyJeLbCNJuA5YHBEbN6OHAYDVwPjI+KJoutvr4g4JPf1GOAEYFREvFavnKz/8RmH9Vd/FRG7A4cBhwMX9/L+9wN2A5b38n7z3gOsiojX0vdGyMn6ARcO69ci4gXg18CHOi+TtJOkiyU9L2m9pJsl7ZUW/zZ9vpQu6xxVYf1dJV0raU2ark2xg4CVufXvq7Buk6SQNC2tu1bSV7v7PZLeIekn6fLTU2RFMb98laTjJZ0N/Bg4KuU/p3NO6VLWNem3b5K0VNI2x8msM1+qsn5N0mjgZOAXFRZ/IU3HAuuBm4HvA58DPkZ2qWpIlUtVFwHjgXFAAHcCF0fE1yUdUsP6pH2PBd4L3CfpiYi4t0r7S4AD0/QusqK4jYi4UdJbwJci4hjY6vLbkIjYLOnE9DsPAjYBHwBeqrJvM8BnHNZ//aukl4CHgAeBf67Q5rPA1RHxbOr/uBCYIqnW/6H6LHB5RKyPiHbgMrKiU8RlEfFaRCwDbgLO7Kb9GcCMiNgQEauB6wruL+8/gT3ICoYiYkVErN2B7dkA4cJh/dXpETEkIt4TEX8XEa9XaLM/8Hzu+/NkZ+H71biPSuvvXzDP1QXX37/COtslIu4jO8O6HlgnaaakPbd3ezZwuHDYQLaGrAO5wwHAZmAd2aWn7Vl/TcEcRhdcf22FdbZbRFwXER8FDiG7ZHXBjmzPBgYXDhvI5gD/W9IYSbuTXc66LfVJtAP/Rdb3UG39iyUNlzQM+CfgpwVz+Lqkd6Y+kbOA27ppfztwoaShkkYB5xXc39skHS7pyHSb7mvAG2S3MJtV5cJhA9ks4BayO6ieI/vDeR5ARPwFmAH8XtJLksZXWP+bQAuwFFgGPJ5iRTwItAILgasi4p5u2l9GdnnqOeCelP/22hP4EbAxbfPPwFU7sD0bIOQXOZn1vh19wNCsnnzGYWZmhbhwmDUYSb9OD+11nr5W79zMwJeqzMysIJ9xmJlZIf1yyJFhw4ZFU1NTvdOweliZhmN6//vrm4dZH/TYY4+9GBHDu2vXLwtHU1MTLS0t9U7D6mHChOzzgQfqmYVZnySpppEIfKnKzMwKceEwM7NCXDjMzKwQFw4zMyvEhcPMzApx4TAzs0JcOMzMrBAXDjMzK8SFw8zMCumXT46bNbKm6b+qy35XXXFKXfZr/Y/POMzMrBAXDjMzK6S0wiFpN0mLJD0habmky1J8jKRHJT0j6TZJu6T4rul7a1relNvWhSm+UtKJZeVsZmbdK/OM403g4xHxEWAcMEnSeOBK4JqIGAtsBM5O7c8GNkbE+4BrUjskHQxMAQ4BJgE/kDSoxLzNzKyK0gpHZF5NXwenKYCPAz9P8dnA6Wl+cvpOWn6cJKX43Ih4MyKeA1qBI8rK28zMqiu1j0PSIElLgPXAAuCPwEsRsTk1aQNGpvmRwGqAtHwTsE8+XmGd/L6mSWqR1NLe3l7GzzEzM0ouHBHxVkSMA0aRnSV8sFKz9KkulnUV77yvmRHRHBHNw4d3+wIrMzPbTr1yV1VEvAQ8AIwHhkjqeH5kFLAmzbcBowHS8r2ADfl4hXXMzKyXlXlX1XBJQ9L8O4DjgRXA/cCnUrOpwJ1pfl76Tlp+X0REik9Jd12NAcYCi8rK28zMqivzyfERwOx0B9ROwO0RcZekp4C5kr4J/AG4MbW/EbhFUivZmcYUgIhYLul24ClgM3BORLxVYt5mZlZFaYUjIpYCh1aIP0uFu6Ii4g3g011sawYwo6dzNDOz4vzkuJmZFeLCYWZmhbhwmJlZIS4cZmZWiAuHmZkV4sJhZmaFuHCYmVkhLhxmZlaIC4eZmRXiwmFmZoW4cJiZWSEuHGZmVogLh5mZFeLCYWZmhbhwmJlZIS4cZmZWiAuHmZkV4sJhZmaFuHCYmVkhLhxmZlaIC4eZmRXiwmFmZoW4cJiZWSEuHGZmVkhphUPSaEn3S1ohabmk81P8UkkvSFqSppNz61woqVXSSkkn5uKTUqxV0vSycjYzs+7tXOK2NwNfjYjHJe0BPCZpQVp2TURclW8s6WBgCnAIsD9wr6SD0uLrgROANmCxpHkR8VSJuZuZWRdKKxwRsRZYm+ZfkbQCGFlllcnA3Ih4E3hOUitwRFrWGhHPAkiam9q6cJiZ1UGv9HFIagIOBR5NoXMlLZU0S9LQFBsJrM6t1pZiXcU772OapBZJLe3t7T38C8zMrEPphUPS7sAdwFci4mXgBuBAYBzZGcl3O5pWWD2qxLcORMyMiOaIaB4+fHiP5G5mZtsqs48DSYPJisatEfELgIhYl1v+I+Cu9LUNGJ1bfRSwJs13FTczs15W5l1VAm4EVkTE1bn4iFyzTwBPpvl5wBRJu0oaA4wFFgGLgbGSxkjahawDfV5ZeZuZWXVlnnEcDXwOWCZpSYp9DThT0jiyy02rgC8DRMRySbeTdXpvBs6JiLcAJJ0L3A0MAmZFxPIS8zYzsyrKvKvqISr3T8yvss4MYEaF+Pxq65mZWe/xk+NmZlaIC4eZmRXiwmFmZoW4cJiZWSEuHGZmVogLh5mZFeLCYWZmhbhwmJlZIS4cZmZWSKmDHJo1qqbpv6p3CmZ9ls84zMysEBcOMzMrxIXDzMwKceEwM7NCXDjMzKwQFw4zMyvEhcPMzApx4TAzs0JcOMzMrBAXDjMzK8RDjpgNEPUcZmXVFafUbd/W83zGYWZmhbhwmJlZIS4cZmZWiAuHmZkVUlrhkDRa0v2SVkhaLun8FN9b0gJJz6TPoSkuSddJapW0VNJhuW1NTe2fkTS1rJzNzKx7ZZ5xbAa+GhEfBMYD50g6GJgOLIyIscDC9B3gJGBsmqYBN0BWaIBLgCOBI4BLOoqNmZn1vpoKh6RvS9pT0mBJCyW9KOlvq60TEWsj4vE0/wqwAhgJTAZmp2azgdPT/GTg5sg8AgyRNAI4EVgQERsiYiOwAJhU8HeamVkPqfWMY2JEvAycCrQBBwEX1LoTSU3AocCjwH4RsRay4gLsm5qNBFbnVmtLsa7infcxTVKLpJb29vZaUzMzs4JqLRyD0+fJwJyI2FDrDiTtDtwBfCUVny6bVohFlfjWgYiZEdEcEc3Dhw+vNT0zMyuo1sLxb5KeBpqBhZKGA290t5KkwWRF49aI+EUKr0uXoEif61O8DRidW30UsKZK3MzM6qCmwhER04GjgOaI+E/gNbI+iS5JEnAjsCIirs4tmgd03Bk1FbgzF/98urtqPLApXcq6G5goaWjqFJ+YYmZmVgdFxqr6INAkKb/OzVXaHw18DlgmaUmKfQ24Arhd0tnAn4BPp2XzyS6FtQJ/Ac4CiIgNkr4BLE7tLi9yqczMzHpWTYVD0i3AgcAS4K0UDqoUjoh4iMr9EwDHVWgfwDldbGsWMKuWXM3MrFy1nnE0AwenP+5mZjaA1do5/iTw7jITMTOzvqHWM45hwFOSFgFvdgQj4rRSsjIzs4ZVa+G4tMwkzMys76ipcETEg5L2Aw5PoUURsb7aOmZm1j/VOlbVGcAisltnzwAelfSpMhMzM7PGVOulqouAwzvOMtKT4/cCPy8rMTMza0y13lW1U6dLU38usK6ZmfUjtZ5x/EbS3cCc9P0zZE96m5nZAFNr5/gFkj5JNoyIgJkR8ctSMzMzs4ZU81hVEXEH2Ui3ZmY2gFUtHJIeiohjJL3C1u/AENnwUnuWmp2ZmTWcqoUjIo5Jn3v0TjpmZtboan2O45ZaYmZm1v/VekvtIfkv6Z0cH+35dMzMrNFVLRySLkz9Gx+W9HKaXgHWseXNfWZmNoBULRwR8a3Uv/GdiNgzTXtExD4RcWEv5WhmZg2k1uc4Lkzv+x4L7JaL/7asxMzMrDHV+urYLwHnA6PIXh87HngY+Hh5qZmZWSOqtXP8fLIh1Z+PiGOBQ4H20rIyM7OGVWvheCMi3gCQtGtEPA28v7y0zMysUdU65EibpCHAvwILJG0E1pSXlpmZNapaO8c/kWYvlXQ/sBfwm9KyMjOzhtXdWFV7VwgvS5+7Axt6PCMzM2to3fVxPAa0pM/OU0u1FSXNkrRe0pO52KWSXpC0JE0n55ZdKKlV0kpJJ+bik1KsVdL04j/RzMx6UneDHI7ZgW3/BPg+cHOn+DURcVU+IOlgYArZ0Cb7A/dKOigtvh44AWgDFkuaFxFP7UBeZma2A2p9juNjleLVHgCMiN9Kaqoxj8nA3Ih4E3hOUitwRFrWGhHPpjzmprYuHGZmdVLrXVUX5OZ3I/uj/hjb9wDguZI+T3ap66sRsREYCTySa9OWYgCrO8WPrLRRSdOAaQAHHHDAdqRlZma1qOk5joj4q9x0AvAhsoEOi7oBOBAYB6wFvpviqrTbKvFKOc6MiOaIaB4+fPh2pGZmZrWo+dWxnbSRFY9CIuLtYiPpR8Bdue2NzjUdxZbnRLqKm5lZHdTax/F/2PJ/+juRnTE8UXRnkkZExNr09RNAxx1X84CfSbqarHN8LLCI7IxjrKQxwAtkHeh/U3S/ZmbWc2o948jfersZmBMRv6+2gqQ5wARgmKQ24BJggqRxZEVoFfBlgIhYLul2sk7vzcA5EfFW2s65wN3AIGBWRCyvMWczMytBrU+Ozy664Yg4s0L4xirtZwAzKsTnA/OL7t/MzMpR6zvHT5X0B0kbOt4CKOnlspMzM7PGU+ulqmuBvwaWRUTFu5rMzGxgqHVY9dXAky4aZmZW6xnHPwLzJT0IvNkRjIirS8nKzMwaVq2FYwbwKtlT47uUl46ZmTW6WgvH3hExsdRMzMysT6i1j+NeSS4cZmZWc+E4B/iNpNd9O66Z2cBW6wOAe5SdiJmZ9Q3dvTr2AxHxtKTDKi2PiMfLScvMzBpVd2ccf0/2jovv5mL5Zzm2530cZmbWh3XXx/FjSe+OiGMj4liy18G+Sjaq7afKTs7MzBpPd4Xjh8B/wNuvj/0WMBvYBMwsNzUzM2tE3V2qGhQRG9L8Z4CZEXEHcIekJeWmZmZmjai7M45BkjqKy3HAfbll2/v2QDMz68O6++M/B3hQ0ovA68DvACS9j+xylZmZDTBVC0dEzJC0EBgB3JMbHXcn4LyykzMzs8bT7eWmiHikQuzfy0nHzMwaXa1DjpiZmQEuHGZmVpALh5mZFeLCYWZmhbhwmJlZIS4cZmZWiAuHmZkVUtqwIZJmAacC6yPiQym2N3Ab0ASsAs6IiI2SBHwPOBn4C/CFjnd9SJoKXJw2+82ImF1Wztb7mqb/qke3N/fZPwMwpYe3a2ZblHnG8RNgUqfYdGBhRIwFFqbvACcBY9M0DbgB3i40lwBHAkcAl0gaWmLOZmbWjdIKR0T8FtjQKTyZbFh20ufpufjNkXkEGCJpBHAisCAiNkTERmAB2xYjMzPrRb3dx7FfRKwFSJ/7pvhIYHWuXVuKdRXfhqRpkloktbS3t/d44mZmlmmUznFViEWV+LbBiJkR0RwRzcOHD+/R5MzMbIveLhzr0iUo0uf6FG8DRufajQLWVImbmVmd9HbhmAdMTfNTgTtz8c8rMx7YlC5l3Q1MlDQ0dYpPTDEzM6uTMm/HnQNMAIZJaiO7O+oK4HZJZwN/Aj6dms8nuxW3lex23LMAImKDpG8Ai1O7y3OvsjUzszoorXBExJldLDquQtsAzuliO7OAWT2YmpmZ7YBG6Rw3M7M+woXDzMwKKe1SlZlZh54eWqZWq644pS777e98xmFmZoW4cJiZWSEuHGZmVogLh5mZFeLCYWZmhbhwmJlZIS4cZmZWiAuHmZkV4sJhZmaFuHCYmVkhLhxmZlaIC4eZmRXiwmFmZoW4cJiZWSEuHGZmVogLh5mZFeLCYWZmhbhwmJlZIS4cZmZWiAuHmZkV4sJhZmaF1KVwSFolaZmkJZJaUmxvSQskPZM+h6a4JF0nqVXSUkmH1SNnMzPL1POM49iIGBcRzen7dGBhRIwFFqbvACcBY9M0Dbih1zM1M7O3NdKlqsnA7DQ/Gzg9F785Mo8AQySNqEeCZmZWv8IRwD2SHpM0LcX2i4i1AOlz3xQfCazOrduWYluRNE1Si6SW9vb2ElM3MxvYdq7Tfo+OiDWS9gUWSHq6SltViMU2gYiZwEyA5ubmbZabmVnPqMsZR0SsSZ/rgV8CRwDrOi5Bpc/1qXkbMDq3+ihgTe9la2Zmeb1eOCS9S9IeHfPAROBJYB4wNTWbCtyZ5ucBn093V40HNnVc0jIzs95Xj0tV+wG/lNSx/59FxG8kLQZul3Q28Cfg06n9fOBkoBX4C3BW76dsZmYder1wRMSzwEcqxP8MHFchHsA5vZCamZnVoJFuxzUzsz7AhcPMzApx4TAzs0JcOMzMrBAXDjMzK8SFw8zMCnHhMDOzQlw4zMysEBcOMzMrpF6j41qDaZr+q3qnYGZ9hM84zMysEBcOMzMrxIXDzMwKcR+HmfVb9ey7W3XFKXXbd9l8xmFmZoW4cJiZWSEuHGZmVogLh5mZFeLCYWZmhbhwmJlZIS4cZmZWiAuHmZkV4sJhZmaFuHCYmVkhLhxmZlZInxmrStIk4HvAIODHEXFFnVMyM+tSvcbJ6o0xsvpE4ZA0CLgeOAFoAxZLmhcRT9U3s57llymZWV/QJwoHcATQGhHPAkiaC0wGSikc/gNuZta1vlI4RgKrc9/bgCPzDSRNA6alr69KWtlLufW0YcCL9U6iQRQ+Fkd1zFx5ao8nU2f+d5Hxcdii4rHQlTu0zffU0qivFA5ViMVWXyJmAjN7J53ySGqJiOZ659EIfCy28LHI+DhsUc9j0VfuqmoDRue+jwLW1CkXM7MBra8UjsXAWEljJO0CTAHm1TknM7MBqU9cqoqIzZLOBe4mux13VkQsr3NaZenzl9t6kI/FFj4WGR+HLep2LBQR3bcyMzNL+sqlKjMzaxAuHGZmVogLRy+RtLekBZKeSZ9Du2g3NbV5RtLUXPyjkpZJapV0nSR1Wu8fJIWkYWX/lh1V1rGQ9B1JT0taKumXkob01m8qQtIkSStT/tMrLN9V0m1p+aOSmnLLLkzxlZJOrHWbjaqnj4Wk0ZLul7RC0nJJ5/fer9l+ZfybSMsGSfqDpLt6NOGI8NQLE/BtYHqanw5cWaHN3sCz6XNomh+ali0ie75NwK+Bk3LrjSa7ceB5YFi9f2u9jgUwEdg5zV9Zabv1nshu7vgj8F5gF+AJ4OBObf4O+GGanwLcluYPTu13Bcak7QyqZZuNOJV0LEYAh6U2ewD/3ujHoozjkFvv74GfAXf1ZM4+4+g9k4HZaX42cHqFNicCCyJiQ0RsBBYAkySNAPaMiIcj+9dwc6f1rwH+kU4PRTawUo5FRNwTEZvT+o+QPe/TaN4ePici/gPoGD4nL398fg4cl86qJgNzI+LNiHgOaE3bq2WbjajHj0VErI2IxwEi4hVgBdnIE42sjH8TSBoFnAL8uKcTduHoPftFxFqA9LlvhTaVhlYZmaa2CnEknQa8EBFPlJF0SUo5Fp18kexspNF09bsqtkmFcBOwT5V1a9lmIyrjWLwtXc45FHi0B3MuQ1nH4Vqy/6H8r55OuE88x9FXSLoXeHeFRRfVuokKsegqLumdadsTa9x+r+ntY9Fp3xcBm4Fba9xXb+o2/yptuopX+h/AvnD2WcaxyFaSdgfuAL4SES9vd4a9o8ePg6RTgfUR8ZikCTuY3zZcOHpQRBzf1TJJ6ySNiIi16XLL+grN2oAJue+jgAdSfFSn+BrgQLLrmk+k/uFRwOOSjoiI/7cDP2WH1eFYdGx7KnAqcFy6lNVoahk+p6NNm6Sdgb2ADd2s2xeH5CnlWEgaTFY0bo2IX5STeo8q4zicBpwm6WRgN2BPST+NiL/tkYzr3TE0UCbgO2zdIfztCm32Bp4j6wwemub3TssWA+PZ0iF8coX1V9E3OsdLORbAJLKh9ofX+zdW+e07k3X0j2FLR+ghndqcw9Ydoben+UPYuiP0WbKO1W632YhTScdCZP1e19b799XzOHRadwI93Dle94M2UCay65ELgWfSZ8cfwWayNxp2tPsiWQdXK3BWLt4MPEl218T3SU/9d9pHXykcpRyL1G41sCRNP6z3b+3i959MdrfPH4GLUuxy4LQ0vxvwL+n3LALem1v3orTeSra+s26bbfaFqaePBXAM2SWcpbl/B9v8T1ajTWX8m8gt7/HC4SFHzMysEN9VZWZmhbhwmJlZIS4cZmZWiAuHmZkV4sJhZmaFuHBYv5NGCb4l931nSe09PkJo9RyaJD2Z5pslXZfmd5V0r6Qlkj4j6X+kUVyXSHpHb+VntiP85Lj1R68BH5L0joh4HTgBeKFeyUREC9CSvh4KDI6IcQCSfghcFRE31bKtNLCdIqLHxx8yq5XPOKy/+jXZyKAAZwJzOhZIepekWZIWp3cVTE7xJkm/k/R4mv57ik+Q9ICkn6f3fdza8Q6QPGXvCXlC0sNkT/qSW/8uSfsCPwXGpTOMLwNnAP8k6dbU9oKU11JJl+XyWiHpB8DjwGhJEyU9nPL8lzQ2E5JWSbosxZdJ+kCK7y7pphRbKumTKV5xO2bVuHBYfzUXmCJpN+DDbD1C6kXAfRFxOHAs8B1J7yIbM+uEiDgM+AxwXW6dQ4GvkL3/4L3A0RX2eRPwvyLiqEoJRcR64EvA7yJiXET8X2AecEFEfFbSRGAs2bDY44CPSvpYWv39wM0RcSjZGdXFwPEp1xay9y50eDHFbwD+IcW+DmyKiP8WER8G7lP20q9q2zGryJeqrF+KiKVpWO0zgfmdFk8kGwCu44/qbsABZIPDfV/SOOAt4KDcOosiog1A0hKgCXioY6GkvYAhEfFgCt0CnFQw7Ylp+kP6vjtZIfkT8HxEPJLi48kK2O/Tic8uwMO57XQM7PcY8Ndp/niyMY4AiIiNaQTVatsxq8iFw/qzecBVZGP17JOLC/hkRKzMN5Z0KbAO+AjZ2fgbucVv5ubfYtv/dsSOD2Uu4FvpTCSfVxPZWUa+3YKIOLOL7XTkms+zUn7dbcesIl+qsv5sFnB5RCzrFL8bOK+jn0LSoSm+F7A2dTx/jmy01ZpExEvAJknHpNBntyPfu4Ev5vorRqZ+kc4eAY6W9L7U7p2SDqrQLu8e4NyOL8re87492zFz4bD+KyLaIuJ7FRZ9AxgMLE23zH4jxX8ATJX0CNllqtcqrFvNWcD1qXP89e3I9x6y90M/LGkZ2StC96jQrh34AjBH0lKyAvCBbjb/TWCopCclPQEcu53bMfPouGZmVozPOMzMrBAXDjMzK8SFw8zMCnHhMDOzQlw4zMysEBcOMzMrxIXDzMwK+f9lUGu1u8BoJgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f986df9ac18>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(p_diffs);\n",
    "plt.axvline(x = obs_diff, color = 'red')\n",
    "plt.title('Plot of p_diffs')\n",
    "plt.xlabel('Mean difference')\n",
    "plt.ylabel('Simulations')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**j.** What proportion of the **p_diffs** are greater than the actual difference observed in the `df2` data?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The proportion of the p_diffs that are greater than the actual difference observed in the df2 data = 0.9100\n"
     ]
    }
   ],
   "source": [
    "p_value = (p_diffs > obs_diff).mean()\n",
    "print('The proportion of the p_diffs that are greater than the actual difference observed in the df2 data =', \"{:.4f}\".format(p_value))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**k.** Please explain in words what you have just computed in part **j** above.  \n",
    " - What is this value called in scientific studies?  \n",
    " - What does this value signify in terms of whether or not there is a difference between the new and old pages? *Hint*: Compare the value above with the \"Type I error rate (0.05)\". "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### This value is called the p-value. p-values are numbers between 0 and 1 which measure the confidence we have in the H1 or alternative hypothesis which says the new page drives more conversions than the old page.\n",
    "#### The closer the p-value to zero, the more confidence we have in the H1 or alternative hypothesis.\n",
    "#### However, we need to set how small does the p-value have to be before we can sufficiently say that the new page drives more conversions than the old page. This means we need to have a threshold. In practice, a commonly used threshold is Type 1 error rate of 0.05. So if our p-value falls below the type 1 error rate, it suggests that we reject the H0 or null hypothesis. That is, we choose the H1 or alternative hypothesis. Consequently if our p-value is above the threshold error rate, then we fail to reject the H0 or null hypothesis. \n",
    "#### Given that no study or experiment is perfect and that random things can aways happen, the setting of threshold of 0.05 means we would only get a 5 False Positive out of 100 experiments. Getting a small p-value when the null is true is also called False Positive or the type 1 error.\n",
    "#### So, our p-value is 0.9100, which is greater than 0.05. This means that we are not confident to reject the null so we stay with the null hypothesis as our decision. That is, we are confident to say that the new page does not drive more conversions than the old page.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "**l. Using Built-in Methods for Hypothesis Testing**<br>\n",
    "We could also use a built-in to achieve similar results.  Though using the built-in might be easier to code, the above portions are a walkthrough of the ideas that are critical to correctly thinking about statistical significance. \n",
    "\n",
    "Fill in the statements below to calculate the:\n",
    "- `convert_old`: number of conversions with the old_page\n",
    "- `convert_new`: number of conversions with the new_page\n",
    "- `n_old`: number of individuals who were shown the old_page\n",
    "- `n_new`: number of individuals who were shown the new_page\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The number of conversions with old page and new page respectively: 17489 & 17264\n",
      "The number of individuals who were shown the old page and new page respectively: 145274 & 145310\n"
     ]
    }
   ],
   "source": [
    "import statsmodels.api as sm\n",
    "\n",
    "# number of conversions with the old_page\n",
    "convert_old = df2.query(\"landing_page == 'old_page' & converted == '1'\").count()[0]\n",
    "\n",
    "# number of conversions with the new_page\n",
    "convert_new = df2.query(\"landing_page == 'new_page' & converted == '1'\").count()[0]\n",
    "\n",
    "# number of individuals who were shown the old_page\n",
    "n_old = df2['landing_page'].value_counts()[1]\n",
    "\n",
    "# number of individuals who received new_page\n",
    "n_new = df2['landing_page'].value_counts()[0]\n",
    "\n",
    "print('The number of conversions with old page and new page respectively:', convert_old, '&', convert_new) \n",
    "print('The number of individuals who were shown the old page and new page respectively:', n_old, '&', n_new)    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**m.** Now use `sm.stats.proportions_ztest()` to compute your test statistic and p-value.  [Here](https://www.statsmodels.org/stable/generated/statsmodels.stats.proportion.proportions_ztest.html) is a helpful link on using the built in.\n",
    "\n",
    "The syntax is: \n",
    "```bash\n",
    "proportions_ztest(count_array, nobs_array, alternative='larger')\n",
    "```\n",
    "where, \n",
    "- `count_array` = represents the number of \"converted\" for each group\n",
    "- `nobs_array` = represents the total number of observations (rows) in each group\n",
    "- `alternative` = choose one of the values from `[‘two-sided’, ‘smaller’, ‘larger’]` depending upon two-tailed, left-tailed, or right-tailed respectively. \n",
    ">**Hint**: <br>\n",
    "It's a two-tailed if you defined $H_1$ as $(p_{new} = p_{old})$. <br>\n",
    "It's a left-tailed if you defined $H_1$ as $(p_{new} < p_{old})$. <br>\n",
    "It's a right-tailed if you defined $H_1$ as $(p_{new} > p_{old})$. \n",
    "\n",
    "The built-in function above will return the z_score, p_value. \n",
    "\n",
    "---\n",
    "### About the two-sample z-test\n",
    "Recall that you have plotted a distribution `p_diffs` representing the\n",
    "difference in the \"converted\" probability  $(p{'}_{new}-p{'}_{old})$  for your two simulated samples 10,000 times. \n",
    "\n",
    "Another way for comparing the mean of two independent and normal distribution is a **two-sample z-test**. You can perform the Z-test to calculate the Z_score, as shown in the equation below:\n",
    "\n",
    "$$\n",
    "Z_{score} = \\frac{ (p{'}_{new}-p{'}_{old}) - (p_{new}  -  p_{old})}{ \\sqrt{ \\frac{\\sigma^{2}_{new} }{n_{new}} + \\frac{\\sigma^{2}_{old} }{n_{old}}  } }\n",
    "$$\n",
    "\n",
    "where,\n",
    "- $p{'}$ is the \"converted\" success rate in the sample\n",
    "- $p_{new}$ and $p_{old}$ are the \"converted\" success rate for the two groups in the population. \n",
    "- $\\sigma_{new}$ and $\\sigma_{new}$ are the standard deviation for the two groups in the population. \n",
    "- $n_{new}$ and $n_{old}$ represent the size of the two groups or samples (it's same in our case)\n",
    "\n",
    "\n",
    ">Z-test is performed when the sample size is large, and the population variance is known. The z-score represents the distance between the two \"converted\" success rates in terms of the standard error. \n",
    "\n",
    "Next step is to make a decision to reject or fail to reject the null hypothesis based on comparing these two values: \n",
    "- $Z_{score}$\n",
    "- $Z_{\\alpha}$ or $Z_{0.05}$, also known as critical value at 95% confidence interval.  $Z_{0.05}$ is 1.645 for one-tailed tests,  and 1.960 for two-tailed test. You can determine the $Z_{\\alpha}$ from the z-table manually. \n",
    "\n",
    "Decide if your hypothesis is either a two-tailed, left-tailed, or right-tailed test. Accordingly, reject OR fail to reject the  null based on the comparison between $Z_{score}$ and $Z_{\\alpha}$. We determine whether or not the $Z_{score}$ lies in the \"rejection region\" in the distribution. In other words, a \"rejection region\" is an interval where the null hypothesis is rejected iff the $Z_{score}$ lies in that region.\n",
    "\n",
    ">Hint:<br>\n",
    "For a right-tailed test, reject null if $Z_{score}$ > $Z_{\\alpha}$. <br>\n",
    "For a left-tailed test, reject null if $Z_{score}$ < $Z_{\\alpha}$. \n",
    "\n",
    "\n",
    "\n",
    "\n",
    "Reference: \n",
    "- Example 9.1.2 on this [page](https://stats.libretexts.org/Bookshelves/Introductory_Statistics/Book%3A_Introductory_Statistics_(Shafer_and_Zhang)/09%3A_Two-Sample_Problems/9.01%3A_Comparison_of_Two_Population_Means-_Large_Independent_Samples), courtesy www.stats.libretexts.org\n",
    "\n",
    "---\n",
    "\n",
    ">**Tip**: You don't have to dive deeper into z-test for this exercise. **Try having an overview of what does z-score signify in general.** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "z_score = -1.3109\n",
      "p_value = 0.9051\n"
     ]
    }
   ],
   "source": [
    "import statsmodels.api as sm\n",
    "# ToDo: Complete the sm.stats.proportions_ztest() method arguments\n",
    "z_score, p_value = sm.stats.proportions_ztest([convert_new, convert_old], [n_new, n_old], alternative = 'larger')\n",
    "print('z_score =',\"{:.4f}\".format(z_score))\n",
    "print('p_value =',\"{:.4f}\".format(p_value))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**n.** What do the z-score and p-value you computed in the previous question mean for the conversion rates of the old and new pages?  Do they agree with the findings in parts **j.** and **k.**?<br><br>\n",
    "\n",
    ">**Tip**: Notice whether the p-value is similar to the one computed earlier. Accordingly, can you reject/fail to reject the null hypothesis? It is important to correctly interpret the test statistic and p-value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### In order to make a decision to reject or fail to reject the null hypothesis, we need to compare the two values of z_score and 𝑍𝛼. \n",
    "#### We defined 𝐻1 : 𝑝𝑛𝑒𝑤 > 𝑝𝑜𝑙𝑑 therefore, it is a right-tailed z test which is a one-tailed test. 𝑍𝛼 or $Z_{0.05}$, also known as critical value at 95% confidence interval is 1.645 for one-tailed tests.\n",
    "#### For a right-tailed test, we reject the null hypothesis if z_𝑠𝑐𝑜𝑟𝑒  > 𝑍𝛼. Our z_score is -1.3109 which is less than 𝑍𝛼 of 1.645.\n",
    "#### Therefore, based on this result, we are not confident to reject the null hypothesis.\n",
    "#### The p-value in part j is 0.9100 and the p-value using the built-in function is 0.9051. The two p-values have a small difference difference but statistically speaking, they agree with the findings in part j. p-value remains greater than the 0.05 threshold and therefore we are not confident to reject the null hypothesis.\n",
    "#### Considering the results of z_score and p-value using the built-in function, we are not confident to reject the null hypothesis same as our findings in part k."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='regression'></a>\n",
    "### Part III - A regression approach\n",
    "\n",
    "### ToDo 3.1 \n",
    "In this final part, you will see that the result you achieved in the A/B test in Part II above can also be achieved by performing regression.<br><br> \n",
    "\n",
    "**a.** Since each row in the `df2` data is either a conversion or no conversion, what type of regression should you be performing in this case?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">**In this case, logistic regression will be performed as it bounds our response to a probability between zero and 1.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b.** The goal is to use **statsmodels** library to fit the regression model you specified in part **a.** above to see if there is a significant difference in conversion based on the page-type a customer receives. However, you first need to create the following two columns in the `df2` dataframe:\n",
    " 1. `intercept` - It should be `1` in the entire column. \n",
    " 2. `ab_page` - It's a dummy variable column, having a value `1` when an individual receives the **treatment**, otherwise `0`.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:1: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n",
      "/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:2: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>ab_page</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>851104</td>\n",
       "      <td>2017-01-21 22:11:48.556739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>804228</td>\n",
       "      <td>2017-01-12 08:01:45.159739</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>661590</td>\n",
       "      <td>2017-01-11 16:55:06.154213</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>853541</td>\n",
       "      <td>2017-01-08 18:28:03.143765</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>864975</td>\n",
       "      <td>2017-01-21 01:52:26.210827</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id                   timestamp      group landing_page  converted  \\\n",
       "0   851104  2017-01-21 22:11:48.556739    control     old_page          0   \n",
       "1   804228  2017-01-12 08:01:45.159739    control     old_page          0   \n",
       "2   661590  2017-01-11 16:55:06.154213  treatment     new_page          0   \n",
       "3   853541  2017-01-08 18:28:03.143765  treatment     new_page          0   \n",
       "4   864975  2017-01-21 01:52:26.210827    control     old_page          1   \n",
       "\n",
       "   intercept  ab_page  \n",
       "0          1        0  \n",
       "1          1        0  \n",
       "2          1        1  \n",
       "3          1        1  \n",
       "4          1        0  "
      ]
     },
     "execution_count": 168,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2['intercept'] = 1\n",
    "df2['ab_page'] = pd.get_dummies(df['group'])['treatment']\n",
    "df2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 290584 entries, 0 to 294477\n",
      "Data columns (total 7 columns):\n",
      "user_id         290584 non-null int64\n",
      "timestamp       290584 non-null object\n",
      "group           290584 non-null object\n",
      "landing_page    290584 non-null object\n",
      "converted       290584 non-null int64\n",
      "intercept       290584 non-null int64\n",
      "ab_page         290584 non-null uint8\n",
      "dtypes: int64(3), object(3), uint8(1)\n",
      "memory usage: 15.8+ MB\n"
     ]
    }
   ],
   "source": [
    "df2.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c.** Use **statsmodels** to instantiate your regression model on the two columns you created in part (b). above, then fit the model to predict whether or not an individual converts. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366118\n",
      "         Iterations 6\n"
     ]
    }
   ],
   "source": [
    "log_mod = sm.Logit(df2['converted'], df2[['intercept', 'ab_page']])\n",
    "results = log_mod.fit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "        <td>Model:</td>              <td>Logit</td>       <td>No. Iterations:</td>    <td>6.0000</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <td>Dependent Variable:</td>     <td>converted</td>    <td>Pseudo R-squared:</td>    <td>0.000</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "         <td>Date:</td>        <td>2022-02-12 23:37</td>       <td>AIC:</td>        <td>212780.3502</td>\n",
       "</tr>\n",
       "<tr>\n",
       "   <td>No. Observations:</td>       <td>290584</td>            <td>BIC:</td>        <td>212801.5095</td>\n",
       "</tr>\n",
       "<tr>\n",
       "       <td>Df Model:</td>              <td>1</td>         <td>Log-Likelihood:</td>  <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "     <td>Df Residuals:</td>         <td>290582</td>          <td>LL-Null:</td>      <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "      <td>Converged:</td>           <td>1.0000</td>           <td>Scale:</td>         <td>1.0000</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>       <th>Coef.</th>  <th>Std.Err.</th>     <th>z</th>      <th>P>|z|</th> <th>[0.025</th>  <th>0.975]</th> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th> <td>-1.9888</td>  <td>0.0081</td>  <td>-246.6690</td> <td>0.0000</td> <td>-2.0046</td> <td>-1.9730</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page</th>   <td>-0.0150</td>  <td>0.0114</td>   <td>-1.3109</td>  <td>0.1899</td> <td>-0.0374</td> <td>0.0074</td> \n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary2.Summary'>\n",
       "\"\"\"\n",
       "                          Results: Logit\n",
       "==================================================================\n",
       "Model:              Logit            No. Iterations:   6.0000     \n",
       "Dependent Variable: converted        Pseudo R-squared: 0.000      \n",
       "Date:               2022-02-12 23:37 AIC:              212780.3502\n",
       "No. Observations:   290584           BIC:              212801.5095\n",
       "Df Model:           1                Log-Likelihood:   -1.0639e+05\n",
       "Df Residuals:       290582           LL-Null:          -1.0639e+05\n",
       "Converged:          1.0000           Scale:            1.0000     \n",
       "-------------------------------------------------------------------\n",
       "              Coef.   Std.Err.      z      P>|z|    [0.025   0.975]\n",
       "-------------------------------------------------------------------\n",
       "intercept    -1.9888    0.0081  -246.6690  0.0000  -2.0046  -1.9730\n",
       "ab_page      -0.0150    0.0114    -1.3109  0.1899  -0.0374   0.0074\n",
       "==================================================================\n",
       "\n",
       "\"\"\""
      ]
     },
     "execution_count": 171,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.summary2()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.98511193960306265"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# exponentiate the coefficient for ab_page\n",
    "np.exp(-0.0150)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0151130646157189"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# compute the reciprocal since we get multiplicative changes less than 1\n",
    "1/np.exp(-0.0150)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**d.** Provide the summary of your model below, and use it as necessary to answer the following questions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The results shows our coefficient for ab_page is -0.0150. In order to interpret it,  we need to exponentiate it. So we get the value of around 0.9851 which is less than 1. When multiplicative changes are less than 1, it is usually useful to calculate the reciprocal. \n",
    "\n",
    "#### Now, our value is around 1.0151. So, the result for ab_page can be interpreted as: The new page received by the treatment group drives 1.0151 times less conversions than the old page received by the control group, holding all other variables constant or the individuals in the treatment group converts 1.0151 times less likely than the individuals in the control group, holding all other variables constant."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**e.** What is the p-value associated with **ab_page**? Why does it differ from the value you found in **Part II**?<br><br>  \n",
    "\n",
    "**Hints**: \n",
    "- What are the null and alternative hypotheses associated with your regression model, and how do they compare to the null and alternative hypotheses in **Part II**? \n",
    "- You may comment on if these hypothesis (Part II vs. Part III) are one-sided or two-sided. \n",
    "- You may also compare the current p-value with the Type I error rate (0.05).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The p-value associated with ab_page is 0.1899. The p-value in part II is 0.9100. It is different than the p-value in Part II because were are testing different hypotheses.\n",
    "\n",
    "#### The hypothesis associated with the regression model is a two-tailed test with null & alternative hypotheses below:\n",
    "#### **$H_0$: $p_{new}$ = $p_{old}$**\n",
    "#### **$H_1$: $p_{new}$ ≠ $p_{old}$**\n",
    "\n",
    "#### The hypothesis we did earlier in Part II is right-tailed which is one of the one-tailed tests with the null & alternative hypotheses below :\n",
    "#### **$H_0$: $p_{new}$ ≤ $p_{old}$**\n",
    "#### **$H_1$: $p_{new}$ > $p_{old}$**\n",
    "\n",
    "#### The p-value we got from the Logit regression model remains greater than the Type 1 error rate of 0.05 threshold and therefore we are not confident to reject the null hypothesis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**f.** Now, you are considering other things that might influence whether or not an individual converts.  Discuss why it is a good idea to consider other factors to add into your regression model.  Are there any disadvantages to adding additional terms into your regression model?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Other factors that can be cosidered that might influence the conversion rate are age and location (city, country-side, country) of the individual. \n",
    "\n",
    "#### Old-age people like those who are 60+ years are not very much comfortable using the internet and websites. Most of them prefer traditional shopping from the physical store where there is human interaction and where they can check the items face to face. Whereas the younger people, they are more proficient with internet usage and they enjoy the comfort of online shopping. \n",
    "\n",
    "#### Location can also be a factor. If you live in the city where shopping centers are everywhere and in close proximity, you might not need to do online shopping regularly, maybe on an urgent needs only. But if you live far from the city, you might find online shopping more efficient than driving all the way to the city just to buy one item of clothing or toy. \n",
    "\n",
    "#### Country can also be a factor. Some countries have a very good internet connection, IT security and huge e-commerce presence country-wide. Countries where majority of citizens are above average and rich tend to use e-commerce more. The old and new pages might also behave differently from country to country.\n",
    "\n",
    "#### It is a good idea to consider these factors to add into the regression model in order to have a more informed decision and avoid the likelihood of getting misguided decision.\n",
    "\n",
    "#### The disadvantages of adding additional factors into regression model could be the additional costs and time the test would need especially if the data is not readily available."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**g. Adding countries**<br> \n",
    "Now along with testing if the conversion rate changes for different pages, also add an effect based on which country a user lives in. \n",
    "\n",
    "1. You will need to read in the **countries.csv** dataset and merge together your `df2` datasets on the appropriate rows. You call the resulting dataframe `df_merged`. [Here](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.join.html) are the docs for joining tables. \n",
    "\n",
    "2. Does it appear that country had an impact on conversion?  To answer this question, consider the three unique values, `['UK', 'US', 'CA']`, in the `country` column. Create dummy variables for these country columns. \n",
    ">**Hint:** Use `pandas.get_dummies()` to create dummy variables. **You will utilize two columns for the three dummy variables.** \n",
    "\n",
    " Provide the statistical output as well as a written response to answer this question."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 172,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>country</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>CA</th>\n",
       "      <td>14499</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>UK</th>\n",
       "      <td>72466</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>US</th>\n",
       "      <td>203619</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         user_id\n",
       "country         \n",
       "CA         14499\n",
       "UK         72466\n",
       "US        203619"
      ]
     },
     "execution_count": 172,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read the countries.csv\n",
    "df_countries = pd.read_csv('countries.csv')\n",
    "df_countries.groupby('country').count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 290584 entries, 0 to 290583\n",
      "Data columns (total 2 columns):\n",
      "user_id    290584 non-null int64\n",
      "country    290584 non-null object\n",
      "dtypes: int64(1), object(1)\n",
      "memory usage: 4.4+ MB\n"
     ]
    }
   ],
   "source": [
    "df_countries.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 174,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>country</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>group</th>\n",
       "      <th>landing_page</th>\n",
       "      <th>converted</th>\n",
       "      <th>intercept</th>\n",
       "      <th>ab_page</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>user_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>834778</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-14 23:08:43.304998</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>928468</th>\n",
       "      <td>US</td>\n",
       "      <td>2017-01-23 14:44:16.387854</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>822059</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-16 14:04:14.719771</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>711597</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-22 03:14:24.763511</td>\n",
       "      <td>control</td>\n",
       "      <td>old_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>710616</th>\n",
       "      <td>UK</td>\n",
       "      <td>2017-01-16 13:14:44.000513</td>\n",
       "      <td>treatment</td>\n",
       "      <td>new_page</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        country                   timestamp      group landing_page  \\\n",
       "user_id                                                               \n",
       "834778       UK  2017-01-14 23:08:43.304998    control     old_page   \n",
       "928468       US  2017-01-23 14:44:16.387854  treatment     new_page   \n",
       "822059       UK  2017-01-16 14:04:14.719771  treatment     new_page   \n",
       "711597       UK  2017-01-22 03:14:24.763511    control     old_page   \n",
       "710616       UK  2017-01-16 13:14:44.000513  treatment     new_page   \n",
       "\n",
       "         converted  intercept  ab_page  \n",
       "user_id                                 \n",
       "834778           0          1        0  \n",
       "928468           0          1        1  \n",
       "822059           1          1        1  \n",
       "711597           0          1        0  \n",
       "710616           0          1        1  "
      ]
     },
     "execution_count": 174,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Join with the df2 dataframe\n",
    "df_merged = df_countries.set_index('user_id').join(df2.set_index('user_id'))\n",
    "df_merged.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "country         0\n",
       "timestamp       0\n",
       "group           0\n",
       "landing_page    0\n",
       "converted       0\n",
       "intercept       0\n",
       "ab_page         0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 175,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_merged.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 290584 entries, 834778 to 934996\n",
      "Data columns (total 7 columns):\n",
      "country         290584 non-null object\n",
      "timestamp       290584 non-null object\n",
      "group           290584 non-null object\n",
      "landing_page    290584 non-null object\n",
      "converted       290584 non-null int64\n",
      "intercept       290584 non-null int64\n",
      "ab_page         290584 non-null uint8\n",
      "dtypes: int64(2), object(4), uint8(1)\n",
      "memory usage: 25.8+ MB\n"
     ]
    }
   ],
   "source": [
    "df_merged.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the necessary dummy variables\n",
    "df_merged[['US', 'UK', 'CA']] = pd.get_dummies(df_merged['country'])[['US', 'UK', 'CA']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The proportions of individuals who converted in the US, UK and CA are 0.1195 , 0.1206 and 0.1153 respectively\n"
     ]
    }
   ],
   "source": [
    "#Does it appear that country had an impact on conversion? \n",
    "#Provide the statistical output as well as a written response to answer this question.\n",
    "US_converted = (df_merged.query(\"US == '1'\")['converted']).mean()\n",
    "UK_converted = (df_merged.query(\"UK == '1'\")['converted']).mean()\n",
    "CA_converted = (df_merged.query(\"CA == '1'\")['converted']).mean()\n",
    "print('The proportions of individuals who converted in the US, UK and CA are',\"{:.4f}\".format(US_converted),',',\"{:.4f}\".format(UK_converted),'and '\"{:.4f}\".format(CA_converted), 'respectively')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### To answer the question, \"Does it appear that country had an impact on conversion?  \n",
    "#### The conversion rates of each country US = 11.95, UK = 12.06 and CA = 11.53 show that the country does not affect the conversion as the proportions are all approximately the same."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**h. Fit your model and obtain the results**<br> \n",
    "Though you have now looked at the individual factors of country and page on conversion, we would now like to look at an interaction between page and country to see if are there significant effects on conversion.  **Create the necessary additional columns, and fit the new model.** \n",
    "\n",
    "\n",
    "Provide the summary results (statistical output), and your conclusions (written response) based on the results. \n",
    "\n",
    ">**Tip**: Conclusions should include both statistical reasoning, and practical reasoning for the situation. \n",
    "\n",
    ">**Hints**: \n",
    "- Look at all of p-values in the summary, and compare against the Type I error rate (0.05). \n",
    "- Can you reject/fail to reject the null hypotheses (regression model)?\n",
    "- Comment on the effect of page and country to predict the conversion.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.366113\n",
      "         Iterations 6\n"
     ]
    }
   ],
   "source": [
    "# Fit your model, and summarize the results\n",
    "log_mod2 = sm.Logit(df_merged['converted'], df_merged[['intercept', 'ab_page', 'US', 'UK']])\n",
    "results2 = log_mod2.fit()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "        <td>Model:</td>              <td>Logit</td>       <td>No. Iterations:</td>    <td>6.0000</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "  <td>Dependent Variable:</td>     <td>converted</td>    <td>Pseudo R-squared:</td>    <td>0.000</td>   \n",
       "</tr>\n",
       "<tr>\n",
       "         <td>Date:</td>        <td>2022-02-12 23:48</td>       <td>AIC:</td>        <td>212781.1253</td>\n",
       "</tr>\n",
       "<tr>\n",
       "   <td>No. Observations:</td>       <td>290584</td>            <td>BIC:</td>        <td>212823.4439</td>\n",
       "</tr>\n",
       "<tr>\n",
       "       <td>Df Model:</td>              <td>3</td>         <td>Log-Likelihood:</td>  <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "     <td>Df Residuals:</td>         <td>290580</td>          <td>LL-Null:</td>      <td>-1.0639e+05</td>\n",
       "</tr>\n",
       "<tr>\n",
       "      <td>Converged:</td>           <td>1.0000</td>           <td>Scale:</td>         <td>1.0000</td>   \n",
       "</tr>\n",
       "</table>\n",
       "<table class=\"simpletable\">\n",
       "<tr>\n",
       "      <td></td>       <th>Coef.</th>  <th>Std.Err.</th>     <th>z</th>     <th>P>|z|</th> <th>[0.025</th>  <th>0.975]</th> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>intercept</th> <td>-2.0300</td>  <td>0.0266</td>  <td>-76.2488</td> <td>0.0000</td> <td>-2.0822</td> <td>-1.9778</td>\n",
       "</tr>\n",
       "<tr>\n",
       "  <th>ab_page</th>   <td>-0.0149</td>  <td>0.0114</td>   <td>-1.3069</td> <td>0.1912</td> <td>-0.0374</td> <td>0.0075</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>US</th>        <td>0.0408</td>   <td>0.0269</td>   <td>1.5161</td>  <td>0.1295</td> <td>-0.0119</td> <td>0.0934</td> \n",
       "</tr>\n",
       "<tr>\n",
       "  <th>UK</th>        <td>0.0506</td>   <td>0.0284</td>   <td>1.7835</td>  <td>0.0745</td> <td>-0.0050</td> <td>0.1063</td> \n",
       "</tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<class 'statsmodels.iolib.summary2.Summary'>\n",
       "\"\"\"\n",
       "                          Results: Logit\n",
       "==================================================================\n",
       "Model:              Logit            No. Iterations:   6.0000     \n",
       "Dependent Variable: converted        Pseudo R-squared: 0.000      \n",
       "Date:               2022-02-12 23:48 AIC:              212781.1253\n",
       "No. Observations:   290584           BIC:              212823.4439\n",
       "Df Model:           3                Log-Likelihood:   -1.0639e+05\n",
       "Df Residuals:       290580           LL-Null:          -1.0639e+05\n",
       "Converged:          1.0000           Scale:            1.0000     \n",
       "-------------------------------------------------------------------\n",
       "               Coef.   Std.Err.     z      P>|z|    [0.025   0.975]\n",
       "-------------------------------------------------------------------\n",
       "intercept     -2.0300    0.0266  -76.2488  0.0000  -2.0822  -1.9778\n",
       "ab_page       -0.0149    0.0114   -1.3069  0.1912  -0.0374   0.0075\n",
       "US             0.0408    0.0269    1.5161  0.1295  -0.0119   0.0934\n",
       "UK             0.0506    0.0284    1.7835  0.0745  -0.0050   0.1063\n",
       "==================================================================\n",
       "\n",
       "\"\"\""
      ]
     },
     "execution_count": 183,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results2.summary2()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "intercept    0.131332\n",
       "ab_page      0.985168\n",
       "US           1.041599\n",
       "UK           1.051944\n",
       "dtype: float64"
      ]
     },
     "execution_count": 184,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.exp(results2.params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### After considering the countries US, UK and CA into the regression model and based on the results, this can be interpreted as:\n",
    "##### If the costumer is in the US, it is 1.04 times as likely to convert or to actually buy the product than the individuals in other countries, holding all other varibles constant. Consequently, if the costumer is in the UK, it is 1.05 times as likely to convert.\n",
    "\n",
    "#### The p- value for the US is 0.1295 which is higher than the type 1 error rate of 0.05; the p-value for UK is 0.0745 which is also higher than the threshold. This suggests that we are not confident to reject the null hypothesis.\n",
    "\n",
    "#### Based on the available data and the results of all the tests done (probability, hypothesis, z-test and regression), the indication remains that we fail to reject the null hypothesis. That is the new page does not drive more conversions than the old page.\n",
    "\n",
    "#### Therefore, we recommend that the e-commerce company keep the old page for the time being. As for the new page, since it has already been created for testing purposes, this can be updated or upgraded and re-run another tests after sometime. This can save time and money on the part of company rather than to launch a new page  which will cost them for the implementation but without significant increase in revenues."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='finalcheck'></a>\n",
    "## Final Check!\n",
    "\n",
    "Congratulations!  You have reached the end of the A/B Test Results project!  You should be very proud of all you have accomplished!\n",
    "\n",
    "\n",
    "\n",
    "<a id='submission'></a>\n",
    "## Submission\n",
    "You may either submit your notebook through the \"SUBMIT PROJECT\" button at the bottom of this workspace, or you may work from your local machine and submit on  the last page of this project lesson.  \n",
    "\n",
    "1. Before you submit your project, you need to create a .html or .pdf version of this notebook in the workspace here. To do that, run the code cell below. If it worked correctly, you should get a return code of 0, and you should see the generated .html file in the workspace directory (click on the orange Jupyter icon in the upper left).\n",
    "\n",
    "\n",
    "2. Alternatively, you can download this report as .html via the **File** > **Download as** submenu, and then manually upload it into the workspace directory by clicking on the orange Jupyter icon in the upper left, then using the Upload button.\n",
    "\n",
    "\n",
    "3. Once you've done this, you can submit your project by clicking on the \"Submit Project\" button in the lower right here. This will create and submit a zip file with this .ipynb doc and the .html or .pdf version you created. Congratulations!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 185,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from subprocess import call\n",
    "call(['python', '-m', 'nbconvert', 'Analyze_ab_test_results_notebook.ipynb'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}