Case.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### COMMENTS\n",
    "\n",
    "**Comments regarding data** \\\n",
    "In my data, it looks like there are none missing in the test data\n",
    "\n",
    "The following features are not included:\n",
    "\n",
    "- Date of birth: lack of time\n",
    "- Full name: no explanatory power\n",
    "- Zip code: I did not really understand it - where do I find it?\n",
    "\n",
    "The feature **On a shift** is not included due to lack of time \\\n",
    "However, my approach would be:\n",
    "Insert a column on X_test indicating if **on shift last day seen** and use this as regressor\n",
    "\n",
    "\n",
    "**Which features have most explanatory power** \\\n",
    "I did not find this, but my approach would be:\n",
    "\n",
    "1) Use X_train and y_train to set up an econometric model using the regressors and dummy variables \\\n",
    "2) Estimate the model with a probit model for this binary classification \\\n",
    "3) Evaluate the regressors model coeffficients, the *Betas* (and if they are significant at some level) \\\n",
    "4) List the most significant coefficients (which have positive and which have negative effect)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from os.path import join as pjoin\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from datetime import datetime, date\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_FOLDER = pjoin(os.getcwd(), \"Data\")\n",
    "files = os.listdir(DATA_FOLDER)\n",
    "assert len(files) == 4, \"Too many files in here\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Investigating, cleaning and merging data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(4680, 13)\n",
      "(10920, 13)\n",
      "(276, 3)\n",
      "(13, 6)\n"
     ]
    }
   ],
   "source": [
    "data_train_explanatory = pd.read_csv(pjoin(DATA_FOLDER, \"data_train_fin.csv\"))\n",
    "data_test_explanatory = pd.read_csv(pjoin(DATA_FOLDER, \"data_test_fin.csv\"))\n",
    "missing_report = pd.read_csv(pjoin(DATA_FOLDER, \"missing_report.csv\"))\n",
    "shift_report = pd.read_csv(pjoin(DATA_FOLDER, \"shift_report.csv\"))\n",
    "\n",
    "print(data_test_explanatory.shape)\n",
    "print(data_train_explanatory.shape)\n",
    "print(missing_report.shape)                                               \n",
    "print(shift_report.shape)                                               "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>full_name</th>\n",
       "      <th>sex</th>\n",
       "      <th>profession</th>\n",
       "      <th>hourly_salary</th>\n",
       "      <th>date_of_birth</th>\n",
       "      <th>weight</th>\n",
       "      <th>height</th>\n",
       "      <th>city_of_origin</th>\n",
       "      <th>last_seen</th>\n",
       "      <th>satisfaction_score</th>\n",
       "      <th>sleep_1</th>\n",
       "      <th>sleep_2</th>\n",
       "      <th>sleep_3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Ryan Weasley</td>\n",
       "      <td>m</td>\n",
       "      <td>bartender</td>\n",
       "      <td>11.9</td>\n",
       "      <td>2/15/83</td>\n",
       "      <td>78.0</td>\n",
       "      <td>185.0</td>\n",
       "      <td>Melbourne</td>\n",
       "      <td>2208-02-23</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8.3</td>\n",
       "      <td>7.2</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jason Bradbury</td>\n",
       "      <td>m</td>\n",
       "      <td>developer</td>\n",
       "      <td>14.3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>83.0</td>\n",
       "      <td>162.0</td>\n",
       "      <td>Seattle</td>\n",
       "      <td>2208-02-18</td>\n",
       "      <td>0.672414</td>\n",
       "      <td>6.6</td>\n",
       "      <td>7.7</td>\n",
       "      <td>7.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>f</td>\n",
       "      <td>developer</td>\n",
       "      <td>14.0</td>\n",
       "      <td>5/25/64</td>\n",
       "      <td>84.0</td>\n",
       "      <td>171.0</td>\n",
       "      <td>Brisbane</td>\n",
       "      <td>2208-02-22</td>\n",
       "      <td>0.597701</td>\n",
       "      <td>8.4</td>\n",
       "      <td>8.1</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Jose Huxley</td>\n",
       "      <td>m</td>\n",
       "      <td>clerk</td>\n",
       "      <td>16.3</td>\n",
       "      <td>8/6/62</td>\n",
       "      <td>NaN</td>\n",
       "      <td>182.0</td>\n",
       "      <td>Sydney</td>\n",
       "      <td>2208-02-24</td>\n",
       "      <td>0.718391</td>\n",
       "      <td>8.2</td>\n",
       "      <td>5.6</td>\n",
       "      <td>9.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Willie Woolf</td>\n",
       "      <td>m</td>\n",
       "      <td>bookkeeper</td>\n",
       "      <td>16.0</td>\n",
       "      <td>2/17/75</td>\n",
       "      <td>94.0</td>\n",
       "      <td>187.0</td>\n",
       "      <td>Vancouver</td>\n",
       "      <td>2208-02-24</td>\n",
       "      <td>0.718391</td>\n",
       "      <td>8.7</td>\n",
       "      <td>7.9</td>\n",
       "      <td>8.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4675</th>\n",
       "      <td>NaN</td>\n",
       "      <td>f</td>\n",
       "      <td>server</td>\n",
       "      <td>10.3</td>\n",
       "      <td>10/3/87</td>\n",
       "      <td>72.0</td>\n",
       "      <td>195.0</td>\n",
       "      <td>Brisbane</td>\n",
       "      <td>2208-02-24</td>\n",
       "      <td>0.574713</td>\n",
       "      <td>8.1</td>\n",
       "      <td>8.4</td>\n",
       "      <td>8.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4676</th>\n",
       "      <td>Ann Martin</td>\n",
       "      <td>f</td>\n",
       "      <td>cashier</td>\n",
       "      <td>9.9</td>\n",
       "      <td>9/13/92</td>\n",
       "      <td>68.0</td>\n",
       "      <td>176.0</td>\n",
       "      <td>Melbourne</td>\n",
       "      <td>2208-02-16</td>\n",
       "      <td>0.465517</td>\n",
       "      <td>8.4</td>\n",
       "      <td>7.9</td>\n",
       "      <td>6.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4677</th>\n",
       "      <td>Raymond Laurie</td>\n",
       "      <td>m</td>\n",
       "      <td>trader</td>\n",
       "      <td>15.0</td>\n",
       "      <td>9/12/96</td>\n",
       "      <td>80.0</td>\n",
       "      <td>190.0</td>\n",
       "      <td>Toronto</td>\n",
       "      <td>2208-02-16</td>\n",
       "      <td>0.781609</td>\n",
       "      <td>8.9</td>\n",
       "      <td>8.6</td>\n",
       "      <td>8.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4678</th>\n",
       "      <td>Donald Jackson</td>\n",
       "      <td>m</td>\n",
       "      <td>plant worker</td>\n",
       "      <td>10.1</td>\n",
       "      <td>10/13/84</td>\n",
       "      <td>87.0</td>\n",
       "      <td>177.0</td>\n",
       "      <td>Brisbane</td>\n",
       "      <td>2208-02-15</td>\n",
       "      <td>0.545977</td>\n",
       "      <td>6.7</td>\n",
       "      <td>7.2</td>\n",
       "      <td>7.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4679</th>\n",
       "      <td>Evelyn Williams</td>\n",
       "      <td>f</td>\n",
       "      <td>developer</td>\n",
       "      <td>13.7</td>\n",
       "      <td>3/26/87</td>\n",
       "      <td>65.0</td>\n",
       "      <td>166.0</td>\n",
       "      <td>London</td>\n",
       "      <td>2208-02-24</td>\n",
       "      <td>0.770115</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.3</td>\n",
       "      <td>6.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4680 rows × 13 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            full_name sex    profession  hourly_salary date_of_birth  weight  \\\n",
       "0        Ryan Weasley   m     bartender           11.9       2/15/83    78.0   \n",
       "1      Jason Bradbury   m     developer           14.3           NaN    83.0   \n",
       "2                 NaN   f     developer           14.0       5/25/64    84.0   \n",
       "3         Jose Huxley   m         clerk           16.3        8/6/62     NaN   \n",
       "4        Willie Woolf   m    bookkeeper           16.0       2/17/75    94.0   \n",
       "...               ...  ..           ...            ...           ...     ...   \n",
       "4675              NaN   f        server           10.3       10/3/87    72.0   \n",
       "4676       Ann Martin   f       cashier            9.9       9/13/92    68.0   \n",
       "4677   Raymond Laurie   m        trader           15.0       9/12/96    80.0   \n",
       "4678   Donald Jackson   m  plant worker           10.1      10/13/84    87.0   \n",
       "4679  Evelyn Williams   f     developer           13.7       3/26/87    65.0   \n",
       "\n",
       "      height city_of_origin   last_seen  satisfaction_score  sleep_1  sleep_2  \\\n",
       "0      185.0      Melbourne  2208-02-23                 NaN      8.3      7.2   \n",
       "1      162.0        Seattle  2208-02-18            0.672414      6.6      7.7   \n",
       "2      171.0       Brisbane  2208-02-22            0.597701      8.4      8.1   \n",
       "3      182.0         Sydney  2208-02-24            0.718391      8.2      5.6   \n",
       "4      187.0      Vancouver  2208-02-24            0.718391      8.7      7.9   \n",
       "...      ...            ...         ...                 ...      ...      ...   \n",
       "4675   195.0       Brisbane  2208-02-24            0.574713      8.1      8.4   \n",
       "4676   176.0      Melbourne  2208-02-16            0.465517      8.4      7.9   \n",
       "4677   190.0        Toronto  2208-02-16            0.781609      8.9      8.6   \n",
       "4678   177.0       Brisbane  2208-02-15            0.545977      6.7      7.2   \n",
       "4679   166.0         London  2208-02-24            0.770115      9.0      9.3   \n",
       "\n",
       "      sleep_3  \n",
       "0         7.0  \n",
       "1         7.2  \n",
       "2         NaN  \n",
       "3         9.8  \n",
       "4         8.1  \n",
       "...       ...  \n",
       "4675      8.2  \n",
       "4676      6.2  \n",
       "4677      8.3  \n",
       "4678      7.7  \n",
       "4679      6.5  \n",
       "\n",
       "[4680 rows x 13 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_test_explanatory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>full_name</th>\n",
       "      <th>sex</th>\n",
       "      <th>profession</th>\n",
       "      <th>hourly_salary</th>\n",
       "      <th>date_of_birth</th>\n",
       "      <th>weight</th>\n",
       "      <th>height</th>\n",
       "      <th>city_of_origin</th>\n",
       "      <th>last_seen</th>\n",
       "      <th>satisfaction_score</th>\n",
       "      <th>sleep_1</th>\n",
       "      <th>sleep_2</th>\n",
       "      <th>sleep_3</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Emily Holmes</td>\n",
       "      <td>f</td>\n",
       "      <td>laborer</td>\n",
       "      <td>14.0</td>\n",
       "      <td>10/2/73</td>\n",
       "      <td>84.0</td>\n",
       "      <td>175.0</td>\n",
       "      <td>Manchester</td>\n",
       "      <td>2208-02-22</td>\n",
       "      <td>0.706897</td>\n",
       "      <td>7.0</td>\n",
       "      <td>8.8</td>\n",
       "      <td>11.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Vincent Harris</td>\n",
       "      <td>m</td>\n",
       "      <td>psychologist</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1/11/72</td>\n",
       "      <td>72.0</td>\n",
       "      <td>171.0</td>\n",
       "      <td>Seattle</td>\n",
       "      <td>2208-02-24</td>\n",
       "      <td>0.701149</td>\n",
       "      <td>7.9</td>\n",
       "      <td>7.6</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Matthew Wilson</td>\n",
       "      <td>m</td>\n",
       "      <td>electrician</td>\n",
       "      <td>14.0</td>\n",
       "      <td>7/6/71</td>\n",
       "      <td>66.0</td>\n",
       "      <td>179.0</td>\n",
       "      <td>Toronto</td>\n",
       "      <td>2208-02-22</td>\n",
       "      <td>0.580460</td>\n",
       "      <td>8.5</td>\n",
       "      <td>6.2</td>\n",
       "      <td>8.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>m</td>\n",
       "      <td>police officer</td>\n",
       "      <td>14.0</td>\n",
       "      <td>11/2/66</td>\n",
       "      <td>82.0</td>\n",
       "      <td>177.0</td>\n",
       "      <td>New York</td>\n",
       "      <td>2208-02-22</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>9.4</td>\n",
       "      <td>8.1</td>\n",
       "      <td>8.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Marilyn Rodriguez</td>\n",
       "      <td>f</td>\n",
       "      <td>clerk</td>\n",
       "      <td>16.2</td>\n",
       "      <td>1/12/65</td>\n",
       "      <td>94.0</td>\n",
       "      <td>185.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2208-02-24</td>\n",
       "      <td>0.729885</td>\n",
       "      <td>8.9</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10915</th>\n",
       "      <td>Larry White</td>\n",
       "      <td>m</td>\n",
       "      <td>cook</td>\n",
       "      <td>12.0</td>\n",
       "      <td>5/18/96</td>\n",
       "      <td>75.0</td>\n",
       "      <td>170.0</td>\n",
       "      <td>London</td>\n",
       "      <td>2208-02-15</td>\n",
       "      <td>0.724138</td>\n",
       "      <td>7.2</td>\n",
       "      <td>7.9</td>\n",
       "      <td>8.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10916</th>\n",
       "      <td>Christopher Wilson</td>\n",
       "      <td>m</td>\n",
       "      <td>police officer</td>\n",
       "      <td>13.7</td>\n",
       "      <td>1/9/80</td>\n",
       "      <td>65.0</td>\n",
       "      <td>175.0</td>\n",
       "      <td>Manchester</td>\n",
       "      <td>2208-02-23</td>\n",
       "      <td>0.614943</td>\n",
       "      <td>7.1</td>\n",
       "      <td>8.2</td>\n",
       "      <td>7.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10917</th>\n",
       "      <td>Scott Laurie</td>\n",
       "      <td>m</td>\n",
       "      <td>server</td>\n",
       "      <td>10.3</td>\n",
       "      <td>4/4/62</td>\n",
       "      <td>66.0</td>\n",
       "      <td>164.0</td>\n",
       "      <td>Vancouver</td>\n",
       "      <td>2208-02-20</td>\n",
       "      <td>0.316092</td>\n",
       "      <td>8.3</td>\n",
       "      <td>6.3</td>\n",
       "      <td>9.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10918</th>\n",
       "      <td>Jason Morrison</td>\n",
       "      <td>m</td>\n",
       "      <td>cook</td>\n",
       "      <td>11.5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>70.0</td>\n",
       "      <td>177.0</td>\n",
       "      <td>Seattle</td>\n",
       "      <td>2208-02-24</td>\n",
       "      <td>0.574713</td>\n",
       "      <td>7.6</td>\n",
       "      <td>7.2</td>\n",
       "      <td>7.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10919</th>\n",
       "      <td>Dennis Hernandez</td>\n",
       "      <td>m</td>\n",
       "      <td>laborer</td>\n",
       "      <td>14.0</td>\n",
       "      <td>4/14/60</td>\n",
       "      <td>78.0</td>\n",
       "      <td>198.0</td>\n",
       "      <td>London</td>\n",
       "      <td>2208-02-17</td>\n",
       "      <td>0.632184</td>\n",
       "      <td>7.7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10920 rows × 13 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                full_name sex      profession  hourly_salary date_of_birth  \\\n",
       "0            Emily Holmes   f         laborer           14.0       10/2/73   \n",
       "1          Vincent Harris   m    psychologist            NaN       1/11/72   \n",
       "2          Matthew Wilson   m     electrician           14.0        7/6/71   \n",
       "3                     NaN   m  police officer           14.0       11/2/66   \n",
       "4       Marilyn Rodriguez   f           clerk           16.2       1/12/65   \n",
       "...                   ...  ..             ...            ...           ...   \n",
       "10915         Larry White   m            cook           12.0       5/18/96   \n",
       "10916  Christopher Wilson   m  police officer           13.7        1/9/80   \n",
       "10917        Scott Laurie   m          server           10.3        4/4/62   \n",
       "10918      Jason Morrison   m            cook           11.5           NaN   \n",
       "10919    Dennis Hernandez   m         laborer           14.0       4/14/60   \n",
       "\n",
       "       weight  height city_of_origin   last_seen  satisfaction_score  sleep_1  \\\n",
       "0        84.0   175.0     Manchester  2208-02-22            0.706897      7.0   \n",
       "1        72.0   171.0        Seattle  2208-02-24            0.701149      7.9   \n",
       "2        66.0   179.0        Toronto  2208-02-22            0.580460      8.5   \n",
       "3        82.0   177.0       New York  2208-02-22            0.666667      9.4   \n",
       "4        94.0   185.0            NaN  2208-02-24            0.729885      8.9   \n",
       "...       ...     ...            ...         ...                 ...      ...   \n",
       "10915    75.0   170.0         London  2208-02-15            0.724138      7.2   \n",
       "10916    65.0   175.0     Manchester  2208-02-23            0.614943      7.1   \n",
       "10917    66.0   164.0      Vancouver  2208-02-20            0.316092      8.3   \n",
       "10918    70.0   177.0        Seattle  2208-02-24            0.574713      7.6   \n",
       "10919    78.0   198.0         London  2208-02-17            0.632184      7.7   \n",
       "\n",
       "       sleep_2  sleep_3  \n",
       "0          8.8     11.3  \n",
       "1          7.6      9.0  \n",
       "2          6.2      8.0  \n",
       "3          8.1      8.5  \n",
       "4          NaN      8.1  \n",
       "...        ...      ...  \n",
       "10915      7.9      8.9  \n",
       "10916      8.2      7.2  \n",
       "10917      6.3      9.3  \n",
       "10918      7.2      7.9  \n",
       "10919      NaN      8.5  \n",
       "\n",
       "[10920 rows x 13 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_train_explanatory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make new column matching other data set's column\n",
    "missing_report['full_name'] = missing_report.apply(lambda row: row.first_name + \" \" + row.last_name, axis=1)\n",
    "missing_report = missing_report.loc[:, [\"missing\", \"full_name\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>missing</th>\n",
       "      <th>full_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Willie Gallager</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Melissa Morrison</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>Gregory Miller</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>Richard White</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>Julia Gilmour</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>271</th>\n",
       "      <td>1</td>\n",
       "      <td>Kevin White</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>272</th>\n",
       "      <td>1</td>\n",
       "      <td>Jeremy Gallager</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>273</th>\n",
       "      <td>1</td>\n",
       "      <td>Abigail Forster</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>274</th>\n",
       "      <td>1</td>\n",
       "      <td>Roy Burgess</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>275</th>\n",
       "      <td>1</td>\n",
       "      <td>Cynthia Huxley</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>276 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     missing         full_name\n",
       "0          1   Willie Gallager\n",
       "1          1  Melissa Morrison\n",
       "2          1    Gregory Miller\n",
       "3          1     Richard White\n",
       "4          1     Julia Gilmour\n",
       "..       ...               ...\n",
       "271        1       Kevin White\n",
       "272        1   Jeremy Gallager\n",
       "273        1   Abigail Forster\n",
       "274        1       Roy Burgess\n",
       "275        1    Cynthia Huxley\n",
       "\n",
       "[276 rows x 2 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "missing_report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No. of depedent variables inserted: 260\n",
      "No. of depedent variables in total: 276\n"
     ]
    }
   ],
   "source": [
    "# Merge Explanatory variables with dependent variable \"missing\"\n",
    "train_set = data_train_explanatory.merge(right=missing_report, how='left', left_on=\"full_name\", right_on='full_name')\n",
    "print(\"No. of depedent variables inserted:\", train_set.missing.count())\n",
    "print(\"No. of depedent variables in total:\", len(missing_report.dropna()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No. of depedent variables inserted: 0\n",
      "No. of depedent variables in total: 276\n"
     ]
    }
   ],
   "source": [
    "# Merge Explanatory variables with dependent variable \"missing\"\n",
    "test_set = data_test_explanatory.merge(right=missing_report, how='left', left_on=\"full_name\", right_on='full_name')\n",
    "print(\"No. of depedent variables inserted:\", test_set.missing.count())\n",
    "print(\"No. of depedent variables in total:\", len(missing_report.dropna()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>full_name</th>\n",
       "      <th>sex</th>\n",
       "      <th>profession</th>\n",
       "      <th>hourly_salary</th>\n",
       "      <th>date_of_birth</th>\n",
       "      <th>weight</th>\n",
       "      <th>height</th>\n",
       "      <th>city_of_origin</th>\n",
       "      <th>last_seen</th>\n",
       "      <th>satisfaction_score</th>\n",
       "      <th>sleep_1</th>\n",
       "      <th>sleep_2</th>\n",
       "      <th>sleep_3</th>\n",
       "      <th>missing</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Ryan Weasley</td>\n",
       "      <td>m</td>\n",
       "      <td>bartender</td>\n",
       "      <td>11.9</td>\n",
       "      <td>2/15/83</td>\n",
       "      <td>78.0</td>\n",
       "      <td>185.0</td>\n",
       "      <td>Melbourne</td>\n",
       "      <td>2208-02-23</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8.3</td>\n",
       "      <td>7.2</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jason Bradbury</td>\n",
       "      <td>m</td>\n",
       "      <td>developer</td>\n",
       "      <td>14.3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>83.0</td>\n",
       "      <td>162.0</td>\n",
       "      <td>Seattle</td>\n",
       "      <td>2208-02-18</td>\n",
       "      <td>0.672414</td>\n",
       "      <td>6.6</td>\n",
       "      <td>7.7</td>\n",
       "      <td>7.2</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>f</td>\n",
       "      <td>developer</td>\n",
       "      <td>14.0</td>\n",
       "      <td>5/25/64</td>\n",
       "      <td>84.0</td>\n",
       "      <td>171.0</td>\n",
       "      <td>Brisbane</td>\n",
       "      <td>2208-02-22</td>\n",
       "      <td>0.597701</td>\n",
       "      <td>8.4</td>\n",
       "      <td>8.1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Jose Huxley</td>\n",
       "      <td>m</td>\n",
       "      <td>clerk</td>\n",
       "      <td>16.3</td>\n",
       "      <td>8/6/62</td>\n",
       "      <td>NaN</td>\n",
       "      <td>182.0</td>\n",
       "      <td>Sydney</td>\n",
       "      <td>2208-02-24</td>\n",
       "      <td>0.718391</td>\n",
       "      <td>8.2</td>\n",
       "      <td>5.6</td>\n",
       "      <td>9.8</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Willie Woolf</td>\n",
       "      <td>m</td>\n",
       "      <td>bookkeeper</td>\n",
       "      <td>16.0</td>\n",
       "      <td>2/17/75</td>\n",
       "      <td>94.0</td>\n",
       "      <td>187.0</td>\n",
       "      <td>Vancouver</td>\n",
       "      <td>2208-02-24</td>\n",
       "      <td>0.718391</td>\n",
       "      <td>8.7</td>\n",
       "      <td>7.9</td>\n",
       "      <td>8.1</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4675</th>\n",
       "      <td>NaN</td>\n",
       "      <td>f</td>\n",
       "      <td>server</td>\n",
       "      <td>10.3</td>\n",
       "      <td>10/3/87</td>\n",
       "      <td>72.0</td>\n",
       "      <td>195.0</td>\n",
       "      <td>Brisbane</td>\n",
       "      <td>2208-02-24</td>\n",
       "      <td>0.574713</td>\n",
       "      <td>8.1</td>\n",
       "      <td>8.4</td>\n",
       "      <td>8.2</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4676</th>\n",
       "      <td>Ann Martin</td>\n",
       "      <td>f</td>\n",
       "      <td>cashier</td>\n",
       "      <td>9.9</td>\n",
       "      <td>9/13/92</td>\n",
       "      <td>68.0</td>\n",
       "      <td>176.0</td>\n",
       "      <td>Melbourne</td>\n",
       "      <td>2208-02-16</td>\n",
       "      <td>0.465517</td>\n",
       "      <td>8.4</td>\n",
       "      <td>7.9</td>\n",
       "      <td>6.2</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4677</th>\n",
       "      <td>Raymond Laurie</td>\n",
       "      <td>m</td>\n",
       "      <td>trader</td>\n",
       "      <td>15.0</td>\n",
       "      <td>9/12/96</td>\n",
       "      <td>80.0</td>\n",
       "      <td>190.0</td>\n",
       "      <td>Toronto</td>\n",
       "      <td>2208-02-16</td>\n",
       "      <td>0.781609</td>\n",
       "      <td>8.9</td>\n",
       "      <td>8.6</td>\n",
       "      <td>8.3</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4678</th>\n",
       "      <td>Donald Jackson</td>\n",
       "      <td>m</td>\n",
       "      <td>plant worker</td>\n",
       "      <td>10.1</td>\n",
       "      <td>10/13/84</td>\n",
       "      <td>87.0</td>\n",
       "      <td>177.0</td>\n",
       "      <td>Brisbane</td>\n",
       "      <td>2208-02-15</td>\n",
       "      <td>0.545977</td>\n",
       "      <td>6.7</td>\n",
       "      <td>7.2</td>\n",
       "      <td>7.7</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4679</th>\n",
       "      <td>Evelyn Williams</td>\n",
       "      <td>f</td>\n",
       "      <td>developer</td>\n",
       "      <td>13.7</td>\n",
       "      <td>3/26/87</td>\n",
       "      <td>65.0</td>\n",
       "      <td>166.0</td>\n",
       "      <td>London</td>\n",
       "      <td>2208-02-24</td>\n",
       "      <td>0.770115</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.3</td>\n",
       "      <td>6.5</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>4680 rows × 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            full_name sex    profession  hourly_salary date_of_birth  weight  \\\n",
       "0        Ryan Weasley   m     bartender           11.9       2/15/83    78.0   \n",
       "1      Jason Bradbury   m     developer           14.3           NaN    83.0   \n",
       "2                 NaN   f     developer           14.0       5/25/64    84.0   \n",
       "3         Jose Huxley   m         clerk           16.3        8/6/62     NaN   \n",
       "4        Willie Woolf   m    bookkeeper           16.0       2/17/75    94.0   \n",
       "...               ...  ..           ...            ...           ...     ...   \n",
       "4675              NaN   f        server           10.3       10/3/87    72.0   \n",
       "4676       Ann Martin   f       cashier            9.9       9/13/92    68.0   \n",
       "4677   Raymond Laurie   m        trader           15.0       9/12/96    80.0   \n",
       "4678   Donald Jackson   m  plant worker           10.1      10/13/84    87.0   \n",
       "4679  Evelyn Williams   f     developer           13.7       3/26/87    65.0   \n",
       "\n",
       "      height city_of_origin   last_seen  satisfaction_score  sleep_1  sleep_2  \\\n",
       "0      185.0      Melbourne  2208-02-23                 NaN      8.3      7.2   \n",
       "1      162.0        Seattle  2208-02-18            0.672414      6.6      7.7   \n",
       "2      171.0       Brisbane  2208-02-22            0.597701      8.4      8.1   \n",
       "3      182.0         Sydney  2208-02-24            0.718391      8.2      5.6   \n",
       "4      187.0      Vancouver  2208-02-24            0.718391      8.7      7.9   \n",
       "...      ...            ...         ...                 ...      ...      ...   \n",
       "4675   195.0       Brisbane  2208-02-24            0.574713      8.1      8.4   \n",
       "4676   176.0      Melbourne  2208-02-16            0.465517      8.4      7.9   \n",
       "4677   190.0        Toronto  2208-02-16            0.781609      8.9      8.6   \n",
       "4678   177.0       Brisbane  2208-02-15            0.545977      6.7      7.2   \n",
       "4679   166.0         London  2208-02-24            0.770115      9.0      9.3   \n",
       "\n",
       "      sleep_3  missing  \n",
       "0         7.0      0.0  \n",
       "1         7.2      0.0  \n",
       "2         NaN      0.0  \n",
       "3         9.8      0.0  \n",
       "4         8.1      0.0  \n",
       "...       ...      ...  \n",
       "4675      8.2      0.0  \n",
       "4676      6.2      0.0  \n",
       "4677      8.3      0.0  \n",
       "4678      7.7      0.0  \n",
       "4679      6.5      0.0  \n",
       "\n",
       "[4680 rows x 14 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make missing a binary classication column\n",
    "train_set.missing = train_set.missing.replace(np.nan, 0)\n",
    "test_set.missing = test_set.missing.replace(np.nan, 0)\n",
    "test_set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>hairdresser</th>\n",
       "      <th>counselor</th>\n",
       "      <th>psychologist</th>\n",
       "      <th>doctor</th>\n",
       "      <th>dentist</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>date</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2208-02-23</th>\n",
       "      <td>Christian Lee</td>\n",
       "      <td>Brandon O'Hara</td>\n",
       "      <td>Zachary Dickens</td>\n",
       "      <td>Patrick Lennon</td>\n",
       "      <td>Katherine Heller</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2208-02-22</th>\n",
       "      <td>Isabella Huxley</td>\n",
       "      <td>Paul Waters</td>\n",
       "      <td>Benjamin Roth</td>\n",
       "      <td>Anthony Vonnegut</td>\n",
       "      <td>Laura Mikkelsen</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2208-02-24</th>\n",
       "      <td>Jason Hernandez</td>\n",
       "      <td>Joseph Watson</td>\n",
       "      <td>Zachary Dickens</td>\n",
       "      <td>Anthony Vonnegut</td>\n",
       "      <td>Philip Gonzalez</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2208-02-18</th>\n",
       "      <td>Isabella McConaughey</td>\n",
       "      <td>Jordan Garcia</td>\n",
       "      <td>Janet Davis</td>\n",
       "      <td>Anthony Vonnegut</td>\n",
       "      <td>Steven Lewis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2208-02-19</th>\n",
       "      <td>Beverly Hernandez</td>\n",
       "      <td>Michael Lee</td>\n",
       "      <td>Frank Jagger</td>\n",
       "      <td>William Walker</td>\n",
       "      <td>Steven Lewis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2208-02-17</th>\n",
       "      <td>Beverly Hernandez</td>\n",
       "      <td>Kathryn Gallager</td>\n",
       "      <td>Zachary Dickens</td>\n",
       "      <td>William Walker</td>\n",
       "      <td>Joan Gilmour</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2208-02-12</th>\n",
       "      <td>Frances Gallager</td>\n",
       "      <td>Jordan Garcia</td>\n",
       "      <td>Kelly Orwell</td>\n",
       "      <td>William Walker</td>\n",
       "      <td>Gerald Roth</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2208-02-13</th>\n",
       "      <td>Sophia Hall</td>\n",
       "      <td>Kelly Granger</td>\n",
       "      <td>Donna Smith</td>\n",
       "      <td>William Walker</td>\n",
       "      <td>Laura Mikkelsen</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2208-02-14</th>\n",
       "      <td>Christian Lee</td>\n",
       "      <td>Abigail Martinez</td>\n",
       "      <td>John Morrison</td>\n",
       "      <td>Anthony Vonnegut</td>\n",
       "      <td>Gerald Roth</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2208-02-20</th>\n",
       "      <td>Isabella McConaughey</td>\n",
       "      <td>Brandon O'Hara</td>\n",
       "      <td>Brandon Lawrence</td>\n",
       "      <td>Anthony Vonnegut</td>\n",
       "      <td>Jennifer Thompson</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2208-02-15</th>\n",
       "      <td>Christian Lee</td>\n",
       "      <td>Bradley Woolf</td>\n",
       "      <td>Frank Jagger</td>\n",
       "      <td>Patrick Lennon</td>\n",
       "      <td>Marie McConaughey</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2208-02-16</th>\n",
       "      <td>Kathryn Mikkelsen</td>\n",
       "      <td>Michael Lee</td>\n",
       "      <td>Brandon Lawrence</td>\n",
       "      <td>Patrick Lennon</td>\n",
       "      <td>Shirley Kafka</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2208-02-21</th>\n",
       "      <td>Christian Jackson</td>\n",
       "      <td>Kathryn Gallager</td>\n",
       "      <td>Joan Griffin</td>\n",
       "      <td>Anthony Vonnegut</td>\n",
       "      <td>Brandon Weasley</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     hairdresser         counselor      psychologist  \\\n",
       "date                                                                   \n",
       "2208-02-23         Christian Lee    Brandon O'Hara   Zachary Dickens   \n",
       "2208-02-22       Isabella Huxley       Paul Waters     Benjamin Roth   \n",
       "2208-02-24       Jason Hernandez     Joseph Watson   Zachary Dickens   \n",
       "2208-02-18  Isabella McConaughey     Jordan Garcia       Janet Davis   \n",
       "2208-02-19     Beverly Hernandez       Michael Lee      Frank Jagger   \n",
       "2208-02-17     Beverly Hernandez  Kathryn Gallager   Zachary Dickens   \n",
       "2208-02-12      Frances Gallager     Jordan Garcia      Kelly Orwell   \n",
       "2208-02-13           Sophia Hall     Kelly Granger       Donna Smith   \n",
       "2208-02-14         Christian Lee  Abigail Martinez     John Morrison   \n",
       "2208-02-20  Isabella McConaughey    Brandon O'Hara  Brandon Lawrence   \n",
       "2208-02-15         Christian Lee     Bradley Woolf      Frank Jagger   \n",
       "2208-02-16     Kathryn Mikkelsen       Michael Lee  Brandon Lawrence   \n",
       "2208-02-21     Christian Jackson  Kathryn Gallager      Joan Griffin   \n",
       "\n",
       "                      doctor            dentist  \n",
       "date                                             \n",
       "2208-02-23    Patrick Lennon   Katherine Heller  \n",
       "2208-02-22  Anthony Vonnegut    Laura Mikkelsen  \n",
       "2208-02-24  Anthony Vonnegut    Philip Gonzalez  \n",
       "2208-02-18  Anthony Vonnegut       Steven Lewis  \n",
       "2208-02-19    William Walker       Steven Lewis  \n",
       "2208-02-17    William Walker       Joan Gilmour  \n",
       "2208-02-12    William Walker        Gerald Roth  \n",
       "2208-02-13    William Walker    Laura Mikkelsen  \n",
       "2208-02-14  Anthony Vonnegut        Gerald Roth  \n",
       "2208-02-20  Anthony Vonnegut  Jennifer Thompson  \n",
       "2208-02-15    Patrick Lennon  Marie McConaughey  \n",
       "2208-02-16    Patrick Lennon      Shirley Kafka  \n",
       "2208-02-21  Anthony Vonnegut    Brandon Weasley  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "shift_report = shift_report.set_index(\"date\")\n",
    "shift_report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def on_shift(name, date):\n",
    "    return 1 if name in shift_report.loc[date, :].values else 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "on_shift(\"Christian Lee\", \"2208-02-23\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_set.shape[1] == test_set.shape[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Missing Data:\n",
    "An approach: \n",
    "\n",
    "- remove all records having a missing data point\n",
    "- Another approch: replace all missing data points with mean of that column\n",
    "\n",
    "Since first approach removes half the data, I *would* go for the other \\\n",
    "Due to lack of time, I went for the second"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_set.shape[1] == test_set.shape[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_set_no_na = train_set.dropna()\n",
    "test_set_no_na = test_set.dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-17-ff03c9c2539a>:1: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  train_set_no_na['on_shift_last_day_seen'] = train_set_no_na.apply(lambda row: on_shift(row['full_name'], row['last_seen']) , axis=1)\n",
      "<ipython-input-17-ff03c9c2539a>:2: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  test_set_no_na['on_shift_last_day_seen'] = test_set_no_na.apply(lambda row: on_shift(row['full_name'], row['last_seen']) , axis=1)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_set_no_na['on_shift_last_day_seen'] = train_set_no_na.apply(lambda row: on_shift(row['full_name'], row['last_seen']) , axis=1)\n",
    "test_set_no_na['on_shift_last_day_seen'] = test_set_no_na.apply(lambda row: on_shift(row['full_name'], row['last_seen']) , axis=1)\n",
    "train_set_no_na['on_shift_last_day_seen'].sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\juv\\Anaconda3\\lib\\site-packages\\pandas\\core\\generic.py:5168: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  self[name] = value\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "4      2208-02-24\n",
       "6      2208-02-17\n",
       "8      2208-02-20\n",
       "11     2208-02-16\n",
       "17     2208-02-24\n",
       "          ...    \n",
       "4674   2208-02-19\n",
       "4676   2208-02-16\n",
       "4677   2208-02-16\n",
       "4678   2208-02-15\n",
       "4679   2208-02-24\n",
       "Name: last_seen, Length: 2372, dtype: datetime64[ns]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_set_no_na.last_seen = train_set_no_na.last_seen.apply(lambda x: datetime.strptime(str(x), \"%Y-%m-%d\")) \n",
    "test_set_no_na.last_seen = test_set_no_na.last_seen.apply(lambda x: datetime.strptime(str(x), \"%Y-%m-%d\")) \n",
    "# train_set_no_na.last_seen\n",
    "test_set_no_na.last_seen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "today = datetime(2208, 2, 24)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_set_no_na.shape[1] == test_set_no_na.shape[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-21-05bd32732cb9>:1: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  train_set_no_na['last_seen_in_days'] = train_set_no_na.apply(lambda row: (today - row.last_seen).days, axis=1)\n",
      "<ipython-input-21-05bd32732cb9>:2: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  test_set_no_na['last_seen_in_days'] = test_set_no_na.apply(lambda row: (today - row.last_seen).days, axis=1)\n"
     ]
    }
   ],
   "source": [
    "train_set_no_na['last_seen_in_days'] = train_set_no_na.apply(lambda row: (today - row.last_seen).days, axis=1)\n",
    "test_set_no_na['last_seen_in_days'] = test_set_no_na.apply(lambda row: (today - row.last_seen).days, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_set_no_na.shape[1] == test_set_no_na.shape[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-23-9fc263082a39>:1: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  train_set_no_na['male'] = train_set_no_na.apply(lambda row: 1 if row.sex == \"m\" else 0, axis=1)\n",
      "<ipython-input-23-9fc263082a39>:2: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  test_set_no_na['male'] = test_set_no_na.apply(lambda row: 1 if row.sex == \"m\" else 0, axis=1)\n"
     ]
    }
   ],
   "source": [
    "train_set_no_na['male'] = train_set_no_na.apply(lambda row: 1 if row.sex == \"m\" else 0, axis=1)\n",
    "test_set_no_na['male'] = test_set_no_na.apply(lambda row: 1 if row.sex == \"m\" else 0, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_set_no_na.shape[1] == test_set_no_na.shape[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# train_set_no_na.profession.nunique()\n",
    "test_set_no_na.profession.nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# train_set_no_na.city_of_origin.nunique()\n",
    "# test_set_no_na"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "length = len(train_set)\n",
    "\n",
    "length = len(train_set_no_na)\n",
    "\n",
    "cols = list(train_set.columns)\n",
    "values = []\n",
    "\n",
    "for column in list(train_set.columns):\n",
    "    values.append(length - train_set_no_na[column].count())\n",
    "    print(f\"Missing values for column {column}: \" + str(length - train_set_no_na[column].count()))\n",
    "    \n",
    "plt.bar(cols, values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### One Hot Encoding\n",
    "To ensure the model to consume the data set I apply OHC formatting the categorical columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bartender</th>\n",
       "      <th>bookkeeper</th>\n",
       "      <th>carpenter</th>\n",
       "      <th>cashier</th>\n",
       "      <th>clerk</th>\n",
       "      <th>cook</th>\n",
       "      <th>counselor</th>\n",
       "      <th>dentist</th>\n",
       "      <th>developer</th>\n",
       "      <th>doctor</th>\n",
       "      <th>...</th>\n",
       "      <th>gardener</th>\n",
       "      <th>hairdresser</th>\n",
       "      <th>janitor</th>\n",
       "      <th>laborer</th>\n",
       "      <th>manager</th>\n",
       "      <th>plant worker</th>\n",
       "      <th>police officer</th>\n",
       "      <th>psychologist</th>\n",
       "      <th>server</th>\n",
       "      <th>trader</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10913</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10914</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10915</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10916</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10917</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5571 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       bartender  bookkeeper  carpenter  cashier  clerk  cook  counselor  \\\n",
       "0              0           0          0        0      0     0          0   \n",
       "2              0           0          0        0      0     0          0   \n",
       "6              0           0          0        0      0     0          0   \n",
       "7              0           0          0        0      0     0          0   \n",
       "10             0           1          0        0      0     0          0   \n",
       "...          ...         ...        ...      ...    ...   ...        ...   \n",
       "10913          0           0          0        0      0     0          0   \n",
       "10914          0           0          1        0      0     0          0   \n",
       "10915          0           0          0        0      0     1          0   \n",
       "10916          0           0          0        0      0     0          0   \n",
       "10917          0           0          0        0      0     0          0   \n",
       "\n",
       "       dentist  developer  doctor  ...  gardener  hairdresser  janitor  \\\n",
       "0            0          0       0  ...         0            0        0   \n",
       "2            0          0       0  ...         0            0        0   \n",
       "6            0          1       0  ...         0            0        0   \n",
       "7            0          0       0  ...         0            0        0   \n",
       "10           0          0       0  ...         0            0        0   \n",
       "...        ...        ...     ...  ...       ...          ...      ...   \n",
       "10913        0          0       0  ...         0            0        0   \n",
       "10914        0          0       0  ...         0            0        0   \n",
       "10915        0          0       0  ...         0            0        0   \n",
       "10916        0          0       0  ...         0            0        0   \n",
       "10917        0          0       0  ...         0            0        0   \n",
       "\n",
       "       laborer  manager  plant worker  police officer  psychologist  server  \\\n",
       "0            1        0             0               0             0       0   \n",
       "2            0        0             0               0             0       0   \n",
       "6            0        0             0               0             0       0   \n",
       "7            0        0             1               0             0       0   \n",
       "10           0        0             0               0             0       0   \n",
       "...        ...      ...           ...             ...           ...     ...   \n",
       "10913        1        0             0               0             0       0   \n",
       "10914        0        0             0               0             0       0   \n",
       "10915        0        0             0               0             0       0   \n",
       "10916        0        0             0               1             0       0   \n",
       "10917        0        0             0               0             0       1   \n",
       "\n",
       "       trader  \n",
       "0           0  \n",
       "2           0  \n",
       "6           0  \n",
       "7           0  \n",
       "10          0  \n",
       "...       ...  \n",
       "10913       0  \n",
       "10914       0  \n",
       "10915       0  \n",
       "10916       0  \n",
       "10917       0  \n",
       "\n",
       "[5571 rows x 21 columns]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "profession_dummies_train = pd.get_dummies(train_set_no_na.profession)\n",
    "profession_dummies_test = pd.get_dummies(test_set_no_na.profession)\n",
    "profession_dummies_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(profession_dummies_train.columns) == list(profession_dummies_test.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"doctor\" in list(profession_dummies_train.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"doctor\" in list(profession_dummies_test.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bartender</th>\n",
       "      <th>bookkeeper</th>\n",
       "      <th>carpenter</th>\n",
       "      <th>cashier</th>\n",
       "      <th>clerk</th>\n",
       "      <th>cook</th>\n",
       "      <th>counselor</th>\n",
       "      <th>dentist</th>\n",
       "      <th>developer</th>\n",
       "      <th>electrician</th>\n",
       "      <th>...</th>\n",
       "      <th>hairdresser</th>\n",
       "      <th>janitor</th>\n",
       "      <th>laborer</th>\n",
       "      <th>manager</th>\n",
       "      <th>plant worker</th>\n",
       "      <th>police officer</th>\n",
       "      <th>psychologist</th>\n",
       "      <th>server</th>\n",
       "      <th>trader</th>\n",
       "      <th>doctor</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4674</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4676</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4677</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4678</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4679</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2372 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      bartender  bookkeeper  carpenter  cashier  clerk  cook  counselor  \\\n",
       "4             0           1          0        0      0     0          0   \n",
       "6             0           0          0        0      0     1          0   \n",
       "8             0           0          0        0      0     0          0   \n",
       "11            0           0          0        0      1     0          0   \n",
       "17            0           0          0        0      0     0          0   \n",
       "...         ...         ...        ...      ...    ...   ...        ...   \n",
       "4674          0           0          0        0      0     0          0   \n",
       "4676          0           0          0        1      0     0          0   \n",
       "4677          0           0          0        0      0     0          0   \n",
       "4678          0           0          0        0      0     0          0   \n",
       "4679          0           0          0        0      0     0          0   \n",
       "\n",
       "      dentist  developer  electrician  ...  hairdresser  janitor  laborer  \\\n",
       "4           0          0            0  ...            0        0        0   \n",
       "6           0          0            0  ...            0        0        0   \n",
       "8           0          0            0  ...            0        0        1   \n",
       "11          0          0            0  ...            0        0        0   \n",
       "17          0          0            0  ...            0        0        0   \n",
       "...       ...        ...          ...  ...          ...      ...      ...   \n",
       "4674        0          0            0  ...            0        0        0   \n",
       "4676        0          0            0  ...            0        0        0   \n",
       "4677        0          0            0  ...            0        0        0   \n",
       "4678        0          0            0  ...            0        0        0   \n",
       "4679        0          1            0  ...            0        0        0   \n",
       "\n",
       "      manager  plant worker  police officer  psychologist  server  trader  \\\n",
       "4           0             0               0             0       0       0   \n",
       "6           0             0               0             0       0       0   \n",
       "8           0             0               0             0       0       0   \n",
       "11          0             0               0             0       0       0   \n",
       "17          0             0               0             0       0       1   \n",
       "...       ...           ...             ...           ...     ...     ...   \n",
       "4674        0             0               1             0       0       0   \n",
       "4676        0             0               0             0       0       0   \n",
       "4677        0             0               0             0       0       1   \n",
       "4678        0             1               0             0       0       0   \n",
       "4679        0             0               0             0       0       0   \n",
       "\n",
       "      doctor  \n",
       "4          0  \n",
       "6          0  \n",
       "8          0  \n",
       "11         0  \n",
       "17         0  \n",
       "...      ...  \n",
       "4674       0  \n",
       "4676       0  \n",
       "4677       0  \n",
       "4678       0  \n",
       "4679       0  \n",
       "\n",
       "[2372 rows x 21 columns]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "profession_dummies_test['doctor'] = 0\n",
    "profession_dummies_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_set_no_na = pd.concat([train_set_no_na, profession_dummies_train], axis=1)\n",
    "test_set_no_na = pd.concat([test_set_no_na, profession_dummies_test], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_set_no_na.shape[1] == test_set_no_na.shape[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "origin_dummies_train = pd.get_dummies(train_set_no_na.city_of_origin)\n",
    "origin_dummies_test = pd.get_dummies(test_set_no_na.city_of_origin)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(origin_dummies_train.columns) == list(origin_dummies_test.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>full_name</th>\n",
       "      <th>sex</th>\n",
       "      <th>profession</th>\n",
       "      <th>hourly_salary</th>\n",
       "      <th>date_of_birth</th>\n",
       "      <th>weight</th>\n",
       "      <th>height</th>\n",
       "      <th>city_of_origin</th>\n",
       "      <th>last_seen</th>\n",
       "      <th>satisfaction_score</th>\n",
       "      <th>...</th>\n",
       "      <th>London</th>\n",
       "      <th>Los Angeles</th>\n",
       "      <th>Manchester</th>\n",
       "      <th>Melbourne</th>\n",
       "      <th>Montreal</th>\n",
       "      <th>New York</th>\n",
       "      <th>Seattle</th>\n",
       "      <th>Sydney</th>\n",
       "      <th>Toronto</th>\n",
       "      <th>Vancouver</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Emily Holmes</td>\n",
       "      <td>f</td>\n",
       "      <td>laborer</td>\n",
       "      <td>14.0</td>\n",
       "      <td>10/2/73</td>\n",
       "      <td>84.0</td>\n",
       "      <td>175.0</td>\n",
       "      <td>Manchester</td>\n",
       "      <td>2208-02-22</td>\n",
       "      <td>0.706897</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Matthew Wilson</td>\n",
       "      <td>m</td>\n",
       "      <td>electrician</td>\n",
       "      <td>14.0</td>\n",
       "      <td>7/6/71</td>\n",
       "      <td>66.0</td>\n",
       "      <td>179.0</td>\n",
       "      <td>Toronto</td>\n",
       "      <td>2208-02-22</td>\n",
       "      <td>0.580460</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Ashley White</td>\n",
       "      <td>f</td>\n",
       "      <td>developer</td>\n",
       "      <td>13.5</td>\n",
       "      <td>12/20/90</td>\n",
       "      <td>93.0</td>\n",
       "      <td>195.0</td>\n",
       "      <td>Vancouver</td>\n",
       "      <td>2208-02-19</td>\n",
       "      <td>0.718391</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Carl Griffin</td>\n",
       "      <td>m</td>\n",
       "      <td>plant worker</td>\n",
       "      <td>9.5</td>\n",
       "      <td>10/8/69</td>\n",
       "      <td>78.0</td>\n",
       "      <td>195.0</td>\n",
       "      <td>Birmingham</td>\n",
       "      <td>2208-02-22</td>\n",
       "      <td>0.425287</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Sean Perez</td>\n",
       "      <td>m</td>\n",
       "      <td>bookkeeper</td>\n",
       "      <td>16.3</td>\n",
       "      <td>12/15/76</td>\n",
       "      <td>88.0</td>\n",
       "      <td>163.0</td>\n",
       "      <td>Seattle</td>\n",
       "      <td>2208-02-17</td>\n",
       "      <td>0.798851</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10913</th>\n",
       "      <td>Keith Smith</td>\n",
       "      <td>m</td>\n",
       "      <td>laborer</td>\n",
       "      <td>14.0</td>\n",
       "      <td>12/3/82</td>\n",
       "      <td>70.0</td>\n",
       "      <td>190.0</td>\n",
       "      <td>Brisbane</td>\n",
       "      <td>2208-02-12</td>\n",
       "      <td>0.701149</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10914</th>\n",
       "      <td>Sarah Perez</td>\n",
       "      <td>f</td>\n",
       "      <td>carpenter</td>\n",
       "      <td>14.3</td>\n",
       "      <td>1/27/81</td>\n",
       "      <td>67.0</td>\n",
       "      <td>177.0</td>\n",
       "      <td>Manchester</td>\n",
       "      <td>2208-02-12</td>\n",
       "      <td>0.770115</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10915</th>\n",
       "      <td>Larry White</td>\n",
       "      <td>m</td>\n",
       "      <td>cook</td>\n",
       "      <td>12.0</td>\n",
       "      <td>5/18/96</td>\n",
       "      <td>75.0</td>\n",
       "      <td>170.0</td>\n",
       "      <td>London</td>\n",
       "      <td>2208-02-15</td>\n",
       "      <td>0.724138</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10916</th>\n",
       "      <td>Christopher Wilson</td>\n",
       "      <td>m</td>\n",
       "      <td>police officer</td>\n",
       "      <td>13.7</td>\n",
       "      <td>1/9/80</td>\n",
       "      <td>65.0</td>\n",
       "      <td>175.0</td>\n",
       "      <td>Manchester</td>\n",
       "      <td>2208-02-23</td>\n",
       "      <td>0.614943</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10917</th>\n",
       "      <td>Scott Laurie</td>\n",
       "      <td>m</td>\n",
       "      <td>server</td>\n",
       "      <td>10.3</td>\n",
       "      <td>4/4/62</td>\n",
       "      <td>66.0</td>\n",
       "      <td>164.0</td>\n",
       "      <td>Vancouver</td>\n",
       "      <td>2208-02-20</td>\n",
       "      <td>0.316092</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5571 rows × 50 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                full_name sex      profession  hourly_salary date_of_birth  \\\n",
       "0            Emily Holmes   f         laborer           14.0       10/2/73   \n",
       "2          Matthew Wilson   m     electrician           14.0        7/6/71   \n",
       "6            Ashley White   f       developer           13.5      12/20/90   \n",
       "7            Carl Griffin   m    plant worker            9.5       10/8/69   \n",
       "10             Sean Perez   m      bookkeeper           16.3      12/15/76   \n",
       "...                   ...  ..             ...            ...           ...   \n",
       "10913         Keith Smith   m         laborer           14.0       12/3/82   \n",
       "10914         Sarah Perez   f       carpenter           14.3       1/27/81   \n",
       "10915         Larry White   m            cook           12.0       5/18/96   \n",
       "10916  Christopher Wilson   m  police officer           13.7        1/9/80   \n",
       "10917        Scott Laurie   m          server           10.3        4/4/62   \n",
       "\n",
       "       weight  height city_of_origin  last_seen  satisfaction_score  ...  \\\n",
       "0        84.0   175.0     Manchester 2208-02-22            0.706897  ...   \n",
       "2        66.0   179.0        Toronto 2208-02-22            0.580460  ...   \n",
       "6        93.0   195.0      Vancouver 2208-02-19            0.718391  ...   \n",
       "7        78.0   195.0     Birmingham 2208-02-22            0.425287  ...   \n",
       "10       88.0   163.0        Seattle 2208-02-17            0.798851  ...   \n",
       "...       ...     ...            ...        ...                 ...  ...   \n",
       "10913    70.0   190.0       Brisbane 2208-02-12            0.701149  ...   \n",
       "10914    67.0   177.0     Manchester 2208-02-12            0.770115  ...   \n",
       "10915    75.0   170.0         London 2208-02-15            0.724138  ...   \n",
       "10916    65.0   175.0     Manchester 2208-02-23            0.614943  ...   \n",
       "10917    66.0   164.0      Vancouver 2208-02-20            0.316092  ...   \n",
       "\n",
       "       London  Los Angeles  Manchester  Melbourne  Montreal  New York  \\\n",
       "0           0            0           1          0         0         0   \n",
       "2           0            0           0          0         0         0   \n",
       "6           0            0           0          0         0         0   \n",
       "7           0            0           0          0         0         0   \n",
       "10          0            0           0          0         0         0   \n",
       "...       ...          ...         ...        ...       ...       ...   \n",
       "10913       0            0           0          0         0         0   \n",
       "10914       0            0           1          0         0         0   \n",
       "10915       1            0           0          0         0         0   \n",
       "10916       0            0           1          0         0         0   \n",
       "10917       0            0           0          0         0         0   \n",
       "\n",
       "       Seattle  Sydney  Toronto  Vancouver  \n",
       "0            0       0        0          0  \n",
       "2            0       0        1          0  \n",
       "6            0       0        0          1  \n",
       "7            0       0        0          0  \n",
       "10           1       0        0          0  \n",
       "...        ...     ...      ...        ...  \n",
       "10913        0       0        0          0  \n",
       "10914        0       0        0          0  \n",
       "10915        0       0        0          0  \n",
       "10916        0       0        0          0  \n",
       "10917        0       0        0          1  \n",
       "\n",
       "[5571 rows x 50 columns]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_set_no_na = pd.concat([train_set_no_na, origin_dummies_train], axis=1)\n",
    "test_set_no_na = pd.concat([test_set_no_na, origin_dummies_test], axis=1)\n",
    "train_set_no_na"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_set_no_na.shape[1] == test_set_no_na.shape[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "columns_not_to_use_anymore = ['full_name', 'date_of_birth', 'sex', 'profession', 'city_of_origin', 'last_seen']\n",
    "estimation_set_train = train_set_no_na.drop(columns_not_to_use_anymore, axis=1)\n",
    "estimation_set_test = test_set_no_na.drop(columns_not_to_use_anymore, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "44"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "estimation_set_train.shape[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "44"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "estimation_set_test.shape[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>hourly_salary</th>\n",
       "      <th>weight</th>\n",
       "      <th>height</th>\n",
       "      <th>satisfaction_score</th>\n",
       "      <th>sleep_1</th>\n",
       "      <th>sleep_2</th>\n",
       "      <th>sleep_3</th>\n",
       "      <th>missing</th>\n",
       "      <th>on_shift_last_day_seen</th>\n",
       "      <th>last_seen_in_days</th>\n",
       "      <th>...</th>\n",
       "      <th>London</th>\n",
       "      <th>Los Angeles</th>\n",
       "      <th>Manchester</th>\n",
       "      <th>Melbourne</th>\n",
       "      <th>Montreal</th>\n",
       "      <th>New York</th>\n",
       "      <th>Seattle</th>\n",
       "      <th>Sydney</th>\n",
       "      <th>Toronto</th>\n",
       "      <th>Vancouver</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>14.0</td>\n",
       "      <td>84.0</td>\n",
       "      <td>175.0</td>\n",
       "      <td>0.706897</td>\n",
       "      <td>7.0</td>\n",
       "      <td>8.8</td>\n",
       "      <td>11.3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>14.0</td>\n",
       "      <td>66.0</td>\n",
       "      <td>179.0</td>\n",
       "      <td>0.580460</td>\n",
       "      <td>8.5</td>\n",
       "      <td>6.2</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>13.5</td>\n",
       "      <td>93.0</td>\n",
       "      <td>195.0</td>\n",
       "      <td>0.718391</td>\n",
       "      <td>7.6</td>\n",
       "      <td>8.2</td>\n",
       "      <td>7.3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>9.5</td>\n",
       "      <td>78.0</td>\n",
       "      <td>195.0</td>\n",
       "      <td>0.425287</td>\n",
       "      <td>9.3</td>\n",
       "      <td>6.8</td>\n",
       "      <td>9.2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>16.3</td>\n",
       "      <td>88.0</td>\n",
       "      <td>163.0</td>\n",
       "      <td>0.798851</td>\n",
       "      <td>8.2</td>\n",
       "      <td>8.0</td>\n",
       "      <td>5.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10913</th>\n",
       "      <td>14.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>190.0</td>\n",
       "      <td>0.701149</td>\n",
       "      <td>9.0</td>\n",
       "      <td>7.7</td>\n",
       "      <td>10.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10914</th>\n",
       "      <td>14.3</td>\n",
       "      <td>67.0</td>\n",
       "      <td>177.0</td>\n",
       "      <td>0.770115</td>\n",
       "      <td>7.4</td>\n",
       "      <td>9.6</td>\n",
       "      <td>7.3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10915</th>\n",
       "      <td>12.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>170.0</td>\n",
       "      <td>0.724138</td>\n",
       "      <td>7.2</td>\n",
       "      <td>7.9</td>\n",
       "      <td>8.9</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10916</th>\n",
       "      <td>13.7</td>\n",
       "      <td>65.0</td>\n",
       "      <td>175.0</td>\n",
       "      <td>0.614943</td>\n",
       "      <td>7.1</td>\n",
       "      <td>8.2</td>\n",
       "      <td>7.2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10917</th>\n",
       "      <td>10.3</td>\n",
       "      <td>66.0</td>\n",
       "      <td>164.0</td>\n",
       "      <td>0.316092</td>\n",
       "      <td>8.3</td>\n",
       "      <td>6.3</td>\n",
       "      <td>9.3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5571 rows × 44 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       hourly_salary  weight  height  satisfaction_score  sleep_1  sleep_2  \\\n",
       "0               14.0    84.0   175.0            0.706897      7.0      8.8   \n",
       "2               14.0    66.0   179.0            0.580460      8.5      6.2   \n",
       "6               13.5    93.0   195.0            0.718391      7.6      8.2   \n",
       "7                9.5    78.0   195.0            0.425287      9.3      6.8   \n",
       "10              16.3    88.0   163.0            0.798851      8.2      8.0   \n",
       "...              ...     ...     ...                 ...      ...      ...   \n",
       "10913           14.0    70.0   190.0            0.701149      9.0      7.7   \n",
       "10914           14.3    67.0   177.0            0.770115      7.4      9.6   \n",
       "10915           12.0    75.0   170.0            0.724138      7.2      7.9   \n",
       "10916           13.7    65.0   175.0            0.614943      7.1      8.2   \n",
       "10917           10.3    66.0   164.0            0.316092      8.3      6.3   \n",
       "\n",
       "       sleep_3  missing  on_shift_last_day_seen  last_seen_in_days  ...  \\\n",
       "0         11.3      0.0                       0                  2  ...   \n",
       "2          8.0      0.0                       0                  2  ...   \n",
       "6          7.3      0.0                       0                  5  ...   \n",
       "7          9.2      0.0                       0                  2  ...   \n",
       "10         5.5      0.0                       0                  7  ...   \n",
       "...        ...      ...                     ...                ...  ...   \n",
       "10913     10.5      0.0                       0                 12  ...   \n",
       "10914      7.3      0.0                       0                 12  ...   \n",
       "10915      8.9      0.0                       0                  9  ...   \n",
       "10916      7.2      0.0                       0                  1  ...   \n",
       "10917      9.3      0.0                       0                  4  ...   \n",
       "\n",
       "       London  Los Angeles  Manchester  Melbourne  Montreal  New York  \\\n",
       "0           0            0           1          0         0         0   \n",
       "2           0            0           0          0         0         0   \n",
       "6           0            0           0          0         0         0   \n",
       "7           0            0           0          0         0         0   \n",
       "10          0            0           0          0         0         0   \n",
       "...       ...          ...         ...        ...       ...       ...   \n",
       "10913       0            0           0          0         0         0   \n",
       "10914       0            0           1          0         0         0   \n",
       "10915       1            0           0          0         0         0   \n",
       "10916       0            0           1          0         0         0   \n",
       "10917       0            0           0          0         0         0   \n",
       "\n",
       "       Seattle  Sydney  Toronto  Vancouver  \n",
       "0            0       0        0          0  \n",
       "2            0       0        1          0  \n",
       "6            0       0        0          1  \n",
       "7            0       0        0          0  \n",
       "10           1       0        0          0  \n",
       "...        ...     ...      ...        ...  \n",
       "10913        0       0        0          0  \n",
       "10914        0       0        0          0  \n",
       "10915        0       0        0          0  \n",
       "10916        0       0        0          0  \n",
       "10917        0       0        0          1  \n",
       "\n",
       "[5571 rows x 44 columns]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "estimation_set_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>hourly_salary</th>\n",
       "      <th>weight</th>\n",
       "      <th>height</th>\n",
       "      <th>satisfaction_score</th>\n",
       "      <th>sleep_1</th>\n",
       "      <th>sleep_2</th>\n",
       "      <th>sleep_3</th>\n",
       "      <th>missing</th>\n",
       "      <th>on_shift_last_day_seen</th>\n",
       "      <th>last_seen_in_days</th>\n",
       "      <th>...</th>\n",
       "      <th>London</th>\n",
       "      <th>Los Angeles</th>\n",
       "      <th>Manchester</th>\n",
       "      <th>Melbourne</th>\n",
       "      <th>Montreal</th>\n",
       "      <th>New York</th>\n",
       "      <th>Seattle</th>\n",
       "      <th>Sydney</th>\n",
       "      <th>Toronto</th>\n",
       "      <th>Vancouver</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>16.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>187.0</td>\n",
       "      <td>0.718391</td>\n",
       "      <td>8.7</td>\n",
       "      <td>7.9</td>\n",
       "      <td>8.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>12.5</td>\n",
       "      <td>70.0</td>\n",
       "      <td>169.0</td>\n",
       "      <td>0.620690</td>\n",
       "      <td>8.8</td>\n",
       "      <td>8.3</td>\n",
       "      <td>7.9</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>13.9</td>\n",
       "      <td>79.0</td>\n",
       "      <td>181.0</td>\n",
       "      <td>0.614943</td>\n",
       "      <td>8.1</td>\n",
       "      <td>8.7</td>\n",
       "      <td>9.6</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>16.0</td>\n",
       "      <td>90.0</td>\n",
       "      <td>178.0</td>\n",
       "      <td>0.850575</td>\n",
       "      <td>8.2</td>\n",
       "      <td>7.8</td>\n",
       "      <td>9.4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>15.3</td>\n",
       "      <td>81.0</td>\n",
       "      <td>174.0</td>\n",
       "      <td>0.660920</td>\n",
       "      <td>8.5</td>\n",
       "      <td>8.5</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4674</th>\n",
       "      <td>14.2</td>\n",
       "      <td>75.0</td>\n",
       "      <td>173.0</td>\n",
       "      <td>0.597701</td>\n",
       "      <td>6.2</td>\n",
       "      <td>7.5</td>\n",
       "      <td>8.2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4676</th>\n",
       "      <td>9.9</td>\n",
       "      <td>68.0</td>\n",
       "      <td>176.0</td>\n",
       "      <td>0.465517</td>\n",
       "      <td>8.4</td>\n",
       "      <td>7.9</td>\n",
       "      <td>6.2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4677</th>\n",
       "      <td>15.0</td>\n",
       "      <td>80.0</td>\n",
       "      <td>190.0</td>\n",
       "      <td>0.781609</td>\n",
       "      <td>8.9</td>\n",
       "      <td>8.6</td>\n",
       "      <td>8.3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4678</th>\n",
       "      <td>10.1</td>\n",
       "      <td>87.0</td>\n",
       "      <td>177.0</td>\n",
       "      <td>0.545977</td>\n",
       "      <td>6.7</td>\n",
       "      <td>7.2</td>\n",
       "      <td>7.7</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4679</th>\n",
       "      <td>13.7</td>\n",
       "      <td>65.0</td>\n",
       "      <td>166.0</td>\n",
       "      <td>0.770115</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.3</td>\n",
       "      <td>6.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2372 rows × 44 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      hourly_salary  weight  height  satisfaction_score  sleep_1  sleep_2  \\\n",
       "4              16.0    94.0   187.0            0.718391      8.7      7.9   \n",
       "6              12.5    70.0   169.0            0.620690      8.8      8.3   \n",
       "8              13.9    79.0   181.0            0.614943      8.1      8.7   \n",
       "11             16.0    90.0   178.0            0.850575      8.2      7.8   \n",
       "17             15.3    81.0   174.0            0.660920      8.5      8.5   \n",
       "...             ...     ...     ...                 ...      ...      ...   \n",
       "4674           14.2    75.0   173.0            0.597701      6.2      7.5   \n",
       "4676            9.9    68.0   176.0            0.465517      8.4      7.9   \n",
       "4677           15.0    80.0   190.0            0.781609      8.9      8.6   \n",
       "4678           10.1    87.0   177.0            0.545977      6.7      7.2   \n",
       "4679           13.7    65.0   166.0            0.770115      9.0      9.3   \n",
       "\n",
       "      sleep_3  missing  on_shift_last_day_seen  last_seen_in_days  ...  \\\n",
       "4         8.1      0.0                       0                  0  ...   \n",
       "6         7.9      0.0                       0                  7  ...   \n",
       "8         9.6      0.0                       0                  4  ...   \n",
       "11        9.4      0.0                       0                  8  ...   \n",
       "17        9.0      0.0                       0                  0  ...   \n",
       "...       ...      ...                     ...                ...  ...   \n",
       "4674      8.2      0.0                       0                  5  ...   \n",
       "4676      6.2      0.0                       0                  8  ...   \n",
       "4677      8.3      0.0                       0                  8  ...   \n",
       "4678      7.7      0.0                       0                  9  ...   \n",
       "4679      6.5      0.0                       0                  0  ...   \n",
       "\n",
       "      London  Los Angeles  Manchester  Melbourne  Montreal  New York  Seattle  \\\n",
       "4          0            0           0          0         0         0        0   \n",
       "6          0            0           0          0         0         0        0   \n",
       "8          0            0           0          0         0         0        0   \n",
       "11         0            0           1          0         0         0        0   \n",
       "17         0            0           1          0         0         0        0   \n",
       "...      ...          ...         ...        ...       ...       ...      ...   \n",
       "4674       0            0           0          0         0         0        0   \n",
       "4676       0            0           0          1         0         0        0   \n",
       "4677       0            0           0          0         0         0        0   \n",
       "4678       0            0           0          0         0         0        0   \n",
       "4679       1            0           0          0         0         0        0   \n",
       "\n",
       "      Sydney  Toronto  Vancouver  \n",
       "4          0        0          1  \n",
       "6          1        0          0  \n",
       "8          0        0          1  \n",
       "11         0        0          0  \n",
       "17         0        0          0  \n",
       "...      ...      ...        ...  \n",
       "4674       0        0          0  \n",
       "4676       0        0          0  \n",
       "4677       0        1          0  \n",
       "4678       0        0          0  \n",
       "4679       0        0          0  \n",
       "\n",
       "[2372 rows x 44 columns]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "estimation_set_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "# NOTE TO SELF:\n",
    "# Date of birth should be included\n",
    "# If you were at shift the day you were last seen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>missing</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5566</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5567</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5568</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5569</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5570</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5571 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      missing\n",
       "0         0.0\n",
       "1         0.0\n",
       "2         0.0\n",
       "3         0.0\n",
       "4         0.0\n",
       "...       ...\n",
       "5566      0.0\n",
       "5567      0.0\n",
       "5568      0.0\n",
       "5569      0.0\n",
       "5570      0.0\n",
       "\n",
       "[5571 rows x 1 columns]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train = estimation_set_train.missing\n",
    "y_train = y_train.reset_index().drop(\"index\", axis=1)\n",
    "y_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>hourly_salary</th>\n",
       "      <th>weight</th>\n",
       "      <th>height</th>\n",
       "      <th>satisfaction_score</th>\n",
       "      <th>sleep_1</th>\n",
       "      <th>sleep_2</th>\n",
       "      <th>sleep_3</th>\n",
       "      <th>on_shift_last_day_seen</th>\n",
       "      <th>last_seen_in_days</th>\n",
       "      <th>male</th>\n",
       "      <th>...</th>\n",
       "      <th>London</th>\n",
       "      <th>Los Angeles</th>\n",
       "      <th>Manchester</th>\n",
       "      <th>Melbourne</th>\n",
       "      <th>Montreal</th>\n",
       "      <th>New York</th>\n",
       "      <th>Seattle</th>\n",
       "      <th>Sydney</th>\n",
       "      <th>Toronto</th>\n",
       "      <th>Vancouver</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>14.0</td>\n",
       "      <td>84.0</td>\n",
       "      <td>175.0</td>\n",
       "      <td>0.706897</td>\n",
       "      <td>7.0</td>\n",
       "      <td>8.8</td>\n",
       "      <td>11.3</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>14.0</td>\n",
       "      <td>66.0</td>\n",
       "      <td>179.0</td>\n",
       "      <td>0.580460</td>\n",
       "      <td>8.5</td>\n",
       "      <td>6.2</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13.5</td>\n",
       "      <td>93.0</td>\n",
       "      <td>195.0</td>\n",
       "      <td>0.718391</td>\n",
       "      <td>7.6</td>\n",
       "      <td>8.2</td>\n",
       "      <td>7.3</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9.5</td>\n",
       "      <td>78.0</td>\n",
       "      <td>195.0</td>\n",
       "      <td>0.425287</td>\n",
       "      <td>9.3</td>\n",
       "      <td>6.8</td>\n",
       "      <td>9.2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>16.3</td>\n",
       "      <td>88.0</td>\n",
       "      <td>163.0</td>\n",
       "      <td>0.798851</td>\n",
       "      <td>8.2</td>\n",
       "      <td>8.0</td>\n",
       "      <td>5.5</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5566</th>\n",
       "      <td>14.0</td>\n",
       "      <td>70.0</td>\n",
       "      <td>190.0</td>\n",
       "      <td>0.701149</td>\n",
       "      <td>9.0</td>\n",
       "      <td>7.7</td>\n",
       "      <td>10.5</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5567</th>\n",
       "      <td>14.3</td>\n",
       "      <td>67.0</td>\n",
       "      <td>177.0</td>\n",
       "      <td>0.770115</td>\n",
       "      <td>7.4</td>\n",
       "      <td>9.6</td>\n",
       "      <td>7.3</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5568</th>\n",
       "      <td>12.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>170.0</td>\n",
       "      <td>0.724138</td>\n",
       "      <td>7.2</td>\n",
       "      <td>7.9</td>\n",
       "      <td>8.9</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5569</th>\n",
       "      <td>13.7</td>\n",
       "      <td>65.0</td>\n",
       "      <td>175.0</td>\n",
       "      <td>0.614943</td>\n",
       "      <td>7.1</td>\n",
       "      <td>8.2</td>\n",
       "      <td>7.2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5570</th>\n",
       "      <td>10.3</td>\n",
       "      <td>66.0</td>\n",
       "      <td>164.0</td>\n",
       "      <td>0.316092</td>\n",
       "      <td>8.3</td>\n",
       "      <td>6.3</td>\n",
       "      <td>9.3</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5571 rows × 43 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      hourly_salary  weight  height  satisfaction_score  sleep_1  sleep_2  \\\n",
       "0              14.0    84.0   175.0            0.706897      7.0      8.8   \n",
       "1              14.0    66.0   179.0            0.580460      8.5      6.2   \n",
       "2              13.5    93.0   195.0            0.718391      7.6      8.2   \n",
       "3               9.5    78.0   195.0            0.425287      9.3      6.8   \n",
       "4              16.3    88.0   163.0            0.798851      8.2      8.0   \n",
       "...             ...     ...     ...                 ...      ...      ...   \n",
       "5566           14.0    70.0   190.0            0.701149      9.0      7.7   \n",
       "5567           14.3    67.0   177.0            0.770115      7.4      9.6   \n",
       "5568           12.0    75.0   170.0            0.724138      7.2      7.9   \n",
       "5569           13.7    65.0   175.0            0.614943      7.1      8.2   \n",
       "5570           10.3    66.0   164.0            0.316092      8.3      6.3   \n",
       "\n",
       "      sleep_3  on_shift_last_day_seen  last_seen_in_days  male  ...  London  \\\n",
       "0        11.3                       0                  2     0  ...       0   \n",
       "1         8.0                       0                  2     1  ...       0   \n",
       "2         7.3                       0                  5     0  ...       0   \n",
       "3         9.2                       0                  2     1  ...       0   \n",
       "4         5.5                       0                  7     1  ...       0   \n",
       "...       ...                     ...                ...   ...  ...     ...   \n",
       "5566     10.5                       0                 12     1  ...       0   \n",
       "5567      7.3                       0                 12     0  ...       0   \n",
       "5568      8.9                       0                  9     1  ...       1   \n",
       "5569      7.2                       0                  1     1  ...       0   \n",
       "5570      9.3                       0                  4     1  ...       0   \n",
       "\n",
       "      Los Angeles  Manchester  Melbourne  Montreal  New York  Seattle  Sydney  \\\n",
       "0               0           1          0         0         0        0       0   \n",
       "1               0           0          0         0         0        0       0   \n",
       "2               0           0          0         0         0        0       0   \n",
       "3               0           0          0         0         0        0       0   \n",
       "4               0           0          0         0         0        1       0   \n",
       "...           ...         ...        ...       ...       ...      ...     ...   \n",
       "5566            0           0          0         0         0        0       0   \n",
       "5567            0           1          0         0         0        0       0   \n",
       "5568            0           0          0         0         0        0       0   \n",
       "5569            0           1          0         0         0        0       0   \n",
       "5570            0           0          0         0         0        0       0   \n",
       "\n",
       "      Toronto  Vancouver  \n",
       "0           0          0  \n",
       "1           1          0  \n",
       "2           0          1  \n",
       "3           0          0  \n",
       "4           0          0  \n",
       "...       ...        ...  \n",
       "5566        0          0  \n",
       "5567        0          0  \n",
       "5568        0          0  \n",
       "5569        0          0  \n",
       "5570        0          1  \n",
       "\n",
       "[5571 rows x 43 columns]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train = estimation_set_train.drop(\"missing\", axis=1)\n",
    "X_train = X_train.reset_index().drop(\"index\", axis=1)\n",
    "X_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.DataFrame(X_train).to_excel(\"X_train.xlsx\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>missing</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2367</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2368</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2369</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2370</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2371</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2372 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      missing\n",
       "0         0.0\n",
       "1         0.0\n",
       "2         0.0\n",
       "3         0.0\n",
       "4         0.0\n",
       "...       ...\n",
       "2367      0.0\n",
       "2368      0.0\n",
       "2369      0.0\n",
       "2370      0.0\n",
       "2371      0.0\n",
       "\n",
       "[2372 rows x 1 columns]"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test = estimation_set_test.missing\n",
    "y_test = y_test.reset_index().drop(\"index\", axis=1)\n",
    "y_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>hourly_salary</th>\n",
       "      <th>weight</th>\n",
       "      <th>height</th>\n",
       "      <th>satisfaction_score</th>\n",
       "      <th>sleep_1</th>\n",
       "      <th>sleep_2</th>\n",
       "      <th>sleep_3</th>\n",
       "      <th>on_shift_last_day_seen</th>\n",
       "      <th>last_seen_in_days</th>\n",
       "      <th>male</th>\n",
       "      <th>...</th>\n",
       "      <th>London</th>\n",
       "      <th>Los Angeles</th>\n",
       "      <th>Manchester</th>\n",
       "      <th>Melbourne</th>\n",
       "      <th>Montreal</th>\n",
       "      <th>New York</th>\n",
       "      <th>Seattle</th>\n",
       "      <th>Sydney</th>\n",
       "      <th>Toronto</th>\n",
       "      <th>Vancouver</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>16.0</td>\n",
       "      <td>94.0</td>\n",
       "      <td>187.0</td>\n",
       "      <td>0.718391</td>\n",
       "      <td>8.7</td>\n",
       "      <td>7.9</td>\n",
       "      <td>8.1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>12.5</td>\n",
       "      <td>70.0</td>\n",
       "      <td>169.0</td>\n",
       "      <td>0.620690</td>\n",
       "      <td>8.8</td>\n",
       "      <td>8.3</td>\n",
       "      <td>7.9</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>13.9</td>\n",
       "      <td>79.0</td>\n",
       "      <td>181.0</td>\n",
       "      <td>0.614943</td>\n",
       "      <td>8.1</td>\n",
       "      <td>8.7</td>\n",
       "      <td>9.6</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>16.0</td>\n",
       "      <td>90.0</td>\n",
       "      <td>178.0</td>\n",
       "      <td>0.850575</td>\n",
       "      <td>8.2</td>\n",
       "      <td>7.8</td>\n",
       "      <td>9.4</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>15.3</td>\n",
       "      <td>81.0</td>\n",
       "      <td>174.0</td>\n",
       "      <td>0.660920</td>\n",
       "      <td>8.5</td>\n",
       "      <td>8.5</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2367</th>\n",
       "      <td>14.2</td>\n",
       "      <td>75.0</td>\n",
       "      <td>173.0</td>\n",
       "      <td>0.597701</td>\n",
       "      <td>6.2</td>\n",
       "      <td>7.5</td>\n",
       "      <td>8.2</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2368</th>\n",
       "      <td>9.9</td>\n",
       "      <td>68.0</td>\n",
       "      <td>176.0</td>\n",
       "      <td>0.465517</td>\n",
       "      <td>8.4</td>\n",
       "      <td>7.9</td>\n",
       "      <td>6.2</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2369</th>\n",
       "      <td>15.0</td>\n",
       "      <td>80.0</td>\n",
       "      <td>190.0</td>\n",
       "      <td>0.781609</td>\n",
       "      <td>8.9</td>\n",
       "      <td>8.6</td>\n",
       "      <td>8.3</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2370</th>\n",
       "      <td>10.1</td>\n",
       "      <td>87.0</td>\n",
       "      <td>177.0</td>\n",
       "      <td>0.545977</td>\n",
       "      <td>6.7</td>\n",
       "      <td>7.2</td>\n",
       "      <td>7.7</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2371</th>\n",
       "      <td>13.7</td>\n",
       "      <td>65.0</td>\n",
       "      <td>166.0</td>\n",
       "      <td>0.770115</td>\n",
       "      <td>9.0</td>\n",
       "      <td>9.3</td>\n",
       "      <td>6.5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2372 rows × 43 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      hourly_salary  weight  height  satisfaction_score  sleep_1  sleep_2  \\\n",
       "0              16.0    94.0   187.0            0.718391      8.7      7.9   \n",
       "1              12.5    70.0   169.0            0.620690      8.8      8.3   \n",
       "2              13.9    79.0   181.0            0.614943      8.1      8.7   \n",
       "3              16.0    90.0   178.0            0.850575      8.2      7.8   \n",
       "4              15.3    81.0   174.0            0.660920      8.5      8.5   \n",
       "...             ...     ...     ...                 ...      ...      ...   \n",
       "2367           14.2    75.0   173.0            0.597701      6.2      7.5   \n",
       "2368            9.9    68.0   176.0            0.465517      8.4      7.9   \n",
       "2369           15.0    80.0   190.0            0.781609      8.9      8.6   \n",
       "2370           10.1    87.0   177.0            0.545977      6.7      7.2   \n",
       "2371           13.7    65.0   166.0            0.770115      9.0      9.3   \n",
       "\n",
       "      sleep_3  on_shift_last_day_seen  last_seen_in_days  male  ...  London  \\\n",
       "0         8.1                       0                  0     1  ...       0   \n",
       "1         7.9                       0                  7     0  ...       0   \n",
       "2         9.6                       0                  4     0  ...       0   \n",
       "3         9.4                       0                  8     0  ...       0   \n",
       "4         9.0                       0                  0     0  ...       0   \n",
       "...       ...                     ...                ...   ...  ...     ...   \n",
       "2367      8.2                       0                  5     0  ...       0   \n",
       "2368      6.2                       0                  8     0  ...       0   \n",
       "2369      8.3                       0                  8     1  ...       0   \n",
       "2370      7.7                       0                  9     1  ...       0   \n",
       "2371      6.5                       0                  0     0  ...       1   \n",
       "\n",
       "      Los Angeles  Manchester  Melbourne  Montreal  New York  Seattle  Sydney  \\\n",
       "0               0           0          0         0         0        0       0   \n",
       "1               0           0          0         0         0        0       1   \n",
       "2               0           0          0         0         0        0       0   \n",
       "3               0           1          0         0         0        0       0   \n",
       "4               0           1          0         0         0        0       0   \n",
       "...           ...         ...        ...       ...       ...      ...     ...   \n",
       "2367            0           0          0         0         0        0       0   \n",
       "2368            0           0          1         0         0        0       0   \n",
       "2369            0           0          0         0         0        0       0   \n",
       "2370            0           0          0         0         0        0       0   \n",
       "2371            0           0          0         0         0        0       0   \n",
       "\n",
       "      Toronto  Vancouver  \n",
       "0           0          1  \n",
       "1           0          0  \n",
       "2           0          1  \n",
       "3           0          0  \n",
       "4           0          0  \n",
       "...       ...        ...  \n",
       "2367        0          0  \n",
       "2368        0          0  \n",
       "2369        1          0  \n",
       "2370        0          0  \n",
       "2371        0          0  \n",
       "\n",
       "[2372 rows x 43 columns]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test = estimation_set_test.drop(\"missing\", axis=1)\n",
    "X_test = X_test.reset_index().drop(\"index\", axis=1)\n",
    "X_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pip._internal import main\n",
    "\n",
    "def import_or_install(package):\n",
    "    try:\n",
    "        __import__(package)\n",
    "    except ImportError:\n",
    "        main(['install', package])\n",
    "        __import__(package)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Feature Scaling\n",
    "from sklearn.preprocessing import StandardScaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "sc = StandardScaler()\n",
    "X_train = sc.fit_transform(X_train)\n",
    "X_test = sc.fit_transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "import_or_install(\"tensorflow\")\n",
    "import_or_install(\"keras\")\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "43"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_dim = X_train.shape[1]\n",
    "input_dim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setting up ANN\n",
    "\n",
    "**Why is used Keras**\n",
    "\n",
    "I have used Keras due to its easy high-level API. \\\n",
    "But I would have implemented **my own ANN** if I had the time. Here, I would have done the chain rule derivations myself, specifying the loss and activation functions explicitly and done the batch Gradient Descent myself. \n",
    "\n",
    "**The implementation**\n",
    "\n",
    "Here I am using Binary Cross Entropy Loss equivalent to Log Loss. \\\n",
    "This is the loss function for binary response using Bernoulli Distribution.\n",
    "I apply Stochastic Gradient Descent Algorithm to tune the weights (batches of 10)\n",
    "\n",
    "This ANN is as follows:\n",
    "\n",
    "- Input Layer \n",
    "- 6 Neurons w. ReLU acitivation\n",
    "- 6 Neurons w. ReLU activation\n",
    "- 1 Neuron w. Sigmoid (Logistic) activation\n",
    "\n",
    "**Note**: The number of layers, neurons and epochs are more or less chosen arbritrarily. \\\n",
    "ReLU is chosen in hidden layers as this emprically converges (to optimal weight) faster than Sigmoid and Tanh\n",
    "\n",
    "*Improvements* \\\n",
    "Adding normalization/regularization constant to weight matrices (\\lambda) to ensure right-size weights. \\\n",
    "E.g. this could be done by using LASSO for penalizing high weight coefficients. \\\n",
    "Or deleting weights equal to 0 (memory efficiency)\n",
    "\n",
    "**Other ML approaches**\n",
    "\n",
    "Other tools for classification:\n",
    "- Gaussian Discriminant Analysis (GDA)\n",
    "- EM (Expectation Maximization) estimation\n",
    "- Kernel functions (for higher dimensionality regression)\n",
    "- Maybe: Naive Bayes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/30\n",
      "558/558 [==============================] - 1s 716us/step - loss: 0.4478 - accuracy: 0.9684\n",
      "Epoch 2/30\n",
      "558/558 [==============================] - 0s 727us/step - loss: 0.0875 - accuracy: 0.9721\n",
      "Epoch 3/30\n",
      "558/558 [==============================] - 0s 723us/step - loss: 0.0867 - accuracy: 0.9683\n",
      "Epoch 4/30\n",
      "558/558 [==============================] - 0s 661us/step - loss: 0.0613 - accuracy: 0.9751\n",
      "Epoch 5/30\n",
      "558/558 [==============================] - 0s 598us/step - loss: 0.0619 - accuracy: 0.9728\n",
      "Epoch 6/30\n",
      "558/558 [==============================] - 0s 671us/step - loss: 0.0597 - accuracy: 0.9736\n",
      "Epoch 7/30\n",
      "558/558 [==============================] - 0s 627us/step - loss: 0.0544 - accuracy: 0.9754\n",
      "Epoch 8/30\n",
      "558/558 [==============================] - 0s 747us/step - loss: 0.0513 - accuracy: 0.9753\n",
      "Epoch 9/30\n",
      "558/558 [==============================] - 0s 609us/step - loss: 0.0517 - accuracy: 0.9732\n",
      "Epoch 10/30\n",
      "558/558 [==============================] - 0s 661us/step - loss: 0.0484 - accuracy: 0.9762\n",
      "Epoch 11/30\n",
      "558/558 [==============================] - 0s 600us/step - loss: 0.0487 - accuracy: 0.9750\n",
      "Epoch 12/30\n",
      "558/558 [==============================] - 0s 650us/step - loss: 0.0467 - accuracy: 0.9795\n",
      "Epoch 13/30\n",
      "558/558 [==============================] - 0s 682us/step - loss: 0.0517 - accuracy: 0.9756\n",
      "Epoch 14/30\n",
      "558/558 [==============================] - 0s 682us/step - loss: 0.0457 - accuracy: 0.9757\n",
      "Epoch 15/30\n",
      "558/558 [==============================] - 0s 636us/step - loss: 0.0463 - accuracy: 0.9745\n",
      "Epoch 16/30\n",
      "558/558 [==============================] - 0s 652us/step - loss: 0.0447 - accuracy: 0.9793\n",
      "Epoch 17/30\n",
      "558/558 [==============================] - 0s 671us/step - loss: 0.0431 - accuracy: 0.9779\n",
      "Epoch 18/30\n",
      "558/558 [==============================] - 0s 580us/step - loss: 0.0418 - accuracy: 0.9797\n",
      "Epoch 19/30\n",
      "558/558 [==============================] - 0s 612us/step - loss: 0.0463 - accuracy: 0.9791\n",
      "Epoch 20/30\n",
      "558/558 [==============================] - 0s 634us/step - loss: 0.0392 - accuracy: 0.9834\n",
      "Epoch 21/30\n",
      "558/558 [==============================] - 0s 691us/step - loss: 0.0407 - accuracy: 0.9798\n",
      "Epoch 22/30\n",
      "558/558 [==============================] - 0s 770us/step - loss: 0.0396 - accuracy: 0.9788\n",
      "Epoch 23/30\n",
      "558/558 [==============================] - 0s 586us/step - loss: 0.0373 - accuracy: 0.9823\n",
      "Epoch 24/30\n",
      "558/558 [==============================] - 0s 771us/step - loss: 0.0426 - accuracy: 0.9811\n",
      "Epoch 25/30\n",
      "558/558 [==============================] - 0s 598us/step - loss: 0.0350 - accuracy: 0.9839\n",
      "Epoch 26/30\n",
      "558/558 [==============================] - 0s 702us/step - loss: 0.0356 - accuracy: 0.9849\n",
      "Epoch 27/30\n",
      "558/558 [==============================] - 1s 981us/step - loss: 0.0355 - accuracy: 0.9856\n",
      "Epoch 28/30\n",
      "558/558 [==============================] - 1s 1ms/step - loss: 0.0373 - accuracy: 0.9825\n",
      "Epoch 29/30\n",
      "558/558 [==============================] - 1s 1ms/step - loss: 0.0359 - accuracy: 0.9850\n",
      "Epoch 30/30\n",
      "558/558 [==============================] - 1s 1ms/step - loss: 0.0366 - accuracy: 0.9844\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x1e6b6e493a0>"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier = Sequential()\n",
    "classifier.add(Dense(units=6, kernel_initializer='uniform', activation='relu', input_dim=input_dim))\n",
    "classifier.add(Dense(units=6, kernel_initializer='uniform', activation='relu'))\n",
    "classifier.add(Dense(units=1, kernel_initializer='uniform', activation='sigmoid'))\n",
    "classifier.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])\n",
    "classifier.fit(x=X_train, y=y_train, batch_size=10, epochs=30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = classifier.predict(X_test)\n",
    "# y_pred = (y_pred > 0.5)     # Like rounding: getting True / False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.6112688],\n",
       "       [0.6112688],\n",
       "       [0.6112688],\n",
       "       ...,\n",
       "       [0.       ],\n",
       "       [0.       ],\n",
       "       [0.       ]], dtype=float32)"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "probabilities = np.sort(y_pred, axis=0)[::-1]\n",
    "probabilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Making the Confusion Matrix\n",
    "# from sklearn.metrics import confusion_matrix\n",
    "# cm = confusion_matrix(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ACCURACY: 0.8975548060708263\n"
     ]
    }
   ],
   "source": [
    "y_pred = (y_pred > 0.5)     # Like rounding: getting True / False\n",
    "prediction_correct = y_pred == y_test\n",
    "accuracy = prediction_correct.mean().values[0]\n",
    "print(\"ACCURACY:\", accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "# POPULATE THIS (with 43 parameters)\n",
    "# characteristic_vector = []\n",
    "# new_prediction = classifier.predict(sc.transform(np.array([characteristic_vector])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### COMMENTS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Comments regarding data** \\\n",
    "In my data, it looks like there are none missing in the test data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following features are not included:\n",
    "\n",
    "- Date of birth: lack of time\n",
    "- Full name: no explanatory power\n",
    "- Zip code: I did not really understand it - where do I find it?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The feature **On a shift** is not included due to lack of time \\\n",
    "However, my approach would be:\n",
    "Insert a column on X_test indicating if **on shift last day seen** and use this as regressor\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Which features have most explanatory power** \\\n",
    "I did not find this, but my approach would be:\n",
    "\n",
    "1) Use X_train and y_train to set up an econometric model using the regressors and dummy variables \\\n",
    "2) Estimate the model with a probit model for this binary classification \\\n",
    "3) Evaluate the regressors model coeffficients, the *Betas* (and if they are significant at some level) \\\n",
    "4) List the most significant coefficients (which have positive and which have negative effect)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}