04.14-Automatic-Imputation-Method-Detection-Sklearn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Automatic selection of best imputation technique with Sklearn\n",
    "\n",
    "In this notebook we will do a grid search over the imputation methods available in Scikit-learn to determine which imputation technique works best for this dataset and the machine learning model of choice.\n",
    "\n",
    "We will also train a very simple machine learning model as part of a small pipeline.\n",
    "\n",
    "We will use the House Price dataset.\n",
    "\n",
    "- To download the dataset please visit the lecture **Datasets** in **Section 1** of the course."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# import classes for imputation\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.impute import SimpleImputer\n",
    "\n",
    "# import extra classes for modelling\n",
    "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
    "from sklearn.linear_model import Lasso\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV\n",
    "\n",
    "np.random.seed(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>Street</th>\n",
       "      <th>Alley</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>...</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>PoolQC</th>\n",
       "      <th>Fence</th>\n",
       "      <th>MiscFeature</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>SaleCondition</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>65.0</td>\n",
       "      <td>8450</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>208500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>20</td>\n",
       "      <td>RL</td>\n",
       "      <td>80.0</td>\n",
       "      <td>9600</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2007</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>181500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>68.0</td>\n",
       "      <td>11250</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>223500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>70</td>\n",
       "      <td>RL</td>\n",
       "      <td>60.0</td>\n",
       "      <td>9550</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2006</td>\n",
       "      <td>WD</td>\n",
       "      <td>Abnorml</td>\n",
       "      <td>140000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>60</td>\n",
       "      <td>RL</td>\n",
       "      <td>84.0</td>\n",
       "      <td>14260</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>2008</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "      <td>250000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 81 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  MSSubClass MSZoning  LotFrontage  LotArea Street Alley LotShape  \\\n",
       "0   1          60       RL         65.0     8450   Pave   NaN      Reg   \n",
       "1   2          20       RL         80.0     9600   Pave   NaN      Reg   \n",
       "2   3          60       RL         68.0    11250   Pave   NaN      IR1   \n",
       "3   4          70       RL         60.0     9550   Pave   NaN      IR1   \n",
       "4   5          60       RL         84.0    14260   Pave   NaN      IR1   \n",
       "\n",
       "  LandContour Utilities  ... PoolArea PoolQC Fence MiscFeature MiscVal MoSold  \\\n",
       "0         Lvl    AllPub  ...        0    NaN   NaN         NaN       0      2   \n",
       "1         Lvl    AllPub  ...        0    NaN   NaN         NaN       0      5   \n",
       "2         Lvl    AllPub  ...        0    NaN   NaN         NaN       0      9   \n",
       "3         Lvl    AllPub  ...        0    NaN   NaN         NaN       0      2   \n",
       "4         Lvl    AllPub  ...        0    NaN   NaN         NaN       0     12   \n",
       "\n",
       "  YrSold  SaleType  SaleCondition  SalePrice  \n",
       "0   2008        WD         Normal     208500  \n",
       "1   2007        WD         Normal     181500  \n",
       "2   2008        WD         Normal     223500  \n",
       "3   2006        WD        Abnorml     140000  \n",
       "4   2008        WD         Normal     250000  \n",
       "\n",
       "[5 rows x 81 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load dataset with all the variables\n",
    "\n",
    "data = pd.read_csv('../houseprice.csv',)\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# find categorical variables\n",
    "# those of type 'Object' in the dataset\n",
    "features_categorical = [c for c in data.columns if data[c].dtypes=='O']\n",
    "\n",
    "# find numerical variables\n",
    "# those different from object and also excluding the target SalePrice\n",
    "features_numerical = [c for c in data.columns if data[c].dtypes!='O' and c !='SalePrice']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MSZoning</th>\n",
       "      <th>Street</th>\n",
       "      <th>Alley</th>\n",
       "      <th>LotShape</th>\n",
       "      <th>LandContour</th>\n",
       "      <th>Utilities</th>\n",
       "      <th>LotConfig</th>\n",
       "      <th>LandSlope</th>\n",
       "      <th>Neighborhood</th>\n",
       "      <th>Condition1</th>\n",
       "      <th>...</th>\n",
       "      <th>GarageType</th>\n",
       "      <th>GarageFinish</th>\n",
       "      <th>GarageQual</th>\n",
       "      <th>GarageCond</th>\n",
       "      <th>PavedDrive</th>\n",
       "      <th>PoolQC</th>\n",
       "      <th>Fence</th>\n",
       "      <th>MiscFeature</th>\n",
       "      <th>SaleType</th>\n",
       "      <th>SaleCondition</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>RL</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Inside</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>CollgCr</td>\n",
       "      <td>Norm</td>\n",
       "      <td>...</td>\n",
       "      <td>Attchd</td>\n",
       "      <td>RFn</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>Y</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>RL</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Reg</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>FR2</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>Veenker</td>\n",
       "      <td>Feedr</td>\n",
       "      <td>...</td>\n",
       "      <td>Attchd</td>\n",
       "      <td>RFn</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>Y</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>RL</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Inside</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>CollgCr</td>\n",
       "      <td>Norm</td>\n",
       "      <td>...</td>\n",
       "      <td>Attchd</td>\n",
       "      <td>RFn</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>Y</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>RL</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>Corner</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>Crawfor</td>\n",
       "      <td>Norm</td>\n",
       "      <td>...</td>\n",
       "      <td>Detchd</td>\n",
       "      <td>Unf</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>Y</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>WD</td>\n",
       "      <td>Abnorml</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>RL</td>\n",
       "      <td>Pave</td>\n",
       "      <td>NaN</td>\n",
       "      <td>IR1</td>\n",
       "      <td>Lvl</td>\n",
       "      <td>AllPub</td>\n",
       "      <td>FR2</td>\n",
       "      <td>Gtl</td>\n",
       "      <td>NoRidge</td>\n",
       "      <td>Norm</td>\n",
       "      <td>...</td>\n",
       "      <td>Attchd</td>\n",
       "      <td>RFn</td>\n",
       "      <td>TA</td>\n",
       "      <td>TA</td>\n",
       "      <td>Y</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>WD</td>\n",
       "      <td>Normal</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 43 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  MSZoning Street Alley LotShape LandContour Utilities LotConfig LandSlope  \\\n",
       "0       RL   Pave   NaN      Reg         Lvl    AllPub    Inside       Gtl   \n",
       "1       RL   Pave   NaN      Reg         Lvl    AllPub       FR2       Gtl   \n",
       "2       RL   Pave   NaN      IR1         Lvl    AllPub    Inside       Gtl   \n",
       "3       RL   Pave   NaN      IR1         Lvl    AllPub    Corner       Gtl   \n",
       "4       RL   Pave   NaN      IR1         Lvl    AllPub       FR2       Gtl   \n",
       "\n",
       "  Neighborhood Condition1  ... GarageType GarageFinish GarageQual GarageCond  \\\n",
       "0      CollgCr       Norm  ...     Attchd          RFn         TA         TA   \n",
       "1      Veenker      Feedr  ...     Attchd          RFn         TA         TA   \n",
       "2      CollgCr       Norm  ...     Attchd          RFn         TA         TA   \n",
       "3      Crawfor       Norm  ...     Detchd          Unf         TA         TA   \n",
       "4      NoRidge       Norm  ...     Attchd          RFn         TA         TA   \n",
       "\n",
       "  PavedDrive PoolQC Fence MiscFeature SaleType SaleCondition  \n",
       "0          Y    NaN   NaN         NaN       WD        Normal  \n",
       "1          Y    NaN   NaN         NaN       WD        Normal  \n",
       "2          Y    NaN   NaN         NaN       WD        Normal  \n",
       "3          Y    NaN   NaN         NaN       WD       Abnorml  \n",
       "4          Y    NaN   NaN         NaN       WD        Normal  \n",
       "\n",
       "[5 rows x 43 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# inspect the categorical variables\n",
    "\n",
    "data[features_categorical].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>MSSubClass</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>LotArea</th>\n",
       "      <th>OverallQual</th>\n",
       "      <th>OverallCond</th>\n",
       "      <th>YearBuilt</th>\n",
       "      <th>YearRemodAdd</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>BsmtFinSF1</th>\n",
       "      <th>...</th>\n",
       "      <th>GarageArea</th>\n",
       "      <th>WoodDeckSF</th>\n",
       "      <th>OpenPorchSF</th>\n",
       "      <th>EnclosedPorch</th>\n",
       "      <th>3SsnPorch</th>\n",
       "      <th>ScreenPorch</th>\n",
       "      <th>PoolArea</th>\n",
       "      <th>MiscVal</th>\n",
       "      <th>MoSold</th>\n",
       "      <th>YrSold</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>60</td>\n",
       "      <td>65.0</td>\n",
       "      <td>8450</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>2003</td>\n",
       "      <td>2003</td>\n",
       "      <td>196.0</td>\n",
       "      <td>706</td>\n",
       "      <td>...</td>\n",
       "      <td>548</td>\n",
       "      <td>0</td>\n",
       "      <td>61</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2008</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>20</td>\n",
       "      <td>80.0</td>\n",
       "      <td>9600</td>\n",
       "      <td>6</td>\n",
       "      <td>8</td>\n",
       "      <td>1976</td>\n",
       "      <td>1976</td>\n",
       "      <td>0.0</td>\n",
       "      <td>978</td>\n",
       "      <td>...</td>\n",
       "      <td>460</td>\n",
       "      <td>298</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>2007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>60</td>\n",
       "      <td>68.0</td>\n",
       "      <td>11250</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>2001</td>\n",
       "      <td>2002</td>\n",
       "      <td>162.0</td>\n",
       "      <td>486</td>\n",
       "      <td>...</td>\n",
       "      <td>608</td>\n",
       "      <td>0</td>\n",
       "      <td>42</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>2008</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>70</td>\n",
       "      <td>60.0</td>\n",
       "      <td>9550</td>\n",
       "      <td>7</td>\n",
       "      <td>5</td>\n",
       "      <td>1915</td>\n",
       "      <td>1970</td>\n",
       "      <td>0.0</td>\n",
       "      <td>216</td>\n",
       "      <td>...</td>\n",
       "      <td>642</td>\n",
       "      <td>0</td>\n",
       "      <td>35</td>\n",
       "      <td>272</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>60</td>\n",
       "      <td>84.0</td>\n",
       "      <td>14260</td>\n",
       "      <td>8</td>\n",
       "      <td>5</td>\n",
       "      <td>2000</td>\n",
       "      <td>2000</td>\n",
       "      <td>350.0</td>\n",
       "      <td>655</td>\n",
       "      <td>...</td>\n",
       "      <td>836</td>\n",
       "      <td>192</td>\n",
       "      <td>84</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>2008</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 37 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  MSSubClass  LotFrontage  LotArea  OverallQual  OverallCond  YearBuilt  \\\n",
       "0   1          60         65.0     8450            7            5       2003   \n",
       "1   2          20         80.0     9600            6            8       1976   \n",
       "2   3          60         68.0    11250            7            5       2001   \n",
       "3   4          70         60.0     9550            7            5       1915   \n",
       "4   5          60         84.0    14260            8            5       2000   \n",
       "\n",
       "   YearRemodAdd  MasVnrArea  BsmtFinSF1  ...  GarageArea  WoodDeckSF  \\\n",
       "0          2003       196.0         706  ...         548           0   \n",
       "1          1976         0.0         978  ...         460         298   \n",
       "2          2002       162.0         486  ...         608           0   \n",
       "3          1970         0.0         216  ...         642           0   \n",
       "4          2000       350.0         655  ...         836         192   \n",
       "\n",
       "   OpenPorchSF  EnclosedPorch  3SsnPorch  ScreenPorch  PoolArea  MiscVal  \\\n",
       "0           61              0          0            0         0        0   \n",
       "1            0              0          0            0         0        0   \n",
       "2           42              0          0            0         0        0   \n",
       "3           35            272          0            0         0        0   \n",
       "4           84              0          0            0         0        0   \n",
       "\n",
       "   MoSold  YrSold  \n",
       "0       2    2008  \n",
       "1       5    2007  \n",
       "2       9    2008  \n",
       "3       2    2006  \n",
       "4      12    2008  \n",
       "\n",
       "[5 rows x 37 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# inspect the numerical variables\n",
    "\n",
    "data[features_numerical].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1022, 80), (438, 80))"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# separate intro train and test set\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    data.drop('SalePrice', axis=1),  # just the features\n",
    "    data['SalePrice'],  # the target\n",
    "    test_size=0.3,  # the percentage of obs in the test set\n",
    "    random_state=0)  # for reproducibility\n",
    "\n",
    "X_train.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We create the preprocessing pipelines for both\n",
    "# numerical and categorical data\n",
    "\n",
    "# adapted from Scikit-learn code available here under BSD3 license:\n",
    "# https://scikit-learn.org/stable/auto_examples/compose/plot_column_transformer_mixed_types.html\n",
    "\n",
    "numeric_transformer = Pipeline(steps=[\n",
    "    ('imputer', SimpleImputer(strategy='median')),\n",
    "    ('scaler', StandardScaler())])\n",
    "\n",
    "categorical_transformer = Pipeline(steps=[\n",
    "    ('imputer', SimpleImputer(strategy='constant', fill_value='missing')),\n",
    "    ('onehot', OneHotEncoder(handle_unknown='ignore'))])\n",
    "\n",
    "preprocessor = ColumnTransformer(\n",
    "    transformers=[\n",
    "        ('numerical', numeric_transformer, features_numerical),\n",
    "        ('categorical', categorical_transformer, features_categorical)])\n",
    "\n",
    "# Note that to initialise the pipeline I pass any argument to the transformers.\n",
    "# Those will be changed during the gridsearch below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Append classifier to preprocessing pipeline.\n",
    "# Now we have a full prediction pipeline.\n",
    "\n",
    "clf = Pipeline(steps=[('preprocessor', preprocessor),\n",
    "                      ('classifier', Lasso(max_iter=2000))])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# now we create the grid with all the parameters that we would like to test\n",
    "\n",
    "param_grid = {\n",
    "    'preprocessor__numerical__imputer__strategy': ['mean', 'median'],\n",
    "    'preprocessor__categorical__imputer__strategy': ['most_frequent', 'constant'],\n",
    "    'classifier__alpha': [10, 100, 200],\n",
    "}\n",
    "\n",
    "grid_search = GridSearchCV(clf, param_grid, cv=5, iid=False, n_jobs=-1, scoring='r2')\n",
    "\n",
    "# cv=3 is the cross-validation\n",
    "# no_jobs =-1 indicates to use all available cpus\n",
    "# scoring='r2' indicates to evaluate using the r squared\n",
    "\n",
    "# for more details in the grid parameters visit:\n",
    "#https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When setting the grid parameters, this is how we indicate the parameters:\n",
    "\n",
    "preprocessor__numerical__imputer__strategy': ['mean', 'median'],\n",
    "\n",
    "the above line of code indicates that I would like to test the mean and the median in the imputer step of the numerical processor.\n",
    "\n",
    "preprocessor__categorical__imputer__strategy': ['most_frequent', 'constant']\n",
    "\n",
    "the above line of code indicates that I would like to test the most frequent or a constant value in the imputer step of the categorical processor\n",
    "\n",
    "classifier__alpha': [0.1, 1.0, 0.5]\n",
    "\n",
    "the above line of code indicates that I want to test those 3 values for the alpha parameter of Lasso. Note that Lasso is the 'classifier' step of our last pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best linear regression from grid search: 0.933\n"
     ]
    }
   ],
   "source": [
    "# and now we train over all the possible combinations of the parameters above\n",
    "grid_search.fit(X_train, y_train)\n",
    "\n",
    "# and we print the best score over the train set\n",
    "print((\"best linear regression from grid search: %.3f\"\n",
    "       % grid_search.score(X_train, y_train)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('preprocessor',\n",
       "                 ColumnTransformer(n_jobs=None, remainder='drop',\n",
       "                                   sparse_threshold=0.3,\n",
       "                                   transformer_weights=None,\n",
       "                                   transformers=[('numerical',\n",
       "                                                  Pipeline(memory=None,\n",
       "                                                           steps=[('imputer',\n",
       "                                                                   SimpleImputer(add_indicator=False,\n",
       "                                                                                 copy=True,\n",
       "                                                                                 fill_value=None,\n",
       "                                                                                 missing_values=nan,\n",
       "                                                                                 strategy='median',\n",
       "                                                                                 verbose=0)),\n",
       "                                                                  ('scaler',\n",
       "                                                                   StandardScaler(copy=True,\n",
       "                                                                                  wit...\n",
       "                                                   'Foundation', 'BsmtQual',\n",
       "                                                   'BsmtCond', 'BsmtExposure',\n",
       "                                                   'BsmtFinType1',\n",
       "                                                   'BsmtFinType2', 'Heating',\n",
       "                                                   'HeatingQC', 'CentralAir',\n",
       "                                                   'Electrical', ...])],\n",
       "                                   verbose=False)),\n",
       "                ('classifier',\n",
       "                 Lasso(alpha=100, copy_X=True, fit_intercept=True,\n",
       "                       max_iter=2000, normalize=False, positive=False,\n",
       "                       precompute=False, random_state=None, selection='cyclic',\n",
       "                       tol=0.0001, warm_start=False))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we can print the best estimator parameters like this\n",
    "grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'classifier__alpha': 100,\n",
       " 'preprocessor__categorical__imputer__strategy': 'constant',\n",
       " 'preprocessor__numerical__imputer__strategy': 'median'}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# and find the best fit parameters like this\n",
    "grid_search.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'classifier__alpha': 10,\n",
       "  'preprocessor__categorical__imputer__strategy': 'most_frequent',\n",
       "  'preprocessor__numerical__imputer__strategy': 'mean'},\n",
       " {'classifier__alpha': 10,\n",
       "  'preprocessor__categorical__imputer__strategy': 'most_frequent',\n",
       "  'preprocessor__numerical__imputer__strategy': 'median'},\n",
       " {'classifier__alpha': 10,\n",
       "  'preprocessor__categorical__imputer__strategy': 'constant',\n",
       "  'preprocessor__numerical__imputer__strategy': 'mean'},\n",
       " {'classifier__alpha': 10,\n",
       "  'preprocessor__categorical__imputer__strategy': 'constant',\n",
       "  'preprocessor__numerical__imputer__strategy': 'median'},\n",
       " {'classifier__alpha': 100,\n",
       "  'preprocessor__categorical__imputer__strategy': 'most_frequent',\n",
       "  'preprocessor__numerical__imputer__strategy': 'mean'},\n",
       " {'classifier__alpha': 100,\n",
       "  'preprocessor__categorical__imputer__strategy': 'most_frequent',\n",
       "  'preprocessor__numerical__imputer__strategy': 'median'},\n",
       " {'classifier__alpha': 100,\n",
       "  'preprocessor__categorical__imputer__strategy': 'constant',\n",
       "  'preprocessor__numerical__imputer__strategy': 'mean'},\n",
       " {'classifier__alpha': 100,\n",
       "  'preprocessor__categorical__imputer__strategy': 'constant',\n",
       "  'preprocessor__numerical__imputer__strategy': 'median'},\n",
       " {'classifier__alpha': 200,\n",
       "  'preprocessor__categorical__imputer__strategy': 'most_frequent',\n",
       "  'preprocessor__numerical__imputer__strategy': 'mean'},\n",
       " {'classifier__alpha': 200,\n",
       "  'preprocessor__categorical__imputer__strategy': 'most_frequent',\n",
       "  'preprocessor__numerical__imputer__strategy': 'median'},\n",
       " {'classifier__alpha': 200,\n",
       "  'preprocessor__categorical__imputer__strategy': 'constant',\n",
       "  'preprocessor__numerical__imputer__strategy': 'mean'},\n",
       " {'classifier__alpha': 200,\n",
       "  'preprocessor__categorical__imputer__strategy': 'constant',\n",
       "  'preprocessor__numerical__imputer__strategy': 'median'}]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# here we can see all the combinations evaluated during the gridsearch\n",
    "grid_search.cv_results_['params']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.84746254, 0.84739594, 0.84814964, 0.8481309 , 0.86624908,\n",
       "       0.86621021, 0.86646886, 0.86651035, 0.86552764, 0.8654755 ,\n",
       "       0.86525292, 0.86523714])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# and here the scores for each of one of the above combinations\n",
    "grid_search.cv_results_['mean_test_score']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best linear regression from grid search: 0.738\n"
     ]
    }
   ],
   "source": [
    "# and finally let's check the performance over the test set\n",
    "print((\"best linear regression from grid search: %.3f\"\n",
    "       % grid_search.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This model overfits to the train set, look at the r2 of 0.93 obtained for the train set vs 0.738 for the test set.\n",
    "\n",
    "We will try to reduce this over-fitting as we progress in the course."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "feml",
   "language": "python",
   "name": "feml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}