04.22-Missing-Indicator-Feature-Engine.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Missing Indicator ==> Feature-Engine\n",
    "\n",
    "\n",
    "### What is Feature-Engine\n",
    "\n",
    "Feature-Engine is an open source python package that I created at the back of this course. \n",
    "\n",
    "- Feature-Engine includes all the feature engineering techniques described in the course\n",
    "- Feature-Engine works like to Scikit-learn, so it is easy to learn\n",
    "- Feature-Engine allows you to implement specific engineering steps to specific feature subsets\n",
    "- Feature-Engine can be integrated with the Scikit-learn pipeline allowing for smooth model building\n",
    "- \n",
    "**Feature-Engine allows you to design and store a feature engineering pipeline with bespoke procedures for different variable groups.**\n",
    "\n",
    "-------------------------------------------------------------------\n",
    "Feature-Engine can be installed via pip ==> pip install feature-engine\n",
    "\n",
    "- Make sure you have installed feature-engine before running this notebook\n",
    "\n",
    "For more information visit:\n",
    "my website\n",
    "\n",
    "## In this demo\n",
    "\n",
    "We will use Feature-Engine to perform mean or median imputation using the Ames House Price Dataset.\n",
    "\n",
    "- To download the dataset visit the lecture **Datasets** in **Section 1** of the course."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# to split the datasets\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "# from feature-engine\n",
    "from feature_engine import missing_data_imputers as mdi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>BsmtQual</th>\n",
       "      <th>FireplaceQu</th>\n",
       "      <th>GarageYrBlt</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>65.0</td>\n",
       "      <td>196.0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2003.0</td>\n",
       "      <td>208500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>80.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>1976.0</td>\n",
       "      <td>181500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>68.0</td>\n",
       "      <td>162.0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>2001.0</td>\n",
       "      <td>223500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>60.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>TA</td>\n",
       "      <td>Gd</td>\n",
       "      <td>1998.0</td>\n",
       "      <td>140000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>84.0</td>\n",
       "      <td>350.0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>250000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   LotFrontage  MasVnrArea BsmtQual FireplaceQu  GarageYrBlt  SalePrice\n",
       "0         65.0       196.0       Gd         NaN       2003.0     208500\n",
       "1         80.0         0.0       Gd          TA       1976.0     181500\n",
       "2         68.0       162.0       Gd          TA       2001.0     223500\n",
       "3         60.0         0.0       TA          Gd       1998.0     140000\n",
       "4         84.0       350.0       Gd          TA       2000.0     250000"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's load the dataset with a selected group of variables\n",
    "\n",
    "cols_to_use = [\n",
    "    'BsmtQual', 'FireplaceQu', 'LotFrontage', 'MasVnrArea', 'GarageYrBlt',\n",
    "    'SalePrice'\n",
    "]\n",
    "\n",
    "data = pd.read_csv('../houseprice.csv', usecols=cols_to_use)\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LotFrontage    0.177397\n",
       "MasVnrArea     0.005479\n",
       "BsmtQual       0.025342\n",
       "FireplaceQu    0.472603\n",
       "GarageYrBlt    0.055479\n",
       "SalePrice      0.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.isnull().mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1022, 5), (438, 5))"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's separate into training and testing set\n",
    "\n",
    "# first drop the target from the feature list\n",
    "cols_to_use.remove('SalePrice')\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(data[cols_to_use],\n",
    "                                                    data['SalePrice'],\n",
    "                                                    test_size=0.3,\n",
    "                                                    random_state=0)\n",
    "X_train.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature-Engine's missing indicator selects all variables by default"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we call the imputer from feature-engine\n",
    "# we don't need to specify anything \n",
    "\n",
    "imputer = mdi.AddNaNBinaryImputer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AddNaNBinaryImputer(variables=['BsmtQual', 'FireplaceQu', 'LotFrontage',\n",
       "                               'MasVnrArea', 'GarageYrBlt'])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we fit the imputer\n",
    "\n",
    "imputer.fit(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['BsmtQual', 'FireplaceQu', 'LotFrontage', 'MasVnrArea', 'GarageYrBlt']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we see that the imputer selected all the variables numerical\n",
    "# and categorical\n",
    "\n",
    "imputer.variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>BsmtQual</th>\n",
       "      <th>FireplaceQu</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>GarageYrBlt</th>\n",
       "      <th>BsmtQual_na</th>\n",
       "      <th>FireplaceQu_na</th>\n",
       "      <th>LotFrontage_na</th>\n",
       "      <th>MasVnrArea_na</th>\n",
       "      <th>GarageYrBlt_na</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>Gd</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>573.0</td>\n",
       "      <td>1998.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>682</th>\n",
       "      <td>Gd</td>\n",
       "      <td>Gd</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1996.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>960</th>\n",
       "      <td>TA</td>\n",
       "      <td>NaN</td>\n",
       "      <td>50.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1384</th>\n",
       "      <td>TA</td>\n",
       "      <td>NaN</td>\n",
       "      <td>60.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1939.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1100</th>\n",
       "      <td>TA</td>\n",
       "      <td>NaN</td>\n",
       "      <td>60.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1930.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     BsmtQual FireplaceQu  LotFrontage  MasVnrArea  GarageYrBlt  BsmtQual_na  \\\n",
       "64         Gd         NaN          NaN       573.0       1998.0            0   \n",
       "682        Gd          Gd          NaN         0.0       1996.0            0   \n",
       "960        TA         NaN         50.0         0.0          NaN            0   \n",
       "1384       TA         NaN         60.0         0.0       1939.0            0   \n",
       "1100       TA         NaN         60.0         0.0       1930.0            0   \n",
       "\n",
       "      FireplaceQu_na  LotFrontage_na  MasVnrArea_na  GarageYrBlt_na  \n",
       "64                 1               1              0               0  \n",
       "682                0               1              0               0  \n",
       "960                1               0              0               1  \n",
       "1384               1               0              0               0  \n",
       "1100               1               0              0               0  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# feature-engine returns a dataframe\n",
    "# with the additional features\n",
    "\n",
    "# no need to contatenate!!\n",
    "\n",
    "tmp = imputer.transform(X_train)\n",
    "tmp.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BsmtQual          0.023483\n",
       "FireplaceQu       0.467710\n",
       "LotFrontage       0.184932\n",
       "MasVnrArea        0.004892\n",
       "GarageYrBlt       0.052838\n",
       "BsmtQual_na       0.000000\n",
       "FireplaceQu_na    0.000000\n",
       "LotFrontage_na    0.000000\n",
       "MasVnrArea_na     0.000000\n",
       "GarageYrBlt_na    0.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's check NA\n",
    "\n",
    "tmp.isnull().mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature-engine allows you to specify variable groups easily"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AddNaNBinaryImputer(variables=['BsmtQual', 'FireplaceQu', 'LotFrontage'])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's do it imputation but this time\n",
    "# and let's select a few variables\n",
    "\n",
    "imputer = mdi.AddNaNBinaryImputer(variables=['BsmtQual', 'FireplaceQu', 'LotFrontage'])\n",
    "\n",
    "imputer.fit(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['BsmtQual', 'FireplaceQu', 'LotFrontage']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# now the imputer uses only the variables we indicated\n",
    "\n",
    "imputer.variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>BsmtQual</th>\n",
       "      <th>FireplaceQu</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>GarageYrBlt</th>\n",
       "      <th>BsmtQual_na</th>\n",
       "      <th>FireplaceQu_na</th>\n",
       "      <th>LotFrontage_na</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>Gd</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>573.0</td>\n",
       "      <td>1998.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>682</th>\n",
       "      <td>Gd</td>\n",
       "      <td>Gd</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1996.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>960</th>\n",
       "      <td>TA</td>\n",
       "      <td>NaN</td>\n",
       "      <td>50.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1384</th>\n",
       "      <td>TA</td>\n",
       "      <td>NaN</td>\n",
       "      <td>60.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1939.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1100</th>\n",
       "      <td>TA</td>\n",
       "      <td>NaN</td>\n",
       "      <td>60.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1930.0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     BsmtQual FireplaceQu  LotFrontage  MasVnrArea  GarageYrBlt  BsmtQual_na  \\\n",
       "64         Gd         NaN          NaN       573.0       1998.0            0   \n",
       "682        Gd          Gd          NaN         0.0       1996.0            0   \n",
       "960        TA         NaN         50.0         0.0          NaN            0   \n",
       "1384       TA         NaN         60.0         0.0       1939.0            0   \n",
       "1100       TA         NaN         60.0         0.0       1930.0            0   \n",
       "\n",
       "      FireplaceQu_na  LotFrontage_na  \n",
       "64                 1               1  \n",
       "682                0               1  \n",
       "960                1               0  \n",
       "1384               1               0  \n",
       "1100               1               0  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# feature-engine returns a dataframe\n",
    "# with the additional features\n",
    "\n",
    "# no need to contatenate!!\n",
    "\n",
    "tmp = imputer.transform(X_train)\n",
    "\n",
    "tmp.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "scrolled": true
   },
   "source": [
    "## Feature-engine can be used with the Scikit-learn pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BsmtQual       0.023483\n",
       "FireplaceQu    0.467710\n",
       "LotFrontage    0.184932\n",
       "MasVnrArea     0.004892\n",
       "GarageYrBlt    0.052838\n",
       "dtype: float64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's check the percentage of NA in each variable\n",
    "\n",
    "X_train.isnull().mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>BsmtQual</th>\n",
       "      <th>FireplaceQu</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>GarageYrBlt</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>Gd</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>573.0</td>\n",
       "      <td>1998.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>682</th>\n",
       "      <td>Gd</td>\n",
       "      <td>Gd</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1996.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>960</th>\n",
       "      <td>TA</td>\n",
       "      <td>NaN</td>\n",
       "      <td>50.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1384</th>\n",
       "      <td>TA</td>\n",
       "      <td>NaN</td>\n",
       "      <td>60.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1939.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1100</th>\n",
       "      <td>TA</td>\n",
       "      <td>NaN</td>\n",
       "      <td>60.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1930.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     BsmtQual FireplaceQu  LotFrontage  MasVnrArea  GarageYrBlt\n",
       "64         Gd         NaN          NaN       573.0       1998.0\n",
       "682        Gd          Gd          NaN         0.0       1996.0\n",
       "960        TA         NaN         50.0         0.0          NaN\n",
       "1384       TA         NaN         60.0         0.0       1939.0\n",
       "1100       TA         NaN         60.0         0.0       1930.0"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are the steps we will concatenate\n",
    "\n",
    "- Add Missing Indicator to all variables\n",
    "- Median Imputation to numerical variables\n",
    "- Missing category imputation to categorical variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe = Pipeline([\n",
    "    ('missing_ind', mdi.AddNaNBinaryImputer()),\n",
    "    ('imputer_mode', mdi.FrequentCategoryImputer(variables=['FireplaceQu', 'BsmtQual'])),\n",
    "    ('imputer_median', mdi.MeanMedianImputer(imputation_method = 'median',\n",
    "                                             variables=['LotFrontage', 'MasVnrArea', 'GarageYrBlt'])),\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('missing_ind',\n",
       "                 AddNaNBinaryImputer(variables=['BsmtQual', 'FireplaceQu',\n",
       "                                                'LotFrontage', 'MasVnrArea',\n",
       "                                                'GarageYrBlt'])),\n",
       "                ('imputer_mode',\n",
       "                 FrequentCategoryImputer(variables=['FireplaceQu',\n",
       "                                                    'BsmtQual'])),\n",
       "                ('imputer_median',\n",
       "                 MeanMedianImputer(imputation_method='median',\n",
       "                                   variables=['LotFrontage', 'MasVnrArea',\n",
       "                                              'GarageYrBlt']))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# fit the pipe\n",
    "pipe.fit(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['BsmtQual', 'FireplaceQu', 'LotFrontage', 'MasVnrArea', 'GarageYrBlt']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# inspect the separate steps\n",
    "pipe.named_steps['missing_ind'].variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'FireplaceQu': 'Gd', 'BsmtQual': 'TA'}"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe.named_steps['imputer_mode'].imputer_dict_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'LotFrontage': 69.0, 'MasVnrArea': 0.0, 'GarageYrBlt': 1979.0}"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe.named_steps['imputer_median'].imputer_dict_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BsmtQual          0.0\n",
       "FireplaceQu       0.0\n",
       "LotFrontage       0.0\n",
       "MasVnrArea        0.0\n",
       "GarageYrBlt       0.0\n",
       "BsmtQual_na       0.0\n",
       "FireplaceQu_na    0.0\n",
       "LotFrontage_na    0.0\n",
       "MasVnrArea_na     0.0\n",
       "GarageYrBlt_na    0.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's transform the data with the pipeline\n",
    "\n",
    "# this pipeline will:\n",
    "#- add the missing indicators\n",
    "#- fill na in the original variables\n",
    "# leaving the dataset ready to use for ML\n",
    "\n",
    "tmp = pipe.transform(X_train)\n",
    "\n",
    "# let's check null values are gone\n",
    "tmp.isnull().mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1022, 10)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tmp.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "feml",
   "language": "python",
   "name": "feml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}