04.17-Arbitrary-Value-Imputation-Feature-Engine.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Arbitrary Imputation ==> Feature-Engine\n",
    "\n",
    "\n",
    "### What is Feature-Engine\n",
    "\n",
    "Feature-Engine is an open source python package that I created at the back of this course. \n",
    "\n",
    "- Feature-Engine includes all the feature engineering techniques described in the course\n",
    "- Feature-Engine works like to Scikit-learn, so it is easy to learn\n",
    "- Feature-Engine allows you to implement specific engineering steps to specific feature subsets\n",
    "- Feature-Engine can be integrated with the Scikit-learn pipeline allowing for smooth model building\n",
    "- \n",
    "**Feature-Engine allows you to design and store a feature engineering pipeline with bespoke procedures for different variable groups.**\n",
    "\n",
    "-------------------------------------------------------------------\n",
    "Feature-Engine can be installed via pip ==> pip install feature-engine\n",
    "\n",
    "- Make sure you have installed feature-engine before running this notebook\n",
    "\n",
    "For more information visit:\n",
    "my website\n",
    "\n",
    "## In this demo\n",
    "\n",
    "We will use Feature-Engine to perform mean or median imputation using the Ames House Price Dataset.\n",
    "\n",
    "- To download the dataset visit the lecture **Datasets** in **Section 1** of the course."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# to split the datasets\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "# from feature-engine\n",
    "from feature_engine import missing_data_imputers as mdi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>BsmtQual</th>\n",
       "      <th>FireplaceQu</th>\n",
       "      <th>GarageYrBlt</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>65.0</td>\n",
       "      <td>196.0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2003.0</td>\n",
       "      <td>208500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>80.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>1976.0</td>\n",
       "      <td>181500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>68.0</td>\n",
       "      <td>162.0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>2001.0</td>\n",
       "      <td>223500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>60.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>TA</td>\n",
       "      <td>Gd</td>\n",
       "      <td>1998.0</td>\n",
       "      <td>140000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>84.0</td>\n",
       "      <td>350.0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>250000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   LotFrontage  MasVnrArea BsmtQual FireplaceQu  GarageYrBlt  SalePrice\n",
       "0         65.0       196.0       Gd         NaN       2003.0     208500\n",
       "1         80.0         0.0       Gd          TA       1976.0     181500\n",
       "2         68.0       162.0       Gd          TA       2001.0     223500\n",
       "3         60.0         0.0       TA          Gd       1998.0     140000\n",
       "4         84.0       350.0       Gd          TA       2000.0     250000"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's load the dataset with a selected group of variables\n",
    "\n",
    "cols_to_use = [\n",
    "    'BsmtQual', 'FireplaceQu', 'LotFrontage', 'MasVnrArea', 'GarageYrBlt',\n",
    "    'SalePrice'\n",
    "]\n",
    "\n",
    "data = pd.read_csv('../houseprice.csv', usecols=cols_to_use)\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LotFrontage    0.177397\n",
       "MasVnrArea     0.005479\n",
       "BsmtQual       0.025342\n",
       "FireplaceQu    0.472603\n",
       "GarageYrBlt    0.055479\n",
       "SalePrice      0.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.isnull().mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the predictor variables contain missing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1022, 5), (438, 5))"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's separate into training and testing set\n",
    "\n",
    "# first drop the target from the feature list\n",
    "cols_to_use.remove('SalePrice')\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(data[cols_to_use],\n",
    "                                                    data['SalePrice'],\n",
    "                                                    test_size=0.3,\n",
    "                                                    random_state=0)\n",
    "X_train.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature-Engine captures the numerical variables automatically"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we call the imputer from feature-engine\n",
    "# we specify the arbitrary value as an argument\n",
    "\n",
    "imputer = mdi.ArbitraryNumberImputer(arbitrary_number = -999)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ArbitraryNumberImputer(arbitrary_number=-999,\n",
       "            variables=['LotFrontage', 'MasVnrArea', 'GarageYrBlt'])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we fit the imputer\n",
    "\n",
    "imputer.fit(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['LotFrontage', 'MasVnrArea', 'GarageYrBlt']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we see that the imputer found the numerical variables to\n",
    "# impute with the arbitrary value\n",
    "\n",
    "imputer.variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-999"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# here we can see the arbitrary value stored\n",
    "imputer.arbitrary_number"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>BsmtQual</th>\n",
       "      <th>FireplaceQu</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>GarageYrBlt</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>Gd</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-999.0</td>\n",
       "      <td>573.0</td>\n",
       "      <td>1998.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>682</th>\n",
       "      <td>Gd</td>\n",
       "      <td>Gd</td>\n",
       "      <td>-999.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1996.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>960</th>\n",
       "      <td>TA</td>\n",
       "      <td>NaN</td>\n",
       "      <td>50.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-999.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1384</th>\n",
       "      <td>TA</td>\n",
       "      <td>NaN</td>\n",
       "      <td>60.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1939.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1100</th>\n",
       "      <td>TA</td>\n",
       "      <td>NaN</td>\n",
       "      <td>60.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1930.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     BsmtQual FireplaceQu  LotFrontage  MasVnrArea  GarageYrBlt\n",
       "64         Gd         NaN       -999.0       573.0       1998.0\n",
       "682        Gd          Gd       -999.0         0.0       1996.0\n",
       "960        TA         NaN         50.0         0.0       -999.0\n",
       "1384       TA         NaN         60.0         0.0       1939.0\n",
       "1100       TA         NaN         60.0         0.0       1930.0"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# feature-engine returns a dataframe\n",
    "\n",
    "tmp = imputer.transform(X_train)\n",
    "tmp.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LotFrontage    0.0\n",
       "MasVnrArea     0.0\n",
       "GarageYrBlt    0.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's check that the numerical variables don't\n",
    "# contain NA any more\n",
    "\n",
    "tmp[imputer.variables].isnull().mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature-engine allows you to specify variable groups easily"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ArbitraryNumberImputer(arbitrary_number=-999,\n",
       "            variables=['LotFrontage', 'MasVnrArea'])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's do it imputation but this time\n",
    "# and let's do it over 2 of the 3 numerical variables\n",
    "\n",
    "imputer = mdi.ArbitraryNumberImputer(arbitrary_number = -999,\n",
    "                                     variables=['LotFrontage', 'MasVnrArea'])\n",
    "\n",
    "imputer.fit(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['LotFrontage', 'MasVnrArea']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# now the imputer uses only the variables we indicated\n",
    "\n",
    "imputer.variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-999"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# and we can see the value assigned to each variable\n",
    "imputer.arbitrary_number"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LotFrontage    0.0\n",
       "MasVnrArea     0.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# feature-engine returns a dataframe\n",
    "\n",
    "tmp = imputer.transform(X_train)\n",
    "\n",
    "# let's check null values are gone\n",
    "tmp[imputer.variables].isnull().mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature-engine can be used with the Scikit-learn pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[<matplotlib.axes._subplots.AxesSubplot object at 0x000000DD72ECCDA0>,\n",
       "        <matplotlib.axes._subplots.AxesSubplot object at 0x000000DD72F63710>],\n",
       "       [<matplotlib.axes._subplots.AxesSubplot object at 0x000000DD72F8C978>,\n",
       "        <matplotlib.axes._subplots.AxesSubplot object at 0x000000DD731F3BE0>]],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEICAYAAABRSj9aAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xu4HFWZ7/Hvz4Sb4ZJAYBuT6AaJFzwRDBmIg6M5glyCY+IzolFGAnLMqOjgkPMMwZkjeH3AMyjiccQoaEAuZkCGKCgywBZxJEIACRCRgBFCApFLAgnX4Hv+WKuTSqd3dvfe3Xt3V36f56mnq1ZVV721e+23q1ddliICMzMrr1cMdQBmZtZaTvRmZiXnRG9mVnJO9GZmJedEb2ZWck70ZmYl50RvvZIUkvYd6jjMbGCc6PsgaaakRZLWS1qdxz8pSUMdW4Wkj0laKmmHQtkeOd4je3nPVEl/kbQuD49I+vxWtvEDSV9qRfzWPiQtl3RYg+/Z7ICgRt1aJ+knzY/W9bJeTvRbIWkO8A3g/wKvArqAjwOHANs3uK7hTQ8wi4jvAiuAzxWKzwGuiYifbyWWlRGxc0TsDLwdOFHSjFbFaduUjXUrD39ba6FW/l9YQUR4qDEAuwHrgb/byjJHA3cATwMPA2cU5nUDAZwIPATclMv/A3gUWAvcBLy58J49gJ/k9d0KfAm4uTD/jcB1wJPAfcAHqrb3FHAAcDiwEhiV500lfRGcmrd9UaWsan8WAJ8tTAewLzAbeAl4EVgH/GSoPx8PrRmA5cBhNco/BizLdW8h8OpcflOuJ+tz3fhgrbpVWM8ZwOXAD3M9/1/ADqQDk5V5OAfYIS9fqbtzgNXAKuCEPK9mvQTmAg8AzwD3Au8rbH8YcDbwOPBH4FM5/uF5/m7A+Xk7j+T/wWFD/bkM+HMd6gDadQCOBDZUKkAvy0wFJpJ+Gb0FeAyYked15wp0ITAC2CmXfxTYpVC57yys77I8vBLYj/TlcXOeNyJPnwAMByblylr8ovg0cHuuwDOq4twAnJW3u1P1PyMwIVfsdxXKAtg3j/8A+NJQfy4eWjtQI9ED78p1bVKuP98kH7hU15M8vVndqlrXGTk5z8j/NzsBXwBuAfYC9gT+G/hiYV0b8jLbAdOAZ9l0ELNFvQSOAV6d1/9B0pfQmDzv46TkPw4YBfwXmyf6/wS+k//f9gJ+C/zDUH8uA/5chzqAdh2AvwcerSr7b2AN8BzwjhrvOQf4eh7vzhVon61sY2ReZjfSkcZLwBsK8zce0ecK+6uq938HOL0wLWARcGXVclNJRz07VpX9Je/P0zmOHwPbF5Zxot/Ghl4S/fnAVwvTO+e62l1dT/J0sW5Vhg/keWdQ+JLIZQ8A0wrTRwDLC+t6jsIBF+nIfkoe77NeAncC0/P4DcXEDRxWSfSkptkXyAdlef6HgBuH+nMZ6OD2sd49AYyWNDwiNgBExF8DSFoBvELSwcCZwP8gtdnvQGqaKXq4MiJpGPBl0hHHnqR/BoDRpCOb4cXlq8ZfCxwsaU2hbDipGYYcX0haSvqpW+3PEfF8VdnKiBiXY9sN+HdgPqlym1W8mvRLEYCIWCfpCWAs6Yuhlo11q4aHq6ZfDfypMP2nXFbxROV/MHuW9GVTk6TjgFNIB1vkZUcXtrW1/7HtgFWFay1eUSPejuOTsb37DenbffpWlrmE1F45PiJ2A84jHVUXFR8P+uG8vsNIR/HduVzAn0k/UYv/HOML4w8Dv4yIkYVh54j4RJ37s9XHlEbE2rw/NU+a9fV+K7WVpCQIgKQRpPNJj/RzfdV1abP1A6/JZQ2vS9Jrge+S2t73iIiRwN1s+r9cxdb/x14ARhf+x3aNiDfXGUvbcqLvRUSsAT4P/Luk90vaWdIrJB1Aar+D1Nb+ZEQ8L+kgUiLfml1IFekJUjv8Vwrbe5nUdHKGpFdKeiNwXOG9PwVeL+kjkrbLw19JelMz9lfSzsBM4J5eFnkM2KcZ27K2t52kHSsD6ST9CZIOyJfwfgVYFBHL8/IDrRuXAv8qaU9Jo0lXj/2wzvdWb3sEKfn/GUDSCaRf3BULgJMljZU0knSBAgARsQr4BXC2pF3z//vrJL2zvzvWLpzotyIivkr6CfjPpHbBx0jt4qeS2us/CXxB0jOkyrmgj1VeSPpZ+gjphNAtVfM/RTrSr1wZcynpi4GIeIZ0Nc1M0tHOo2w6udpfr65c55zj2h04tpdlzwf2k7RG0n8OYJvW/q4htYtXhr8B/g9wBemI+HWkelhxBjA/140P9GN7XwJuA+4ClpCaieq9Nn6zehkR95KuqvkN6f91IvDrwvLfJSXzu0hXzF1D+iX9cp5/HKkZ9l7SVWyXA2P6sU9tRfmEg7UhSWcBr4qIWUMdi1kZSToKOC8iXtvnwh3MR/RtRNIbJb1FyUGka/CvHOq4zMpC0k6SpkkaLmkscDrbwP+YE3172YXUTr+e1Ax0NnDVkEZkVi4inXt7itR0s5TN7ygvJTfdmJmVnI/ozcxKrs8bpvLlVTeRru4YDlweEadL2pt0u/7upLPkH4mIF/PlVxcCB5IuI/xg4TKsmkaPHh3d3d01561fv54RI0bUnFdG3t/+W7x48eMRsWdTVtZiW6vz7aYT6+S2EnPddb6OW6IF7JzHtyPdYj+F1IY8M5efB3wij3+SdBYb0iVYP+prGwceeGD05sYbb+x1Xhl5f/sPuC3a4Hbzeoat1fl204l1cluJud4632fTTV7fujy5XR6C9KCjy3P5fNJDiiDd+Tk/j18OHNpOz243M9vW1PWsm/yMlsWkR9Z+i/QQojWx6fkTK0jPvSC/PgwQERskrSXdLv141Tpnkx4zSldXFz09PTW3vW7dul7nlZH318yara5EH+n2/APyLcNXArVuu69cvlPr6H2LS3siYh4wD2Dy5MkxderUmtvu6emht3ll5P01s2Zr6OmVEbFGUg+pjX5k4cmO49j0EKIVpAcFrci9x+xG6qzAbDPdc69mzsQNHD/36rrfs/zMo1sYUefqbuBvWOG/5bajzzb6/KChkXl8J9KTF5cCNwLvz4vNYtONPQvzNHn+DfmkgZmZDYF6jujHkB5YNIz0xbAgIn4q6V7gstwx7x2khwuRXy+SVOl2bGatlZqZ2eDoM9FHxF3AW2uUPwgcVKP8eVLHGmZm1gZ8Z6xZLyQNk3SHpJ/m6b0lLZJ0v6QfSdo+l++Qp5fl+d1DGbdZNSd6s96dTDofVXEWqU/gCaSHYp2Yy08EnoqIfYGv5+XM2oYTvVkNksYBRwPfy9PCNwlah3Ln4Ga1nUPqWWyXPL0Hg3STYH/Mmbih74Wq1Lv9TrypzTFvzonerIqk9wCrI2KxpKmV4hqLtuQmwf5o5F6EiuXH1rf9TrypzTFvzonebEuHAO+VNA3YEdiVdITvmwStI7mN3qxKRJwWEeMiopt0H8gNEXEsvknQOpQTvVn9TgVOyTcD7sHmNwnukctPAeYOUXxmNbnpxmwrIqIH6MnjvknQOpKP6M3MSs6J3sys5JzozcxKzonezKzknOjNzErOid7MrOSc6M3MSs6J3sys5JzozcxKzonezKzknOjNzErOid7MrOT6TPSSxku6UdJSSfdIOjmX7y7putxR8nWSRuVySTo3d5R8l6RJrd4JMzPrXT1H9BuAORHxJmAKcJKk/UiPYr0+d5R8PZsezXoUMCEPs4FvNz1qMzOrW5+JPiJWRcTtefwZYCmpj8xih8jVHSVfGMktpF55xjQ9cjMzq0tDz6OX1A28FVgEdEXEKkhfBpL2yott7Cg5q3SivKpqXXV1lNyJnfwOxLa0v3MmbqBrp8Y6tt5W/jZmzVR3ope0M3AF8JmIeFqq1R9yWrRGWb87Su7ETn4HYlva3+PnXs2ciRs4e0n9xxv1dmhtZpvUddWNpO1ISf7iiPhxLn6s0iSTX1fn8kpHyRXFTpTNzGyQ1XPVjUh9Yi6NiK8VZhU7RK7uKPm4fPXNFGBtpYnHzMwGXz2/mQ8BPgIskXRnLvsscCawQNKJwENs6jPzGmAasAx4FjihqRGbmVlD+kz0EXEztdvdAQ6tsXwAJw0wLjMzaxLfGWtmVnJO9GZmJedEb2ZWck70ZmYl50RvZlZyTvRmZiXnRG9mVnJO9GZmJedEb2ZWck70ZmYl19Dz6M2sPLrnXl3XcnMmbuD4wrLLzzy6VSFZi/iI3qyK+0m2svERfYnVe8RW5KM1YFM/ybdL2gVYLOk64HhSP8lnSppL6if5VDbvJ/lgUj/JBw9J5GY1ONFbU/TnS6Vd5f4TKt1kPiOp2E/y1LzYfKCHlOg39pMM3CJppKQx7ofB2oUTfYcoUyLtJM3sJ9lsqDjRm/Wi2f0kS5oNzAbo6upqakfnjXSw3qjqDtw7oYP2devWdUScRa2M2YnerIat9ZOcj+Yb7ic5IuYB8wAmT54czewE/vgW/uKr7sC9Ezpo7+npoZl/38HQyph91Y1ZFfeTbGXjI3qzLbmfZCsVJ3qzKu4n2crGTTdmZiXnRG9mVnJ9JnpJF0haLenuQplvBTcz6xD1HNH/ADiyqmwu6VbwCcD1eRo2vxV8NulWcDMzG0J9JvqIuAl4sqp4OukWcPLrjEL5hZHcAozM1xubmdkQ6e9VNwO+FbzeuwQ78Q63gehtf1t552NRf//W/Y2v+q7LvmxLdcGsWZp9eWVdt4JD/XcJduIdbgPR2/628s7HzSxZ38839q8qVd912ZdOuCtzoPxcI2u2/l5181ilSaY/t4Kbmdng6W+i963gZmYdos/fzJIuJT2De7SkFcDp+FZwM7OO0Weij4gP9TLLt4KbmXUA3xlrZlZyTvRmZiXnRG9mVnJO9GZmJedEb2ZWck70ZmYl5x6mhsDWbnGfM3HD4D3uwMy2CT6iNzMrOSd6M7OSc6I3Mys5J3ozs5JzojczKzknejOzkvPllVl/evVZfubRLYjEzKy5fERvZlZyPqIfAPftaWadwEf0ZmYl50RvZlZypWy6cZOKmdkmPqI3Myu5Uh7Rm1nr+FLkzuMjejOzkmtJopd0pKT7JC2TNLcV2zBrN6731q6a3nQjaRjwLeDdwArgVkkLI+Le/qxvySNr3RGHtb1m13uzZmpFG/1BwLKIeBBA0mXAdMAV3srM9X4rButKOJ8LqK0ViX4s8HBhegVwcPVCkmYDs/PkOkn39bK+0cDjTY2wjf2j93erdNZWZ792oPEMQJ/1voE631Y6qU4W6kfHxFzQn5jrqvOtSPSqURZbFETMA+b1uTLptoiY3IzAOoH3t2P1We/rrfPtphM/I8e8uVacjF0BjC9MjwNWtmA7Zu3E9d7aVisS/a3ABEl7S9oemAksbMF2zNqJ6721raY33UTEBkmfAq4FhgEXRMQ9A1hlx/3UHSDvbwdqQb1vJ534GTnmAkVs0XxuZmYl4jtjzcxKzonezKzkBj3RS7pA0mpJdxfK9pf0G0lLJP1E0q6FeaflW8rvk3REobwjbjdvZH8ldUt6TtKdeTiv8J4D8/LLJJ0rqdblfENO0nhJN0paKukeSSfn8t0lXSfp/vw6Kpcr788ySXdJmlRY16y8/P2SZg3VPpVZL/Wz4c9qkGNuWh0b5Lh3lPRbSb/LcX8+l+8taVGO+0f5ZD6SdsjTy/L87n5vPCIGdQDeAUwC7i6U3Qq8M49/FPhiHt8P+B2wA7A38ADpRNewPL4PsH1eZr/B3pcW7G93cbmq9fwWeBvpeu2fAUcN9b71EucYYFIe3wX4Q/4cvwrMzeVzgbPy+LS8PwKmAIty+e7Ag/l1VB4fNdT7V7ahl/rZ0GfVqXVsCOIWsHMe3w5YlONZAMzM5ecBn8jjnwTOy+MzgR/1e9tDtMObJTTgaTadGB4P3JvHTwNOKyx3bU52bwOuLZRvtly7DQ3sb81Enyv27wvTHwK+M9T7Vee+X0V6/st9wJjC/tyXx78DfKiw/H15/mb7WL2ch6Z+RtX1s6HPqg3i71cdG+KYXwncTrp7+nFgeC7fmNsq+S6PD8/LqT/ba5c2+ruB9+bxY9h040mt28rHbqW8U/S2vwB7S7pD0i8l/U0uG0vax4qO2N/8U/OtpCOXrohYBZBf98qLlfUz7mSNflZDZoB1bNBJGibpTmA1cB2pZWJNRGyoEdvGuPP8tcAe/dluuyT6jwInSVpM+in2Yi7v7bbyuh6z0MZ6299VwGsi4q3AKcAluf2+4/ZX0s7AFcBnIuLprS1ao6wMn3EZtdVn0oQ6Nugi4uWIOIB05/RBwJtqLZZfmxZ3WyT6iPh9RBweEQcCl5K+5aD328o7+nbz3vY3Il6IiCfy+OJc/nrS/o4rrKKt91fSdqR/wIsj4se5+DFJY/L8MaQjGijpZ9zhGv2sBl2T6tiQiYg1QA+pjX6kpMrNq8XYNsad5+8GPNmf7bVFope0V359BfCvpBMSkG4hn5nPPu8NTCCdlGz57eaSlkt6UdLoqvI7JUV/z4BL+o6kBXl84/5KeoukFyrbk7QPaX8fzD9Dn5E0JV9tcxypXbLW+nskPSVph/7EN1A5vvOBpRHxtcKshUDlyplZbIp/IXBcvjJiCrA27++1wOGSRuWrJw7PZdZ6jX5Wg6qJdWxQSdpT0sg8vhNwGLAUuBF4f16sOu7K/rwfuCFyg33DhuAkxKWkJoqXSN9YJwInk86c/wE4k8IJB+BfSEe291G40oR0Jv0Ped6/tCDO5Xmbny6UTcxlAXT3c3+/DLwA3F/cX+DfgN8A95CuIrod+NvCeiaT2vYfAP5f5W9EPomTx7uBl0nf+sf0EdewFn2+b89/n7uAO/MwjdS2eH3e7+uB3fPyInXY8QCwBJhcWNdHgWV5OGGw6+q2MPTy/9jwZzXIMTetjg1y3G8B7shx3w18LpfvQzqAXQb8B7BDLt8xTy/L8/fp77b9CIReSFoOfA+YHhF/lcv+DXgK+BLpcs835/HXkU6UnB8RZ+Rld8zvP4p0Oej9wHsi4jGl55B/OSIuzMsOI510+XhELJR0BulyseeB9wEPAbMi4rZCbN8GjgXeAIyI9KyVzwFHkE5MvT4i3lPYnx8Az5GeX/1OUqcYvyJ98XyAdAnrlcA/RcRz+Sj6ItJVAcOBX+f4iieFzawDtEXTTRu7BdhV0ptyMv4g8MPC/PWkZpSRwNHAJyTNyPNmkdrUxpOOND5OSrQAF+b3VRxGuq72Z4Wy9wKX5XUvJB3FF30ob3NkbDpjfxxwcR6OkNRV9Z4PkxL7LsDNwFmkcwAHAPuSzvJ/Li/7CuD7pC+G1+TYq2Mwsw7gRN+3i0gJ9N3A74FHKjMioicilkTEXyLiLtLP4Hfm2S+REvy+kc60L45NVwZcBLxTUuUE63HAJRHxUmG7N0fENRHxcl5+/6q4zo2IhyPiOQBJbycl5QWx6UTuh6vec1VE/Doi/kJqPvoY6Qj+yYh4BvgK6XwHEfFERFwREc/meV8u7JuZdRAn+r5dREqYx5OOxDeSdHC+FfvPktaSjtpHF953LXCZpJWSvpqvFCAiHgJuAv4+XyI2A5hftd1HC+PPAjsWzszD5tcFQ/oF8YuIqHRFdgmbTuTUes+epJs2FktaI2kN8PNcjqRX5hPHf5L0dI53ZP5lY2YdxIm+DxHxJ+CPpJM9P66afQmpWWV8ROxGulpI+X0vRcTnI2I/4K+B97B5c838PP13wB8j4vZGQ6uM5DP4HyD9SnhU0qPAPwH7S9q/1ntId9k9B7w5IkbmYbeI2DnPn0Nq/z84InYl3SpPZf/MrHM40dfnROBdEbG+qnwX4MmIeF7SQRSaSiT9T0kT8xHw06SmnJcL772C1H7/ebY8mm/UjLzu/Ujt7QeQbsT4FZt/uWyUm2++C3y9cHnrWG16cNwupC+CNZJ2B04fYIxmNkSc6OsQEQ9Urnip8kngC5KeIZ3EXFCY9yrgclKSXwr8ksKJ3PylUUn2Fw8wxFnA9yPioYh4tDKQTp4eW9XkU3Qq6dKtW3LzzH+RjuIBzgF2Ih3530Jq1jGzDuTLK83MSs5H9GZmJedEb2ZWck70ZmYl50RvZlZyvV2NMahGjx4d3d3dNeetX7+eESNGDG5ADXKMzTOQOBcvXvx4ROzZ5JDMOl5bJPru7m5uu63W1YvQ09PD1KlTBzegBjnG5hlInJL+1NxozMrBTTdmZiXnRG9mVnJO9GZmJdcWbfRbs+SRtRw/9+qG3rP8zKNbFI2ZWefxEb2ZWck50ZuZlZwTvZlZyTnRm5mVnBO9mVnJOdGbmZWcE72ZWck50ZuZlZwTvZlZyTnRm5mVnBO9mVnJOdGbmZWcE72ZWck50ZuZlVxdiV7SSEmXS/q9pKWS3iZpd0nXSbo/v47Ky0rSuZKWSbpL0qTW7oKZmW1NvUf03wB+HhFvBPYHlgJzgesjYgJwfZ4GOAqYkIfZwLebGrGZmTWkz0QvaVfgHcD5ABHxYkSsAaYD8/Ni84EZeXw6cGEktwAjJY1peuRmZlaXenqY2gf4M/B9SfsDi4GTga6IWAUQEask7ZWXHws8XHj/ily2qrhSSbNJR/x0dXXR09NTc+NdO8GciRvq3R+AXtfVKuvWrRv0bTaqE2KEzonTrJPUk+iHA5OAT0fEIknfYFMzTS2qURZbFETMA+YBTJ48OaZOnVpzZd+8+CrOXtJYj4fLj629rlbp6emht/jbRSfECJ0Tp1knqaeNfgWwIiIW5enLSYn/sUqTTH5dXVh+fOH944CVzQnXzMwa1Weij4hHgYclvSEXHQrcCywEZuWyWcBVeXwhcFy++mYKsLbSxGNmZoOv3jaRTwMXS9oeeBA4gfQlsUDSicBDwDF52WuAacAy4Nm8rJmZDZG6En1E3AlMrjHr0BrLBnDSAOMyM7Mm8Z2xZmYl50RvZlZyTvRmZiXnRG9mVnJO9GZmJedEb2ZWck70ZmYl50RvZlZyTvRmZiXnRG9mVnJO9GZmJedEb2ZWck70ZmYl50RvZlZyTvRmZiVXd6KXNEzSHZJ+mqf3lrRI0v2SfpQ7JUHSDnl6WZ7f3ZrQzcysHo0c0Z8MLC1MnwV8PSImAE8BJ+byE4GnImJf4Ot5OTMzGyJ1JXpJ44Cjge/laQHvInUUDjAfmJHHp+dp8vxD8/JmZjYE6u0z9hzgn4Fd8vQewJqI2JCnVwBj8/hY4GGAiNggaW1e/vHiCiXNBmYDdHV10dPTU3PDXTvBnIkbas7rTW/rapV169YN+jYb1QkxQufEadZJ+kz0kt4DrI6IxZKmVoprLBp1zNtUEDEPmAcwefLkmDp1avUiAHzz4qs4e0m930fJ8mNrr6tVenp66C3+dtEJMULnxGnWSerJoIcA75U0DdgR2JV0hD9S0vB8VD8OWJmXXwGMB1ZIGg7sBjzZ9MjNzKwufbbRR8RpETEuIrqBmcANEXEscCPw/rzYLOCqPL4wT5Pn3xARWxzRm5nZ4BjIdfSnAqdIWkZqgz8/l58P7JHLTwHmDixEMzMbiIYavyOiB+jJ4w8CB9VY5nngmCbEZmZmTeA7Y83MSs6J3sys5JzozcxKzonezKzknOjNzErOid7MrOSc6M3MSs6J3sys5JzozcxKzonezKzknOjNzErOid7MrOSc6M3MSs6J3sys5JzozcxKrs9EL2m8pBslLZV0j6STc/nukq6TdH9+HZXLJelcScsk3SVpUqt3wszMelfPEf0GYE5EvAmYApwkaT9Sz1HXR8QE4Ho29SR1FDAhD7OBbzc9ajMzq1s9fcauiojb8/gzwFJgLDAdmJ8Xmw/MyOPTgQsjuYXUifiYpkduZmZ1aagrQUndwFuBRUBXRKyC9GUgaa+82Fjg4cLbVuSyVVXrmk064qerq4uenp6a2+zaCeZM3NBImL2uq1XWrVs36NtsVCfECJ0Tp1knqTvRS9oZuAL4TEQ8LanXRWuUxRYFEfOAeQCTJ0+OqVOn1lzZNy++irOXNPR9xPJja6+rVXp6eugt/nbRCTFC58Rp1knquupG0nakJH9xRPw4Fz9WaZLJr6tz+QpgfOHt44CVzQnXzMwaVc9VNwLOB5ZGxNcKsxYCs/L4LOCqQvlx+eqbKcDaShOPmZkNvnraRA4BPgIskXRnLvsscCawQNKJwEPAMXneNcA0YBnwLHBCUyOuQ/fcqxt+z/Izj25BJGZmQ6/PRB8RN1O73R3g0BrLB3DSAOMyM7Mm8Z2xZmYl50RvZlZyTvRmZiXnRG9mVnJO9GZmJedEb2ZWck70ZmYl50RvZlZyTvRmZiXnRG9mVnKNPf+3xPx8HDMrKx/Rm5mVnBO9mVnJOdGbmZWcE72ZWcm1JNFLOlLSfZKWSZrbim2YmVl9mn7VjaRhwLeAd5P6j71V0sKIuLfZ2xpqlSt15kzcwPH9uGqnEb7Cx8z6qxWXVx4ELIuIBwEkXQZMB0qX6AeTL/80s/5qRaIfCzxcmF4BHFy9kKTZwOw8uU7Sfb2sbzTweFMjbLJ/bNMYddZmk20ZYw0DifO1zQzErCxakehr9S8bWxREzAPm9bky6baImNyMwFrFMTZPp8Rp1klacTJ2BTC+MD0OWNmC7ZiZWR1akehvBSZI2lvS9sBMYGELtmNmZnVoetNNRGyQ9CngWmAYcEFE3DOAVfbZvNMGHGPzdEqcZh1DEVs0n5uZWYn4zlgzs5JzojczK7m2TfTt9BgFScslLZF0p6Tbctnukq6TdH9+HZXLJencHPddkia1MK4LJK2WdHehrOG4JM3Ky98vadYgxHiGpEfy3/NOSdMK807LMd4n6YhCedvUB7OOExFtN5BO4j4A7ANsD/wO2G8I41kOjK4q+yowN4/PBc7K49OAn5HuJ5gCLGphXO8AJgF39zcuYHfgwfw6Ko+PanGMZwD/u8ay++XPegdg71wHhrVbffDgodOGdj2i3/gYhYh4Eag8RqGdTAfm5/H5wIxC+YWR3AKMlDSmFQFExE3AkwOM6wjguoh4MiKeAq4DjmxxjL2ZDlwWES9ExB+BZaS60An1waxttWuir/UYhbFDFAukO3t/IWlxfnQDQFdErALIr3vl8qGOvdG4hireT+UmpAsqzUttGKNZKbRroq/rMQqD6JCImAQcBZwk6R0JvsdZAAABRUlEQVRbWbbdYq/oLa6hiPfbwOuAA4BVwNm5vJ1iNCuNdk30bfUYhYhYmV9XA1eSmhIeqzTJ5NfVefGhjr3RuAY93oh4LCJejoi/AN8l/T3bKkazMmnXRN82j1GQNELSLpVx4HDg7hxP5QqVWcBVeXwhcFy+ymUKsLbSlDJIGo3rWuBwSaNyE8rhuaxlqs5ZvI/096zEOFPSDpL2BiYAv6WN6oNZJ2rF0ysHLJr/GIWB6AKulATp73VJRPxc0q3AAkknAg8Bx+TlryFd4bIMeBY4oVWBSboUmAqMlrQCOB04s5G4IuJJSV8kJVOAL0REvSdP+xvjVEkHkJpflgP/kGO5R9ICUt8FG4CTIuLlvJ52qQ9mHcePQDAzK7l2bboxM7MmcaI3Mys5J3ozs5JzojczKzknejOzknOiNzMrOSd6M7OS+/9gE4z8TUmgKAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 4 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# let's look at the distributions to determine the\n",
    "# arbitraty values to use\n",
    "\n",
    "X_train.hist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe = Pipeline([\n",
    "    ('imputer_999', mdi.ArbitraryNumberImputer(arbitrary_number = -999,\n",
    "                                             variables = ['LotFrontage', 'MasVnrArea'])),\n",
    "     \n",
    "    ('imputer_minus1', mdi.ArbitraryNumberImputer(arbitrary_number = -1,\n",
    "                                          variables = ['GarageYrBlt'])),\n",
    "     ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "     steps=[('imputer_999', ArbitraryNumberImputer(arbitrary_number=-999,\n",
       "            variables=['LotFrontage', 'MasVnrArea'])), ('imputer_minus1', ArbitraryNumberImputer(arbitrary_number=-1, variables=['GarageYrBlt']))])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe.fit(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-999"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe.named_steps['imputer_999'].arbitrary_number"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe.named_steps['imputer_minus1'].arbitrary_number"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BsmtQual       0.023483\n",
       "FireplaceQu    0.467710\n",
       "LotFrontage    0.000000\n",
       "MasVnrArea     0.000000\n",
       "GarageYrBlt    0.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's transform the data with the pipeline\n",
    "tmp = pipe.transform(X_train)\n",
    "\n",
    "# let's check null values are gone\n",
    "tmp.isnull().mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are no more null values for the 3 imputed numerical variables."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}