04.19-Frequent-Category-Imputation-Feature-Engine.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Frequent Category Imputation ==> Feature-Engine\n",
    "\n",
    "\n",
    "### What is Feature-Engine\n",
    "\n",
    "Feature-Engine is an open source python package that I created at the back of this course. \n",
    "\n",
    "- Feature-Engine includes all the feature engineering techniques described in the course\n",
    "- Feature-Engine works like to Scikit-learn, so it is easy to learn\n",
    "- Feature-Engine allows you to implement specific engineering steps to specific feature subsets\n",
    "- Feature-Engine can be integrated with the Scikit-learn pipeline allowing for smooth model building\n",
    "- \n",
    "**Feature-Engine allows you to design and store a feature engineering pipeline with bespoke procedures for different variable groups.**\n",
    "\n",
    "-------------------------------------------------------------------\n",
    "Feature-Engine can be installed via pip ==> pip install feature-engine\n",
    "\n",
    "- Make sure you have installed feature-engine before running this notebook\n",
    "\n",
    "For more information visit:\n",
    "my website\n",
    "\n",
    "## In this demo\n",
    "\n",
    "We will use Feature-Engine to perform mean or median imputation using the Ames House Price Dataset.\n",
    "\n",
    "- To download the dataset visit the lecture **Datasets** in **Section 1** of the course."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# to split the datasets\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "# from feature-engine\n",
    "from feature_engine import missing_data_imputers as mdi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>BsmtQual</th>\n",
       "      <th>FireplaceQu</th>\n",
       "      <th>GarageYrBlt</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>65.0</td>\n",
       "      <td>196.0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2003.0</td>\n",
       "      <td>208500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>80.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>1976.0</td>\n",
       "      <td>181500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>68.0</td>\n",
       "      <td>162.0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>2001.0</td>\n",
       "      <td>223500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>60.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>TA</td>\n",
       "      <td>Gd</td>\n",
       "      <td>1998.0</td>\n",
       "      <td>140000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>84.0</td>\n",
       "      <td>350.0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>250000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   LotFrontage  MasVnrArea BsmtQual FireplaceQu  GarageYrBlt  SalePrice\n",
       "0         65.0       196.0       Gd         NaN       2003.0     208500\n",
       "1         80.0         0.0       Gd          TA       1976.0     181500\n",
       "2         68.0       162.0       Gd          TA       2001.0     223500\n",
       "3         60.0         0.0       TA          Gd       1998.0     140000\n",
       "4         84.0       350.0       Gd          TA       2000.0     250000"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's load the dataset with a selected group of variables\n",
    "\n",
    "cols_to_use = [\n",
    "    'BsmtQual', 'FireplaceQu', 'LotFrontage', 'MasVnrArea', 'GarageYrBlt',\n",
    "    'SalePrice'\n",
    "]\n",
    "\n",
    "data = pd.read_csv('../houseprice.csv', usecols=cols_to_use)\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LotFrontage    0.177397\n",
       "MasVnrArea     0.005479\n",
       "BsmtQual       0.025342\n",
       "FireplaceQu    0.472603\n",
       "GarageYrBlt    0.055479\n",
       "SalePrice      0.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.isnull().mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the predictor variables contain missing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1022, 5), (438, 5))"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's separate into training and testing set\n",
    "\n",
    "# first drop the target from the feature list\n",
    "cols_to_use.remove('SalePrice')\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(data[cols_to_use],\n",
    "                                                    data['SalePrice'],\n",
    "                                                    test_size=0.3,\n",
    "                                                    random_state=0)\n",
    "X_train.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature-Engine captures the categorical variables automatically"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we call the imputer from feature-engine\n",
    "# we don't need to specify anything \n",
    "\n",
    "imputer = mdi.FrequentCategoryImputer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FrequentCategoryImputer(variables=['BsmtQual', 'FireplaceQu'])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we fit the imputer\n",
    "\n",
    "imputer.fit(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['BsmtQual', 'FireplaceQu']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we see that the imputer found the categorical variables to\n",
    "# impute with the frequent category\n",
    "\n",
    "imputer.variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'BsmtQual': 'TA', 'FireplaceQu': 'Gd'}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# here we can see the values that will be used\n",
    "# to replace NA for each variable\n",
    "\n",
    "imputer.imputer_dict_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>BsmtQual</th>\n",
       "      <th>FireplaceQu</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>TA</td>\n",
       "      <td>Gd</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  BsmtQual FireplaceQu\n",
       "0       TA          Gd"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's check those values agains the train data\n",
    "\n",
    "X_train[imputer.variables].mode()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>BsmtQual</th>\n",
       "      <th>FireplaceQu</th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>GarageYrBlt</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>Gd</td>\n",
       "      <td>Gd</td>\n",
       "      <td>NaN</td>\n",
       "      <td>573.0</td>\n",
       "      <td>1998.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>682</th>\n",
       "      <td>Gd</td>\n",
       "      <td>Gd</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1996.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>960</th>\n",
       "      <td>TA</td>\n",
       "      <td>Gd</td>\n",
       "      <td>50.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1384</th>\n",
       "      <td>TA</td>\n",
       "      <td>Gd</td>\n",
       "      <td>60.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1939.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1100</th>\n",
       "      <td>TA</td>\n",
       "      <td>Gd</td>\n",
       "      <td>60.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1930.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     BsmtQual FireplaceQu  LotFrontage  MasVnrArea  GarageYrBlt\n",
       "64         Gd          Gd          NaN       573.0       1998.0\n",
       "682        Gd          Gd          NaN         0.0       1996.0\n",
       "960        TA          Gd         50.0         0.0          NaN\n",
       "1384       TA          Gd         60.0         0.0       1939.0\n",
       "1100       TA          Gd         60.0         0.0       1930.0"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# feature-engine returns a dataframe\n",
    "\n",
    "tmp = imputer.transform(X_train)\n",
    "tmp.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BsmtQual       0.0\n",
       "FireplaceQu    0.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's check that the numerical variables don't\n",
    "# contain NA any more\n",
    "\n",
    "tmp[imputer.variables].isnull().mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature-engine allows you to specify variable groups easily"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FrequentCategoryImputer(variables=['BsmtQual'])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's do it imputation but this time\n",
    "# and let's do it over 1 of the 2 categorical variables\n",
    "\n",
    "imputer = mdi.FrequentCategoryImputer(variables=['BsmtQual'])\n",
    "\n",
    "imputer.fit(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['BsmtQual']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# now the imputer uses only the variables we indicated\n",
    "\n",
    "imputer.variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'BsmtQual': 'TA'}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# and we can see the value assigned to each variable\n",
    "imputer.imputer_dict_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BsmtQual    0.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# feature-engine returns a dataframe\n",
    "\n",
    "tmp = imputer.transform(X_train)\n",
    "\n",
    "# let's check null values are gone\n",
    "tmp[imputer.variables].isnull().mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It worked!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}