04.12-Missing-Category-Imputation-Sklearn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Missing Category imputation with Scikit-learn: SimpleImputer\n",
    "\n",
    "Scikit-learn provides a class to make most of the most common data imputation techniques.\n",
    "\n",
    "The **SimpleImputer** class provides basic strategies for imputing missing values, including:\n",
    "\n",
    "- Mean and median imputation for numerical variables\n",
    "- Most frequent category imputation for categorical variables\n",
    "- Arbitrary value imputation for both categorical and numerical variables\n",
    "\n",
    "### Advantages\n",
    "\n",
    "- Simple to use if applied to the entire dataframe\n",
    "- Maintained by the scikit-learn developers: good quality code\n",
    "- Fast computation (it uses numpy for calculations)\n",
    "- Allows for grid search over the various imputation techniques\n",
    "- Allows for different missing values encodings (you can indicate if the missing values are np.nan, or zeroes, etc)\n",
    "\n",
    "### Limitations\n",
    "\n",
    "- Returns a numpy array instead of a pandas dataframe, inconvenient for data analysis\n",
    "- Needs to use additional classes to select which features to impute  ==>\n",
    "    - requires more lines of code\n",
    "    - additional classes still in beta (may change without warning)\n",
    "    - not so straightforward to use anymore.\n",
    "\n",
    "### More details about the transformers\n",
    "\n",
    "- [SimpleImputer](https://scikit-learn.org/stable/modules/generated/sklearn.impute.SimpleImputer.html#sklearn.impute.SimpleImputer)\n",
    "- [ColumnTransformer](https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html)\n",
    "- [Stackoverflow](https://stackoverflow.com/questions/54160370/how-to-use-sklearn-column-transformer)\n",
    "\n",
    "\n",
    "## In this demo:\n",
    "\n",
    "We will do **missing category imputation with Scikit-learn** using the Ames House Price Dataset.\n",
    "\n",
    "- To download the dataset please refer to the lecture **Datasets** in **Section 1** of this course."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# these are the objects we need to impute missing data\n",
    "# with sklearn\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.impute import SimpleImputer\n",
    "\n",
    "# to split the datasets\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>BsmtQual</th>\n",
       "      <th>FireplaceQu</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Gd</td>\n",
       "      <td>NaN</td>\n",
       "      <td>208500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>181500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>223500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>TA</td>\n",
       "      <td>Gd</td>\n",
       "      <td>140000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>250000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  BsmtQual FireplaceQu  SalePrice\n",
       "0       Gd         NaN     208500\n",
       "1       Gd          TA     181500\n",
       "2       Gd          TA     223500\n",
       "3       TA          Gd     140000\n",
       "4       Gd          TA     250000"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's load the dataset with a few categorical columns\n",
    "\n",
    "# these are categorical columns and the target SalePrice\n",
    "cols_to_use = ['BsmtQual', 'FireplaceQu', 'SalePrice']\n",
    "\n",
    "data = pd.read_csv('../houseprice.csv', usecols=cols_to_use)\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BsmtQual       0.025342\n",
       "FireplaceQu    0.472603\n",
       "SalePrice      0.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's inspect the percentage of missing values in each variable\n",
    "\n",
    "data.isnull().mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The variables LotFrontage, MasVnrArea and GarageYrBlt contain missing data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1022, 2), (438, 2))"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's separate into training and testing set\n",
    "\n",
    "# first let's remove the target from the features\n",
    "cols_to_use.remove('SalePrice')\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    data[cols_to_use],  # just the features\n",
    "    data['SalePrice'],  # the target\n",
    "    test_size=0.3,  # the percentage of obs in the test set\n",
    "    random_state=0)  # for reproducibility\n",
    "\n",
    "X_train.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BsmtQual       0.023483\n",
       "FireplaceQu    0.467710\n",
       "dtype: float64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's check the misssing data again\n",
    "X_train.isnull().mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Gd', 'TA', 'Fa', nan, 'Ex'], dtype=object)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's inspect the values of the categorical variable\n",
    "X_train['BsmtQual'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([nan, 'Gd', 'TA', 'Fa', 'Po', 'Ex'], dtype=object)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's inspect the values of the categorical variable\n",
    "X_train['FireplaceQu'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SimpleImputer(copy=True, fill_value='Missing', missing_values=nan,\n",
       "       strategy='constant', verbose=0)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now we impute the missing values with SimpleImputer\n",
    "\n",
    "# create an instance of the simple imputer\n",
    "# we indicate that we want to impute by replacing NA\n",
    "# with 'Missing'\n",
    "\n",
    "imputer = SimpleImputer(strategy='constant', \n",
    "                       fill_value = 'Missing')\n",
    "\n",
    "# we fit the imputer to the train set\n",
    "# the imputer will learn the median of all variables\n",
    "imputer.fit(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Missing', 'Missing'], dtype=object)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we can look at the learnt modes like this:\n",
    "imputer.statistics_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([['Gd', 'Missing'],\n",
       "       ['Gd', 'Gd'],\n",
       "       ['TA', 'Missing'],\n",
       "       ...,\n",
       "       ['Missing', 'Missing'],\n",
       "       ['Gd', 'TA'],\n",
       "       ['Gd', 'Missing']], dtype=object)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# and now we impute the train and test set\n",
    "\n",
    "# NOTE: the data is returned as a numpy array!!!\n",
    "X_train = imputer.transform(X_train)\n",
    "X_test = imputer.transform(X_test)\n",
    "\n",
    "X_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>BsmtQual</th>\n",
       "      <th>FireplaceQu</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Gd</td>\n",
       "      <td>Missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Gd</td>\n",
       "      <td>Gd</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>TA</td>\n",
       "      <td>Missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>TA</td>\n",
       "      <td>Missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>TA</td>\n",
       "      <td>Missing</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  BsmtQual FireplaceQu\n",
       "0       Gd     Missing\n",
       "1       Gd          Gd\n",
       "2       TA     Missing\n",
       "3       TA     Missing\n",
       "4       TA     Missing"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# encode the train set back to a dataframe:\n",
    "\n",
    "X_train = pd.DataFrame(X_train, columns=cols_to_use)\n",
    "X_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Gd', 'TA', 'Fa', 'Missing', 'Ex'], dtype=object)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train['BsmtQual'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BsmtQual       0.0\n",
       "FireplaceQu    0.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.isnull().mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**A MASSIVE NOTE OF CAUTION**:\n",
    "\n",
    "Note that when using SimpleImputer and setting the parameters to:\n",
    "- strategy='constant'\n",
    "- fill_value = 'Missing'\n",
    "\n",
    "If your dataframe contains variables that are numerical and categorical, NA in both will be replaced by 'Missing\" therefore converting your numerical variables into categorical, which is probably not what you are after.\n",
    "\n",
    "Most datasets contain both numerical and categorical variables, so very likely you will have to use a column transformer as shown in previous notebooks and as I also show below again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>BsmtQual</th>\n",
       "      <th>FireplaceQu</th>\n",
       "      <th>GarageYrBlt</th>\n",
       "      <th>SalePrice</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>65.0</td>\n",
       "      <td>196.0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2003.0</td>\n",
       "      <td>208500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>80.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>1976.0</td>\n",
       "      <td>181500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>68.0</td>\n",
       "      <td>162.0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>2001.0</td>\n",
       "      <td>223500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>60.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>TA</td>\n",
       "      <td>Gd</td>\n",
       "      <td>1998.0</td>\n",
       "      <td>140000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>84.0</td>\n",
       "      <td>350.0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>TA</td>\n",
       "      <td>2000.0</td>\n",
       "      <td>250000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   LotFrontage  MasVnrArea BsmtQual FireplaceQu  GarageYrBlt  SalePrice\n",
       "0         65.0       196.0       Gd         NaN       2003.0     208500\n",
       "1         80.0         0.0       Gd          TA       1976.0     181500\n",
       "2         68.0       162.0       Gd          TA       2001.0     223500\n",
       "3         60.0         0.0       TA          Gd       1998.0     140000\n",
       "4         84.0       350.0       Gd          TA       2000.0     250000"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's load the dataset with both numerical and categorical variables\n",
    "\n",
    "cols_to_use = [\n",
    "    'BsmtQual', 'FireplaceQu', 'LotFrontage', 'MasVnrArea', 'GarageYrBlt',\n",
    "    'SalePrice'\n",
    "]\n",
    "\n",
    "data = pd.read_csv('../houseprice.csv', usecols=cols_to_use)\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1022, 5), (438, 5))"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's separate into training and testing set\n",
    "\n",
    "# first drop the target from the feature list\n",
    "cols_to_use.remove('SalePrice')\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(data[cols_to_use],\n",
    "                                                    data['SalePrice'],\n",
    "                                                    test_size=0.3,\n",
    "                                                    random_state=0)\n",
    "X_train.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BsmtQual       0.023483\n",
       "FireplaceQu    0.467710\n",
       "LotFrontage    0.184932\n",
       "MasVnrArea     0.004892\n",
       "GarageYrBlt    0.052838\n",
       "dtype: float64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's look at the missing values\n",
    "\n",
    "X_train.isnull().mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this demo, I will impute the numerical variables by the mean, and the categorical variables by the most frequent category."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# first we need to make lists, indicating which features\n",
    "# will be imputed with each method\n",
    "\n",
    "features_numeric = ['LotFrontage', 'MasVnrArea', 'GarageYrBlt']\n",
    "features_categoric = ['BsmtQual', 'FireplaceQu']\n",
    "\n",
    "# then we instantiate the imputers, within a pipeline\n",
    "# we create one imputer for numerical and one imputer\n",
    "# for categorical\n",
    "\n",
    "imputer_numeric = Pipeline(steps=[\n",
    "    ('imputer', SimpleImputer(strategy='mean')),\n",
    "])\n",
    "\n",
    "imputer_categoric = Pipeline(\n",
    "    steps=[('imputer',\n",
    "            SimpleImputer(strategy='constant', fill_value='Missing'))])\n",
    "\n",
    "# then we put the features list and the transformers together\n",
    "# using the column transformer\n",
    "\n",
    "preprocessor = ColumnTransformer(transformers=[('imputer_numeric',\n",
    "                                                imputer_numeric,\n",
    "                                                features_numeric),\n",
    "                                               ('imputer_categoric',\n",
    "                                                imputer_categoric,\n",
    "                                                features_categoric)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ColumnTransformer(n_jobs=None, remainder='drop', sparse_threshold=0.3,\n",
       "         transformer_weights=None,\n",
       "         transformers=[('imputer_numeric', Pipeline(memory=None,\n",
       "     steps=[('imputer', SimpleImputer(copy=True, fill_value=None, missing_values=nan, strategy='mean',\n",
       "       verbose=0))]), ['LotFrontage', 'MasVnrArea', 'GarageYrBlt']), ('imputer_categoric', Pipeline(memory=None,\n",
       "     steps=[('imputer', SimpleImputer(copy=True, fill_value='Missing', missing_values=nan,\n",
       "       strategy='constant', verbose=0))]), ['BsmtQual', 'FireplaceQu'])])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# now we fit the preprocessor\n",
    "preprocessor.fit(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('imputer_numeric', Pipeline(memory=None,\n",
       "       steps=[('imputer', SimpleImputer(copy=True, fill_value=None, missing_values=nan, strategy='mean',\n",
       "         verbose=0))]), ['LotFrontage', 'MasVnrArea', 'GarageYrBlt']),\n",
       " ('imputer_categoric', Pipeline(memory=None,\n",
       "       steps=[('imputer', SimpleImputer(copy=True, fill_value='Missing', missing_values=nan,\n",
       "         strategy='constant', verbose=0))]), ['BsmtQual', 'FireplaceQu'])]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we can explore the transformers like this:\n",
    "preprocessor.transformers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  69.66866747,  103.55358899, 1978.01239669])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# and we can look at the parameters learnt like this:\n",
    "\n",
    "# for the numerical imputer\n",
    "preprocessor.named_transformers_['imputer_numeric'].named_steps['imputer'].statistics_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Missing', 'Missing'], dtype=object)"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# for the categorical imputer\n",
    "preprocessor.named_transformers_['imputer_categoric'].named_steps['imputer'].statistics_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# and now we can impute the data\n",
    "# remember it returs a numpy array\n",
    "\n",
    "X_train = preprocessor.transform(X_train)\n",
    "X_test = preprocessor.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LotFrontage</th>\n",
       "      <th>MasVnrArea</th>\n",
       "      <th>GarageYrBlt</th>\n",
       "      <th>BsmtQual</th>\n",
       "      <th>FireplaceQu</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>69.6687</td>\n",
       "      <td>573</td>\n",
       "      <td>1998</td>\n",
       "      <td>Gd</td>\n",
       "      <td>Missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>69.6687</td>\n",
       "      <td>0</td>\n",
       "      <td>1996</td>\n",
       "      <td>Gd</td>\n",
       "      <td>Gd</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>50</td>\n",
       "      <td>0</td>\n",
       "      <td>1978.01</td>\n",
       "      <td>TA</td>\n",
       "      <td>Missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>60</td>\n",
       "      <td>0</td>\n",
       "      <td>1939</td>\n",
       "      <td>TA</td>\n",
       "      <td>Missing</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>60</td>\n",
       "      <td>0</td>\n",
       "      <td>1930</td>\n",
       "      <td>TA</td>\n",
       "      <td>Missing</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  LotFrontage MasVnrArea GarageYrBlt BsmtQual FireplaceQu\n",
       "0     69.6687        573        1998       Gd     Missing\n",
       "1     69.6687          0        1996       Gd          Gd\n",
       "2          50          0     1978.01       TA     Missing\n",
       "3          60          0        1939       TA     Missing\n",
       "4          60          0        1930       TA     Missing"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# now we convert the result into a dataframe\n",
    "pd.DataFrame(X_train,\n",
    "             columns=features_numeric+features_categoric).head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LotFrontage    0.0\n",
       "MasVnrArea     0.0\n",
       "GarageYrBlt    0.0\n",
       "BsmtQual       0.0\n",
       "FireplaceQu    0.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# now we convert the result into a dataframe\n",
    "# and explore the missing values\n",
    "# there should be none\n",
    "\n",
    "X_train = pd.DataFrame(X_train,\n",
    "             columns=features_numeric+features_categoric)\n",
    "\n",
    "X_train.isnull().mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "feml",
   "language": "python",
   "name": "feml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}