06.09-Comparison-categorical-encoding-techniques.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparison of Categorical Variable Encodings\n",
    "\n",
    "In this lecture, we will compare the performance of the different feature categorical encoding techniques we learned so far.\n",
    "\n",
    "We will compare:\n",
    "\n",
    "- One hot encoding\n",
    "- Replacing labels by the count\n",
    "- Ordering labels according to target\n",
    "- Mean Encoding\n",
    "- WoE\n",
    "\n",
    "Using the titanic dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "from sklearn.metrics import roc_auc_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pclass</th>\n",
       "      <th>survived</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>fare</th>\n",
       "      <th>cabin</th>\n",
       "      <th>embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>29.0000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>211.3375</td>\n",
       "      <td>B5</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>male</td>\n",
       "      <td>0.9167</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>151.5500</td>\n",
       "      <td>C22</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>female</td>\n",
       "      <td>2.0000</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>151.5500</td>\n",
       "      <td>C22</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>male</td>\n",
       "      <td>30.0000</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>151.5500</td>\n",
       "      <td>C22</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>female</td>\n",
       "      <td>25.0000</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>151.5500</td>\n",
       "      <td>C22</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   pclass  survived     sex      age  sibsp  parch      fare cabin embarked\n",
       "0       1         1  female  29.0000      0      0  211.3375    B5        S\n",
       "1       1         1    male   0.9167      1      2  151.5500   C22        S\n",
       "2       1         0  female   2.0000      1      2  151.5500   C22        S\n",
       "3       1         0    male  30.0000      1      2  151.5500   C22        S\n",
       "4       1         0  female  25.0000      1      2  151.5500   C22        S"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's load the titanic dataset\n",
    "\n",
    "# we will only use these columns in the demo\n",
    "cols = ['pclass', 'age', 'sibsp', 'parch', 'fare',\n",
    "        'sex', 'cabin', 'embarked', 'survived']\n",
    "\n",
    "data = pd.read_csv('../titanic.csv', usecols=cols)\n",
    "\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pclass         0\n",
       "survived       0\n",
       "sex            0\n",
       "age          263\n",
       "sibsp          0\n",
       "parch          0\n",
       "fare           1\n",
       "cabin       1014\n",
       "embarked       2\n",
       "dtype: int64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's check for missing data\n",
    "\n",
    "data.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop observations with NA in Fare and embarked\n",
    "\n",
    "data.dropna(subset=['fare', 'embarked'], inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pclass</th>\n",
       "      <th>survived</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>fare</th>\n",
       "      <th>cabin</th>\n",
       "      <th>embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>female</td>\n",
       "      <td>29.0000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>211.3375</td>\n",
       "      <td>B</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>male</td>\n",
       "      <td>0.9167</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>151.5500</td>\n",
       "      <td>C</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>female</td>\n",
       "      <td>2.0000</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>151.5500</td>\n",
       "      <td>C</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>male</td>\n",
       "      <td>30.0000</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>151.5500</td>\n",
       "      <td>C</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>female</td>\n",
       "      <td>25.0000</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>151.5500</td>\n",
       "      <td>C</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   pclass  survived     sex      age  sibsp  parch      fare cabin embarked\n",
       "0       1         1  female  29.0000      0      0  211.3375     B        S\n",
       "1       1         1    male   0.9167      1      2  151.5500     C        S\n",
       "2       1         0  female   2.0000      1      2  151.5500     C        S\n",
       "3       1         0    male  30.0000      1      2  151.5500     C        S\n",
       "4       1         0  female  25.0000      1      2  151.5500     C        S"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now we extract the first letter of the cabin\n",
    "\n",
    "data['cabin'] = data['cabin'].astype(str).str[0]\n",
    "\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop observations with cabin = T, they are too few\n",
    "\n",
    "data = data[data['cabin'] != 'T']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((913, 8), (392, 8))"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's divide into train and test set\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    data.drop(labels='survived', axis=1),  # predictors\n",
    "    data['survived'],  # target\n",
    "    test_size=0.3,\n",
    "    random_state=0)\n",
    "\n",
    "X_train.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's replace null values in numerical variables by the mean\n",
    "\n",
    "\n",
    "def impute_na(df, variable, value):\n",
    "    df[variable].fillna(value, inplace=True)\n",
    "\n",
    "\n",
    "impute_na(X_test, 'age', X_train['age'].mean())\n",
    "impute_na(X_train, 'age',  X_train['age'].mean())\n",
    "# note how I impute first the test set, this way the value of\n",
    "# the median used will be the same for both train and test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pclass</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>fare</th>\n",
       "      <th>cabin</th>\n",
       "      <th>embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>402</th>\n",
       "      <td>2</td>\n",
       "      <td>female</td>\n",
       "      <td>30.00000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>13.8583</td>\n",
       "      <td>n</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>698</th>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>18.00000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.6625</td>\n",
       "      <td>n</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1291</th>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>29.79847</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.7125</td>\n",
       "      <td>n</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1229</th>\n",
       "      <td>3</td>\n",
       "      <td>male</td>\n",
       "      <td>27.00000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.6625</td>\n",
       "      <td>n</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>118</th>\n",
       "      <td>1</td>\n",
       "      <td>male</td>\n",
       "      <td>29.79847</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>26.5500</td>\n",
       "      <td>D</td>\n",
       "      <td>S</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      pclass     sex       age  sibsp  parch     fare cabin embarked\n",
       "402        2  female  30.00000      1      0  13.8583     n        C\n",
       "698        3    male  18.00000      0      0   8.6625     n        S\n",
       "1291       3    male  29.79847      0      0   8.7125     n        S\n",
       "1229       3    male  27.00000      0      0   8.6625     n        S\n",
       "118        1    male  29.79847      0      0  26.5500     D        S"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(pclass      0\n",
       " sex         0\n",
       " age         0\n",
       " sibsp       0\n",
       " parch       0\n",
       " fare        0\n",
       " cabin       0\n",
       " embarked    0\n",
       " dtype: int64, pclass      0\n",
       " sex         0\n",
       " age         0\n",
       " sibsp       0\n",
       " parch       0\n",
       " fare        0\n",
       " cabin       0\n",
       " embarked    0\n",
       " dtype: int64)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's check that we have no missing data after NA imputation\n",
    "\n",
    "X_train.isnull().sum(), X_test.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### One Hot Encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pclass</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>fare</th>\n",
       "      <th>sex_male</th>\n",
       "      <th>cabin_B</th>\n",
       "      <th>cabin_C</th>\n",
       "      <th>cabin_D</th>\n",
       "      <th>cabin_E</th>\n",
       "      <th>cabin_F</th>\n",
       "      <th>cabin_G</th>\n",
       "      <th>cabin_n</th>\n",
       "      <th>embarked_Q</th>\n",
       "      <th>embarked_S</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>402</th>\n",
       "      <td>2</td>\n",
       "      <td>30.00000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>13.8583</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>698</th>\n",
       "      <td>3</td>\n",
       "      <td>18.00000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.6625</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1291</th>\n",
       "      <td>3</td>\n",
       "      <td>29.79847</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.7125</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1229</th>\n",
       "      <td>3</td>\n",
       "      <td>27.00000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.6625</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>118</th>\n",
       "      <td>1</td>\n",
       "      <td>29.79847</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>26.5500</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      pclass       age  sibsp  parch     fare  sex_male  cabin_B  cabin_C  \\\n",
       "402        2  30.00000      1      0  13.8583         0        0        0   \n",
       "698        3  18.00000      0      0   8.6625         1        0        0   \n",
       "1291       3  29.79847      0      0   8.7125         1        0        0   \n",
       "1229       3  27.00000      0      0   8.6625         1        0        0   \n",
       "118        1  29.79847      0      0  26.5500         1        0        0   \n",
       "\n",
       "      cabin_D  cabin_E  cabin_F  cabin_G  cabin_n  embarked_Q  embarked_S  \n",
       "402         0        0        0        0        1           0           0  \n",
       "698         0        0        0        0        1           0           1  \n",
       "1291        0        0        0        0        1           0           1  \n",
       "1229        0        0        0        0        1           0           1  \n",
       "118         1        0        0        0        0           0           1  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def get_OHE(df):\n",
    "\n",
    "    df_OHE = pd.concat(\n",
    "        [df[['pclass', 'age', 'sibsp', 'parch', 'fare']],\n",
    "         pd.get_dummies(df[['sex', 'cabin', 'embarked']], drop_first=True)],\n",
    "        axis=1)\n",
    "\n",
    "    return df_OHE\n",
    "\n",
    "\n",
    "X_train_OHE = get_OHE(X_train)\n",
    "X_test_OHE = get_OHE(X_test)\n",
    "\n",
    "X_train_OHE.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pclass</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>fare</th>\n",
       "      <th>sex_male</th>\n",
       "      <th>cabin_B</th>\n",
       "      <th>cabin_C</th>\n",
       "      <th>cabin_D</th>\n",
       "      <th>cabin_E</th>\n",
       "      <th>cabin_F</th>\n",
       "      <th>cabin_G</th>\n",
       "      <th>cabin_n</th>\n",
       "      <th>embarked_Q</th>\n",
       "      <th>embarked_S</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>586</th>\n",
       "      <td>2</td>\n",
       "      <td>29.00000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>26.0000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200</th>\n",
       "      <td>1</td>\n",
       "      <td>46.00000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>75.2417</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>831</th>\n",
       "      <td>3</td>\n",
       "      <td>40.00000</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>46.9000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1149</th>\n",
       "      <td>3</td>\n",
       "      <td>29.79847</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7.7208</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>393</th>\n",
       "      <td>2</td>\n",
       "      <td>25.00000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>31.5000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      pclass       age  sibsp  parch     fare  sex_male  cabin_B  cabin_C  \\\n",
       "586        2  29.00000      1      0  26.0000         0        0        0   \n",
       "200        1  46.00000      0      0  75.2417         1        0        1   \n",
       "831        3  40.00000      1      6  46.9000         1        0        0   \n",
       "1149       3  29.79847      0      0   7.7208         0        0        0   \n",
       "393        2  25.00000      0      0  31.5000         1        0        0   \n",
       "\n",
       "      cabin_D  cabin_E  cabin_F  cabin_G  cabin_n  embarked_Q  embarked_S  \n",
       "586         0        0        0        0        1           0           1  \n",
       "200         0        0        0        0        0           0           0  \n",
       "831         0        0        0        0        1           0           1  \n",
       "1149        0        0        0        0        1           1           0  \n",
       "393         0        0        0        0        1           0           1  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test_OHE.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Count encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pclass</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>fare</th>\n",
       "      <th>cabin</th>\n",
       "      <th>embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>402</th>\n",
       "      <td>2</td>\n",
       "      <td>326</td>\n",
       "      <td>30.00000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>13.8583</td>\n",
       "      <td>702</td>\n",
       "      <td>184</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>698</th>\n",
       "      <td>3</td>\n",
       "      <td>587</td>\n",
       "      <td>18.00000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.6625</td>\n",
       "      <td>702</td>\n",
       "      <td>647</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1291</th>\n",
       "      <td>3</td>\n",
       "      <td>587</td>\n",
       "      <td>29.79847</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.7125</td>\n",
       "      <td>702</td>\n",
       "      <td>647</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1229</th>\n",
       "      <td>3</td>\n",
       "      <td>587</td>\n",
       "      <td>27.00000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.6625</td>\n",
       "      <td>702</td>\n",
       "      <td>647</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>118</th>\n",
       "      <td>1</td>\n",
       "      <td>587</td>\n",
       "      <td>29.79847</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>26.5500</td>\n",
       "      <td>33</td>\n",
       "      <td>647</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      pclass  sex       age  sibsp  parch     fare  cabin  embarked\n",
       "402        2  326  30.00000      1      0  13.8583    702       184\n",
       "698        3  587  18.00000      0      0   8.6625    702       647\n",
       "1291       3  587  29.79847      0      0   8.7125    702       647\n",
       "1229       3  587  27.00000      0      0   8.6625    702       647\n",
       "118        1  587  29.79847      0      0  26.5500     33       647"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def categorical_to_counts(df_train, df_test):\n",
    "\n",
    "    # make a temporary copy of the original dataframes\n",
    "    df_train_temp = df_train.copy()\n",
    "    df_test_temp = df_test.copy()\n",
    "\n",
    "    for col in ['sex', 'cabin', 'embarked']:\n",
    "\n",
    "        # make dictionary mapping category to counts\n",
    "        counts_map = df_train_temp[col].value_counts().to_dict()\n",
    "\n",
    "        # remap the labels to their counts\n",
    "        df_train_temp[col] = df_train_temp[col].map(counts_map)\n",
    "        df_test_temp[col] = df_test_temp[col].map(counts_map)\n",
    "\n",
    "    return df_train_temp, df_test_temp\n",
    "\n",
    "\n",
    "X_train_count, X_test_count = categorical_to_counts(X_train, X_test)\n",
    "\n",
    "X_train_count.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ordered Integer Encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pclass</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>fare</th>\n",
       "      <th>cabin</th>\n",
       "      <th>embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>402</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>30.00000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>13.8583</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>698</th>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>18.00000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.6625</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1291</th>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>29.79847</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.7125</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1229</th>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>27.00000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.6625</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>118</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>29.79847</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>26.5500</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      pclass  sex       age  sibsp  parch     fare  cabin  embarked\n",
       "402        2    1  30.00000      1      0  13.8583      0         2\n",
       "698        3    0  18.00000      0      0   8.6625      0         0\n",
       "1291       3    0  29.79847      0      0   8.7125      0         0\n",
       "1229       3    0  27.00000      0      0   8.6625      0         0\n",
       "118        1    0  29.79847      0      0  26.5500      5         0"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def categories_to_ordered(df_train, df_test, y_train, y_test):\n",
    "\n",
    "    # make a temporary copy of the datasets\n",
    "    df_train_temp = pd.concat([df_train, y_train], axis=1).copy()\n",
    "    df_test_temp = pd.concat([df_test, y_test], axis=1).copy()\n",
    "\n",
    "    for col in ['sex', 'cabin', 'embarked']:\n",
    "\n",
    "        # order categories according to target mean\n",
    "        ordered_labels = df_train_temp.groupby(\n",
    "            [col])['survived'].mean().sort_values().index\n",
    "\n",
    "        # create the dictionary to map the ordered labels to an ordinal number\n",
    "        ordinal_label = {k: i for i, k in enumerate(ordered_labels, 0)}\n",
    "\n",
    "        # remap the categories  to these ordinal numbers\n",
    "        df_train_temp[col] = df_train[col].map(ordinal_label)\n",
    "        df_test_temp[col] = df_test[col].map(ordinal_label)\n",
    "\n",
    "    # remove the target\n",
    "    df_train_temp.drop(['survived'], axis=1, inplace=True)\n",
    "    df_test_temp.drop(['survived'], axis=1, inplace=True)\n",
    "\n",
    "    return df_train_temp, df_test_temp\n",
    "\n",
    "\n",
    "X_train_ordered, X_test_ordered = categories_to_ordered(\n",
    "    X_train, X_test, y_train, y_test)\n",
    "\n",
    "X_train_ordered.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mean Encoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pclass</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>fare</th>\n",
       "      <th>cabin</th>\n",
       "      <th>embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>402</th>\n",
       "      <td>2</td>\n",
       "      <td>0.730061</td>\n",
       "      <td>30.00000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>13.8583</td>\n",
       "      <td>0.292023</td>\n",
       "      <td>0.516304</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>698</th>\n",
       "      <td>3</td>\n",
       "      <td>0.173765</td>\n",
       "      <td>18.00000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.6625</td>\n",
       "      <td>0.292023</td>\n",
       "      <td>0.332303</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1291</th>\n",
       "      <td>3</td>\n",
       "      <td>0.173765</td>\n",
       "      <td>29.79847</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.7125</td>\n",
       "      <td>0.292023</td>\n",
       "      <td>0.332303</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1229</th>\n",
       "      <td>3</td>\n",
       "      <td>0.173765</td>\n",
       "      <td>27.00000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.6625</td>\n",
       "      <td>0.292023</td>\n",
       "      <td>0.332303</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>118</th>\n",
       "      <td>1</td>\n",
       "      <td>0.173765</td>\n",
       "      <td>29.79847</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>26.5500</td>\n",
       "      <td>0.696970</td>\n",
       "      <td>0.332303</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      pclass       sex       age  sibsp  parch     fare     cabin  embarked\n",
       "402        2  0.730061  30.00000      1      0  13.8583  0.292023  0.516304\n",
       "698        3  0.173765  18.00000      0      0   8.6625  0.292023  0.332303\n",
       "1291       3  0.173765  29.79847      0      0   8.7125  0.292023  0.332303\n",
       "1229       3  0.173765  27.00000      0      0   8.6625  0.292023  0.332303\n",
       "118        1  0.173765  29.79847      0      0  26.5500  0.696970  0.332303"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def categories_to_mean(df_train, df_test, y_train, y_test):\n",
    "\n",
    "    # make a temporary copy of the datasets\n",
    "    df_train_temp = pd.concat([df_train, y_train], axis=1).copy()\n",
    "    df_test_temp = pd.concat([df_test, y_test], axis=1).copy()\n",
    "\n",
    "    for col in ['sex', 'cabin', 'embarked']:\n",
    "\n",
    "        # calculate mean target per category\n",
    "        ordered_labels = df_train_temp.groupby(\n",
    "            [col])['survived'].mean().to_dict()\n",
    "\n",
    "        # remap the categories to target mean\n",
    "        df_train_temp[col] = df_train[col].map(ordered_labels)\n",
    "        df_test_temp[col] = df_test[col].map(ordered_labels)\n",
    "\n",
    "    # remove the target\n",
    "    df_train_temp.drop(['survived'], axis=1, inplace=True)\n",
    "    df_test_temp.drop(['survived'], axis=1, inplace=True)\n",
    "\n",
    "    return df_train_temp, df_test_temp\n",
    "\n",
    "\n",
    "X_train_mean, X_test_mean = categories_to_mean(\n",
    "    X_train, X_test, y_train, y_test)\n",
    "\n",
    "X_train_mean.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### WoE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pclass</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>fare</th>\n",
       "      <th>cabin</th>\n",
       "      <th>embarked</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>402</th>\n",
       "      <td>2</td>\n",
       "      <td>0.994934</td>\n",
       "      <td>30.00000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>13.8583</td>\n",
       "      <td>-0.885580</td>\n",
       "      <td>0.065241</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>698</th>\n",
       "      <td>3</td>\n",
       "      <td>-1.559176</td>\n",
       "      <td>18.00000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.6625</td>\n",
       "      <td>-0.885580</td>\n",
       "      <td>-0.697788</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1291</th>\n",
       "      <td>3</td>\n",
       "      <td>-1.559176</td>\n",
       "      <td>29.79847</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.7125</td>\n",
       "      <td>-0.885580</td>\n",
       "      <td>-0.697788</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1229</th>\n",
       "      <td>3</td>\n",
       "      <td>-1.559176</td>\n",
       "      <td>27.00000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8.6625</td>\n",
       "      <td>-0.885580</td>\n",
       "      <td>-0.697788</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>118</th>\n",
       "      <td>1</td>\n",
       "      <td>-1.559176</td>\n",
       "      <td>29.79847</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>26.5500</td>\n",
       "      <td>0.832909</td>\n",
       "      <td>-0.697788</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      pclass       sex       age  sibsp  parch     fare     cabin  embarked\n",
       "402        2  0.994934  30.00000      1      0  13.8583 -0.885580  0.065241\n",
       "698        3 -1.559176  18.00000      0      0   8.6625 -0.885580 -0.697788\n",
       "1291       3 -1.559176  29.79847      0      0   8.7125 -0.885580 -0.697788\n",
       "1229       3 -1.559176  27.00000      0      0   8.6625 -0.885580 -0.697788\n",
       "118        1 -1.559176  29.79847      0      0  26.5500  0.832909 -0.697788"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def categories_to_woe(df_train, df_test, y_train, y_test):\n",
    "\n",
    "    # make a temporary copy of the datasets\n",
    "    df_train_temp = pd.concat([df_train, y_train], axis=1).copy()\n",
    "    df_test_temp = pd.concat([df_test, y_test], axis=1).copy()\n",
    "\n",
    "    for col in ['sex', 'cabin', 'embarked']:\n",
    "\n",
    "        # create df containing the different parts of the WoE equation\n",
    "        # prob survived =1\n",
    "        prob_df = pd.DataFrame(df_train_temp.groupby([col])['survived'].mean())\n",
    "\n",
    "        # prob survived = 0\n",
    "        prob_df['died'] = 1-prob_df.survived\n",
    "\n",
    "        # calculate WoE\n",
    "        prob_df['WoE'] = np.log(prob_df.survived/prob_df.died)\n",
    "\n",
    "        # capture woe in dictionary\n",
    "        woe = prob_df['WoE'].to_dict()\n",
    "\n",
    "        # re-map the labels to WoE\n",
    "        df_train_temp[col] = df_train[col].map(woe)\n",
    "        df_test_temp[col] = df_test[col].map(woe)\n",
    "\n",
    "    # drop the target\n",
    "    df_train_temp.drop(['survived'], axis=1, inplace=True)\n",
    "    df_test_temp.drop(['survived'], axis=1, inplace=True)\n",
    "\n",
    "    return df_train_temp, df_test_temp\n",
    "\n",
    "\n",
    "X_train_woe, X_test_woe = categories_to_woe(X_train, X_test, y_train, y_test)\n",
    "\n",
    "X_train_woe.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random Forest Performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a function to build random forests and compare performance in train and test set\n",
    "\n",
    "\n",
    "def run_randomForests(X_train, X_test, y_train, y_test):\n",
    "\n",
    "    rf = RandomForestClassifier(n_estimators=50, random_state=39, max_depth=3)\n",
    "    rf.fit(X_train, y_train)\n",
    "\n",
    "    print('Train set')\n",
    "    pred = rf.predict_proba(X_train)\n",
    "    print(\n",
    "        'Random Forests roc-auc: {}'.format(roc_auc_score(y_train, pred[:, 1])))\n",
    "\n",
    "    print('Test set')\n",
    "    pred = rf.predict_proba(X_test)\n",
    "    print(\n",
    "        'Random Forests roc-auc: {}'.format(roc_auc_score(y_test, pred[:, 1])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Random Forests roc-auc: 0.8488938507340109\n",
      "Test set\n",
      "Random Forests roc-auc: 0.8072730715135779\n"
     ]
    }
   ],
   "source": [
    "# OHE\n",
    "run_randomForests(X_train_OHE, X_test_OHE, y_train, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Random Forests roc-auc: 0.8654552920644698\n",
      "Test set\n",
      "Random Forests roc-auc: 0.8194309206967434\n"
     ]
    }
   ],
   "source": [
    "# counts\n",
    "run_randomForests(X_train_count, X_test_count, y_train, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Random Forests roc-auc: 0.8669027820552304\n",
      "Test set\n",
      "Random Forests roc-auc: 0.8219733852645245\n"
     ]
    }
   ],
   "source": [
    "# ordered labels\n",
    "run_randomForests(X_train_ordered, X_test_ordered, y_train, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Random Forests roc-auc: 0.867010573863053\n",
      "Test set\n",
      "Random Forests roc-auc: 0.8207562479714378\n"
     ]
    }
   ],
   "source": [
    "# mean encoding\n",
    "run_randomForests(X_train_mean, X_test_mean, y_train, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Random Forests roc-auc: 0.867010573863053\n",
      "Test set\n",
      "Random Forests roc-auc: 0.8207562479714378\n"
     ]
    }
   ],
   "source": [
    "# woe\n",
    "run_randomForests(X_train_woe, X_test_woe, y_train, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Comparing the roc_auc values on the test sets, we can see that one hot encoding has the worse performance. This makes sense because trees do not perform well in datasets with big feature spaces.\n",
    "\n",
    "The remaining encodings returned similar performances. This also makes sense, because trees are non-linear models, so target guided encodings may not necessarily improve the model performance\n",
    "\n",
    "### Logistic Regression Performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def run_logistic(X_train, X_test, y_train, y_test):\n",
    "\n",
    "    # function to train and test the performance of logistic regression\n",
    "    logit = LogisticRegression(random_state=44, C=0.01)\n",
    "    logit.fit(X_train, y_train)\n",
    "\n",
    "    print('Train set')\n",
    "    pred = logit.predict_proba(X_train)\n",
    "    print(\n",
    "        'Logistic Regression roc-auc: {}'.format(roc_auc_score(y_train, pred[:, 1])))\n",
    "\n",
    "    print('Test set')\n",
    "    pred = logit.predict_proba(X_test)\n",
    "    print(\n",
    "        'Logistic Regression roc-auc: {}'.format(roc_auc_score(y_test, pred[:, 1])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Logistic Regression roc-auc: 0.8253644389693051\n",
      "Test set\n",
      "Logistic Regression roc-auc: 0.7973601644487721\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Sole\\Anaconda3\\envs\\feml\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n"
     ]
    }
   ],
   "source": [
    "# OHE\n",
    "run_logistic(X_train_OHE, X_test_OHE, y_train, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Logistic Regression roc-auc: 0.7814777743558157\n",
      "Test set\n",
      "Logistic Regression roc-auc: 0.7226279346532511\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Sole\\Anaconda3\\envs\\feml\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n"
     ]
    }
   ],
   "source": [
    "# counts\n",
    "run_logistic(X_train_count, X_test_count, y_train, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Logistic Regression roc-auc: 0.8233831228826608\n",
      "Test set\n",
      "Logistic Regression roc-auc: 0.804581845721086\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Sole\\Anaconda3\\envs\\feml\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n"
     ]
    }
   ],
   "source": [
    "# ordered labels\n",
    "run_logistic(X_train_ordered, X_test_ordered, y_train, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Logistic Regression roc-auc: 0.7913792218458064\n",
      "Test set\n",
      "Logistic Regression roc-auc: 0.762712322838905\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Sole\\Anaconda3\\envs\\feml\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n"
     ]
    }
   ],
   "source": [
    "# mean encoding\n",
    "run_logistic(X_train_mean, X_test_mean, y_train, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Logistic Regression roc-auc: 0.8450082127091674\n",
      "Test set\n",
      "Logistic Regression roc-auc: 0.8166450286703452\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Sole\\Anaconda3\\envs\\feml\\lib\\site-packages\\sklearn\\linear_model\\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n"
     ]
    }
   ],
   "source": [
    "# woe\n",
    "run_logistic(X_train_woe, X_test_woe, y_train, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For Logistic regression, the best performances are obtained with one hot encoding, as it preserves linear relationships with variables and target, and also with weight of evidence, and ordered encoding.\n",
    "\n",
    "Note however how count encoding, returns the worse performance as it does not create a monotonic relationship between variables and target, and in this case, mean target encoding is probably causing over-fitting."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}