03.2-Cardinality.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cardinality\n",
    "\n",
    "The values of a categorical variable are selected from a group of categories, also called labels. For example, in the variable _gender_ the categories or labels are male and female, whereas in the variable _city_ the labels can be London, Manchester, Brighton and so on.\n",
    "\n",
    "Different categorical variables contain different number of labels or categories. The variable gender contains only 2 labels, but a variable like city or postcode, can contain a huge number of different labels.\n",
    "\n",
    "The number of different labels within a categorical variable is known as cardinality. A high number of labels within a variable is known as __high cardinality__.\n",
    "\n",
    "\n",
    "### Are multiple labels in a categorical variable a problem?\n",
    "\n",
    "High cardinality may pose the following problems: \n",
    "\n",
    "- Variables with too many labels tend to dominate over those with only a few labels, particularly in **Tree based** algorithms.\n",
    "\n",
    "- A big number of labels within a variable may introduce noise with little, if any, information, therefore making machine learning models prone to over-fit.\n",
    "\n",
    "- Some of the labels may only be present in the training data set, but not in the test set, therefore machine learning algorithms may over-fit to the training set.\n",
    "\n",
    "- Contrarily, some labels may appear only in the test set, therefore leaving the machine learning algorithms unable to perform a calculation over the new (unseen) observation.\n",
    "\n",
    "\n",
    "In particular, **tree methods can be biased towards variables with lots of labels** (variables with high cardinality). Thus, their performance may be affected by high cardinality.\n",
    "\n",
    "Below, I will show the effect of high cardinality of variables on the performance of different machine learning algorithms, and how a quick fix to reduce the number of labels, without any sort of data insight, already helps to boost performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## In this Demo:\n",
    "\n",
    "We will:\n",
    "\n",
    "- Learn how to quantify cardinality\n",
    "- See examples of high and low cardinality variables\n",
    "- Understand the effect of cardinality when preparing train and test sets\n",
    "- Visualise the effect of cardinality on Machine Learning Model performance\n",
    "\n",
    "We will use the Titanic dataset.\n",
    "\n",
    "- To download the dataset, please refer to the **Datasets** lecture in **Section 1** of the course."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# to build machine learning models\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.ensemble import AdaBoostClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "\n",
    "# to evaluate the models\n",
    "from sklearn.metrics import roc_auc_score\n",
    "\n",
    "# to separate data into train and test\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>pclass</th>\n",
       "      <th>survived</th>\n",
       "      <th>name</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>sibsp</th>\n",
       "      <th>parch</th>\n",
       "      <th>ticket</th>\n",
       "      <th>fare</th>\n",
       "      <th>cabin</th>\n",
       "      <th>embarked</th>\n",
       "      <th>boat</th>\n",
       "      <th>body</th>\n",
       "      <th>home.dest</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Allen, Miss. Elisabeth Walton</td>\n",
       "      <td>female</td>\n",
       "      <td>29.0000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>24160</td>\n",
       "      <td>211.3375</td>\n",
       "      <td>B5</td>\n",
       "      <td>S</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>St Louis, MO</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Allison, Master. Hudson Trevor</td>\n",
       "      <td>male</td>\n",
       "      <td>0.9167</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>113781</td>\n",
       "      <td>151.5500</td>\n",
       "      <td>C22</td>\n",
       "      <td>S</td>\n",
       "      <td>11</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Montreal, PQ / Chesterville, ON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>Allison, Miss. Helen Loraine</td>\n",
       "      <td>female</td>\n",
       "      <td>2.0000</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>113781</td>\n",
       "      <td>151.5500</td>\n",
       "      <td>C22</td>\n",
       "      <td>S</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Montreal, PQ / Chesterville, ON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>Allison, Mr. Hudson Joshua Creighton</td>\n",
       "      <td>male</td>\n",
       "      <td>30.0000</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>113781</td>\n",
       "      <td>151.5500</td>\n",
       "      <td>C22</td>\n",
       "      <td>S</td>\n",
       "      <td>NaN</td>\n",
       "      <td>135.0</td>\n",
       "      <td>Montreal, PQ / Chesterville, ON</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>Allison, Mrs. Hudson J C (Bessie Waldo Daniels)</td>\n",
       "      <td>female</td>\n",
       "      <td>25.0000</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>113781</td>\n",
       "      <td>151.5500</td>\n",
       "      <td>C22</td>\n",
       "      <td>S</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Montreal, PQ / Chesterville, ON</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0  pclass  survived  \\\n",
       "0           0       1         1   \n",
       "1           1       1         1   \n",
       "2           2       1         0   \n",
       "3           3       1         0   \n",
       "4           4       1         0   \n",
       "\n",
       "                                              name     sex      age  sibsp  \\\n",
       "0                    Allen, Miss. Elisabeth Walton  female  29.0000      0   \n",
       "1                   Allison, Master. Hudson Trevor    male   0.9167      1   \n",
       "2                     Allison, Miss. Helen Loraine  female   2.0000      1   \n",
       "3             Allison, Mr. Hudson Joshua Creighton    male  30.0000      1   \n",
       "4  Allison, Mrs. Hudson J C (Bessie Waldo Daniels)  female  25.0000      1   \n",
       "\n",
       "   parch  ticket      fare cabin embarked boat   body  \\\n",
       "0      0   24160  211.3375    B5        S    2    NaN   \n",
       "1      2  113781  151.5500   C22        S   11    NaN   \n",
       "2      2  113781  151.5500   C22        S  NaN    NaN   \n",
       "3      2  113781  151.5500   C22        S  NaN  135.0   \n",
       "4      2  113781  151.5500   C22        S  NaN    NaN   \n",
       "\n",
       "                         home.dest  \n",
       "0                     St Louis, MO  \n",
       "1  Montreal, PQ / Chesterville, ON  \n",
       "2  Montreal, PQ / Chesterville, ON  \n",
       "3  Montreal, PQ / Chesterville, ON  \n",
       "4  Montreal, PQ / Chesterville, ON  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's load the titanic dataset\n",
    "\n",
    "data = pd.read_csv('../titanic.csv')\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The categorical variables in this dataset are Name, Sex, Ticket, Cabin and Embarked.\n",
    "\n",
    "---------------\n",
    "**Note** that Ticket and Cabin contain both letters and numbers, so they could be treated as Mixed Variables. For this demonstration, I will treat them as categorical."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of categories in the variable Name: 1307\n",
      "Number of categories in the variable Gender: 2\n",
      "Number of categories in the variable Ticket: 929\n",
      "Number of categories in the variable Cabin: 182\n",
      "Number of categories in the variable Embarked: 4\n",
      "Total number of passengers in the Titanic: 1309\n"
     ]
    }
   ],
   "source": [
    "# let's inspect the cardinality, this is the number\n",
    "# of different labels, for the different categorical variables\n",
    "\n",
    "print('Number of categories in the variable Name: {}'.format(\n",
    "    len(data.name.unique())))\n",
    "\n",
    "print('Number of categories in the variable Gender: {}'.format(\n",
    "    len(data.sex.unique())))\n",
    "\n",
    "print('Number of categories in the variable Ticket: {}'.format(\n",
    "    len(data.ticket.unique())))\n",
    "\n",
    "print('Number of categories in the variable Cabin: {}'.format(\n",
    "    len(data.cabin.unique())))\n",
    "\n",
    "print('Number of categories in the variable Embarked: {}'.format(\n",
    "    len(data.embarked.unique())))\n",
    "\n",
    "print('Total number of passengers in the Titanic: {}'.format(len(data)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the variable Sex contains only 2 categories and Embarked 4 (low cardinality), the variables Ticket, Name and Cabin, as expected, contain a huge number of different labels (high cardinality).\n",
    "\n",
    "To demonstrate the effect of high cardinality in train and test sets and machine learning performance, I will work with the variable Cabin. I will create a new variable with reduced cardinality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['B5', 'C22', 'E12', 'D7', 'A36', 'C101', nan, 'C62', 'B35', 'A23',\n",
       "       'B58', 'D15', 'C6', 'D35', 'C148', 'C97', 'B49', 'C99', 'C52', 'T',\n",
       "       'A31', 'C7', 'C103', 'D22', 'E33', 'A21', 'B10', 'B4', 'E40',\n",
       "       'B38', 'E24', 'B51', 'B96', 'C46', 'E31', 'E8', 'B61', 'B77', 'A9',\n",
       "       'C89', 'A14', 'E58', 'E49', 'E52', 'E45', 'B22', 'B26', 'C85',\n",
       "       'E17', 'B71', 'B20', 'A34', 'C86', 'A16', 'A20', 'A18', 'C54',\n",
       "       'C45', 'D20', 'A29', 'C95', 'E25', 'C111', 'C23', 'E36', 'D34',\n",
       "       'D40', 'B39', 'B41', 'B102', 'C123', 'E63', 'C130', 'B86', 'C92',\n",
       "       'A5', 'C51', 'B42', 'C91', 'C125', 'D10', 'B82', 'E50', 'D33',\n",
       "       'C83', 'B94', 'D49', 'D45', 'B69', 'B11', 'E46', 'C39', 'B18',\n",
       "       'D11', 'C93', 'B28', 'C49', 'B52', 'E60', 'C132', 'B37', 'D21',\n",
       "       'D19', 'C124', 'D17', 'B101', 'D28', 'D6', 'D9', 'B80', 'C106',\n",
       "       'B79', 'C47', 'D30', 'C90', 'E38', 'C78', 'C30', 'C118', 'D36',\n",
       "       'D48', 'D47', 'C105', 'B36', 'B30', 'D43', 'B24', 'C2', 'C65',\n",
       "       'B73', 'C104', 'C110', 'C50', 'B3', 'A24', 'A32', 'A11', 'A10',\n",
       "       'B57', 'C28', 'E44', 'A26', 'A6', 'A7', 'C31', 'A19', 'B45', 'E34',\n",
       "       'B78', 'B50', 'C87', 'C116', 'C55', 'D50', 'E68', 'E67', 'C126',\n",
       "       'C68', 'C70', 'C53', 'B19', 'D46', 'D37', 'D26', 'C32', 'C80',\n",
       "       'C82', 'C128', 'E39', 'D', 'F4', 'D56', 'F33', 'E101', 'E77', 'F2',\n",
       "       'D38', 'F', 'E121', 'E10', 'G6', 'F38'], dtype=object)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's explore the values / categories of Cabin\n",
    "\n",
    "# we know from the previous cell that there are 148\n",
    "# different cabins, therefore the variable\n",
    "# is highly cardinal\n",
    "\n",
    "data.cabin.unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now reduce the cardinality of the variable. How? instead of using the entire **cabin** value, I will capture only the \n",
    "first letter.\n",
    "\n",
    "***Rationale***: the first letter indicates the deck on which the cabin was located, and is therefore an indication of both social class status and proximity to the surface of the Titanic. Both are known to improve the probability of survival."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cabin</th>\n",
       "      <th>Cabin_reduced</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>B5</td>\n",
       "      <td>B</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>C22</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>C22</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>C22</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>C22</td>\n",
       "      <td>C</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  cabin Cabin_reduced\n",
       "0    B5             B\n",
       "1   C22             C\n",
       "2   C22             C\n",
       "3   C22             C\n",
       "4   C22             C"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's capture the first letter of Cabin\n",
    "data['Cabin_reduced'] = data['cabin'].astype(str).str[0]\n",
    "\n",
    "data[['cabin', 'Cabin_reduced']].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of categories in the variable Cabin: 182\n",
      "Number of categories in the variable Cabin reduced: 9\n"
     ]
    }
   ],
   "source": [
    "print('Number of categories in the variable Cabin: {}'.format(\n",
    "    len(data.cabin.unique())))\n",
    "\n",
    "print('Number of categories in the variable Cabin reduced: {}'.format(\n",
    "    len(data.Cabin_reduced.unique())))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We reduced the number of different labels from 182 to 9."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((916, 3), (393, 3))"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's separate into training and testing set\n",
    "# in order to build machine learning models\n",
    "\n",
    "use_cols = ['cabin', 'Cabin_reduced', 'sex']\n",
    "\n",
    "# this functions comes from scikit-learn\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    data[use_cols], \n",
    "    data['survived'],  \n",
    "    test_size=0.3,\n",
    "    random_state=0)\n",
    "\n",
    "X_train.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### High cardinality leads to uneven distribution of categories in train and test sets\n",
    "\n",
    "When a variable is highly cardinal, often some categories land only on the training set, or only on the testing set. If present only in the training set, they may lead to over-fitting. If present only on the testing set, the machine learning algorithm will not know how to handle them, as it has not seen them during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "113"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's find out labels present only in the training set\n",
    "\n",
    "unique_to_train_set = [\n",
    "    x for x in X_train.cabin.unique() if x not in X_test.cabin.unique()\n",
    "]\n",
    "\n",
    "len(unique_to_train_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 113 Cabins only present in the training set, and not in the testing set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "36"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's find out labels present only in the test set\n",
    "\n",
    "unique_to_test_set = [\n",
    "    x for x in X_test.cabin.unique() if x not in X_train.cabin.unique()\n",
    "]\n",
    "\n",
    "len(unique_to_test_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Variables with high cardinality tend to have values (i.e., categories) present in the training set, that are not present in the test set, and vice versa. This will bring problems at the time of training (due to over-fitting) and scoring of new data (how should the model deal with unseen categories?).\n",
    "\n",
    "This problem is almost overcome by reducing the cardinality of the variable. See below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's find out labels present only in the training set\n",
    "# for Cabin with reduced cardinality\n",
    "\n",
    "unique_to_train_set = [\n",
    "    x for x in X_train['Cabin_reduced'].unique()\n",
    "    if x not in X_test['Cabin_reduced'].unique()\n",
    "]\n",
    "\n",
    "len(unique_to_train_set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's find out labels present only in the test set\n",
    "# for Cabin with reduced cardinality\n",
    "\n",
    "unique_to_test_set = [\n",
    "    x for x in X_test['Cabin_reduced'].unique()\n",
    "    if x not in X_train['Cabin_reduced'].unique()\n",
    "]\n",
    "\n",
    "len(unique_to_test_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Observe how by reducing the cardinality there is now only 1 label in the training set that is not present in the test set. And no label in the test set that is not contained in the training set as well.\n",
    "\n",
    "### Effect of cardinality on Machine Learning Model Performance\n",
    "\n",
    "In order to evaluate the effect of categorical variables in machine learning models, I will quickly replace the categories by numbers. See below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{nan: 0,\n",
       " 'E36': 1,\n",
       " 'C68': 2,\n",
       " 'E24': 3,\n",
       " 'C22': 4,\n",
       " 'D38': 5,\n",
       " 'B50': 6,\n",
       " 'A24': 7,\n",
       " 'C111': 8,\n",
       " 'F': 9,\n",
       " 'C6': 10,\n",
       " 'C87': 11,\n",
       " 'E8': 12,\n",
       " 'B45': 13,\n",
       " 'C93': 14,\n",
       " 'D28': 15,\n",
       " 'D36': 16,\n",
       " 'C125': 17,\n",
       " 'B35': 18,\n",
       " 'T': 19,\n",
       " 'B73': 20,\n",
       " 'B57': 21,\n",
       " 'A26': 22,\n",
       " 'A18': 23,\n",
       " 'B96': 24,\n",
       " 'G6': 25,\n",
       " 'C78': 26,\n",
       " 'C101': 27,\n",
       " 'D9': 28,\n",
       " 'D33': 29,\n",
       " 'C128': 30,\n",
       " 'E50': 31,\n",
       " 'B26': 32,\n",
       " 'B69': 33,\n",
       " 'E121': 34,\n",
       " 'C123': 35,\n",
       " 'B94': 36,\n",
       " 'A34': 37,\n",
       " 'D': 38,\n",
       " 'C39': 39,\n",
       " 'D43': 40,\n",
       " 'E31': 41,\n",
       " 'B5': 42,\n",
       " 'D17': 43,\n",
       " 'F33': 44,\n",
       " 'E44': 45,\n",
       " 'D7': 46,\n",
       " 'A21': 47,\n",
       " 'D34': 48,\n",
       " 'A29': 49,\n",
       " 'D35': 50,\n",
       " 'A11': 51,\n",
       " 'B51': 52,\n",
       " 'D46': 53,\n",
       " 'E60': 54,\n",
       " 'C30': 55,\n",
       " 'D26': 56,\n",
       " 'E68': 57,\n",
       " 'A9': 58,\n",
       " 'B71': 59,\n",
       " 'D37': 60,\n",
       " 'F2': 61,\n",
       " 'C55': 62,\n",
       " 'C89': 63,\n",
       " 'C124': 64,\n",
       " 'C23': 65,\n",
       " 'C126': 66,\n",
       " 'E49': 67,\n",
       " 'E46': 68,\n",
       " 'D19': 69,\n",
       " 'B58': 70,\n",
       " 'C82': 71,\n",
       " 'B52': 72,\n",
       " 'C92': 73,\n",
       " 'E45': 74,\n",
       " 'C65': 75,\n",
       " 'E25': 76,\n",
       " 'B3': 77,\n",
       " 'D40': 78,\n",
       " 'C91': 79,\n",
       " 'B102': 80,\n",
       " 'B61': 81,\n",
       " 'A20': 82,\n",
       " 'B36': 83,\n",
       " 'C7': 84,\n",
       " 'B77': 85,\n",
       " 'D20': 86,\n",
       " 'C148': 87,\n",
       " 'C105': 88,\n",
       " 'E38': 89,\n",
       " 'B86': 90,\n",
       " 'C132': 91,\n",
       " 'C86': 92,\n",
       " 'A14': 93,\n",
       " 'C54': 94,\n",
       " 'A5': 95,\n",
       " 'B49': 96,\n",
       " 'B28': 97,\n",
       " 'B24': 98,\n",
       " 'C2': 99,\n",
       " 'F4': 100,\n",
       " 'A6': 101,\n",
       " 'C83': 102,\n",
       " 'B42': 103,\n",
       " 'A36': 104,\n",
       " 'C52': 105,\n",
       " 'D56': 106,\n",
       " 'C116': 107,\n",
       " 'B19': 108,\n",
       " 'E77': 109,\n",
       " 'E101': 110,\n",
       " 'B18': 111,\n",
       " 'C95': 112,\n",
       " 'D15': 113,\n",
       " 'E33': 114,\n",
       " 'B30': 115,\n",
       " 'D21': 116,\n",
       " 'E10': 117,\n",
       " 'C130': 118,\n",
       " 'D6': 119,\n",
       " 'C51': 120,\n",
       " 'D30': 121,\n",
       " 'E67': 122,\n",
       " 'C110': 123,\n",
       " 'C103': 124,\n",
       " 'C90': 125,\n",
       " 'C118': 126,\n",
       " 'C97': 127,\n",
       " 'D47': 128,\n",
       " 'E34': 129,\n",
       " 'B4': 130,\n",
       " 'D50': 131,\n",
       " 'C62': 132,\n",
       " 'E17': 133,\n",
       " 'B41': 134,\n",
       " 'C49': 135,\n",
       " 'C85': 136,\n",
       " 'B20': 137,\n",
       " 'C28': 138,\n",
       " 'E63': 139,\n",
       " 'C99': 140,\n",
       " 'D49': 141,\n",
       " 'A10': 142,\n",
       " 'A16': 143,\n",
       " 'B37': 144,\n",
       " 'C80': 145,\n",
       " 'B78': 146}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's re-map Cabin into numbers so we can use it to train ML models\n",
    "\n",
    "# I will replace each cabin by a number\n",
    "# to quickly demonstrate the effect of\n",
    "# labels on machine learning algorithms\n",
    "\n",
    "##############\n",
    "# Note: this is neither the only nor the best\n",
    "# way to encode categorical variables into numbers\n",
    "# there is more on these techniques in the section\n",
    "# \"Encoding categorical variales\"\n",
    "##############\n",
    "\n",
    "cabin_dict = {k: i for i, k in enumerate(X_train.cabin.unique(), 0)}\n",
    "cabin_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Cabin_mapped</th>\n",
       "      <th>cabin</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>501</th>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>588</th>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>402</th>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1193</th>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>686</th>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>971</th>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>117</th>\n",
       "      <td>1</td>\n",
       "      <td>E36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>540</th>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294</th>\n",
       "      <td>2</td>\n",
       "      <td>C68</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>261</th>\n",
       "      <td>3</td>\n",
       "      <td>E24</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Cabin_mapped cabin\n",
       "501              0   NaN\n",
       "588              0   NaN\n",
       "402              0   NaN\n",
       "1193             0   NaN\n",
       "686              0   NaN\n",
       "971              0   NaN\n",
       "117              1   E36\n",
       "540              0   NaN\n",
       "294              2   C68\n",
       "261              3   E24"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# replace the labels in Cabin, using the dic created above\n",
    "X_train.loc[:, 'Cabin_mapped'] = X_train.loc[:, 'cabin'].map(cabin_dict)\n",
    "X_test.loc[:, 'Cabin_mapped'] = X_test.loc[:, 'cabin'].map(cabin_dict)\n",
    "\n",
    "X_train[['Cabin_mapped', 'cabin']].head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see how NaN takes the value 0 in the new variable, E36 takes the value 1, C68 takes the value 2, and so on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Cabin_reduced</th>\n",
       "      <th>cabin</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>501</th>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>588</th>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>402</th>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1193</th>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>686</th>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>971</th>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>117</th>\n",
       "      <td>1</td>\n",
       "      <td>E36</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>540</th>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294</th>\n",
       "      <td>2</td>\n",
       "      <td>C68</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>261</th>\n",
       "      <td>1</td>\n",
       "      <td>E24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>587</th>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>489</th>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>C22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>405</th>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1284</th>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>338</th>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>356</th>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>985</th>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>182</th>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1027</th>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Cabin_reduced cabin\n",
       "501               0   NaN\n",
       "588               0   NaN\n",
       "402               0   NaN\n",
       "1193              0   NaN\n",
       "686               0   NaN\n",
       "971               0   NaN\n",
       "117               1   E36\n",
       "540               0   NaN\n",
       "294               2   C68\n",
       "261               1   E24\n",
       "587               0   NaN\n",
       "489               0   NaN\n",
       "2                 2   C22\n",
       "405               0   NaN\n",
       "1284              0   NaN\n",
       "338               0   NaN\n",
       "356               0   NaN\n",
       "985               0   NaN\n",
       "182               0   NaN\n",
       "1027              0   NaN"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Now I will replace the letters in the reduced cabin variable\n",
    "# with the same procedure\n",
    "\n",
    "# create replace dictionary\n",
    "cabin_dict = {k: i for i, k in enumerate(X_train['Cabin_reduced'].unique(), 0)}\n",
    "\n",
    "# replace labels by numbers with dictionary\n",
    "X_train.loc[:, 'Cabin_reduced'] = X_train.loc[:, 'Cabin_reduced'].map(\n",
    "    cabin_dict)\n",
    "X_test.loc[:, 'Cabin_reduced'] = X_test.loc[:, 'Cabin_reduced'].map(cabin_dict)\n",
    "\n",
    "X_train[['Cabin_reduced', 'cabin']].head(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see now that E36 and E24 take the same number, 1, because we are capturing only the letter. They both start with E."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "501     1\n",
       "588     1\n",
       "402     1\n",
       "1193    0\n",
       "686     1\n",
       "Name: sex, dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# re-map the categorical variable Sex into numbers\n",
    "\n",
    "X_train.loc[:, 'sex'] = X_train.loc[:, 'sex'].map({'male': 0, 'female': 1})\n",
    "X_test.loc[:, 'sex'] = X_test.loc[:, 'sex'].map({'male': 0, 'female': 1})\n",
    "\n",
    "X_train.sex.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Cabin_mapped     0\n",
       "Cabin_reduced    0\n",
       "sex              0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check if there are missing values in these variables\n",
    "\n",
    "X_train[['Cabin_mapped', 'Cabin_reduced', 'sex']].isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Cabin_mapped     41\n",
       "Cabin_reduced     0\n",
       "sex               0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test[['Cabin_mapped', 'Cabin_reduced', 'sex']].isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the test set, there are now 30 missing values for the highly cardinal variable. These were introduced when encoding the categories into numbers. How? Many categories exist only in the test set. Thus, when we created our encoding dictionary using only the train set, we did not generate a number to replace those labels present only in the test set. As a consequence, they were encoded as NaN. We will see in future notebooks how to tackle this problem. For now, I will fill those missing values with -1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(147, 9)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's check the number of different categories in the encoded variables\n",
    "len(X_train.Cabin_mapped.unique()), len(X_train.Cabin_reduced.unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the above we note immediately that from the original 148 cabins in the dataset, only 121 are present in the training set. We also see how we reduced the number of different categories to just 9 in our previous step.\n",
    "\n",
    "Let's go ahead and evaluate the effect of labels in machine learning algorithms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random Forests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Random Forests roc-auc: 0.853790650048556\n",
      "Test set\n",
      "Random Forests roc-auc: 0.7691361097284443\n"
     ]
    }
   ],
   "source": [
    "# model built on data with high cardinality for cabin\n",
    "\n",
    "# call the model\n",
    "rf = RandomForestClassifier(n_estimators=200, random_state=39)\n",
    "\n",
    "# train the model\n",
    "rf.fit(X_train[['Cabin_mapped', 'sex']], y_train)\n",
    "\n",
    "# make predictions on train and test set\n",
    "pred_train = rf.predict_proba(X_train[['Cabin_mapped', 'sex']])\n",
    "pred_test = rf.predict_proba(X_test[['Cabin_mapped', 'sex']].fillna(0))\n",
    "\n",
    "print('Train set')\n",
    "print('Random Forests roc-auc: {}'.format(roc_auc_score(y_train, pred_train[:,1])))\n",
    "print('Test set')\n",
    "print('Random Forests roc-auc: {}'.format(roc_auc_score(y_test, pred_test[:,1])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We observe that the performance of the Random Forests on the training set is quite superior to its performance in the test set. This indicates that the model is over-fitting, which means that it does a great job at predicting the outcome on the dataset it was trained on, but it lacks the power to generalise the prediction to unseen data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Random Forests roc-auc: 0.8163420365403872\n",
      "Test set\n",
      "Random Forests roc-auc: 0.8017670482827277\n"
     ]
    }
   ],
   "source": [
    "# model built on data with low cardinality for cabin\n",
    "\n",
    "# call the model\n",
    "rf = RandomForestClassifier(n_estimators=200, random_state=39)\n",
    "\n",
    "# train the model\n",
    "rf.fit(X_train[['Cabin_reduced', 'sex']], y_train)\n",
    "\n",
    "# make predictions on train and test set\n",
    "pred_train = rf.predict_proba(X_train[['Cabin_reduced', 'sex']])\n",
    "pred_test = rf.predict_proba(X_test[['Cabin_reduced', 'sex']])\n",
    "\n",
    "print('Train set')\n",
    "print('Random Forests roc-auc: {}'.format(roc_auc_score(y_train, pred_train[:,1])))\n",
    "print('Test set')\n",
    "print('Random Forests roc-auc: {}'.format(roc_auc_score(y_test, pred_test[:,1])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see now that the Random Forests no longer over-fit to the training set. In addition, the model is much better at generalising the predictions (compare the roc-auc of this model on the test set vs the roc-auc of the model above also in the test set: 0.81 vs 0.80).\n",
    "\n",
    "**I would like to point out, that likely we can overcome the effect of high cardinality by adjusting the hyper-parameters of the random forests. That goes beyond the scope of this course. Here, I want to show you that given a same model, with identical hyper-parameters, high cardinality may cause the model to over-fit**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### AdaBoost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Adaboost roc-auc: 0.8296861713101102\n",
      "Test set\n",
      "Adaboost roc-auc: 0.7604391350035948\n"
     ]
    }
   ],
   "source": [
    "# model build on data with plenty of categories in Cabin\n",
    "\n",
    "# call the model\n",
    "ada = AdaBoostClassifier(n_estimators=200, random_state=44)\n",
    "\n",
    "# train the model\n",
    "ada.fit(X_train[['Cabin_mapped', 'sex']], y_train)\n",
    "\n",
    "# make predictions on train and test set\n",
    "pred_train = ada.predict_proba(X_train[['Cabin_mapped', 'sex']])\n",
    "pred_test = ada.predict_proba(X_test[['Cabin_mapped', 'sex']].fillna(0))\n",
    "\n",
    "print('Train set')\n",
    "print('Adaboost roc-auc: {}'.format(roc_auc_score(y_train, pred_train[:,1])))\n",
    "print('Test set')\n",
    "print('Adaboost roc-auc: {}'.format(roc_auc_score(y_test, pred_test[:,1])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Adaboost roc-auc: 0.8161256723642566\n",
      "Test set\n",
      "Adaboost roc-auc: 0.8001078480172557\n"
     ]
    }
   ],
   "source": [
    "# model build on data with fewer categories in Cabin Variable\n",
    "\n",
    "# call the model\n",
    "ada = AdaBoostClassifier(n_estimators=200, random_state=44)\n",
    "\n",
    "# train the model\n",
    "ada.fit(X_train[['Cabin_reduced', 'sex']], y_train)\n",
    "\n",
    "# make predictions on train and test set\n",
    "pred_train = ada.predict_proba(X_train[['Cabin_reduced', 'sex']])\n",
    "pred_test = ada.predict_proba(X_test[['Cabin_reduced', 'sex']].fillna(0))\n",
    "\n",
    "print('Train set')\n",
    "print('Adaboost roc-auc: {}'.format(roc_auc_score(y_train, pred_train[:,1])))\n",
    "print('Test set')\n",
    "print('Adaboost roc-auc: {}'.format(roc_auc_score(y_test, pred_test[:,1])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Similarly, the Adaboost model trained on the variable with high cardinality is overfit to the train set. Whereas the Adaboost trained on the low cardinal variable is not overfitting and therefore does a better job in generalising the predictions.\n",
    "\n",
    "In addition, building an AdaBoost on a model with less categories in Cabin, is a) simpler and b) should a different category in the test set appear, by taking just the front letter of cabin, the ML model will know how to handle it because it was seen during training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Logistic regression roc-auc: 0.8133909298124677\n",
      "Test set\n",
      "Logistic regression roc-auc: 0.7750815773463858\n"
     ]
    }
   ],
   "source": [
    "# model build on data with plenty of categories in Cabin variable\n",
    "\n",
    "# call the model\n",
    "logit = LogisticRegression(random_state=44, solver='lbfgs')\n",
    "\n",
    "# train the model\n",
    "logit.fit(X_train[['Cabin_mapped', 'sex']], y_train)\n",
    "\n",
    "# make predictions on train and test set\n",
    "pred_train = logit.predict_proba(X_train[['Cabin_mapped', 'sex']])\n",
    "pred_test = logit.predict_proba(X_test[['Cabin_mapped', 'sex']].fillna(0))\n",
    "\n",
    "print('Train set')\n",
    "print('Logistic regression roc-auc: {}'.format(roc_auc_score(y_train, pred_train[:,1])))\n",
    "print('Test set')\n",
    "print('Logistic regression roc-auc: {}'.format(roc_auc_score(y_test, pred_test[:,1])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Logistic regression roc-auc: 0.8123468468695123\n",
      "Test set\n",
      "Logistic regression roc-auc: 0.8008268347989602\n"
     ]
    }
   ],
   "source": [
    "# model build on data with fewer categories in Cabin Variable\n",
    "\n",
    "# call the model\n",
    "logit = LogisticRegression(random_state=44, solver='lbfgs')\n",
    "\n",
    "# train the model\n",
    "logit.fit(X_train[['Cabin_reduced', 'sex']], y_train)\n",
    "\n",
    "# make predictions on train and test set\n",
    "pred_train = logit.predict_proba(X_train[['Cabin_reduced', 'sex']])\n",
    "pred_test = logit.predict_proba(X_test[['Cabin_reduced', 'sex']].fillna(0))\n",
    "\n",
    "print('Train set')\n",
    "print('Logistic regression roc-auc: {}'.format(roc_auc_score(y_train, pred_train[:,1])))\n",
    "print('Test set')\n",
    "print('Logistic regression roc-auc: {}'.format(roc_auc_score(y_test, pred_test[:,1])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can draw the same conclusion for Logistic Regression: reducing the cardinality improves the performance and generalisation of the algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gradient Boosted Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Gradient Boosted Trees roc-auc: 0.862631390919749\n",
      "Test set\n",
      "Gradient Boosted Trees roc-auc: 0.7733117637298823\n"
     ]
    }
   ],
   "source": [
    "# model build on data with plenty of categories in Cabin variable\n",
    "\n",
    "# call the model\n",
    "gbc = GradientBoostingClassifier(n_estimators=300, random_state=44)\n",
    "\n",
    "# train the model\n",
    "gbc.fit(X_train[['Cabin_mapped', 'sex']], y_train)\n",
    "\n",
    "# make predictions on train and test set\n",
    "pred_train = gbc.predict_proba(X_train[['Cabin_mapped', 'sex']])\n",
    "pred_test = gbc.predict_proba(X_test[['Cabin_mapped', 'sex']].fillna(0))\n",
    "\n",
    "print('Train set')\n",
    "print('Gradient Boosted Trees roc-auc: {}'.format(roc_auc_score(y_train, pred_train[:,1])))\n",
    "print('Test set')\n",
    "print('Gradient Boosted Trees roc-auc: {}'.format(roc_auc_score(y_test, pred_test[:,1])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set\n",
      "Gradient Boosted Trees roc-auc: 0.816719415917359\n",
      "Test set\n",
      "Gradient Boosted Trees roc-auc: 0.8015181682429069\n"
     ]
    }
   ],
   "source": [
    "# model build on data with plenty of categories in Cabin variable\n",
    "\n",
    "# call the model\n",
    "gbc = GradientBoostingClassifier(n_estimators=300, random_state=44)\n",
    "\n",
    "# train the model\n",
    "gbc.fit(X_train[['Cabin_reduced', 'sex']], y_train)\n",
    "\n",
    "# make predictions on train and test set\n",
    "pred_train = gbc.predict_proba(X_train[['Cabin_reduced', 'sex']])\n",
    "pred_test = gbc.predict_proba(X_test[['Cabin_reduced', 'sex']].fillna(0))\n",
    "\n",
    "print('Train set')\n",
    "print('Gradient Boosted Trees roc-auc: {}'.format(roc_auc_score(y_train, pred_train[:,1])))\n",
    "print('Test set')\n",
    "print('Gradient Boosted Trees roc-auc: {}'.format(roc_auc_score(y_test, pred_test[:,1])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gradient Boosted trees are indeed over-fitting to the training set in those cases where the variable Cabin has a lot of labels. This was expected as tree methods tend to be biased to variables with plenty of categories.\n",
    "\n",
    "**That is all for this demonstration. I hope you enjoyed the notebook, and see you in the next one.**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}