08.02-Equal-frequency-discretisation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discretisation\n",
    "\n",
    "Discretisation is the process of transforming continuous variables into discrete variables by creating a set of contiguous intervals that span the range of the variable's values. Discretisation is also called **binning**, where bin is an alternative name for interval.\n",
    "\n",
    "\n",
    "### Discretisation helps handle outliers and may improve value spread in skewed variables\n",
    "\n",
    "Discretisation helps handle outliers by placing these values into the lower or higher intervals, together with the remaining inlier values of the distribution. Thus, these outlier observations no longer differ from the rest of the values at the tails of the distribution, as they are now all together in the same interval / bucket. In addition, by creating appropriate bins or intervals, discretisation can help spread the values of a skewed variable across a set of bins with equal number of observations.\n",
    "\n",
    "\n",
    "### Discretisation approaches\n",
    "\n",
    "There are several approaches to transform continuous variables into discrete ones. Discretisation methods fall into 2 categories: **supervised and unsupervised**. Unsupervised methods do not use any information, other than the variable distribution, to create the contiguous bins in which the values will be placed. Supervised methods typically use target information in order to create the bins or intervals.\n",
    "\n",
    "\n",
    "####  Unsupervised discretisation methods\n",
    "\n",
    "- Equal width discretisation\n",
    "- Equal frequency discretisation\n",
    "- K-means discretisation\n",
    "\n",
    "#### Supervised discretisation methods\n",
    "\n",
    "- Discretisation using decision trees\n",
    "\n",
    "\n",
    "In this lecture, I will describe **equal frequency discretisation**.\n",
    "\n",
    "\n",
    "## Equal frequency discretisation\n",
    "\n",
    "Equal frequency discretisation divides the scope of possible values of the variable into N bins, where each bin carries the same amount of observations. This is particularly useful for skewed variables as it spreads the observations over the different bins equally. We find the interval boundaries by determining the quantiles.\n",
    "\n",
    "Equal frequency discretisation using quantiles consists of dividing the continuous variable into N quantiles, N to be defined by the user.\n",
    "\n",
    "Equal frequency binning is straightforward to implement and by spreading the values of the observations more evenly it may help boost the algorithm's performance. This arbitrary binning may also disrupt the relationship with the target.\n",
    "\n",
    "## In this demo\n",
    "\n",
    "We will learn how to perform equal frequency discretisation using the Titanic dataset with\n",
    "\n",
    "- pandas and NumPy\n",
    "- Feature-engine\n",
    "- Scikit-learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Titanic dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "from sklearn.preprocessing import KBinsDiscretizer\n",
    "\n",
    "from feature_engine.discretisers import EqualFrequencyDiscretiser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>survived</th>\n",
       "      <th>age</th>\n",
       "      <th>fare</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>29.0000</td>\n",
       "      <td>211.3375</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0.9167</td>\n",
       "      <td>151.5500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>2.0000</td>\n",
       "      <td>151.5500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>30.0000</td>\n",
       "      <td>151.5500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>25.0000</td>\n",
       "      <td>151.5500</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   survived      age      fare\n",
       "0         1  29.0000  211.3375\n",
       "1         1   0.9167  151.5500\n",
       "2         0   2.0000  151.5500\n",
       "3         0  30.0000  151.5500\n",
       "4         0  25.0000  151.5500"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load the numerical variables of the Titanic Dataset\n",
    "\n",
    "data = pd.read_csv('../titanic.csv',\n",
    "                   usecols=['age', 'fare', 'survived'])\n",
    "\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((916, 2), (393, 2))"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's separate into train and test set\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    data[['age', 'fare']],\n",
    "    data['survived'],\n",
    "    test_size=0.3,\n",
    "    random_state=0)\n",
    "\n",
    "X_train.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The variables Age and Fare contain missing data, that I will fill by extracting a random sample of the variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def impute_na(data, variable):\n",
    "    # function to fill NA with a random sample\n",
    "\n",
    "    df = data.copy()\n",
    "\n",
    "    # random sampling\n",
    "    df[variable+'_random'] = df[variable]\n",
    "\n",
    "    # extract the random sample to fill the na\n",
    "    random_sample = X_train[variable].dropna().sample(\n",
    "        df[variable].isnull().sum(), random_state=0)\n",
    "\n",
    "    # pandas needs to have the same index in order to merge datasets\n",
    "    random_sample.index = df[df[variable].isnull()].index\n",
    "    df.loc[df[variable].isnull(), variable+'_random'] = random_sample\n",
    "\n",
    "    return df[variable+'_random']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# replace NA in both train and test sets\n",
    "\n",
    "X_train['age'] = impute_na(data, 'age')\n",
    "X_test['age'] = impute_na(data, 'age')\n",
    "\n",
    "X_train['fare'] = impute_na(data, 'fare')\n",
    "X_test['fare'] = impute_na(data, 'fare')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAeoAAAEICAYAAACQ4bezAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAGrFJREFUeJzt3X+wZGV95/H3R0YF8ceA6N3ZGcrRhfJHOSvqFOKym70BTQCNuFWQ1WUFLHbnj9UEN7OrY7YqZrdSW1i1BH8kZTIRI0YiKmqggDJSyF3L3UgUQQZBl1GnnJHR0QjokMRk9Lt/9LlwZ6bvzP3Rffvp2+9XVdft85ynu7+3T5/z7ec5z3k6VYUkSWrTE0YdgCRJmp+JWpKkhpmoJUlqmIlakqSGmaglSWqYiVqSpIaZqCVplUjy/CR3Jflpkt8cdTwajDWjDkCSNDBvB2aq6qWjDkSDY4taklaP5wBfX+yDkthoa5iJegIk2ZbkW1132H1J/k1XfkySK5P8KMl3krw1Sc3utEmekeTqJHuTfC/J7yU5ZrT/jaR+knwe+GXgD5LsT3J51w3+kyS7k/zunLobu339siTfBT7flZ+R5P8meTjJ15JMj+Sf0UH8FjUZvgX8K+D7wIXAR5OcApwPnAucBjwKfPKQx10D/AA4BTgeuAnYDfzxyoQtaaGq6qwkM8BHq+qDXZK9mF4L+8XArUnurqq/mPOwfw28EPhFkvXAzcCbgM8CZwOfSvKCqvrhCv4rOoQt6glQVZ+sqger6hdV9XHgAeB04NeB91bVnqp6CLhi9jFJpugl8bdV1aNVtQ+4CnjDCP4FSYtUVTNVtaPb7+8BPkYvMc/1u93+/XfAvwduqapbusfcCnwFOG+FQ9chbFFPgCQXA78FbOyKngqcBPxTei3kWXPvPwd4IrA3yWzZEw6pI6lRSV5B78v3i4EnAU/m8F6zQ/f5C5P82pyyJwK3DzNOHZ2JepVL8hzgT+h1Y/1VVf08yd1AgL3AhjnVT55zfzfwM+CkqjqwUvFKGpg/B/4AOLeq/j7Je+h9QZ9r7s8n7gb+rKr+40oFqIWx63v1O57ezvhDgCRvpvcNG+ATwOVJ1idZC7xj9kFVtRf4HHBlkqcneUKSf5bk0K4zSW16GvDjLkmfDvy7o9T/KPBrSX61G2h6bJLpJBuO8jgNmYl6lauq+4Argb+iNzBsE/B/utV/Qi8Z3wPcBdwCHAB+3q2/mF6X2X3AQ8D1wLqVil3Ssvwn4H8k+SnwO/S+mM+rqnbTG2D62/S+2O8G/ivmiZFLVR29liZCknOBP6qq54w6FklSj9+UJliS45Kcl2RNd2nGu4DPjDouSdLjbFFPsCRPAf438ALg7+hdQ3l5Vf1kpIFJkh5jopYkqWF2fUuS1LAmrqM+6aSTauPGjfOuf/TRRzn++ONXLqABM/7RGpf477zzzh9V1bNGHcewHW1/h7a2WSuxtBIHtBPLOMexqP29qkZ+e/nLX15Hcvvttx9xfeuMf7TGJX7gK9XA/jjs29H296q2tlkrsbQSR1U7sYxzHIvZ3+36liSpYSZqSZIaZqKWJKlhR03UST6UZF+Se+eUnZjk1iQPdH9P6MqT5H1Jdia5J8nLhhm8JEmr3UJa1B8GzjmkbBtwW1WdCtzWLUPv94tP7W5bgA8MJkxJkibTURN1VX0B+PEhxecD13T3rwFeP6f8I92gti8Ba5P4Iw6SJC3RUs9RT1XvZxBnfw7x2V35eg7+IfI9XZkkSVqCQU94kj5lfecoTbKFXvc4U1NTzMzMzPuk+/fvP+L61hn/aI17/JIm21IT9Q+SrKuqvV3X9r6ufA9w8px6G4AH+z1BVW0HtgNs3ry5pqen532xmZkZjrS+dcY/WuMev6TJttREfSNwCXBF9/eGOeVvTXId8Argkdku8km243uPcOm2mw8r33XFa0YQjdQ+9xnpcUdN1Ek+BkwDJyXZQ+83i68APpHkMuC7wIVd9VuA84CdwN8Cbx5CzJIkTYyjJuqqeuM8q87uU7eAtyw3KEmS1OPMZJIOkmRXkh1J7k7yla7MSY6kETFRS+rnl6vqtKra3C07yZE0IiZqSQvhJEfSiAz6OmpJ46+AzyUp4I+7SykPmuQoydEmOTroao/FzJsAMHUcbN104LDyUVwP38p1+K3EAe3EMilxmKglHerMqnqwS8a3JvnGEeouaJKjxcybAPD+a2/gyh2HH552XXTkxw1DK9fhtxIHtBPLpMRh17ekg1TVg93ffcBngNPpJjkCWOokR5KWxkQt6TFJjk/ytNn7wK8A9/L4JEdw+CRHF3ejv8/ASY6kgbPrW9JcU8BnkkDv+PDnVfXZJF/GSY6kkTBRS3pMVX0beEmf8r/BSY6kkbDrW5KkhpmoJUlqmIlakqSGmaglSWqYg8nG3MY+v9kL/m6vJK0WtqglSWqYiVqSpIaZqCVJapiJWpKkhjmYrEEOEJMkzbJFLUlSw0zUkiQ1zEQtSVLDTNSSJDXMRC1JUsNM1JIkNcxELUlSw0zUkiQ1zEQtSVLDTNSSJDXMRC1JUsNM1JIkNcxELUlSw0zUkiQ1bFmJOsl/TvL1JPcm+ViSY5M8N8kdSR5I8vEkTxpUsJIkTZolJ+ok64HfBDZX1YuBY4A3AO8GrqqqU4GHgMsGEagkSZNouV3fa4DjkqwBngLsBc4Cru/WXwO8fpmvIUnSxFqz1AdW1feS/C/gu8DfAZ8D7gQerqoDXbU9wPp+j0+yBdgCMDU1xczMzLyvtX///iOub93UcbB104HDyuf7n/rVna/+Yuou1bi//+Mev6TJtuREneQE4HzgucDDwCeBc/tUrX6Pr6rtwHaAzZs31/T09LyvNTMzw5HWt+79197AlTsOf6t3XTTdt/6l227uW96v/mLqLtW4v//jHr+kybacru9XAd+pqh9W1T8Cnwb+BbC26woH2AA8uMwYJUmaWMtJ1N8FzkjylCQBzgbuA24HLujqXALcsLwQJUmaXEtO1FV1B71BY18FdnTPtR14B/BbSXYCzwSuHkCcklZQkmOS3JXkpm6572WXSZ7cLe/s1m8cZdzSarSsUd9V9a6qekFVvbiq3lRVP6uqb1fV6VV1SlVdWFU/G1SwklbM5cD9c5bnu+zyMuChqjoFuKqrJ2mAnJlM0kGSbABeA3ywWw7zX3Z5frdMt/7srr6kAVnyqG9Jq9Z7gLcDT+uWn8n8l12uB3YDVNWBJI909X809wkXczkmLP6SxmFq5fK+VuKAdmKZlDhM1JIek+S1wL6qujPJ9Gxxn6q1gHWPFyzickxY/CWNw9TK5X2txAHtxDIpcZioJc11JvC6JOcBxwJPp9fCXptkTdeqnnvZ5R7gZGBPd1nmM4Afr3zY0urlOWpJj6mqd1bVhqraSG/u/s9X1UXMf9nljd0y3frPV1XfSY4kLY2JWtJCzHfZ5dXAM7vy3wK2jSg+adWy61tSX1U1A8x0978NnN6nzt8DF65oYNKEsUUtSVLDTNSSJDXMRC1JUsNM1JIkNcxELUlSw0zUkiQ1zEQtSVLDTNSSJDXMRC1JUsNM1JIkNcxELUlSw0zUkiQ1zEQtSVLDTNSSJDXMRC1JUsNM1JIkNcxELUlSw0zUkiQ1zEQtSVLDTNSSJDXMRC1JUsPWjDqASbZx282jDkGS1Dhb1JIkNcxELUlSw0zUkiQ1zHPUY8Rz2pI0eZbVok6yNsn1Sb6R5P4kr0xyYpJbkzzQ/T1hUMFKkjRpltv1/V7gs1X1AuAlwP3ANuC2qjoVuK1bliRJS7DkRJ3k6cAvAVcDVNU/VNXDwPnANV21a4DXLzdISZIm1XLOUT8P+CHwp0leAtwJXA5MVdVegKram+TZ/R6cZAuwBWBqaoqZmZl5X2j//v1HXN+6qeNg66YDK/qag3y/xv39H/f4JU225STqNcDLgN+oqjuSvJdFdHNX1XZgO8DmzZtrenp63rozMzMcaX3r3n/tDVy5Y2XH7e26aHpgzzXu7/+4xy9psi0ne+wB9lTVHd3y9fQS9Q+SrOta0+uAfcsNclzMNyp766YVDkRaoiTHAl8Ankzv+HB9Vb0ryXOB64ATga8Cb6qqf0jyZOAjwMuBvwH+bVXtGknw0iq15HPUVfV9YHeS53dFZwP3ATcCl3RllwA3LCtCSSvpZ8BZVfUS4DTgnCRnAO8GruoGiT4EXNbVvwx4qKpOAa7q6kkaoOWO+v4N4Nok99Dbqf8ncAXw6iQPAK/uliWNgerZ3y0+sbsVcBa9XjM4eJDo3MGj1wNnJ8kKhStNhGWdOK2qu4HNfVadvZznlTQ6SY6hNzj0FOAPgW8BD1fV7IjIPcD67v56YDdAVR1I8gjwTOBHhzznggePwvwDMEcxKLCVwYitxAHtxDIpcTgzmaSDVNXPgdOSrAU+A7ywX7Xub7/Wcx1WsIjBozD/AMxBDpJcqFYGI7YSB7QTy6TE4Vzfkvrq5kWYAc4A1iaZzZwbgAe7+3uAkwG69c8AfryykUqrm4la0mOSPKtrSZPkOOBV9GYcvB24oKs2d5Do3MGjFwCfr6rDWtSSls6ub0lzrQOu6c5TPwH4RFXdlOQ+4LokvwfcRTcjYff3z5LspNeSfsMogpZWMxO1pMdU1T3AS/uUfxs4vU/53wMXrkBo0sSy61uSpIaZqCVJapiJWpKkhpmoJUlqmIlakqSGmaglSWqYiVqSpIaZqCVJapgTnkyYjdtu7lu+64rXrHAkkqSFsEUtSVLDTNSSJDXMRC1JUsNM1JIkNcxELUlSw0zUkiQ1zEQtSVLDTNSSJDXMRC1JUsNM1JIkNcxELUlSw0zUkiQ1zEQtSVLDTNSSJDXMRC1JUsNM1JIkNcxELUlSw0zUkiQ1zEQtSVLDlp2okxyT5K4kN3XLz01yR5IHknw8yZOWH6YkSZNpzQCe43LgfuDp3fK7gauq6rokfwRcBnxgAK+jRdi47eaBPc/WTQe49JDn23XFawby/JKkI1tWizrJBuA1wAe75QBnAdd3Va4BXr+c15AkaZItt0X9HuDtwNO65WcCD1fVgW55D7C+3wOTbAG2AExNTTEzMzPvi+zfv/+I61uxddOBvuVTx82/rhXzvb9bNx3oG/84bI9Z4/L5aUGSk4GPAP8E+AWwvarem+RE4OPARmAX8OtV9VD35fy9wHnA3wKXVtVXRxG7tFotOVEneS2wr6ruTDI9W9ynavV7fFVtB7YDbN68uaanp/tVA3pJ4UjrW3Fo9/CsrZsOcOWOQZxlGJ5dF033Lb+06/o+NP756rdoXD4/jTgAbK2qryZ5GnBnkluBS4HbquqKJNuAbcA7gHOBU7vbK+id5nrFSCKXVqnlZI8zgdclOQ84lt456vcAa5Os6VrVG4AHlx+mpJVQVXuBvd39nya5n16v2PnAdFftGmCGXqI+H/hIVRXwpSRrk6zrnkfSACw5UVfVO4F3AnQt6v9SVRcl+SRwAXAdcAlwwwDilLTCkmwEXgrcAUzNJt+q2pvk2V219cDuOQ+bPd11UKJezKkumP900ShOYbRy6qSVOKCdWCYljmH0x74DuC7J7wF3AVcP4TUkDVGSpwKfAt5WVT/pnYruX7VP2WGnuxZzqgvg/dfe0Pd00ShOubRy6qSVOKCdWCYljoEk6qqaodcVRlV9Gzh9EM8raeUleSK9JH1tVX26K/7BbJd2knXAvq58D3DynId7uksaMGcmk/SYbhT31cD9VfX7c1bdSO9UFhx8SutG4OL0nAE84vlpabDaHoosaaWdCbwJ2JHk7q7st4ErgE8kuQz4LnBht+4Wepdm7aR3edabVzZcafUzUS/BoGb9klpTVV+k/3lngLP71C/gLUMNSppwJmpJY2O+L8lOaavVzHPUkiQ1zEQtSVLDTNSSJDXMRC1JUsNM1JIkNcxELUlSw0zUkiQ1zEQtSVLDTNSSJDXMmckEOC2qJLXKFrUkSQ0zUUuS1DATtSRJDfMctVaEv3okSUtji1qSpIaZqCVJapiJWpKkhpmoJUlqmIlakqSGOepbA+UMZ5I0WLaoJUlqmIlakqSGjU3XtxNmSJImkS1qSZIaZqKWJKlhJmpJkho2NueotTr1G3vguANJepwtakmSGmailiSpYUtO1ElOTnJ7kvuTfD3J5V35iUluTfJA9/eEwYUrSdJkWU6L+gCwtapeCJwBvCXJi4BtwG1VdSpwW7csaUwk+VCSfUnunVPW9wt4et6XZGeSe5K8bHSRS6vTkhN1Ve2tqq92938K3A+sB84HrumqXQO8frlBSlpRHwbOOaRsvi/g5wKndrctwAdWKEZpYgxk1HeSjcBLgTuAqaraC71knuTZ8zxmC70dm6mpKWZmZuZ9/v3797N108/7rjvS44Zl66YDi6o/ddziH9OSfvG//9ob+tbdumn5rzfobbp///6RfE7GVVV9odun5zofmO7uXwPMAO/oyj9SVQV8KcnaJOtmjwGSlm/ZiTrJU4FPAW+rqp8kWdDjqmo7sB1g8+bNNT09PW/dmZkZrvzio33X7bpo/scNy6WL/IWorZsOcOWO8b0SbqXjH/Q2nZmZ4UifLy3IfF/A1wO759Tb05WZqKUBWdbRN8kT6SXpa6vq013xD2a/USdZB+xbbpCSmtXvm3kdVmkRPWiw+F6oYfaYtNIj00oc0E4skxLHkhN1ek3nq4H7q+r356y6EbgEuKL727+PVNI4me8L+B7g5Dn1NgAPHvrgxfSgQe/UymJ6cYbZs9ZKj0wrcUA7sUxKHMsZ9X0m8CbgrCR3d7fz6CXoVyd5AHh1tyxpvM1+AYeDv4DfCFzcjf4+A3jE89PSYC25RV1VX6R/txfA2Ut9XkmjleRj9AaOnZRkD/Auel+4P5HkMuC7wIVd9VuA84CdwN8Cb17xgKVVbnxHOGnV8rfHR6uq3jjPqsO+gHejvd8y3IikyeYUopIkNcxELUlSw0zUkiQ1bOzPUS/2fKa/fzw5Zrf11k0HDpqkxu0taZzYopYkqWEmakmSGmailiSpYSZqSZIaNvaDyaTFckIVSePEFrUkSQ0zUUuS1DATtSRJDfMcNZ6zHBduJ0mTyBa1JEkNM1FLktQwu74lrVrznS6Zj6dR1CJb1JIkNcwWtaSxt9iWszROVm2idsedHG5rSauZXd+SJDVs1baoB8GWmiRp1GxRS5LUMBO1JEkNM1FLktQwE7UkSQ0zUUuS1DBHfUsdf51LUotsUUuS1DATtSRJDbPrW5I6/U5/fPic40cQifQ4E7UkjRHHUkweE7V0FIuZStaD5eqz43uPcOkAphNu6bNhsh8vQzlHneScJN9MsjPJtmG8hqR2uM9LwzPwFnWSY4A/BF4N7AG+nOTGqrpv0K8ltWYSWyru89JwDaPr+3RgZ1V9GyDJdcD5gDuttDq5zy/AsL/E+Wt/o7Fx281s3XTgsNMjg/xynqoa2JMBJLkAOKeq/kO3/CbgFVX11kPqbQG2dIvPB755hKc9CfjRQANdWcY/WuMS/3Oq6lmjDmKxFrLPL3J/h7a2WSuxtBIHtBPLOMex4P19GC3q9Ck77NtAVW0Hti/oCZOvVNXm5QY2KsY/WuMe/xg46j6/mP0d2tpmrcTSShzQTiyTEscwBpPtAU6es7wBeHAIryOpDe7z0hANI1F/GTg1yXOTPAl4A3DjEF5HUhvc56UhGnjXd1UdSPJW4C+BY4APVdXXl/m0C+4ya5Txj9a4x9+0CdjnW4mllTignVgmIo6BDyaTJEmD449ySJLUMBO1JEkNaz5Rj9vUhElOTnJ7kvuTfD3J5V35iUluTfJA9/eEUcc6nyTHJLkryU3d8nOT3NHF/vFuwFCTkqxNcn2Sb3Tb4JXj9N5rZff5JB9Ksi/JvXPK+n5e0vO+Lq57krxsgHEs6rgx5FiOTfLXSb7WxfLfu/K+x4EkT+6Wd3brNw4qlu75F3Q8WoE4diXZkeTuJF/pylZk+zSdqPP41ITnAi8C3pjkRaON6qgOAFur6oXAGcBbupi3AbdV1anAbd1yqy4H7p+z/G7gqi72h4DLRhLVwrwX+GxVvQB4Cb3/Y5ze+4k2gn3+w8A5h5TN93k5Fzi1u20BPjDAOBZ73BhmLD8DzqqqlwCnAeckOYP5jwOXAQ9V1SnAVV29QVro8WjYcQD8clWdNuea6ZXZPlXV7A14JfCXc5bfCbxz1HEt8n+4gd4cyN8E1nVl64Bvjjq2eeLd0H3gzgJuojeZxY+ANf22SUs34OnAd+gGSc4pH4v33tto9nlgI3DvnOW+nxfgj4E39qs3hJiOeNxYqViApwBfBV4x33GA3mj/V3b313T1MqDXX/DxaJhxdM+5CzjpkLIV2T5Nt6iB9cDuOct7urKx0HW9vBS4A5iqqr0A3d9njy6yI3oP8HbgF93yM4GHq+pAt9zyNnge8EPgT7uusg8mOZ7xee/Vxj4/3+dlRWJb4HFjqLF03c13A/uAW4FvMf9x4LFYuvWP0DtuDMJijkfDjAN6s+19Lsmd6U2JCyu0fVpP1AuajrRFSZ4KfAp4W1X9ZNTxLESS1wL7qurOucV9qra6DdYALwM+UFUvBR7Fbu5x0/LnbeixLeK4MdRYqurnVXUavRbt6cALj/B6Q4llCcejYW+fM6vqZfS6td+S5JeOUHegsbSeqMdyasIkT6S3s11bVZ/uin+QZF23fh29b6qtORN4XZJdwHX0upveA6xNMjs5TsvbYA+wp6ru6Javp5e4x+G9V08L+/x8n5ehxrbI48aKvE9V9TAwQ++8+XzHgcdi6dY/A/jxAF5+scejYcUBQFU92P3dB3yG3heYFdk+rSfqsZuaMEmAq4H7q+r356y6Ebiku38JvXNQTamqd1bVhqraSO+9/nxVXQTcDlzQVWsydoCq+j6wO8nzu6Kz6f3UYvPvvR7Twj4/3+flRuDibkTvGcAjs92ey7WE48YwY3lWkrXd/eOAV9EbzDXfcWBujBfQO24suyW7hOPRUOIASHJ8kqfN3gd+BbiXldo+gxx4MIwbcB7w/+idI/lvo45nAfH+S3pdHPcAd3e38+idK7kNeKD7e+KoYz3K/zEN3NTdfx7w18BO4JPAk0cd3xHiPg34Svf+/wVwwri995N+W8l9HvgYsBf4R3qtoMvm+7zQ6878wy6uHcDmAcaxqOPGkGP558BdXSz3Ar/Tlfc9DgDHdss7u/XPG8J2OurxaJhxdK/5te729dnP5UptH6cQlSSpYa13fUuSNNFM1JIkNcxELUlSw0zUkiQ1zEQtSVLDTNSSJDXMRC1JUsP+P8SqNwmoGuz2AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# let's explore the distribution of age\n",
    "\n",
    "X_train[['age', 'fare']].hist(bins=30, figsize=(8,4))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Equal frequency discretisation with pandas and NumPy\n",
    "\n",
    "The interval limits are the quantile limits. We can find those out with pandas qcut."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>501</th>\n",
       "      <td>(0.166, 16.0]</td>\n",
       "      <td>13.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>588</th>\n",
       "      <td>(0.166, 16.0]</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>402</th>\n",
       "      <td>(28.0, 31.0]</td>\n",
       "      <td>30.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1193</th>\n",
       "      <td>(20.0, 22.25]</td>\n",
       "      <td>21.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>686</th>\n",
       "      <td>(20.0, 22.25]</td>\n",
       "      <td>22.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>971</th>\n",
       "      <td>(0.166, 16.0]</td>\n",
       "      <td>16.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>117</th>\n",
       "      <td>(28.0, 31.0]</td>\n",
       "      <td>30.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>540</th>\n",
       "      <td>(0.166, 16.0]</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>294</th>\n",
       "      <td>(42.0, 50.0]</td>\n",
       "      <td>49.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>261</th>\n",
       "      <td>(31.0, 36.0]</td>\n",
       "      <td>35.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                age   age\n",
       "501   (0.166, 16.0]  13.0\n",
       "588   (0.166, 16.0]   4.0\n",
       "402    (28.0, 31.0]  30.0\n",
       "1193  (20.0, 22.25]  21.0\n",
       "686   (20.0, 22.25]  22.0\n",
       "971   (0.166, 16.0]  16.0\n",
       "117    (28.0, 31.0]  30.0\n",
       "540   (0.166, 16.0]   2.0\n",
       "294    (42.0, 50.0]  49.0\n",
       "261    (31.0, 36.0]  35.0"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's use pandas qcut (quantile cut) and I indicate that\n",
    "# we want 10 bins.\n",
    "\n",
    "# retbins = True indicates that I want to capture the limits\n",
    "# of each interval (so I can then use them to cut the test set)\n",
    "\n",
    "Age_disccretised, intervals = pd.qcut(\n",
    "    X_train['age'], 10, labels=None, retbins=True, precision=3, duplicates='raise')\n",
    "\n",
    "pd.concat([Age_disccretised, X_train['age']], axis=1).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see in the above output how by discretising using quantiles, we placed each Age observation within one interval. For example, age 29 was placed in the 26-30 interval, whereas age 63 was placed into the 49-80 interval. \n",
    "\n",
    "Note how the interval widths are different.\n",
    "\n",
    "We can visualise the interval cut points below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.1667, 16.    , 20.    , 22.25  , 25.    , 28.    , 31.    ,\n",
       "       36.    , 42.    , 50.    , 74.    ])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "intervals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And because we generated the bins using the quantile cut method, we should have roughly the same amount of observations per bin. See below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(31.0, 36.0]     106\n",
       "(0.166, 16.0]    103\n",
       "(25.0, 28.0]      95\n",
       "(22.25, 25.0]     94\n",
       "(42.0, 50.0]      92\n",
       "(28.0, 31.0]      91\n",
       "(16.0, 20.0]      91\n",
       "(36.0, 42.0]      84\n",
       "(20.0, 22.25]     81\n",
       "(50.0, 74.0]      79\n",
       "Name: age, dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# roughly the same number of passengers per interval\n",
    "\n",
    "Age_disccretised.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Q1', 'Q2', 'Q3', 'Q4', 'Q5', 'Q6', 'Q7', 'Q8', 'Q9', 'Q10']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we can also add labels instead of having the interval boundaries, to the bins, as follows:\n",
    "\n",
    "labels = ['Q'+str(i) for i in range(1,11)]\n",
    "labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "501     Q1\n",
       "588     Q1\n",
       "402     Q6\n",
       "1193    Q3\n",
       "686     Q3\n",
       "Name: age, dtype: category\n",
       "Categories (10, object): [Q1 < Q2 < Q3 < Q4 ... Q7 < Q8 < Q9 < Q10]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Age_disccretised, intervals = pd.qcut(X_train['age'], 10, labels=labels,\n",
    "                                      retbins=True,\n",
    "                                      precision=3, duplicates='raise')\n",
    "\n",
    "Age_disccretised.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>fare</th>\n",
       "      <th>Age_disc_label</th>\n",
       "      <th>Age_disc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1139</th>\n",
       "      <td>38.0</td>\n",
       "      <td>7.8958</td>\n",
       "      <td>Q8</td>\n",
       "      <td>(36.0, 42.0]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>533</th>\n",
       "      <td>21.0</td>\n",
       "      <td>21.0000</td>\n",
       "      <td>Q3</td>\n",
       "      <td>(20.0, 22.25]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>459</th>\n",
       "      <td>42.0</td>\n",
       "      <td>27.0000</td>\n",
       "      <td>Q8</td>\n",
       "      <td>(36.0, 42.0]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1150</th>\n",
       "      <td>34.0</td>\n",
       "      <td>14.5000</td>\n",
       "      <td>Q7</td>\n",
       "      <td>(31.0, 36.0]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>393</th>\n",
       "      <td>25.0</td>\n",
       "      <td>31.5000</td>\n",
       "      <td>Q4</td>\n",
       "      <td>(22.25, 25.0]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1189</th>\n",
       "      <td>4.0</td>\n",
       "      <td>16.7000</td>\n",
       "      <td>Q1</td>\n",
       "      <td>(0.167, 16.0]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>48.0</td>\n",
       "      <td>26.5500</td>\n",
       "      <td>Q9</td>\n",
       "      <td>(42.0, 50.0]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>231</th>\n",
       "      <td>52.0</td>\n",
       "      <td>30.5000</td>\n",
       "      <td>Q10</td>\n",
       "      <td>(50.0, 74.0]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>330</th>\n",
       "      <td>57.0</td>\n",
       "      <td>13.0000</td>\n",
       "      <td>Q10</td>\n",
       "      <td>(50.0, 74.0]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>887</th>\n",
       "      <td>32.0</td>\n",
       "      <td>8.1125</td>\n",
       "      <td>Q7</td>\n",
       "      <td>(31.0, 36.0]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       age     fare Age_disc_label       Age_disc\n",
       "1139  38.0   7.8958             Q8   (36.0, 42.0]\n",
       "533   21.0  21.0000             Q3  (20.0, 22.25]\n",
       "459   42.0  27.0000             Q8   (36.0, 42.0]\n",
       "1150  34.0  14.5000             Q7   (31.0, 36.0]\n",
       "393   25.0  31.5000             Q4  (22.25, 25.0]\n",
       "1189   4.0  16.7000             Q1  (0.167, 16.0]\n",
       "5     48.0  26.5500             Q9   (42.0, 50.0]\n",
       "231   52.0  30.5000            Q10   (50.0, 74.0]\n",
       "330   57.0  13.0000            Q10   (50.0, 74.0]\n",
       "887   32.0   8.1125             Q7   (31.0, 36.0]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# to transform the test set:\n",
    "# we use pandas cut method (instead of qcut) and\n",
    "# pass the quantile edges calculated in the training set\n",
    "\n",
    "X_test['Age_disc_label'] = pd.cut(x = X_test['age'], bins=intervals, labels=labels)\n",
    "X_test['Age_disc'] = pd.cut(x = X_test['age'], bins=intervals)\n",
    "\n",
    "X_test.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x727f594198>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAFCCAYAAADhQCbjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XmcHVWd/vHPAwkCgiDSIAKZgCLCqGwR8cc4KoiCjBIUUHQ0Kk6c329kZByVuIz7guMyjIooI0hY3FgEhlUGAV/qgCQQhbC6IEZZAoKAuLA8vz+qmtw0t9M33X3rnq5+3q9XXrm3bnWfJ7fS36576pxTsk1EREx9aww6QERETI4U9IiIlkhBj4hoiRT0iIiWSEGPiGiJFPSIiJZIQY+IaIkU9IiIlkhBj4hoiRlNNrbxxht79uzZTTYZETHlLV68+E7bQ2Pt12hBnz17NosWLWqyyYiIKU/Sr3rZL10uEREtkYIeEdESKegRES2Rgh4R0RIp6BERLZGCHhHREinoEREtkYIeEdESKegRES3R6EzRscxecM6Ev8fNR+w7CUkiIqaenKFHRLRECnpEREukoEdEtEQKekRES6SgR0S0RE8FXdKGkk6VdL2k6yQ9T9JGki6UdFP99xP7HTYiIkbX6xn6fwLn234GsANwHbAAuMj2NsBF9fOIiBiQMQu6pCcAfwscC2D7L7bvAfYDFta7LQTm9itkRESMrZcz9K2B5cDXJF0l6auSHg9savtWgPrvTfqYMyIixtBLQZ8B7AwcbXsn4A+sRveKpPmSFklatHz58nHGjIiIsfRS0JcBy2xfXj8/larA3y5pM4D67zu6fbHtY2zPsT1naGjMm1ZHRMQ4jVnQbd8G/FrStvWmPYFrgbOAefW2ecCZfUkYERE96XVxrkOBkyWtBfwCeBPVL4NvSzoEuAU4sD8RIyKiFz0VdNtLgDldXtpzcuNERMR4ZaZoRERLpKBHRLRECnpEREukoEdEtEQKekRES6SgR0S0RAp6RERLpKBHRLRECnpEREukoEdEtEQKekRES6SgR0S0RAp6RERLpKBHRLRECnpEREukoEdEtEQKekRES6SgR0S0RAp6RERLpKBHRLRECnpEREukoEdEtEQKekRES6SgR0S0xIxedpJ0M3Af8DDwkO05kjYCvgXMBm4GDrJ9d39iRkTEWFbnDP1Ftne0Pad+vgC4yPY2wEX184iIGJCJdLnsByysHy8E5k48TkREjFevBd3AdyUtljS/3rap7VsB6r836UfAiIjoTU996MDutn8raRPgQknX99pA/QtgPsCsWbPGETEiInrR0xm67d/Wf98BfAfYFbhd0mYA9d93jPK1x9ieY3vO0NDQ5KSOiIjHGLOgS3q8pPWHHwMvAa4BzgLm1bvNA87sV8iIiBhbL10umwLfkTS8/9dtny/pCuDbkg4BbgEO7F/MiIgYy5gF3fYvgB26bL8L2LMfoSIiYvVlpmhEREukoEdEtEQKekRES6SgR0S0RAp6RERLpKBHRLRECnpEREukoEdEtEQKekRES6SgR0S0RAp6RERL9LoeekQUYPaCcyb8PW4+Yt9JSBIlyhl6RERLpKBHRLRECnpEREukoEdEtEQKekRES6SgR0S0RIYtdjHRoWEZFhYRg5Az9IiIlkhBj4hoiRT0iIiWSEGPiGiJFPSIiJboeZSLpDWBRcBvbP+dpK2AbwIbAVcCr7f9l/7EjEHJiJ+IqWN1ztDfDlzX8fxTwH/Y3ga4GzhkMoNFRMTq6amgS9oC2Bf4av1cwB7AqfUuC4G5/QgYERG96fUM/Ujg3cAj9fMnAffYfqh+vgzYvNsXSpovaZGkRcuXL59Q2IiIGN2YBV3S3wF32F7cubnLru729baPsT3H9pyhoaFxxoyIiLH0clF0d+AVkl4GrA08geqMfUNJM+qz9C2A3/YvZkREjGXMM3Tb77G9he3ZwGuA79l+HXAxcEC92zzgzL6ljIiIMU1kHPrhwDsk/YyqT/3YyYkUERHjsVqrLdq+BLikfvwLYNfJjxQREeORmaIRES2Rgh4R0RIp6BERLZGCHhHRErkFXaEmuigWZGGsiOkmZ+gRES2Rgh4R0RIp6BERLZE+9IiYknKd6bFyhh4R0RIp6BERLZEul4iICSjpvrs5Q4+IaIkU9IiIlkhBj4hoiRT0iIiWSEGPiGiJFPSIiJZIQY+IaIkU9IiIlkhBj4hoicwUjehRSTMCI7rJGXpEREukoEdEtMSYBV3S2pJ+LOknkpZK+nC9fStJl0u6SdK3JK3V/7gRETGaXs7Q/wzsYXsHYEdgb0m7AZ8C/sP2NsDdwCH9ixkREWMZs6C7cn/9dGb9x8AewKn19oXA3L4kjIiInvQ0ykXSmsBi4GnAUcDPgXtsP1TvsgzYfJSvnQ/MB5g1a9ZE80ZEATLip0w9XRS1/bDtHYEtgF2B7brtNsrXHmN7ju05Q0ND408aERGrtFqjXGzfA1wC7AZsKGn4DH8L4LeTGy0iIlZHL6NchiRtWD9eB3gxcB1wMXBAvds84Mx+hYyIiLH10oe+GbCw7kdfA/i27bMlXQt8U9LHgKuAY/uYM6axifbXQvpsY3oYs6Db/imwU5ftv6DqT4+IiAJkpmhEREukoEdEtEQKekRES6SgR0S0RAp6RERLpKBHRLRECnpEREukoEdEtEQKekRES6SgR0S0RAp6RERLpKBHRLRECnpEREukoEdEtEQKekRES6SgR0S0RAp6RERLpKBHRLRECnpEREukoEdEtEQKekRES6SgR0S0RAp6RERLjFnQJW0p6WJJ10laKunt9faNJF0o6ab67yf2P25ERIymlzP0h4B/tb0dsBvwT5K2BxYAF9neBriofh4REQMyZkG3favtK+vH9wHXAZsD+wEL690WAnP7FTIiIsa2Wn3okmYDOwGXA5vavhWqog9sMtnhIiKidz0XdEnrAacBh9m+dzW+br6kRZIWLV++fDwZIyKiBz0VdEkzqYr5ybZPrzffLmmz+vXNgDu6fa3tY2zPsT1naGhoMjJHREQXvYxyEXAscJ3tz3W8dBYwr348Dzhz8uNFRESvZvSwz+7A64GrJS2pt70XOAL4tqRDgFuAA/sTMSIiejFmQbf9A0CjvLzn5MaJiIjxykzRiIiWSEGPiGiJFPSIiJZIQY+IaIkU9IiIlkhBj4hoiRT0iIiWSEGPiGiJFPSIiJZIQY+IaIkU9IiIlkhBj4hoiRT0iIiWSEGPiGiJFPSIiJZIQY+IaIkU9IiIlkhBj4hoiRT0iIiWSEGPiGiJFPSIiJZIQY+IaIkU9IiIlkhBj4hoiTELuqTjJN0h6ZqObRtJulDSTfXfT+xvzIiIGEsvZ+jHA3uP2LYAuMj2NsBF9fOIiBigMQu67e8DvxuxeT9gYf14ITB3knNFRMRqGm8f+qa2bwWo/95k8iJFRMR49P2iqKT5khZJWrR8+fJ+NxcRMW2Nt6DfLmkzgPrvO0bb0fYxtufYnjM0NDTO5iIiYizjLehnAfPqx/OAMycnTkREjFcvwxa/AfwvsK2kZZIOAY4A9pJ0E7BX/TwiIgZoxlg72D54lJf2nOQsERExAZkpGhHREinoEREtkYIeEdESKegRES2Rgh4R0RIp6BERLZGCHhHREinoEREtkYIeEdESKegRES2Rgh4R0RIp6BERLZGCHhHREinoEREtkYIeEdESKegRES2Rgh4R0RIp6BERLZGCHhHREinoEREtkYIeEdESKegRES2Rgh4R0RIp6BERLTGhgi5pb0k3SPqZpAWTFSoiIlbfuAu6pDWBo4B9gO2BgyVtP1nBIiJi9UzkDH1X4Ge2f2H7L8A3gf0mJ1ZERKyuiRT0zYFfdzxfVm+LiIgBkO3xfaF0IPBS22+pn78e2NX2oSP2mw/Mr59uC9ww/rgAbAzcOcHvMVElZIAyciTDCiXkKCEDlJGjhAwwOTn+yvbQWDvNmEADy4AtO55vAfx25E62jwGOmUA7K5G0yPacyfp+UzVDKTmSoawcJWQoJUcJGZrOMZEulyuAbSRtJWkt4DXAWZMTKyIiVte4z9BtPyTpbcAFwJrAcbaXTlqyiIhYLRPpcsH2ucC5k5SlV5PWfTMBJWSAMnIkwwol5CghA5SRo4QM0GCOcV8UjYiIsmTqf0RES6SgR0S0xIT60PtJ0kY97PaI7Xv6HmbASnkvJO3cw24P2r66nzlKIOkdPez2B9tfaXOGUnKUkKGEHMX2oUv6E9W4dq1itzVtz+pzjs/3sNu9tt/fxwylvBf3UQ1XXVWOrWzP7mOGgR+POsetwNGs+r14ne2ntzlDKTlKyFBCjmLP0IHrbO+0qh0kXdVAjv2AD4yxzwKgnwWklPfiCtt7jJHje33OUMLxADjR9kdWtYOkx0+DDKXkKCHDwHOUfIa+tu0/TXSfSchxmO0jJ7rPBDMU8V6UoITjEVGqYgs6gCRRreq4OWCqbocfu+TQDZG0ke3fDaDdDYC9WfmYXDAdrmWMJOmlwFxWfi/OtH3+dMpQSo4SMgw6R7EFXdJLgC8BNwG/qTdvATwN+H+2v9tQjhnAIcD+wFPoOEDAsbYfbCDD+21/rH68PXAGMJOqn+7Vti/vd4a67TcAHwS+y8rHZC/gw7ZPaCDDwI9HneNI4OnACVTrGkH1XrwBuMn226dDhlJylJChhBwlF/TrgH1s3zxi+1bAuba3ayjHN4B7gIWsfIDmARvZfnUDGa60vXP9+Bzgi7bPk7QrcKTt/9PvDHXbNwDPHXk2LumJwOX9vuBUtzXw41HnuLHbv7f+VHmj7W2mQ4ZScpSQoYQcJV8UncGKH9hOv6E6O23Kzra3HbFtGXCZpBsbzDHsKbbPA7D9Y0nrNNi2qM6IR3qEVV/Vn0ylHI8/SdrV9o9HbH8O0NS1jBIylJKjhAwDz1FyQT8OuELSN1lxI40tqVZ1PLbBHHfXa7+fZvsRAElrAAcCdzeUYWtJZ1EVzS0krWv7gfq1Jn+5fRy4UtJ3WXFMZlF1uXy0oQwlHA+ANwJHS1qfFSceWwL31q9Nlwyl5Cghw8BzFNvlAiBpO6phaptTFbNlwFm2r20ww2zgU8AerCgYGwIXAwts/7KBDC8YsWmx7fslbQocYPuofmfoyPJE4KWsfEwusN1IMS3heIzI82Q63gvbtzXZfikZSslRQoZB5ii6oJdG0pOo3rMS7oIy7eV4RKxsSq7lIulDg2jX9l2dxaP+Ldx3ktaU9FZJH5W0+4jX+j2BpieSGl+qdFDHYyySrkyGSgk5SsgAzeSYkgUdWDzoALWm+vK/ArwAuAv4vKTPdbz2yoYyjKWva2T0qMlrK6MaHpE03TNAGTlKyADN5EiXyxQg6ae2n10/nkE1Pn9j4GDgsrGWBYjpQdITgG2AXzR1TaM09XWlRyf02L59wJGQ9Arbjdyes9hRLpL2By61/TtJQ8BngZ2Aa4F/td1tSGO/sgx6xupaww9sPwTMl/QB4HvAeg1lQNKzbf+0fjwTOJzqfbkG+FjHyJt+55hFtQDXPfVF0jnA9bavaaL9sUi62vazGmjnJOAw23fWsxO/CtxAda/fd9o+pd8Z6hxbAp+m+vk4D/j08AQvSWfYnttAhh2BLwMb0DHpTdI9VBMRG+l2kTTyE7OAo+oTMWyf3tf2Sz1Dl3St7e3rx98CLgNOAV5MtVrZXg3lGPiM1foH96SRU4clvQU42nYjQxdHTHD6LPAk4GtU05yfZPsNDWRYALwV+DPwGeCdwA+B3ahmin5uFV8+mTlG6+oS8GXbQw1kePQXh6QfAa+1fbOkjYGLbO/Q7wx12xcCp1H9jB4C7AK83PZdkq5q4hOkpCXAW0fOmpa0G/CVBt+Lh4DzgTtYMTfjAOBUwLbf3Nf2Cy7oNwxPIJG02PYuHa8tsb1jQzmKmLFags4fzvoH6Dm2H6w/wfxkuFuozxmWUp2RrwvcDGxte7mqFewut/3MfmeoczwInEz3iVYH2F6/gQxLgefZvlfSD4C/7Ribv9T2X/c7Q93WSj+Pkv4eeA/wCuCURvqOpZtGm4Up6We2n9bvDHVbzwGOoCrgX7ZtSb+0vVUT7Rfb5QJcIukjwCfrx3NtnyHpRcDvG8xRyozVriQ9ucGxthvUXWFrAI8b/lhd/6dt6szgYdt/lPQX4I9UF4qx/Yfq90pjfgp8pls3j6QXN5Thw8DFko6i+pRyiqQzqcboN7kg1Ux1rPZp+yRJtwEXAE0sWQtwnqplMU5g5YmIb6DB98L2FZL2Ag4FvifpcLr/0u+Lks/QZwLvA4Y/omwB/AH4b6oJJLc0lOM9wEFAtxmr37b9ySZyjEbSObb3baitr43YtMD27fVwwZNt79lAhuOprik8HngAGP6Iuwewvu2D+p2hzvF84Ffd/h9KmmN7UUM5tgHeQrUg1PDJxxm2L2ii/TrDvwBX2r50xPadgH9vsHt0H7pPRDy3ifa75HkKcCQwx/bWjbRZakHvpGrJ1hm27xpQ+9tTfXwc2IzVqNQXlw6kOus5leqi7GuBW4CjbP9hgPEiBmpKFPSRJD3D9vWDzjEd1UPjhmz/fMT2R0fATAeS1gXeRvWL5QtUn9heCVwPfMT2/QPI8GrgVU1mWEW2rqsO9rG9UkZgnQ6cTvUpqfH3f6pOLGpkLXSoCpikT0o6UdLBI177UkMZniXpMkm/lnSMqvVUhl8buapbP3McRFUsTpO0tL4ANOz4hjJ0Ho/XjnitkeNROx7YFNgKOIfqQu1nqD7BHT2gDM8ZQAYk3Sfp3vrPfaruPfvU4e0NxTi+4/ERVKPQPgusQzWcsSnPpRr1dYukb0vaX9JaY33RZCn2DF2j3wxYwDzbT2gox2lUQxYvo+rPf5BqeNifO4fx9TnDD4CP1RneArwJeIXtnzc1LKzOsYRqxM+tqtZiPwF4r+3TGxyeNvDjUedYYnvHeoTPrcBm9cXhJkf8DDxDneMLVOO/3zU8kafJkR11ewMfgdWZQ9Vqi3OpJv89Bzgb+Ea/hzmXPMrlTcC/Uo03HungLtv65am2X1U/PkPS+6iuXr+iwQzrdYxB/4ykxcD5kl5Pg1fQgTVt3wqPrsX+IuBsSVs0mKOE4/GouoCe6/rMqOERP0VksH2opF2Ab0g6A/gizf6/hDJGYEH977Z9H3AicKKkjagGViygz70LJRf0K4BrbP9o5AtqdnGux0laY3h8r+2PS1oGfJ/mZmlK0ga2f19nuFjSq6gmc2zUUAaA+yQ9dbj/vD5TfyHVLfEaGfNMGccDYJGk9Wzf3zlZRNJTgfumUQYAbC+uh2u+DbgUWLvJ9us2h3+pXyZp044RWE2uxvmYfnNX9/79Mg10/ZTc5bIR8KemLmasIse/A9+1/T8jtu8NfGG0yQyTnOG1VOtzXDZi+yzg32z/Q78z1O3tAPzB9s9GbJ8JHGT75AYyDPx4jEWSPOAfrEFmkLQZsNOghgtOZ8UW9IiIWD1TdZTLtKIpsB56RAxeCvrUMBXWQ4+IAUtBnxp2tf1a20dSjXNdT9Lpkh7HihXdogCSNquPy7TOUEqOEjI0mWPKFXRJn5B0uKr7SQ4yxxxJmzfU3ErrodueDyyh4fXQRyNpoaSjJTWy0uEoGZo8HqtyInC9pM9M8wyl5CghQ2M5ptxFUUlzgacCO7iB9bdXkWMh8GzgRtuv7nNbRayHPhpVM0ZnUX2SOHxAGRo7Hj1kEbC97aXTOUMpOUrI0FSOKVfQSyNp/XoSQRSgyeNR/4AO8k5WRWQoJUcJGQado9iCXl/4O832DwvIsgGwNysfoAts3zPQYA1TOQsgDfx4qIw7WQ08Qyk5SshQQo6SC/py4FfAEPAtqnUQrhpAjjcAH6Sastt5gPYCPmz7hKYzDYrKuAVdEcdDBdzJqoQMpeQoIUMJOUqe+r/M9hxVC/i/BjhJ0prAN6iK+40N5XgfsMvIsz9VKx5eTrVA1XTROaJmT1YsgPR94CcNZSjleJRwJ6sSMpSSo4QMA89RckEfXuTmJuCjwEclPZtqYa5zqT7CNEF0X2joEQY8ZLCeYv07290WMOuHEhZAKuV4HAdcIanbnayOnUYZSslRQoaB5yi5y6WxZWHHyDEP+ADVR/zhAzSL6iP+R20fP6BoSPofqhE/p9l+ZwPtlXALumKOh6Tt6H7Ls8buZFVChlJylJBh0DlKLujreYB3XOlUf5x/KSsfoAts3z3QYJQzJKtJJR+PiEEquaCvBTw4PNRH1drbOwPX2j5voOEGoKAhWbkF3Rgkfcj2h6Z7hlJylJChqRwlzxS9AtgQQNK7gI9T3U7qHZI+OchgwyQd01A7L6EaBvUh4GXAvsCHgZvq1xqhAm5BtypNHY8eLB50AMrIAGXkKCEDNJCj5DP0a2w/s368CHi+7T+quuv7lW7ollKrImkX2/0/SOUMyRr4LejGyNfI8YgoVcln6Pd2rA1yJyvugDKDQnI3WDxKGZK10i3ogBcB75P0zzR/y7HHaLKYS9pa0nGSPiZpPUn/JekaSadImt1UjlXk+0CDbe2v6oY0SBqSdIKkqyV9S9XtCZvK8VJJh4x8/yW9uftX9CXDDFVLXZ8v6aeSfiLpPEn/WE/G66siCuMo/hE4WdIJwB1Ut9s6DvgB8ImmQkh6m6SN68dPk/R9SfdIulzSsxqKMTwU6nBJr63/HE417rrJIVn3qbq9GVDdgg54IdUV/UZuQVcPXR1+PFPS+yWdpWrRtnWbyFA7nqpb8H6qG1ZfD+wDnE91vAbtLQ229XFXt1mD6n6iV1G9F+dRTTzrO0mfoJqj8CzgIkmHdrz8tiYy1E4EduSx3aM7ACf1u/Fiu1ygurED8BLg6aw4S216ivdS239dPz4H+Krt76i6l+bHbe++ym8weTkGPiRLZdyCbuCzVeu2O+8yf4vtWd1e63OGe0d7CVjHdiPzTCTdYHvb+vFi27t0vLbE9o4NZLia6rZ3D0naEPg6cIPtf2myO7Dzvejy2o22n97P9kueWITth6l+yw9yVEvne7SJ7e8A2L5E0vpNhbB9HXBdU+2NkqHrbNB6glHfi3mthNmqAI9IejqwAbCupDm2F0l6GrBmQxnuofr33z7yBUm/7rJ/v1wi6SPAJ+vHc22fUY9M+31DGWbYfgjA9j2SXg4cI+kUOpafbsDdkg6kmhvyCICkNYADgb4Pqy25y2VUkpos8KdKOl7S1sB3JB0maZakNwG3NJijK0kfGnQGaHSEyQZ1n+2rGDFblWb78d8N/DfVheG5wHsk/Qz4EfBvDWU4AfirUV77ekMZoOrSeAS4gapwnS7pPuAfgNc3lOHnkl4w/MT2w7YPqTM1Mmig9hrgAOB2STdKugm4jerOYq/pd+PFdrlI2nm0l4CzbW/WYJY3Av+Xalbm46hmKJ4BfMp2U2cgXUl6ue3/HmSGOkdTI34GPlt1NPW1lrvrT5bTkqqVMGfYvqvhdtcBsP3HLq9tbvs3j/2qvmd6ElWNvbOxNgsu6A8Dl9J9fY7dbK/TcKSIlaiASVb1LzJs3yZpCHg+Vd9xozOHS8kxItMnbL93UO3XGbYCdqKaEHl939sruKBfA+xfL8418rVf295yALFG5tjL9oUNtDMDOATYH3gKK2aKngkcO9zt0ECONalGT2wBnO+Oteolvd/2xxrKUUIhPQg4kmoE1kzgjbavqF979MJtnzO8FVhAddLzKeCNwFJgd+DfbTcyAqqEHJI+P3ITVXfPCQC2/7nfGeocZ9ieWz/ej+r/yCVU78Un+r7WkO0i/1D1Q207ymtzB52vznFLQ+18Azga2I2qmG5RPz4a+FaD/96vUvXNHkY16+1zHa9d2VCGg6h+mS2hKhrPaTpD3dYSYLP68a5UwxZfWT+/qqEMVwPrUo30uR94cr39icCSBt+LgeegGvV1EvAGYF79Z/nw4wbfi6s6Hv8I2Kp+vDHwk363X+woF9unruK1M5rKIems0V6i+g/chJ392KFQy4DLJDW1LjxU9wx9NoCkLwJfknQ61ZLGTS1d+16q9dCHZ6ueKOm9tk9vMAOMmGRVj+g4u55I09TH3gdd3SXqAUk/t31bneduNbeccSk5tqNaZntv4F22fyPpg7YXNtT+sM5/7wzbvwSwfaekR/rdeLEFfVUk7Wz7yoaaez7w91RnHivFoDoza8JAh0J1eHT4l6shYvNVzUj8HrBeQxlKKKRQT7Jy3e1T/4J5IdXF8kYmWVENnZzpqstt3+GNktam2RFsA8/h6j6yh0nahepmOOc01fYIO9TzAwQ8TtKTXV1XWIsmhrM29VFkkj/W/FeDbZ0HvGiU177fUIbZVLfhWw7cWP+5o962VYPvxUnA3l22v4XqLK2JDD8Cnjpi2/rARcCfG3wvdgC26bJ9JvC6hjLMojoLHLl9c+DFDb4XReToaFfAPwEnNd32KjJtCDyv3+0Ue1E0uhvEUKiS1LNVH/CIi+VNzlat25PH+OHpZZ+pnqGUHCVkKCFH0ROLJG0g6dWS3iHpX+rHGzacYcx+2V72mYQcT6g/4t/VWczVsbZJE4ZzdNneVI6fjizmUM1WHS7mTRwP4GJJh0qa1blR0lqS9pC0kOqCXNszlJKjhAwDz1HsGbrKubv7JcBpwJm2b+nYvhbwN1QH52L3cThSCUPkSslRwvGo21sbeDPwOmArqmn4a1P1k34XOMr2krZnKCXHKBnWoTppnTbvRckF/QbguR7l7u7u8yI3He2V8J+1iHXIS8hRwvHokmkm1bC0P478/zqdMpSSo4QMg8pRckG/kWqM8e9HbN8AWGR7mwFkGsh/FElX235Wx/PNgLOBhVRnyU2doReRo6P9In5wI0pR8rDFjwNXSup6d/dBBHI1LOvWATRdwhC5knJQtz+o4xFRpGLP0OHR7pVpf3f3gkZ2FJEjIrortqAPevhPSUp5L0rJERHdlTxssZRhSCUo5b0oJUdEdFHyGXoRw5BKUMrIjlJyRER3xRb0ThnNsEIp70UpOSJihSlR0CMiYmwl96FHRMRqSEGPiGiJFPSIiJZIQY8pSdL+kizpGX1sY7aqe9siaY4ee9/KiKKkoMdUdTDwA+A1TTRme5EbutFwxHiloMeUI2k9qrveEGnfAAACJUlEQVSoH0Jd0CWtIelLkpZKOlvSuZIOqF/bRdKlkhZLuqBeVGy0772LpJ9I+l+qu94Mb3+hpLPrxy+QtKT+c5Wk9evt75Z0df31R/TvHYjoruTFuSJGMxc43/aNkn4naWdga6pb9T0L2AS4DjiuHi//BWA/28slvZpq4bc3j/K9vwYcavtSSZ8eZZ93Av9k+4f1L5c/SdqnzvVc2w9I2miS/q0RPUtBj6noYKobbQB8s34+EzjF1U20b5N0cf36tsAzgQtV3choTUZZobFemnlD25fWm04E9umy6w+Bz0k6GTjd9jJJLwa+ZvsBANu/m+C/MWK1paDHlKLqnqp7AM+UZKoCbeA7o30JsNT283r59vX3WiXbR6i6q/zLgMvqYt7T10b0U/rQY6o5ADjB9l/Znm17S+CXwJ3Aq+q+9E2BF9b73wAMSXoeVEsWSOq6dnu9hMHvJf1Nvel13far14S/2vangEXAM6jWsnmzpHXrfdLlEo3LGXpMNQcDIy84ngZsR7Ve/jXAjcDlwO9t/6W+OPr5uktlBlV3zdJRvv+bqPreHwAuGGWfwyS9CHgYuBY4z/afJe0ILJL0F+Bc4L3j/UdGjEfWconWkLSe7fvrbpkfA7vbvm3QuSKakjP0aJOzJW0IrAV8NMU8ppucoce0JOkoqrHsnf7T9tcGkSdiMqSgR0S0REa5RES0RAp6RERLpKBHRLRECnpEREukoEdEtMT/B9Fc8LgGg7F9AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# let's check that we have equal frequency (equal number of observations per bin)\n",
    "X_test.groupby('Age_disc')['age'].count().plot.bar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the top intervals have less observations. This may happen with skewed distributions if we try to divide in a high number of intervals. To make the value spread  more homogeneous, we should discretise in less intervals.\n",
    "\n",
    "## Equal frequency discretisation with Feature-Engine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((916, 2), (393, 2))"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's separate into train and test set\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    data[['age', 'fare']],\n",
    "    data['survived'],\n",
    "    test_size=0.3,\n",
    "    random_state=0)\n",
    "\n",
    "X_train.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# replace NA in both  train and test sets\n",
    "\n",
    "X_train['age'] = impute_na(data, 'age')\n",
    "X_test['age'] = impute_na(data, 'age')\n",
    "\n",
    "X_train['fare'] = impute_na(data, 'fare')\n",
    "X_test['fare'] = impute_na(data, 'fare')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "EqualFrequencyDiscretiser(q=10, return_object=False, variables=['age', 'fare'])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# with feature engine we can automate the process for many variables\n",
    "# in one line of code\n",
    "\n",
    "disc = EqualFrequencyDiscretiser(q=10, variables = ['age', 'fare'])\n",
    "\n",
    "disc.fit(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'age': [-inf,\n",
       "  16.0,\n",
       "  20.0,\n",
       "  22.25000000000003,\n",
       "  25.0,\n",
       "  28.0,\n",
       "  31.0,\n",
       "  36.0,\n",
       "  42.0,\n",
       "  50.0,\n",
       "  inf],\n",
       " 'fare': [-inf,\n",
       "  7.55,\n",
       "  7.7958,\n",
       "  8.05,\n",
       "  10.5,\n",
       "  14.4542,\n",
       "  21.075,\n",
       "  26.55,\n",
       "  40.125,\n",
       "  79.025,\n",
       "  inf]}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# in the binner dict, we can see the limits of the intervals. Note\n",
    "# that the intervals have different widths\n",
    "\n",
    "disc.binner_dict_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# transform train and text\n",
    "\n",
    "train_t = disc.transform(X_train)\n",
    "test_t = disc.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>fare</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>501</th>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>588</th>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>402</th>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1193</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>686</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      age  fare\n",
       "501     0     5\n",
       "588     0     6\n",
       "402     5     4\n",
       "1193    2     1\n",
       "686     2     1"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_t.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'Number of observations per bin')"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XucVXW9//HXm5sIItqA/hQkSKm85mVEzfJWIHaBLDX02DGPj7CTnCyP/YLzK/JSv0PHUvNkFwrKMkLDy49TUyB5KTsqN1EZ0ByRZEsnEBVFxQQ+vz/WGtxuh1mLYdbeW+b9fDzmMXut9f2u9RnE+bC+V0UEZmZm7elW6wDMzKz+OVmYmVkmJwszM8vkZGFmZpmcLMzMLJOThZmZZXKyMDOzTE4WZmaWycnCzMwy9ah1AJ1lwIABMXTo0FqHYWb2lrJo0aJnImJgVrmdJlkMHTqUhQsX1joMM7O3FEl/yVPOzVBmZpbJycLMzDI5WZiZWaadps/CzKwjXnvtNUqlEhs3bqx1KIXq3bs3gwcPpmfPnh2q72RhZl1aqVSiX79+DB06FEm1DqcQEcG6desolUoMGzasQ/cotBlK0mhJj0lqkTSxjesnSFosaZOkMyquDZE0V9JyScskDS0yVjPrmjZu3EhDQ8NOmygAJNHQ0LBDb0+FJQtJ3YHrgdOAg4CzJR1UUewp4NPAjDZu8TPgqog4EBgBrCkqVjPr2nbmRNFqR3/GIpuhRgAtEbECQNJMYCywrLVARKxMr20pr5gmlR4RcUdabkOBcZqZWYYik8UgYFXZcQk4JmfddwLPS7oVGAbMAyZGxObODdHM7I2GTvxNp95v5ZQPt3v9+eefZ8aMGXzuc5/brvt+6EMfYsaMGeyxxx47El5uRSaLtt55ImfdHsD7gSNImqpuImmumvaGB0jjgfEAQ4YM6Wicr7usf44y63f8OWZmqeeff57vfe97b0oWmzdvpnv37tus19TUVHRob1BkB3cJ2K/seDCwejvqPhgRKyJiE3A7cGRloYiYGhGNEdE4cGDm0iZmZnVn4sSJPPHEExx++OEcffTRnHzyyZxzzjkceuihAHzsYx/jqKOO4uCDD2bq1Klb6w0dOpRnnnmGlStXcuCBB/KZz3yGgw8+mFGjRvHKK690epxFJosFwHBJwyT1AsYBs7ej7p6SWjPAKZT1dZiZ7SymTJnC/vvvz5IlS7jqqquYP38+3/jGN1i2LPmVN336dBYtWsTChQu57rrrWLdu3Zvu8fjjj3PRRRfR3NzMHnvswS233NLpcRaWLNI3ggnAHGA5cHNENEu6QtIYAElHSyoBZwI/lNSc1t0MXAr8XtIjJE1aPyoqVjOzejFixIg3zIW47rrreM973sOxxx7LqlWrePzxx99UZ9iwYRx++OEAHHXUUaxcubLT4yp0Ul5ENAFNFecml31eQNI81VbdO4DDiozPzKze9O3bd+vnu+++m3nz5nHffffRp08fTjrppDbnSuyyyy5bP3fv3v0t1wxlZmYZ+vXrx4svvtjmtfXr17PnnnvSp08fHn30Ue6///4qR/c6L/dhZlYma6hrZ2toaOD444/nkEMOYdddd2Xvvffeem306NH84Ac/4LDDDuNd73oXxx57bFVjK+dkYWZWYzNmtLWIRdK89Nvf/rbNa639EgMGDGDp0qVbz1966aWdHh+4GcrMzHJwsjAzs0xOFmZmlsnJwszMMjlZmJlZJicLMzPL5KGzZmbl8qw+vV33a3+l6o4uUQ5w7bXXMn78ePr06dPR6HLzm4WZWQ21LlHeEddeey0vv/xyJ0fUNr9ZmJnVUPkS5SNHjmSvvfbi5ptv5tVXX+X000/n8ssv56WXXuKss86iVCqxefNmvvrVr/K3v/2N1atXc/LJJzNgwADuuuuuQuN0sjAzq6EpU6awdOlSlixZwty5c5k1axbz588nIhgzZgx/+MMfWLt2Lfvuuy+/+U2yi9/69evp378/V199NXfddRcDBgwoPE43Q5mZ1Ym5c+cyd+5cjjjiCI488kgeffRRHn/8cQ499FDmzZvHl7/8Zf74xz/Sv38n96vk4DcLM7M6ERFMmjSJCy+88E3XFi1aRFNTE5MmTWLUqFFMnjy5jTsUx28WZmY1VL5E+amnnsr06dPZsGEDAE8//TRr1qxh9erV9OnTh3PPPZdLL72UxYsXv6lu0XK9WUh6LzC0vHxE/KygmMzMaidjqGtnK1+i/LTTTuOcc87huOOOA2C33XbjxhtvpKWlhS996Ut069aNnj178v3vfx+A8ePHc9ppp7HPPvsU3sGtiGi/gPRzYH9gCbA5PR0R8fnMm0ujge8A3YEfR8SUiusnANeS7Ig3LiJmVVzfnWRL1tsiYkJ7z2psbIyFCxdmhdS+POOrq/wXycyKtXz5cg488MBah1EVbf2skhZFRGNW3TxvFo3AQZGVVSpI6g5cD4wESsACSbMjYllZsaeAT5Pst92WK4F7tue5ZmbW+fL0WSwF/lcH7j0CaImIFRHxd2AmMLa8QESsjIiHgS2VlSUdBewNzO3As83MrBPlebMYACyTNB94tfVkRIzJqDcIWFV2XAKOyROUpG7At4FPAR/IU8fMrKMiAkm1DqNQ29k49CZ5ksVlHbx3W3/yeaP9HNAUEava+w8oaTwwHmDIkCHbHaCZWe/evVm3bh0NDQ07bcKICNatW0fv3r07fI/MZBERHe0zKAH7lR0PBlbnrHsc8H5JnwN2A3pJ2hAREytimwpMhaSDu4NxmlkXNnjwYEqlEmvXrq11KIXq3bs3gwcP7nD9bSYLSfdGxPskvcgb3whEMhpq94x7LwCGSxoGPA2MA87JE1RE/ENZHJ8GGisThZlZZ+jZsyfDhg2rdRh1b5sd3BHxvvR7v4jYveyrX45EQURsAiYAc0iGv94cEc2SrpA0BkDS0ZJKwJnADyU1d8YPZWZmnSvvpLwjgfeRvGHcGxEP5qkXEU1AU8W5yWWfF5A0T7V3j58CP83zPDMzK0bm0FlJk4EbgAaSkVE/lfSVogMzM7P6kefN4mzgiIjYCCBpCrAY+HqRgZmZWf3IMylvJVA+3moX4IlCojEzs7rU3mio/yTpo3gVaJZ0R3o8Eri3OuGZmVk9aK8ZqnVVvkXAbWXn7y4sGjMzq0vbTBYRcUM1AzEzs/rlzY/MzCyTk4WZmWVqN1lI6i7pqmoFY2Zm9andZBERm4GjtLMuxWhmZrnkmZT3IPD/JP0KeKn1ZETcWlhUZmZWV/Iki7cB64BTys4F4GRhZtZF5NnP4vxqBGJmZvUrz0KC75T0e0lL0+PDvJCgmVnXkmfo7I+AScBrABHxMMlGRmZm1kXkSRZ9ImJ+xblNRQRjZmb1KU+yeEbS/qRbq0o6A/hroVGZmVldyZMsLgJ+CLxb0tPAF4DP5rm5pNGSHpPUIulNe2hLOkHSYkmb0iTUev5wSfdJapb0sKRP5vx5zMysAHlGQ60APiipL9AtIl7Mc2NJ3YHrSZY0LwELJM2OiGVlxZ4CPg1cWlH9ZeAfI+JxSfsCiyTNiYjn8zzbzMw6V2aykNQAfI10D25J9wJXRMS6jKojgJY02SBpJjAW2JosImJlem1LecWI+HPZ59WS1gADAScLM7MayNMMNRNYC3wCOCP9fFOOeoOAVWXHpfTcdpE0AuiFd+czM6uZPMnibRFxZUQ8mX59HdgjR7221pOK7QlO0j7Az4HzI2JLG9fHS1ooaeHatWu359ZmZrYd8iSLuySNk9Qt/ToL+E2OeiVgv7LjwcDqvIFJ2j19zlci4v62ykTE1IhojIjGgQMH5r21mZltpzzJ4kJgBsle3K+SNEtdIulFSS+0U28BMFzSMEm9SCbyzc4TVFr+NuBnEfGrPHXMzKw4mckiIvpFRLeI6Jl+dUvP9YuI3duptwmYAMwBlgM3R0SzpCskjQGQdLSkEnAm8ENJzWn1s4ATgE9LWpJ+Hb6DP6uZmXVQnlVnOywimoCminOTyz4vIGmeqqx3I3BjkbGZmVl+hSYLM3trGTqx/e7IlVM+XKVIrN54D24zM8uUZ1Le/kApIl6VdBJwGEnHsyfImXU1l/XPuL6+OnFY1eV5s7gF2CzpAGAaMIxkdJSZmXUReZLFlnRk0+nAtRHxRWCfYsMyM7N6kqeD+zVJZwPnAR9Nz/UsLqTiZHbe9a5SIGZmbzF53izOB44DvhERT0oahoe1mpl1KXmWKF8GfL7s+ElgSpFBmZlZfckzGup44DLg7Wl5ARER7yg2tC7MI07MrM7k6bOYBnwRWARsLjYcMzOrR3mSxfqI+G3hkZiZWd3KkyzuknQVcCvJqrMARMTiwqIyM7O6kidZHJN+byw7F8ApnR+OmZnVozyjoU6uRiBmZla/MudZSOov6erW7UslfVtSxnAdMzPbmeRphpoOLCXZkAjgU8BPgI8XFZTZVh5GbFYX8iSL/SPiE2XHl0taUlRAZtXmPRzMsuVJFq9Iel9E3AtbJ+m9kufmkkYD3wG6Az+OiCkV108AriVZ9nxcRMwqu3Ye8JX08OsRcUOeZ9pbR9YvafB6XWb1Ik+y+GfghrSfQsCzwKezKknqDlwPjARKwAJJs9PlQ1o9ld7r0oq6bwO+RjICK4BFad3ncsRrZmadLM9oqCXAeyTtnh6/kPPeI4CWiFgBIGkmMBbYmiwiYmV6bUtF3VOBOyLi2fT6HcBo4Jc5n21mZp1om8lC0rkRcaOkSyrOAxARV2fcexCwquy4xOtzNrK0VXdQzrpmZjvMfVlv1N6bRd/0e782rkWOe6uD9XLXlTQeGA8wZMiQnLeuLbfTW1ty/b3oYr+crL5sM1lExA/Tj/Mi4k/l19JO7iwlYL+y48HA6pxxlYCTKure3UaMU4GpAI2NjXkTkZmZbac8mx/9Z85zlRYAwyUNk9QLGAfMzhnXHGCUpD0l7QmMSs+ZmVkNtNdncRzwXmBgRb/F7iRDYdsVEZskTSD5Jd8dmB4RzZKuABZGxGxJRwO3AXsCH5V0eUQcHBHPSrqSJOEAXNHa2W1mZtXXXp9FL2C3tEx5v8ULwBl5bh4RTUBTxbnJZZ8XkDQxtVV3OsnscTMzq7H2+izuAe6R9NOI+EsVYzIzszqTZ1Ley+l+FgcDW8fpRISXKDcz6yLyJItfADcBHwE+C5wHrC0yKKsDWQv4gRfxM6sHVVpsM89oqIaImAa8FhH3RMQ/Acd2ytPNzOwtIc+bxWvp979K+jDJXIk2O6XNzGznlCdZfD1dRPBfSeZX7A58sdCozKzLesvMZu9ie63kSRYPRMR6YD3gLVbNzLqgPH0W/y1prqQL0tnUZmbWxWQmi4gYTrIJ0cEk+0r8WtK5hUdmZmZ1I8+bBRExPyIuIdmj4lnAu9aZmXUhmX0W6aZHp5MsBLg/yVpOIwqOywqWuVa/l0k3szJ5OrgfAm4nWczvvoLjMTOzOtRuskj30b4tbYIyM7MqqqfN0trts4iIzcB7qhOKmZnVqzzNUEskzQZ+BbzUejIibi0sKjMzqyt5ksXbgHVA+SqzAThZmJl1EZnJIiLOr0YgZmZWvzLnWUh6p6TfS1qaHh8m6St5bi5ptKTHJLVImtjG9V0k3ZRef0DS0PR8T0k3SHpE0nJJk7bvxzIzs86UZ1Lej4BJpKvPRsTDJHMu2pWOpLoeOA04CDhb0kEVxS4AnouIA4BrgG+m588EdomIQ4GjgAtbE4mZmVVfnj6LPhExX1L5uU056o0AWiJiBYCkmcBYYFlZmbHAZennWcB3lTwogL6SegC7An8n2fvbrOvqYqucWn3J82bxjKT9SX6BI+kM4K856g0CVpUdl9JzbZaJiE0kK9s2kCSOl9LnPAV8KyKerXyApPGSFkpauHatN+8zMytKnjeLi4CpwLslPQ08CeRZSFBtnIucZUYAm4F9gT2BP0qa1/qWsrVgxNQ0NhobGyvvbWZmnSTPaKgVwAcl9QW6RcSLOe9dAvYrOx5MssteW2VKaZNTf5KFCs8BfhcRrwFrJP0JaARWYGZmVZdnNNTF6WKCLwPXSFosaVSOey8AhksaJqkXSaf47Ioys4Hz0s9nAHdGRJA0PZ2iRF+SPb8fzfcjmZlZZ8vTZ/FPEfECMArYCzgfmJJVKe2DmADMAZYDN0dEs6QrJI1Ji00DGiS1AJcArcNrrwd2A5aSJJ2fpKOwzMysBvL0WbT2K3yI5Jf2Q6oYGrUtEdEENFWcm1z2eSPJMNnKehvaOm9mBnhkWA3kebNYJGkuSbKYI6kfsKXYsMzMrJ7kebO4ADgcWBERL0tqIGmKMjOzLiLPaKgt6ezpcyUFcG9E3FZ0YGZ1w00eZrlGQ30P+CzwCEmH84WSri86MDMzqx95mqFOBA5Jh7Qi6QaSxGFmZl1Eng7ux4AhZcf7AR7GambWhWzzzULSf5EsvdEfWC5pfnppBPDfVYjNzMzqRHvNUN+qWhRmZlbXtpksIuKe1s+S9gaOTg/nR8SaogMzM7P6kWc01FnAfJIZ1WcBD6TLlJuZWReRZzTU/wGObn2bkDQQmEey54SZmXUBeUZDdatodlqXs56Zme0k8rxZ/E7SHOCX6fEnqVgc0MzMdm55lvv4kqSPA+8jWYF2qpf7MDPrWvK8WRARtwK3FhyLmZnVKfc9mJlZJicLMzPLtM1kIen36fdvdvTmkkZLekxSi6SJbVzfRdJN6fUH0qXQW68dJuk+Sc2SHpHUu6NxmJnZjmmvz2IfSScCYyTN5PXtVQGIiMXt3VhSd5K9tEcCJWCBpNkRsays2AXAcxFxgKRxwDeBT0rqAdwIfCrdxrUBeG17fzgzM+sc7SWLycBEYDBwdcW1AE7JuPcIoCUiVgCkCWcsUJ4sxgKXpZ9nAd9N9/ceBTwcEQ8BRMS6zJ/EzMwK097aULOAWZK+GhFXduDeg4BVZccl4JhtlYmITZLWAw3AO4FI53cMBGZGxH9UPkDSeGA8wJAhQyovm5lZJ8kzz+JKSWOAE9JTd0fEr3PcW22ci5xlepDM6zgaeBn4vaRFEfH7itimAlMBGhsbK+9tZmadJM9Cgv8OXEzSfLQMuDg9l6VEslFSq8HA6m2VSfsp+gPPpufviYhnIuJlkhnjR+Z4ppmZFSDP0NkPAyMjYnpETAdGp+eyLACGSxomqRcwDphdUWY2cF76+QzgznT71jnAYZL6pEnkRN7Y12FmZlWUawY3sAfJv/gh+dd/prQPYgLJL/7uwPSIaJZ0BbAwImYD04CfS2pJ7z8urfucpKtJEk4ATRHxm7w/lJmZda48yeLfgQcl3UXSx3ACMCnPzSOiiYpFByNictnnjST7ZLRV90aS4bNmZlZjeTq4fynpbpLOZgFfjoj/KTowMzOrH3kXEvwrb+5vMDOzLsJrQ5mZWSYnCzMzy9RuspDUTdLSagVjZmb1qd1kERFbgIckeS0NM7MuLE8H9z5As6T5wEutJyNiTGFRmZlZXcmTLC4vPAozM6treeZZ3CPp7cDwiJgnqQ/JjGwzM+si8iwk+BmSvSZ+mJ4aBNxeZFBmZlZf8gydvQg4HngBICIeB/YqMigzM6sveZLFqxHx99aDdBVY7x1hZtaF5EkW90j6N2BXSSOBXwH/VWxYZmZWT/Iki4nAWuAR4EKSVWS/UmRQZmZWX/KMhtoi6QbgAZLmp8fSDYrMzKyLyEwWkj4M/AB4gmSJ8mGSLoyI3xYdnJmZ1Yc8zVDfBk6OiJMi4kTgZOCaPDeXNFrSY5JaJE1s4/oukm5Krz8gaWjF9SGSNki6NM/zzMysGHmSxZqIaCk7XgGsyaokqTtwPXAacBBwtqSDKopdADwXEQeQJKBvVly/BvAbjJlZjW2zGUrSx9OPzZKagJtJ+izOJNkbO8sIoCUiVqT3mwmMBZaVlRkLXJZ+ngV8V5IiIiR9jCQxvYSZmdVUe30WHy37/DfgxPTzWmDPHPceBKwqOy4Bx2yrTERskrQeaJD0CvBlYCTgJigzsxrbZrKIiPN38N5q67Y5y1wOXBMRG6S2iqSVpfHAeIAhQ7yKuplZUfKMhhoG/AswtLx8jiXKS8B+ZceDgdXbKFNKZ4b3B54leQM5Q9J/AHsAWyRtjIjvlleOiKnAVIDGxkYP5zUzK0ieJcpvB6aRzNresh33XgAMT5PN08A44JyKMrOB84D7gDOAO9M5HO9vLSDpMmBDZaIwM7PqyZMsNkbEddt747QPYgIwh2RJ8+kR0SzpCmBhRMwmSUI/l9RC8kYxbnufY2ZmxcuTLL4j6WvAXODV1pMRsTirYkQ0kSwPUn5uctnnjSSjq9q7x2U5YjQzswLlSRaHAp8CTuH1ZqhIj83MrAvIkyxOB95Rvky5mZl1LXlmcD9EMiLJzMy6qDxvFnsDj0pawBv7LLKGzpqZ2U4iT7L4WuFRmJlZXcuzn8U91QjEzMzqV54Z3C/y+jIdvYCewEsRsXuRgZmZWf3I82bRr/w4XQ12RGERmZlZ3ckzGuoNIuJ2PMfCzKxLydMM9fGyw25AI29ePdbMzHZieUZDle9rsQlYSbJpkZmZdRF5+ix2dF8LMzN7i2tvW9XJ27oGRERcWUA8ZmZWh9p7s2hr7+u+wAVAA+BkYWbWRbS3req3Wz9L6gdcDJwPzAS+va16Zma282m3z0LS24BLgH8AbgCOjIjnqhGYmZnVj/b6LK4CPk6yx/WhEbGhalGZmVldaW9S3r8C+wJfAVZLeiH9elHSC3luLmm0pMcktUia2Mb1XSTdlF5/QNLQ9PxISYskPZJ+9yRAM7Maaq/PYrtnd5eT1B24HhgJlIAFkmZHxLKyYhcAz0XEAZLGAd8EPgk8A3w0IlZLOoRkH+9BOxKPmZl13A4lhAwjgJaIWJHusjeTN0/mG0vSFwIwC/iAJEXEgxGxOj3fDPSWtEuBsZqZWTuKTBaDgFVlxyXe/HawtUxEbALWkwzLLfcJ4MGIeBUzM6uJPMt9dJTaOFe5plS7ZSQdTNI0NarNB0jjgfEAQ4YM6ViUZmaWqcg3ixKwX9nxYGD1tspI6gH0B55NjwcDtwH/GBFPtPWAiJgaEY0R0Thw4MBODt/MzFoVmSwWAMMlDZPUCxgHzK4oMxs4L/18BnBnRISkPYDfAJMi4k8FxmhmZjkUlizSPogJJCOZlgM3R0SzpCskjUmLTQMaJLWQTP5rHV47ATgA+KqkJenXXkXFamZm7Suyz4KIaAKaKs5NLvu8ETizjXpfB75eZGxmZpZfkc1QZma2k3CyMDOzTE4WZmaWycnCzMwyOVmYmVkmJwszM8vkZGFmZpmcLMzMLJOThZmZZXKyMDOzTE4WZmaWycnCzMwyOVmYmVkmJwszM8vkZGFmZpmcLMzMLJOThZmZZSo0WUgaLekxSS2SJrZxfRdJN6XXH5A0tOzapPT8Y5JOLTJOMzNrX2HJQlJ34HrgNOAg4GxJB1UUuwB4LiIOAK4BvpnWPQgYBxwMjAa+l97PzMxqoMg3ixFAS0SsiIi/AzOBsRVlxgI3pJ9nAR+QpPT8zIh4NSKeBFrS+5mZWQ0UmSwGAavKjkvpuTbLRMQmYD3QkLOumZlVSY8C7602zkXOMnnqImk8MD493CDpse2K8M3BDACeabfQ5W2F1rky46iHGOolji4SQ73EUQ8x1Esc9RBDJ8Xx9jzPKTJZlID9yo4HA6u3UaYkqQfQH3g2Z10iYiowtbMClrQwIho7635v5TjqIYZ6iaMeYqiXOOohhnqJox5iqGYcRTZDLQCGSxomqRdJh/XsijKzgfPSz2cAd0ZEpOfHpaOlhgHDgfkFxmpmZu0o7M0iIjZJmgDMAboD0yOiWdIVwMKImA1MA34uqYXkjWJcWrdZ0s3AMmATcFFEbC4qVjMza1+RzVBERBPQVHFuctnnjcCZ26j7DeAbRcbXhk5r0tpB9RBHPcQA9RFHPcQA9RFHPcQA9RFHPcQAVYpDSauPmZnZtnm5DzMzy+RkkcpamqRKMUyXtEbS0lo8P41hP0l3SVouqVnSxTWIobek+ZIeSmO4vNoxlMXSXdKDkn5dwxhWSnpE0hJJC2sYxx6SZkl6NP37cVyVn/+u9M+g9esFSV+oZgxlsXwx/bu5VNIvJfWuQQwXp89vrsafg5uh2Lo0yZ+BkSTDdhcAZ0fEsirHcQKwAfhZRBxSzWeXxbAPsE9ELJbUD1gEfKyafxbpLP6+EbFBUk/gXuDiiLi/WjGUxXIJ0AjsHhEfqfbz0xhWAo0R0f68l+LjuAH4Y0T8OB3h2Ccinq9RLN2Bp4FjIuIvVX72IJK/kwdFxCvpYJymiPhpFWM4hGRVjBHA34HfAf8cEY8X9Uy/WSTyLE1SuIj4A8mosJqJiL9GxOL084vAcqo8ez4SG9LDnulX1f9VI2kw8GHgx9V+dr2RtDtwAskIRiLi77VKFKkPAE9UO1GU6QHsms4P60Mb88AKdiBwf0S8nK5+cQ9wepEPdLJIeHmRNqSrAB8BPFCDZ3eXtARYA9wREVWPAbgW+N/Alho8u1wAcyUtSlctqIV3AGuBn6TNcj+W1LdGsUAyzP6XtXhwRDwNfAt4CvgrsD4i5lY5jKXACZIaJPUBPsQbJzJ3OieLRK7lRboSSbsBtwBfiIgXqv38iNgcEYeTzN4fkb52V42kjwBrImJRNZ+7DcdHxJEkKzhflDZXVlsP4Ejg+xFxBPASUKu+vV7AGOBXNXr+niQtD8OAfYG+ks6tZgwRsZxkle47SJqgHiKZk1YYJ4tEruVFuoq0n+AW4BcRcWstY0mbOu4mWaq+mo4HxqT9BTOBUyTdWOUYAIiI1en3NcBt1GYF5hJQKnvDm0WSPGrhNGBxRPytRs//IPBkRKyNiNeAW4H3VjuIiJgWEUdGxAkkzdeF9VeAk0WrPEuTdAlp5/I0YHlEXF2jGAZK2iP9vCvJ/5yPVjOGiJgUEYMjYijJ34c7I6Kq/3oEkNQ3HWhA2uwziqQJoqoi4n+AVZLelZ76AMkKC7VwNjVqgko9BRwrqU/6/8sHSPr2qkrSXun3IcDHKfjPpNAZ3G8V21qapNpxSPolcBIwQFIJ+FpETKtyGMcDnwIeSfsMAP4tnY1fLfsAN6QjXroBN0d4t7rsAAAByElEQVREzYau1tjewG3J7yR6ADMi4nc1iuVfgF+k/6BaAZxf7QDS9vmRwIXVfnariHhA0ixgMUnTz4PUZjb3LZIagNdIlkR6rsiHeeismZllcjOUmZllcrIwM7NMThZmZpbJycLMzDI5WZiZWSYnCzMzy+RkYWZmmZwszDqBpNvThf6aWxf7k3SBpD9LulvSjyR9Nz0/UNItkhakX8fXNnqzbJ6UZ9YJJL0tIp5NlydZAJwK/Ilk/aQXgTuBhyJigqQZwPci4t50qYY5EXFgzYI3y8HLfZh1js9Lat1PYD+SJVPuiYhnAST9Cnhnev2DwEHpEh4Au0vql+4fYlaXnCzMdpCkk0gSwHER8bKku4HHSDaoaUu3tOwr1YnQbMe5z8Jsx/UHnksTxbuBY0l2TztR0p7pbmqfKCs/F5jQeiDp8KpGa9YBThZmO+53QA9JDwNXAveT7A/9f0l2GZxHspz3+rT854FGSQ9LWgZ8tvohm20fd3CbFUTSbhGxIX2zuI1k6fvbah2XWUf4zcKsOJele4IsBZ4Ebq9xPGYd5jcLMzPL5DcLMzPL5GRhZmaZnCzMzCyTk4WZmWVysjAzs0xOFmZmlun/A58b3J+mRsZDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# and now let's explore the number of observations per bucket\n",
    "\n",
    "t1 = train_t.groupby(['age'])['age'].count() / len(train_t)\n",
    "t2 = test_t.groupby(['age'])['age'].count() / len(test_t)\n",
    "\n",
    "tmp = pd.concat([t1, t2], axis=1)\n",
    "tmp.columns = ['train', 'test']\n",
    "tmp.plot.bar()\n",
    "plt.xticks(rotation=0)\n",
    "plt.ylabel('Number of observations per bin')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'Number of observations per bin')"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3X2cVnWd//HXW0AQRLQB/SFIkFKbqamNmKtLpmmQm2ihoVlqPn7jbrq5tpawvyJvatO11HW1kpIiDdFQi3IKtNTuTO7E5EaWEUlG2kRUFBUV/Pz+OGfscrzmug7DnOs6zryfj8f1mHPz/Z7zuRDnw/neHUUEZmZmlexQ7wDMzKz4nCzMzKwqJwszM6vKycLMzKpysjAzs6qcLMzMrKpck4WkcZJWSmqRNLnM+bGSFkvaImliu3MjJM2TtELSckkj84zVzMw6lluykNQLuA4YD+wLnCJp33bFHgfOAGaWucQPgSsi4t3AGODJvGI1M7PKeud47TFAS0SsBpA0C5gALG8rEBFr0nOvlVZMk0rviLgrLbcpxzjNzKyKPJPFMGBtyX4rcGjGuu8EnpV0OzAKuBuYHBFbO6owePDgGDlyZCdDNTPrmRYtWvRURAypVi7PZKEyx7KuLdIb+AfgIJKmqltImqtueMMNpCagCWDEiBEsXLiws7GamfVIkv6cpVyeHdytwF4l+8OBddtQ98GIWB0RW4CfAAe3LxQR0yKiMSIahwypmhjNzKyT8kwWC4DRkkZJ2hGYBMzZhrq7SWrLAEdR0tdhZma1lVuySJ8IzgXmAiuAWyNimaRLJB0PIOkQSa3AScD1kpaldbcCFwC/kvQwSZPWd/OK1czMKlN3WaK8sbEx3GdhZtvq1VdfpbW1lc2bN9c7lFz169eP4cOH06dPnzccl7QoIhqr1c+zg9vMrPBaW1sZOHAgI0eORCo3LuetLyLYsGEDra2tjBo1qlPX8HIfZtajbd68mYaGhm6bKAAk0dDQsF1PT04WZtbjdedE0WZ7v6OThZmZVeU+CzOzEiMn39ml11tz2XEVzz/77LPMnDmTz372s9t03Y985CPMnDmTXXfddXvCy8zJwsq7aFCGMhvzj8Osm3v22Wf51re+9aZksXXrVnr16tVhvebm5rxDewMnCzOzOpo8eTKPPvooBx54IH369GHnnXdm6NChLFmyhOXLl3PCCSewdu1aNm/ezHnnnUdTUxMAI0eOZOHChWzatInx48dzxBFH8Ic//IFhw4bx05/+lJ122qlL43SfhZlZHV122WXsvffeLFmyhCuuuIL58+fzta99jeXLk0Urpk+fzqJFi1i4cCHXXHMNGzZseNM1Vq1axTnnnMOyZcvYddddue2227o8Tj9ZmJkVyJgxY94wF+Kaa67hjjvuAGDt2rWsWrWKhoaGN9QZNWoUBx54IADve9/7WLNmTZfH5WRhZlYgAwYMeH373nvv5e677+b++++nf//+HHnkkWXnSvTt2/f17V69evHSSy91eVxuhjIzq6OBAwfy/PPPlz23ceNGdtttN/r3788jjzzCH//4xxpH9zd+sjAzK1FtqGtXa2ho4PDDD2e//fZjp512Yo899nj93Lhx4/jOd77DAQccwLve9S7e//731zS2Uk4WZmZ1NnPmzLLH+/btyy9+8Yuy59r6JQYPHszSpUtfP37BBRd0eXzgZigzM8vAycLMzKpysjAzs6qcLMzMrConCzMzqyrXZCFpnKSVklokTS5zfqykxZK2SJpY5vwukp6QdG2ecZqZWWW5DZ2V1Au4DjgGaAUWSJoTEctLij0OnAF0NNbrUuC+vGI0M3uTLCsub9P1Kq/O3NklygGuvvpqmpqa6N+/f2ejyyzPJ4sxQEtErI6IV4BZwITSAhGxJiL+BLzWvrKk9wF7APNyjNHMrK7alijvjKuvvpoXX3yxiyMqL89JecOAtSX7rcChWSpK2gH4JvAp4OiuD83MrBhKlyg/5phj2H333bn11lt5+eWXOfHEE7n44ot54YUXOPnkk2ltbWXr1q18+ctf5q9//Svr1q3jgx/8IIMHD+aee+7JNc48k0W5F75GxrqfBZojYm2l98ZKagKaAEaMGLHNAZqZ1dtll13G0qVLWbJkCfPmzWP27NnMnz+fiOD444/nN7/5DevXr2fPPffkzjuTt/ht3LiRQYMGceWVV3LPPfcwePDg3OPMsxmqFdirZH84sC5j3cOAcyWtAb4BfFrSZe0LRcS0iGiMiMYhQ4Zsb7xmZnU1b9485s2bx0EHHcTBBx/MI488wqpVq9h///25++67ufDCC/ntb3/LoEFd3K+SQZ5PFguA0ZJGAU8Ak4BTs1SMiE+2bUs6A2iMiDeNpjIz604igilTpnD22We/6dyiRYtobm5mypQpHHvssUydOrWmseX2ZBERW4BzgbnACuDWiFgm6RJJxwNIOkRSK3AScL2kZXnFY2ZWRKVLlH/4wx9m+vTpbNq0CYAnnniCJ598knXr1tG/f39OO+00LrjgAhYvXvymunnL9GQh6e+BkaXlI+KH1epFRDPQ3O7Y1JLtBSTNU5Wu8QPgB1niNDPbblWGuna10iXKx48fz6mnnsphhx0GwM4778xNN91ES0sLX/jCF9hhhx3o06cP3/72twFoampi/PjxDB06NPcObkVU7nOWdCOwN7AE2Joejoj4XK6RbaPGxsZYuHBhvcPoPrKMNa/x/1RmeVixYgXvfve76x1GTZT7rpIWRURjtbpZniwagX2jWlYxM7NuK0ufxVLg/+QdiJmZFVeWJ4vBwHJJ84GX2w5GxPG5RWVmVkMRQaU5Xd3B9jYOZUkWF23XHczMCqxfv35s2LCBhoaGbpswIoINGzbQr1+/Tl+jarKICC/kZ2bd1vDhw2ltbWX9+vX1DiVX/fr1Y/jwioNPK+owWUj6XUQcIel53rhMh0hGQ+3S6buamRVEnz59GDVqVL3DKLwOk0VEHJH+HFi7cMzMrIiyTso7GDiC5AnjdxHxYK5RmZlZoVQdOitpKjADaCAZGfUDSV/KOzAzMyuOLE8WpwAHRcRmgHT118XAV/MMzMzMiiPLpLw1QOl4q77Ao7lEY2ZmhVRpNNR/k/RRvAwsk3RXun8M8LvahGdmZkVQqRmqbVW+RcAdJcfvzS0aMzMrpEpDZ2fUMhAzMyuuPF+ramZm3YSThZmZVVVx6KykXsBlEfGFGsVjNTJy8p0Vz6/p/HpjZtYNVXyyiIitwPvUyaUYJY2TtFJSi6TJZc6PlbRY0hZJE0uOHyjpfknLJP1J0ic6c38zM+saWSblPQj8VNKPgRfaDkbE7ZUqpU8l15EMtW0FFkiaExHLS4o9DpwBXNCu+ovApyNilaQ9gUWS5kbEsxniNTOzLpYlWbwN2AAcVXIsgIrJAhgDtETEagBJs4AJwOvJIiLWpOdeK60YEf9Tsr1O0pPAEMDJwsysDrK8z+LMTl57GLC2ZL8VOHRbLyJpDLAjnjVuZlY3WRYSfKekX0lamu4fkHEhwXL9HNv0Xj9JQ4EbgTMj4rUy55skLZS0sLu/uMTMrJ6yDJ39LjAFeBUgIv4ETMpQrxXYq2R/OLAua2CSdgHuBL4UEX8sVyYipkVEY0Q0DhkyJOulzcxsG2VJFv0jYn67Y1sy1FsAjJY0StKOJAlmTpag0vJ3AD+MiB9nqWNmZvnJkiyekrQ3aRNSOsT1L9UqRcQW4FxgLrACuDUilkm6RNLx6bUOkdQKnARcL2lZWv1kYCxwhqQl6efAbf1yZmbWNbKMhjoHmAb8naQngMeAT2a5eEQ0A83tjk0t2V5A0jzVvt5NwE1Z7mFmVhcXDapyfmNt4qiRLKOhVgMfkjQA2CEins8/LDMzK5Iso6EaJF0D/Ba4V9J/SWrIPzQzMyuKLH0Ws4D1wMeBien2LXkGZWZmxZJpBndEXFqy/1VJJ+QVkJmZFU+WJ4t7JE2StEP6OZlk/oOZmfUQWZLF2cBMkndxv0zSLPV5Sc9Lei7P4MzMrBiyjIYaWItAzMysuPymPDMzq8rJwszMqnKyMDOzqrJMyttbUt90+0hJn5O0a/6hmZlZUWR5srgN2CppH+AGYBTJ6CgzM+shsiSL19IVZE8Ero6I84Gh+YZlZmZFkiVZvCrpFOB04OfpsT75hWRmZkWTJVmcCRwGfC0iHpM0Ci8fbmbWo2SZlLcc+FzJ/mPAZXkGZWZmxVI1WUg6HLgIeHtaXkBExDvyDc2sNkZOrrzU2ZrLjqtRJGbFlWXV2RuA84FFwNZ8w8mXfymYmXVOlmSxMSJ+kXskZmZWWFmXKL9C0mGSDm77ZLm4pHGSVkpqkTS5zPmxkhZL2iJpYrtzp0talX5Oz/h9zMwsB1meLA5NfzaWHAvgqEqVJPUCrgOOAVqBBZLmpB3mbR4HzgAuaFf3bcBX0nsGsCit+0yGeM26p4sGVTm/sWfEUKQ4epAso6E+2MlrjwFaImI1gKRZwATg9WQREWvSc6+1q/th4K6IeDo9fxcwDri5k7GYmdl2yDIaahDJv/LHpofuAy6JiGqpexiwtmS/lb89pVRTru6wMrE1AU0AI0aMyHhpM+tI1UEg/WoUiAHV/3tA7QbmZGmGmg4sBU5O9z8FfB/4WJV6KnMsMsaVqW5ETAOmATQ2Nma9tplZVU6cb5QlWewdER8v2b9Y0pIM9VqBvUr2hwPrMsbVChzZru69GeuamVkXy5IsXpJ0RET8Dl6fpPdShnoLgNHp8iBPAJOAUzPGNRf4D0m7pfvHAlMy1jWzt7BMTS897F/1RZAlWfwzMCPtuxDwNMkIpooiYoukc0l+8fcCpkfEMkmXAAsjYo6kQ4A7gN2Aj0q6OCLeExFPS7qUJOFA0kfy9DZ/uwIqUhukmVlWWUZDLQHeK2mXdP+5rBePiGagud2xqSXbC0iamMrVnU7SX2JmZnXWYbKQdFpE3CTp8+2OAxARV+Ycm5mZFUSlJ4sB6c+BZc555JFZF3I7vRVdh8kiIq5PN++OiN+Xnks7uc3y55m6ZoWQZW2o/854zMzMuqlKfRaHAX8PDGnXb7ELyegmMzPrISr1WewI7JyWKe23eA6YWLaGmZl1S5X6LO4D7pP0g4j4cw1jqp9q7ePgNnIz65GyTMp7UdIVwHuA18djRETFJcrNzKz7yNLB/SPgEWAUcDGwhr/NrDYzsx4gS7JoiIgbgFcj4r6I+Azw/pzjMjOzAsnSDPVq+vMvko4jWTm27BIdZmbWPWVJFl9NFxH8N5L5FbsA5+calZmZFUqWZPFA+la8jUBnX7FqZmZ5qNEqB1n6LP4gaZ6ks0reL2FmZj1I1WQREaOBL5EMnV0k6eeSTss9MjMzK4wsTxZExPyI+DwwhuTlRzNyjcrMzAqlarKQtIuk0yX9AvgD8BeSpGFmZj1Elg7uh4CfkLza9P6c4zEzswKq+GQhqRdwR0Sc35lEIWmcpJWSWiRNLnO+r6Rb0vMPSBqZHu8jaYakhyWtkDRlW+9tZmZdp+KTRURslfTezlw4TTTXAccArcACSXMiYnlJsbOAZyJiH0mTgMuBTwAnAX0jYn9J/YHlkm6OiDWdicVsu/gFTGaZmqGWSJoD/Bh4oe1gRNxepd4YoCUiVgNImgVMAEqTxQTgonR7NnCtkpd8BzBAUm9gJ+AVkqXRzcysDrIki7cBG4DSVWYDqJYshgFrS/ZbgUM7KhMRWyRtBBpIEscEks70/sD5EfF0+xtIagKaAEaMGJHhq5iZWWdUTRYRcWYnr61yl8tYZgywFdgT2A34raS7255SSmKbBkwDaGxsbH9tMzPrIlmGzr5T0q8kLU33D5D0pQzXbgX2KtkfTrIIYdkyaZPTIJJ5HKcCv4yIVyPiSeD3QGOGe5qZWQ6yTMr7LjCFdPXZiPgTMClDvQXAaEmjJO2Y1pnTrswc4PR0eyLw64gI4HHgKCUGkCyJ/kiGe5qZWQ6y9Fn0j4j5Sb/z67ZUq5T2QZwLzAV6AdMjYpmkS4CFETEHuAG4UVILyRNFWxK6Dvg+sJSkqer7aZKybmTk5DurllnTr2oRM6uBLMniKUl7k/Y3SJpI0vFcVUQ0A83tjk0t2d5MMky2fb1N5Y6bmVl9ZEkW55B0Iv+dpCeAxwAvJGhm1oNkGQ21GvhQ2newQ0Q8n39YZmZWJFlGQ50naRfgReAqSYslHZt/aGZmVhRZRkN9JiKeA44FdgfOBC7LNSozMyuULMmibRjUR0hGJT1E+cl0ZmbWTWVJFoskzSNJFnMlDQReyzcsMzMrkiyjoc4CDgRWR8SLkhpImqIsL17l1MwKJstoqNfS90ycJimA30XEHXkHZmZmxZFlNNS3gH8CHiaZUX22pOvyDszMzIojSzPUB4D90jWbkDSDJHGYmVkPkaWDeyVQ+rKIvQCv02Rm1oN0+GQh6Wck60ENAlZImp+eGgP8oQaxmZlZQVRqhvpGzaIwM7NC6zBZRMR9bduS9gAOSXfnpy8kMjOzHiLLaKiTgfkkS4afDDyQLlNuZmY9RJbRUP8POKTtaULSEOBuYHaegZmZWXFkGQ21Q7tmpw0Z65mZWTeR5cnil5LmAjen+5+g3dvvzMyse6v6hBARXwCuBw4A3gtMi4gLs1xc0jhJKyW1SJpc5nxfSbek5x9IlxVpO3eApPslLZP0sCS/jdnMrE6yPFkQEbcDt2/LhSX1Aq4DjgFagQWS5kTE8pJiZwHPRMQ+kiYBlwOfkNQbuAn4VEQ8lC5e+Oq23N/MzLpOnn0PY4CWiFgdEa8As4AJ7cpMAGak27OBoyWJ5EVLf0rfnUFEbIiIrTnGamZmFeSZLIYBa0v2W9NjZctExBZgI9AAvBMISXPT17h+sdwNJDVJWihp4fr167v8C5iZWaLDZCHpV+nPyzt57XJv04uMZXoDRwCfTH+eKOnoNxWMmBYRjRHROGTIkE6GaWZm1VTqsxgq6QPA8ZJm0e4Xe0QsrnLtVpJFB9sMB9Z1UKY17acYBDydHr8vIp4CkNQMHAz8qso9zcwsB5WSxVRgMskv+SvbnQvgqCrXXgCMljQKeAKYBJzarswc4HTgfmAi8OuIiHSo7hcl9QdeIVkm/arqX8fMzPJQaW2o2cBsSV+OiEu39cIRsUXSucBcoBcwPSKWSboEWBgRc4AbgBsltZA8UUxK6z4j6UqShBNAc0Tcua0xmJlZ18jyWtVLJR0PjE0P3RsRP89y8Yhopt0EvoiYWrK9mWTNqXJ1byIZPmtmZnWWZSHBrwPnAcvTz3npMTMz6yGyTMo7DjgwIl6D11+r+iAwJc/AzMysOLLOs9i1ZHtQHoGYmVlxZXmy+DrwoKR7SIbPjsVPFWZmPUqWDu6bJd1L8qY8ARdGxP/mHZiZmRVH1oUE/0IyJ8LMzHogv8TIzMyqcrIwM7OqKiYLSTtIWlqrYMzMrJgqJot0bsVDkkbUKB4zMyugLB3cQ4FlkuYDL7QdjIjjc4vKzMwKJUuyuDj3KMzMrNCyzLO4T9LbgdERcXe6bHiv/EMzM7OiyLKQ4P8leT/29emhYcBP8gzKzMyKJcvQ2XOAw4HnACJiFbB7nkGZmVmxZEkWL0fEK2076etP279L28zMurEsyeI+Sf8O7CTpGODHwM/yDcvMzIokS7KYDKwHHgbOJnnz3ZfyDMrMzIqlarJIJ+bNAC4lGUY7IyIyNUNJGidppaQWSZPLnO8r6Zb0/AOSRrY7P0LSJkkXZLmfmZnlI8toqOOAR4FrgGuBFknjM9TrBVwHjAf2BU6RtG+7YmcBz0TEPsBVwOXtzl8F/KLavczMLF9ZJuV9E/hgRLQASNobuJPqv8THAC0RsTqtNwuYQPIe7zYTgIvS7dnAtZIUESHpBGA1JbPGzcysPrL0WTzZlihSq4EnM9QbBqwt2W9Nj5UtExFbgI1Ag6QBwIV49riZWSF0+GQh6WPp5jJJzcCtJENmTwIWZLi2yhxr39fRUZmLgasiYpNUrsjrMTYBTQAjRnitQzOzvFRqhvpoyfZfgQ+k2+uB3TJcuxXYq2R/OLCugzKt6fyNQcDTwKHAREn/CewKvCZpc0RcW1o5IqYB0wAaGxs998PMLCcdJouIOHM7r70AGC1pFPAEMAk4tV2ZOcDpwP3ARODX6Uirf2grIOkiYFP7RGFmZrVTtYM7/WX/L8DI0vLVliiPiC2SzgXmkiw8OD0ilkm6BFgYEXOAG4AbJbWQPFFM6uwXMTOz/GQZDfUTkl/qPwNe25aLR0QzySS+0mNTS7Y3k/SBVLrGRdtyTzMz63pZksXmiLgm90jMzKywsiSL/5L0FWAe8HLbwYhYnFtUZmZWKFmSxf7Ap4Cj+FszVKT7ZmbWA2RJFicC7yhdptzMzHqWLDO4HyKZ62BmZj1UlieLPYBHJC3gjX0WFYfOmplZ95ElWXwl9yjMzKzQqiaLiLivFoGYmVlxZZnB/Tx/WwBwR6AP8EJE7JJnYGZmVhxZniwGlu6n75kYk1tEZmZWOFlGQ71BRPwEz7EwM+tRsjRDfaxkdwegkTe/l8LMzLqxLKOhSt9rsQVYQ/I6VDMz6yGy9Fls73stzMzsLa7Sa1WndnQOiIi4NId4zMysgCo9WbxQ5tgA4CygAXCyMDPrISq9VvWbbduSBgLnAWcCs4BvdlTPzMy6n4p9FpLeBnwe+CQwAzg4Ip6pRWBmZlYcHc6zkHQFsAB4Htg/Ii7a1kQhaZyklZJaJE0uc76vpFvS8w9IGpkeP0bSIkkPpz89r8PMrI4qTcr7N2BP4EvAOknPpZ/nJT1X7cKSegHXAeOBfYFTJO3brthZwDMRsQ9wFXB5evwp4KMRsT9wOnDjtnwpMzPrWpX6LLZ5dnc7Y4CWiFgNIGkWyfyM5SVlJgAXpduzgWslKSIeLCmzDOgnqW9EvIyZmdXc9iaESoYBa0v2W9NjZctExBZgI8lIq1IfBx50ojAzq58sM7g7S2WOtV8mpGIZSe8haZo6tuwNpCagCWDEiBGdi9LMzKrK88miFdirZH84sK6jMpJ6A4OAp9P94cAdwKcj4tFyN4iIaRHRGBGNQ4YM6eLwzcysTZ7JYgEwWtIoSTsCk4A57crMIenABpgI/DoiQtKuwJ3AlIj4fY4xmplZBrkli7QP4lxgLrACuDUilkm6RFLb+7tvABoktZDM52gbXnsusA/wZUlL0s/uecVqZmaV5dlnQUQ0A83tjk0t2d4MnFSm3leBr+YZm5mZZZdnM5SZmXUTThZmZlaVk4WZmVXlZGFmZlU5WZiZWVVOFmZmVpWThZmZVeVkYWZmVTlZmJlZVU4WZmZWlZOFmZlV5WRhZmZVOVmYmVlVThZmZlaVk4WZmVXlZGFmZlU5WZiZWVVOFmZmVlWuyULSOEkrJbVImlzmfF9Jt6TnH5A0suTclPT4SkkfzjNOMzOrLLdkIakXcB0wHtgXOEXSvu2KnQU8ExH7AFcBl6d19wUmAe8BxgHfSq9nZmZ1kOeTxRigJSJWR8QrwCxgQrsyE4AZ6fZs4GhJSo/PioiXI+IxoCW9npmZ1UGeyWIYsLZkvzU9VrZMRGwBNgINGeuamVmN9M7x2ipzLDKWyVIXSU1AU7q7SdLKbYrwzcEMBp6qWOjicqF1rapxFCGGosTRQ2IoShxFiKEocRQhhi6K4+1Z7pNnsmgF9irZHw6s66BMq6TewCDg6Yx1iYhpwLSuCljSwoho7KrrvZXjKEIMRYmjCDEUJY4ixFCUOIoQQy3jyLMZagEwWtIoSTuSdFjPaVdmDnB6uj0R+HVERHp8UjpaahQwGpifY6xmZlZBbk8WEbFF0rnAXKAXMD0ilkm6BFgYEXOAG4AbJbWQPFFMSusuk3QrsBzYApwTEVvzitXMzCrLsxmKiGgGmtsdm1qyvRk4qYO6XwO+lmd8ZXRZk9Z2KkIcRYgBihFHEWKAYsRRhBigGHEUIQaoURxKWn3MzMw65uU+zMysKieLVLWlSWoUw3RJT0paWo/7pzHsJekeSSskLZN0Xh1i6CdpvqSH0hgurnUMJbH0kvSgpJ/XMYY1kh6WtETSwjrGsauk2ZIeSf9+HFbj+78r/TNo+zwn6V9rGUNJLOenfzeXSrpZUr86xHBeev9ltfhzcDMUry9N8j/AMSTDdhcAp0TE8hrHMRbYBPwwIvar5b1LYhgKDI2IxZIGAouAE2r5Z5HO4h8QEZsk9QF+B5wXEX+sVQwlsXweaAR2iYh/rPX90xjWAI0RUXneS/5xzAB+GxHfS0c49o+IZ+sUSy/gCeDQiPhzje89jOTv5L4R8VI6GKc5In5Qwxj2I1kVYwzwCvBL4J8jYlVe9/STRSLL0iS5i4jfkIwKq5uI+EtELE63nwdWUOPZ85HYlO72ST81/1eNpOHAccD3an3vopG0CzCWZAQjEfFKvRJF6mjg0VonihK9gZ3S+WH9KTMPLGfvBv4YES+mq1/cB5yY5w2dLBJeXqSMdBXgg4AH6nDvXpKWAE8Cd0VEzWMArga+CLxWh3uXCmCepEXpqgX18A5gPfD9tFnue5IG1CkWSIbZ31yPG0fEE8A3gMeBvwAbI2JejcNYCoyV1CCpP/AR3jiRucs5WSQyLS/Sk0jaGbgN+NeIeK7W94+IrRFxIMns/THpY3fNSPpH4MmIWFTL+3bg8Ig4mGQF53PS5spa6w0cDHw7Ig4CXgDq1be3I3A88OM63X83kpaHUcCewABJp9UyhohYQbJK910kTVAPkcxJy42TRSLT8iI9RdpPcBvwo4i4vZ6xpE0d95IsVV9LhwPHp/0Fs4CjJN1U4xgAiIh16c8ngTuozwrMrUBryRPebJLkUQ/jgcUR8dc63f9DwGMRsT4iXgVuB/6+1kFExA0RcXBEjCVpvs6tvwKcLNpkWZqkR0g7l28AVkTElXWKYYikXdPtnUj+53ykljFExJSIGB4RI0n+Pvw6Imr6r0cASQPSgQakzT7HkjRB1FRE/C+wVtK70kNHk6ywUA+nUKcmqNTjwPsl9U//fzmapG+vpiTtnv4cAXyMnP9Mcp3B/VbR0dIktY5D0s3AkcBgSa3AVyLihhqHcTjwKeDhtM8A4N/T2fi1MhSYkY542QG4NSLqNnS1zvYA7kh+J9GL5grqAAABpklEQVQbmBkRv6xTLP8C/Cj9B9Vq4MxaB5C2zx8DnF3re7eJiAckzQYWkzT9PEh9ZnPfJqkBeJVkSaRn8ryZh86amVlVboYyM7OqnCzMzKwqJwszM6vKycLMzKpysjAzs6qcLMy2k6TPpauw/qjesZjlxUNnzbaTpEeA8RHxWIayvdOF38zeUjwpz2w7SPoOySJ7c9LlQCYAOwEvAWdGxEpJZ5CsXtsPGECydMgXgJOBvsAdEfGVesRvlpWThdl2iIh/kjQO+CDJewW+ma4I8CHgP4CPp0UPAw6IiKclHQuMJlnjSSSJZmy6RL1ZITlZmHWdQSTLlIwmWbW4T8m5uyKi7V0lx6afB9P9nUmSh5OFFZaThVnXuRS4JyJOTN8Fcm/JuRdKtgV8PSKur11oZtvHo6HMus4gkld9ApxRodxc4DPpO0OQNKxtBVGzonKyMOs6/wl8XdLvSVYvLit9q9pM4H5JD5O8G2JgbUI06xwPnTUzs6r8ZGFmZlU5WZiZWVVOFmZmVpWThZmZVeVkYWZmVTlZmJlZVU4WZmZWlZOFmZlV9f8BzmyXudpUV0cAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "t1 = train_t.groupby(['fare'])['fare'].count() / len(train_t)\n",
    "t2 = test_t.groupby(['fare'])['fare'].count() / len(test_t)\n",
    "\n",
    "tmp = pd.concat([t1, t2], axis=1)\n",
    "tmp.columns = ['train', 'test']\n",
    "tmp.plot.bar()\n",
    "plt.xticks(rotation=0)\n",
    "plt.ylabel('Number of observations per bin')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note how equal frequency discretisation obtains a better value spread across the different intervals.\n",
    "\n",
    "## Equal frequency discretisation with Scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((916, 2), (393, 2))"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's separate into train and test set\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    data[['age', 'fare']],\n",
    "    data['survived'],\n",
    "    test_size=0.3,\n",
    "    random_state=0)\n",
    "\n",
    "X_train.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# replace NA in both  train and test sets\n",
    "\n",
    "X_train['age'] = impute_na(data, 'age')\n",
    "X_test['age'] = impute_na(data, 'age')\n",
    "\n",
    "X_train['fare'] = impute_na(data, 'fare')\n",
    "X_test['fare'] = impute_na(data, 'fare')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KBinsDiscretizer(encode='ordinal', n_bins=10, strategy='quantile')"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "disc = KBinsDiscretizer(n_bins=10, encode='ordinal', strategy='quantile')\n",
    "\n",
    "disc.fit(X_train[['age', 'fare']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([array([ 0.1667, 16.    , 20.    , 22.25  , 25.    , 28.    , 31.    ,\n",
       "       36.    , 42.    , 50.    , 74.    ]),\n",
       "       array([  0.    ,   7.55  ,   7.7958,   8.05  ,  10.5   ,  14.4542,\n",
       "        21.075 ,  26.55  ,  40.125 ,  79.025 , 512.3292])], dtype=object)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "disc.bin_edges_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>fare</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   age  fare\n",
       "0  0.0   5.0\n",
       "1  0.0   6.0\n",
       "2  5.0   4.0\n",
       "3  2.0   1.0\n",
       "4  2.0   1.0"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_t = disc.transform(X_train[['age', 'fare']])\n",
    "\n",
    "train_t = pd.DataFrame(train_t, columns = ['age', 'fare'])\n",
    "\n",
    "train_t.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_t = disc.transform(X_test[['age', 'fare']])\n",
    "\n",
    "test_t = pd.DataFrame(test_t, columns = ['age', 'fare'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'Number of observations per bin')"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XucVXW9//HXW0QuykUH9KeggZcu3g7qRJoe85IK9guy1NBjqcffD/upvyzTn3hOkXg50anUPKlFSnk5eEMxyjHIvJSZwqBogBqIJCMWIyqCigp+fn+sNbrdzOy9GGbtWTLv5+OxH7Mu37X2e/Zs/bDWd63vUkRgZmZWyWadHcDMzIrPxcLMzKpysTAzs6pcLMzMrCoXCzMzq8rFwszMqnKxMDOzqlwszMysKhcLMzOravPODtBRBgwYEEOGDOnsGGZmHypz5sx5KSIGVmu3yRSLIUOG0NjY2NkxzMw+VCT9LUs7n4YyM7Oqci0WkkZIekbSIknjWll/sKTHJK2VdGwr6/tKekHST/LMaWZmleVWLCR1A64CRgK7AydI2r2s2fPAKcCUNnZzMfBgXhnNzCybPPsshgOLImIxgKRbgNHAgpYGEbEkXfdu+caS9gO2A34L1OeY08y6sHfeeYempibWrFnT2VFy1bNnTwYPHkz37t3btX2exWIQsLRkvgn4VJYNJW0G/Aj4CnB4x0czM0s0NTXRp08fhgwZgqTOjpOLiGDFihU0NTUxdOjQdu0jzz6L1j71rE9aOgNoiIillRpJGiupUVJjc3PzBgc0M1uzZg11dXWbbKEAkERdXd1GHT3leWTRBOxYMj8YWJZx2wOAf5Z0BrAVsIWk1RHxgU7yiJgETAKor6/3I//MrF025ULRYmN/xzyLxWxgN0lDgReAMcCJWTaMiH9pmZZ0ClBfXijMzKx2cisWEbFW0lnADKAbMDki5ku6CGiMiOmSPglMA7YGPi9pQkTskVcmM7Nqhoy7u0P3t2Ti5yquf/XVV5kyZQpnnHHGBu336KOPZsqUKfTv339j4mWW6x3cEdEANJQtG18yPZvk9FSlffwS+GUO8dZ3Yb8MbVbmn8PMuoxXX32Vq6++er1isW7dOrp169bmdg0NDW2uy8MmM9yHmdmH0bhx43j22WcZNmwY3bt3Z6uttmL77bdn7ty5LFiwgC984QssXbqUNWvWcPbZZzN27Fjg/SGOVq9ezciRIznooIN4+OGHGTRoEL/61a/o1atXh+b0cB9mZp1o4sSJ7LLLLsydO5cf/OAHzJo1i0svvZQFC5Jb0iZPnsycOXNobGzkyiuvZMWKFevtY+HChZx55pnMnz+f/v37c8cdd3R4Th9ZmJkVyPDhwz9wL8SVV17JtGnTAFi6dCkLFy6krq7uA9sMHTqUYcOGAbDffvuxZMmSDs/lYmFmViBbbrnle9MPPPAA9957L3/+85/p3bs3hxxySKv3SvTo0eO96W7duvHmm292eC6fhjIz60R9+vRh1apVra5buXIlW2+9Nb179+bpp5/mkUceqXG69/nIwsysRLVLXTtaXV0dBx54IHvuuSe9evViu+22e2/diBEj+OlPf8ree+/Nxz72Mfbff/+aZivlYmFm1smmTGl94O0ePXpwzz33tLqupV9iwIABzJs3773l5557bofnA5+GMjOzDFwszMysKhcLMzOrysXCzMyqcrEwM7OqXCzMzKwqXzprZlYqy+jTG7S/yiNVt3eIcoArrriCsWPH0rt37/amy8xHFmZmnahliPL2uOKKK3jjjTc6OFHrfGRhZtaJSocoP+KII9h222257bbbeOuttzjmmGOYMGECr7/+OscffzxNTU2sW7eO73znO/zjH/9g2bJlHHrooQwYMID7778/15wuFmZmnWjixInMmzePuXPnMnPmTKZOncqsWbOICEaNGsUf/vAHmpub2WGHHbj77uQpfitXrqRfv35cdtll3H///QwYMCD3nD4NZWZWEDNnzmTmzJnss88+7Lvvvjz99NMsXLiQvfbai3vvvZfzzz+fP/7xj/Tr18H9Khn4yMLMrCAiggsuuIDTTz99vXVz5syhoaGBCy64gCOPPJLx48e3sof8+MjCzKwTlQ5RftRRRzF58mRWr14NwAsvvMDy5ctZtmwZvXv35qSTTuLcc8/lscceW2/bvGU6spD0aWBIafuIuCGnTGZmnafKpa4drXSI8pEjR3LiiSdywAEHALDVVltx0003sWjRIs477zw222wzunfvzjXXXAPA2LFjGTlyJNtvv33uHdyKiMoNpBuBXYC5wLp0cUTE16vuXBoB/BjoBlwbERPL1h8MXAHsDYyJiKnp8mHANUDf9D0vjYhbK71XfX19NDY2VotUWZbrq2v8RTKzfD311FN84hOf6OwYNdHa7yppTkTUV9s2y5FFPbB7VKsqZSR1A64CjgCagNmSpkfEgpJmzwOnAOUDsL8BfDUiFkraAZgjaUZEvLohGczMrGNkKRbzgP8BvLiB+x4OLIqIxQCSbgFGA+8Vi4hYkq57t3TDiPhryfQyScuBgYCLhZlZJ8hSLAYACyTNAt5qWRgRo6psNwhYWjLfBHxqQwNKGg5sATy7oduamWUREUjq7Bi52sCTQ+vJUiwubOe+W/vkN/RU1vbAjcDJEfFuK+vHAmMBdtppp/ZkNLMSQ8bdXXF9rZ9PXQs9e/ZkxYoV1NXVbbIFIyJYsWIFPXv2bPc+qhaLiHiwnftuAnYsmR8MLMu6saS+wN3AtyPikTayTQImQdLB3c6cZtaFDR48mKamJpqbmzs7Sq569uzJ4MGD2719m8VC0kMRcZCkVXzwiEAkV0P1rbLv2cBukoYCLwBjgBOzhJK0BTANuCEibs+yjZlZe3Tv3p2hQ4d2dozCa/OmvIg4KP3ZJyL6lrz6ZCgURMRa4CxgBvAUcFtEzJd0kaRRAJI+KakJOA74maT56ebHAwcDp0iam76GbdRvamZm7Zb1prx9gYNIjjAeiojHs2wXEQ1AQ9my8SXTs0lOT5VvdxNwU5b3MDOz/FUd7kPSeOB6oI7kyqhfSvp23sHMzKw4shxZnADsExFrACRNBB4DLskzmJmZFUeWgQSXAKXXW/XA9zyYmXUpla6G+i+SPoq3gPmSfpfOHwE8VJt41mk8TpaZlah0GqplVL45JJextnggtzRmZlZIbRaLiLi+lkHMzKy4/PAjMzOryo9VLaJq/QXuKzCzGqtYLNJnUkyMiPNqlMesS6o2gB9smoP42YdHxdNQEbEO2E+b6lCMZmaWSZbTUI8Dv5J0O/B6y8KIuDO3VGZmVihZisU2wArgsJJlAbhYmJl1EVmeZ3FqLYKYmVlxVS0Wkj4KXANsFxF7StobGBURHhvqQ6zqE9Ha/0AtM9sEZbnP4ufABcA7ABHxJMmDjMzMrIvIUix6R8SssmVr8whjZmbFlKVYvCRpF9JHq0o6Fngx11RmZlYoWa6GOhOYBHxc0gvAc8C/5JrKzMwKJcvVUIuBz0raEtgsIlblH8vMzIp0Z3+Wq6HqgO+SPoNb0kPARRGxIu9wZmZWRY3GksvSZ3EL0Ax8CTg2nb61Q97dzMw+FLIUi20i4uKIeC59XQL0z7JzSSMkPSNpkaRxraw/WNJjktamHeel606WtDB9nZzt1zEzszxk6eC+X9IY4LZ0/lig6om0dMTaq0gew9oEzJY0PSIWlDR7HjgFOLds221ITn3Vk1yFNSfd9pUMec3M8tfFHiWQ5cjidGAKybO43yI5LXWOpFWSXquw3XBgUUQsjoi30+1GlzaIiCXpTX7vlm17FPC7iHg5LRC/A0Zk+o3MzKzDZbkaqk879z0IWFoy3wR8aiO2HdTOHGZmG8xD4nxQno9Vbe0ZGNGR20oaK6lRUmNzc/MGhTMzs+zyLBZNwI4l84OBZR25bURMioj6iKgfOHBgu4OamVlleRaL2cBukoZK2oJk8MHpGbedARwpaWtJWwNHpsvMzKwTZLkpbxegKSLeknQIsDdwQ0S8Wmm7iFgr6SyS/8l3AyZHxHxJFwGNETFd0ieBacDWwOclTYiIPSLiZUkXkxQcSG4CfLndv2WBZLojs4udCzWz4sty6ewdQL2kXYHrSI4OpgBHV9swIhqAhrJl40umZ5OcYmpt28nA5Az5zDZK1Y7MGg2nYFZkWU5DvRsRa4FjgCsi4pvA9vnGMjOzIslyZPGOpBOAk4HPp8u65xfJzLqyIg2eZ+/LcmRxKnAAcGlEPCdpKHBTvrHMzKxIstyUtwD4esn8c8DEPEOZmVmxZLka6kDgQuAjaXsBERE75xut4/mOTPtQK8JYREXIYJ0iS5/FdcA3gTnAunzjmJlZEWUpFisj4p7ck5iZWWFlHaL8B8CdJKPOAhARj+WWyszMCiVLsWgZKba+ZFkAh3V8HDOzDNx3UnNZroY6tBZBzMysuKreZyGpn6TLWoYCl/QjSVXKupmZbUqy3JQ3GVgFHJ++XgN+kWcoMzMrlix9FrtExJdK5idImptXIDMzK54sRxZvSjqoZSa9Se/N/CKZmVnRZDmy+D/A9Wk/hYCXgVPyDGVmZsWS5WqoucA/Seqbzr+WeyozMyuUNouFpJMi4iZJ55QtByAiLss5m5mZFUSlI4st0599WlkXOWQxM7OCarNYRMTP0sl7I+JPpevSTm4zM+sislwN9V8Zl5mZ2SaqUp/FAcCngYFl/RZ9gW55BzMzs+Ko1GexBbBV2qa03+I14Ng8Q5mZWbFU6rN4EHhQ0i8j4m/t2bmkEcCPSY5Ero2IiWXrewA3APsBK4AvR8QSSd2Ba4F904w3RMT32pPBzMw2Xpab8t5In2exB/Deg0cjouIQ5ZK6AVcBRwBNwGxJ09Nnerc4DXglInaVNAb4PvBl4DigR0TsJak3sEDSzRGxZAN+NzMz6yBZOrj/G3gaGApMAJYAszNsNxxYFBGLI+Jt4BZgdFmb0cD16fRU4HAlN3IEsKWkzYFewNskp7/MzKwTZCkWdRFxHfBORDwYEf8K7J9hu0HA0pL5pnRZq20iYi2wEqgjKRyvAy8CzwM/jIiXy99A0tiWodObm5szRDIzs/bIUizeSX++KOlzkvYBBmfYTq0sK7+Zr602w4F1wA4kRzTfkrTzeg0jJkVEfUTUDxw4MEMkMzNrjyx9Fpekgwh+i+T+ir7ANzNs1wTsWDI/GFjWRpum9JRTP5KBCk8EfhsR7wDLJf2J5LGuizO8r5mZdbAsRxaPRsTKiJgXEYdGxH4RMT3DdrOB3SQNlbQFMAYo3246cHI6fSxwX0QEyamnw5TYkuS019OZfiMzM+twWYrFw5JmSjpN0tZZd5z2QZwFzACeAm6LiPmSLpI0Km12HVAnaRFwDjAuXX4VyT0e80iKzi8i4sms721mZh0ryxDlu0kaTnJk8O+SFgC3RMRNGbZtABrKlo0vmV5Dcpls+XarW1tuZmadI0ufBRExC5gl6T+Ay0gud61aLMw22oX9qqxfWZscZl1c1dNQkvpKOlnSPcDDJJezDs89mZmZFUaWI4sngLuAiyLizznnMTOzAqpYLNIhO6ZFxDmV2pmZ2aatYrGIiHWS/qlWYcwKyf0mZplOQ82VNB24nWQIDgAi4s7cUpmZWaFkKRbbkAwfXjrKbAAuFmZmXUSW+yxOrUUQMzMrriyXzn5U0u8lzUvn95b07fyjmZlZUWQ5DfVz4DzgZwAR8aSkKcAleQazTd+QcXdXbbOkZ9UmZlYDWcaG6p3ewV1qbR5hzMysmLIUi5ck7UL6LApJx5LcxW1mZl1EltNQZwKTgI9LegF4Djgp11RmZlYoWa6GWgx8Nn2uxGYRsSr/WGZmViRZroY6W1Jf4A3gckmPSToy/2hmZlYUWfos/jUiXgOOBLYFTgUm5prKzMwKJUuxUPrzaJIn1j1RsszMzLqALMVijqSZJMVihqQ+wLv5xjIzsyLJcjXUacAwYHFEvCGpjuRUlJmZdRFZroZ6V9IQ4CRJATwUEdPyDmZmZsWR5Wqoq4GvAX8B5gGnS7oq72BmZlYcWU5DfQbYMyJa7uC+nqRwVCVpBPBjoBtwbURMLFvfA7gB2I9kGPQvR8SSdN3eJONR9SXpI/lkRKzJ8r5mZtaxsnRwPwPsVDK/I/BktY3SR7JeBYwEdgdOkLR7WbPTgFciYlfgcuD76babAzcBX4uIPYBDgHcyZDUzsxy0eWQh6dck40H1A56S1DKY4HDg4Qz7Hg4sSu8AR9ItwGhgQUmb0cCF6fRU4CeSRHJPx5PpZbpExIqsv5CZmXW8SqehfriR+x4ELC2ZbwI+1VabiFgraSVQB3wUCEkzgIHALRHxn+VvIGksMBZgp512Kl9tZmYdpM1iEREPtkxL2g74ZDo7KyKWZ9h3azfuRcY2mwMHpe/5BvB7SXMi4vdlGSeRDHJIfX19+b7NzKyDZLka6nhgFnAccDzwaDpMeTVNJP0bLQYDy9pqk/ZT9ANeTpc/GBEvRcQbQAOwb4b3NDOzHGTp4P53kiuRTo6Ir5L0RXwnw3azgd0kDZW0BTAGmF7WZjpwcjp9LHBfetXVDGBvSb3TIvIZPtjXYWZmNZTl0tnNyk47rSBDkUn7IM4i+R9/N2ByRMyXdBHQGBHTgeuAGyUtIjmiGJNu+4qky0gKTgANEVH9GZxmZpaLLMXit2lH883p/JdJTgtVFREN5W0jYnzJ9BqS01utbXsTyeWzZmbWybIM93GepC+SdDgLmOThPszMupYsRxZExJ3AnTlnMTOzgsrSwW1mZl2ci4WZmVXVZrGQ9Pv05/drF8fMzIqoUp/F9pI+A4xKx3X6wN3WEfFYrsnMzKwwKhWL8cA4kjuvLytbF8BheYUyM7NiqTQ21FRgqqTvRMTFNcxkZmYFk+U+i4sljQIOThc9EBG/yTeWmZkVSZaBBL8HnE0yNtMC4Ox0mZmZdRFZbsr7HDAsIt6F9x6r+jhwQZ7BzMysOLLeZ9G/ZLpfHkHMzKy4shxZfA94XNL9JJfPHoyPKszMupQsHdw3S3qA5Kl1As6PiL/nHczMzIoj60CCL7L+g4vMzKyL8NhQZmZWlYuFmZlVVbFYSNpM0rxahTEzs2KqWCzSeyuekLRTjfKYmVkBZeng3h6YL2kW8HrLwogYlVsqMzMrlCzFYkLuKczMrNCqdnBHxIPAEqB7Oj0byPQsC0kjJD0jaZGkca2s7yHp1nT9o5KGlK3fSdJqSedmeT8zM8tHloEE/zcwFfhZumgQcFeG7boBVwEjgd2BEyTtXtbsNOCViNgVuBwofyrf5cA91d7LzMzyleXS2TOBA4HXACJiIbBthu2GA4siYnFEvA3cAowuazMauD6dngocLkkAkr4ALAbmZ3gvMzPLUZZi8Vb6P3sAJG1O8qS8agYBS0vmm9JlrbaJiLXASqBO0pbA+bi/xMysELIUiwcl/RvQS9IRwO3ArzNsp1aWlReZttpMAC6PiNUV30AaK6lRUmNzc3OGSGZm1h5ZisU4oBn4C3A60AB8O8N2TcCOJfODgWVttUmPWPoBLwOfAv5T0hLgG8C/STqr/A0iYlJE1EdE/cCBAzNEMjOz9sgy6uy76QOPHiX5V/8zEZHlNNRsYDdJQ4EXgDHAiWVtpgMnA38GjgXuS/f9zy0NJF0IrI6In2R4TzMzy0HVYiHpc8BPgWdJThsNlXR6RFS8Siki1qZHAzOAbsDkiJgv6SKgMSKmA9cBN0paRHJEMWbjfh0zM8tDlpvyfgQcGhGLACTtAtxNhktaI6KB5LRV6bLxJdNrgOOq7OPCDBnNzCxHWfoslrcUitRiYHlOeczMrIDaPLKQ9MV0cr6kBuA2kj6L40j6I8zMrIuodBrq8yXT/wA+k043A1vnlsjMzAqnzWIREafWMoiZmRVXlquhhgL/FxhS2t5DlJuZdR1Zroa6i+QS118D7+Ybx8zMiihLsVgTEVfmnsTMzAorS7H4saTvAjOBt1oWRkSmZ1qYmdmHX5ZisRfwFeAw3j8NFem8mZl1AVmKxTHAzqXDlJuZWdeS5Q7uJ4D+eQcxM7PiynJksR3wtKTZfLDPwpfOmpl1EVmKxXdzT2FmZoWW5XkWD9YiiJmZFVeWO7hX8f7jULcAugOvR0TfPIOZmVlxZDmy6FM6L+kLwPDcEpmZWeFkuRrqAyLiLnyPhZlZl5LlNNQXS2Y3A+p5/7SUmZl1AVmuhip9rsVaYAkwOpc0ZmZWSFn6LPxcCzOzLq7SY1XHV9guIuLiHPKYmVkBVergfr2VF8BpwPlZdi5phKRnJC2SNK6V9T0k3Zquf1TSkHT5EZLmSPpL+tMd6mZmnajSY1V/1DItqQ9wNnAqcAvwo7a2K9mmG3AVcATQBMyWND0iFpQ0Ow14JSJ2lTQG+D7wZeAl4PMRsUzSnsAMYNCG/nJmZtYxKl46K2kbSZcAT5IUln0j4vyIWJ5h38OBRRGxOB2x9hbW7xgfDVyfTk8FDpekiHg8Ipaly+cDPSX1yPg7mZlZB2uzWEj6ATAbWAXsFREXRsQrG7DvQcDSkvkm1j86eK9NRKwFVgJ1ZW2+BDweEW9hZmadotKRxbeAHYBvA8skvZa+Vkl6LcO+1cqy8vszKraRtAfJqanTW30DaaykRkmNzc3NGSKZmVl7tFksImKziOgVEX0iom/Jq0/GcaGagB1L5gcDy9pqI2lzoB/wcjo/GJgGfDUinm0j46SIqI+I+oEDB2aIZGZm7bHBw31sgNnAbpKGStoCGANML2szHTg5nT4WuC8iQlJ/4G7ggoj4U44Zzcwsg9yKRdoHcRbJlUxPAbdFxHxJF0lqeXDSdUCdpEXAOUDL5bVnAbsC35E0N31tm1dWMzOrLMtwH+0WEQ1AQ9my8SXTa4DjWtnuEuCSPLOZmVl2eZ6GMjOzTYSLhZmZVeViYWZmVblYmJlZVS4WZmZWlYuFmZlV5WJhZmZVuViYmVlVLhZmZlaVi4WZmVXlYmFmZlW5WJiZWVUuFmZmVpWLhZmZVeViYWZmVblYmJlZVS4WZmZWlYuFmZlV5WJhZmZVuViYmVlVLhZmZlZVrsVC0ghJz0haJGlcK+t7SLo1Xf+opCEl6y5Ilz8j6ag8c5qZWWW5FQtJ3YCrgJHA7sAJknYva3Ya8EpE7ApcDnw/3XZ3YAywBzACuDrdn5mZdYI8jyyGA4siYnFEvA3cAowuazMauD6dngocLknp8lsi4q2IeA5YlO7PzMw6QZ7FYhCwtGS+KV3WapuIWAusBOoybmtmZjWyeY77VivLImObLNsiaSwwNp1dLemZDUq4fpgBwEsVG01oLVrHqpqjCBmKkqOLZChKjiJkKEqOImTooBwfyfI+eRaLJmDHkvnBwLI22jRJ2hzoB7yccVsiYhIwqaMCS2qMiPqO2t+HOUcRMhQlRxEyFCVHETIUJUcRMtQyR56noWYDu0kaKmkLkg7r6WVtpgMnp9PHAvdFRKTLx6RXSw0FdgNm5ZjVzMwqyO3IIiLWSjoLmAF0AyZHxHxJFwGNETEduA64UdIikiOKMem28yXdBiwA1gJnRsS6vLKamVlleZ6GIiIagIayZeNLptcAx7Wx7aXApXnma0WHndLaSEXIUYQMUIwcRcgAxchRhAxQjBxFyAA1yqHkrI+ZmVnbPNyHmZlV1SWLxcYMQ9KBGSZLWi5pXhvrJenKNMOTkvbNIcOOku6X9JSk+ZLOrnUOST0lzZL0RJphQittcv97lLxXN0mPS/pNZ+SQtETSXyTNldTYyvrcvxfp+/SXNFXS0+n344Ba5pD0sfQzaHm9JukbtcxQ8j7fTL+b8yTdLKln2fpafC/OTt9/fvnnkK7P/7OIiC71IulsfxbYGdgCeALYvazNGcBP0+kxwK055DgY2BeY18b6o4F7SO452R94NIcM2wP7ptN9gL+28lnkmiPd71bpdHfgUWD/Wv89St7rHGAK8JtW1tXie7EEGFBhfe7fi/R9rgf+Vzq9BdC/M3Kk79UN+DvwkVpnILkZ+DmgVzp/G3BKLb8XwJ7APKA3ST/zvcButf4suuKRxcYMQ9JhIuIPJFeAtWU0cEMkHgH6S9q+gzO8GBGPpdOrgKdY/075XHOk+12dznZPX+Udabn/PQAkDQY+B1zbRpOa5Kgi9++FpL4k/5i5DiAi3o6IV2udo8ThwLMR8bdOyrA50EvJvWC9Wf+er7y/F58AHomINyIZ6eJB4JhWMuT6WXTFYrExw5DUUk2HPEkPnfch+Zd9TXOkp37mAsuB30VEmxly/ntcAfw/4N021tciRwAzJc1RMkJBmxlSeXwvdgaagV+kp+SulbRlJ+RoMQa4uZXluWeIiBeAHwLPAy8CKyNiZls5cvpezAMOllQnqTfJUcSOZW1y/yy6YrHYmGFIaqlmGSRtBdwBfCMiXqt1johYFxHDSO7UHy5pz1pnkPQ/geURMadSs7xzAAdGxL4kozWfKengTsiwOckp0msiYh/gdaC8b68m308lN/SOAm5vbXXeGSRtTfKv9qHADsCWkk6qZY6IeIpkRO7fAb8lOXW+tpYZoGsWiw0ZhgR9cBiSWso05MnGktSdpFD8d0Tc2Vk5ANJTHQ+QDEvfaoYc/x4HAqMkLSE5NXmYpJtqnSMilqU/lwPTWH+05Vr8PZqAppIjvKkkxaPWOSApmo9FxD9aWVeLDJ8FnouI5oh4B7gT+HRbOXL8XlwXEftGxMHpvhe2lSHV4Z9FVywWGzMMSS1NB76aXuWwP8nh74sd+QbpedXrgKci4rLOyCFpoKT+6XQvkv84n24lQ65/j4i4ICIGR8QQku/EfRFR/i/IXHNI2lJSn5Zp4EiSUxDlGXL9XkTE34Glkj6WLjqcZDSFmuZInUDrp6BqleF5YH9JvdP/Xg4n6dsrz5Hr91PStunPnYAvsv5nkv9n0dE95h+GF8k5v7+SXBX17+myi4BR6XRPksPeRSRjUu2cQ4abSc6BvkPyr4LTgK8BX0vXi+ThUc8CfwHqc8hwEMmh6pPA3PR1dC1zAHsDj6cZ5gHjO+PvUZbpENKroWqZg6Sv4In0Nb/ku1nT70X6PsOAxvTvchewdSd8P3sDK4B+Jcs647OYQPIPmHlLfXi1AAABlklEQVTAjUCPTvj/xR9JCvYTwOGd8Vn4Dm4zM6uqK56GMjOzDeRiYWZmVblYmJlZVS4WZmZWlYuFmZlV5WJhZmZVuViYmVlVLhZmHUDSXengf/NbBgCUdJqkv0p6QNLPJf0kXT5Q0h2SZqevAzs3vVl1vinPrANI2iYiXk6HLJkNHAX8iWRMpVXAfcATEXGWpCnA1RHxUDp8w4yI+ESnhTfLYPPODmC2ifi6pJZnDOwIfAV4MCJeBpB0O/DRdP1ngd1LHnnQV1KfSJ4pYlZILhZmG0nSISQF4ICIeEPSA8AzJA+tac1mads3a5PQbOO5z8Js4/UDXkkLxcdJHmvZG/iMpK3TYau/VNJ+JnBWy4ykYTVNa9YOLhZmG++3wOaSngQuBh4BXgD+g+TJg/eSjBi6Mm3/daBe0pOSFpCMHmpWaO7gNsuJpK0iYnV6ZDENmBwR0zo7l1l7+MjCLD8Xps8Wnwc8R/JcCLMPJR9ZmJlZVT6yMDOzqlwszMysKhcLMzOrysXCzMyqcrEwM7OqXCzMzKyq/w+vyko482xHJgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "t1 = train_t.groupby(['age'])['age'].count() / len(train_t)\n",
    "t2 = test_t.groupby(['age'])['age'].count() / len(test_t)\n",
    "\n",
    "tmp = pd.concat([t1, t2], axis=1)\n",
    "tmp.columns = ['train', 'test']\n",
    "tmp.plot.bar()\n",
    "plt.xticks(rotation=0)\n",
    "plt.ylabel('Number of observations per bin')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'Number of observations per bin')"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xuc1XW97/HXW0RuItiAHgSJCal9vKVEhOkxL6mgJ9FSI/Okbs/Bdmp00ROcjMTLCbPM3JnGTtqmIRppTYmJbi9dNLkoKqAGIsmIOxAVRUUFPvuP3290tVgz68cwv7UWs97Px2M95ndf75lZ8Jnf7/v9fX+KCMzMzNqyQ7UDmJlZ7XOxMDOzslwszMysLBcLMzMry8XCzMzKcrEwM7OyXCzMzKwsFwszMyvLxcLMzMrasdoBOkq/fv1iyJAh1Y5hZrZdWbBgwYsR0b/cdp2mWAwZMoT58+dXO4aZ2XZF0t+ybOfLUGZmVpaLhZmZleViYWZmZXWaNotS3nnnHZqbm9mwYUO1o+Sue/fuDBo0iK5du1Y7ipl1Qp26WDQ3N9O7d2+GDBmCpGrHyU1EsHbtWpqbm2lsbKx2HDPrhDr1ZagNGzbQ0NDQqQsFgCQaGhrq4gzKzKqjUxcLoNMXihb18n2aWXV0+mJhZmbbrlO3WRQbMvGODj3eiqnHld3mlVdeYcaMGXzpS1/aqmMfe+yxzJgxg759+7Y3nplZh8m1WEgaDfwQ6AL8NCKmFq0/FLgK2B8YFxGz0uUHANcCuwCbgMsi4pY8s+bllVde4cc//vEWxWLTpk106dKl1f1mz56ddzRLlfsjIssfBWadXW7FQlIX4BrgKKAZmCepKSKWFGz2HHAGcH7R7m8AX4iIpZL2ABZIuisiXskrb14mTpzIM888wwEHHEDXrl3ZeeedGTBgAAsXLmTJkiWccMIJrFy5kg0bNjBhwgTGjx8PvDd8yfr16xkzZgyHHHIIDz74IAMHDuQ3v/kNPXr0qPJ3Zmb1JM82i5HAsohYHhFvAzOBsYUbRMSKiHgc2Fy0/K8RsTSdXgWsBsoOdFWLpk6dytChQ1m4cCFXXHEFc+fO5bLLLmPJkqRmTp8+nQULFjB//nyuvvpq1q5du8Uxli5dyjnnnMPixYvp27cvv/rVryr9bZhZncuzWAwEVhbMN6fLtoqkkcBOwDMdlKuqRo4c+Q/3Qlx99dV8+MMfZtSoUaxcuZKlS5dusU9jYyMHHHAAAB/5yEdYsWJFpeKamQH5tlmU6ssZW3UAaQBwI3B6RGwusX48MB5g8ODB7clYcb169Xp3+v777+eee+7hoYceomfPnhx22GEl75Xo1q3bu9NdunThzTffrEjWmnBRnzLr11Umh1mdy/PMohnYs2B+ELAq686SdgHuAC6MiL+U2iYipkXEiIgY0b9/bV6l6t27N6+99lrJdevWrWPXXXelZ8+ePPXUU/zlLyW/TTOzqsvzzGIeMExSI/A8MA44NcuOknYCbgd+HhG/7KhA1ejV0tDQwMEHH8y+++5Ljx492H333d9dN3r0aK677jr2339/PvShDzFq1KiK5zMzyyK3YhERGyWdC9xF0nV2ekQslnQxMD8imiR9lKQo7Ap8StKUiNgHOAU4FGiQdEZ6yDMiYmFeefM0Y8aMksu7devGnXfeWXJdS7tEv379WLRo0bvLzz+/uOOYmVn+cr3PIiJmA7OLlk0umJ5HcnmqeL+bgJvyzGZmZtnV1R3cZmYdps46X3hsKDMzK8tnFmZWU7KM4eYhWCrPZxZmZlaWzyzMbPtTZ+0FtaC+ikW5D9hWH6/8B7K9Q5QDXHXVVYwfP56ePXu2J52ZWYfxZaictQxR3h5XXXUVb7zxRgcnMjPbevV1ZlEFhUOUH3XUUey2227ceuutvPXWW5x44olMmTKF119/nVNOOYXm5mY2bdrEt771Lf7+97+zatUqDj/8cPr168d9991X7W/FzOqYi0XOpk6dyqJFi1i4cCFz5sxh1qxZzJ07l4jg+OOP5w9/+ANr1qxhjz324I47kl4g69ato0+fPlx55ZXcd9999OvXr8rfhZnVO1+GqqA5c+YwZ84cDjzwQIYPH85TTz3F0qVL2W+//bjnnnv4xje+wR//+Ef69OngthUzs23kM4sKiggmTZrE2WefvcW6BQsWMHv2bCZNmsTRRx/N5MmTSxzBzKw6fGaRs8Ihyo855himT5/O+vXrAXj++edZvXo1q1atomfPnpx22mmcf/75PPLII1vsa2ZWTZnOLCR9HBhSuH1E/DynTPmpQt/rwiHKx4wZw6mnnspBBx0EwM4778xNN93EsmXLuOCCC9hhhx3o2rUr1157LQDjx49nzJgxDBgwwA3cZlZVZYuFpBuBocBCYFO6OIDtr1hUSfEQ5RMmTPiH+aFDh3LMMcdssd95553Heeedl2s2M7MsspxZjAD2joiteiSqmZl1HlnaLBYB/y3vIGZmVruynFn0A5ZImgu81bIwIo7PLVUHiggkVTtG7nziZ2Z5ylIsLso7RF66d+/O2rVraWho6NQFIyJYu3Yt3bt3r3YUM+ukyhaLiHigEkHyMGjQIJqbm1mzZk21o+Sue/fuDBq0xRNqzcw6RKvFQtKfIuIQSa+R9H56dxUQEbFL7um2UdeuXWlsbKx2DDOz7V6rxSIiDkm/9q5cHDMzq0VZb8obDhxCcobxp4h4NNdUZrXED9oxK991VtJk4AaggaRn1L9LujDvYGZmVjuynFl8DjgwIjYASJoKPAJcmmcwMzOrHVluylsBFPbJ7AY8k0saMzOrSW31hvpXkjaKt4DFku5O548C/pTl4JJGAz8EugA/jYipResPBa4C9gfGRcSsgnWnAy2Xuy6NiBuyflNmnZLbTqyK2roMNT/9ugC4vWD5/VkOLKkLcA1JcWkG5klqioglBZs9B5wBnF+07/uAb5OMSxXAgnTfl7O8t5mZday2us5u61/yI4FlEbEcQNJMYCzwbrGIiBXpus1F+x4D3B0RL6Xr7wZGAzdvYyYzM2uHPJ+UNxBYWTDfDHxsG/YdWLyRpPHAeIDBgwe3L6VZDRgy8Y6y26zwaC4VVe53UonfR6bPxdTj8g9Cvk/KKzUYU9bR7jLtGxHTImJERIzo37//VoUzM7Ps2jyzSNsdpkbEBe04djOwZ8H8IGDVVux7WNG+97cjg7VXucZUcIOqWR1p88wiIjYBH1H7hmydBwyT1ChpJ2Ac0JRx37uAoyXtKmlX4Oh0mZmZVUGWNotHgd9I+iXwesvCiLitrZ0iYqOkc0n+k+8CTI+IxZIuBuZHRJOkj5L0tNoV+JSkKRGxT0S8JOkSkoIDcHFLY7eZmVVelmLxPmAtcETBsgDaLBYAETEbmF20bHLB9DySS0yl9p0OTM+Qz8zMcpbleRZnViKImW0HfGNg3SpbLCR9ELgW2D0i9pW0P3B8RGx3Y0OV7QpXoS5oZmbbmyxdZ/8NmAS8AxARj5M0VpuZWZ3I0mbRMyLmFnWI2phTnk6vlm6yMTPLKsuZxYuShpLeFCfpJOCFXFOZmVlNyXJmcQ4wDfgnSc8DzwKfzzWV1QUPcWG2/cjSG2o58ElJvYAdIuK1/GOZmVktyfJY1QZJVwN/BO6X9ENJDflHMzOzWpHlMtRM4A/AZ9L5zwO3AJ/MK5TlrxZG1DSz7UemO7gj4pKC+UslnZBXIDMzqz1ZekPdJ2mcpB3S1ylA+ZZJMzPrNLIUi7OBGSTP4n6L5LLU1yS9JunVPMOZmVltyNIbqnclgpiZWe3K87Gq2x8/8MfMrKQ8H6tqZmadhIuFmZmVleWmvKGSuqXTh0n6sqS++UczM7NakaXN4lfACEl7AdeTPEd7BnBsnsHMzCyDCj2QKstlqM0RsRE4EbgqIr4KDOiQdzczs+1ClmLxjqTPAacDv0uXdc0vkpmZ1ZosxeJM4CDgsoh4VlIjcFO+sczMrJZkuSlvCfDlgvlngal5hjKz6vAAk9aassVC0sHARcD70+0FRER8IN9oZmZWK7L0hroe+CqwANiUbxwzM6tFWYrFuoi4M/ckZmZWs7IOUX6FpIMkDW95ZTm4pNGSnpa0TNLEEuu7SbolXf+wpCHp8q6SbpD0hKQnJU3aqu/KzMw6VJYzi4+lX0cULAvgiLZ2ktQFuAY4CmgG5klqShvMW5wFvBwRe0kaB1wOfBY4GegWEftJ6gkskXRzRKzI8k2ZmVnHytIb6vB2HnsksCwilgNImgmMBQqLxViSxnOAWcCPJImkGPWStCPQA3gb8LMzzMyqJMvYUH0kXSlpfvr6vqQMY3kzEFhZMN+cLiu5TXqX+DqggaRwvA68ADwHfC8iXiqRbXxLrjVr1mSIZGZm7ZGlzWI68BpwSvp6FfhZhv1UYllk3GYkSc+rPYBG4OuStuiqGxHTImJERIzo379/hkhmZtYeWdoshkbEZwrmp0hamGG/ZmDPgvlBwKpWtmlOLzn1AV4CTgV+HxHvAKsl/ZmkzWR5hvc1M7MOluXM4k1Jh7TMpDfpvZlhv3nAMEmNknYCxpGMWFuoiWTMKYCTgHsjIkguPR2hRC9gFPBUhvc0M7McZDmz+BfghrSdQiR/+Z9RbqeI2CjpXOAuoAswPSIWS7oYmB8RTSQ3/N0oaVl63HHp7teQXOpalL7nzyLi8a36zszMrMNk6Q21EPiwpF3S+cy9kiJiNjC7aNnkgukNJN1ki/dbX2q5mZlVR6vFQtJpEXGTpK8VLQcgIq7MOZuZmdWIts4seqVfe5dYV9yryczMOrFWi0VE/CSdvCci/ly4Lm3kNjOzOpGlN9S/ZlxmZmadVFttFgcBHwf6F7Vb7ELSu8nMzOpEW20WOwE7p9sUtlu8SnJPhJmZ1Ym22iweAB6Q9O8R8bcKZjIzsxqT5aa8NyRdAewDvPsE3ohoc4hyMzPrPLI0cP+CZKiNRmAKsIJkKA8zM6sTWc4sGiLiekkTCi5NPZB3sLp2UZkR4C9aV5kcZmapLMXinfTrC5KOIxk5dlB+kczMrNZkKRaXpoMIfp3k/opdgK/mmsrMzGpKlmLxcESsI3mKXXsfsWpmZtuxLA3cD0qaI+ksSbvmnsjMzGpO2WIREcOAC0m6zi6Q9DtJp+WezMzMakaWMwsiYm5EfI3k2dgvATfkmsrMzGpK2WIhaRdJp0u6E3gQeIGkaJiZWZ3I0sD9GPBr4OKIeCjnPGZmVoPaLBaSugC3p5egzMysTrV5GSoiNgEfrlAWMzOrUVkuQy2U1AT8Eni9ZWFE3JZbKjMzqylZisX7gLVA4SizAbhYmJnVibLFIiLOrEQQMzOrXVm6zn5Q0n9IWpTO7y/pwvyjmZlZrchyU96/AZNIR5+NiMeBcXmGMjOz2pKlWPSMiLlFyzZmObik0ZKelrRM0sQS67tJuiVd/7CkIQXr9pf0kKTFkp6Q1L14fzMzq4wsxeJFSUNJGrWRdBLJXdxtSu/RuAYYA+wNfE7S3kWbnQW8HBF7AT8ALk/33RG4CfhiROwDHMZ7z9UwM7MKy1IszgF+AvyTpOeBrwD/kmG/kcCyiFgeEW8DM4GxRduM5b1xpmYBR0oScDTweEQ8BhARa9N7PszMrAqy9IZaDnxSUi9gh4h4LeOxBwIrC+abgY+1tk1EbJS0DmgAPgiEpLuA/sDMiPhu8RtIGg+MBxg8eHDGWGZmtrWy9IaaIGkX4A3gB5IekXR0hmOrxLLIuM2OwCHA59OvJ0o6cosNI6ZFxIiIGNG/f/8MkczMrD2yXIb654h4leTS0G7AmcDUDPs1A3sWzA8ieX53yW3Sdoo+JEOgNwMPRMSLEfEGMBsYnuE9zcwsB1mKRctf/8cCP0vbEUqdERSbBwyT1ChpJ5Lutk1F2zQBp6fTJwH3RkQAdwH7S+qZFpFPAEsyvKeZmeUgy3AfCyTNARqBSZJ6A5vL7ZS2QZxL8h9/F2B6RCyWdDEwPyKagOuBGyUtIzmjGJfu+7KkK0kKTgCzI+KOdnx/ZmbWAbIUi7OAA4DlEfGGpAaSS1FlRcRskktIhcsmF0xvAE5uZd+bSLrPmplZlWXpDbU5vVnuNEkB/Ckibs87mJmZ1Y4svaF+DHwReAJYBJwt6Zq8g5mZWe3IchnqE8C+acMzkm4gKRxmZlYnsvSGehoovONtT+DxfOKYmVktavXMQtJvSXoi9QGelNQymOBI4MEKZDMzsxrR1mWo71UshZmZ1bRWi0VEPNAyLWl34KPp7NyIWJ13MDMzqx1ZekOdAswluR/iFODhdJhyMzOrE1l6Q30T+GjL2YSk/sA9JEOKm5lZHcjSG2qHostOazPuZ2ZmnUSWM4vfp8+VuDmd/yxFQ3iYmVnnlmW4jwskfZrkuRICpnm4DzOz+pLlzIKIuA24LecsZmZWo9z2YGZmZblYmJlZWa0WC0n/kX69vHJxzMysFrXVZjFA0ieA4yXNpOhRqhHxSK7JzMysZrRVLCYDE4FBwJVF6wI4Iq9QZmZWW9oaG2oWMEvStyLikgpmMjOzGpPlPotLJB0PHJouuj8ifpdvLDMzqyVZBhL8DjABWJK+JqTLzMysTmS5Ke844ICI2AzvPlb1UWBSnsHMzKx2ZL3Pom/BdJ88gpiZWe3KcmbxHeBRSfeRdJ89FJ9VmJnVlbJnFhFxMzCKZGyo24CDImJmloNLGi3paUnLJE0ssb6bpFvS9Q9LGlK0frCk9ZLOz/J+ZmaWj6wDCb4ANG3NgSV1Aa4BjgKagXmSmiJiScFmZwEvR8ReksYBl5MMgd7iB8CdW/O+ZmbW8fIcG2oksCwilkfE28BMYGzRNmOBG9LpWcCRkgQg6QRgObA4x4xmZpZBnsViILCyYL45XVZym4jYCKwDGiT1Ar4BTMkxn5mZZdRmsZC0g6RF7Ty2SiyLjNtMAX4QEevL5Bsvab6k+WvWrGlnTDMzK6fNNouI2CzpMUmDI+K5rTx2M7BnwfwgYFUr2zRL2pGkW+5LwMeAkyR9l6Tb7mZJGyLiR0X5pgHTAEaMGFFciMzMrINkaeAeACyWNBd4vWVhRBxfZr95wDBJjcDzwDjg1KJtmoDTgYeAk4B7IyKA/9GygaSLgPXFhcLMzConS7FoV7tBRGyUdC5wF9AFmB4RiyVdDMyPiCbgeuBGSctIzijGtee9zMwsX1kGEnxA0vuBYRFxj6SeJP/5lxURs4HZRcsmF0xvAE4uc4yLsryXmZnlJ8tAgv+HpFvrT9JFA4Ff5xnKzMxqS5aus+cABwOvAkTEUmC3PEOZmVltyVIs3kpvqgMg7bXknkdmZnUkS7F4QNL/A3pIOgr4JfDbfGOZmVktyVIsJgJrgCeAs0karC/MM5SZmdWWLL2hNqcPPHqY5PLT0+m9EGZmVifKFgtJxwHXAc+QDM/RKOnsiPBosGZmdSLLTXnfBw6PiGUAkoYCd+Chw83M6kaWNovVLYUitRxYnVMeMzOrQa2eWUj6dDq5WNJs4FaSNouTScZ9MjOzOtHWZahPFUz/HfhEOr0G2DW3RGZmVnNaLRYRcWYlg5iZWe3K0huqETgPGFK4fYYhys3MrJPI0hvq1yRDif8W2JxvHDMzq0VZisWGiLg69yRmZlazshSLH0r6NjAHeKtlYUQ8klsqMzOrKVmKxX7A/wKO4L3LUJHOm5lZHchSLE4EPlA4TLmZmdWXLHdwPwb0zTuImZnVrixnFrsDT0maxz+2WbjrrJlZnchSLL6dewozM6tpWZ5n8UAlgpiZWe3Kcgf3a7z3zO2dgK7A6xGxS57BzMysdmQ5s+hdOC/pBGBkbonMzKzmZOkN9Q8i4tf4Hgszs7qS5TLUpwtmdwBG8N5lqXL7jgZ+CHQBfhoRU4vWdwN+DnwEWAt8NiJWSDoKmEpy2ett4IKIuDfLe5qZWcfL0huq8LkWG4EVwNhyO0nqAlwDHAU0A/MkNUXEkoLNzgJejoi9JI0DLgc+C7wIfCoiVknaF7gLGJghq5mZ5SBLm0V7n2sxElgWEcsBJM0kKTKFxWIscFE6PQv4kSRFxKMF2ywGukvqFhFvYWZmFdfWY1Unt7FfRMQlZY49EFhZMN8MfKy1bSJio6R1QAPJmUWLzwCPulCYmVVPW2cWr5dY1ovk0lEDUK5YqMSy4raONreRtA/JpamjS76BNB4YDzB48OAycczMrL3aeqzq91umJfUGJgBnAjOB77e2X4FmYM+C+UHAqla2aZa0I9AHeCl9z0HA7cAXIuKZVjJOA6YBjBgxIlOju5mZbb02u85Kep+kS4HHSQrL8Ij4RkSsznDsecAwSY2SdgLGAU1F2zQBp6fTJwH3RkRI6gvcAUyKiD9vxfdjZmY5aLVYSLqC5D/814D9IuKiiHg564EjYiNwLklPpieBWyNisaSLJbUMQng90CBpGfA1YGK6/FxgL+Bbkhamr9229pszM7OO0VabxddJRpm9EPim9G7zgkgauMsO9xERs4HZRcsmF0xvAE4usd+lwKXljm9mZpXRVpvFVt/dbWZmnZMLgpmZleViYWZmZblYmJlZWS4WZmZWlouFmZmV5WJhZmZluViYmVlZLhZmZlaWi4WZmZXlYmFmZmW5WJiZWVkuFmZmVpaLhZmZleViYWZmZblYmJlZWS4WZmZWlouFmZmV5WJhZmZluViYmVlZLhZmZlaWi4WZmZXlYmFmZmW5WJiZWVkuFmZmVlauxULSaElPS1omaWKJ9d0k3ZKuf1jSkIJ1k9LlT0s6Js+cZmbWttyKhaQuwDXAGGBv4HOS9i7a7Czg5YjYC/gBcHm6797AOGAfYDTw4/R4ZmZWBXmeWYwElkXE8oh4G5gJjC3aZixwQzo9CzhSktLlMyPirYh4FliWHs/MzKogz2IxEFhZMN+cLiu5TURsBNYBDRn3NTOzCtkxx2OrxLLIuE2WfZE0Hhifzq6X9PRWJdwyTD/gxTY3mlIqWscqm6MWMtRKjjrJUCs5aiFDreSohQwdlOP9Wd4nz2LRDOxZMD8IWNXKNs2SdgT6AC9l3JeImAZM66jAkuZHxIiOOt72nKMWMtRKjlrIUCs5aiFDreSohQyVzJHnZah5wDBJjZJ2Immwbirapgk4PZ0+Cbg3IiJdPi7tLdUIDAPm5pjVzMzakNuZRURslHQucBfQBZgeEYslXQzMj4gm4HrgRknLSM4oxqX7LpZ0K7AE2AicExGb8spqZmZty/MyFBExG5hdtGxywfQG4ORW9r0MuCzPfCV02CWtbVQLOWohA9RGjlrIALWRoxYyQG3kqIUMUKEcSq76mJmZtc7DfZiZWVl1WSy2ZRiSDswwXdJqSYtaWS9JV6cZHpc0PIcMe0q6T9KTkhZLmlDpHJK6S5or6bE0w5QS2+T++yh4ry6SHpX0u2rkkLRC0hOSFkqaX2J97p+L9H36Spol6an083FQJXNI+lD6M2h5vSrpK5XMUPA+X00/m4sk3Sype9H6SnwuJqTvv7j455Cuz/9nERF19SJpbH8G+ACwE/AYsHfRNl8CrkunxwG35JDjUGA4sKiV9ccCd5LcczIKeDiHDAOA4el0b+CvJX4WueZIj7tzOt0VeBgYVenfR8F7fQ2YAfyuxLpKfC5WAP3aWJ/75yJ9nxuA/51O7wT0rUaO9L26AP8JvL/SGUhuBn4W6JHO3wqcUcnPBbAvsAjoSdLOfA8wrNI/i3o8s9iWYUg6TET8gaQHWGvGAj+PxF+AvpIGdHCGFyLikXT6NeBJtrxTPtcc6XHXp7Nd01dxQ1ruvw8ASYOA44CftrJJRXKUkfvnQtIuJH/MXA8QEW9HxCuVzlHgSOCZiPhblTLsCPRQci9YT7a85yvvz8V/B/4SEW9EMtLFA8CJJTLk+rOox2KxLcOQVFJFhzxJT50PJPnLvqI50ks/C4HVwN0R0WqGnH8fVwH/F9jcyvpK5AhgjqQFSkYoaDVDKo/PxQeANcDP0ktyP5XUqwo5WowDbi6xPPcMEfE88D3gOeAFYF1EzGktR06fi0XAoZIaJPUkOYvYs2ib3H8W9VgstmUYkkqqWAZJOwO/Ar4SEa9WOkdEbIqIA0ju1B8pad9KZ5D0P4HVEbGgrc3yzgEcHBHDSUZrPkfSoVXIsCPJJdJrI+JA4HWguG2vIp9PJTf0Hg/8stTqvDNI2pXkr/ZGYA+gl6TTKpkjIp4kGZH7buD3JJfON1YyA9RnsdiaYUjQPw5DUkmZhjzZVpK6khSKX0TEbdXKAZBe6rifZFj6khly/H0cDBwvaQXJpckjJN1U6RwRsSr9uhq4nS1HW67E76MZaC44w5tFUjwqnQOSovlIRPy9xLpKZPgk8GxErImId4DbgI+3liPHz8X1ETE8Ig5Nj720tQypDv9Z1GOx2JZhSCqpCfhC2sthFMnp7wsd+QbpddXrgScj4spq5JDUX1LfdLoHyT/Op0pkyPX3ERGTImJQRAwh+UzcGxHFf0HmmkNSL0m9W6aBo0kuQRRnyPVzERH/CayU9KF00ZEkoylUNEfqc5S+BFWpDM8BoyT1TP+9HEnStlecI9fPp6Td0q+DgU+z5c8k/59FR7eYbw8vkmt+fyXpFfXNdNnFwPHpdHeS095lJGNSfSCHDDeTXAN9h+SvgrOALwJfTNeL5OFRzwBPACNyyHAIyanq48DC9HVsJXMA+wOPphkWAZOr8fsoynQYaW+oSuYgaSt4LH0tLvhsVvRzkb7PAcD89Pfya2DXKnw+ewJrgT4Fy6rxs5hC8gfMIuBGoFsV/r/4I0nBfgw4sho/C9/BbWZmZdXjZSgzM9tKLhZmZlaWi4WZmZXlYmFmZmW5WJiZWVkuFmbbSNKXlYzM+otqZzHLi7vOmm0jSU8BYyLi2Qzb7hjJ+EFm25VcH6tq1tlJuo7kZrqmdHiQsUAP4E3gzIh4WtIZJKPZdgd6kQwlcgFwCskNXrdHxLerkd8sKxcLs20QEV+UNBo4HHgb+H5EbJT0SeD/A59JNz0I2D8iXpJ0NDCMZNwnkRSaQyMZtt6sJrlYmHWcPsANkoaRDKPStWDd3RHRMrjc0enr0XR+Z5Li4WJhNcvFwqzjXALcFxEnps8Hub9g3esF0wK+ExE/qVw0s23j3lBmHacP8Hw6fUapLwKPAAAAbklEQVQb290F/HP6HBEkDWwZVdSsVrlYmHWc7wLfkfRnkudGlxTJk9ZmAA9JeoLkeRG9KxPRrH3cddbMzMrymYWZmZXlYmFmZmW5WJiZWVkuFmZmVpaLhZmZleViYWZmZblYmJlZWS4WZmZW1n8B6QeD2dbTQ8IAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "t1 = train_t.groupby(['fare'])['fare'].count() / len(train_t)\n",
    "t2 = test_t.groupby(['fare'])['fare'].count() / len(test_t)\n",
    "\n",
    "tmp = pd.concat([t1, t2], axis=1)\n",
    "tmp.columns = ['train', 'test']\n",
    "tmp.plot.bar()\n",
    "plt.xticks(rotation=0)\n",
    "plt.ylabel('Number of observations per bin')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "feml",
   "language": "python",
   "name": "feml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}