WF Interest Rate Predictions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting Loan Interest Rates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 1**:\n",
    "Clean and prepare the data. There are truly missing and simulated missing values. Clean them.\n",
    "\n",
    "**Step 2**:\n",
    "Build a machine learning model in Python to predict the interest rates\n",
    "assigned to loans. Structure your code into two parts: a first part that\n",
    "trains the model on the training data, and a second part runs the model on the\n",
    "training data and the test data.\n",
    "\n",
    "**Step 3**:\n",
    "Submit code for cleaning, prepping and modeling the data, along with a\n",
    "discussion of how you designed and implemented your model. Include a discussion\n",
    "of the performance of the model on the training data, for which you can\n",
    "compare the actual loan interest rate with the rate predicted by your model.\n",
    "Show the RMSE your model achieves on the training data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Step 1: Import, clean and prepare the data:\n",
    "\n",
    "Let's first import the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# import necessary python modules for analysis\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn import preprocessing\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import LinearRegression, RidgeCV\n",
    "from sklearn.metrics import mean_squared_error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# import training_data.csv via pandas module\n",
    "train = pd.read_csv(\"training_data.csv\", low_memory=False)\n",
    "\n",
    "# import test_data.csv via pandas module\n",
    "test = pd.read_csv(\"test_data.csv\", low_memory=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import the metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Variable</th>\n",
       "      <th>Definition</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>X1</td>\n",
       "      <td>Interest Rate on the loan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>X2</td>\n",
       "      <td>A unique id for the loan.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>X3</td>\n",
       "      <td>A unique id assigned for the borrower.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>X4</td>\n",
       "      <td>Loan amount requested</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>X5</td>\n",
       "      <td>Loan amount funded</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>X6</td>\n",
       "      <td>Investor-funded portion of loan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>X7</td>\n",
       "      <td>Number of payments (36 or 60)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>X8</td>\n",
       "      <td>Loan grade</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>X9</td>\n",
       "      <td>Loan subgrade</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>X10</td>\n",
       "      <td>Employer or job title (self-filled)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>X11</td>\n",
       "      <td>Number of years employed (0 to 10; 10 = 10 or ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>X12</td>\n",
       "      <td>Home ownership status: RENT, OWN, MORTGAGE, OT...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>X13</td>\n",
       "      <td>Annual income of borrower</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>X14</td>\n",
       "      <td>Income verified, not verified, or income sourc...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>X15</td>\n",
       "      <td>Date loan was issued</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>X16</td>\n",
       "      <td>Reason for loan provided by borrower</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>X17</td>\n",
       "      <td>Loan category, as provided by borrower</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>X18</td>\n",
       "      <td>Loan title, as provided by borrower</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>X19</td>\n",
       "      <td>First 3 numbers of zip code</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>X20</td>\n",
       "      <td>State of borrower</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>X21</td>\n",
       "      <td>A ratio calculated using the borrower’s total ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>X22</td>\n",
       "      <td>The number of 30+ days past-due incidences of ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>X23</td>\n",
       "      <td>Date the borrower's earliest reported credit l...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>X24</td>\n",
       "      <td>Number of inquiries by creditors during the pa...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>X25</td>\n",
       "      <td>Number of months since the borrower's last del...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>X26</td>\n",
       "      <td>Number of months since the last public record.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>X27</td>\n",
       "      <td>Number of open credit lines in the borrower's ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>X28</td>\n",
       "      <td>Number of derogatory public records</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>X29</td>\n",
       "      <td>Total credit revolving balance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>X30</td>\n",
       "      <td>Revolving line utilization rate, or the amount...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>X31</td>\n",
       "      <td>The total number of credit lines currently in ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>X32</td>\n",
       "      <td>The initial listing status of the loan. Possib...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Variable                                         Definition\n",
       "0        X1                          Interest Rate on the loan\n",
       "1        X2                          A unique id for the loan.\n",
       "2        X3             A unique id assigned for the borrower.\n",
       "3        X4                              Loan amount requested\n",
       "4        X5                                 Loan amount funded\n",
       "5        X6                    Investor-funded portion of loan\n",
       "6        X7                      Number of payments (36 or 60)\n",
       "7        X8                                         Loan grade\n",
       "8        X9                                      Loan subgrade\n",
       "9       X10                Employer or job title (self-filled)\n",
       "10      X11  Number of years employed (0 to 10; 10 = 10 or ...\n",
       "11      X12  Home ownership status: RENT, OWN, MORTGAGE, OT...\n",
       "12      X13                          Annual income of borrower\n",
       "13      X14  Income verified, not verified, or income sourc...\n",
       "14      X15                               Date loan was issued\n",
       "15      X16               Reason for loan provided by borrower\n",
       "16      X17             Loan category, as provided by borrower\n",
       "17      X18                Loan title, as provided by borrower\n",
       "18      X19                        First 3 numbers of zip code\n",
       "19      X20                                  State of borrower\n",
       "20      X21  A ratio calculated using the borrower’s total ...\n",
       "21      X22  The number of 30+ days past-due incidences of ...\n",
       "22      X23  Date the borrower's earliest reported credit l...\n",
       "23      X24  Number of inquiries by creditors during the pa...\n",
       "24      X25  Number of months since the borrower's last del...\n",
       "25      X26     Number of months since the last public record.\n",
       "26      X27  Number of open credit lines in the borrower's ...\n",
       "27      X28                Number of derogatory public records\n",
       "28      X29                     Total credit revolving balance\n",
       "29      X30  Revolving line utilization rate, or the amount...\n",
       "30      X31  The total number of credit lines currently in ...\n",
       "31      X32  The initial listing status of the loan. Possib..."
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# import metadata.csv via pandas module to get the metadata on the data set variables\n",
    "metadata = pd.read_csv(\"metadata.csv\")\n",
    "metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will look at the data types of each column and look for missing/null values within the data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 400000 entries, 0 to 399999\n",
      "Data columns (total 32 columns):\n",
      "X1     338990 non-null object\n",
      "X2     399999 non-null float64\n",
      "X3     399999 non-null float64\n",
      "X4     399999 non-null object\n",
      "X5     399999 non-null object\n",
      "X6     399999 non-null object\n",
      "X7     399999 non-null object\n",
      "X8     338730 non-null object\n",
      "X9     338730 non-null object\n",
      "X10    376018 non-null object\n",
      "X11    399999 non-null object\n",
      "X12    338639 non-null object\n",
      "X13    338972 non-null float64\n",
      "X14    399999 non-null object\n",
      "X15    399999 non-null object\n",
      "X16    123561 non-null object\n",
      "X17    399999 non-null object\n",
      "X18    399982 non-null object\n",
      "X19    399999 non-null object\n",
      "X20    399999 non-null object\n",
      "X21    399999 non-null float64\n",
      "X22    399999 non-null float64\n",
      "X23    399999 non-null object\n",
      "X24    399999 non-null float64\n",
      "X25    181198 non-null float64\n",
      "X26    51155 non-null float64\n",
      "X27    399999 non-null float64\n",
      "X28    399999 non-null float64\n",
      "X29    399999 non-null float64\n",
      "X30    399733 non-null object\n",
      "X31    399999 non-null float64\n",
      "X32    399999 non-null object\n",
      "dtypes: float64(12), object(20)\n",
      "memory usage: 97.7+ MB\n"
     ]
    }
   ],
   "source": [
    "train.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 80000 entries, 0 to 79999\n",
      "Data columns (total 32 columns):\n",
      "X1     0 non-null float64\n",
      "X2     80000 non-null int64\n",
      "X3     80000 non-null int64\n",
      "X4     80000 non-null object\n",
      "X5     80000 non-null object\n",
      "X6     80000 non-null object\n",
      "X7     80000 non-null object\n",
      "X8     80000 non-null object\n",
      "X9     80000 non-null object\n",
      "X10    75606 non-null object\n",
      "X11    80000 non-null object\n",
      "X12    80000 non-null object\n",
      "X13    80000 non-null float64\n",
      "X14    80000 non-null object\n",
      "X15    80000 non-null object\n",
      "X16    15 non-null object\n",
      "X17    80000 non-null object\n",
      "X18    80000 non-null object\n",
      "X19    80000 non-null object\n",
      "X20    80000 non-null object\n",
      "X21    80000 non-null float64\n",
      "X22    80000 non-null int64\n",
      "X23    80000 non-null object\n",
      "X24    80000 non-null int64\n",
      "X25    41296 non-null float64\n",
      "X26    13839 non-null float64\n",
      "X27    80000 non-null int64\n",
      "X28    80000 non-null int64\n",
      "X29    80000 non-null int64\n",
      "X30    79970 non-null object\n",
      "X31    80000 non-null int64\n",
      "X32    80000 non-null object\n",
      "dtypes: float64(5), int64(8), object(19)\n",
      "memory usage: 19.5+ MB\n"
     ]
    }
   ],
   "source": [
    "test.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Drop X2, X3 as they are unique identifiers for each loan and borrower and not useful for our model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# drop X2 and X3 from training and test set\n",
    "train.drop(['X2', 'X3'], axis=1, inplace=True)\n",
    "test.drop(['X2', 'X3'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### X1, our dependent variable (loan interest rate), is missing about 61000 values\n",
    "\n",
    "We will create a data set where the rows containing these missing values are removed and we'll create another data set where we impute the missing values with the average of the loan interest rate column. We will eventually run our prediction model(s) on both data sets to decipher which method is better for predicting.\n",
    "\n",
    "Note: With more time, perhaps a better method of imputing these missing values would be to partition the training data into clusters using, say, a KNN clustering algorithm and, the cluster that has an average or centroid that's closest to the centroid of the cluster of datum with these missing interest rate values, would be the cluster that we would take the corresponding average interest rate of and impute the original missing interest rate values with. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# remove '%' from interest rate column and change dtype to numeric\n",
    "train['X1'] = pd.to_numeric(train['X1'].str.replace('%', ''))/100\n",
    "\n",
    "# drop the rows with missing interest rate values\n",
    "train = train.dropna(subset=['X1'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There seems to be one row within the training data set with all missing values besides the \n",
    "loan interest rate value. Here we will remove this row:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train.dropna(subset=['X4'], inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are over 61,000 missing values for the X8 and X9 columns, which correspond to loan grades and loan subgrades. As these are categorical variables, a good way to impute these missing values may be, again, via the unsupervised clustering method mentioned above with the missing interest rate values. But, in the interest of time, we will drop these rows from the data frame instead. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# drop X8 and X9 columns: loan grade and subgrade\n",
    "train.dropna(subset=['X8', 'X9'], inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove column X16: Reason for applying for loan\n",
    "\n",
    "As more than 50% of its values are missing and it is the borrower's reason for applying for the loan; this is another relatively unique text data type that will most likely not be useful for our model. Our test set only has 15 out 80000 values available for this column as well, so it is definitely not a useful for our model, where the ultimate goal is to predict on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# remove X16 from training set\n",
    "train.drop(['X16'], axis=1, inplace=True)\n",
    "\n",
    "# remove X16 from test set\n",
    "test.drop(['X16'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove column X10: Borrower-provided job title"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Column X10, which corresponds to borrower-entered employer or job title, is missing almost 20000 values. This is also another unique character data type that won't be as useful to our model, so we will remove the variable altogether from both the test and training data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# remove X10 from training set\n",
    "train.drop(['X10'], axis=1, inplace=True)\n",
    "\n",
    "# remove X10 variable from tests set\n",
    "test.drop(['X10'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove rows where X13 column has missing values: Annual Income\n",
    "As we cannot impute these missing values since it would create false information in a feature that, intuitively, would seem to be a good predictor of loan interest rates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# drop rows where there are missing values for X12 column from data set\n",
    "train.dropna(subset=['X13'], inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove X18 column: Loan Title\n",
    "This column is not useful for our predictive model as it is the borrower-provided loan title; it's another text data type unique to each observation with seemingly no impact on our target variable (interest rate)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# remove X10 from training set\n",
    "train.drop(['X18'], axis=1, inplace=True)\n",
    "\n",
    "# remove X10 variable from test set\n",
    "test.drop(['X18'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove X25 and X26 from data sets: # of months since last delinquency and public record\n",
    "\n",
    "I will delete these variables with reluctance as, intuitively, both of these variables would seem to be correlated with the interest rate assigned to a loan, but they both have more than 50% of their values missing in both the training and test data, we will remove the columns altogether."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# remove X10 from training set\n",
    "train.drop(['X25', 'X26'], axis=1, inplace=True)\n",
    "\n",
    "# remove X10 variable from test set\n",
    "test.drop(['X25', 'X26'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove rows with missing values in X30 column: credit utilization percentage...\n",
    "\n",
    "since there are only about 154 missing values in this column, we will delete these rows from both the training and the test sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# drop rows where there are missing values for X30 column from data set\n",
    "train.dropna(subset=['X30'], inplace=True)\n",
    "\n",
    "# convert X30 to numeric data type\n",
    "test['X30'] = pd.to_numeric(test['X30'].str.replace('%', ''))\n",
    "\n",
    "# fill rows in test data where there are missing values for X30 column with median values\n",
    "test['X30'] = test['X30'].fillna(test.X30.median(axis=0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove X17: Loan Category (provided by Borrower) from data sets...\n",
    "as the category of a loan, intuitively, doesn't seem to be an important feature that would be as relative to the interest rate compare to other features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# remove X10 from training set\n",
    "train.drop(['X17'], axis=1, inplace=True)\n",
    "\n",
    "# remove X10 variable from test set\n",
    "test.drop(['X17'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove X1 (interest rate) variable from test set\n",
    "\n",
    "This is the target variable that we are predicting for our test set, so we remove this empty column (and will replace it with our predictions later)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test.drop(['X1'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove X11 (Years of Employment) column from training and test sets...\n",
    "\n",
    "as both data sets have quite a few 'n/a' values and we do not want to remove any rows from the test set we are predicting for."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['10+ years' '8 years' '< 1 year' '7 years' '2 years' '5 years' '1 year'\n",
      " '6 years' '9 years' '3 years' '4 years' 'n/a'] ['< 1 year' '1 year' '10+ years' '6 years' '9 years' '3 years' '5 years'\n",
      " '2 years' '4 years' '8 years' '7 years' 'n/a']\n"
     ]
    }
   ],
   "source": [
    "print(test.X11.unique(), train.X11.unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of 'n/a' values in X11 column in test set:  4382\n",
      "Number of 'n/a' values in X11 column in training set:  10570\n"
     ]
    }
   ],
   "source": [
    "print(\"Number of 'n/a' values in X11 column in test set: \", len(test[test.X11=='n/a']))\n",
    "print(\"Number of 'n/a' values in X11 column in training set: \", len(train[train.X11=='n/a']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# drop X11 from trainig and test sets\n",
    "test.drop(['X11'], axis=1, inplace=True)\n",
    "train.drop(['X11'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove X15 (Date Loan was Issued)\n",
    "Even though I think base interest rates may oscillate during certains months of any year, like seasons, this feature doesn't seem to be as strong as some of the other features, intuitively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# drop X15 from trainig and test sets\n",
    "test.drop(['X15'], axis=1, inplace=True)\n",
    "train.drop(['X15'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove X20 column (State)\n",
    "\n",
    "Since we already have a more granular location feature being used for our model: first three of zip code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# drop X20 from trainig and test sets\n",
    "test.drop(['X20'], axis=1, inplace=True)\n",
    "train.drop(['X20'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove X23 column (Date of Earliest Line of Credit) from training and test set\n",
    "\n",
    "This could be converted into years, days, or even seconds since earliest line of credit in integer format, but for this model we will exclude this column. Furthermore, we could have also created a new feature by subtracting this feature (converted into an integer) from the date the loan was issued to have a relative picture how long credit establishment effects interest rates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# drop X19 and X20 from training and test sets\n",
    "test.drop(['X23'], axis=1, inplace=True)\n",
    "train.drop(['X23'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove X32 column (Initial Listing Status of Loan) from training and test set\n",
    "\n",
    "As I do not think these initial listing status' would be available when predicting an interest rate for new customers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# drop X19 and X20 from trainig and test sets\n",
    "test.drop(['X32'], axis=1, inplace=True)\n",
    "train.drop(['X32'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove X12 Column (Home Ownership Status)\n",
    "\n",
    "With reluctance as this feature does seem, intuitively, to correlate with loan interest rate, I'm not exactly sure it would be as strong of a predictor as other variables in the data set. \n",
    "\n",
    "Also, since there are a few ambiguous categories (i.e. NONE, OTHER, & ANY) AND nan values, it may make more sense to remove the column altogether as we don't want to combine or impute these values at the cost of creating incorrect signal for our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unique categories in Home Ownership variable:  ['RENT' 'OWN' 'MORTGAGE' 'NONE' nan 'OTHER' 'ANY']\n"
     ]
    }
   ],
   "source": [
    "print(\"Unique categories in Home Ownership variable: \", train.X12.unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# drop X12 from trainig and test sets\n",
    "test.drop(['X12'], axis=1, inplace=True)\n",
    "train.drop(['X12'], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert string columns to appropriate numeric data types in training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train['X4'] = train['X4'].str.replace('$', '')\n",
    "train['X4'] = pd.to_numeric(train['X4'].str.replace(',', ''))\n",
    "train['X5'] = train['X5'].str.replace('$', '')\n",
    "train['X5'] = pd.to_numeric(train['X5'].str.replace(',', ''))\n",
    "train['X6'] = train['X6'].str.replace('$', '')\n",
    "train['X6'] = pd.to_numeric(train['X6'].str.replace(',', ''))\n",
    "train['X7'] = pd.to_numeric(train['X7'].str.replace(' months', ''))\n",
    "train['X19'] = pd.to_numeric(train['X19'].str.replace('x', ''))\n",
    "train['X21'] = train['X21']/100 # convert dti percentage to ratio\n",
    "train['X13'] = train['X13'].astype(int) # convert to interger type\n",
    "train['X22'] = train['X22'].astype(int) # convert to interger type\n",
    "train['X24'] = train['X24'].astype(int) # convert to interger type\n",
    "train['X27'] = train['X27'].astype(int) # convert to interger type\n",
    "train['X28'] = train['X28'].astype(int) # convert to interger type\n",
    "train['X29'] = train['X29'].astype(int) # convert to interger type\n",
    "train['X30'] = train['X30'].str.replace('%', '')\n",
    "train['X30'] = pd.to_numeric(train['X30'].str.replace(',', ''))/100\n",
    "train['X31'] = train['X31'].astype(int) # convert to interger type"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create integer labels for categorical variables in training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# instatiate sklearn's labelencoder\n",
    "le = preprocessing.LabelEncoder()\n",
    "\n",
    "# fit label encorder and return encoded integer labels for categorical string features\n",
    "train['X8'] = le.fit_transform(train['X8'].values)\n",
    "train['X9'] = le.fit_transform(train['X9'].values)\n",
    "train['X14'] = le.fit_transform(train['X14'].values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Look over training data values, types, and summary statistics \n",
    "\n",
    "To ensure there are no abnormalities (missing or outlier values) and all data types of features are as intended"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 242983 entries, 0 to 399999\n",
      "Data columns (total 18 columns):\n",
      "X1     242983 non-null float64\n",
      "X4     242983 non-null int64\n",
      "X5     242983 non-null int64\n",
      "X6     242983 non-null int64\n",
      "X7     242983 non-null int64\n",
      "X8     242983 non-null int64\n",
      "X9     242983 non-null int64\n",
      "X13    242983 non-null int64\n",
      "X14    242983 non-null int64\n",
      "X19    242983 non-null int64\n",
      "X21    242983 non-null float64\n",
      "X22    242983 non-null int64\n",
      "X24    242983 non-null int64\n",
      "X27    242983 non-null int64\n",
      "X28    242983 non-null int64\n",
      "X29    242983 non-null int64\n",
      "X30    242983 non-null float64\n",
      "X31    242983 non-null int64\n",
      "dtypes: float64(3), int64(15)\n",
      "memory usage: 35.2 MB\n"
     ]
    }
   ],
   "source": [
    "train.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>X1</th>\n",
       "      <th>X4</th>\n",
       "      <th>X5</th>\n",
       "      <th>X6</th>\n",
       "      <th>X7</th>\n",
       "      <th>X8</th>\n",
       "      <th>X9</th>\n",
       "      <th>X13</th>\n",
       "      <th>X14</th>\n",
       "      <th>X19</th>\n",
       "      <th>X21</th>\n",
       "      <th>X22</th>\n",
       "      <th>X24</th>\n",
       "      <th>X27</th>\n",
       "      <th>X28</th>\n",
       "      <th>X29</th>\n",
       "      <th>X30</th>\n",
       "      <th>X31</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.1189</td>\n",
       "      <td>25000</td>\n",
       "      <td>25000</td>\n",
       "      <td>19080</td>\n",
       "      <td>36</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>85000</td>\n",
       "      <td>0</td>\n",
       "      <td>941</td>\n",
       "      <td>0.1948</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>28854</td>\n",
       "      <td>0.521</td>\n",
       "      <td>42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.1071</td>\n",
       "      <td>7000</td>\n",
       "      <td>7000</td>\n",
       "      <td>673</td>\n",
       "      <td>36</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>65000</td>\n",
       "      <td>2</td>\n",
       "      <td>112</td>\n",
       "      <td>0.1429</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>33623</td>\n",
       "      <td>0.767</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.1699</td>\n",
       "      <td>25000</td>\n",
       "      <td>25000</td>\n",
       "      <td>24725</td>\n",
       "      <td>36</td>\n",
       "      <td>3</td>\n",
       "      <td>17</td>\n",
       "      <td>70000</td>\n",
       "      <td>0</td>\n",
       "      <td>100</td>\n",
       "      <td>0.1050</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>19878</td>\n",
       "      <td>0.663</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.1311</td>\n",
       "      <td>1200</td>\n",
       "      <td>1200</td>\n",
       "      <td>1200</td>\n",
       "      <td>36</td>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>54000</td>\n",
       "      <td>2</td>\n",
       "      <td>777</td>\n",
       "      <td>0.0547</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>2584</td>\n",
       "      <td>0.404</td>\n",
       "      <td>31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.1357</td>\n",
       "      <td>10800</td>\n",
       "      <td>10800</td>\n",
       "      <td>10692</td>\n",
       "      <td>36</td>\n",
       "      <td>2</td>\n",
       "      <td>12</td>\n",
       "      <td>32000</td>\n",
       "      <td>2</td>\n",
       "      <td>67</td>\n",
       "      <td>0.1163</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>14</td>\n",
       "      <td>0</td>\n",
       "      <td>3511</td>\n",
       "      <td>0.256</td>\n",
       "      <td>40</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       X1     X4     X5     X6  X7  X8  X9    X13  X14  X19     X21  X22  X24  \\\n",
       "0  0.1189  25000  25000  19080  36   1   8  85000    0  941  0.1948    0    0   \n",
       "1  0.1071   7000   7000    673  36   1   9  65000    2  112  0.1429    0    0   \n",
       "2  0.1699  25000  25000  24725  36   3  17  70000    0  100  0.1050    0    0   \n",
       "3  0.1311   1200   1200   1200  36   2  11  54000    2  777  0.0547    0    0   \n",
       "4  0.1357  10800  10800  10692  36   2  12  32000    2   67  0.1163    0    1   \n",
       "\n",
       "   X27  X28    X29    X30  X31  \n",
       "0   10    0  28854  0.521   42  \n",
       "1    7    0  33623  0.767    7  \n",
       "2   10    0  19878  0.663   17  \n",
       "3    5    0   2584  0.404   31  \n",
       "4   14    0   3511  0.256   40  "
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>X1</th>\n",
       "      <th>X4</th>\n",
       "      <th>X5</th>\n",
       "      <th>X6</th>\n",
       "      <th>X7</th>\n",
       "      <th>X8</th>\n",
       "      <th>X9</th>\n",
       "      <th>X13</th>\n",
       "      <th>X14</th>\n",
       "      <th>X19</th>\n",
       "      <th>X21</th>\n",
       "      <th>X22</th>\n",
       "      <th>X24</th>\n",
       "      <th>X27</th>\n",
       "      <th>X28</th>\n",
       "      <th>X29</th>\n",
       "      <th>X30</th>\n",
       "      <th>X31</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>242983.000000</td>\n",
       "      <td>242983.000000</td>\n",
       "      <td>242983.000000</td>\n",
       "      <td>242983.000000</td>\n",
       "      <td>242983.0000</td>\n",
       "      <td>242983.000000</td>\n",
       "      <td>242983.000000</td>\n",
       "      <td>2.429830e+05</td>\n",
       "      <td>242983.000000</td>\n",
       "      <td>242983.000000</td>\n",
       "      <td>242983.000000</td>\n",
       "      <td>242983.000000</td>\n",
       "      <td>242983.000000</td>\n",
       "      <td>242983.000000</td>\n",
       "      <td>242983.000000</td>\n",
       "      <td>2.429830e+05</td>\n",
       "      <td>242983.000000</td>\n",
       "      <td>242983.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.139406</td>\n",
       "      <td>14267.531062</td>\n",
       "      <td>14237.459102</td>\n",
       "      <td>14163.732150</td>\n",
       "      <td>42.4365</td>\n",
       "      <td>1.812530</td>\n",
       "      <td>11.041484</td>\n",
       "      <td>7.312063e+04</td>\n",
       "      <td>0.945399</td>\n",
       "      <td>514.726899</td>\n",
       "      <td>0.170043</td>\n",
       "      <td>0.275233</td>\n",
       "      <td>0.818831</td>\n",
       "      <td>11.120967</td>\n",
       "      <td>0.153064</td>\n",
       "      <td>1.594181e+04</td>\n",
       "      <td>0.562925</td>\n",
       "      <td>24.979838</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.043811</td>\n",
       "      <td>8260.262613</td>\n",
       "      <td>8246.054200</td>\n",
       "      <td>8270.265388</td>\n",
       "      <td>10.6324</td>\n",
       "      <td>1.320213</td>\n",
       "      <td>6.531771</td>\n",
       "      <td>5.657148e+04</td>\n",
       "      <td>0.831274</td>\n",
       "      <td>316.807345</td>\n",
       "      <td>0.076893</td>\n",
       "      <td>0.784393</td>\n",
       "      <td>1.060841</td>\n",
       "      <td>4.887522</td>\n",
       "      <td>0.519584</td>\n",
       "      <td>1.923258e+04</td>\n",
       "      <td>0.237849</td>\n",
       "      <td>11.468621</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.054200</td>\n",
       "      <td>500.000000</td>\n",
       "      <td>500.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>36.0000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.000000e+03</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.109900</td>\n",
       "      <td>8000.000000</td>\n",
       "      <td>8000.000000</td>\n",
       "      <td>8000.000000</td>\n",
       "      <td>36.0000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>4.500000e+04</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>223.000000</td>\n",
       "      <td>0.112700</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>6.465000e+03</td>\n",
       "      <td>0.394000</td>\n",
       "      <td>17.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.136800</td>\n",
       "      <td>12000.000000</td>\n",
       "      <td>12000.000000</td>\n",
       "      <td>12000.000000</td>\n",
       "      <td>36.0000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>6.274400e+04</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>480.000000</td>\n",
       "      <td>0.167100</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>10.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.177400e+04</td>\n",
       "      <td>0.579000</td>\n",
       "      <td>23.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>0.167800</td>\n",
       "      <td>20000.000000</td>\n",
       "      <td>20000.000000</td>\n",
       "      <td>19875.000000</td>\n",
       "      <td>60.0000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>15.000000</td>\n",
       "      <td>8.800000e+04</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>826.000000</td>\n",
       "      <td>0.225000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>14.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.022100e+04</td>\n",
       "      <td>0.749000</td>\n",
       "      <td>32.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>0.260600</td>\n",
       "      <td>35000.000000</td>\n",
       "      <td>35000.000000</td>\n",
       "      <td>35000.000000</td>\n",
       "      <td>60.0000</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>34.000000</td>\n",
       "      <td>7.500000e+06</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>999.000000</td>\n",
       "      <td>0.399900</td>\n",
       "      <td>29.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>62.000000</td>\n",
       "      <td>63.000000</td>\n",
       "      <td>2.568995e+06</td>\n",
       "      <td>8.923000</td>\n",
       "      <td>118.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  X1             X4             X5             X6  \\\n",
       "count  242983.000000  242983.000000  242983.000000  242983.000000   \n",
       "mean        0.139406   14267.531062   14237.459102   14163.732150   \n",
       "std         0.043811    8260.262613    8246.054200    8270.265388   \n",
       "min         0.054200     500.000000     500.000000       0.000000   \n",
       "25%         0.109900    8000.000000    8000.000000    8000.000000   \n",
       "50%         0.136800   12000.000000   12000.000000   12000.000000   \n",
       "75%         0.167800   20000.000000   20000.000000   19875.000000   \n",
       "max         0.260600   35000.000000   35000.000000   35000.000000   \n",
       "\n",
       "                X7             X8             X9           X13            X14  \\\n",
       "count  242983.0000  242983.000000  242983.000000  2.429830e+05  242983.000000   \n",
       "mean       42.4365       1.812530      11.041484  7.312063e+04       0.945399   \n",
       "std        10.6324       1.320213       6.531771  5.657148e+04       0.831274   \n",
       "min        36.0000       0.000000       0.000000  3.000000e+03       0.000000   \n",
       "25%        36.0000       1.000000       6.000000  4.500000e+04       0.000000   \n",
       "50%        36.0000       2.000000      10.000000  6.274400e+04       1.000000   \n",
       "75%        60.0000       3.000000      15.000000  8.800000e+04       2.000000   \n",
       "max        60.0000       6.000000      34.000000  7.500000e+06       2.000000   \n",
       "\n",
       "                 X19            X21            X22            X24  \\\n",
       "count  242983.000000  242983.000000  242983.000000  242983.000000   \n",
       "mean      514.726899       0.170043       0.275233       0.818831   \n",
       "std       316.807345       0.076893       0.784393       1.060841   \n",
       "min         7.000000       0.000000       0.000000       0.000000   \n",
       "25%       223.000000       0.112700       0.000000       0.000000   \n",
       "50%       480.000000       0.167100       0.000000       0.000000   \n",
       "75%       826.000000       0.225000       0.000000       1.000000   \n",
       "max       999.000000       0.399900      29.000000       8.000000   \n",
       "\n",
       "                 X27            X28           X29            X30  \\\n",
       "count  242983.000000  242983.000000  2.429830e+05  242983.000000   \n",
       "mean       11.120967       0.153064  1.594181e+04       0.562925   \n",
       "std         4.887522       0.519584  1.923258e+04       0.237849   \n",
       "min         1.000000       0.000000  0.000000e+00       0.000000   \n",
       "25%         8.000000       0.000000  6.465000e+03       0.394000   \n",
       "50%        10.000000       0.000000  1.177400e+04       0.579000   \n",
       "75%        14.000000       0.000000  2.022100e+04       0.749000   \n",
       "max        62.000000      63.000000  2.568995e+06       8.923000   \n",
       "\n",
       "                 X31  \n",
       "count  242983.000000  \n",
       "mean       24.979838  \n",
       "std        11.468621  \n",
       "min         2.000000  \n",
       "25%        17.000000  \n",
       "50%        23.000000  \n",
       "75%        32.000000  \n",
       "max       118.000000  "
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# use this cell for looking at columns once converted to numeric data types\n",
    "train.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert string columns values to appropriate numeric data types in test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test['X4'] = test['X4'].str.replace('$', '')\n",
    "test['X4'] = pd.to_numeric(test['X4'].str.replace(',', ''))\n",
    "test['X5'] = test['X5'].str.replace('$', '')\n",
    "test['X5'] = pd.to_numeric(test['X5'].str.replace(',', ''))\n",
    "test['X6'] = test['X6'].str.replace('$', '')\n",
    "test['X6'] = pd.to_numeric(test['X6'].str.replace(',', ''))\n",
    "test['X7'] = pd.to_numeric(test['X7'].str.replace(' months', ''))\n",
    "test['X19'] = pd.to_numeric(test['X19'].str.replace('x', ''))\n",
    "test['X21'] = test['X21']/100 # convert dti percentage to ratio\n",
    "test['X13'] = test['X13'].astype(int) # convert to interger type\n",
    "test['X30'] = test['X30']/100 # convert revolving utilization to ratio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create integer labels for categorical variables in test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# instatiate sklearn's labelencoder\n",
    "le = preprocessing.LabelEncoder()\n",
    "\n",
    "# fit label encorder and return encoded integer labels for categorical string features\n",
    "test['X8'] = le.fit_transform(test['X8'].values)\n",
    "test['X9'] = le.fit_transform(test['X9'].values)\n",
    "test['X14'] = le.fit_transform(test['X14'].values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Separate the training data predictors from the targets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# extract target variable (interest rate) from training data\n",
    "target = train['X1']\n",
    "\n",
    "# remove interest rate column from training data\n",
    "predictors = train.drop(['X1'], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Build Machine Learning model in Python to predict the interest rates assigned to loans. \n",
    "\n",
    "Structure your code into two parts: a first part that trains the model on the training data, and a second part runs the model on the training data and the test data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train the RidgeCV regression model on the training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10.0"
      ]
     },
     "execution_count": 132,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# instantiate linear regression object\n",
    "ridge = RidgeCV(cv=10)\n",
    "\n",
    "# fit or train the linear regression model on the training set and store parameters\n",
    "ridge.fit(predictors, target)\n",
    "\n",
    "# show the alpha parameter used in final ridgeCV model\n",
    "ridge.alpha_\n",
    "\n",
    "# show the coefficients of each variable\n",
    "# ridge.coef_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predict interest rates on training and test data - Display training prediction RMSE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE on training data:  0.00903810307548\n"
     ]
    }
   ],
   "source": [
    "# use trained RidgeCV regression model to predict interest rates of training and test data\n",
    "train_pred = ridge.predict(predictors)\n",
    "\n",
    "ridge_test_pred = ridge.predict(test)\n",
    "\n",
    "# print RMSE of training predictions\n",
    "print('RMSE on training data: ', np.sqrt(mean_squared_error(target, train_pred)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An RMSE of 0.009 doesn't seem to be bad, especially for a general linear regression model using that only uses ridge regression to account for potential multicollineararity between features. Also, this linear model is pretty fast in computation time and seemingly straightforward in deploying and maintaing within production. However, without having RMSE's of other models to compare it to, it is difficult to understand the true value of or whether or not this model is \"good\" or even \"fair\". So we will try another machine learning technique to compare the models RMSE scores."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Extreme Gradient Boosted Decision Tree Model using XGBoost\n",
    "\n",
    "Extreme gradient boosting is a Kaggle-popular machine learning technique that can be used on supervised learning problems like classification and regression problems. The model creates an ensemble of relatively weak prediction models or \"learners\" (usually decision trees, but linear models can be used as well). The weak learners (decision trees with, say, one or two logical conditions applied to samples of the data) are built sequentially and the output of each learner corresponds to the weight the algorithm gives to the outcomes in the subsequent learner, at least in classification; in regression, however, at each iteration, a new tree learns the gradients of the residuals between the target values and the current predicted values, and then the algorithm conducts gradient descent based on the learned gradients. That is, at each iteration, the learners are combined and start to predict the training data more accurately, becoming increasingly complex along the way.\n",
    "\n",
    "Finally, each learner is applied a weight  and a combined model is built for final analysis/predicting.\n",
    "\n",
    "XGBoost is a specific implementation of gradient boosted decision trees designed for speed and performance. It is most known and used for its 'regularized boosting' technique, its fast, parallel processing in its decision tree building (leading to potential scalable production), and its relatively high customization options with hyperparameters. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gradient Boosting Explained:\n",
    "\n",
    "\n",
    "* Make a set of predictions: $\\hat{y}[i]$\n",
    "\n",
    "\n",
    "* The \"error\" in our predictions is: $E(y, \\hat{y})$ (\"MSE\" loss function used by default)\n",
    "\n",
    "\n",
    "* \"Adjust\" $\\hat{y}$ to reduce error: $\\hat{y}[i] = \\hat{y}[i] + alpha * f[i]$\n",
    "\n",
    "\n",
    "* $f[i] = \\bigtriangledown E(y, \\hat{y})$ (i.e. the loss function)\n",
    "\n",
    "\n",
    "* Each learner is estimating the gradient of the loss function \n",
    "\n",
    "\n",
    "* Gradient Descent: take sequence of steps to reduce (descend) the Error\n",
    "\n",
    "\n",
    "* Each predictor is summed together, weighted by the alpha step size or \"learning rate\" to create the final model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import xgboost as xgb\n",
    "from xgboost import plot_importance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Split the training set into a training and validation set to make sure our model doesn't significantly overfit our training data and perform poorly on our test data and create parameter dictionary to use in the model (manually, trial-and-error tuned)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train the xgb model on a train_test_split of the training data with a manually-tuned set of hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\tTest-rmse:0.327271\n",
      "Will train until Test-rmse hasn't improved in 10 rounds.\n",
      "[1]\tTest-rmse:0.294572\n",
      "[2]\tTest-rmse:0.265146\n",
      "[3]\tTest-rmse:0.238664\n",
      "[4]\tTest-rmse:0.214834\n",
      "[5]\tTest-rmse:0.193388\n",
      "[6]\tTest-rmse:0.174092\n",
      "[7]\tTest-rmse:0.156728\n",
      "[8]\tTest-rmse:0.141103\n",
      "[9]\tTest-rmse:0.127046\n",
      "[10]\tTest-rmse:0.1144\n",
      "[11]\tTest-rmse:0.103024\n",
      "[12]\tTest-rmse:0.092792\n",
      "[13]\tTest-rmse:0.083591\n",
      "[14]\tTest-rmse:0.075317\n",
      "[15]\tTest-rmse:0.067879\n",
      "[16]\tTest-rmse:0.061194\n",
      "[17]\tTest-rmse:0.055186\n",
      "[18]\tTest-rmse:0.049792\n",
      "[19]\tTest-rmse:0.044949\n",
      "[20]\tTest-rmse:0.040604\n",
      "[21]\tTest-rmse:0.036708\n",
      "[22]\tTest-rmse:0.033219\n",
      "[23]\tTest-rmse:0.030096\n",
      "[24]\tTest-rmse:0.027307\n",
      "[25]\tTest-rmse:0.024816\n",
      "[26]\tTest-rmse:0.022597\n",
      "[27]\tTest-rmse:0.020624\n",
      "[28]\tTest-rmse:0.018874\n",
      "[29]\tTest-rmse:0.017319\n",
      "[30]\tTest-rmse:0.015954\n",
      "[31]\tTest-rmse:0.01475\n",
      "[32]\tTest-rmse:0.013698\n",
      "[33]\tTest-rmse:0.01278\n",
      "[34]\tTest-rmse:0.011981\n",
      "[35]\tTest-rmse:0.011295\n",
      "[36]\tTest-rmse:0.010704\n",
      "[37]\tTest-rmse:0.010191\n",
      "[38]\tTest-rmse:0.00976\n",
      "[39]\tTest-rmse:0.009394\n",
      "[40]\tTest-rmse:0.009091\n",
      "[41]\tTest-rmse:0.008831\n",
      "[42]\tTest-rmse:0.008618\n",
      "[43]\tTest-rmse:0.008442\n",
      "[44]\tTest-rmse:0.008287\n",
      "[45]\tTest-rmse:0.008162\n",
      "[46]\tTest-rmse:0.008062\n",
      "[47]\tTest-rmse:0.007967\n",
      "[48]\tTest-rmse:0.007899\n",
      "[49]\tTest-rmse:0.00782\n",
      "[50]\tTest-rmse:0.007771\n",
      "[51]\tTest-rmse:0.007733\n",
      "[52]\tTest-rmse:0.007698\n",
      "[53]\tTest-rmse:0.007656\n",
      "[54]\tTest-rmse:0.00763\n",
      "[55]\tTest-rmse:0.007601\n",
      "[56]\tTest-rmse:0.007567\n",
      "[57]\tTest-rmse:0.007553\n",
      "[58]\tTest-rmse:0.007521\n",
      "[59]\tTest-rmse:0.007496\n",
      "[60]\tTest-rmse:0.007479\n",
      "[61]\tTest-rmse:0.007472\n",
      "[62]\tTest-rmse:0.007466\n",
      "[63]\tTest-rmse:0.007461\n",
      "[64]\tTest-rmse:0.007441\n",
      "[65]\tTest-rmse:0.007416\n",
      "[66]\tTest-rmse:0.007412\n",
      "[67]\tTest-rmse:0.00741\n",
      "[68]\tTest-rmse:0.007405\n",
      "[69]\tTest-rmse:0.007402\n",
      "[70]\tTest-rmse:0.007382\n",
      "[71]\tTest-rmse:0.007371\n",
      "[72]\tTest-rmse:0.007367\n",
      "[73]\tTest-rmse:0.007354\n",
      "[74]\tTest-rmse:0.007351\n",
      "[75]\tTest-rmse:0.00735\n",
      "[76]\tTest-rmse:0.007346\n",
      "[77]\tTest-rmse:0.007344\n",
      "[78]\tTest-rmse:0.00734\n",
      "[79]\tTest-rmse:0.007329\n",
      "[80]\tTest-rmse:0.00732\n",
      "[81]\tTest-rmse:0.00732\n",
      "[82]\tTest-rmse:0.007317\n",
      "[83]\tTest-rmse:0.007303\n",
      "[84]\tTest-rmse:0.007296\n",
      "[85]\tTest-rmse:0.007287\n",
      "[86]\tTest-rmse:0.007286\n",
      "[87]\tTest-rmse:0.007285\n",
      "[88]\tTest-rmse:0.007278\n",
      "[89]\tTest-rmse:0.007277\n",
      "[90]\tTest-rmse:0.007275\n",
      "[91]\tTest-rmse:0.007264\n",
      "[92]\tTest-rmse:0.007263\n",
      "[93]\tTest-rmse:0.007261\n",
      "[94]\tTest-rmse:0.00726\n",
      "[95]\tTest-rmse:0.007254\n",
      "[96]\tTest-rmse:0.007249\n",
      "[97]\tTest-rmse:0.007248\n",
      "[98]\tTest-rmse:0.007247\n",
      "[99]\tTest-rmse:0.00724\n",
      "[100]\tTest-rmse:0.00724\n",
      "[101]\tTest-rmse:0.007238\n",
      "[102]\tTest-rmse:0.007234\n",
      "[103]\tTest-rmse:0.007232\n",
      "[104]\tTest-rmse:0.007231\n",
      "[105]\tTest-rmse:0.007229\n",
      "[106]\tTest-rmse:0.007227\n",
      "[107]\tTest-rmse:0.007226\n",
      "[108]\tTest-rmse:0.007225\n",
      "[109]\tTest-rmse:0.007221\n",
      "[110]\tTest-rmse:0.007218\n",
      "[111]\tTest-rmse:0.007217\n",
      "[112]\tTest-rmse:0.007217\n",
      "[113]\tTest-rmse:0.007216\n",
      "[114]\tTest-rmse:0.007213\n",
      "[115]\tTest-rmse:0.007211\n",
      "[116]\tTest-rmse:0.007209\n",
      "[117]\tTest-rmse:0.007206\n",
      "[118]\tTest-rmse:0.007203\n",
      "[119]\tTest-rmse:0.007202\n",
      "[120]\tTest-rmse:0.007201\n",
      "[121]\tTest-rmse:0.007197\n",
      "[122]\tTest-rmse:0.007195\n",
      "[123]\tTest-rmse:0.007193\n",
      "[124]\tTest-rmse:0.007193\n",
      "[125]\tTest-rmse:0.00719\n",
      "[126]\tTest-rmse:0.00719\n",
      "[127]\tTest-rmse:0.007189\n",
      "[128]\tTest-rmse:0.007187\n",
      "[129]\tTest-rmse:0.007186\n",
      "[130]\tTest-rmse:0.007186\n",
      "[131]\tTest-rmse:0.007181\n",
      "[132]\tTest-rmse:0.007176\n",
      "[133]\tTest-rmse:0.007175\n",
      "[134]\tTest-rmse:0.007173\n",
      "[135]\tTest-rmse:0.007172\n",
      "[136]\tTest-rmse:0.00717\n",
      "[137]\tTest-rmse:0.00717\n",
      "[138]\tTest-rmse:0.007168\n",
      "[139]\tTest-rmse:0.007165\n",
      "[140]\tTest-rmse:0.007164\n",
      "[141]\tTest-rmse:0.007163\n",
      "[142]\tTest-rmse:0.007163\n",
      "[143]\tTest-rmse:0.00716\n",
      "[144]\tTest-rmse:0.007157\n",
      "[145]\tTest-rmse:0.007156\n",
      "[146]\tTest-rmse:0.007155\n",
      "[147]\tTest-rmse:0.007154\n",
      "[148]\tTest-rmse:0.007154\n",
      "[149]\tTest-rmse:0.007151\n",
      "[150]\tTest-rmse:0.007151\n",
      "[151]\tTest-rmse:0.007151\n",
      "[152]\tTest-rmse:0.007151\n",
      "[153]\tTest-rmse:0.00715\n",
      "[154]\tTest-rmse:0.00715\n",
      "[155]\tTest-rmse:0.00715\n",
      "[156]\tTest-rmse:0.007149\n",
      "[157]\tTest-rmse:0.007147\n",
      "[158]\tTest-rmse:0.007146\n",
      "[159]\tTest-rmse:0.007146\n",
      "[160]\tTest-rmse:0.007141\n",
      "[161]\tTest-rmse:0.007138\n",
      "[162]\tTest-rmse:0.007138\n",
      "[163]\tTest-rmse:0.007138\n",
      "[164]\tTest-rmse:0.007136\n",
      "[165]\tTest-rmse:0.007136\n",
      "[166]\tTest-rmse:0.007134\n",
      "[167]\tTest-rmse:0.007133\n",
      "[168]\tTest-rmse:0.007133\n",
      "[169]\tTest-rmse:0.007132\n",
      "[170]\tTest-rmse:0.007129\n",
      "[171]\tTest-rmse:0.007129\n",
      "[172]\tTest-rmse:0.007127\n",
      "[173]\tTest-rmse:0.007127\n",
      "[174]\tTest-rmse:0.007126\n",
      "[175]\tTest-rmse:0.007124\n",
      "[176]\tTest-rmse:0.007121\n",
      "[177]\tTest-rmse:0.00712\n",
      "[178]\tTest-rmse:0.007119\n",
      "[179]\tTest-rmse:0.007118\n",
      "[180]\tTest-rmse:0.007116\n",
      "[181]\tTest-rmse:0.007116\n",
      "[182]\tTest-rmse:0.007116\n",
      "[183]\tTest-rmse:0.007116\n",
      "[184]\tTest-rmse:0.007115\n",
      "[185]\tTest-rmse:0.00711\n",
      "[186]\tTest-rmse:0.007109\n",
      "[187]\tTest-rmse:0.007108\n",
      "[188]\tTest-rmse:0.007107\n",
      "[189]\tTest-rmse:0.007107\n",
      "[190]\tTest-rmse:0.007106\n",
      "[191]\tTest-rmse:0.007106\n",
      "[192]\tTest-rmse:0.007106\n",
      "[193]\tTest-rmse:0.007106\n",
      "[194]\tTest-rmse:0.007105\n",
      "[195]\tTest-rmse:0.007105\n",
      "[196]\tTest-rmse:0.007105\n",
      "[197]\tTest-rmse:0.007105\n",
      "[198]\tTest-rmse:0.007105\n",
      "[199]\tTest-rmse:0.007105\n",
      "[200]\tTest-rmse:0.007104\n",
      "[201]\tTest-rmse:0.007104\n",
      "[202]\tTest-rmse:0.007104\n",
      "[203]\tTest-rmse:0.007103\n",
      "[204]\tTest-rmse:0.007103\n",
      "[205]\tTest-rmse:0.007103\n",
      "[206]\tTest-rmse:0.007102\n",
      "[207]\tTest-rmse:0.007102\n",
      "[208]\tTest-rmse:0.0071\n",
      "[209]\tTest-rmse:0.007095\n",
      "[210]\tTest-rmse:0.007096\n",
      "[211]\tTest-rmse:0.007095\n",
      "[212]\tTest-rmse:0.007095\n",
      "[213]\tTest-rmse:0.007093\n",
      "[214]\tTest-rmse:0.00709\n",
      "[215]\tTest-rmse:0.00709\n",
      "[216]\tTest-rmse:0.007089\n",
      "[217]\tTest-rmse:0.007089\n",
      "[218]\tTest-rmse:0.007088\n",
      "[219]\tTest-rmse:0.007088\n",
      "[220]\tTest-rmse:0.007088\n",
      "[221]\tTest-rmse:0.007088\n",
      "[222]\tTest-rmse:0.007088\n",
      "[223]\tTest-rmse:0.007086\n",
      "[224]\tTest-rmse:0.007086\n",
      "[225]\tTest-rmse:0.007086\n",
      "[226]\tTest-rmse:0.007086\n",
      "[227]\tTest-rmse:0.007086\n",
      "[228]\tTest-rmse:0.007085\n",
      "[229]\tTest-rmse:0.007083\n",
      "[230]\tTest-rmse:0.007082\n",
      "[231]\tTest-rmse:0.007082\n",
      "[232]\tTest-rmse:0.007082\n",
      "[233]\tTest-rmse:0.007081\n",
      "[234]\tTest-rmse:0.007079\n",
      "[235]\tTest-rmse:0.007078\n",
      "[236]\tTest-rmse:0.007075\n",
      "[237]\tTest-rmse:0.007072\n",
      "[238]\tTest-rmse:0.007071\n",
      "[239]\tTest-rmse:0.007071\n",
      "[240]\tTest-rmse:0.00707\n",
      "[241]\tTest-rmse:0.007071\n",
      "[242]\tTest-rmse:0.00707\n",
      "[243]\tTest-rmse:0.00707\n",
      "[244]\tTest-rmse:0.00707\n",
      "[245]\tTest-rmse:0.00707\n",
      "[246]\tTest-rmse:0.00707\n",
      "[247]\tTest-rmse:0.007067\n",
      "[248]\tTest-rmse:0.007066\n",
      "[249]\tTest-rmse:0.007066\n",
      "[250]\tTest-rmse:0.007066\n",
      "[251]\tTest-rmse:0.007065\n",
      "[252]\tTest-rmse:0.007063\n",
      "[253]\tTest-rmse:0.007063\n",
      "[254]\tTest-rmse:0.007063\n",
      "[255]\tTest-rmse:0.007061\n",
      "[256]\tTest-rmse:0.00706\n",
      "[257]\tTest-rmse:0.007058\n",
      "[258]\tTest-rmse:0.007056\n",
      "[259]\tTest-rmse:0.007056\n",
      "[260]\tTest-rmse:0.007055\n",
      "[261]\tTest-rmse:0.007055\n",
      "[262]\tTest-rmse:0.007055\n",
      "[263]\tTest-rmse:0.007054\n",
      "[264]\tTest-rmse:0.007054\n",
      "[265]\tTest-rmse:0.007052\n",
      "[266]\tTest-rmse:0.007052\n",
      "[267]\tTest-rmse:0.007052\n",
      "[268]\tTest-rmse:0.007051\n",
      "[269]\tTest-rmse:0.00705\n",
      "[270]\tTest-rmse:0.007049\n",
      "[271]\tTest-rmse:0.007049\n",
      "[272]\tTest-rmse:0.007049\n",
      "[273]\tTest-rmse:0.007049\n",
      "[274]\tTest-rmse:0.007049\n",
      "[275]\tTest-rmse:0.007049\n",
      "[276]\tTest-rmse:0.007049\n",
      "[277]\tTest-rmse:0.007049\n",
      "[278]\tTest-rmse:0.007049\n",
      "[279]\tTest-rmse:0.007048\n",
      "[280]\tTest-rmse:0.007046\n",
      "[281]\tTest-rmse:0.007045\n",
      "[282]\tTest-rmse:0.007044\n",
      "[283]\tTest-rmse:0.007044\n",
      "[284]\tTest-rmse:0.007044\n",
      "[285]\tTest-rmse:0.007042\n",
      "[286]\tTest-rmse:0.007042\n",
      "[287]\tTest-rmse:0.007041\n",
      "[288]\tTest-rmse:0.007041\n",
      "[289]\tTest-rmse:0.007041\n",
      "[290]\tTest-rmse:0.007041\n",
      "[291]\tTest-rmse:0.007041\n",
      "[292]\tTest-rmse:0.007041\n",
      "[293]\tTest-rmse:0.00704\n",
      "[294]\tTest-rmse:0.007039\n",
      "[295]\tTest-rmse:0.007039\n",
      "[296]\tTest-rmse:0.007037\n",
      "[297]\tTest-rmse:0.007036\n",
      "[298]\tTest-rmse:0.007035\n",
      "[299]\tTest-rmse:0.007034\n",
      "[300]\tTest-rmse:0.007034\n",
      "[301]\tTest-rmse:0.007034\n",
      "[302]\tTest-rmse:0.007034\n",
      "[303]\tTest-rmse:0.007032\n",
      "[304]\tTest-rmse:0.007032\n",
      "[305]\tTest-rmse:0.00703\n",
      "[306]\tTest-rmse:0.007029\n",
      "[307]\tTest-rmse:0.007028\n",
      "[308]\tTest-rmse:0.007028\n",
      "[309]\tTest-rmse:0.007028\n",
      "[310]\tTest-rmse:0.007028\n",
      "[311]\tTest-rmse:0.007026\n",
      "[312]\tTest-rmse:0.007024\n",
      "[313]\tTest-rmse:0.007025\n",
      "[314]\tTest-rmse:0.007024\n",
      "[315]\tTest-rmse:0.007023\n",
      "[316]\tTest-rmse:0.007023\n",
      "[317]\tTest-rmse:0.007023\n",
      "[318]\tTest-rmse:0.007023\n",
      "[319]\tTest-rmse:0.007023\n",
      "[320]\tTest-rmse:0.007023\n",
      "[321]\tTest-rmse:0.007023\n",
      "[322]\tTest-rmse:0.007022\n",
      "[323]\tTest-rmse:0.007022\n",
      "[324]\tTest-rmse:0.007022\n",
      "[325]\tTest-rmse:0.007021\n",
      "[326]\tTest-rmse:0.007021\n",
      "[327]\tTest-rmse:0.00702\n",
      "[328]\tTest-rmse:0.00702\n",
      "[329]\tTest-rmse:0.00702\n",
      "[330]\tTest-rmse:0.00702\n",
      "[331]\tTest-rmse:0.00702\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[332]\tTest-rmse:0.007019\n",
      "[333]\tTest-rmse:0.007019\n",
      "[334]\tTest-rmse:0.007018\n",
      "[335]\tTest-rmse:0.007018\n",
      "[336]\tTest-rmse:0.007017\n",
      "[337]\tTest-rmse:0.007017\n",
      "[338]\tTest-rmse:0.007017\n",
      "[339]\tTest-rmse:0.007018\n",
      "[340]\tTest-rmse:0.007018\n",
      "[341]\tTest-rmse:0.007018\n",
      "[342]\tTest-rmse:0.007018\n",
      "[343]\tTest-rmse:0.007018\n",
      "[344]\tTest-rmse:0.007016\n",
      "[345]\tTest-rmse:0.007016\n",
      "[346]\tTest-rmse:0.007016\n",
      "[347]\tTest-rmse:0.007015\n",
      "[348]\tTest-rmse:0.007016\n",
      "[349]\tTest-rmse:0.007015\n",
      "[350]\tTest-rmse:0.007014\n",
      "[351]\tTest-rmse:0.007014\n",
      "[352]\tTest-rmse:0.007014\n",
      "[353]\tTest-rmse:0.007014\n",
      "[354]\tTest-rmse:0.007014\n",
      "[355]\tTest-rmse:0.007014\n",
      "[356]\tTest-rmse:0.007014\n",
      "[357]\tTest-rmse:0.007014\n",
      "[358]\tTest-rmse:0.007012\n",
      "[359]\tTest-rmse:0.007012\n",
      "[360]\tTest-rmse:0.007012\n",
      "[361]\tTest-rmse:0.007012\n",
      "[362]\tTest-rmse:0.007011\n",
      "[363]\tTest-rmse:0.007011\n",
      "[364]\tTest-rmse:0.00701\n",
      "[365]\tTest-rmse:0.00701\n",
      "[366]\tTest-rmse:0.00701\n",
      "[367]\tTest-rmse:0.00701\n",
      "[368]\tTest-rmse:0.00701\n",
      "[369]\tTest-rmse:0.00701\n",
      "[370]\tTest-rmse:0.007009\n",
      "[371]\tTest-rmse:0.007009\n",
      "[372]\tTest-rmse:0.007009\n",
      "[373]\tTest-rmse:0.007009\n",
      "[374]\tTest-rmse:0.007008\n",
      "[375]\tTest-rmse:0.007008\n",
      "[376]\tTest-rmse:0.007008\n",
      "[377]\tTest-rmse:0.007008\n",
      "[378]\tTest-rmse:0.007008\n",
      "[379]\tTest-rmse:0.007008\n",
      "[380]\tTest-rmse:0.007009\n",
      "[381]\tTest-rmse:0.007008\n",
      "[382]\tTest-rmse:0.007008\n",
      "[383]\tTest-rmse:0.007007\n",
      "[384]\tTest-rmse:0.007007\n",
      "[385]\tTest-rmse:0.007005\n",
      "[386]\tTest-rmse:0.007004\n",
      "[387]\tTest-rmse:0.007004\n",
      "[388]\tTest-rmse:0.007004\n",
      "[389]\tTest-rmse:0.007004\n",
      "[390]\tTest-rmse:0.007004\n",
      "[391]\tTest-rmse:0.007004\n",
      "[392]\tTest-rmse:0.007004\n",
      "[393]\tTest-rmse:0.007004\n",
      "[394]\tTest-rmse:0.007004\n",
      "[395]\tTest-rmse:0.007002\n",
      "[396]\tTest-rmse:0.007002\n",
      "[397]\tTest-rmse:0.007001\n",
      "[398]\tTest-rmse:0.007\n",
      "[399]\tTest-rmse:0.007\n",
      "[400]\tTest-rmse:0.007\n",
      "[401]\tTest-rmse:0.007\n",
      "[402]\tTest-rmse:0.006998\n",
      "[403]\tTest-rmse:0.006998\n",
      "[404]\tTest-rmse:0.006998\n",
      "[405]\tTest-rmse:0.006997\n",
      "[406]\tTest-rmse:0.006997\n",
      "[407]\tTest-rmse:0.006996\n",
      "[408]\tTest-rmse:0.006996\n",
      "[409]\tTest-rmse:0.006996\n",
      "[410]\tTest-rmse:0.006996\n",
      "[411]\tTest-rmse:0.006996\n",
      "[412]\tTest-rmse:0.006996\n",
      "[413]\tTest-rmse:0.006995\n",
      "[414]\tTest-rmse:0.006995\n",
      "[415]\tTest-rmse:0.006995\n",
      "[416]\tTest-rmse:0.006995\n",
      "[417]\tTest-rmse:0.006994\n",
      "[418]\tTest-rmse:0.006994\n",
      "[419]\tTest-rmse:0.006994\n",
      "[420]\tTest-rmse:0.006994\n",
      "[421]\tTest-rmse:0.006994\n",
      "[422]\tTest-rmse:0.006994\n",
      "[423]\tTest-rmse:0.006994\n",
      "[424]\tTest-rmse:0.006994\n",
      "[425]\tTest-rmse:0.006993\n",
      "[426]\tTest-rmse:0.006993\n",
      "[427]\tTest-rmse:0.006992\n",
      "[428]\tTest-rmse:0.006992\n",
      "[429]\tTest-rmse:0.006992\n",
      "[430]\tTest-rmse:0.006992\n",
      "[431]\tTest-rmse:0.006992\n",
      "[432]\tTest-rmse:0.006992\n",
      "[433]\tTest-rmse:0.006992\n",
      "[434]\tTest-rmse:0.006992\n",
      "[435]\tTest-rmse:0.006991\n",
      "[436]\tTest-rmse:0.006992\n",
      "[437]\tTest-rmse:0.006991\n",
      "[438]\tTest-rmse:0.006991\n",
      "[439]\tTest-rmse:0.006991\n",
      "[440]\tTest-rmse:0.00699\n",
      "[441]\tTest-rmse:0.006991\n",
      "[442]\tTest-rmse:0.00699\n",
      "[443]\tTest-rmse:0.00699\n",
      "[444]\tTest-rmse:0.00699\n",
      "[445]\tTest-rmse:0.00699\n",
      "[446]\tTest-rmse:0.00699\n",
      "[447]\tTest-rmse:0.00699\n",
      "[448]\tTest-rmse:0.00699\n",
      "[449]\tTest-rmse:0.00699\n",
      "[450]\tTest-rmse:0.00699\n",
      "Stopping. Best iteration:\n",
      "[440]\tTest-rmse:0.00699\n",
      "\n",
      "Model's best RMSE on test set:  0.00699\n",
      "Model's best iteration:  441\n"
     ]
    }
   ],
   "source": [
    "# split training data into training and validation sets\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    predictors,\n",
    "    target,\n",
    "    test_size=.2,\n",
    "    random_state=42)\n",
    "\n",
    "# xgb models only accept 'DMatrix' input; convert the data here\n",
    "dtrain = xgb.DMatrix(X_train, label=y_train)\n",
    "dtest = xgb.DMatrix(X_test, label=y_test)\n",
    "\n",
    "# parameter grid for xgb model\n",
    "param_dict = {\n",
    "    # Parameters that we are going to tune. See discussion below for explanation of \n",
    "    # parameters and tuning\n",
    "    'max_depth':6, \n",
    "    'min_child_weight': 1,\n",
    "    'eta':.1,\n",
    "    'subsample': 1,\n",
    "    'colsample_bytree': 1,\n",
    "    # Other parameters\n",
    "    'objective':'reg:linear',\n",
    "    'eval_metric':'rmse'\n",
    "}\n",
    "\n",
    "# train XGB model on split training data using split test data\n",
    "num_boost_round=999\n",
    "model = xgb.train(\n",
    "    param_dict,\n",
    "    dtrain,\n",
    "    num_boost_round=num_boost_round,\n",
    "    evals=[(dtest, \"Test\")],\n",
    "    early_stopping_rounds=10\n",
    ")\n",
    "\n",
    "print(\"Model's best RMSE on test set: \", model.best_score)\n",
    "print(\"Model's best iteration: \", model.best_iteration+1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE of XGB model on training data:  0.00663681909919\n"
     ]
    }
   ],
   "source": [
    "# convert training data to DMatrix object\n",
    "dpred=xgb.DMatrix(predictors)\n",
    "\n",
    "# predict RMSE of Gradient Boosted Decision Tree model on training data\n",
    "print(\"RMSE of XGB model on training data: \",\n",
    "      np.sqrt(mean_squared_error(model.predict(dpred), target)))\n",
    "\n",
    "# predict RMSE of model on test data\n",
    "GBDT_test_pred = model.predict(xgb.DMatrix(test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot importance of features via F-score provided by GBDT model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEWCAYAAABMoxE0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XucznX+//HHy2ElE5G0hpDkNAfjbL5OI2tUTmnaNpEo\ntdpsOtDqJ0VpU1iybDpItKGU0jmtDLstiRrHGmwUxrGcBsXw+v3x+czVNTPXNedrPnPxut9u181c\nn+vz+VzPz5Xm7focnh9RVYwxxpjCKuN1AGOMMeHNBhJjjDFFYgOJMcaYIrGBxBhjTJHYQGKMMaZI\nbCAxxhhTJDaQGFNIIjJTRMZ4ncMYr4ldR2JKmojsAC4DzvhNbqiqaUVYZwLwT1WtXbR04UlEXgF2\nqeojXmcx5x/7RmK80ktVI/wehR5EioOIlPPy/YtCRMp6ncGc32wgMaWKiLQTkf+KyGERWed+08h8\nbbCIfCMix0TkOxH5ozu9EvARECki6e4jUkReEZHxfssniMguv+c7ROQvIrIeOC4i5dzl3hKRAyKy\nXUTuzSWrb/2Z6xaRh0Rkv4jsEZHrReQ6EdkiIj+JyP/zW3asiLwpIq+72/OViDTze72JiCS7n8Mm\nEemd7X2fE5EPReQ4cAfQH3jI3fb33PlGicj/3PVvFpG+fusYJCL/EZFJInLI3dZr/V6vJiKzRSTN\nff0dv9d6ikiKm+2/IhKb7//A5pxkA4kpNUSkFvABMB6oBowA3hKRS91Z9gM9gcrAYGCKiLRQ1ePA\ntUBaIb7h9AN6ABcDZ4H3gHVALaArcJ+IdM/nun4LXOAu+yjwIjAAaAl0BB4Vkfp+8/cBFrrbOg94\nR0TKi0h5N8cSoAbwZ+A1EWnkt+wtwJPARcBc4DXgGXfbe7nz/M993yrAOOCfIlLTbx1tgVSgOvAM\nMEtExH3tVeBCIMrNMAVARFoALwN/BC4BngfeFZEK+fyMzDnIBhLjlXfcf9Ee9vvX7gDgQ1X9UFXP\nquqnwBrgOgBV/UBV/6eO5Ti/aDsWMcc0Vd2pqieB1sClqvq4qp5S1e9wBoOb87mu08CTqnoaWIDz\nC/pZVT2mqpuATYD/v97Xquqb7vx/wxmE2rmPCGCCm+Mz4H2cQS/TYlX93P2cfg4URlUXqmqaO8/r\nwFagjd8s36vqi6p6BpgD1AQucweba4GhqnpIVU+7nzfAncDzqvqFqp5R1TnAL25mc54K2/3CJuxd\nr6r/yjatLvB7EenlN608sAzA3fXyGNAQ5x9BFwIbiphjZ7b3jxSRw37TygL/zue6fnR/KQOcdP/c\n5/f6SZwBIsd7q+pZd7dbZOZrqnrWb97vcb7pBModkIgMBB4A6rmTInAGt0x7/d7/hPtlJALnG9JP\nqnoowGrrAreJyJ/9pv3GL7c5D9lAYkqTncCrqnpn9hfcXSdvAQNx/jV+2v0mk7krJtDph8dxBptM\nvw0wj/9yO4HtqnpVYcIXwuWZP4hIGaA2kLlL7nIRKeM3mNQBtvgtm317szwXkbo436a6AitV9YyI\npPDr55WbnUA1EblYVQ8HeO1JVX0yH+sx5wnbtWVKk38CvUSku4iUFZEL3IPYtXH+1VsBOABkuN9O\nEv2W3QdcIiJV/KalANe5B45/C9yXx/uvBo66B+AruhmiRaR1sW1hVi1F5Ab3jLH7cHYRrQK+wBkE\nH3KPmSQAvXB2lwWzD/A//lIJZ3A5AM6JCkB0fkKp6h6ckxf+ISJV3Qyd3JdfBIaKSFtxVBKRHiJy\nUT632ZyDbCAxpYaq7sQ5AP3/cH4B7gRGAmVU9RhwL/AGcAjnYPO7fst+C8wHvnOPu0TiHDBeB+zA\nOZ7yeh7vfwbnF3YcsB04CLyEc7A6FBYDf8DZnluBG9zjEaeA3jjHKQ4C/wAGutsYzCygaeYxJ1Xd\nDEwGVuIMMjHA5wXIdivOMZ9vcU5yuA9AVdfgHCeZ7ubeBgwqwHrNOcguSDTGAyIyFmigqgO8zmJM\nUdk3EmOMMUViA4kxxpgisV1bxhhjisS+kRhjjCmSsL+O5OKLL9YGDRp4HaPQjh8/TqVKlbyOUSjh\nnB0sv9csv7fWrl17UFUvzXvOvIX9QHLZZZexZs0ar2MUWnJyMgkJCV7HKJRwzg6W32uW31si8n1x\nrct2bRljjCkSG0iMMcYUiQ0kxhhjisQGEmOMMUViA4kxxpgisYHEGGNMkdhAYowxHvr5559p06YN\nzZo1IyoqisceewyA6dOn06BBA0SEgwcP+uY/cuQIvXr18s0/e/ZsAFJSUoiPjycqKorY2Fhefz3X\nsutiVeLXkYjI5cAKoKWq/iQiVYGvgATgTzj3zwZ4wr09qDHGnLMqVKjAZ599RkREBKdPn6ZDhw5c\ne+21tG/fnp49e+a4VmXGjBk0bdqU9957jwMHDtCoUSP69+/PhRdeyNy5c7nqqqtIS0ujZcuWdO/e\nnYsvvjjk21DiA4mq7hSR54AJwF3uny/g3HSnBc69ICoAy0XkI1U9WtIZjTGmpIgIERHOHZhPnz7N\n6dOnERGaN28edP5jx46hqqSnp1OtWjXKlStHw4YNffNERkZSo0YNDhw4cG4OJK4pwFoRuQ/oAPwZ\nGA4sV9UMnDvgrQOuwbmRUVAnT5+h3qgPQp03ZB6MyWBQmOYP5+xg+b1m+WHHBGcHzJkzZ2jZsiXb\ntm3jnnvuoW3btkGXGTZsGL179yYyMpJjx47x+uuvU6ZM1qMUq1ev5tSpU1x55ZVFypdfngwk7v22\nRwIfA4mqesodOB4Tkb/h3Ge7C7A50PIichfOtxmqV7+UR2MySih58busovMXMhyFc3aw/F6z/E7N\nSqapU6eSnp7OmDFjaNy4MVdccQXgHEP5/PPPqVLFuVHn8uXLqV69OvPmzSMtLY0hQ4bw0ksv+Xq/\nfvzxR+6//35GjRrFihUripQvv7zs2roW2IOzS+tTVV3i3hv7vzi3WV0JBPyvpKov4OwOo079Bjp5\nQ/hWhj0Yk0G45g/n7GD5vWb5YUf/hBzT1q5dy48//sjgwYMBuOCCC2jfvj3Vq1cHYOLEiYwaNYqO\nHTsCMGvWLC699FLatGnD0aNHSUhIYPLkyfz+978vUrYCUdUSf+AcB9kE1AF+AGoGmGcecF1e62rY\nsKGGs2XLlnkdodDCObuq5fea5Xfs379fDx06pKqqJ06c0A4dOuh7773ne71u3bp64MAB3/OhQ4fq\nY489pqqqe/fu1cjISD1w4ID+8ssvevXVV+uUKVPy9b7AGi2m3+klfvqviAjwHHCfqv4ATAQmiUhZ\nEbnEnScWiAWWlHQ+Y4wpSXv27KFLly7ExsbSunVrunXrRs+ePZk2bRq1a9dm165dxMbGMmTIEADG\njBnDf//7X2JiYujatStPP/001atX54033mDFihW88sorxMXFERcXR0pKSolsgxffK+8EflDVT93n\n/wAG4Rx0f84ZZzgKDFDnwLsxxpyzYmNj+frrr3NMv/fee7n33ntzTI+MjGTJkpz/xh4wYAADBgwI\nSca8eHH6r+/4hvv8DNDSfdq0pPMYY4wpGruy3RhjTJHYQGKMMaZIbCAxxhTZzp076dKlC02aNCEq\nKopnn30WcA4Mx8bGEhcXR2JiImlpaYBz/UTPnj19B4Uff/zxLOs7c+YMzZs3p2fPniW+LabgSlvX\nlgIvAZe7P1+nqjtKOqMxpmDKlSvH5MmTadGiBceOHaNly5Z069aNkSNH8sQTTwAwbdo0Hn/8cWbO\nnAlATEwMK1euDLi+Z599liZNmnD0qDUkhYMS/0aiqjtxTv+d4E6aALygqt8Dc4GJqtoEaAPsL+l8\nxpiCq1mzJi1atADgoosuokmTJuzevZvKlSv75jl+/DjuWZm52rVrFx988IHvdFdT+pWari0RaQqU\nyzwtWFXT87Mi69ryTjhnB8tfXDL7onzPd+zg66+/9vVFjR49mrlz51KlShWWLVvmm2/z5s00a9aM\nyMhIJk2aRFRUFAD33XcfzzzzDMeOHSu5jTBF4skxElU9DYzEGVDuU9VTQEPgsIgsEpGvRWSiiJT1\nIp8xpnDS09NJSkpi6tSpvm8jTz75JDt37qR///5Mnz4dgBYtWrBgwQLWrVvHn//8Z66//noA3n//\nfWrUqEHLli2DvocpfUpN15abpSPQHKc25XWcCxVnZV/QShtLh3DODpa/uGQWD2ZkZPDwww/Ttm1b\nqlWrlqWQEOCKK67g4YcfpkuXLoBzQD05OZkLL7yQY8eOsXjxYt544w2WLFnCokWLOHXqFCdOnKBb\nt26MHj26hLcqb+np6Tm28bxVXF0rBXkQoGsLaAck+81zKzAjr3VZ15Z3wjm7quUvTmfPntVbb71V\nhw8fnmX6li1bfD9PmzZNk5KSVFV1z549+tlnn6mq6hdffKGXX365nj17Nsuyy5Yt0x49eoQ4eeGV\nps+/MCjGri0vztrK0rUlIhOBScBAoKqIXKqqB4CrgTUlnc8YU3Cff/45r776KjExMcTFxQHw17/+\nlVmzZpGamkqZMmWoW7eu74ytN998k0mTJlGlShUqVqzIggUL8nUg3pROpa1rawSw1B1s1gIvepDP\nGFNAHTp0yNyTkMV1110XcP5hw4YRHR2d4zay/hISEnJ93ZQepa1rC5zWX2OMMWHCrmw3xhhTJDaQ\nGGOMKRIbSIwxBVbQbq1vv/2W+Ph4KlSowKRJk7Ks6/bbb6dGjRpER0eX+HaY4uHFHRIvF5HtIlLN\nfV7VfV7XfV5ZRHaLyPSSzmaMyZ/Mbq1vvvmGVatWMWPGDDZv3szIkSNZv349KSkp9OzZ01fGWK1a\nNaZNm8aIESNyrGvQoEF8/PHHJb0JphiVtq4tgCeA5SWdyxiTfwXt1qpRowatW7emfPnyOdbVqVMn\nqlWrVjLBTUiUmq4tABFpCVwGfAy0ys+KrGvLO+GcHSx/YRW2W8ucu0pN15aIlAEmu9ONMWEgv91a\n5txWmrq2/gR8qKo787rC1bq2Sodwzg6Wv7AK260FzreXihUrkpycnKWrau/evRw/fjysuqusa8tP\ncXWtFORB4K6t19yfdwAHgaPAhLzWZV1b3gnn7KqWvygK2q2V6bHHHtOJEyeqatb827dv16ioqNAF\nDoFw//vDudi1par9/eYZBLRS1VElnc8Yk7eCdmvt3buXVq1acfToUcqUKcPUqVN5/vnnAejXrx/J\nyckcPHiQ2rVrM27cOO644w7Pts0UXKnp2hKRzqpqZ2sZEwYK2q3129/+ll27dmWZlrlbaP78+cWe\nz5Ss0ti1haq+ArxSosGMMcYUil3ZbowxpkhsIDHGGFMkNpAYY4wpEhtIjDlHBStW/Omnn+jWrRtX\nXXUV3bp149ChQwBMnDiRuLg44uLiiI6OpmzZsvz0008A1KtXz3eGVqtW+SqdMOeRkAwkuRQzdhaR\nlSKySUTWi8gf/JYZJiLbRERFpHoochlzPglWrDhhwgS6du3K1q1b6dq1KxMmOLV3I0eOJCUlhZSU\nFJ566ik6d+6cpQNr2bJlpKSksGaN3QHbZBWSs7bUuTo9s5jxLvfPF3CuZB+oqltFJBKnb+sTVT0M\nfA68DyQX5L2sa8s74Zwdzu38Oyb0oGbNmtSsWRPIWqy4ePFi36m3t912GwkJCTz99NNZlp8/fz79\n+vULaX5z7gjlrq0pQDu/YsbJqrpFVbcCqGoasB+41H3+taruCGEeY85b/sWK+/bt8w0wNWvWZP/+\n/VnmPXHiBB9//DFJSUm+aSJCYmIiLVu25IUXXsAYfyG7jkRVT4vISJwm30RVPeX/uoi0AX4D/K+g\n67aurdIhnLPDuZ3fvwPq5MmTDB8+nCFDhvDVV1+RkZGR5fXszz/77DMaN27M+vXrfdMmTpxI9erV\nOXToECNGjODkyZM0a9asSPnDvasq3PMXq+LqWgn0AKYCacD92abXBFKBdgGW2QFUz+97WNeWd8I5\nu+r5kf/UqVOamJiokydP9k1r2LChpqWlqapqWlqaZv9/6Prrr9fXXnst6Dr9+7KK4nz4/EszirFr\nK2S7tkQkDugGtAPuF5Ga7vTKwAfAI6q6KlTvb8z5TlW54447aNKkCQ888IBveu/evZkzZw4Ac+bM\noU+fPr7Xjhw5wvLly7NMO378OMeOHfP9vGTJErstrskiJLu2ghUzishg4G1grqouDMV7G2McwYoV\nR40axU033cSsWbOoU6cOCxf++r/i22+/TWJiIpUqVfJN27dvH3379gWc3WC33HIL11xzTclujCnV\nQnWMJGAxI/Aw0Am4xG34BRikqikici/wEPBbYL2IfKiqQ0KUz5hzXrBiRYClS5cGnD5o0CAGDRqU\nZVr9+vVZt25dcccz55BQnf6bWzHjuCDLTAOmhSKPMcaY0LEr240xxhSJDSTGGGOKxAYSY8LM7bff\nTo0aNRg8eLBv2rp164iPjycmJoZevXpx9OhR32vr168nPj6eqKgoYmJi+PnnnwFYu3YtMTExNGjQ\ngHvvvTfo8RRj8hLK03+D9W3VFZGPReSwiLyfbZlZIrLO7eF6U0QiQpXPmHA1aNAgPv744yzThgwZ\nwoQJE9iwYQN9+/Zl4sSJgHOW1YABA5g5cyabNm0iOTmZ8uXLA3D33XfzwgsvsHXrVrZu3Zpjncbk\nV8gGElXdiXMK8AR30gTgBVX9HpgI3BpgsftVtZmqxgI/AMNClc+YcNWpU6csZYoAqampdOrUCYBu\n3brx1ltvAbBkyRJiY2N9V6FfcskllC1blj179nD06FHi4+MREQYOHMg777xTshtizhmhvtXuFJxi\nxsy+rT8DqOpSEUnIPrOqHgXfdSgVgTy/a1tpo3fCOTuEX/4dE3oEfS06Opp3332XPn36sHDhQnbu\n3AnAli1bEBG6d+/OgQMHuPnmm3nooYfYvXs3tWvX9i1fu3Ztdu/eHfJtMOemkA4kmkffViAiMhu4\nDtgMPBhkHuvaKgXCOTuEX37/Xqe9e/dy9uxZ37ShQ4cyfvx4Ro4cSfv27SlTpgzJycmkpqbyr3/9\ni5kzZ1KhQgUefPBBypYtS6VKlTh06JBv+fXr1/PTTz+VaHdUuHdVhXv+YlVcXSvBHgTv20oA3g+y\nTFmcixgH57V+69ryTjhnVw3v/Nu3b9d69eoFfC01NVVbt26tqqrz58/X2267zffa448/rs8884ym\npaVpo0aNfNPnzZund911V0gzZxfOn79q+OcnHLq2IHjfVl7UuYDxdSApr3mNMfiq4M+ePcv48eMZ\nOnQoAN27d2f9+vWcOHGCjIwMli9fTtOmTalZsyYXXXQRq1atQlWZO3duln4tYwoilGdtZenbwjnA\nPim3+UWkgd+yvYBvQ5XPmHDVr18/4uPj2blzJ7Vr12bWrFnMnz+fhg0b0rhxYyIjI32nBletWpUH\nHniA1q1bExcXR4sWLejRwznW8txzzzFkyBAaNGjAlVdeybXXXuvlZpkwFspjJAH7tkSkMzAeaAxE\niMgu4A7gU2CO2w4swDrg7hDmMyYszZ8/H3COmSQkJPimDx8+POD8AwYMYMCAATmmt2rVio0bN4Yk\nozm/hPLGVrn1bXUMslj7UOUxxhgTGnZluzHGmCKxgcQYY0yR2EBiTBjJ7Nnyv0NhsJ6tHTt2ULFi\nReLi4oiLi/OdyQWQkJBAo0aNfK9lnvVlTGF41bW1VkRSRGSTiAz1W6aliGwQkW0iMs09e8sY4ypI\nzxbAlVdeSUpKCikpKcycOTPLcq+99prvtRo1apRIfnNuKvGuLWAP8H+qGge0BUaJSKQ7z3M4V6xf\n5T7sfp7G+ClIz5YxJaXEu7Y0a01KBdzBzL1YsbKqrnSfzwWuBz7K7Q2sa8s74Zwdwi9/sK6tYD1b\nANu3b6d58+ZUrlyZ8ePH07HjrydMDh48mLJly5KUlMQjjzyC7QAwheVJ15aIXA58ADQARqpqmoi0\nAnb5Lb4LqBVovda1VTqEc3YIv/yZvU579+7l+PHjvq6nYD1bp06dYt68eVSpUoXU1FSSkpKYPXs2\nlSpV4p577uHSSy/lxIkTPPbYY5w4cYLu3buX6PaEe1dVuOcvVsXVtRLsQZCuLfe1SGA1cBnQGviX\n32sdgffyWr91bXknnLOrhm/+7du3a1RUVMD8/j1b2XXu3Fm//PLLHNNnz56t99xzT3HHzFO4fv6Z\nwj0/50rXlqqmAZvcQWMXUNvv5druAGSMyUWwnq0DBw5w5swZAL777ju2bt1K/fr1ycjI4ODBgwCc\nPn2a999/P8tZYMYUVIl3bYlIbRGp6M5TFedq9lRV3QMcE5F27rIDgcWhymdMOMrs2UpNTeX3v/99\nrj1bK1as8N3U6sYbb2TmzJlUq1aNX375he7duxMbG0tcXBy1atXizjvv9HjLTDgr8a4tnF6tJBFR\nnE6tSaq6wZ3nbuAVnJtafUQeB9qNOd9k9mxB1q6tQD1bSUlJJCXlLNCuVKkSa9euDVlGc/7xqmtr\nXJBl1gD2HdsYY8KIXdlujDGmSGwgMcYYUyQ2kBhTygXr17rnnnty9Gt9+umntGzZkpiYGFq2bMln\nn32WY329e/e2s7RMsQrJQJJLz1ZnEVnpdmytF5E/+C1ztYh8JSIbRWSOiIT6qntjwkKwfq0777wz\nR79W9erVee+999iwYQNz5szh1ltvzbLcokWLiIiIKLHs5vwQkoFEc+/ZGqiqUTg9WlNF5GIRKQPM\nAW5W1Wjge+C2UGQzJtwE69dq1qwZkLVfq3nz5kRGOtV1UVFR/Pzzz/zyyy+AcyX23/72Nx555JES\nTG/OB6H8V3+uPVvq1KLsBy4FygO/qOoW9+VPgYeBWXm9iXVteSecs0N45M+tX+vzzz+nS5cuOfq1\nMr311ls0b96cChUqADBmzBgefPBBLrzwwpBmNuefUJ7+G7BnK5OItAF+A/wPUKC8iLRyTwG+Ebg8\n2Lqta6t0COfsEB75s/drZT4fOnQoU6ZMYe7cuVn6tTJt376dRx55hGeeeYbk5GS2bdvGF198QZ8+\nfVi1alWWdXkl3Luqwj1/sSqurpVAD4L0bAE1gVSgnd+0eODfON1b44Gv8/Me1rXlnXDOrhpe+TP7\ntfxl5s/er7Vz50696qqr9D//+Y9v2j/+8Q+tWbOm1q1bV2vVqqXly5fXzp07l0T0oMLp8w8k3PMT\nDl1bwXq2RKQyTvPvI6q6ym9AW6mqHVW1DbAC2BqqbMaEu2D9WocPH6ZHjx489dRTtG/f3jf/3Xff\nTVpaGjt27OA///kPDRs2tH9Nm2ITqrO2gvVs/QZ4G5irqguzLVPD/bMC8Bcg6+3cjDlP+fdr1a5d\n29evdeutt+bo15o+fTrbtm3jiSeesNvomhITqmMkwXq2HgY6AZeIyCD3tUGqmgKMFJGeOIPbc6qa\n8wR4Y85D/v1a/po1a+br2sr0yCOP5HlWVr169di4cWNxxTOm4AOJ29h7uaquDzaPFq5nayQwsqB5\njDHGeCtfu7ZEJFlEKrsXGK4DZovI30IbzRhjTDjI7zGSKqp6FLgBmK2qLYHfhS6WMcaYcJHfgaSc\ne9bVTcD7IcxjTIkL1GU1duxYatWq5Ttg/eGHH/pee+qpp2jQoAGNGjXik08+8U2fMmUKUVFRREdH\n069fP37++ecS3Q5jvJLfgeRx4BPgf6r6pYjUJ5fTc3Pp2qorIh+LyGEReT/bMta1ZTwRqMsK4P77\n7yclJYWUlBSuu+46ADZv3syCBQvYtGkTH3/8MX/60584c+YMu3fvZtq0aaxZs4aNGzdy5swZFixY\nUNKbYown8jWQqOpCVY1V1bvd59+pas5br/06f8CuLVX9HudU4CxNcta1ZbwUqMsqmMWLF3PzzTdT\noUIFrrjiCho0aMDq1asByMjI4OTJk2RkZHDixAlf55Ux57p8/atfRBriDAyXqWq0iMQCvVV1fC6L\n5ejaAlDVpSKSkG3eS7CurbATztnByZ+Qy+vTp09n7ty5tGrVismTJ1O1alV2795Nu3btfPPUrl2b\n3bt3Ex8fz4gRI6hTpw4VK1YkMTGRxMTEkG+DMaVBfncfvYhzau7zAKq6XkTm4VSZBKR5dG1lcxDr\n2go74ZwdnPzBuqxiY2OZNWsWIsLLL7/MLbfcwl/+8hd27drFN99845tvz549bNq0iQoVKjBnzhz+\n+c9/EhERwdixYxk9ejTdunULWf5w73qy/OeO/A4kF6rqaueCdZ/8/Aa5Fqc6PhrnW0ZAqqoicjMw\nxb2yfUlu6/e/TqVRo0b65/598hGldEpOTuambBeVhYtwzg5O/swL+nbs2EGlSpVyXOAHUL9+fXr2\n7ElCQgIrV64E8M331FNPkZiYyK5du2jevDnXX389AGlpaaxatSrg+kKRPxxZ/nNHfg+2HxSRK3Fa\nehGRG3EGiKCCdW0FY11bpjTZs+fXv95vv/2274yu3r17s2DBAn755Re2b9/O1q1badOmDXXq1GHV\nqlWcOHECVWXp0qU0adLEq/jGlKj8fiO5B+cbQGMR2Q1sB/oHmzl715aITAQm5bFMDVXd79e19WQ+\nsxlTJP369SM5OZmDBw9Su3Ztxo0bR3JyMikpKYgI9erV4/nnnwecm0XddNNNNG3alHLlyjFjxgzK\nli1L27ZtufHGG2nRogXlypWjefPm3HXXXR5vmTElI8+BxD2jqpWq/k5EKgFlVPVYHosF7NoSkc44\nx1UaAxEisgu4Q1U/wbq2jEcCdVndcccdQecfPXo0o0ePzjF93LhxjBsXsAHImHNangOJqp4VkWHA\nG6p6PD8rzaNrq2OQZaxryxhjwlB+j5F8KiIj3AsNq2U+QprMGGNMWMjvMZLb3T/v8ZumQP3ijWOM\nMSbc5PfK9isCPGwQMfkWqM/qp59+olu3blx11VV069aNQ4cOAXDo0CH69u1LbGwsbdq0yXLvjGef\nfZbo6GiioqKYOnVqiW+HMSan/NbIDwz0yGOZ3Pq21opIiohsEpGhfss8KSI7RSS9aJtlSptAfVYT\nJkyga9eubN26la5duzJhgtOo89e//pW4uDjWr1/P3LlzGT58OAAbN27kxRdfZPXq1axbt47333+f\nrVvtLHHhQMxcAAAgAElEQVRjvJbfYySt/R4dgbFA79wWCNa3hXP9yf+pahzQFhglIpmlRO8BbQqQ\n34SJQH1Wixcv5rbbnEq12267jXfeeQdwihG7du0KQOPGjdmxYwf79u3jm2++oV27dlx44YWUK1eO\nzp078/bbb5fshhhjcsjXMRJV/bP/cxGpAryaj0Vz9G1lq0qpgN9gpqqr3PXnJxZgXVteym/2HRN6\nBJy+b98+atZ0rlOtWbOm797izZo1Y9GiRXTo0IHVq1fz/fffs2vXLqKjoxk9ejQ//vgjFStW5MMP\nP6RVq1bFt0HGmEIpbFX7CeCqvGYK1rclIpcDHwANgJGqmlaQN7eurdIhv9mD9VllZGRk6SrKfN6+\nfXumT59OgwYNqF+/Pg0aNODrr7+mQYMG9OnTh/j4eCpWrEjdunXZu3dvofuOwr0ryfJ7K9zzFytV\nzfOBs8vpXffxPvAd8HQ+l50KpAH3B3gtEliN0yrsPz09P+tWVRo2bKjhbNmyZV5HKLSCZt++fbtG\nRUX5njds2FDT0tJUVTUtLU0D/bc8e/as1q1bV48cOZLjtYcfflhnzJhRsNB+wvmzV7X8Xgv3/MAa\nzefv2bwe+T1GMgmY7D6eAjqp6l/yWiivvi11volsIshFiubc1rt3b+bMmQPAnDlz6NPHKd88fPgw\np045e0BfeuklOnXqROXKlQF8u79++OEHFi1aRL9+/TxIbozxl99dW9dlHzhE5OncBpNgfVsi8hfg\nR1U9KSJVgfbA3wqZ34SJQH1Wo0aN4qabbmLWrFnUqVOHhQsXAvDNN98wcOBAypYtS9OmTZk169fb\n0iQlJfHjjz9Svnx5ZsyYQdWqVb3aJGOMK78DSTecIkV/1waY5i9g3xZwB5AkIgoIMElVNwCIyDPA\nLcCFbg/XS6o6Np8ZTSkWqM8KYOnSpTmmxcfHBz2t99///nex5jLGFF2uA4mI3A38CagvIuv9XroI\n+Dy3ZTX3vq2AzXaq+hDwUN6xjTHGlBZ5fSOZB3yEc1xklN/0Y6r6U8hSGWOMCRu5DiSqegQ4AvQD\n554hwAU4FfARqvpD6CMaY4wpzfJbkdJLRLbi3NBqObAD55uKMQH7r9atW0d8fDwxMTH06tWLo0eP\nAs4tbStWrEhcXBxxcXEMHTo0t1UbY8JAfk//HY9zCu8WVb0C6Eoux0hy6dnqLCIr3Y6t9SLyB79l\n/u32b6WISJqIvFOE7TIlJFj/1ZAhQ5gwYQIbNmygb9++TJw40bfMlVdeSUpKCikpKcycOdPD9MaY\n4pDfgeS0qv4IlBGRMqq6DIgLNrPm3rM1UFWjgGuAqSJysbtMR1WNU6eDayWwqFBbZEpUsP6r1NRU\nOnXqBEC3bt146623PE5qjAmV/J7+e1hEIoB/A6+JyH4gr26MXHu2VDXNXc+lwOHM6SJyEXA1MDg/\nwaxryzuvXFMpaP9VdHQ07777Ln369GHhwoXs3LnTt9z27dtp3rw5lStXZvz48XTsaNejGhPOxLlS\nPo+ZnHu1n8T5BtMfqAK85n5LyW257vzas/VpttfaAHOAKFU96zd9INBbVW/MZb3+XVstH536Yp7b\nUFpdVhH2nfQ6ReFcUaUsERERfPDBByxevNjXf1WhQgV69erF3//+d44cOUL79u1ZtGgRixcv5tSp\nU5w8eZIqVaqQmprKmDFjmD17NpUqVSrx/Onp6URERJT4+xYXy++tcM/fpUuXtapaPK2n+e1SAeoC\nv3N/vhC4KB/LBOzZAmoCqUC7AMt8BCTlN5d1bXknUPZA/VepqanaunXrgOvo3Lmzfvnll6GIl6dw\n/uxVLb/Xwj0/Jd21JSJ3Am8Cz7uTagG5HgwP1rMlIpVxmn8fUbc23m+ZS3DuRxKe+3rOU4H6rzKn\nnT17lvHjx/vOzjpw4ABnzpwB4LvvvmPr1q3Ur2832zQmnOX3YPs9OJ1YRwFUdStQI9jM2Xu2gMye\nrd8AbwNzVXVhgEV/D7yvqj/nfxOM15KSkmjatCm9evXy9V/Nnz+fhg0b0rhxYyIjIxk82DnktWLF\nCmJjY2nWrBk33ngjM2fOzHHDK2NMeMnvwfZfVPVU5g2nRKQckNvBlWA9Ww8DnYBLRGSQ+9ogVU1x\nf76ZX8/0MmEiUP/V8OHDfbfI9ZeUlERSUlJJxDLGlJD8DiTLReT/ARVFpBtO/9Z7wWbWQvRsufMl\n5DOPMcaYUiK/u7ZGAQeADcAfgQ+BR0IVyhhjTPjIq/23jqr+oM7puS+6D2OMMcYnr28kvjOzRMQu\nTTY5FKRn68cff6RLly5EREQwbNgwL2MbY4pRXgOJ+P1cLOdo5tLDVVdEnnF7uL4RkWmSeXTflErb\nt28vUM/WBRdcwBNPPMGkSZM8Tm6MKU55DSQa5OdC0+A9XLVwTjGOBaKB1kDn4nhPExrff/99gXq2\nKlWqRIcOHbjgggu8jG2MKWZ5nbXVTESO4nwzqej+jPtcVbVyId83Rw8XzlldFwC/cddfHtiX14qs\na8s745pcwbx58wrUs2WMOffkdWOrsqF4U1U9LSIj+bWH6xSwUkSW4TQECzBdVb8JtHy2ri0ejcmr\nP7L0uqyiM5iEo0uqXEKfPn2Ij4/39Wzt3buXoUOHMn78eEaOHEn79u0pU6YMycnJvuW+/fZbdu/e\nnWWaF9LT0z3PUBSW31vhnr9YFVfXSkEfZOvhAhrgVKNEuI+VQKe81mNdW97Jnj2/PVuzZ8/We+65\nJ9Tx8hTOn72q5fdauOenpLu2iluQHq6+wCpVTVfVdJzyxnZe5DP5V5CeLWPMuanEB5JgPVzAD0Bn\nESknIuVxDrQH3LVlSo+C9GwB1KtXjwceeIBXXnmF2rVrs3nzZg/TG2OKQ34rUopTsB6uF4D/4Vw9\nr8DHqhq0hsWUDgXp2QLnnu3GmHNLiQ8kmnsP1/KSzmOMMaZoPDlGYowx5txhA4kxxpgisYHEMGXK\nFKKiooiOjqZfv378/PPP9O/fn0aNGhEdHc3tt9/O6dOnAThy5Ai9evWiWbNmDBo0iNmzZ3uc3hjj\ntZANJHl0an0sIodF5P0gy/5dRNJDlc38avfu3UybNo01a9awceNGzpw5w4IFC+jfvz/ffvstGzZs\n4OTJk7z00ksAzJgxg6ZNm7Ju3TqmTp3Kgw8+yKlTpzzeCmOMl0I2kGiQTi1V/R7nlN9bAy0nIq2A\ni0OVy+SUkZHByZMnycjI4MSJE0RGRnLdddchIogIbdq0YdeuXQCICMeOHUNVOXnyJNWqVaNcOS9O\n/jPGlBah/g0QqFMLVV0qIgnZZxaRsjiDzC04Fyjmybq2Cm/HhB7UqlWLESNGUKdOHSpWrEhiYiKJ\niYm+eU6fPs2rr77Ks88+C8CwYcPo3bs3kZGRHD58mDfffJMyZWwPqTHns5AOJBq4Uys3w4B3VXVP\nbg3y1rVVPJKTkzl27Bhz5szhn//8JxEREYwdO5bRo0fTrVs3ACZNmkT9+vU5c+YMycnJLF++nOrV\nqzNv3jxfZfxLL71EpUqVPNmGogj3riTL761wz1+cSmKfxLU4RYzRwKfBZhKRSOD3QEJeK/S/FqVO\n/QY6eUP47lp5MCYDr/Lv6J/AwoULad68Oddffz0AaWlprFq1ioSEBMaNG0e5cuV44403fN86Jk6c\nyKhRo+jYsSMiQpMmTbj00ktp06aNJ9tQFMnJySQkJHgdo9Asv7fCPX9xCulvsGydWv8RkQWquifI\n7M1xihu3ud9GLhSRbaraILf3qFi+LKkTehRn7BKVnJzMjv4Jnr1/nTp1WLVqFSdOnKBixYosXbqU\nVq1a8dJLL/HJJ5+wdOnSLLuu6tSpw9KlS+nYsSM//fQTqamp1K9fLPc8M8aEqZANJNk7tUQks1Or\nf6D5VfUD4Ld+y6fnNYiYomvbti033ngjLVq0oFy5cjRv3py77rqLSpUqUbduXeLj4wG44YYbePTR\nRxkzZgyDBg0iJiaG9PR0nn76aapXr+7xVhhjvBTKbyQBO7VEpDMwHmgMRIjILuAOVf0khFlMLsaN\nG8e4ceOyTMvICHzcJjIykiVLlgD21d4Y4wjZQJJHp1bHfCwfEaJoxhhjipGdt2mMMaZIbCAxxhhT\nJDaQhJF69eoRExNDXFwcrVq1AmDdunXEx8cTExNDr169OHr0aJZlfvjhByIiIpg0aZIXkY0x5wEv\n7pAYrIPrNhFJ8Xv8LCLXl3S+0m7ZsmWkpKSwZs0aAIYMGcKECRPYsGEDffv2ZeLEiVnmv//++7n2\n2mu9iGqMOU+U+ECSSwfXHFWNU9U44GrgBLCkpPOFm9TUVDp16gRAt27deOutt3yvvfPOO9SvX5+o\nqCiv4hljzgNeXRIesIPLz43AR6p6Iq8VnQ9dWzvcCy5FhMTERESEP/7xj9x1111ER0fz7rvv0qdP\nHxYuXMjOnTsBOH78OE8//TSffvqp7dYyxoSUqKo3byzSnV87uD7N9tpnwN9UNVjNvH/XVstHp74Y\n6rghc1lF2Hcy93lialUB4ODBg1SvXp1Dhw4xYsQI7r33XqpWrcrf//53jhw5Qvv27Vm0aBGLFy/m\nueeeo3HjxnTp0oVXXnmFihUr8oc//KFYs6enpxMREb5naVt+b1l+b3Xp0mWtqrYqjnV5WVIVsINL\nRGoCMUDQCxTPt66tQBUq69at4/Tp0wwcOJCBAwcCsGXLFjZt2kRCQgJjxozhiy++YM6cORw+fJgy\nZcoQFRXFsGHDii17uF+QaPm9ZfnPHZ78Bs6jg+sm4G1VPZ2fdZ0vXVvHjx/n7NmzXHTRRRw/fpwl\nS5bw6KOPsn//fmrUqMHZs2cZP348Q4cOBeDf//63b9mxY8cSERFRrIOIMcZk8uKsrSwdXDj3H/Hf\nid8PmF/SuUq7ffv20aFDB5o1a0abNm3o0aMH11xzDfPnz6dhw4Y0btyYyMhIBg8e7HVUY8x5xotv\nJLl1cH0PXA4s9yBXqVa/fn3WrVuXY/rw4cMZPnx4rsuOHTs2RKmMMcaDgSSPDi6AWiWdyRhjTOHZ\nle3GGGOKxAYSY4wxRWIDSSkVqFdr7Nix1KpVi7i4OOLi4vjwww998z/11FM0aNCARo0a8ckndmsX\nY0zJCeUdEi8HVgAtVfUnEakKfAUMwqlFqQycAZ5U1dezLft3YPD5fk+SZcuW5bj74P3338+IESOy\nTNu8eTMLFixg06ZNpKWl8bvf/Y4tW7ZQtmzZkoxrjDlPhewbSbBOLZyLEAeqahRwDTBVRC7OXE5E\nWgEXY/Jt8eLF3HzzzVSoUIErrriCBg0asHr1aq9jGWPOE6E+aytHp5aqnsp8UVXTRGQ/cClwWETK\n4lxXcgvQNz9vcK51beXWqwUwffp05s6dS6tWrZg8eTJVq1Zl9+7dtGvXzreO2rVrs3v37pLdEGPM\neSvkXVt5dGq1AeYAUap6VkSGA2VUdYqIpAfbtXUud23l1qt1+eWXU6VKFUSEl19+mR9//JG//OUv\nTJ06laioKLp16wbAM888Q9u2bencuXNIs4d715Dl95bl91a4dW3l1qn1KnCbO4hEAr8HEvJa4bnc\ntZVbr9YNN9zgm1a/fn169uxJQkICK1euBPD1/jz11FMkJiYSHx8f0uzh3jVk+b1l+c8dIf0NHKxT\nS0QqAx8Aj6jqKnf25kADYJvTosKFIrJNVRvk9h7nYtdWsF6tPXv2ULNmTQDefvttoqOjAejduze3\n3HILDzzwAGlpaWzdupU2bdqU9KYYY85ToTxrK0unlohMBCaJyGDgbWCuqi7MnF9VPwB+67d8el6D\nyLlq37599O3rHCLKyMjglltu4ZprruHWW28lJSUFEaFevXo8//zzAERFRXHTTTfRtGlTypUrx4wZ\nM+yMLWNMiQnlN5KAnVrAw0An4BIRGeS+NkhVU0KYJawE69V69dVXgy4zevRoRo8eHcpYxhgTUMgG\nkjw6tcblY/nwPYpljDHnEbuy3RhjTJHYQGKMMaZIbCAphQL1bI0ZM4bY2Fji4uJITEwkLS3NN39y\ncjJxcXFERUWF/NoRY4zJzos7JF4uIttFpJr7vKr7vK6InBGRFPfxbklnK02WLVtGSkoKa9asAWDk\nyJGsX7+elJQUevbsyeOPPw7A4cOH+dOf/sS7777Lpk2bWLhwYW6rNcaYYufFja12ikhmB9dd7p8v\nqOr3InJSVeNKOlM4qFy5su/n48eP415rw7x587jhhhuoU6cOADVq1PAknzHm/OXVJeE5OrgKu6Jz\nqWsrr56t0aNHM3fuXKpUqcKyZcsA2LJlC6dPnyYhIYFjx44xfPhwBg4c6M3GGGPOS54cI1HV08BI\nnAHlPr8ixwtEZI2IrBKR673IVhp8/vnnfPXVV3z00UfMmDGDFStWAPDkk0+yc+dO+vfvz/Tp0wHn\ngsW1a9fywQcf8Mknn/DEE0+wZcsWL+MbY84zXpZUBergquM2AtcHPhORDar6v+wLZitt5NGYjJLK\nXOwuq+h8KwHnoHmmzMGgefPmzJ8/n7Nnz/peu+KKK3j44Yfp0qULp06donHjxnz55ZcAXHXVVcyb\nN69EOoDS09OzZA43lt9blv8coqol/gDigE1AHeAHoGaAeV4BbsxrXQ0bNtRwtmzZsizP09PT9ejR\no76f4+Pj9aOPPtItW7b45pk2bZomJSWpqurmzZv16quv1tOnT+vx48c1KipKN2zY4En2cGP5vWX5\nvQWs0WL6nV7i30hy6eAaBpxQ1V9EpDrQHnimpPN5LVjPVlJSEqmpqZQpU4a6desyc+ZMAJo0acI1\n11xDbGwsZcqUYciQIb4yR2OMKQle7NoK1sEVC0wXkbM4x24mqOpmD/J5KljP1ltvvRV0mZEjRzJy\n5MhQxjLGmKC8OP03tw6umJLOY4wxpmjsynZjjDFFYgOJMcaYIrGBpIB27txJly5daNKkCVFRUTz7\n7LMApKSk0K5dO18/1urVqwE4cuQIvXr1olmzZkRFRTF79mwv4xtjTLELyUCSS59WZxFZKSKbRGS9\niPzBb5nXRCRVRDaKyMsiUj4U2YqqXLlyTJ48mW+++YZVq1YxY8YMNm/ezEMPPcRjjz1GSkoKjz/+\nOA899BAAM2bMoGnTpqxbt47k5GQefPBBTp06lce7GGNM+AjJQKKqO3FO8Z3gTpqAc4B9DzBQVaOA\na4CpInKxO89rQGOcA+4VgSGhyFZUNWvWpEWLFgBcdNFFNGnShN27dyMiHD16FHC+hURGRgJO3cmx\nY8dQVdLT06lWrRrlynl5HagxxhSvUP5Gy9Gnpb9WoaDOFez7gUuBw6r6YeZrIrIaqJ2fNynJrq3M\nLizf8x07+Prrr2nbti1Tp06le/fujBgxgrNnz/Lf//4XgGHDhtG7d28iIyM5duwYr7/+OmXK2B5F\nY8y5I2S/0TR4nxYAItIG+A3wv2zTywO3Ah+HKltxSE9PJykpialTp1K5cmWee+45pkyZws6dO5ky\nZQp33HEHAJ988glxcXGkpaWRkpLCsGHDfN9cjDHmXCDOlfIhWrnIVOAmYKKqTvGbXhNIBm5T1VXZ\nlnkROK6q9+WyXv+urZaPTn0xBOlziqlVBXCuOH/44Ydp3bo1N910EwA9e/bkvffeQ0RQVXr27MkH\nH3zAqFGjuOWWW4iNjQXggQce4M4776RJkyaAMyBFRITn7enDOTtYfq9Zfm916dJlraq2KpaVFVfX\nSvYHQfq0gMrAV8DvAyzzGPAOUCa/71PSXVtnz57VW2+9VYcPH55leuPGjX3dO//617+0RYsWqqo6\ndOhQfeyxx1RVde/evRoZGakHDhzwLRfOfT3hnF3V8nvN8nuL0t61lUuf1mDgbWCuqi7MtswQoDvQ\nVVXP5lhpKfH555/z6quv+m6FC/DXv/6VF198keHDh5ORkcEFF1zACy84F++PGTOGQYMGERMTg6ry\n9NNPU716dS83wRhjilWoDrYH69N6GOgEXCIig9zXBqlqCjAT+B5Y6d79b5GqPh6ifIXWoUOHzG9P\nOaxduzbHtMjISJYsWRLqWMYY45mQDCSae5/WuCDL2DmxxhgThuw8VGOMMUViA4kxxpgisYHEGGNM\nkdhAEkCwYsaFCxcSFRVFmTJlWLNmTZZl1q9fT3x8PFFRUcTExPDzzz97Ed0YY0pcyAaSQhY3dhWR\nr0QkRUT+IyINQpUvN8GKGaOjo1m0aBGdOnXKMn9GRgYDBgxg5syZbNq0ieTkZMqXL5Wdk8YYU+xC\nWZFSmOLG54D+qhoHzAMeCVW+3AQrZmzSpAmNGjXKMf+SJUuIjY2lWbNmAFxyySWULVu2RDMbY4xX\nQn3KbYGKGwHFufIdoAqQltcbFHdpY27FjMFs2bIFEaF79+4cOHCAm2++2Vcjb4wx57qQDiSqelpE\nRuIUMCZq3sWNQ4APReQkcBRoF2i92bq2eDQmo9gyJycn+34+efIkw4cPZ8iQIXz11Ve+6YcPH2bt\n2rWkp6cDkJqayr/+9S9mzpxJhQoVePDBBylbtiwtW7bMvvoc0tPTs7xnOAnn7GD5vWb5zyHF1bUS\n7AFMxflmcX+26TWBVKCd37RFQFv355HAS3mtP1RdW6dOndLExESdPHlyjtc6d+6sX375pe/5/Pnz\n9bbbbvM9f/zxx/WZZ57J1/uEc19POGdXtfxes/zeohi7tkJ61paIxAHdcL5Z3O+2/iIilYEPgEfU\nbf8VkUuBZqr6hbv468D/hTJfMKrKHXfcQZMmTXjggQfynL979+6sX7+eEydOkJGRwfLly2natGkJ\nJDXGGO+F8qytLMWNQGZx428IXNx4CKgiIg3d592Ab0KVLzeZxYyfffYZcXFxxMXF8eGHH/L2229T\nu3ZtVq5cSY8ePejevTsAVatW5YEHHqB169bExcXRokULevTokce7GGPMuSGUx0gKXNwoIncCb4nI\nWZyB5fYQ5gsqt2LGvn37Bpw+YMAABgwYEMpYxhhTKoVsINHCFTe+jfNtxRhjTJiwK9uNMcYUiQ0k\nxhhjisQGEmOMMUViA4kxxpgisYHEGGNMkdhAYowxpkgk2PUS4UJEjuFUrYSr6sBBr0MUUjhnB8vv\nNcvvrUaqelFxrCjU7b8lIVVVW3kdorBEZE245g/n7GD5vWb5vSUia/KeK39s15YxxpgisYHEGGNM\nkZwLA8kLec9SqoVz/nDODpbfa5bfW8WWP+wPthtjjPHWufCNxBhjjIdsIDHGGFMkYTuQiMg1IpIq\nIttEZJTXeYIRkR0iskFEUjJPtxORaiLyqYhsdf+s6k4XEZnmbtN6EWnhQd6XRWS/iGz0m1bgvCJy\nmzv/VhG5zeP8Y0Vkt/vfIEVErvN77WE3f6qIdPebXuJ/v0TkchFZJiLfiMgmERnuTg+Lzz+X/OHy\n+V8gIqtFZJ2bf5w7/QoR+cL9LF93b86HiFRwn29zX6+X13Z5lP8VEdnu9/nHudOL7+9Pcd2ztyQf\nQFngf0B94DfAOqCp17mCZN0BVM827RlglPvzKOBp9+frgI8Awbk98Rce5O0EtAA2FjYvUA34zv2z\nqvtzVQ/zjwVGBJi3qft3pwJwhft3qqxXf7+AmkAL9+eLgC1uxrD4/HPJHy6fvwAR7s/lgS/cz/UN\n4GZ3+kzgbvfnPwEz3Z9vBl7Pbbs8zP8KcGOA+Yvt70+4fiNpA2xT1e9U9RSwAOjjcaaC6APMcX+e\nA1zvN32uOlYBF4t7n/uSoqorgJ+yTS5o3u7Ap6r6k6oeAj4Frgl9+qD5g+kDLFDVX1R1O7AN5++W\nJ3+/VHWPqn7l/nwM51bTtQiTzz+X/MGUts9fVTXdfVrefShwNfCmOz3755/53+VNoKuICMG3y6v8\nwRTb359wHUhqATv9nu8i97+wXlJgiYisFZG73GmXqeoecP7nA2q400vrdhU0b2ncjmHu1/eXM3cN\nUYrzu7tJmuP8qzLsPv9s+SFMPn8RKSsiKcB+nF+g/wMOq2pGgCy+nO7rR4BLKEX5VTXz83/S/fyn\niEiF7Pmz5Sxw/nAdSCTAtNJ6HnN7VW0BXAvcIyKdcpk3nLYLguctbdvxHHAlEAfsASa700tlfhGJ\nAN4C7lPVo7nNGmBaacwfNp+/qp5R1TigNs63iCa5ZCn1+UUkGngYaAy0xtld9Rd39mLLH64DyS7g\ncr/ntYE0j7LkSlXT3D/349yPvg2wL3OXlfvnfnf20rpdBc1bqrZDVfe5/4OdBV7k190MpS6/iJTH\n+SX8mqoucieHzecfKH84ff6ZVPUwkIxz7OBiEcnsJfTP4svpvl4FZ7dqacp/jbvLUVX1F2A2Ifj8\nw3Ug+RK4yj2b4jc4B7re9ThTDiJSSUQuyvwZSAQ24mTNPBPiNmCx+/O7wED3bIp2wJHMXRoeK2je\nT4BEEanq7sZIdKd5Ittxpr44/w3AyX+ze/bNFcBVwGo8+vvl7l+fBXyjqn/zeyksPv9g+cPo879U\nRC52f64I/A7nOM8y4EZ3tuyff+Z/lxuBz9Q5Wh1su7zI/63fP0IE5/iO/+dfPH9/iuNsAS8eOGcc\nbMHZhzna6zxBMtbHOXtjHbApMyfOftSlwFb3z2r661kXM9xt2gC08iDzfJzdD6dx/mVyR2HyArfj\nHGTcBgz2OP+rbr717v88Nf3mH+3mTwWu9fLvF9ABZxfCeiDFfVwXLp9/LvnD5fOPBb52c24EHnWn\n18cZCLYBC4EK7vQL3Ofb3Nfr57VdHuX/zP38NwL/5Nczu4rt749VpBhjjCmScN21ZYwxppSwgcQY\nY0yR2EBijDGmSGwgMcYYUyQ2kBhjjCmScnnPYsz5R0TO4JwSmel6Vd3hURxjSjU7/deYAEQkXVUj\nSvD9yumvfU7GhBXbtWVMIYhITRFZ4d7fYaOIdHSnXyMiX7n3hFjqTqsmIu+4pXmrRCTWnT5WRF4Q\nkSXAXLdwb6KIfOnO+0cPN9GYfLNdW8YEVtFtUQXYrqp9s71+C/CJqj4pImWBC0XkUpwuqU6qul1E\nquQHpNQAAAFASURBVLnzjgO+VtXrReRqYC5OgSFAS6CDqp5026GPqGprt6H1cxFZok4VuTGllg0k\nxgR2Up0W1WC+BF52SwrfUdUUEUkAVmT+4lfVzPuidACS3GmficglIlLFfe1dVT3p/pwIxIpIZq9T\nFZyeJhtITKlmA4kxhaCqK9xbAvQAXhWRicBhAtdt51bLfTzbfH9WVc8KLo0pDDtGYkwhiEhdYL+q\nvojTeNsCWAl0dhtf8du1tQLo705LAA5q4PuMfALc7X7LQUQauq3RxpRq9o3EmMJJAEaKyGkgHRio\nqgfc4xyLRKQMzn1DuuHcs3y2iKwHTvBr9Xh2LwH1gK/cyu8D/HpbV2NKLTv91xhjTJHYri1jjDFF\nYgOJMcaYIrGBxBhjTJHYQGKMMaZIbCAxxhhTJDaQGGOMKRIbSIwxxhTJ/wcHHsqDfRMyfQAAAABJ\nRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1a2f66d128>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot F-score of each feature using xgboost's .plot_importance() method\n",
    "plt.show(plot_importance(model))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interestingly, it looks like the loan subgrade is the most important feature in predicting interest loan rates with the amount requested and invester funded portion of the loans coming in at the third and second most important. The number of 30 day+ delinquencies in payments and the number of derogatory public records were the least important features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# write out csv for each model's test predictions\n",
    "linear_test_pred = pd.DataFrame(ridge_test_pred, columns=['RidgePredictions'])\n",
    "\n",
    "XGB_test_pred = pd.DataFrame(GBDT_test_pred, columns=['XGBPredictions'])\n",
    "\n",
    "test_predictions = pd.concat([linear_test_pred, XGB_test_pred], axis=1)\n",
    "\n",
    "test_predictions.to_csv(\"Ryan_Schaub_test_predictions.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discussion of Model Selection, Design, and Results:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ridge Regression Model:\n",
    "\n",
    "The first model I selected to predict interest rates was a ridge regression model using scikitlearn's RidgeCV() esitmator. Ridge regression is a multiple linear regression model, where multiple linear regression assumes that the target, numerical variable (interest rate) is a combined linear function of multiple numeric features. Ridge regression is advantageous to a base linear regression estimator in that it penalizes or regularizes (using the L2 norm) the model's coefficient values for specific features that correlate highly with other features or predictors, also known as multicolinearity. \n",
    "\n",
    "Scikit-learn's RidgeCV() object allows for the alpha parameter, or regularization strength parameter, to be optimized via cross-validation selection, where each of the alpha parameters given (I used the default parameter values) are tested on the training data and the alpha parameter that produces the highest accuracy scores will be stored in the model object for predictions. An advantage of using linear regression models is their relative ease of use, fast computation times, and straightforward deployment and maintenance in production environments.\n",
    "\n",
    "\n",
    "### Ridge Regression Performance:\n",
    "\n",
    "The RidgeCV regression model (with 10-fold cv technique used to optimize alpha parameter) produced an RMSE of 0.0090 when predicting on the training data. This RMSE implies that the root average of the model's squared residual errors in prediction is roughly the equivalent of plus or minus 0.90 of the the true value of the interest rate, since the interest rate was scaled down during preprocessing (divided by 100). \n",
    "\n",
    "A potential improvement to the linear regression model would be to attempt to regress on the interest rate using an elastic net regularization model, which combines L2 and L1 norm regularization methods in order to optimize coefficient values of features in detecting the true, combined feature signal. This may not always be a better solution, but, in production, it may be worth verifying.\n",
    "\n",
    "### Extreme Gradient Boosted Decision Tree Model:\n",
    "\n",
    "As discussed above, extreme gradient boosting is a very popular machine learning model being used ubiquitously in regression and classification problems, specifically in Kaggle-based competitions. The model's have proven to produce high performance measures as it uses a sequential series of \"weaker\" decision tree models to learn the data's signal using gradient descent on the loss function of each sub-model. The model then combines the accuracies of each model, using a weighted sum of predictions, to produce a best-fitting or highest-performing model. The speed of the models are valued as well.\n",
    "\n",
    "I specifically used XGBoost's python module xgboost as it provides relatively high customization options for hyperparameters and relatively fast processing with its built-in parellel processing when building decision trees. Many of the model's default parameters work well for this regression problem, but there were a few that I manually tuned via trial-and-error efforts. Specifically:\n",
    "\n",
    "* **max_depth:** the max_depth of each decision tree built. This is used to control over-fitting of the training data as higher depths will fit to more noise in the training data. Final value used: 6\n",
    "\n",
    "\n",
    "* **min_child_weight:** this is the sum of the weights defined for each child of each learner. This is used to also control overfitting of training data. (higher values == more regularization for each sample in a tree). Final value used: 1\n",
    "\n",
    "\n",
    "* **eta:** this is essentially the learning rate of the model or the alpha value when gradient descent occurs. This makes the model more robust by shrinking the weights on each step for gradient descent. Final value used: .1\n",
    "\n",
    "\n",
    "* **subsample:** the fraction of observations to be randomly sampled for each tree. Lower values make the algorithm more conservative and prevents overfitting but too small values might lead to under-fitting. Final value: 1\n",
    "\n",
    "\n",
    "* **colsample_bytree:** similar to subsample except it is the fraction of columns to be randomly sampled for each tree. Final value: 1\n",
    "\n",
    "\n",
    "* **num_boosting_rounds:** the number of iterations or trees built to produce the best accuracy measure when training the model and testing its accuracy on allocated test data. Final value: 441\n",
    "\n",
    "\n",
    "* **early_stopping_rounds:** this threshold tells the model how many test accuracy scores to look back upon during training iterations. If the testing scores don't improve after 10 consecutive iterations, the model will stop and take the best score. Final value: 10\n",
    "\n",
    "The hyperparameters of the model were tuned in a less-than-optimal way in exchange for computational efforts; that is, I manually tuned each of the above hyperparameters with a bit of intuition and trial-and-error efforts (running the model with different values set and choosing the best performing parameter values). A much more optimal way of tuning the model's parameters would be to do a cross-validation search of the best combination of parameters, most likely including choices for other available, tunable parameters as well. However, in the interest of computational time (and our time), I manually tuned the model instead. \n",
    "\n",
    "It should be noted that I trained the XGB model on only 90% of the training data as the other 10% was randomly sampled from the training data and allocated as test data to optimize the general fit of the model during training. This helps ensure that overfitting doesn't occur on the training data.\n",
    "\n",
    "### Extreme Gradient Boosting Performance:\n",
    "\n",
    "The RMSE of the XGB model on the training data was roughly 0.0066. This is lower than the ridge regression model's RMSE, but does come at the cost of computational time for training and predicting. It would be interesting to see if the above-mentioned improvement suggestions would allow for an increase in performance that may be worth the extra computational time in production. Although both models were regularized and optimized to prevent overfitting as much as possible, I would expect the RMSE values on the test set predictions from both models to be slightly higher than their respective training set RMSE's as the models are naturally going to predict better on the data that they were trained on.\n",
    "\n",
    "### Reflection and Improvement Suggestions:\n",
    "\n",
    "Although they do outperform some models I've seen on Kaggle, I believe both of the models created can be improved for better performance, which would be ideal for production. The extreme gradient boosting decision tree model outperforms the regression model with an RMSE of 0.0066 compared to RidgeCV's 0.0090, but it does come at the cost of computational time and effort. Furthermore, I believe my preprocessing techniques were limited to my inuitive (and learned) understanding of features that may influence interest rates of loans. I removed quite a few columns from the data that may have been useful indicators. The above plot of F-scores for each feature used in predicting shows that loan subgrades are highly correlated with interest rates while the number of delinquencies and derogatory public records were the least important features. \n",
    "\n",
    "Upon reflection, I would have given every feature its chance in predicting interest rates and used a more empirical of statistical techniques in evaluating features for preprocessing, like using extreme gradient boosting F-scores to detect and display the most important features. From there I could potentially choose more important features and omit less important features, as well as engineer new features out of highly important features to create stronger signals. For example, I would look futher into the relationship between loan subgrades and interest rate by generating a scatter plot of the two variables to discern if there is a specific relationship between the two and alter the loan subgrade feature or create a new feature using this feature to model the relationship better when predicing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}