Unit 2Assessment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Unit 2 Assessment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Akshit RamPershad- U87835413\n",
    "## Sanjana Thinderu- U16484810\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this assignment, we will focus on salary prediction. The data set for this assignment includes information on job postings. Use this data set to see if you can predict the salary of a job posting (i.e., the `Salary` column in the data set) based on the job description and other attributes of the job. This is important, because this model can make a salary recommendation as soon as a job posting is entered into a system."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Description of Variables\n",
    "\n",
    "The description of variables are provided in \"Jobs - Data Dictionary.docx\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Goal\n",
    "\n",
    "Use the **jobs_alldata.csv** data set and build models to predict **salary**.\n",
    "\n",
    "**Be careful: this is a REGRESSION task**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submission:\n",
    "\n",
    "Please save and submit this Jupyter notebook file. The correctness of the code matters for your grade. **Readability and organization of your code is also important.** You may lose points for submitting unreadable/undecipherable code. Therefore, use markdown cells to create sections, and use comments where necessary.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Important hints:\n",
    "\n",
    "* This assignment requires you to work with a text-based column in addition to regular numeric/categorical columns. So you will have to pay attention to your pipelines during data processing.\n",
    "* You can do your data prep before or after the train/test split. Regardless, you should use train_test_split only once. If you find yourself using it twice, it means you are doing something wrong.\n",
    "* Recommended approach: \n",
    "    * import the data and perform the train/test split - like we always do. \n",
    "    * identify the names of numeric, categorical, feature engineered, and text columns - like we always do\n",
    "    * create individual pipelines for each type of column - like we always do. For the text pipeline, I would recommend the TFIDF Vectorizer with SVDs. Though, you can also use TFIDF Vectorizer with top N terms (without SVDs).\n",
    "    * combine all pipelines using the column transformer - like we always do "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Section 1: (6 points in total)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Prep (4 points)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set Up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Common imports\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Salary</th>\n",
       "      <th>Job Description</th>\n",
       "      <th>Location</th>\n",
       "      <th>Min_years_exp</th>\n",
       "      <th>Technical</th>\n",
       "      <th>Comm</th>\n",
       "      <th>Travel</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>67206</td>\n",
       "      <td>Civil Service Title: Regional Director Mental ...</td>\n",
       "      <td>Remote</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>88313</td>\n",
       "      <td>The New York City Comptrollerâ€™s Office Burea...</td>\n",
       "      <td>Remote</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>10-15 hrs</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>81315</td>\n",
       "      <td>With minimal supervision from the Deputy Commi...</td>\n",
       "      <td>East campus</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>5-10 hrs</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>76426</td>\n",
       "      <td>OPEN TO CURRENT BUSINESS PROMOTION COORDINATOR...</td>\n",
       "      <td>East campus</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>55675</td>\n",
       "      <td>Only candidates who are permanent in the Princ...</td>\n",
       "      <td>Southeast campus</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>5-10 hrs</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Salary                                    Job Description  \\\n",
       "0   67206  Civil Service Title: Regional Director Mental ...   \n",
       "1   88313  The New York City Comptrollerâ€™s Office Burea...   \n",
       "2   81315  With minimal supervision from the Deputy Commi...   \n",
       "3   76426  OPEN TO CURRENT BUSINESS PROMOTION COORDINATOR...   \n",
       "4   55675  Only candidates who are permanent in the Princ...   \n",
       "\n",
       "           Location  Min_years_exp  Technical  Comm     Travel  \n",
       "0            Remote              5          2     3         No  \n",
       "1            Remote              5          2     4  10-15 hrs  \n",
       "2       East campus              5          3     3   5-10 hrs  \n",
       "3       East campus              1          1     3         No  \n",
       "4  Southeast campus              1          1     3   5-10 hrs  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#We will predict the \"Salary\" value in the data set:\n",
    "\n",
    "jobs = pd.read_csv(\"jobs_alldata.csv\")\n",
    "jobs.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split the data into train and test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "train_set, test_set = train_test_split(jobs, test_size=0.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Checking for missing values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Salary             0\n",
       "Job Description    0\n",
       "Location           0\n",
       "Min_years_exp      0\n",
       "Technical          0\n",
       "Comm               0\n",
       "Travel             0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_set.isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Salary             0\n",
       "Job Description    0\n",
       "Location           0\n",
       "Min_years_exp      0\n",
       "Technical          0\n",
       "Comm               0\n",
       "Travel             0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_set.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Prep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from sklearn.preprocessing import FunctionTransformer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Select the Target Variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_y = train_set[['Salary']]\n",
    "test_y = test_set[['Salary']]\n",
    "\n",
    "train_inputs = train_set.drop(['Salary'], axis=1)\n",
    "test_inputs = test_set.drop(['Salary'], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sklearn: Text preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: nltk in /Users/sanjanathinderu/opt/anaconda3/lib/python3.9/site-packages (3.7)\n",
      "Requirement already satisfied: click in /Users/sanjanathinderu/opt/anaconda3/lib/python3.9/site-packages (from nltk) (8.0.4)\n",
      "Requirement already satisfied: regex>=2021.8.3 in /Users/sanjanathinderu/opt/anaconda3/lib/python3.9/site-packages (from nltk) (2022.3.15)\n",
      "Requirement already satisfied: tqdm in /Users/sanjanathinderu/opt/anaconda3/lib/python3.9/site-packages (from nltk) (4.64.0)\n",
      "Requirement already satisfied: joblib in /Users/sanjanathinderu/opt/anaconda3/lib/python3.9/site-packages (from nltk) (1.1.0)\n"
     ]
    }
   ],
   "source": [
    "# In order to use NLTK, you need to install it first. If you don't have it already installed, convert this\n",
    "# cell to \"Code\" from the toolbar and then execute it to install it.\n",
    "!pip install nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     /Users/sanjanathinderu/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n",
      "[nltk_data] Downloading package wordnet to\n",
      "[nltk_data]     /Users/sanjanathinderu/nltk_data...\n",
      "[nltk_data]   Package wordnet is already up-to-date!\n",
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     /Users/sanjanathinderu/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package omw-1.4 to\n",
      "[nltk_data]     /Users/sanjanathinderu/nltk_data...\n",
      "[nltk_data]   Package omw-1.4 is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "import re\n",
    "\n",
    "nltk.download('stopwords')\n",
    "nltk.download('wordnet')\n",
    "nltk.download('punkt')\n",
    "nltk.download('omw-1.4')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Custom tokenizer "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def nltk_tokenizer(dataset):\n",
    "\n",
    "    #Create a blank list\n",
    "    returned_data = []\n",
    "\n",
    "    # For each row in dataset, we will read the text, tokenize it, remove stopwords, lemmatize it, \n",
    "    # and append it to the list\n",
    "\n",
    "    for text in dataset:\n",
    "        text = re.sub(r'[!\"#$%&()*+,-./:;<=>?[\\]^_`{|}~]', ' ', text).lower()\n",
    "        words= nltk.tokenize.word_tokenize(text)\n",
    "        words = [w for w in words if w.isalpha()]  # include only letter-based terms, exclude all terms with numbers\n",
    "        words = [w for w in words if len(w)>1 and w not in stopwords.words('english')] #include terms that have more than 2 chars and not included in stopwords\n",
    "        lemmatizer = nltk.stem.WordNetLemmatizer()\n",
    "        words = [lemmatizer.lemmatize(w) for w in words]\n",
    "        returned_data.append(' '.join(words))\n",
    "    \n",
    "    return returned_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.decomposition import TruncatedSVD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "    def new_col(df):\n",
    "        #Create a copy so that we don't overwrite the existing dataframe\n",
    "        df1 = df.copy()\n",
    "\n",
    "        # First, conver the dataframe column to a numpy array. Then, call the ravel function to make it one-dimensional\n",
    "        return np.array(df).ravel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([63407,\n",
       "       \"Only candidates who are permanent in the Computer Systems Manager title, or those who can provide proof of successful registration for the October 2018 Open Competitive or Promotional Exam may apply.  Failure to do so will result in your disqualification.   The Department of Design and Construction, Division of Public Buildings seeks a Director of Data Analytics. The Data Analytics team is responsible for providing descriptive, diagnostic and predictive data insights based on data from across the Agency and external sources. The dataset includes basic project management data such as schedule and budget, as well as other external and internal data from information systems and sensors, including uncensored data. The Director of Data Analytics will leverage business acumen and knowledge of data science and modern technical tools, to develop data-informed strategies and action related to strengthening of Agency business performance related to the delivery of quality capital projects on time and within budget in a safe manner. The Director will aggregate and diagnose data to identified trends and risks in performance metrics. S/he will also oversee all internal and external Agency reporting and ensure effective data flow to external dashboards such as Capital Projects Dashboard, and Open Data Portal. The Director should have a good understanding of NYC Open Data initiatives. S/he will promote best practices in data management and data-informed thinking. In performance of above responsibilities, will interact with portfolio and project managers within DDC and the counterparts from the Mayor's Office of Operations, Mayor's Office of Data Analytics and academia. The position reports to the Executive Director of Project Controls. The selected candidate will lead and develop a staff of up to eight data analysts and data scientists.\",\n",
       "       'HQ', ..., 2, 3, 'No'], dtype=object)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_col(train_set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_column = ['Job Description']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create new train and test data \n",
    "\n",
    "new_train = nltk_tokenizer(train_set)\n",
    "\n",
    "new_test = nltk_tokenizer(test_set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert to pandas\n",
    "\n",
    "new_train_df = pd.DataFrame(new_train, columns=['Job Description'])\n",
    "\n",
    "new_test_df = pd.DataFrame(new_test, columns=['Job Description'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparison of original and new train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Salary                                                         63407\n",
       "Job Description    Only candidates who are permanent in the Compu...\n",
       "Location                                                          HQ\n",
       "Min_years_exp                                                      1\n",
       "Technical                                                          3\n",
       "Comm                                                               4\n",
       "Travel                                                            No\n",
       "Name: 429, dtype: object"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# original train data - first observation\n",
    "\n",
    "train_set.iloc[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'salary'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# new train data - first observation\n",
    "new_train[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use Scikit-Learn to create the term-by-doc matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "#TfidfVectorizer includes pre-processing, tokenization, filtering stop words\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "tfidf_vect = TfidfVectorizer(stop_words='english', max_features=500)\n",
    "\n",
    "train_x_tfidf = tfidf_vect.fit_transform(new_train_df['Job Description'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform the TfidfVectorizer transformation\n",
    "# Be careful: We are using the train fit to transform the test data set. Otherwise, the test data \n",
    "# features will be very different and match the train set!!!\n",
    "\n",
    "test_x_tfidf = tfidf_vect.transform(new_test_df['Job Description'])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((7, 10), (7, 10))"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_x_tfidf.shape, test_x_tfidf.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Engineering (1 point)\n",
    "\n",
    "Create one NEW feature from existing data. You either transform a single variable, or create a new variable from existing ones. \n",
    "\n",
    "Grading: \n",
    "- 0.5 points for creating the new feature correctly\n",
    "- 0.5 points for the justification of the new feature (i.e., why did you create this new feature)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We have created a new copy of the existing data frame and created a new variable, Technical Skill from the existing variable Technical. Using characteristics such as bins, labels, etc, we constructed a new column called Technical Skill from the original column in order to improve the predicted accuracy of the models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "## created a new column - Technical and Comm "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    1689.000000\n",
       "mean        2.280639\n",
       "std         1.213319\n",
       "min         1.000000\n",
       "25%         1.000000\n",
       "50%         2.000000\n",
       "75%         3.000000\n",
       "max         5.000000\n",
       "Name: Technical, dtype: float64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's analyze \"Technical\"\n",
    "train_inputs['Technical'].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2    567\n",
       "1    521\n",
       "3    364\n",
       "5    157\n",
       "4     80\n",
       "Name: Technical, dtype: int64"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_inputs['Technical'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD4CAYAAAAXUaZHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAASBElEQVR4nO3db4xddX7f8fdnDWURswUjtlML0xqpVlVYazfLiFIhReMQBTesYh4EyRHZmorISkWkjUrVmjxolAdWeUIUFYJaK6zWFWxGFgm1xYYkyGG0qlRC1ptNvIalWMGixitbWcAbbxGV6bcP5nh0d/547p25d+7sj/dLGt1zf+d3z/mer+585syZe++kqpAkteVT4y5AkjR8hrskNchwl6QGGe6S1CDDXZIadNW4CwC46aabatu2bat+/I9+9COuu+664RU0JNY1GOsajHUNpsW6jh8//rdV9dklV1bV2L/uuOOOWotXXnllTY8fFesajHUNxroG02JdwLdqmVz1sowkNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDVoQ3z8gEZv2/5vzC8/uuMSD3X3Tz9+37hKkjRCnrlLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhrUxD/rOPHuhfl/PtHLf0Qh6ZPKM3dJapDhLkkNMtwlqUF9hXuS00lOJPlOkm91YzcmeTnJW93t5p75jyU5leTNJPeOqnhJ0tIGOXPfWVVfqKqp7v5+4FhVbQeOdfdJchuwB7gd2AU8nWTTEGuWJK1gLZdldgOHuuVDwP094zNV9VFVvQ2cAu5cw34kSQNKVa08KXkbeB8o4L9W1cEkH1TVDT1z3q+qzUmeAl6tqme78WeAl6rq+QXb3AfsA5icnLxjZmZm1Qdx/r0LnPtw8fiOm69f9TaH4eLFi0xMTIy1hstOvHthfnnyWub7Ne4e9dpI/eplXYOxrsGspa6dO3ce77ma8mP6fZ373VV1Nsk/AF5O8r0rzM0SY4t+glTVQeAgwNTUVE1PT/dZymJPPneEJ04sPpTTD65+m8MwOzvLWo5rmHrfB/Dojkvz/Rp3j3ptpH71sq7BWNdgRlVXX5dlqupsd3seeIG5yyznkmwB6G7Pd9PPALf0PHwrcHZYBUuSVrZiuCe5LslnLi8DPwd8FzgK7O2m7QWOdMtHgT1JrklyK7AdeG3YhUuSltfPZZlJ4IUkl+d/var+OMlfAIeTPAy8AzwAUFUnkxwGXgcuAY9U1ccjqV6StKQVw72q/gb4/BLjPwDuWeYxB4ADa65OkrQqvkNVkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDeo73JNsSvKXSV7s7t+Y5OUkb3W3m3vmPpbkVJI3k9w7isIlScsb5Mz9K8AbPff3A8eqajtwrLtPktuAPcDtwC7g6SSbhlOuJKkffYV7kq3AfcDv9QzvBg51y4eA+3vGZ6rqo6p6GzgF3DmUaiVJfUlVrTwpeR74T8BngH9XVV9K8kFV3dAz5/2q2pzkKeDVqnq2G38GeKmqnl+wzX3APoDJyck7ZmZmVn0Q59+7wLkPF4/vuPn6VW9zGC5evMjExMRYa7jsxLsX5pcnr2W+X+PuUa+N1K9e1jUY6xrMWurauXPn8aqaWmrdVSs9OMmXgPNVdTzJdB/7yxJji36CVNVB4CDA1NRUTU/3s+mlPfncEZ44sfhQTj+4+m0Ow+zsLGs5rmF6aP835pcf3XFpvl/j7lGvjdSvXtY1GOsazKjqWjHcgbuBX0jy88Cngb+f5FngXJItVfX9JFuA8938M8AtPY/fCpwdZtGSpCtb8Zp7VT1WVVurahtzfyj9s6r6ZeAosLebthc40i0fBfYkuSbJrcB24LWhVy5JWlY/Z+7LeRw4nORh4B3gAYCqOpnkMPA6cAl4pKo+XnOlkqS+DRTuVTULzHbLPwDuWWbeAeDAGmuTJK2S71CVpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBa3kTk/SJcOLdCz/22TyXnX78vjFUI/XHM3dJapDhLkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQSuGe5JPJ3ktyV8lOZnkt7rxG5O8nOSt7nZzz2MeS3IqyZtJ7h3lAUiSFuvnzP0j4Geq6vPAF4BdSe4C9gPHqmo7cKy7T5LbgD3A7cAu4Okkm0ZQuyRpGSuGe8252N29uvsqYDdwqBs/BNzfLe8GZqrqo6p6GzgF3DnMoiVJV5aqWnnS3Jn3ceCfAL9bVf8hyQdVdUPPnPeranOSp4BXq+rZbvwZ4KWqen7BNvcB+wAmJyfvmJmZWfVBnH/vAuc+XDy+4+brV73NYbh48SITExNjreGyE+9emF+evJb5fo27R702Ur96+fwajHUNZi117dy583hVTS217qp+NlBVHwNfSHID8EKSz11hepbaxBLbPAgcBJiamqrp6el+SlnSk88d4YkTiw/l9IOr3+YwzM7OspbjGqaH9n9jfvnRHZfm+zXuHvXaSP3q5fNrMNY1mFHVNdCrZarqA2CWuWvp55JsAehuz3fTzgC39DxsK3B2rYVKkvrXz6tlPtudsZPkWuBnge8BR4G93bS9wJFu+SiwJ8k1SW4FtgOvDbluSdIV9HNZZgtwqLvu/ingcFW9mOR/AoeTPAy8AzwAUFUnkxwGXgcuAY90l3UkSetkxXCvqr8GfmqJ8R8A9yzzmAPAgTVXJ0laFd+hKkkNMtwlqUGGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAatGO5JbknySpI3kpxM8pVu/MYkLyd5q7vd3POYx5KcSvJmkntHeQCSpMX6OXO/BDxaVf8MuAt4JMltwH7gWFVtB4519+nW7QFuB3YBTyfZNIriJUlLWzHcq+r7VfXtbvnvgDeAm4HdwKFu2iHg/m55NzBTVR9V1dvAKeDOIdctSbqCVFX/k5NtwDeBzwHvVNUNPever6rNSZ4CXq2qZ7vxZ4CXqur5BdvaB+wDmJycvGNmZmbVB3H+vQuc+3Dx+I6br1/1Nofh4sWLTExMjLWGy068e2F+efJa5vs17h712kj96uXzazDWNZi11LVz587jVTW11Lqr+t1IkgngD4Bfr6ofJll26hJji36CVNVB4CDA1NRUTU9P91vKIk8+d4QnTiw+lNMPrn6bwzA7O8tajmuYHtr/jfnlR3dcmu/XuHvUayP1q5fPr8FY12BGVVdfr5ZJcjVzwf5cVf1hN3wuyZZu/RbgfDd+Bril5+FbgbPDKVeS1I9+Xi0T4Bngjar67Z5VR4G93fJe4EjP+J4k1yS5FdgOvDa8kiVJK+nnsszdwJeBE0m+0439BvA4cDjJw8A7wAMAVXUyyWHgdeZeafNIVX087MIlSctbMdyr6n+w9HV0gHuWecwB4MAa6pIkrYHvUJWkBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1yHCXpAb1/amQkqTV29bzyay9vrbrupHszzN3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJatCK4Z7kq0nOJ/luz9iNSV5O8lZ3u7ln3WNJTiV5M8m9oypckrS8fs7cvwbsWjC2HzhWVduBY919ktwG7AFu7x7zdJJNQ6tWktSXFcO9qr4JvLdgeDdwqFs+BNzfMz5TVR9V1dvAKeDO4ZQqSepXqmrlSck24MWq+lx3/4OquqFn/ftVtTnJU8CrVfVsN/4M8FJVPb/ENvcB+wAmJyfvmJmZWfVBnH/vAuc+XDy+4+brV73NYbh48SITExNjreGyE+9emF+evJb5fo27R702Ur96+fwajHUtrfd7sNet129adV07d+48XlVTS627alVbXF6WGFvyp0dVHQQOAkxNTdX09PSqd/rkc0d44sTiQzn94Oq3OQyzs7Os5biG6aGe/7z+6I5L8/0ad496baR+9fL5NRjrWlrv92Cvr+26biR1rfbVMueSbAHobs9342eAW3rmbQXOrr48SdJqrDbcjwJ7u+W9wJGe8T1JrklyK7AdeG1tJUqSBrXiZZkkvw9MAzclOQP8JvA4cDjJw8A7wAMAVXUyyWHgdeAS8EhVfTyi2iVJy1gx3Kvql5ZZdc8y8w8AB9ZSlCRpbXyHqiQ1yHCXpAYN+6WQkj6Bti14qe3ll/2dfvy+cZX0ieeZuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWqQ4S5JDTLcJalBhrskNchwl6QGGe6S1CDDXZIaZLhLUoMMd0lqkOEuSQ0y3CWpQYa7JDXIcJekBhnuktQgw12SGmS4S1KDDHdJapDhLkkNMtwlqUGGuyQ1aGThnmRXkjeTnEqyf1T7kSQtNpJwT7IJ+F3gXwK3Ab+U5LZR7EuStNioztzvBE5V1d9U1f8FZoDdI9qXJGmBVNXwN5r8IrCrqn6lu/9l4J9X1a/1zNkH7Ovu/lPgzTXs8ibgb9fw+FGxrsFY12CsazAt1vWPq+qzS624avX1XFGWGPuxnyJVdRA4OJSdJd+qqqlhbGuYrGsw1jUY6xrMJ62uUV2WOQPc0nN/K3B2RPuSJC0wqnD/C2B7kluT/D1gD3B0RPuSJC0wkssyVXUpya8BfwJsAr5aVSdHsa/OUC7vjIB1Dca6BmNdg/lE1TWSP6hKksbLd6hKUoMMd0lq0E9MuCf5apLzSb67zPok+c/dxx38dZIvbpC6ppNcSPKd7us/rkNNtyR5JckbSU4m+coSc9a9X33WNY5+fTrJa0n+qqvrt5aYM67nVz+1rXvPuv1uSvKXSV5cYt1Y+tVHXWPpVbfv00lOdPv91hLrh9uzqvqJ+AJ+Gvgi8N1l1v888BJzr7G/C/jzDVLXNPDiOvdqC/DFbvkzwP8Cbht3v/qsaxz9CjDRLV8N/Dlw17j7NUBt696zbr//Fvj6UvseV7/6qGssver2fRq46Qrrh9qzn5gz96r6JvDeFabsBv5bzXkVuCHJlg1Q17qrqu9X1be75b8D3gBuXjBt3fvVZ13rruvBxe7u1d3XwlcajOv51U9t6y7JVuA+4PeWmTKWfvVR10Y21J79xIR7H24G/nfP/TNsgODo/Ivu1+qXkty+njtOsg34KebO+HqNtV9XqAvG0K/uV/nvAOeBl6tqw/Srj9pg/Xv2O8C/B/7fMuvH1a/f4cp1wfi+Hwv40yTHM/fxKwsNtWcthfuKH3kwJt9m7vMfPg88Cfz39dpxkgngD4Bfr6ofLly9xEPWpV8r1DWWflXVx1X1BebeTX1nks8tmDK2fvVR27r2LMmXgPNVdfxK05YYG2m/+qxrbN+PwN1V9UXmPi33kSQ/vWD9UHvWUrhvyI88qKofXv61uqr+CLg6yU2j3m+Sq5kL0Oeq6g+XmDKWfq1U17j61bP/D4BZYNeCVWN/fi1X2xh6djfwC0lOM/eJrz+T5NkFc8bRrxXrGufzq6rOdrfngReY+/TcXkPtWUvhfhT4V91fnO8CLlTV98ddVJJ/mCTd8p3M9fwHI95ngGeAN6rqt5eZtu796qeuMfXrs0lu6JavBX4W+N6CaWN5fvVT23r3rKoeq6qtVbWNuY8W+bOq+uUF09a9X/3UNY7nV7ev65J85vIy8HPAwlfYDbVno/pUyKFL8vvM/aX7piRngN9k7o9LVNV/Af6Iub82nwL+D/CvN0hdvwj8mySXgA+BPdX9aXyE7ga+DJzortUC/Abwj3rqGke/+qlrHP3aAhzK3D+Z+RRwuKpeTPKrPXWN5fnVZ23j6NkiG6RfK9U1rl5NAi90P1euAr5eVX88yp758QOS1KCWLstIkjqGuyQ1yHCXpAYZ7pLUIMNdkhpkuEtSgwx3SWrQ/wcFp8w0mErYgwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "train_inputs['Technical'].hist(bins=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def new_col(df):\n",
    "    #Create a copy so that we don't overwrite the existing dataframe\n",
    "    df1 = df.copy()\n",
    "    \n",
    "    df1['Technicalskill'] = pd.cut(df1['Technical'],\n",
    "                                       bins=[0,0.5,1,5,15,50,10000],  #bins=[exclusive, inclusive]\n",
    "                                       labels=False, \n",
    "                                       include_lowest=True,\n",
    "                                       ordered=True)\n",
    "    \n",
    "#     You can also do this if you want categorical values:    \n",
    "#     df1['num_reviews_binned'] = pd.cut(df1['number_of_reviews'],\n",
    "#                                        bins=[0,0.5,1,5,15,50,10000], \n",
    "#                                        labels=['None','Very few','Few','Medium','Many','Too many'], \n",
    "#                                        include_lowest=True,\n",
    "#                                        ordered=False)\n",
    "\n",
    "    \n",
    "    return df1[['Technicalskill']]\n",
    "    # You can use this to check whether the calculation is made correctly:\n",
    "    #return df1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Technicalskill</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>429</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1185</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2116</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2127</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>458</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1638</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1095</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1130</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1294</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>860</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1689 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      Technicalskill\n",
       "429                2\n",
       "1185               2\n",
       "2116               2\n",
       "2127               2\n",
       "458                1\n",
       "...              ...\n",
       "1638               2\n",
       "1095               2\n",
       "1130               2\n",
       "1294               2\n",
       "860                2\n",
       "\n",
       "[1689 rows x 1 columns]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Let's test the new function:\n",
    "\n",
    "# Send train set to the function we created\n",
    "new_col(train_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Identify the numerical and categorical columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Job Description    object\n",
       "Location           object\n",
       "Min_years_exp       int64\n",
       "Technical           int64\n",
       "Comm                int64\n",
       "Travel             object\n",
       "dtype: object"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_inputs.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Identify the numerical columns\n",
    "numeric_columns = train_inputs.select_dtypes(include=[np.number]).columns.to_list()\n",
    "\n",
    "# Identify the categorical columns\n",
    "categorical_columns = train_inputs.select_dtypes('object').columns.to_list()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Min_years_exp', 'Technical', 'Comm']"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "numeric_columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Job Description', 'Location', 'Travel']"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "categorical_columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "transformed_columns = ['Technical']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "numeric_transformer = Pipeline(steps=[\n",
    "                ('imputer', SimpleImputer(strategy='median')),\n",
    "                ('scaler', StandardScaler())])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "categorical_transformer = Pipeline(steps=[\n",
    "    ('imputer', SimpleImputer(strategy='constant', fill_value='unknown')),\n",
    "    ('onehot', OneHotEncoder(handle_unknown='ignore'))])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "number_svd_components = 400\n",
    "text_transformer = Pipeline(steps=[\n",
    "                ('my_new_column', FunctionTransformer(new_col)),\n",
    "                ('text', TfidfVectorizer(stop_words='english')),\n",
    "                ('svd', TruncatedSVD(n_components=number_svd_components, n_iter=10))\n",
    "            ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_new_column = Pipeline(steps=[('my_new_column', FunctionTransformer(new_col)),\n",
    "                                ('imputer', SimpleImputer(strategy='median')),\n",
    "                                ('scaler', StandardScaler())])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "preprocessor = ColumnTransformer([\n",
    "        ('num', numeric_transformer, numeric_columns),\n",
    "        ('cat', categorical_transformer, categorical_columns),\n",
    "        ('trans', my_new_column, transformed_columns)],\n",
    "        remainder='drop')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Transform: fit_transform() for TRAIN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<1689x1213 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 11823 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Fit and transform the train data\n",
    "train_x = preprocessor.fit_transform(train_inputs)\n",
    "\n",
    "train_x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-1.12955755,  0.59306224,  0.9780514 , ...,  0.        ,\n",
       "         1.        ,  0.66787846],\n",
       "       [ 1.0979062 ,  0.59306224, -1.25579419, ...,  0.        ,\n",
       "         0.        ,  0.66787846],\n",
       "       [-1.12955755,  0.59306224,  2.09497419, ...,  0.        ,\n",
       "         1.        ,  0.66787846],\n",
       "       ...,\n",
       "       [-1.12955755,  0.59306224, -1.25579419, ...,  1.        ,\n",
       "         0.        ,  0.66787846],\n",
       "       [ 0.54104026, -0.23136749, -0.1388714 , ...,  0.        ,\n",
       "         1.        ,  0.66787846],\n",
       "       [-1.12955755, -0.23136749, -0.1388714 , ...,  0.        ,\n",
       "         1.        ,  0.66787846]])"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_x.toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1689, 1213)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_x.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tranform: transform() for TEST"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<724x1213 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 4738 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Transform the test data\n",
    "test_x = preprocessor.transform(test_inputs)\n",
    "\n",
    "test_x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-1.12955755, -0.23136749, -0.1388714 , ...,  0.        ,\n",
       "         1.        ,  0.66787846],\n",
       "       [ 0.54104026,  0.59306224, -2.37271699, ...,  1.        ,\n",
       "         0.        ,  0.66787846],\n",
       "       [ 0.54104026,  0.59306224, -1.25579419, ...,  0.        ,\n",
       "         1.        ,  0.66787846],\n",
       "       ...,\n",
       "       [-1.12955755, -0.23136749,  0.9780514 , ...,  0.        ,\n",
       "         1.        ,  0.66787846],\n",
       "       [ 0.54104026,  2.24192171, -1.25579419, ...,  0.        ,\n",
       "         0.        ,  0.66787846],\n",
       "       [ 1.0979062 , -1.05579722,  0.9780514 , ...,  0.        ,\n",
       "         1.        , -1.4972784 ]])"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_x.toarray()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(724, 1213)"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_x.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Find the Baseline (1 point)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DummyRegressor()"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.dummy import DummyRegressor\n",
    "\n",
    "dummy_regr = DummyRegressor(strategy=\"mean\")\n",
    "\n",
    "dummy_regr.fit(train_x, train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import mean_squared_error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Acuracy Values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Baseline Train RMSE: 29582.833221810994\n"
     ]
    }
   ],
   "source": [
    "#Baseline Train RMSE\n",
    "dummy_train_pred = dummy_regr.predict(train_x)\n",
    "\n",
    "baseline_train_mse = mean_squared_error(train_y, dummy_train_pred)\n",
    "\n",
    "baseline_train_rmse = np.sqrt(baseline_train_mse)\n",
    "\n",
    "print('Baseline Train RMSE: {}' .format(baseline_train_rmse))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Baseline Test RMSE: 28294.892856870818\n"
     ]
    }
   ],
   "source": [
    "#Baseline Test RMSE\n",
    "dummy_test_pred = dummy_regr.predict(test_x)\n",
    "\n",
    "baseline_test_mse = mean_squared_error (test_y, dummy_test_pred)\n",
    "\n",
    "baseline_test_rmse = np.sqrt(baseline_test_mse)\n",
    "\n",
    "print('Baseline Test RMSE: {}' .format(baseline_test_rmse))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Section 2: (5 points in total)\n",
    "\n",
    "Build the following models:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decision Tree: (1 point)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeRegressor(max_depth=5, min_samples_leaf=10)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.tree import DecisionTreeRegressor\n",
    "\n",
    "tree_reg = DecisionTreeRegressor(max_depth=5, min_samples_leaf = 10) \n",
    "\n",
    "tree_reg.fit(train_x, train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train RMSE: 29181.176569565057\n"
     ]
    }
   ],
   "source": [
    "#Train RMSE\n",
    "train_pred = tree_reg.predict(train_x)\n",
    "\n",
    "train_mse = mean_squared_error(train_y, train_pred)\n",
    "\n",
    "train_rmse = np.sqrt(train_mse)\n",
    "\n",
    "print('Train RMSE: {}' .format(train_rmse))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test RMSE: 28599.71696809674\n"
     ]
    }
   ],
   "source": [
    "#Test RMSE\n",
    "test_pred = tree_reg.predict(test_x)\n",
    "\n",
    "test_mse = mean_squared_error(test_y, test_pred)\n",
    "\n",
    "test_rmse = np.sqrt(test_mse)\n",
    "\n",
    "print('Test RMSE: {}' .format(test_rmse))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# Max depth = 1     Train RMSE = 29203.1467   Test RMSE = 28284.3275\n",
      "# Max depth = 2     Train RMSE = 28880.2708   Test RMSE = 28275.4515\n",
      "# Max depth = 3     Train RMSE = 28578.1936   Test RMSE = 28267.6451\n",
      "# Max depth = 4     Train RMSE = 28305.0797   Test RMSE = 28260.9236\n",
      "# Max depth = 5     Train RMSE = 28079.9618   Test RMSE = 27727.7252\n",
      "# Max depth = 6     Train RMSE = 27854.1164   Test RMSE = 27721.5988\n",
      "# Max depth = 7     Train RMSE = 27638.5576   Test RMSE = 27716.1534\n",
      "# Max depth = 8     Train RMSE = 27422.6082   Test RMSE = 27712.6219\n",
      "# Max depth = 9     Train RMSE = 27225.9236   Test RMSE = 27709.5011\n",
      "# Max depth = 10     Train RMSE = 27035.3843   Test RMSE = 27267.6641\n"
     ]
    }
   ],
   "source": [
    "train_error = []\n",
    "test_error = []\n",
    "\n",
    "for x in range(1,11):\n",
    "    tree_reg3 = DecisionTreeRegressor(max_depth=x)\n",
    "    tree_reg3.fit(train_x, train_y)\n",
    "    reg_train_predictions = tree_reg3.predict(train_x)\n",
    "    reg_test_predictions = tree_reg3.predict(test_x)\n",
    "    train_rmse = round(np.sqrt(mean_squared_error (train_y, reg_train_predictions)),4)\n",
    "    test_rmse = round(np.sqrt(mean_squared_error (test_y, reg_test_predictions)),4)\n",
    "    print('# Max depth = {}'.format(x) + \"     \" +'Train RMSE = {}'.format(train_rmse) + \"   \"\n",
    "         'Test RMSE = {}'.format(test_rmse))\n",
    "    \n",
    "    train_error.append(train_rmse)\n",
    "    test_error.append(test_rmse)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7feddba247c0>"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEGCAYAAABPdROvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA0J0lEQVR4nO3dd3gVVfrA8e97c1NooQaFJHRQepCIoVcVC0VFja6ogCKIdMuia1t39+cqgiACooiAUpSygIIivYQWikCkhR4p0gklkHJ+f9wJXDAkgeRmktz38zzzcObMnJn3XiBvZubMOWKMQSmllLpVDrsDUEoplbdpIlFKKZUlmkiUUkpliSYSpZRSWaKJRCmlVJY47Q4gp5UqVcpUqFDB7jCUUipPWb9+/XFjTFBa27wukVSoUIHo6Gi7w1BKqTxFRPbfaJve2lJKKZUlmkiUUkpliSYSpZRSWeJ1z0iUUupmJSYmEhcXR0JCgt2heFxAQAAhISH4+vpmuo0mEqWUykBcXBxFihShQoUKiIjd4XiMMYYTJ04QFxdHxYoVM91Ob20ppVQGEhISKFmyZL5OIgAiQsmSJW/6yksTiVJKZUJ+TyKpbuVzaiLJpH3Hz/Pfn7eTnKLD7iullDtNJJn0S8wRRi3ZzUsT13PhcpLd4SilvMiJEycICwsjLCyM22+/neDg4Cvrly9fTrdtdHQ0ffr08Wh8+rA9k15qXpkCfj68NzuGyDGr+eq5cEoXCbA7LKWUFyhZsiSbNm0C4L333qNw4cK8+uqrV7YnJSXhdKb94zw8PJzw8HCPxqdXJDfh2YYVGNM5nF1Hz/HoyChi/4y3OySllJd6/vnnGTBgAC1btuSNN95g7dq1NGrUiHr16tGoUSN27NgBwJIlS3j44YcBVxLq2rUrLVq0oFKlSgwfPjxbYtErkpvUpsZtTH0pgq7fRPPoyCjGPBtORKWSdoellMoh78+J4fdDZ7P1mDXKBvJuu5o33W7nzp0sWLAAHx8fzp49y7Jly3A6nSxYsIA333yT6dOn/6XN9u3bWbx4MfHx8dxxxx307Nnzpt4ZSYtekdyCOiHFmPlyI0oHBtB57Br+t/EPu0NSSnmhxx9/HB8fHwDOnDnD448/Tq1atejfvz8xMTFptnnooYfw9/enVKlSlC5dmqNHj2Y5Dr0iuUWhJQoyvUcjuk+Mpt/UTcSdukCvllW8pougUt7qVq4cPKVQoUJXym+//TYtW7Zk5syZ7Nu3jxYtWqTZxt/f/0rZx8eHpKSsdx7SK5IsKFrQlwndGtAxrCyD5+9k0IwtJCan2B2WUsoLnTlzhuDgYAC++eabHD23JpIs8nf6MPTJMHq3qsKUdQfpNj6a+IREu8NSSnmZ119/nUGDBtG4cWOSk5Nz9NxijHe9YBceHm48NbHV1HUHeHPmVqqWLsy4LndTpmgBj5xHKZWztm3bRvXq1e0OI8ek9XlFZL0xJs1+xHpFko2evLsc456/m7hTF3nk8yi2Hc7enh1KKZUbaSLJZs2qBfH9Sw0BeHz0KpbtPGZzREop5VmaSDygRtlAZvZqREjxAnT5Zh3frztod0hKKeUxmkg8pEzRAvzQoyGNKpfk9embGfzLDrzteZRSyjtoIvGgIgG+fP383TwZHsqIxbH0n7qJS0k525tCKaU8TV9I9DBfHwcfPlabciUL8vEvOzhyNoEvngmnaMGsDUmglFK5hV6R5AARoVfLKgyLDGPD/tM8NjqKgycv2B2WUiqPyMow8uAauDEqKspj8XkskYhIqIgsFpFtIhIjIn2t+roiskpEtojIHBEJdGszSERiRWSHiNzvVl/f2j9WRIaLNQ6JiPiLyFSrfo2IVPDU58kOHcKCmdCtAX+eTeCRkVFsjjttd0hKqTwgdRj5TZs20aNHD/r3739l3c/PL8P2eTaRAEnAQGNMdSAC6CUiNYCvgL8bY2oDM4HXAKxtkUBNoC0wUkR8rGONAroDVa2lrVXfDThljKkCDAX+68HPky0iKpVkxsuNCPB18OQXq1nwe9YHTFNKeZ/169fTvHlz6tevz/3338/hw4cBGD58ODVq1KBOnTpERkayb98+Ro8ezdChQwkLC2P58uXZHovHnpEYYw4Dh61yvIhsA4KBO4Bl1m6/Ar8AbwMdgCnGmEvAXhGJBRqIyD4g0BizCkBEJgAdgXlWm/esY00DRoiImFzePapK6SLMfLkx3cavo/vEaN5rX5NnG1awOyylVGbM+zsc2ZK9x7y9NjzwYaZ3N8bQu3dvZs2aRVBQEFOnTuWtt97i66+/5sMPP2Tv3r34+/tz+vRpihUrRo8ePf4yGVZ2ypGH7dYtp3rAGmAr0B6YBTwOhFq7BQOr3ZrFWXWJVvn6+tQ2BwGMMUkicgYoCRy/7vzdcV3RUK5cuWz6VFkTVMSfKd0j6DN5E+/MiuHgyQsMeqA6DoeOHqyUSt+lS5fYunUr9957LwDJycmUKVMGgDp16vC3v/2Njh070rFjxxyJx+OJREQKA9OBfsaYsyLSFRguIu8As4HUJ0Vp/QQ16dSn1+baCmPGAGPANdbWzX0Czyno5+SLzvX54Mff+XL5Xv44fZEhT4QR4OuTcWOllD1u4srBU4wx1KxZk1WrVv1l208//cSyZcuYPXs2H3zwwQ3nJclOHu21JSK+uJLId8aYGQDGmO3GmPuMMfWBycBua/c4rl6dAIQAh6z6kDTqr2kjIk6gKHDSM5/GM3wcwrvtavCPh6ozb+sRnv5yNSfOXbI7LKVULubv78+xY8euJJLExERiYmJISUnh4MGDtGzZko8++ojTp09z7tw5ihQpQny856YG92SvLQHGAtuMMUPc6ktbfzqAfwCjrU2zgUirJ1ZFXA/V11rPWuJFJMI65rO4boultnnOKncCFuX25yNpERFeaFqJkU/fRcyhszw2Koq9x8/bHZZSKpdyOBxMmzaNN954g7p16xIWFkZUVBTJyck888wz1K5dm3r16tG/f3+KFStGu3btmDlzpscetntsGHkRaQIsB7YAqbM9vYkrQfSy1mcAg1J/+IvIW0BXXD2++hlj5ln14cA3QAFcD9l7G2OMiAQAE3E9fzkJRBpj9qQXlyeHkc8O6/ef4sUJ0Rhj+Oq5cOqXL2F3SEp5PR1GPv1h5HU+klxo3/HzdPlmHX+cvsjQJ8J4qE4Zu0NSyqtpItH5SPKcCqUKMb1nI+oEF6XXpA2MWbZbB3xUSuVamkhyqRKF/Pj2hXt4qHYZ/jN3O+/MiiFJ54NXyjbe8svcrXxOHbQxFwvw9eGzp+oRUqIAXyzdw6HTF/ns6XoU9NO/NqVyUkBAACdOnKBkyZJYIzTlS8YYTpw4QUBAwE21059IuZzDIQx6oDqhxQvyzqytPPnFasY+H07pIjf3F62UunUhISHExcVx7Fj+n/E0ICCAkJCQjHd0ow/b85DF2/+k16QNFCvgy5hnw6kVXNTukJRSXkIftucTLe8sfWU++E6jo5jz26EMWiillOdpIsljagUXZdYrTagdXJTekzfy0c/bSUnxrqtKpVTuookkDwoq4s93L0TwVINyjFyymxcnRBOfkGh3WEopL6WJJI/yczr4zyO1+KBjLZbuPMYjI3VYFaWUPTSR5GEiQueI8nz7wj2cPH+ZDiNWsHRn/u9VopTKXTSR5AMRlUoyq1djyhYrQJdxa/ly2R6veXlKKWU/TST5RGiJgsx4uRFta93Ov+duY+D3v5GQmGx3WEopL6CJJB8p6Ofk86fvYuC91Zix8Q+e/GIVR84k2B2WUiqf00SSz4gIvVtXZUzn+sT+eY52I1aw4cApu8NSSuVjmkjyqftq3s7MXo0p6OdD5Ber+T76oN0hKaXyKU0k+Vi124owq1djGlQswevTNvP+HB1BWCmV/TSR5HPFCvrxTZe76dq4IuNW7uO5cWs5df6y3WEppfIRTSRewOnj4J12Nfi4Ux3W7T1Fh89XsuNIvN1hKaXyCU0kXuTx8FCmvBTBxcRkHh25kl9ijtgdklIqH9BE4mXuKlecOa80oUrpwrw0cT3DFuzSQR+VUlmiicQL3V40gKkvNeTResEMXbCTXpM2cP5Skt1hKaXyKE0kXirA14dPnqjLPx6qzi8xR3hsVBQHT16wOyylVB6kicSLiQgvNK3EuC4NOHT6Iu1HrCBq93G7w1JK5TGaSBTNqwUx65UmlCzsT+exaxkftU8HfVRKZZomEgVAxVKFmPlyI1pUC+Ld2TEMmrGFy0n68qJSKmOaSNQVRQJ8+fLZcHq1rMyUdQd5+svVHIu/ZHdYSqlcThOJuobDIbx2/5189lQ9th46Q/sRK9gSd8busJRSuZgmEpWmdnXLMq1HIwToNDqKWZv+sDskpVQupYlE3VCt4KLM7t2EuiHF6DtlEx/O206yvryolLqOJhKVrlKF/fn2hXt4+p5yjF66m27j13HmYqLdYSmlchFNJCpDfk4H/3mkNv/qWIsVu47zyMiV7D52zu6wlFK5hCYSlWnPRJTn2xfu4fSFRDqOWMnCbUftDkkplQtoIlE3JaJSSWa/0pjQEgXpNj6af//0u75vopSX00SiblpI8YLMeLkRz0SU48vle3l8dBQHTug4XUp5K00k6pYE+Prwr461GfW3u9hz/DwPDV/OnN8O2R2WUsoGmkhUljxQuwxz+zSlym2F6T15I4NmbObi5WS7w1JK5SCPJRIRCRWRxSKyTURiRKSvVR8mIqtFZJOIRItIA6u+gohctOo3ichot2PVF5EtIhIrIsNFRKx6fxGZatWvEZEKnvo86sZCSxTk+5ca0qN5ZSavPUiHz1ew86hO5auUt/DkFUkSMNAYUx2IAHqJSA3gI+B9Y0wY8I61nmq3MSbMWnq41Y8CugNVraWtVd8NOGWMqQIMBf7rwc+j0uHr4+DvD9zJhK4NOHn+Mu1HrGDy2gM6irBSXsBjicQYc9gYs8EqxwPbgGDAAIHWbkWBdG+si0gZINAYs8q4fipNADpamzsA463yNKB16tWKskezakHM7duU8PIlGDRjC70nb+Rsgr7AqFR+liPPSKxbTvWANUA/4GMROQgMBga57VpRRDaKyFIRaWrVBQNxbvvEWXWp2w4CGGOSgDNAyTTO3926jRZ97NixbPtcKm2liwQwoWsDXrv/DuZtPcLDw1fw28HTdoellPIQjycSESkMTAf6GWPOAj2B/saYUKA/MNba9TBQzhhTDxgATBKRQCCtK4zU+yXpbbtaYcwYY0y4MSY8KCgoax9IZYrDIfRqWYWp3SNISk6h0+govlq+hxQdq0upfMejiUREfHElke+MMTOs6ueA1PIPQAMAY8wlY8wJq7we2A1Uw3UFEuJ22BCu3g6LA0Ktczlx3So76anPo25eeIUSzO3blJZ3lOZfP22j2/h1nDinc5wolZ94steW4Lra2GaMGeK26RDQ3Cq3AnZZ+weJiI9VroTrofoeY8xhIF5EIqxjPgvMstrPxpWYADoBi4w+3c11ihX044vO9Xm/fU1Wxp7gweHLWbX7hN1hKaWyiXjq566INAGWA1uA1DE03gTOAsMAJ5AAvGyMWS8ijwH/xNXbKxl41xgzxzpWOPANUACYB/Q2xhgRCQAm4nr+chKINMbsSS+u8PBwEx0dnZ0fVd2EmENn6D1pI/tOnKd3q6r0aV0VH4f2j1AqtxOR9caY8DS3edsv8JpI7Hf+UhJvz9rKjA1/0KBiCYZH1uP2ogF2h6WUSkd6iUTfbFc5rpC/kyFPhPHJ43XZ+scZHhi2jEXbdSRhpfIqTSTKNo/VD2FO7ybcXrQAXb+J5oMfdSRhpfIiTSTKVpWDCjPz5UY817A8Y1fspdPoKPafOG93WEqpm6CJRNkuwNeH9zvUYvQz9dl3/DwPDV/BbB1JWKk8QxOJyjXa1rqduX2bcsftRegzeSNvTNORhJXKCzSRqFwlpHhBpnaPoFfLyny//iDtR6xgxxEdSVip3EwTicp1nD4OXrv/TiZ2vYdTFxJpP2IFk9boSMJK5VaaSFSu1aRqKeb1bUqDiiV4c+YWXtGRhJXKlTSRqFwtqIg/47s04I22d/Lz1iM8NHw5m3QkYaVyFU0kKtdzOISeLSrz/UsNSUmBTqOi+HKZjiSsVG6hiUTlGfXLF2dun6a0qX4b/567ja46krBSuYImEpWnFC3oy6hn7uKDjrWI2n2CB4YtZ/H2P+0OSymvpolE5TkiQueI8szq1ZjAAr50+WYdL02M5o/TF+0OTSmvpKP/ZlbMTNgwEXz8wMcXnP5Xyz5+aZTTqrPKTv+bb6dT0afpclIKXy7fw2eLdiEIfVpXpVuTivg59XckpbJTeqP/OnM6mDwr6TIknIHkS5CcCMmX3f68ruwJjtQk5He17OO8cdkndT/3srX8pZyFNg7n1TZXyr7g8HErO13rHuDndNCrZRU6hJXln3N+578/b2f6hjj+2aEmjSqX8sg5lVLX0iuS7GbMjRPMXxLQpYz3TbrBPimJVjl1Pem68uWr2933TUk9VtLVco4QV0JJTUQOnxuUnVeT4pXyjRKUDzgLQIMXIegOABZtP8q7s2M4ePIiHcLK8taD1SkdqHOdKJVVOrGVG53Y6jrGQEqyW3JK+muick8+f9mWdHVJ3Tcl2a2cdDVpXdkn+eoxrim7HyeTx714EgLLQo8V4FcIgITEZEYu2c3oJbvxdzoYcF81OkeUx+mjt7uUulWaSNxoIsln9q2Abx6G8C7w8NBrNu09fp53Z8ewbOcxapQJ5IOOtahfvrhNgSqVt+kMiSr/qtAEGr0C0V/DzvnXbKpYqhDju9zNyL/dxcnzl3lsVBRvTNvMyfM5dTtPKe+QYSIREYeINMqJYJS6Ja3ehtI1YVYvOH/imk0iwoO1y7BwYHNealaJ6RviaPXJEiatOaBvxiuVTTJMJMaYFOCTHIhFqVvj9IdHx0DCaZjTx/Xc5zqF/J0MerA6c/s2pdptRXhz5hYeHRXF1j/O5Hy8SuUzmb21NV9EHhPRlxlULnV7LWj1D9j+I/w2+Ya7VbutCFO7RzDkibrEnbpA+xEreHfWVs5c1FGFlbpVmXrYLiLxQCEgGbgICGCMMYGeDS/76cP2fCwlGca3g8OboedKKF4+3d3PXExkyPwdTFy9nxKF/HnroTvpGBaM/r6k1F9l+WG7MaaIMcZhjPE1xgRa63kuiah8zuEDj4x2lWf2cCWWdBQt4Mv7HWox+5UmBBcvQP+pvxE5ZjU7j+qMjErdjEz32hKR9iIy2Foe9mRQSt2yYuXgwY/hQBSsGpGpJrWCizKzZyP+80htth+J58Fhy/m/uds4fynJw8EqlT9kKpGIyIdAX+B3a+lr1SmV+9SNhOrtYeEHcGRLppo4HMLT95Rj8asteOyuEL5Ytoc2Q5Yyb8thneJXqQxk9hnJZiDM6sGFiPgAG40xdTwcX7bTZyRe4vwJGNUQCpaCFxeB780Nk7J+/0n+8b8Yth0+S7NqQbzfviYVSxXyULBK5X7Z9UJiMbdy0SxFpJSnFSoJHT6HP2Ng8b9uunn98iWY80pj3m1Xgw37T3H/0GUM+XUnCYnpP3dRyhtlNpH8B9goIt+IyHhgvVWnVO5V9V4I7wZRI2Dv8ptu7vRx0KVxRRYNbM4DtW9n+MJd3Dt0qU6kpdR1MvVmO5ACRAAzrKWhMWaKh2NTKuvu+wBKVHL14kq4tZcPSwcGMCyyHpNeuAc/HwddvllH9wnRxJ26kM3BKpU3ZfbN9leMMYeNMbONMbOMMUdyIDalss6vkOut9/jDMPf1LB2qUZVSzOvbjDfa3snyXcdpM2QpI5fEcjkpJZuCVSpvyuytrV9F5FURCRWREqmLRyNTKruEhEOz12DzFNdMl1ng53TQs0VlFgxsTvNqQXz08w4eGLaMqNjj2RSsUnlPZntt7U2j2hhjKmV/SJ6lvba8VHIijL0PTu2FnqsgsEy2HHbx9j95d3YMB05eoH3dsvzjIZ1IS+VPWeq1ZT0j+bsxpuJ1S55LIsqL+fi6bnElJrhGCc6md0Na3lma+f2b0ad1VX7eeoTWnyxl3Mq9JCXr7S7lPTL7jKRXDsSilGeVqup6+L57Iaz7KtsOG+Drw4B7q/FL/2aElSvG+3N+p/2IlWw4cCrbzqFUbqbPSJR3ufsFqNwa5r8Nx3dl66ErlirEhK4Nrkyk9ejIKAbN2MwpnUhL5XP6jER5n7OHXW+9F68I3ea7bntls3OXkhi2YCdfr9xHYICTvz9wJ4/XD8Xh0JGFVd6UHaP/Xv98JMNnJNbVy2IR2SYiMSLS16oPE5HVIrJJRKJFpIFbm0EiEisiO0Tkfrf6+iKyxdo2PHVeFBHxF5GpVv0aEamQmc+jvFxgGXj4Uzi0AZYN9sgpCvs7eeuhGvzUpwmVgwrzxvQtdBodxe+HznrkfErZKd1EIiKvu5Ufv25bRm+2JwEDjTHVcb3M2EtEagAfAe8bY8KAd6x1rG2RQE2gLTDSGtMLYBTQHahqLW2t+m7AKWNMFWAo8N8MYlLKpWZHqBMJyz6GOM9dod55eyDfv9SQjzvVYd+JC7QbsYJ/zvmd+ASdSEvlHxldkUS6lQddt60t6bBeYNxgleOBbUAwYIDUuUyKAoescgdgijHmkjFmLxALNBCRMkCgMWaVcd2HmwB0dGsz3ipPA1rrLI4q0x78CALLwozucPm8x07jcAiPh4eyaGBzIu8OZVzUXlp/spQ5vx3SkYVVvpBRIpEblNNav/FBXLec6gFrgH7AxyJyEBjM1QQVDBx0axZn1QVb5evrr2ljjEkCzgAl0zh/d+s2WvSxY8cyG7bK7wKKuibCOrnH9fDdw4oV9OPfj9Rm5suNKR3oT+/JG+k8di17jp3z+LmV8qSMEom5QTmt9TSJSGFgOtDPGHMW6An0N8aEAv2Bsam73uD8N6pPr821FcaMMcaEG2PCg4KCMhO28hYVmkCjVyB6LOz6NUdOGRZajFm9mvDPDjX5Le40bT9dzifzd3Dxso4srPKmjBJJXRE5a83ZXscqp67XzujgIuKLK4l8Z4yZYVU/h2vgR4AfgNSH7XFAqFvzEFy3veKs8vX117QRESeuW2UnM4pLqWu0ehtK13S9qHj+RI6c0schPNuwAgsHNuehOmX4bFEs9w5dysJtR3Pk/Eplp3QTiTHGx22OdqdVTl1Pt8+k9axiLLDNGDPEbdMhoLlVbgWkduafDURaPbEq4nqovtYYcxiIF5EI65jPArPc2jxnlTsBi4zedFY3y+nveuv94in4sW+2vfWeGaWLBDD0yTAmvxhBgK8P3cZH68jCKs/J1Hskt3RgkSbAcmALrmHoAd4EzgLDACeQALxsjFlvtXkL6Iqrx1c/Y8w8qz4c+AYoAMwDehtjjIgEABNxPX85CUQaY/akF5e+R6JuaOUw+PUd6Dgawp7K8dNfTkrh65V7GbZgFwZDn9ZVeaFJJfycNzP/nFKekd57JB5LJLmVJhJ1QynJML4dHN4MPVdC8fK2hPHH6Yv8c04Mv8QcpXJQIT7oWItGlUvZEotSqbJrql2l8jeHD3Qc5SrP7OFKLDYILlaALzqH8/Xz4VxOTuHpL9fQb8pG/oxPsCUepTKiiUQpd8XLu94vORAFq0bYGkqrO2/j1/7N6dOqCnO3HKH14KWMj9pHcop33UVQuZ8mEqWuV/cpqN4OFn4AR7bYGkqArw8D7ruDn/s1JaxcMd6dHUP7ESvYqCMLq1xEE4lS1xOBh4dBgeIw4yXXHCY2qxRUmAldG/D503dx/NwlHh0VxaAZWzh9QUcWVvbTRKJUWgqVhA6fw58xsPhfdkcDgIjwUJ0yLBjQnK6NK/J99EFafbKU76MPkqK3u5SNNJEodSPV7oPwrhA1AvYutzuaK4oE+PL2wzX4sXcTKpYqxOvTNvPEF6vYdlhHFlb20ESiVHru+xeUqAj/6wkJZ+yO5hrVywTyw0sN+ahTHXYfO8fDn63ggx9/56yOLKxymCYSpdLjVwge/RLOHoJ5b9gdzV84HMIT4aEsGtiCJ8JD+XrlXloNXsL36/R2l8o5mkiUykhIODR7FX6bDDH/szuaNBUv5Mf/PVqbWb0aU65EQV6fvpkOn69k/X4dek55niYSpTKj2WtQth782A/ij9gdzQ3VCSnG9J6N+PTJMP6MT+CxUavoN2UjR87Y3/NM5V+aSJTKDB9f1y2uxATXKMG5eGghEaFjvWAWDWxBr5aVmbv1CK0+WcLni2NJSNSh6lX200SiVGaVqgr3fQCxC1zzl+RyhfydvHb/nSzo35wmVUrx8S87uG/oMn6JOaIzM6pspYlEqZtx9wtQuTX88g84Hmt3NJlSrmRBxjwbzrfd7iHA18FLE9fzzNg17Dwab3doKp/QRKLUzRBxvajoGwAzXoTkvNPVtknVUszt05T32tVgS9wZHhi2nPdmx3DmQt75DCp30kSi1M0KLAMPfwqHNsDyT+yO5qY4fRw837giS15ryVMNQpmwah8tBi9m4ur9OhikumWaSJS6FTU7Qp1IWPoRxK23O5qbVqKQH//qWJsfezel2m1FePt/W3lo+HJW78mZqYZV/qKJRKlb9eBHEFjWdYvr8nm7o7klNcoGMqV7BJ8/fRfxCUlEjllNr+826FS/6qZoIlHqVgUUdU2EdXIPzH/b7mhumftgkP3bVGPh9qO0/mQpQ3/dycXL2l1YZUyn2lUqq355yzUJlm8hcDhdMy06nK53T1LLDic4rlv36Hb3eh+Q69av2e4EcVwpHz2fxOhl+/h1xwlKFSnIy63v4N6aZRH3/VPbi8PVAUHlezpnuxtNJCrbJV2CNV/AuaOQknR1SU66dv0vS7Kr15f7esr160lu+1jrqfvkFg5n2olKHK56cYDDce26ONz2SWv9+nbudde1u6YunXZXFrnxcTJaHD5X28uNziHX7e8DFRq7rmDzsPQSiTOng1Eq33H6Q+M+OXtOY8CkuCWtRLdEZCUck+yWfFITkdu6Sb4uQV23r0kmJTmR6D3HWPT7IS4lJhJeLpCWVUpQ0Hnd+dM8fsq1S0qy27pVTrlu/fr9Uj/XDdu51yVb38sNjpX6nV1/vtTFk26rBV1/Bv8inj2PTTSRKJUXiVhXAT6Av8dO4wAa1Ic7LiTy6cKd9Fm1n0KHfOjXphqdG5bH1ycfPWa9kpzTSDIZLVfamL9uO7YdZnSH6S9C5HfW31n+ore2lFKZtutoPP/88XeW7zpO1dKFeaddDZpWDbI7rNxv7Zcw91Vo1Ns1x00elN6trXz064RSytOq3laECV0bMKZzfS4lpdB57FpenBDN/hN5s/tzjmnwItz9IkR9Bhsm2B1NttNEopS6KSLCfTVv59cBzXi97R2sjD3OvUOW8d+ft3P+Ui7qBJDbtP0QKreCHwfAvhV2R5OtNJEopW6Jv9OHl1tUYfGrLXi4ThlGLdlNy8FLmLEhTmdnTIuPEzqNc03dPPUZ1/tH+YQmEqVUltwWGMCQJ8OY8XIjyhQNYMD3v/HY6Ch+O3ja7tBynwLF4OmpgMCkJ+HiaZsDyh6aSJRS2eKucsWZ+XJjPu5Uh4MnL9Lh85W89sNv/BmvszNeo0QlePJbOLkXpnVxvW+Ux2kiUUplG4dDeDw8lMWvNuelZpX436Y/aDV4KWOW7eZykoff1chLKjSGh4fC7kXw89/tjibLNJEopbJdkQBfBj1YnV/6NaNBxRL8Z+522n66jMXb/7Q7tNzjrs6u7sDrvnR1D87DNJEopTymUlBhvn7+bsZ1uRuALt+so8u4tew5ds7myHKJNu/DHQ/CvDcgdqHd0dwyTSRKKY9reUdpfu7XjLcerM66fae4/9Nl/GfuNuITvHx2RocPPPollK4OPzwPx3bYHdEt0USilMoRfk4HLzarxOJXW/BIvWDGLNtDy8FL+T76oHd3F/YvDE9NAWcATHoCzue9ycU0kSilclRQEX8+6lSXWb0aE1qiAK9P28wjI1ey4cApu0OzT7FQiJwEZw/D950h6bLdEd0UTSRKKVvUDS3G9B6NGPJEXQ6fSeDRkVEMmLqJo2e9tLtw6N3QcSTsXwk/9XcNAJlHaCJRStnG4RAevSuERa+2oGeLyvy4+TCtBi9h5JJYLiV54eyMtTtB8zdg47eucbnyCI8lEhEJFZHFIrJNRGJEpK9VP1VENlnLPhHZZNVXEJGLbttGux2rvohsEZFYERku4pqSTUT8rePFisgaEangqc+jlPKcwv5O3mh7J/P7N6Nh5VJ89PMO7hu6jAW/H8XbRiin+d+hRkf49R3YPtfuaDLFk1ckScBAY0x1IALoJSI1jDFPGmPCjDFhwHRghlub3anbjDE93OpHAd2BqtbS1qrvBpwyxlQBhgL/9eDnUUp5WIVShfjquXAmdG2A0yG8MCGa58atI/ZPL+ou7HBAx1FQNgymvwBHttgdUYY8lkiMMYeNMRuscjywDQhO3W5dVTwBTE7vOCJSBgg0xqwyrl9NJgAdrc0dgPFWeRrQOvVqRSmVdzWrFsTP/Zrx9sM12HjgFG0/XcYHP/7OmYte0l3Yr6CrJ1dAUZgUCfFH7Y4oXTnyjMS65VQPWONW3RQ4aozZ5VZXUUQ2ishSEWlq1QUDcW77xHE1IQUDBwGMMUnAGaBk9n8CpVRO8/Vx0K1JRRa/2oLHw0P4euVeWg1ewpS1B0j2hu7CRW6Hp6fAxZMw5WlIvGh3RDfk8UQiIoVx3cLqZ4w567bpKa69GjkMlDPG1AMGAJNEJBBI6woj9V9RetvcY+guItEiEn3s2LFb+RhKKZuUKuzP/z1ahzmvNKFiqUL8fcYWOny+guh9J+0OzfPK1IVHx8Af0TDrlVzbk8ujiUREfHElke+MMTPc6p3Ao8DU1DpjzCVjzAmrvB7YDVTDdQUS4nbYEOCQVY4DQt2OWRT4y78uY8wYY0y4MSY8KEinBVUqL6oVXJQfejRkWGQYx+Mv02n0KvpO2cjhM7n3N/VsUb0dtH4Xtk6DpR/ZHU2aPNlrS4CxwDZjzJDrNrcBthtj4tz2DxIRH6tcCddD9T3GmMNAvIhEWMd8FphlNZsNPGeVOwGLjNd18VDKe4gIHcKCWfRqc3q3qsK8rUdoNXgpIxbtIiExH3cXbtIf6j4FS/4DW2dkvH8O8+QVSWOgM9DKrUvvg9a2SP76kL0ZsFlEfsP14LyHMSb16qIn8BUQi+tKZZ5VPxYoKSKxuG6H5f3xmJVSGSro52TgfXewcEBzmlUrxeD5O7l36FJ+3nokf3YXFoF2wyA0Av7XE/5Yb3dE15B8+aWnIzw83ERHR9sdhlIqG62MPc77c2LYefQcjauU5N12Nal2WxG7w8p+54/Dly1dQ6i8uAiKBmfcJpuIyHpjTHha2/TNdqVUnte4Sinm9mnKe+1qsCXuDA8MW857s2M4cyGfdRcuVAqe/h4un4fJka4/cwFNJEqpfMHp4+D5xhVZ8lpLIu8OZcKqfTT7eDGjl+7m4uV89PykdHV4fBwc3QozukOK/TNPaiJRSuUrJQr58e9HavNj76bcVa4YH87bTovBi5m05gCJyfb/0M0WVe+F+/8D23+ERf+0OxpNJEqp/KlG2UDGdWnA1O4RhBQvyJszt3DvkKXM+e1Q/pj/5J4eUL8LrBgKmybZGoomEqVUvnZPpZJM69GQr54Nx9/pQ+/JG2k3YgVLdvyZt3t4icCDH0PF5jC7D+xfZVsomkiUUvmeiNCmxm3M7duUoU/W5czFRJ4ft47IMatZvz8PT6jl4wtPjIfi5WHq3+DkXlvC0ESilPIaPg7hkXohLBrYgvfb12T3sXM8NiqKFydEs/NovN3h3ZoCxeGpqZCS7OrJlXA24zbZTBOJUsrr+DkdPNeoAktfa8mr91Vj9e4T3P/pMgZ8v4mDJy/YHd7NK1UFnpgAJ2JhWldITsrR02siUUp5rUL+Tl5pVZVlr7fkxaaVXDM0frKE92bHcPzcJbvDuzmVmsODgyH2V5j/jxw9tSYSpZTXK17IjzcfrM7S11rQqX4IE1fvp9lHixkyfwfxCXnopcbwLhDxMqwZBdFf59hpdYgUpZS6zu5j5xgyfyc/bTlM8YK+9GpZhWciyhPg62N3aBlLfVYSuxA6z4BKLbLlsOkNkaKJRCmlbmBz3Gk+/mUHy3cdp0zRAPq1qcpjd4Xg9MnlN3MSzsLX98PZP+CFhVCqapYPqWNtKaXULagTUoyJ3e5h0gv3UDowgDemb+G+T5cxb8vh3P0OSkCga6pehy9MehIueHYSME0kSimVgUZVSvG/lxsx+pn6OETo+d0GOny+kpWxx+0O7caKl4fI7+DMQfj+WUj23LMeTSRKKZUJIkLbWrfzc9+mfNSpDsfjL/G3r9bwt69W89vB03aHl7ZyEdD+M9i3HH4a6LGpejWRKKXUTXD6OHgiPJRFr7bg7YdrsO1wPB0+X0nPb9cT++c5u8P7q7qR0GQAbBgP0WM9cgqnR46qlFL5XICvD92aVOSJ8BDGrtjLl8v28EvMETrVD6Ffm2qULVbA7hCvavW2689qD3jk8NprSymlssGJc5f4fPFuvl29HwSejSjPyy2rUKKQn92hZQvt/utGE4lSypPiTl3g0wW7mLEhjoJ+Tl5sWoluTStS2D9v3wDSROJGE4lSKifsOhrP4Pk7+CXmKCUK+dGjeSU6R1SggF8eeKkxDZpI3GgiUUrlpI0HTjHk150s33WcoCL+9GpRmafuKYe/M28lFE0kbjSRKKXssHbvSQbP38HavScpWzSA3q2r0ql+CL65/S15iyYSN5pIlFJ2McawMvYEg+fvYNPB05QrUZC+ravSsV4wPg6xO7x06RApSimVC4gITaqWYubLjfj6+XCKBDgZ+MNv3Dt0KbPz8FzymkiUUiqHiQit7ryNOa80YfQzd+F0CH0mb+TB4cv5JeZI7h7HKw2aSJRSyiYOh9C2Vhnm9W3GsMgwLiWl8NLE9bQfsZLFO/7MMwlFE4lSStnMxyF0CAvm1/7N+LhTHU5duEyXcet4bFQUUbl5YEiLPmxXSqlc5nJSCj+sP8hnC2M5cjaBhpVKMvC+aoRXKGFbTNpry40mEqVUXpGQmMzktQf4fPFujp+7RPNqQQy4txp1Q4vleCyaSNxoIlFK5TUXLicxcdV+Ri/dzakLibSpfhsD7q1GjbKBORaDJhI3mkiUUnlVfEIi41bu48vle4hPSOKhOmXo36YqVUoX8fi5NZG40USilMrrzlxI5Mvlexi3ci8XE5PpGBZM3zZVKV+ykMfOqYnEjSYSpVR+cfL8Zb5Yupvxq/aRmGx4vH4IvVtXJdgDc6FoInGjiUQpld/8eTaBkUt2M2nNAQAiG4TSq2UVbgsMyLZzaCJxo4lEKZVfHTp9kc8WxfJD9EF8HELniPL0aFGZUoX9s3xsTSRuNJEopfK7AycuMGzhLmZujCPA14fnG1Wge7NKFCt467M1aiJxo4lEKeUtdh87x6cLdvHj5kMU9nPy70dr075u2Vs6li2j/4pIqIgsFpFtIhIjIn2t+qkissla9onIJrc2g0QkVkR2iMj9bvX1RWSLtW24iIhV728dL1ZE1ohIBU99HqWUymsqBxXms6fqMa9vUxpVKUmFkgU9ch5PTiKcBAw0xmwQkSLAehH51RjzZOoOIvIJcMYq1wAigZpAWWCBiFQzxiQDo4DuwGpgLtAWmAd0A04ZY6qISCTwX+DK8ZVSSsGdtwfyRec0LyayhceuSIwxh40xG6xyPLANCE7dbl1VPAFMtqo6AFOMMZeMMXuBWKCBiJQBAo0xq4zrPtwEoKNbm/FWeRrQOvVqRSmlVM7IkdF/rVtO9YA1btVNgaPGmF3WejBw0G17nFUXbJWvr7+mjTEmCdfVTck0zt9dRKJFJPrYsWNZ/jxKKaWu8ngiEZHCwHSgnzHmrNump7h6NQKQ1pWESac+vTbXVhgzxhgTbowJDwoKylzgSimlMsWTz0gQEV9cSeQ7Y8wMt3on8ChQ3233OCDUbT0EOGTVh6RR794mzjpmUeBkNn8MpZRS6fBkry0BxgLbjDFDrtvcBthujHG/ZTUbiLR6YlUEqgJrjTGHgXgRibCO+Swwy63Nc1a5E7DIeFt/ZqWUspknr0gaA52BLW5dfN80xszF1TvL/bYWxpgYEfke+B1Xj69eVo8tgJ7AN0ABXL215ln1Y4GJIhKL60ok0mOfRimlVJr0hUSllFIZsuWFRKWUUt7B665IROQYsP8Wm5cCjmdjOHmdfh/X0u/jKv0urpUfvo/yxpg0u716XSLJChGJvtGlnTfS7+Na+n1cpd/FtfL796G3tpRSSmWJJhKllFJZoonk5oyxO4BcRr+Pa+n3cZV+F9fK19+HPiNRSimVJXpFopRSKks0kSillMoSTSSZJCJtrZkbY0Xk73bHY5cbzXzp7UTER0Q2isiPdsdiNxEpJiLTRGS79e+kod0x2UVE+lv/T7aKyGQRCbA7Jk/QRJIJIuIDfA48ANQAnrJmdPRGqTNfVgcigF5e/F2464tr8jYFw4CfjTF3AnXx0u9FRIKBPkC4MaYW4EM+HQ9QE0nmNABijTF7jDGXgSm4Zmf0OhnNfOmNRCQEeAj4yu5Y7CYigUAzXAOqYoy5bIw5bWtQ9nICBaxpLgpydQqMfEUTSebcaPZGr3aDmS+90afA60CKzXHkBpWAY8A461bfVyJSyO6g7GCM+QMYDBwADgNnjDHz7Y3KMzSRZE6mZmL0JunMfOlVRORh4E9jzHq7Y8klnMBdwChjTD3gPOCVzxRFpDiuOxcVgbJAIRF5xt6oPEMTSebcaPZGr3SjmS+9VGOgvYjsw3XLs5WIfGtvSLaKA+KMMalXqdNwJRZv1AbYa4w5ZoxJBGYAjWyOySM0kWTOOqCqiFQUET9cD8xm2xyTLTKY+dLrGGMGGWNCjDEVcP27WGSMyZe/dWaGMeYIcFBE7rCqWuOarM4bHQAiRKSg9f+mNfm044FH52zPL4wxSSLyCvALrp4XXxtjYmwOyy7pzXypFEBv4Dvrl649QBeb47GFMWaNiEwDNuDq7biRfDpUig6RopRSKkv01pZSSqks0USilFIqSzSRKKWUyhJNJEoppbJEE4lSSqks0USilM1EpIWInLGGFNkhIsusN+Zv9XgVRORpt/XnRWRE9kSr1F9pIlEqd1hujKlnjLkD14ixI0Sk9S0eqwLwdEY7KZVdNJEolQ7rt/vt1uCDW0XkOxFpIyIrRWSXiDSw9msgIlHWVUVU6pvdIjJARL62yrWtYxRM75zGmE3AP4FXrHZBIjJdRNZZS2Or/j0RmSgii6xYXrQO8SHQVEQ2iUh/q66siPxs7fdRdn9PyssZY3TRRZcbLLh+u08CauP6xWs98DWugTw7AP+z9gsEnFa5DTDdKjuAZcAjQDTQOI1ztAB+vK4uDNcwNACTgCZWuZxb/XvAb0ABoBSuEarLXn884Hlcb5gXBQKA/UCo3d+tLvln0SFSlMrYXmPMFgARiQEWGmOMiGzBlWjA9UN6vIhUxTUytC+AMSZFRJ4HNgNfGGNWZvKc7iNOtwFquIZrAiBQRIpY5VnGmIvARRFZjGvunNNpHG+hMeaM9Rl+B8pz7dQISt0yTSRKZeySWznFbT2Fq/+HPgAWG2MeseZpWeLWpipwDtfVQmbV4+oAfw6goZUwrrASy/VjHN1ozCP3z5CM/t9X2UifkSiVPYoCf1jl51MrRaQorqlnmwElRaRTRgcSkTrA27imdwaYj/W8xNoe5rZ7BxEJEJGSuG5prQPigSIolUM0kSiVPT4C/k9EVuIaITrVUGCkMWYn0A34UERKp9G+aWr3X1wJpI8xZqG1rQ8QLiKbrdtSPdzarQV+AlYDHxhjDuG6jZYkIr+5PWxXymN09F+l8igReQ84Z4wZbHcsyrvpFYlSSqks0SsSpZRSWaJXJEoppbJEE4lSSqks0USilFIqSzSRKKWUyhJNJEoppbLk/wFc1OsuFmdBWgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(train_error, label='Train')\n",
    "plt.plot(test_error, label='Test')\n",
    "plt.xlabel(\"max Depth\")\n",
    "plt.ylabel(\"Error\")\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Voting regressor (1 points):\n",
    "\n",
    "The voting regressor should have at least 3 individual models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/sanjanathinderu/opt/anaconda3/lib/python3.9/site-packages/sklearn/ensemble/_voting.py:528: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n",
      "/Users/sanjanathinderu/opt/anaconda3/lib/python3.9/site-packages/sklearn/linear_model/_stochastic_gradient.py:1503: ConvergenceWarning: Maximum number of iteration reached before convergence. Consider increasing max_iter to improve the fit.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "VotingRegressor(estimators=[('dt', DecisionTreeRegressor(max_depth=20)),\n",
       "                            ('svr', SVR(C=10, epsilon=0.01)),\n",
       "                            ('sgd', SGDRegressor(max_iter=10000))])"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.tree import DecisionTreeRegressor\n",
    "from sklearn.linear_model import SGDRegressor \n",
    "from sklearn.svm import SVR \n",
    "from sklearn.ensemble import VotingRegressor\n",
    "\n",
    "\n",
    "dtree_reg = DecisionTreeRegressor(max_depth=20)\n",
    "svm_reg = SVR(kernel=\"rbf\", C=10, epsilon=0.01, gamma='scale') \n",
    "sgd_reg = SGDRegressor(max_iter=10000, tol=1e-3)\n",
    "\n",
    "voting_reg = VotingRegressor(\n",
    "            estimators=[('dt', dtree_reg), \n",
    "                        ('svr', svm_reg), \n",
    "                        ('sgd', sgd_reg)])\n",
    "\n",
    "voting_reg.fit(train_x, train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train RMSE: 19167.15999352064\n"
     ]
    }
   ],
   "source": [
    "#Train RMSE\n",
    "train_pred = voting_reg.predict(train_x)\n",
    "\n",
    "train_mse = mean_squared_error(train_y, train_pred)\n",
    "\n",
    "train_rmse = np.sqrt(train_mse)\n",
    "\n",
    "print('Train RMSE: {}' .format(train_rmse))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test RMSE: 23419.458279162285\n"
     ]
    }
   ],
   "source": [
    "#Test RMSE\n",
    "test_pred = voting_reg.predict(test_x)\n",
    "\n",
    "test_mse = mean_squared_error(test_y, test_pred)\n",
    "\n",
    "test_rmse = np.sqrt(test_mse)\n",
    "\n",
    "print('Test RMSE: {}' .format(test_rmse))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/sanjanathinderu/opt/anaconda3/lib/python3.9/site-packages/sklearn/ensemble/_voting.py:528: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n",
      "/Users/sanjanathinderu/opt/anaconda3/lib/python3.9/site-packages/sklearn/linear_model/_stochastic_gradient.py:1503: ConvergenceWarning: Maximum number of iteration reached before convergence. Consider increasing max_iter to improve the fit.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "VotingRegressor(estimators=[('dt', DecisionTreeRegressor(max_depth=20)),\n",
       "                            ('svr', SVR(C=10, epsilon=0.01)),\n",
       "                            ('sgd', SGDRegressor(max_iter=10000))],\n",
       "                weights=[0.4, 0.4, 0.2])"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dtree_reg = DecisionTreeRegressor(max_depth=20)\n",
    "svm_reg = SVR(kernel=\"rbf\", C=10, epsilon=0.01, gamma='scale') \n",
    "sgd_reg = SGDRegressor(max_iter=10000, tol=1e-3)\n",
    "\n",
    "voting_reg = VotingRegressor(\n",
    "            estimators=[('dt', dtree_reg), ('svr', svm_reg), ('sgd', sgd_reg)],\n",
    "                        weights=[0.4, 0.4, 0.2])\n",
    "\n",
    "voting_reg.fit(train_x, train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train RMSE: 22119.202724423536\n"
     ]
    }
   ],
   "source": [
    "#Train RMSE\n",
    "train_pred = voting_reg.predict(train_x)\n",
    "\n",
    "train_mse = mean_squared_error(train_y, train_pred)\n",
    "\n",
    "train_rmse = np.sqrt(train_mse)\n",
    "\n",
    "print('Train RMSE: {}' .format(train_rmse))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test RMSE: 24760.32586419216\n"
     ]
    }
   ],
   "source": [
    "#Test RMSE\n",
    "test_pred = voting_reg.predict(test_x)\n",
    "\n",
    "test_mse = mean_squared_error(test_y, test_pred)\n",
    "\n",
    "test_rmse = np.sqrt(test_mse)\n",
    "\n",
    "print('Test RMSE: {}' .format(test_rmse))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A Boosting model: (1 point)\n",
    "\n",
    "Build either an Adaboost or a GradientBoost model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adaboost Model "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/sanjanathinderu/opt/anaconda3/lib/python3.9/site-packages/sklearn/utils/validation.py:993: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "AdaBoostRegressor(base_estimator=DecisionTreeRegressor(max_depth=10),\n",
       "                  learning_rate=0.1, n_estimators=500)"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.ensemble import AdaBoostRegressor \n",
    "\n",
    "\n",
    "ada_reg = AdaBoostRegressor( \n",
    "            DecisionTreeRegressor(max_depth=10), n_estimators=500, \n",
    "            learning_rate=0.1) \n",
    "\n",
    "ada_reg.fit(train_x, train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train RMSE: 29647.778202916063\n"
     ]
    }
   ],
   "source": [
    "#Train RMSE\n",
    "train_pred = ada_reg.predict(train_x)\n",
    "\n",
    "train_mse = mean_squared_error(train_y, train_pred)\n",
    "\n",
    "train_rmse = np.sqrt(train_mse)\n",
    "\n",
    "print('Train RMSE: {}' .format(train_rmse))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test RMSE: 30331.892618689868\n"
     ]
    }
   ],
   "source": [
    "#Test RMSE\n",
    "test_pred = ada_reg.predict(test_x)\n",
    "\n",
    "test_mse = mean_squared_error(test_y, test_pred)\n",
    "\n",
    "test_rmse = np.sqrt(test_mse)\n",
    "\n",
    "print('Test RMSE: {}' .format(test_rmse))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/sanjanathinderu/opt/anaconda3/lib/python3.9/site-packages/sklearn/utils/validation.py:993: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "AdaBoostRegressor(base_estimator=DecisionTreeRegressor(max_depth=50),\n",
       "                  learning_rate=0.1, n_estimators=200)"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ada_reg = AdaBoostRegressor( \n",
    "            DecisionTreeRegressor(max_depth=50), n_estimators=200, \n",
    "            learning_rate=0.1) \n",
    "\n",
    "ada_reg.fit(train_x, train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train RMSE: 21756.294213326528\n"
     ]
    }
   ],
   "source": [
    "#Train RMSE\n",
    "train_pred = ada_reg.predict(train_x)\n",
    "\n",
    "train_mse = mean_squared_error(train_y, train_pred)\n",
    "\n",
    "train_rmse = np.sqrt(train_mse)\n",
    "\n",
    "print('Train RMSE: {}' .format(train_rmse))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test RMSE: 24428.1207329502\n"
     ]
    }
   ],
   "source": [
    "#Test RMSE\n",
    "test_pred = ada_reg.predict(test_x)\n",
    "\n",
    "test_mse = mean_squared_error(test_y, test_pred)\n",
    "\n",
    "test_rmse = np.sqrt(test_mse)\n",
    "\n",
    "print('Test RMSE: {}' .format(test_rmse))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural network: (1 point)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/sanjanathinderu/opt/anaconda3/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:1599: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n",
      "/Users/sanjanathinderu/opt/anaconda3/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (1000) reached and the optimization hasn't converged yet.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "MLPRegressor(max_iter=1000)"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.neural_network import MLPRegressor\n",
    "\n",
    "mlp_reg = MLPRegressor(hidden_layer_sizes=(100,), max_iter=1000)\n",
    "\n",
    "mlp_reg.fit(train_x, train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train RMSE: 61618.87517523439\n"
     ]
    }
   ],
   "source": [
    "#Train RMSE\n",
    "train_pred = mlp_reg.predict(train_x)\n",
    "\n",
    "train_mse = mean_squared_error(train_y, train_pred)\n",
    "\n",
    "train_rmse = np.sqrt(train_mse)\n",
    "\n",
    "print('Train RMSE: {}' .format(train_rmse))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test RMSE: 60589.11433440012\n"
     ]
    }
   ],
   "source": [
    "#Test RMSE\n",
    "test_pred = mlp_reg.predict(test_x)\n",
    "\n",
    "test_mse = mean_squared_error(test_y, test_pred)\n",
    "\n",
    "test_rmse = np.sqrt(test_mse)\n",
    "\n",
    "print('Test RMSE: {}' .format(test_rmse))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/sanjanathinderu/opt/anaconda3/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:1599: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n",
      "/Users/sanjanathinderu/opt/anaconda3/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (1000) reached and the optimization hasn't converged yet.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "MLPRegressor(hidden_layer_sizes=(1000,), max_iter=1000)"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "mlp_reg = MLPRegressor(hidden_layer_sizes=(1000,), max_iter=1000)\n",
    "\n",
    "mlp_reg.fit(train_x, train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train RMSE: 22917.014986427355\n"
     ]
    }
   ],
   "source": [
    "#Train RMSE\n",
    "train_pred = mlp_reg.predict(train_x)\n",
    "\n",
    "train_mse = mean_squared_error(train_y, train_pred)\n",
    "\n",
    "train_rmse = np.sqrt(train_mse)\n",
    "\n",
    "print('Train RMSE: {}' .format(train_rmse))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test RMSE: 25201.24775531519\n"
     ]
    }
   ],
   "source": [
    "#Test RMSE\n",
    "test_pred = mlp_reg.predict(test_x)\n",
    "\n",
    "test_mse = mean_squared_error(test_y, test_pred)\n",
    "\n",
    "test_rmse = np.sqrt(test_mse)\n",
    "\n",
    "print('Test RMSE: {}' .format(test_rmse))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Grid search (1 points)\n",
    "\n",
    "Perform either a full or randomized grid search on any model you want. There has to be at least two parameters for the search. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Randomized Grid Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 10 candidates, totalling 50 fits\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "RandomizedSearchCV(cv=5, estimator=DecisionTreeRegressor(),\n",
       "                   param_distributions=[{'max_depth': array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,\n",
       "       18, 19]),\n",
       "                                         'min_samples_leaf': array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,\n",
       "       18, 19])}],\n",
       "                   return_train_score=True, scoring='neg_mean_squared_error',\n",
       "                   verbose=1)"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "\n",
    "param_grid = [\n",
    "    {'min_samples_leaf': np.arange(1, 20), \n",
    "     'max_depth': np.arange(1,20)}\n",
    "  ]\n",
    "\n",
    "tree_reg = DecisionTreeRegressor()\n",
    "\n",
    "grid_search = RandomizedSearchCV(tree_reg, param_grid, cv=5, n_iter=10,\n",
    "                           scoring='neg_mean_squared_error', verbose=1,\n",
    "                           return_train_score=True)\n",
    "\n",
    "grid_search.fit(train_x, train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "29603.386504628426 {'min_samples_leaf': 2, 'max_depth': 17}\n",
      "30177.029175556403 {'min_samples_leaf': 8, 'max_depth': 7}\n",
      "30095.85056123195 {'min_samples_leaf': 14, 'max_depth': 6}\n",
      "30239.460277124766 {'min_samples_leaf': 12, 'max_depth': 7}\n",
      "32183.96822406894 {'min_samples_leaf': 3, 'max_depth': 19}\n",
      "30194.521443365637 {'min_samples_leaf': 11, 'max_depth': 5}\n",
      "29808.34379257369 {'min_samples_leaf': 14, 'max_depth': 3}\n",
      "29671.231869773113 {'min_samples_leaf': 10, 'max_depth': 2}\n",
      "29602.004644167973 {'min_samples_leaf': 5, 'max_depth': 3}\n",
      "29494.435734104834 {'min_samples_leaf': 5, 'max_depth': 1}\n"
     ]
    }
   ],
   "source": [
    "cvres = grid_search.cv_results_\n",
    "\n",
    "for mean_score, params in zip(cvres[\"mean_test_score\"], cvres[\"params\"]):\n",
    "    print(np.sqrt(-mean_score), params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'min_samples_leaf': 5, 'max_depth': 1}"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeRegressor(max_depth=1, min_samples_leaf=5)"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train RMSE: 29468.612804209773\n"
     ]
    }
   ],
   "source": [
    "#Train RMSE\n",
    "train_pred = grid_search.best_estimator_.predict(train_x)\n",
    "\n",
    "train_mse = mean_squared_error(train_y, train_pred)\n",
    "\n",
    "train_rmse = np.sqrt(train_mse)\n",
    "\n",
    "print('Train RMSE: {}' .format(train_rmse))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test RMSE: 28254.600330453908\n"
     ]
    }
   ],
   "source": [
    "#Test RMSE\n",
    "test_pred = grid_search.best_estimator_.predict(test_x)\n",
    "\n",
    "test_mse = mean_squared_error(test_y, test_pred)\n",
    "\n",
    "test_rmse = np.sqrt(test_mse)\n",
    "\n",
    "print('Test RMSE: {}' .format(test_rmse))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Discussion (4 points in total)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## List the train and test values of each model you built (1 points)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Baseline:\n",
    "train: 29582.833221810994\n",
    "Test: 28294.892856870818\n",
    "\n",
    "Decision  Tree:\n",
    "Train: 29181.176569565057\n",
    "Test: 28599.71696809674\n",
    "\n",
    "Voting Regission:\n",
    "Train: 19167.15999352064\n",
    "Test: 23419.458279162285\n",
    "\n",
    "Train: 22119.202724423536\n",
    "Test:  24760.32586419216\n",
    "\n",
    "Adaboost model:\n",
    "Train: 29647.778202916063\n",
    "Test:  30331.892618689868\n",
    "\n",
    "Train: 21756.294213326528\n",
    "Test:  24428.1207329502\n",
    "\n",
    "\n",
    "Neural Networks: \n",
    "Train: 61618.87517523439\n",
    "Test: 60589.11433440012\n",
    "\n",
    "Train: 22917.014986427355\n",
    "Test:  25201.24775531519\n",
    "\n",
    "Grid Search:\n",
    "Train: 29468.612804209773\n",
    "Test: 28254.600330453908\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Which model performs the best and why? (0.5 points) \n",
    "## How does it compare to baseline? (0.5 points)\n",
    "\n",
    "Hint: The best model is the one that has the lowest TEST RMSE value (regardless of any of the training values). If you select your model based on TRAIN values, you will lose points."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "The best model is Voting Regression with the TEST RMSE as 23419.4582. Compared with the baseline there is a difference of 4875. So the Voting Regression is better. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Is there any evidence of overfitting in the best model, why or why not? If there is, what did you do about it? (1 point)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Voting Regression is the best model. There is a significant difference between the test and train RMSE. There is approximately a 4000 difference. I tried using weight time to reduce overfitting, but it didn't make much of a difference."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Is there any overfitting in the other models (besides the best model), why or why not? If there is, what did you do about it? (1 point)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Aside from Voting Regression, the Neural Network model has served an overfitting. The test RMSE is 60589.1143344001 and the train RMSE is 61618.87517523439. I tried changing the values to get a better model and there was a significant difference for it as the test value is 25201.24775531519 and train is 22917.014986427355. The overall difference is nearly 2300. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}