Unit 1 Assignment Akshit Ram Pershad.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Akshit Ram Pershad\n",
    "### U87835413"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Unit 1 Assignment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this assignment, we will focus on sports analytics. This data set is made available by http://www.baseball-reference.com. It contains data about professional baseball (MLB) games played in the 2016 season. There are 2,427 games in the data set. Each row represents a single game. The goal is to predict the attendance at a home team’s game. This is an important task because most franchises want to predict the number of attendees for a variety of reasons including profits."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Description of Variables\n",
    "\n",
    "The description of variables are provided in \"Baseball - Data Dictionary.docx\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Goal\n",
    "\n",
    "Use the **baseball.csv** data set and build a model to predict **attendance_binary**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submission:\n",
    "\n",
    "Please save and submit this Jupyter notebook file. The correctness of the code matters for your grade. **Readability and organization of your code is also important.** You may lose points for submitting unreadable/undecipherable code. Therefore, use markdown cells to create sections, and use comments where necessary.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Section 1: (6 points in total)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Prep (5.5 points)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1703,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Common imports\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Get the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1704,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>attendance_binary</th>\n",
       "      <th>previous_attendance</th>\n",
       "      <th>previous_away_team_errors</th>\n",
       "      <th>previous_away_team_hits</th>\n",
       "      <th>previous_away_team_runs</th>\n",
       "      <th>game_type</th>\n",
       "      <th>previous_game_type</th>\n",
       "      <th>previous_home_team_errors</th>\n",
       "      <th>previous_home_team_hits</th>\n",
       "      <th>previous_home_team_runs</th>\n",
       "      <th>game_day</th>\n",
       "      <th>previous_game_day</th>\n",
       "      <th>temperature</th>\n",
       "      <th>wind_speed</th>\n",
       "      <th>sky</th>\n",
       "      <th>previous_game_duration</th>\n",
       "      <th>previous_homewin</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>43683</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>Night Game</td>\n",
       "      <td>Day Game</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>Monday</td>\n",
       "      <td>55</td>\n",
       "      <td>24</td>\n",
       "      <td>Overcast</td>\n",
       "      <td>2.933333</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>45785</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>2</td>\n",
       "      <td>Night Game</td>\n",
       "      <td>Day Game</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>3</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>Monday</td>\n",
       "      <td>48</td>\n",
       "      <td>7</td>\n",
       "      <td>Unknown</td>\n",
       "      <td>2.800000</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>48282</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>Night Game</td>\n",
       "      <td>Day Game</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>Monday</td>\n",
       "      <td>65</td>\n",
       "      <td>10</td>\n",
       "      <td>Cloudy</td>\n",
       "      <td>3.383333</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>21830</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>6</td>\n",
       "      <td>Day Game</td>\n",
       "      <td>Night Game</td>\n",
       "      <td>0</td>\n",
       "      <td>15</td>\n",
       "      <td>11</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>Tuesday</td>\n",
       "      <td>77</td>\n",
       "      <td>0</td>\n",
       "      <td>In Dome</td>\n",
       "      <td>3.233333</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>49289</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>Night Game</td>\n",
       "      <td>Day Game</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>Tuesday</td>\n",
       "      <td>Monday</td>\n",
       "      <td>81</td>\n",
       "      <td>12</td>\n",
       "      <td>Cloudy</td>\n",
       "      <td>2.633333</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   attendance_binary  previous_attendance  previous_away_team_errors  \\\n",
       "0                  0                43683                          2   \n",
       "1                  0                45785                          0   \n",
       "2                  0                48282                          0   \n",
       "3                  0                21830                          0   \n",
       "4                  0                49289                          2   \n",
       "\n",
       "   previous_away_team_hits  previous_away_team_runs   game_type  \\\n",
       "0                        6                        2  Night Game   \n",
       "1                        7                        2  Night Game   \n",
       "2                        8                        4  Night Game   \n",
       "3                        9                        6    Day Game   \n",
       "4                        4                        2  Night Game   \n",
       "\n",
       "  previous_game_type  previous_home_team_errors  previous_home_team_hits  \\\n",
       "0           Day Game                          0                        6   \n",
       "1           Day Game                          0                       10   \n",
       "2           Day Game                          2                        4   \n",
       "3         Night Game                          0                       15   \n",
       "4           Day Game                          1                        1   \n",
       "\n",
       "   previous_home_team_runs   game_day previous_game_day  temperature  \\\n",
       "0                        6  Wednesday            Monday           55   \n",
       "1                        3  Wednesday            Monday           48   \n",
       "2                        3  Wednesday            Monday           65   \n",
       "3                       11  Wednesday           Tuesday           77   \n",
       "4                        3    Tuesday            Monday           81   \n",
       "\n",
       "   wind_speed       sky  previous_game_duration  previous_homewin  \n",
       "0          24  Overcast                2.933333                 1  \n",
       "1           7   Unknown                2.800000                 1  \n",
       "2          10    Cloudy                3.383333                 0  \n",
       "3           0   In Dome                3.233333                 1  \n",
       "4          12    Cloudy                2.633333                 1  "
      ]
     },
     "execution_count": 1704,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "baseball = pd.read_csv(\"baseball.csv\")\n",
    "baseball.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Split the data into train and test 70/30"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1705,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "train, test = train_test_split(baseball, test_size=0.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check the missing values in Train and Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1706,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "attendance_binary            0\n",
       "previous_attendance          0\n",
       "previous_away_team_errors    0\n",
       "previous_away_team_hits      0\n",
       "previous_away_team_runs      0\n",
       "game_type                    0\n",
       "previous_game_type           0\n",
       "previous_home_team_errors    0\n",
       "previous_home_team_hits      0\n",
       "previous_home_team_runs      0\n",
       "game_day                     0\n",
       "previous_game_day            0\n",
       "temperature                  0\n",
       "wind_speed                   0\n",
       "sky                          0\n",
       "previous_game_duration       0\n",
       "previous_homewin             0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 1706,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.isna().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1707,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "attendance_binary            0\n",
       "previous_attendance          0\n",
       "previous_away_team_errors    0\n",
       "previous_away_team_hits      0\n",
       "previous_away_team_runs      0\n",
       "game_type                    0\n",
       "previous_game_type           0\n",
       "previous_home_team_errors    0\n",
       "previous_home_team_hits      0\n",
       "previous_home_team_runs      0\n",
       "game_day                     0\n",
       "previous_game_day            0\n",
       "temperature                  0\n",
       "wind_speed                   0\n",
       "sky                          0\n",
       "previous_game_duration       0\n",
       "previous_homewin             0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 1707,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.isna().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Prep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1708,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.preprocessing import OneHotEncoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Separating the target variable and input variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1709,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_targets = train[[ 'attendance_binary']] \n",
    "test_targets = test[[ 'attendance_binary']]\n",
    "\n",
    "train_inputs = train.drop(['attendance_binary'],  axis=1)\n",
    "test_inputs = test.drop(['attendance_binary'] , axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1710,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>previous_attendance</th>\n",
       "      <th>previous_away_team_errors</th>\n",
       "      <th>previous_away_team_hits</th>\n",
       "      <th>previous_away_team_runs</th>\n",
       "      <th>game_type</th>\n",
       "      <th>previous_game_type</th>\n",
       "      <th>previous_home_team_errors</th>\n",
       "      <th>previous_home_team_hits</th>\n",
       "      <th>previous_home_team_runs</th>\n",
       "      <th>game_day</th>\n",
       "      <th>previous_game_day</th>\n",
       "      <th>temperature</th>\n",
       "      <th>wind_speed</th>\n",
       "      <th>sky</th>\n",
       "      <th>previous_game_duration</th>\n",
       "      <th>previous_homewin</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>270</th>\n",
       "      <td>19072</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>6</td>\n",
       "      <td>Night Game</td>\n",
       "      <td>Night Game</td>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "      <td>7</td>\n",
       "      <td>Friday</td>\n",
       "      <td>Thursday</td>\n",
       "      <td>76</td>\n",
       "      <td>7</td>\n",
       "      <td>Cloudy</td>\n",
       "      <td>3.100000</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2096</th>\n",
       "      <td>40725</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "      <td>Day Game</td>\n",
       "      <td>Night Game</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>5</td>\n",
       "      <td>Thursday</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>61</td>\n",
       "      <td>16</td>\n",
       "      <td>Cloudy</td>\n",
       "      <td>3.050000</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1055</th>\n",
       "      <td>34036</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>Night Game</td>\n",
       "      <td>Night Game</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "      <td>Saturday</td>\n",
       "      <td>Friday</td>\n",
       "      <td>84</td>\n",
       "      <td>10</td>\n",
       "      <td>Cloudy</td>\n",
       "      <td>2.433333</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1014</th>\n",
       "      <td>27134</td>\n",
       "      <td>1</td>\n",
       "      <td>14</td>\n",
       "      <td>7</td>\n",
       "      <td>Night Game</td>\n",
       "      <td>Night Game</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>5</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>Tuesday</td>\n",
       "      <td>91</td>\n",
       "      <td>11</td>\n",
       "      <td>Cloudy</td>\n",
       "      <td>3.083333</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1320</th>\n",
       "      <td>41571</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "      <td>Night Game</td>\n",
       "      <td>Day Game</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>4</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>Saturday</td>\n",
       "      <td>68</td>\n",
       "      <td>17</td>\n",
       "      <td>Sunny</td>\n",
       "      <td>3.050000</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1638</th>\n",
       "      <td>32946</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>Night Game</td>\n",
       "      <td>Day Game</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>Tuesday</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>83</td>\n",
       "      <td>6</td>\n",
       "      <td>Cloudy</td>\n",
       "      <td>2.816667</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1095</th>\n",
       "      <td>40419</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>2</td>\n",
       "      <td>Day Game</td>\n",
       "      <td>Night Game</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>3</td>\n",
       "      <td>Wednesday</td>\n",
       "      <td>Tuesday</td>\n",
       "      <td>88</td>\n",
       "      <td>15</td>\n",
       "      <td>Sunny</td>\n",
       "      <td>2.800000</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1130</th>\n",
       "      <td>25011</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>Day Game</td>\n",
       "      <td>Night Game</td>\n",
       "      <td>1</td>\n",
       "      <td>12</td>\n",
       "      <td>11</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>Saturday</td>\n",
       "      <td>78</td>\n",
       "      <td>0</td>\n",
       "      <td>In Dome</td>\n",
       "      <td>3.033333</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1294</th>\n",
       "      <td>39594</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>8</td>\n",
       "      <td>Day Game</td>\n",
       "      <td>Night Game</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>Sunday</td>\n",
       "      <td>Saturday</td>\n",
       "      <td>77</td>\n",
       "      <td>9</td>\n",
       "      <td>Cloudy</td>\n",
       "      <td>3.266667</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>860</th>\n",
       "      <td>20458</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>2</td>\n",
       "      <td>Night Game</td>\n",
       "      <td>Night Game</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>Tuesday</td>\n",
       "      <td>Monday</td>\n",
       "      <td>80</td>\n",
       "      <td>13</td>\n",
       "      <td>Sunny</td>\n",
       "      <td>2.900000</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1698 rows × 16 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      previous_attendance  previous_away_team_errors  previous_away_team_hits  \\\n",
       "270                 19072                          1                        8   \n",
       "2096                40725                          0                        6   \n",
       "1055                34036                          1                        4   \n",
       "1014                27134                          1                       14   \n",
       "1320                41571                          0                        8   \n",
       "...                   ...                        ...                      ...   \n",
       "1638                32946                          1                        8   \n",
       "1095                40419                          0                        9   \n",
       "1130                25011                          1                        6   \n",
       "1294                39594                          0                       11   \n",
       "860                 20458                          2                        7   \n",
       "\n",
       "      previous_away_team_runs   game_type previous_game_type  \\\n",
       "270                         6  Night Game         Night Game   \n",
       "2096                        4    Day Game         Night Game   \n",
       "1055                        1  Night Game         Night Game   \n",
       "1014                        7  Night Game         Night Game   \n",
       "1320                        2  Night Game           Day Game   \n",
       "...                       ...         ...                ...   \n",
       "1638                        1  Night Game           Day Game   \n",
       "1095                        2    Day Game         Night Game   \n",
       "1130                        0    Day Game         Night Game   \n",
       "1294                        8    Day Game         Night Game   \n",
       "860                         2  Night Game         Night Game   \n",
       "\n",
       "      previous_home_team_errors  previous_home_team_hits  \\\n",
       "270                           1                       11   \n",
       "2096                          0                       10   \n",
       "1055                          0                        6   \n",
       "1014                          0                       10   \n",
       "1320                          1                        8   \n",
       "...                         ...                      ...   \n",
       "1638                          0                        4   \n",
       "1095                          1                        9   \n",
       "1130                          1                       12   \n",
       "1294                          0                        7   \n",
       "860                           0                        4   \n",
       "\n",
       "      previous_home_team_runs   game_day previous_game_day  temperature  \\\n",
       "270                         7     Friday          Thursday           76   \n",
       "2096                        5   Thursday         Wednesday           61   \n",
       "1055                        5   Saturday            Friday           84   \n",
       "1014                        5  Wednesday           Tuesday           91   \n",
       "1320                        4     Sunday          Saturday           68   \n",
       "...                       ...        ...               ...          ...   \n",
       "1638                        2    Tuesday            Sunday           83   \n",
       "1095                        3  Wednesday           Tuesday           88   \n",
       "1130                       11     Sunday          Saturday           78   \n",
       "1294                        4     Sunday          Saturday           77   \n",
       "860                         4    Tuesday            Monday           80   \n",
       "\n",
       "      wind_speed      sky  previous_game_duration  previous_homewin  \n",
       "270            7   Cloudy                3.100000                 1  \n",
       "2096          16   Cloudy                3.050000                 1  \n",
       "1055          10   Cloudy                2.433333                 1  \n",
       "1014          11   Cloudy                3.083333                 0  \n",
       "1320          17    Sunny                3.050000                 1  \n",
       "...          ...      ...                     ...               ...  \n",
       "1638           6   Cloudy                2.816667                 1  \n",
       "1095          15    Sunny                2.800000                 1  \n",
       "1130           0  In Dome                3.033333                 1  \n",
       "1294           9   Cloudy                3.266667                 0  \n",
       "860           13    Sunny                2.900000                 1  \n",
       "\n",
       "[1698 rows x 16 columns]"
      ]
     },
     "execution_count": 1710,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1711,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1698, 19)"
      ]
     },
     "execution_count": 1711,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_x.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1712,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>attendance_binary</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>270</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2096</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1055</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1014</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1320</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1638</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1095</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1130</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1294</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>860</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1698 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      attendance_binary\n",
       "270                   0\n",
       "2096                  1\n",
       "1055                  1\n",
       "1014                  0\n",
       "1320                  1\n",
       "...                 ...\n",
       "1638                  1\n",
       "1095                  1\n",
       "1130                  0\n",
       "1294                  1\n",
       "860                   0\n",
       "\n",
       "[1698 rows x 1 columns]"
      ]
     },
     "execution_count": 1712,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_targets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1713,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1698, 1)"
      ]
     },
     "execution_count": 1713,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_targets.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Identify the numerical, binary and categorical columns manually "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1714,
   "metadata": {},
   "outputs": [],
   "source": [
    "numeric_columns = ['previous_attendance','previous_away_team_errors', 'previous_away_team_hits','previous_away_team_runs',\n",
    "                   'previous_home_team_errors', 'previous_home_team_hits', 'previous_home_team_runs', 'temperature',\n",
    "                   'wind_speed','previous_game_duration']\n",
    " \n",
    "binary_columns = ['previous_homewin']\n",
    " \n",
    "categorical_columns = ['sky']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1715,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['previous_attendance',\n",
       " 'previous_away_team_errors',\n",
       " 'previous_away_team_hits',\n",
       " 'previous_away_team_runs',\n",
       " 'previous_home_team_errors',\n",
       " 'previous_home_team_hits',\n",
       " 'previous_home_team_runs',\n",
       " 'temperature',\n",
       " 'wind_speed',\n",
       " 'previous_game_duration']"
      ]
     },
     "execution_count": 1715,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "numeric_columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1716,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['previous_homewin']"
      ]
     },
     "execution_count": 1716,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "binary_columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1717,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['sky']"
      ]
     },
     "execution_count": 1717,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "categorical_columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1718,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Numeric transformer:\n",
    "\n",
    "numeric_transformer = Pipeline(steps=[\n",
    "                ('imputer', SimpleImputer(strategy='median')),\n",
    "                ('scaler', StandardScaler())])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1719,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Categorical transformer:\n",
    "\n",
    "categorical_transformer = Pipeline(steps=[\n",
    "    ('imputer', SimpleImputer(strategy='constant', fill_value='unknown')),\n",
    "    ('onehot', OneHotEncoder(handle_unknown='ignore'))])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1720,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Binary transformer:\n",
    "\n",
    "binary_transformer = Pipeline(steps=[\n",
    "    ('imputer', SimpleImputer(strategy='most_frequent'))])\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1721,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Column transformer:\n",
    "\n",
    "preprocessor = ColumnTransformer([\n",
    "        ('num', numeric_transformer, numeric_columns),\n",
    "        ('cat', categorical_transformer, categorical_columns),\n",
    "        ('binary', binary_transformer, binary_columns)],\n",
    "        remainder='drop')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Performing Fit and transform for the train data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1722,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-1.12371621,  0.57666325, -0.20719118, ...,  0.        ,\n",
       "         0.        ,  1.        ],\n",
       "       [ 1.05787587, -0.72716391, -0.78017264, ...,  0.        ,\n",
       "         0.        ,  1.        ],\n",
       "       [ 0.38394295,  0.57666325, -1.35315411, ...,  0.        ,\n",
       "         0.        ,  1.        ],\n",
       "       ...,\n",
       "       [-0.52534761,  0.57666325, -0.78017264, ...,  0.        ,\n",
       "         0.        ,  1.        ],\n",
       "       [ 0.94392488, -0.72716391,  0.65228102, ...,  0.        ,\n",
       "         0.        ,  0.        ],\n",
       "       [-0.98407336,  1.88049041, -0.49368191, ...,  1.        ,\n",
       "         0.        ,  1.        ]])"
      ]
     },
     "execution_count": 1722,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_x = preprocessor.fit_transform(train_inputs)\n",
    "train_x\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1723,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1698, 19)"
      ]
     },
     "execution_count": 1723,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_x.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Performing Transform for the test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1724,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.0814842 ,  1.88049041, -1.06666338, ...,  1.        ,\n",
       "         0.        ,  1.        ],\n",
       "       [ 1.17001331, -0.72716391, -1.63964484, ...,  0.        ,\n",
       "         0.        ,  0.        ],\n",
       "       [-0.29905768, -0.72716391, -0.20719118, ...,  0.        ,\n",
       "         1.        ,  0.        ],\n",
       "       ...,\n",
       "       [-0.49582715, -0.72716391,  1.22526249, ...,  0.        ,\n",
       "         0.        ,  0.        ],\n",
       "       [ 0.26445059, -0.72716391, -0.20719118, ...,  0.        ,\n",
       "         0.        ,  1.        ],\n",
       "       [-1.71775245, -0.72716391, -1.63964484, ...,  0.        ,\n",
       "         0.        ,  0.        ]])"
      ]
     },
     "execution_count": 1724,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_x = preprocessor.transform(test_inputs)\n",
    "test_x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1725,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(729, 19)"
      ]
     },
     "execution_count": 1725,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_x.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Find the Baseline (0.5 point)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1726,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-398 {color: black;background-color: white;}#sk-container-id-398 pre{padding: 0;}#sk-container-id-398 div.sk-toggleable {background-color: white;}#sk-container-id-398 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-398 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-398 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-398 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-398 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-398 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-398 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-398 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-398 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-398 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-398 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-398 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-398 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-398 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-398 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-398 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-398 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-398 div.sk-item {position: relative;z-index: 1;}#sk-container-id-398 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-398 div.sk-item::before, #sk-container-id-398 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-398 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-398 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-398 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-398 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-398 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-398 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-398 div.sk-label-container {text-align: center;}#sk-container-id-398 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-398 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-398\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>DummyClassifier(strategy=&#x27;most_frequent&#x27;)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-398\" type=\"checkbox\" checked><label for=\"sk-estimator-id-398\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">DummyClassifier</label><div class=\"sk-toggleable__content\"><pre>DummyClassifier(strategy=&#x27;most_frequent&#x27;)</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "DummyClassifier(strategy='most_frequent')"
      ]
     },
     "execution_count": 1726,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.dummy import DummyClassifier\n",
    "\n",
    "dummy_clf = DummyClassifier(strategy=\"most_frequent\")\n",
    "\n",
    "dummy_clf.fit(train_x, train_targets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1727,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1944,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Baseline Train Accuracy: 0.5141342756183745\n"
     ]
    }
   ],
   "source": [
    "#Baseline Train Accuracy\n",
    "dummy_train_pred = dummy_clf.predict(train_x)\n",
    "\n",
    "baseline_train_acc = accuracy_score(train_targets, dummy_train_pred)\n",
    "\n",
    "print('Baseline Train Accuracy: {}' .format(baseline_train_acc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1945,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Baseline Test Accuracy: 0.5294924554183813\n"
     ]
    }
   ],
   "source": [
    "#Baseline Test Accuracy\n",
    "dummy_test_pred = dummy_clf.predict(test_x)\n",
    "\n",
    "baseline_test_acc = accuracy_score(test_targets, dummy_test_pred)\n",
    "\n",
    "print('Baseline Test Accuracy: {}' .format(baseline_test_acc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Section 2: (3 points in total)\n",
    "\n",
    "Build three different SVM models (by changing the kernels, regularization, etc.). Generate their training and test values. Each model is worth 1 point. \n",
    "\n",
    "(Add cells as needed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SVM Model 1:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train an SVM model with linear kernel\n"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# Case 1 with default parameters, the Train accuracy is 0.820 and Test accuracy is 0.799\n",
    "\n",
    "from sklearn.svm import SVC\n",
    "lin_svm = SVC(kernel=\"linear\")\n",
    "lin_svm.fit(train_x, train_targets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1825,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\validation.py:1143: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-428 {color: black;background-color: white;}#sk-container-id-428 pre{padding: 0;}#sk-container-id-428 div.sk-toggleable {background-color: white;}#sk-container-id-428 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-428 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-428 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-428 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-428 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-428 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-428 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-428 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-428 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-428 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-428 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-428 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-428 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-428 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-428 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-428 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-428 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-428 div.sk-item {position: relative;z-index: 1;}#sk-container-id-428 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-428 div.sk-item::before, #sk-container-id-428 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-428 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-428 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-428 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-428 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-428 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-428 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-428 div.sk-label-container {text-align: center;}#sk-container-id-428 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-428 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-428\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>SVC(C=30, coef0=3, kernel=&#x27;linear&#x27;)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-428\" type=\"checkbox\" checked><label for=\"sk-estimator-id-428\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SVC</label><div class=\"sk-toggleable__content\"><pre>SVC(C=30, coef0=3, kernel=&#x27;linear&#x27;)</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "SVC(C=30, coef0=3, kernel='linear')"
      ]
     },
     "execution_count": 1825,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Case 2 : Updated default parameters with degree as 3, coef as 2 and C as 10.\n",
    "           #The Train accuracy is 0.819 and Test accuracy is 0.799\n",
    "    \n",
    "from sklearn.svm import SVC\n",
    "lin_svm = SVC(kernel=\"linear\", degree=3, coef0=3, C=30)\n",
    "lin_svm.fit(train_x, train_targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accuracy Check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1826,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1827,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8197879858657244"
      ]
     },
     "execution_count": 1827,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Predict the train values\n",
    "train_y_pred = lin_svm.predict(train_x)\n",
    "\n",
    "#Train accuracy\n",
    "accuracy_score(train_targets, train_y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1828,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7997256515775034"
      ]
     },
     "execution_count": 1828,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Predict the test values\n",
    "test_y_pred = lin_svm.predict(test_x)\n",
    "\n",
    "#Test accuracy\n",
    "accuracy_score(test_targets, test_y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Therefore, the best best fit for SVM model for linear Kernel is Case 2 when the parameters are set to kernel=\"linear\", degree=3, coef0=3, C=30. which gives the best Accuracy values for train as 0.819 and test as 0.799"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SVM Model 2:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train an SVM model with RBF kernel"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "## Case 1 : Upadeted the default parameters with C as 40 and Gamma as Scale\n",
    "            ## The accuracy values of Train is 0.967 and Test is 0.743, Which is the case of Overfitting. \n",
    "    \n",
    "rbf_svm = SVC(kernel=\"rbf\", C=40, gamma='scale')\n",
    "rbf_svm.fit(train_x, train_targets)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "## Case 2 : Updated the default parameters with C as 10 and Gamma as scale \n",
    "        ## The accuracy value of Train is 0.929 and Test is 0.758, Which is the case of Overfitting. \n",
    "\n",
    "rbf_svm = SVC(kernel=\"rbf\", C=10, gamma='scale')\n",
    "rbf_svm.fit(train_x, train_targets)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "## Case 3 : Updated the default parameters with C as 10 and gamma as Auto \n",
    "        ## The accuracy values of Train is 0.886 and Test is 0.775, which is the case of overfitting.\n",
    "        \n",
    "rbf_svm = SVC(kernel=\"rbf\", C=10, gamma='auto')\n",
    "rbf_svm.fit(train_x, train_targets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1821,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\validation.py:1143: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-427 {color: black;background-color: white;}#sk-container-id-427 pre{padding: 0;}#sk-container-id-427 div.sk-toggleable {background-color: white;}#sk-container-id-427 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-427 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-427 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-427 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-427 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-427 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-427 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-427 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-427 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-427 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-427 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-427 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-427 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-427 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-427 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-427 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-427 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-427 div.sk-item {position: relative;z-index: 1;}#sk-container-id-427 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-427 div.sk-item::before, #sk-container-id-427 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-427 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-427 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-427 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-427 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-427 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-427 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-427 div.sk-label-container {text-align: center;}#sk-container-id-427 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-427 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-427\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>SVC(C=0.1, gamma=&#x27;auto&#x27;)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-427\" type=\"checkbox\" checked><label for=\"sk-estimator-id-427\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SVC</label><div class=\"sk-toggleable__content\"><pre>SVC(C=0.1, gamma=&#x27;auto&#x27;)</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "SVC(C=0.1, gamma='auto')"
      ]
     },
     "execution_count": 1821,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Case 4 : Updated the default parameters with C as 0.1 and gamma as Auto \n",
    "        ## The accuracy values of Train is 0.829 and Test is 0.801 \n",
    "        \n",
    "rbf_svm = SVC(kernel=\"rbf\", C=0.1, gamma='auto')\n",
    "rbf_svm.fit(train_x, train_targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accuracy Check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1822,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1823,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8297997644287397"
      ]
     },
     "execution_count": 1823,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Predict the train values\n",
    "train_y_pred = rbf_svm.predict(train_x)\n",
    "\n",
    "#Train accuracy\n",
    "accuracy_score(train_targets, train_y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1824,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8010973936899863"
      ]
     },
     "execution_count": 1824,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Predict the test values\n",
    "test_y_pred = rbf_svm.predict(test_x)\n",
    "\n",
    "#Test accuracy\n",
    "accuracy_score(test_targets, test_y_pred) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Therefore, the best best fit for SVM model for RBK Kernel is Case 4 when the parameters are set to C as 0.1 and gamma as Auto.\n",
    "#### which gives the best Accuracy values for train as 0.829 and test as 0.801"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SVM Model 3:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train an SVM model with poly kernel"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "## Case 1 : Updated the parameters with degree as 3, coef as 10 and c as 30 \n",
    "        ## The accuracy values of Train is 0.929 and Test is 0.743, overfitting.\n",
    "    \n",
    "from sklearn.svm import SVC\n",
    "pol_svm = SVC(kernel=\"poly\", degree=3, coef0=10, C=30)\n",
    "pol_svm.fit(train_x, train_targets)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "## Case 2 : Updated the parameters with degree as 2, coef as 10 and c as 30 \n",
    "        ## The accuracy values of Train is 0.838 and Test is 0.788, overfitting\n",
    "    \n",
    "from sklearn.svm import SVC\n",
    "pol_svm = SVC(kernel=\"poly\", degree=2, coef0=10, C=30)\n",
    "pol_svm.fit(train_x, train_targets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1856,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\validation.py:1143: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-437 {color: black;background-color: white;}#sk-container-id-437 pre{padding: 0;}#sk-container-id-437 div.sk-toggleable {background-color: white;}#sk-container-id-437 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-437 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-437 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-437 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-437 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-437 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-437 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-437 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-437 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-437 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-437 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-437 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-437 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-437 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-437 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-437 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-437 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-437 div.sk-item {position: relative;z-index: 1;}#sk-container-id-437 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-437 div.sk-item::before, #sk-container-id-437 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-437 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-437 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-437 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-437 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-437 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-437 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-437 div.sk-label-container {text-align: center;}#sk-container-id-437 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-437 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-437\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>SVC(C=10, coef0=0.1, degree=2, kernel=&#x27;poly&#x27;)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-437\" type=\"checkbox\" checked><label for=\"sk-estimator-id-437\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SVC</label><div class=\"sk-toggleable__content\"><pre>SVC(C=10, coef0=0.1, degree=2, kernel=&#x27;poly&#x27;)</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "SVC(C=10, coef0=0.1, degree=2, kernel='poly')"
      ]
     },
     "execution_count": 1856,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Case 3 : Updated the parameters with degree as 2, coef as 0.1 and c as 0.1 \n",
    "        ## The accuracy values of Train is 0.830 and Test is 0.790\n",
    "    \n",
    "from sklearn.svm import SVC\n",
    "pol_svm = SVC(kernel=\"poly\", degree=2, coef0=0.1, C=10)\n",
    "pol_svm.fit(train_x, train_targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accuracy Check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1857,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1858,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.839811542991755"
      ]
     },
     "execution_count": 1858,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Predict the train values\n",
    "train_y_pred = pol_svm.predict(train_x)\n",
    "\n",
    "#Train accuracy\n",
    "accuracy_score(train_targets, train_y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1859,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7901234567901234"
      ]
     },
     "execution_count": 1859,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Predict the test values\n",
    "test_y_pred = pol_svm.predict(test_x)\n",
    "\n",
    "#Test accuracy\n",
    "accuracy_score(test_targets, test_y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Therefore, the best best fit for SVM model for poly Kernel is Case 3, when the parameters are set to kernel=\"poly\", degree=2, coef0=0.1, C=10 which gives the best Accuracy values for train as 0.830 and test as 0.790"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hence, Based on the predictions of the all 3 SVM models, RBF gives us the best accuracy value of Test which is 0.80. when the parameters are set to kernel=\"rbf\", C=0.1, gamma='auto'. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Section 3: (3 points in total)\n",
    "\n",
    "Build two different SGD models (by changing the penalty, etc. or adding polynomial terms) and one LogisticRregression model. Generate their training and test values. Each model is worth 1 point.\n",
    "\n",
    "(Add cells as needed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SGD Model 1:\n",
    "#### In SGD Model 1, we are changing the learning rate to find out the best test accuracy value with penalty as none. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1872,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\validation.py:1143: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-440 {color: black;background-color: white;}#sk-container-id-440 pre{padding: 0;}#sk-container-id-440 div.sk-toggleable {background-color: white;}#sk-container-id-440 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-440 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-440 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-440 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-440 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-440 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-440 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-440 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-440 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-440 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-440 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-440 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-440 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-440 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-440 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-440 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-440 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-440 div.sk-item {position: relative;z-index: 1;}#sk-container-id-440 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-440 div.sk-item::before, #sk-container-id-440 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-440 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-440 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-440 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-440 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-440 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-440 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-440 div.sk-label-container {text-align: center;}#sk-container-id-440 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-440 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-440\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>SGDClassifier(eta0=0.01, max_iter=100, penalty=None)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-440\" type=\"checkbox\" checked><label for=\"sk-estimator-id-440\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SGDClassifier</label><div class=\"sk-toggleable__content\"><pre>SGDClassifier(eta0=0.01, max_iter=100, penalty=None)</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "SGDClassifier(eta0=0.01, max_iter=100, penalty=None)"
      ]
     },
     "execution_count": 1872,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Case 1: max_iter=100, penalty= none, eta0=0.01\n",
    "## The accuracy values for Train is 0.794 and Test is 0.770\n",
    "\n",
    "from sklearn.linear_model import SGDClassifier \n",
    "        \n",
    "# tol = stopping criterion\n",
    "# eta0 = learning rate\n",
    "# penalty = regularization term\n",
    "# max_iter = number of passes over training data (i.e., epochs)\n",
    "\n",
    "sgd_logreg = SGDClassifier(max_iter=100, penalty= None , eta0=0.01)\n",
    "\n",
    "sgd_logreg.fit(train_x, train_targets)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "#### Case 2: max_iter=100, penalty= none, eta0=0.02\n",
    "#### The accuracy values for Train is 0.770 and Test is 0.746\n",
    "\n",
    "\n",
    "from sklearn.linear_model import SGDClassifier \n",
    "        \n",
    "### tol = stopping criterion\n",
    "### eta0 = learning rate\n",
    "### penalty = regularization term\n",
    "### max_iter = number of passes over training data (i.e., epochs)\n",
    "\n",
    "sgd_logreg = SGDClassifier(max_iter=100, penalty= None , eta0=0.02)\n",
    "\n",
    "sgd_logreg.fit(train_x, train_targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicted vs. Actual values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1873,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 1, 1, 0,\n",
       "       1, 1, 1, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0,\n",
       "       1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 0, 1, 0, 0, 1, 0, 1,\n",
       "       0, 0, 1, 1, 0, 1, 1, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 1, 0, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1,\n",
       "       1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 0, 1, 1,\n",
       "       0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 0, 1, 0, 0, 1, 1, 0, 1, 1, 0, 0, 1,\n",
       "       0, 0, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 1, 1,\n",
       "       1, 1, 1, 0, 1, 0, 0, 0, 1, 0, 1, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 1,\n",
       "       0, 1, 1, 1, 0, 0, 1, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 0, 1,\n",
       "       1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 0, 0, 0, 1, 0,\n",
       "       0, 1, 0, 0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 1, 1, 1, 0, 1, 1,\n",
       "       1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 0,\n",
       "       1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 1, 0, 1, 1,\n",
       "       1, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 1, 0, 1,\n",
       "       0, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0,\n",
       "       1, 1, 0, 1, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1,\n",
       "       1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 1, 0, 0, 1, 1, 0,\n",
       "       0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 1,\n",
       "       0, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 1, 0, 1,\n",
       "       1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 0, 0, 1, 0, 1, 1, 1, 1, 0,\n",
       "       1, 0, 0, 0, 1, 1, 0, 1, 1, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 0, 0,\n",
       "       0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0,\n",
       "       0, 0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1,\n",
       "       0, 0, 1, 0, 1, 1, 0, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0,\n",
       "       0, 1, 0, 0, 1, 0, 0, 1, 1, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 1,\n",
       "       0, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0,\n",
       "       0, 0, 1, 1, 0, 1, 0, 0, 1, 1, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 0,\n",
       "       1, 1, 1, 0, 1, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1,\n",
       "       0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 1,\n",
       "       0, 1, 1, 1, 1, 0, 0, 1, 1, 0, 1, 0, 0, 1, 1, 0, 0, 1, 1, 0, 1, 1,\n",
       "       0, 1, 0], dtype=int64)"
      ]
     },
     "execution_count": 1873,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sgd_logreg.predict(test_x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1874,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Predicted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>724</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>725</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>726</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>727</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>728</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>729 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     Predicted\n",
       "0            1\n",
       "1            1\n",
       "2            0\n",
       "3            1\n",
       "4            1\n",
       "..         ...\n",
       "724          1\n",
       "725          1\n",
       "726          0\n",
       "727          1\n",
       "728          0\n",
       "\n",
       "[729 rows x 1 columns]"
      ]
     },
     "execution_count": 1874,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a new DataFrame\n",
    "\n",
    "predictions = pd.DataFrame(sgd_logreg.predict(test_x), columns=['Predicted'])\n",
    "    \n",
    "predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1875,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Predicted</th>\n",
       "      <th>Actual</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>724</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>725</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>726</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>727</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>728</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>729 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     Predicted  Actual\n",
       "0            1       1\n",
       "1            1       1\n",
       "2            0       0\n",
       "3            1       1\n",
       "4            1       1\n",
       "..         ...     ...\n",
       "724          1       0\n",
       "725          1       1\n",
       "726          0       0\n",
       "727          1       0\n",
       "728          0       0\n",
       "\n",
       "[729 rows x 2 columns]"
      ]
     },
     "execution_count": 1875,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add the actual to the same DataFrame\n",
    "\n",
    "predictions['Actual'] = np.array(test_targets)\n",
    "    \n",
    "predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accuracy Check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1876,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7968197879858657"
      ]
     },
     "execution_count": 1876,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Predict the train values\n",
    "train_y_pred = sgd_logreg.predict(train_x)\n",
    "\n",
    "#Train accuracy\n",
    "accuracy_score(train_targets, train_y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1877,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7709190672153635"
      ]
     },
     "execution_count": 1877,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Predict the test values\n",
    "test_y_pred = sgd_logreg.predict(test_x)\n",
    "\n",
    "#Test accuracy\n",
    "accuracy_score(test_targets, test_y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Therefor, When increasing the learing rate eta0=0.01 their has been and increase in the accuracy value of test and provieded us with best the accuracy value of test i.e 0.770 when compared to the case2 ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Polynomial Regression with Degree2: \n",
    "\n",
    "This is done by creating the polynomial \"variables\" of the existing variables, then fitting them in a regular regression model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1940,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "\n",
    "# Create second degree terms and interaction terms\n",
    "poly_features = PolynomialFeatures(degree=2).fit(train_x)\n",
    "\n",
    "train_x_poly = poly_features.transform(train_x)\n",
    "\n",
    "test_x_poly = poly_features.transform(test_x)\n",
    "\n",
    "#Mind you, this will create the polynomial terms of the categorical variables too\n",
    "\n",
    "#if degree=3, then it creates all combinations: a, a^2, a^3, b, b^2, b^3, a.b, a^2.b, a.b^2, a^2.b^2 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1941,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\validation.py:1143: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\linear_model\\_logistic.py:458: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  n_iter_i = _check_optimize_result(\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-457 {color: black;background-color: white;}#sk-container-id-457 pre{padding: 0;}#sk-container-id-457 div.sk-toggleable {background-color: white;}#sk-container-id-457 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-457 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-457 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-457 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-457 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-457 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-457 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-457 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-457 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-457 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-457 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-457 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-457 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-457 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-457 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-457 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-457 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-457 div.sk-item {position: relative;z-index: 1;}#sk-container-id-457 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-457 div.sk-item::before, #sk-container-id-457 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-457 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-457 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-457 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-457 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-457 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-457 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-457 div.sk-label-container {text-align: center;}#sk-container-id-457 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-457 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-457\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LogisticRegression()</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-457\" type=\"checkbox\" checked><label for=\"sk-estimator-id-457\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression()</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "LogisticRegression()"
      ]
     },
     "execution_count": 1941,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log_reg = LogisticRegression()\n",
    "\n",
    "log_reg.fit(train_x_poly, train_targets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1942,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8368669022379269"
      ]
     },
     "execution_count": 1942,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Predict the train values\n",
    "train_y_pred = log_reg.predict(train_x_poly)\n",
    "\n",
    "#Train accuracy\n",
    "accuracy_score(train_targets, train_y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1943,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7832647462277091"
      ]
     },
     "execution_count": 1943,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Predict the test values\n",
    "test_y_pred = log_reg.predict(test_x_poly)\n",
    "\n",
    "#Test accuracy\n",
    "accuracy_score(test_targets, test_y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's regularize the polynomial regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1888,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\validation.py:1143: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-444 {color: black;background-color: white;}#sk-container-id-444 pre{padding: 0;}#sk-container-id-444 div.sk-toggleable {background-color: white;}#sk-container-id-444 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-444 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-444 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-444 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-444 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-444 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-444 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-444 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-444 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-444 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-444 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-444 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-444 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-444 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-444 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-444 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-444 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-444 div.sk-item {position: relative;z-index: 1;}#sk-container-id-444 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-444 div.sk-item::before, #sk-container-id-444 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-444 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-444 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-444 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-444 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-444 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-444 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-444 div.sk-label-container {text-align: center;}#sk-container-id-444 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-444 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-444\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LogisticRegression(C=0.01)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-444\" type=\"checkbox\" checked><label for=\"sk-estimator-id-444\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression(C=0.01)</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "LogisticRegression(C=0.01)"
      ]
     },
     "execution_count": 1888,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# C is the inverse of alpha. It can only be positive\n",
    "# Lower C means more regularization\n",
    "\n",
    "log_reg = LogisticRegression(penalty='l2', \n",
    "                              C=0.01)\n",
    "\n",
    "log_reg.fit(train_x_poly, train_targets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1889,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8239104829210836"
      ]
     },
     "execution_count": 1889,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Predict the train values\n",
    "train_y_pred = log_reg.predict(train_x_poly)\n",
    "\n",
    "#Train accuracy\n",
    "accuracy_score(train_targets, train_y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1939,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7818930041152263"
      ]
     },
     "execution_count": 1939,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Predict the train values\n",
    "test_y_pred = log_reg.predict(test_x_poly)\n",
    "\n",
    "#Train accuracy\n",
    "accuracy_score(test_targets, test_y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SGD Model 2:\n",
    "#### In SGD model 2, we are changing the learning rate and penalty to l1 to find out the best test accuracy value."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "### Case 1 : max_iter=1000, penalty='l1', eta0=0.01\n",
    "### The Accuracy values of Train is 0.817 and Test is 0.799\n",
    "\n",
    "from sklearn.linear_model import SGDClassifier \n",
    "\n",
    "### tol = stopping criterion\n",
    "### eta0 = learning rate\n",
    "### penalty = regularization term\n",
    "### max_iter = number of passes over training data (i.e., epochs)\n",
    "\n",
    "sgd_logreg = SGDClassifier(max_iter=1000, penalty='l1', eta0=0.01) \n",
    "\n",
    "sgd_logreg.fit(train_x, train_targets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1897,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\validation.py:1143: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-446 {color: black;background-color: white;}#sk-container-id-446 pre{padding: 0;}#sk-container-id-446 div.sk-toggleable {background-color: white;}#sk-container-id-446 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-446 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-446 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-446 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-446 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-446 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-446 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-446 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-446 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-446 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-446 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-446 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-446 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-446 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-446 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-446 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-446 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-446 div.sk-item {position: relative;z-index: 1;}#sk-container-id-446 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-446 div.sk-item::before, #sk-container-id-446 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-446 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-446 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-446 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-446 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-446 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-446 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-446 div.sk-label-container {text-align: center;}#sk-container-id-446 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-446 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-446\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>SGDClassifier(eta0=0.03, penalty=&#x27;l1&#x27;)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-446\" type=\"checkbox\" checked><label for=\"sk-estimator-id-446\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SGDClassifier</label><div class=\"sk-toggleable__content\"><pre>SGDClassifier(eta0=0.03, penalty=&#x27;l1&#x27;)</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "SGDClassifier(eta0=0.03, penalty='l1')"
      ]
     },
     "execution_count": 1897,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Case 2 :: max_iter=1000, penalty='l1', eta0=0.03\n",
    "## The Accuracy values of Train is 0.806 and Test is 0.796\n",
    "\n",
    "from sklearn.linear_model import SGDClassifier \n",
    "\n",
    "# tol = stopping criterion\n",
    "# eta0 = learning rate\n",
    "# penalty = regularization term\n",
    "# max_iter = number of passes over training data (i.e., epochs)\n",
    "\n",
    "sgd_logreg = SGDClassifier(max_iter=1000, penalty='l1', eta0=0.03) \n",
    "\n",
    "sgd_logreg.fit(train_x, train_targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicted vs. Actual values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1892,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0,\n",
       "       1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0,\n",
       "       1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 0, 1, 0, 1, 0, 1,\n",
       "       1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 0, 1, 0, 0, 0, 0, 1,\n",
       "       0, 0, 1, 1, 0, 1, 1, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1,\n",
       "       1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 1, 1,\n",
       "       1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1,\n",
       "       0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 1,\n",
       "       0, 0, 0, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 0, 1, 0, 1, 1, 0, 1, 1,\n",
       "       1, 0, 1, 0, 1, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 0, 0, 0, 1, 1, 1,\n",
       "       0, 1, 1, 1, 0, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 1, 0, 1,\n",
       "       1, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0,\n",
       "       0, 1, 0, 0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 1, 1, 0, 0, 1, 1,\n",
       "       1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 0, 0, 1, 0, 1, 0, 0,\n",
       "       1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 1,\n",
       "       1, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 0, 1,\n",
       "       0, 0, 0, 1, 1, 0, 1, 0, 1, 0, 1, 1, 0, 0, 1, 0, 1, 1, 1, 1, 0, 0,\n",
       "       1, 1, 0, 1, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1,\n",
       "       1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0, 1, 0, 0, 0, 1, 0,\n",
       "       0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 1,\n",
       "       0, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1,\n",
       "       1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0,\n",
       "       1, 0, 0, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0,\n",
       "       0, 1, 0, 0, 0, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0,\n",
       "       0, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 1, 1, 0, 1, 1, 1, 1,\n",
       "       1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1,\n",
       "       0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 0,\n",
       "       1, 1, 0, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 1,\n",
       "       0, 1, 0, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 0, 1, 0, 0, 0,\n",
       "       0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 0, 0, 1, 0, 1, 1, 1, 0,\n",
       "       0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1,\n",
       "       0, 1, 1, 0, 1, 1, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 1, 0, 1,\n",
       "       0, 1, 1, 0, 1, 0, 0, 1, 1, 0, 1, 0, 0, 1, 1, 0, 0, 1, 0, 0, 1, 1,\n",
       "       0, 1, 0], dtype=int64)"
      ]
     },
     "execution_count": 1892,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sgd_logreg.predict(test_x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1893,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Predicted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>724</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>725</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>726</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>727</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>728</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>729 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     Predicted\n",
       "0            1\n",
       "1            1\n",
       "2            0\n",
       "3            1\n",
       "4            1\n",
       "..         ...\n",
       "724          1\n",
       "725          1\n",
       "726          0\n",
       "727          1\n",
       "728          0\n",
       "\n",
       "[729 rows x 1 columns]"
      ]
     },
     "execution_count": 1893,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a new DataFrame\n",
    "\n",
    "predictions = pd.DataFrame(sgd_logreg.predict(test_x), columns=['Predicted'])\n",
    "\n",
    "predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1894,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Predicted</th>\n",
       "      <th>Actual</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>724</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>725</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>726</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>727</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>728</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>729 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     Predicted  Actual\n",
       "0            1       1\n",
       "1            1       1\n",
       "2            0       0\n",
       "3            1       1\n",
       "4            1       1\n",
       "..         ...     ...\n",
       "724          1       0\n",
       "725          1       1\n",
       "726          0       0\n",
       "727          1       0\n",
       "728          0       0\n",
       "\n",
       "[729 rows x 2 columns]"
      ]
     },
     "execution_count": 1894,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add the actual to the same DataFrame\n",
    "\n",
    "predictions['Actual'] = np.array(test_targets)\n",
    "\n",
    "predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1895,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.806831566548881"
      ]
     },
     "execution_count": 1895,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Predict the train values\n",
    "train_y_pred = sgd_logreg.predict(train_x)\n",
    "\n",
    "#Train accuracy\n",
    "accuracy_score(train_targets, train_y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1896,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7969821673525377"
      ]
     },
     "execution_count": 1896,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Predict the test values\n",
    "test_y_pred = sgd_logreg.predict(test_x)\n",
    "\n",
    "#Test accuracy\n",
    "accuracy_score(test_targets, test_y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Therefor, When increasing the learing rate eta0=0.03 their has been and increase in the accuracy value of test and \n",
    "#### provieded us with best the accuracy value of test i.e 0.796 when compared to the previous case1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Polynomial Regression with Degree as 3:\n",
    "\n",
    "This is done by creating the polynomial \"variables\" of the existing variables, then fitting them in a regular regression model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1927,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "\n",
    "# Create second degree terms and interaction terms\n",
    "poly_features = PolynomialFeatures(degree=3).fit(train_x)\n",
    "\n",
    "train_x_poly = poly_features.transform(train_x)\n",
    "\n",
    "test_x_poly = poly_features.transform(test_x)\n",
    "\n",
    "#Mind you, this will create the polynomial terms of the categorical variables too\n",
    "\n",
    "#if degree=3, then it creates all combinations: a, a^2, a^3, b, b^2, b^3, a.b, a^2.b, a.b^2, a^2.b^2 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1928,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\validation.py:1143: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\linear_model\\_logistic.py:458: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  n_iter_i = _check_optimize_result(\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-455 {color: black;background-color: white;}#sk-container-id-455 pre{padding: 0;}#sk-container-id-455 div.sk-toggleable {background-color: white;}#sk-container-id-455 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-455 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-455 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-455 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-455 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-455 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-455 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-455 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-455 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-455 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-455 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-455 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-455 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-455 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-455 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-455 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-455 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-455 div.sk-item {position: relative;z-index: 1;}#sk-container-id-455 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-455 div.sk-item::before, #sk-container-id-455 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-455 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-455 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-455 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-455 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-455 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-455 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-455 div.sk-label-container {text-align: center;}#sk-container-id-455 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-455 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-455\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LogisticRegression()</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-455\" type=\"checkbox\" checked><label for=\"sk-estimator-id-455\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression()</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "LogisticRegression()"
      ]
     },
     "execution_count": 1928,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log_reg = LogisticRegression()\n",
    "\n",
    "log_reg.fit(train_x_poly, train_targets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1929,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9411071849234394"
      ]
     },
     "execution_count": 1929,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Predict the train values\n",
    "train_y_pred = log_reg.predict(train_x_poly)\n",
    "\n",
    "#Train accuracy\n",
    "accuracy_score(train_targets, train_y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1930,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7215363511659808"
      ]
     },
     "execution_count": 1930,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Predict the test values\n",
    "test_y_pred = log_reg.predict(test_x_poly)\n",
    "\n",
    "#Test accuracy\n",
    "accuracy_score(test_targets, test_y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's regularize the polynomial regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1931,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\validation.py:1143: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-456 {color: black;background-color: white;}#sk-container-id-456 pre{padding: 0;}#sk-container-id-456 div.sk-toggleable {background-color: white;}#sk-container-id-456 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-456 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-456 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-456 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-456 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-456 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-456 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-456 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-456 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-456 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-456 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-456 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-456 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-456 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-456 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-456 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-456 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-456 div.sk-item {position: relative;z-index: 1;}#sk-container-id-456 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-456 div.sk-item::before, #sk-container-id-456 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-456 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-456 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-456 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-456 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-456 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-456 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-456 div.sk-label-container {text-align: center;}#sk-container-id-456 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-456 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-456\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LogisticRegression(C=0.01)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-456\" type=\"checkbox\" checked><label for=\"sk-estimator-id-456\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression(C=0.01)</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "LogisticRegression(C=0.01)"
      ]
     },
     "execution_count": 1931,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# C is the inverse of alpha. It can only be positive\n",
    "# Lower C means more regularization\n",
    "\n",
    "log_reg = LogisticRegression(penalty='l2', \n",
    "                              C=0.01)\n",
    "\n",
    "log_reg.fit(train_x_poly, train_targets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1932,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8621908127208481"
      ]
     },
     "execution_count": 1932,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Predict the train values\n",
    "train_y_pred = log_reg.predict(train_x_poly)\n",
    "\n",
    "#Train accuracy\n",
    "accuracy_score(train_targets, train_y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1938,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7818930041152263"
      ]
     },
     "execution_count": 1938,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Predict the test values\n",
    "test_y_pred = log_reg.predict(test_x_poly)\n",
    "\n",
    "#Test accuracy\n",
    "accuracy_score(test_targets, test_y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### After changing the degree value in polynomial regression we can observe an change in the accuracy values of train and test and it is a clear indication of overfitting as there is significant difference between Train and Test accuracy values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LogisticRegression Model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1914,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\linear_model\\_logistic.py:1173: FutureWarning: `penalty='none'`has been deprecated in 1.2 and will be removed in 1.4. To keep the past behaviour, set `penalty=None`.\n",
      "  warnings.warn(\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\validation.py:1143: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-453 {color: black;background-color: white;}#sk-container-id-453 pre{padding: 0;}#sk-container-id-453 div.sk-toggleable {background-color: white;}#sk-container-id-453 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-453 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-453 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-453 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-453 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-453 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-453 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-453 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-453 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-453 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-453 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-453 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-453 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-453 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-453 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-453 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-453 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-453 div.sk-item {position: relative;z-index: 1;}#sk-container-id-453 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-453 div.sk-item::before, #sk-container-id-453 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-453 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-453 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-453 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-453 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-453 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-453 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-453 div.sk-label-container {text-align: center;}#sk-container-id-453 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-453 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-453\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>LogisticRegression(penalty=&#x27;none&#x27;)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-453\" type=\"checkbox\" checked><label for=\"sk-estimator-id-453\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression(penalty=&#x27;none&#x27;)</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "LogisticRegression(penalty='none')"
      ]
     },
     "execution_count": 1914,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "log_reg = LogisticRegression(penalty='none')\n",
    "\n",
    "log_reg.fit(train_x, train_targets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicted vs. Actual values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1915,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 1, 1, 0,\n",
       "       1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0,\n",
       "       1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 0, 1, 0, 1, 0, 1,\n",
       "       1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 0, 1, 1, 0, 1, 0, 0, 0, 0, 1,\n",
       "       0, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 1,\n",
       "       1, 0, 1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 1, 1,\n",
       "       1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 0, 1, 1,\n",
       "       0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 0, 1, 0, 0, 1, 1, 0, 0, 1, 0, 0, 1,\n",
       "       0, 0, 0, 1, 1, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 1, 0, 1, 1, 0, 1, 1,\n",
       "       1, 0, 1, 0, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 0, 0, 0, 1, 1, 1,\n",
       "       0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 0, 1,\n",
       "       1, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0,\n",
       "       0, 1, 0, 0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 1, 1, 1, 0, 1, 1,\n",
       "       1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 0, 1, 0, 1, 0, 0,\n",
       "       1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 1,\n",
       "       1, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 0, 1,\n",
       "       0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 1, 0, 1, 1, 1, 1, 0, 0,\n",
       "       1, 1, 0, 1, 1, 1, 0, 1, 0, 1, 1, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1,\n",
       "       1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 0, 1, 0, 0, 1, 1, 0,\n",
       "       0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 0, 1, 1,\n",
       "       0, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1,\n",
       "       1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0,\n",
       "       1, 0, 0, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0,\n",
       "       0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0,\n",
       "       0, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 1, 1, 0, 1, 1, 1, 1,\n",
       "       1, 1, 1, 0, 1, 0, 1, 1, 0, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1,\n",
       "       0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 0,\n",
       "       0, 1, 0, 0, 1, 0, 0, 1, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1,\n",
       "       0, 1, 0, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 1, 0, 1, 0, 1, 1, 0, 0, 0,\n",
       "       0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 0,\n",
       "       1, 1, 1, 0, 1, 1, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1,\n",
       "       0, 1, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 1, 0, 1,\n",
       "       0, 1, 1, 0, 1, 0, 0, 1, 1, 0, 1, 0, 0, 1, 1, 0, 0, 1, 1, 0, 1, 1,\n",
       "       0, 1, 0], dtype=int64)"
      ]
     },
     "execution_count": 1915,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log_reg.predict(test_x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1916,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Predicted</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>724</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>725</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>726</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>727</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>728</th>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>729 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     Predicted\n",
       "0            1\n",
       "1            1\n",
       "2            0\n",
       "3            1\n",
       "4            1\n",
       "..         ...\n",
       "724          1\n",
       "725          1\n",
       "726          0\n",
       "727          1\n",
       "728          0\n",
       "\n",
       "[729 rows x 1 columns]"
      ]
     },
     "execution_count": 1916,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a new DataFrame\n",
    "\n",
    "predictions = pd.DataFrame(log_reg.predict(test_x), columns=['Predicted'])\n",
    "\n",
    "predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1917,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Predicted</th>\n",
       "      <th>Actual</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>724</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>725</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>726</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>727</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>728</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>729 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     Predicted  Actual\n",
       "0            1       1\n",
       "1            1       1\n",
       "2            0       0\n",
       "3            1       1\n",
       "4            1       1\n",
       "..         ...     ...\n",
       "724          1       0\n",
       "725          1       1\n",
       "726          0       0\n",
       "727          1       0\n",
       "728          0       0\n",
       "\n",
       "[729 rows x 2 columns]"
      ]
     },
     "execution_count": 1917,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Add the actual to the same DataFrame\n",
    "\n",
    "predictions['Actual'] = np.array(test_targets)\n",
    "\n",
    "predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Calculate the overall accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1918,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1919,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8191990577149588"
      ]
     },
     "execution_count": 1919,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Predict the train values\n",
    "train_y_pred = log_reg.predict(train_x)\n",
    "\n",
    "#Train accuracy\n",
    "accuracy_score(train_targets, train_y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1920,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8052126200274349"
      ]
     },
     "execution_count": 1920,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Predict the test values\n",
    "test_y_pred = log_reg.predict(test_x)\n",
    "\n",
    "#Test accuracy\n",
    "accuracy_score(test_targets, test_y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classification Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1921,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[270,  73],\n",
       "       [ 69, 317]], dtype=int64)"
      ]
     },
     "execution_count": 1921,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "#We usually create the confusion matrix on test set\n",
    "confusion_matrix(test_targets, test_y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classification Report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1922,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.80      0.79      0.79       343\n",
      "           1       0.81      0.82      0.82       386\n",
      "\n",
      "    accuracy                           0.81       729\n",
      "   macro avg       0.80      0.80      0.80       729\n",
      "weighted avg       0.81      0.81      0.81       729\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "\n",
    "#We usually create the classification report on test set\n",
    "print(classification_report(test_targets, test_y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Discussion (3 points in total)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## List the train and test values of each model you built (1 point)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Baseline Accuracy:\n",
    "###### Train Accuracy: 0.51\n",
    "###### Test Accuracy: 0.52\n",
    "-----------------------------------------------------------\n",
    "### SVM Model With Linear Kernel\n",
    "###### Train Accuracy: 0.81\n",
    "###### Test Accuracy: 0.79\n",
    "\n",
    "### SVM Model with RBF Kernel\n",
    "##### Train Accuracy:0.82\n",
    "##### Test Accuracy: 0.80\n",
    "\n",
    "### SVM Model with Poly Kernel\n",
    "##### Train Accuracy: 0.83\n",
    "##### Test Accuracy: 0.79\n",
    "\n",
    "### SGD Model 1 \n",
    "##### Train Accuracy: 0.79\n",
    "##### Test Accuracy: 0.77\n",
    "\n",
    "### Polynomial Regression with Degree2:\n",
    "##### Train Accuracy:0.83\n",
    "##### Test Accuracy: 0.78\n",
    "\n",
    "### After Regularizing the polynomial Regression\n",
    "##### Train Accuracy: 0.82\n",
    "##### Test Accuracy: 0.78\n",
    "\n",
    "----------------\n",
    "\n",
    "### SGD Model 2 \n",
    "##### Train Accuracy: 0.80\n",
    "##### Test Accuracy: 0.79\n",
    "\n",
    "### Polynomial Regression with Degree3: \n",
    "##### Train Accuracy:0.94\n",
    "##### Test Accuracy: 0.72\n",
    "\n",
    "### After Regularizing the polynomial Regression\n",
    "##### Train Accuracy: 0.86\n",
    "##### Test Accuracy: 0.78\n",
    "\n",
    "------------------------------------------------------\n",
    "\n",
    "### Logistic Regression \n",
    "##### Train Accuracy: 0.81\n",
    "##### Test Accuracy: 0.80"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Which model performs the best and why? (0.5 points) How does it compare to baseline? (0.5 points)\n",
    "\n",
    "Hint: The best model is the one that has the highest TEST score (regardless of any of the training values). If you select your model based on TRAIN values, you will lose points."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SVM Model with RBF Kernel has the best accuracy value i.e 0.80 when compared to other models. The baseline value of Test is 0.52. Baseline is used to identify the effectiveness of the model and The RBF Model test predections are above the baseline and it is considered to be a good fit. If the RBF model test accuracy was below 0.52 it will be considered as a bad model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Is there any evidence of overfitting in the best model, why or why not? If there is, what did you do about it? (0.5 points)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yes, Based on the analysis we can oberver that when increasing the value of c from 0.01 to 40 and changing the gamma from auto to scale, there has been an significant difference between the test and train values of the model which can be identified as a case of overfitting.\n",
    "\n",
    "Based on the prediction of the model, I identified when the gama was set to Auto and the C value was set to a lower value, the difference between the train and test values was reduced and provided me with the best accuracy value of test. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Is there any evidence of overfitting in the other models (besides the best model), why or why not? If there is, what did you do about it? (0.5 points)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Yes, We can observe overfitting in two models which are:\n",
    " \n",
    "1.  SVM Model with Poly Kernel: \n",
    "\n",
    "When the value of degree was increased, the difference between accuracy value of Train and Test increased when C=30,    coef=10. Therefor by analyzing the change in the parameters, and reducting the values to Degree=2, coef=0.1, c=0.1, we got the best accuracy values of Test and Train. \n",
    "\n",
    "2.  SGD Model 2 in Polynomial Regression when degree is set as 3:\n",
    "\n",
    "When the value of degree was set to 3, the difference between accuracy value of Train and Test increased and it had a gap of almost 22%, but when we performed regularization on the polynomial regression the accuracy value of Test and Train reduced moreover the gap between the both value was also reduce to 8%.\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}