K Nearest Neighbor.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "\n",
    "# K Nearest Neighbor\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "**K Nearest Neighbor** is a supervised learning algorithm where the data is 'trained' with data points corresponding to their classification to predict the class of a given data point, it takes into account the classes of the 'K' nearest data points and chooses the class in which the majority of the 'K' nearest data points belong to as the predicted class.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "**Visualization of the K-Nearest Neighbor algorithm**\n",
    "\n",
    "<img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-ML0101EN-SkillsNetwork/labs/Module%203/images/KNN_Diagram.png\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "In this case, we have data points of Class A and B. We want to predict what the star (test data point) is. If we consider a k value of 3 (3 nearest data points), we will obtain a prediction of Class B. Yet if we consider a k value of 6, we will obtain a prediction of Class A.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "In this sense, it is important to consider the value of k. Hopefully from this diagram, we should get a sense of what the K-Nearest Neighbors algorithm is. It considers the 'K' Nearest Neighbors (data points) when it predicts the classification of the test point.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting scikit-learn==0.23.1\n",
      "  Downloading scikit_learn-0.23.1-cp37-cp37m-manylinux1_x86_64.whl (6.8 MB)\n",
      "     |████████████████████████████████| 6.8 MB 23.0 MB/s            \n",
      "\u001b[?25hRequirement already satisfied: scipy>=0.19.1 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from scikit-learn==0.23.1) (1.7.3)\n",
      "Requirement already satisfied: numpy>=1.13.3 in /home/jupyterlab/conda/envs/python/lib/python3.7/site-packages (from scikit-learn==0.23.1) (1.21.4)\n",
      "Collecting joblib>=0.11\n",
      "  Downloading joblib-1.1.0-py2.py3-none-any.whl (306 kB)\n",
      "     |████████████████████████████████| 306 kB 70.1 MB/s            \n",
      "\u001b[?25hCollecting threadpoolctl>=2.0.0\n",
      "  Downloading threadpoolctl-3.1.0-py3-none-any.whl (14 kB)\n",
      "Installing collected packages: threadpoolctl, joblib, scikit-learn\n",
      "  Attempting uninstall: scikit-learn\n",
      "    Found existing installation: scikit-learn 0.20.1\n",
      "    Uninstalling scikit-learn-0.20.1:\n",
      "      Successfully uninstalled scikit-learn-0.20.1\n",
      "Successfully installed joblib-1.1.0 scikit-learn-0.23.1 threadpoolctl-3.1.0\n"
     ]
    }
   ],
   "source": [
    "!pip install scikit-learn==0.23.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn import preprocessing\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "<div id=\"about_dataset\">\n",
    "    <h2>Dataset</h2>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Imagine a telecommunications provider has segmented its customer base by service usage patterns, categorizing the customers into four groups. If demographic data can be used to predict group membership, the company can customize offers for individual prospective customers. It is a classification problem. The dataset is given with predefined labels. We need to build a model to be used to predict class of a new or unknown case.\n",
    "\n",
    "The example focuses on using demographic data such as region, age, and marital to predict usage patterns.\n",
    "\n",
    "The target field, called **custcat** has four possible values that correspond to the four customer groups as follows:\n",
    "- Basic Service\n",
    "- E-Service\n",
    "- Plus Service\n",
    "- Total Service\n",
    "\n",
    "Our objective is to build a classifier to predict the class of unknown cases. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Let's download the dataset. To download the data, we will use !wget to download it from IBM Object Storage.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2022-02-07 21:05:00--  https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-ML0101EN-SkillsNetwork/labs/Module%203/data/teleCust1000t.csv\n",
      "Resolving cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud (cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud)... 169.63.118.104\n",
      "Connecting to cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud (cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud)|169.63.118.104|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 36047 (35K) [text/csv]\n",
      "Saving to: ‘teleCust1000t.csv’\n",
      "\n",
      "teleCust1000t.csv   100%[===================>]  35.20K  17.7KB/s    in 2.0s    \n",
      "\n",
      "2022-02-07 21:05:02 (17.7 KB/s) - ‘teleCust1000t.csv’ saved [36047/36047]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget -O teleCust1000t.csv https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-ML0101EN-SkillsNetwork/labs/Module%203/data/teleCust1000t.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Load Data From CSV File\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>region</th>\n",
       "      <th>tenure</th>\n",
       "      <th>age</th>\n",
       "      <th>marital</th>\n",
       "      <th>address</th>\n",
       "      <th>income</th>\n",
       "      <th>ed</th>\n",
       "      <th>employ</th>\n",
       "      <th>retire</th>\n",
       "      <th>gender</th>\n",
       "      <th>reside</th>\n",
       "      <th>custcat</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>13</td>\n",
       "      <td>44</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>64.0</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>11</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>136.0</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>68</td>\n",
       "      <td>52</td>\n",
       "      <td>1</td>\n",
       "      <td>24</td>\n",
       "      <td>116.0</td>\n",
       "      <td>1</td>\n",
       "      <td>29</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>33</td>\n",
       "      <td>33</td>\n",
       "      <td>0</td>\n",
       "      <td>12</td>\n",
       "      <td>33.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>23</td>\n",
       "      <td>30</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>30.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   region  tenure  age  marital  address  income  ed  employ  retire  gender  \\\n",
       "0       2      13   44        1        9    64.0   4       5     0.0       0   \n",
       "1       3      11   33        1        7   136.0   5       5     0.0       0   \n",
       "2       3      68   52        1       24   116.0   1      29     0.0       1   \n",
       "3       2      33   33        0       12    33.0   2       0     0.0       1   \n",
       "4       2      23   30        1        9    30.0   1       2     0.0       0   \n",
       "\n",
       "   reside  custcat  \n",
       "0       2        1  \n",
       "1       6        4  \n",
       "2       2        3  \n",
       "3       1        1  \n",
       "4       4        3  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('teleCust1000t.csv')\n",
    "df.head()\n",
    "# df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### Let’s see how many of each class is in our data set\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3    281\n",
       "1    266\n",
       "4    236\n",
       "2    217\n",
       "Name: custcat, dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['custcat'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "**281 Plus Service, 266 Basic-service, 236 Total Service and 217 E-Service customers**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can easily explore our data using visualization techniques:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[<AxesSubplot:title={'center':'income'}>]], dtype=object)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAEICAYAAAC9E5gJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAASIUlEQVR4nO3dfbBcdX3H8feHYMES5aFgJk2iN9SUEWTqwy3qoPam0IKihNpi41AbKk7qFDva2qlBOq39g05sp452kDqpMEaxXFKQISPDKJN6q51qkSBPAVOiBAjEpCIIUUsb/PaPPddu0nuTu7m7e3fh/Zq5s2d/+ztnP3tY7ueesw9JVSFJem47bK4DSJLmnmUgSbIMJEmWgSQJy0CShGUgScIy0LNMki1JxuY6hzRs4ucMJEkeGUiSLAM9uyTZnuTMJB9OsiHJZ5I81Zw+Gm2btyTJ55P8Z5LHklzejB+W5M+SPJhkd7P+0c1tI0kqye8leTjJ40nek+SXk9yV5InJ7bTdz7uS3NfM/WKSl/R3j0gzYxno2excYBw4BtgITP7Cnwd8AXgQGAEWNfMALmx+lgMnAvMn12vzGmAZ8NvAx4BLgTOBU4C3J/mV5n7OAz4EvA04AfgqcE03H6DULb5moGeVJNuBdwOvB15fVWc24ycDm6vq+UleR6scFlbV3v3W3wRcX1VXNNdPAu4Bng8sBh4AFlfVI83tjwF/UFXXNtevB75aVR9LcjNwXVVd2dx2GLAHeFlVPdjL/SB1yiMDPZt9t235R8CRSQ4HlgAP7l8EjZ+ndcQw6UHgcGBB29iutuUfT3F9frP8EuDjzemjJ4DvA6F1JCINFMtAz0UPAy9uimF/j9L6JT7pxcBe9v2F38n9/H5VHdP28/yq+rdD2JbUU5aBnotuBXYCa5McleTIJKc3t10D/FGSpUnmA38FXDvNUcTBfBK4JMkpAEmOTnJ+Nx6A1G2WgZ5zquoZ4K3AS4GHgB20XgwGuAr4LPAVWq8P/Bfwh4d4PzcAHwHGkzxJ67WHN80qvNQjvoAsSfLIQJJkGUiSsAwkSVgGkiRaH6aZc8cff3yNjIx0vN4Pf/hDjjrqqO4H6jFz988wZgZz99uw5t68efP3quqEbmxrIMpgZGSE2267reP1JiYmGBsb636gHjN3/wxjZjB3vw1r7iRd+1oTTxNJkiwDSZJlIEnCMpAkYRlIkrAMJElYBpIkLANJEpaBJIkB+QTybI2suWnK8e1rz+lzEkkaTh4ZSJIsA0mSZSBJwjKQJGEZSJKwDCRJWAaSJCwDSRKWgSQJy0CShGUgScIykCRhGUiSsAwkSVgGkiQsA0kSHZRBknlJvpnkC83145LckuT+5vLYtrmXJNmWZGuSs3oRXJLUPZ0cGbwPuK/t+hpgU1UtAzY110lyMrASOAU4G7giybzuxJUk9cKMyiDJYuAc4FNtwyuA9c3yeuC8tvHxqnq6qh4AtgGndSWtJKknZnpk8DHgT4GftI0tqKqdAM3li5rxRcDDbfN2NGOSpAF1+MEmJHkLsLuqNicZm8E2M8VYTbHd1cBqgAULFjAxMTGDTe9rz549TExM8IFT9055+6Fssx8mcw+bYcw9jJnB3P02rLm76aBlAJwOnJvkzcCRwAuTXA3sSrKwqnYmWQjsbubvAJa0rb8YeHT/jVbVOmAdwOjoaI2NjXUcfmJigrGxMS5cc9OUt2+/oPNt9sNk7mEzjLmHMTOYu9+GNXc3HfQ0UVVdUlWLq2qE1gvD/1xVvwNsBFY101YBNzbLG4GVSY5IshRYBtza9eSSpK6ZyZHBdNYCG5JcBDwEnA9QVVuSbADuBfYCF1fVM7NOKknqmY7KoKomgIlm+THgjGnmXQZcNstskqQ+8RPIkiTLQJJkGUiSsAwkSVgGkiQsA0kSloEkCctAkoRlIEnCMpAkYRlIkrAMJElYBpIkLANJEpaBJAnLQJKEZSBJwjKQJGEZSJKwDCRJWAaSJCwDSRKWgSQJy0CShGUgScIykCRhGUiSsAwkSVgGkiQsA0kSloEkCctAkoRlIEnCMpAkYRlIkrAMJElYBpIkLANJEjMogyRHJrk1yZ1JtiT5y2b8uCS3JLm/uTy2bZ1LkmxLsjXJWb18AJKk2ZvJkcHTwK9W1S8BrwDOTvJaYA2wqaqWAZua6yQ5GVgJnAKcDVyRZF4PskuSuuSgZVAte5qrz2t+ClgBrG/G1wPnNcsrgPGqerqqHgC2Aad1M7QkqbtSVQef1PrLfjPwUuATVfXBJE9U1TFtcx6vqmOTXA58vaqubsavBG6uquv22+ZqYDXAggULXj0+Pt5x+D179jB//nzufuQHU95+6qKjO95mP0zmHjbDmHsYM4O5+21Ycy9fvnxzVY12Y1uHz2RSVT0DvCLJMcANSV5+gOmZahNTbHMdsA5gdHS0xsbGZhJlHxMTE4yNjXHhmpumvH37BZ1vsx8mcw+bYcw9jJnB3P02rLm7qaN3E1XVE8AErdcCdiVZCNBc7m6m7QCWtK22GHh0tkElSb0zk3cTndAcEZDk+cCZwLeAjcCqZtoq4MZmeSOwMskRSZYCy4Bbu5xbktRFMzlNtBBY37xucBiwoaq+kORrwIYkFwEPAecDVNWWJBuAe4G9wMXNaSZJ0oA6aBlU1V3AK6cYfww4Y5p1LgMum3U6SVJf+AlkSZJlIEmyDCRJWAaSJCwDSRKWgSQJy0CShGUgScIykCRhGUiSmOFXWA+rkem+2nrtOX1OIkmDzSMDSZJlIEmyDCRJWAaSJCwDSRKWgSQJy0CShGUgScIykCRhGUiSsAwkSVgGkiQsA0kSloEkCctAkoRlIEnCMpAkYRlIkrAMJElYBpIkLANJEpaBJAnLQJKEZSBJwjKQJGEZSJKwDCRJzKAMkixJ8uUk9yXZkuR9zfhxSW5Jcn9zeWzbOpck2ZZka5KzevkAJEmzN5Mjg73AB6rqZcBrgYuTnAysATZV1TJgU3Od5raVwCnA2cAVSeb1IrwkqTsOWgZVtbOqbm+WnwLuAxYBK4D1zbT1wHnN8gpgvKqerqoHgG3AaV3OLUnqolTVzCcnI8BXgJcDD1XVMW23PV5Vxya5HPh6VV3djF8J3FxV1+23rdXAaoAFCxa8enx8vOPwe/bsYf78+dz9yA86Wu/URUd3fF/dNJl72Axj7mHMDObut2HNvXz58s1VNdqNbR0+04lJ5gPXA++vqieTTDt1irH/1zhVtQ5YBzA6OlpjY2MzjfJTExMTjI2NceGamzpab/sFnd9XN03mHjbDmHsYM4O5+21Yc3fTjN5NlOR5tIrgc1X1+WZ4V5KFze0Lgd3N+A5gSdvqi4FHuxNXktQLM3k3UYArgfuq6qNtN20EVjXLq4Ab28ZXJjkiyVJgGXBr9yJLkrptJqeJTgfeCdyd5I5m7EPAWmBDkouAh4DzAapqS5INwL203ol0cVU90+3gkqTuOWgZVNW/MvXrAABnTLPOZcBls8glSeojP4EsSbIMJEmWgSQJy0CShGUgScIykCRhGUiSsAwkSVgGkiQsA0kSloEkCctAkoRlIEnCMpAkYRlIkrAMJElYBpIkLANJEpaBJAnLQJKEZSBJwjKQJGEZSJKwDCRJWAaSJCwDSRKWgSQJy0CShGUgScIykCRhGUiSsAwkSVgGkiTg8LkOMBdG1tw05fj2tef0OYkkDQaPDCRJloEkyTKQJDGDMkhyVZLdSe5pGzsuyS1J7m8uj2277ZIk25JsTXJWr4JLkrpnJkcGnwbO3m9sDbCpqpYBm5rrJDkZWAmc0qxzRZJ5XUsrSeqJg5ZBVX0F+P5+wyuA9c3yeuC8tvHxqnq6qh4AtgGndSeqJKlXUlUHn5SMAF+oqpc315+oqmPabn+8qo5Ncjnw9aq6uhm/Eri5qq6bYpurgdUACxYsePX4+HjH4ffs2cP8+fO5+5EfdLzuVE5ddHRXtnMwk7mHzTDmHsbMYO5+G9bcy5cv31xVo93YVrc/Z5ApxqZsm6paB6wDGB0drbGxsY7vbGJigrGxMS6c5nMDndp+QecZDsVk7mEzjLmHMTOYu9+GNXc3Heq7iXYlWQjQXO5uxncAS9rmLQYePfR4kqR+ONQy2AisapZXATe2ja9MckSSpcAy4NbZRZQk9dpBTxMluQYYA45PsgP4C2AtsCHJRcBDwPkAVbUlyQbgXmAvcHFVPdOj7JKkLjloGVTVO6a56Yxp5l8GXDabUJKk/vITyJIky0CSZBlIkrAMJElYBpIkLANJEpaBJAnLQJKEZSBJwjKQJGEZSJKwDCRJWAaSJCwDSRKWgSQJy0CShGUgSWIG/9LZc8nImpumHN++9pw+J5Gk/vLIQJJkGUiSLANJEpaBJAnLQJKE7yaaEd9lJOnZziMDSZJlIEmyDCRJWAaSJCwDSRKWgSQJy0CShJ8zmBU/fyDp2cIjA0mSZSBJ8jRRX02eVvrAqXu5sO0Uk6eVJM01y6AHpnstQZIGlaeJJEmWgSSph6eJkpwNfByYB3yqqtb26r6GXTdPK/n6g6RD0ZMySDIP+ATwa8AO4BtJNlbVvb24Px2cn4mQdCC9OjI4DdhWVd8BSDIOrAAsgx7r9Cij05Lo1vzp9Ho7/TBoWQ+Uxz8G+mfQ/yBLVXV/o8lvAWdX1bub6+8EXlNV722bsxpY3Vw9Cdh6CHd1PPC9WcadC+bun2HMDObut2HNfVJVvaAbG+rVkUGmGNundapqHbBuVneS3FZVo7PZxlwwd/8MY2Ywd78Nc+5ubatX7ybaASxpu74YeLRH9yVJmqVelcE3gGVJlib5GWAlsLFH9yVJmqWenCaqqr1J3gt8kdZbS6+qqi09uKtZnWaaQ+bun2HMDObut+d87p68gCxJGi5+AlmSZBlIkoa0DJKcnWRrkm1J1sx1nnZJliT5cpL7kmxJ8r5m/MNJHklyR/Pz5rZ1Lmkey9YkZ81h9u1J7m7y3daMHZfkliT3N5fHDlLuJCe17dM7kjyZ5P2DuL+TXJVkd5J72sY63r9JXt38d9qW5O+STPVW7l7n/psk30pyV5IbkhzTjI8k+XHbfv/kXOSeJnPHz4kB2dfXtmXenuSOZry7+7qqhuqH1gvS3wZOBH4GuBM4ea5zteVbCLyqWX4B8B/AycCHgT+ZYv7JzWM4AljaPLZ5c5R9O3D8fmN/DaxpltcAHxm03Ps9N74LvGQQ9zfwRuBVwD2z2b/ArcDraH2e52bgTXOQ+9eBw5vlj7TlHmmft992+pZ7mswdPycGYV/vd/vfAn/ei309jEcGP/2qi6r6b2Dyqy4GQlXtrKrbm+WngPuARQdYZQUwXlVPV9UDwDZaj3FQrADWN8vrgfPaxgct9xnAt6vqwQPMmbPcVfUV4PtT5Jnx/k2yEHhhVX2tWv/Xf6Ztnb7lrqovVdXe5urXaX2WaFr9zj3Nvp7OQO/rSc1f928HrjnQNg419zCWwSLg4bbrOzjwL9s5k2QEeCXw783Qe5vD6qvaTgcM0uMp4EtJNqf1dSEAC6pqJ7SKDnhRMz5IuSetZN//UQZ9f0Pn+3dRs7z/+Fx6F62/PictTfLNJP+S5A3N2KDk7uQ5MSiZJ70B2FVV97eNdW1fD2MZHPSrLgZBkvnA9cD7q+pJ4O+BXwBeAeykdbgHg/V4Tq+qVwFvAi5O8sYDzB2k3KT14cZzgX9qhoZhfx/IdDkHKn+SS4G9wOeaoZ3Ai6vqlcAfA/+Y5IUMRu5OnxODkLndO9j3j52u7uthLIOB/6qLJM+jVQSfq6rPA1TVrqp6pqp+AvwD/3dqYmAeT1U92lzuBm6glXFXc9g5efi5u5k+MLkbbwJur6pdMBz7u9Hp/t3Bvqdk5ix/klXAW4ALmtMRNKdaHmuWN9M6//6LDEDuQ3hOzHnmSUkOB94GXDs51u19PYxlMNBfddGc17sSuK+qPto2vrBt2m8Ak+8W2AisTHJEkqXAMlov/vRVkqOSvGBymdYLhPc0+VY101YBNzbLA5G7zT5/NQ36/m7T0f5tTiU9leS1zXPtd9vW6Zu0/vGqDwLnVtWP2sZPSOvfMyHJiU3u7wxC7k6fE4OQuc2ZwLeq6qenf7q+r3v5ynivfoA303qXzreBS+c6z37ZXk/rkOwu4I7m583AZ4G7m/GNwMK2dS5tHstWevxuhQPkPpHWOyruBLZM7lfg54BNwP3N5XGDlLvJ8bPAY8DRbWMDt79pldVO4H9o/fV20aHsX2CU1i+ybwOX03yTQJ9zb6N1nn3yOf7JZu5vNs+fO4HbgbfORe5pMnf8nBiEfd2Mfxp4z35zu7qv/ToKSdJQniaSJHWZZSBJsgwkSZaBJAnLQJKEZSBJwjKQJAH/C+ch7ZFQjClTAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df.hist(column='income', bins=50)\n",
    "# df['income'].hist(bins=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "**Feature Set**\n",
    "\n",
    "Let's define feature sets, X:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['region', 'tenure', 'age', 'marital', 'address', 'income', 'ed',\n",
       "       'employ', 'retire', 'gender', 'reside', 'custcat'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use scikit-learn library, we have to convert the Pandas data frame to a Numpy array:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  2.,  13.,  44.,   1.,   9.,  64.,   4.,   5.,   0.,   0.,   2.],\n",
       "       [  3.,  11.,  33.,   1.,   7., 136.,   5.,   5.,   0.,   0.,   6.],\n",
       "       [  3.,  68.,  52.,   1.,  24., 116.,   1.,  29.,   0.,   1.,   2.],\n",
       "       [  2.,  33.,  33.,   0.,  12.,  33.,   2.,   0.,   0.,   1.,   1.],\n",
       "       [  2.,  23.,  30.,   1.,   9.,  30.,   1.,   2.,   0.,   0.,   4.]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = df[['region', 'tenure','age', 'marital', 'address', 'income', 'ed', 'employ','retire', 'gender', 'reside']] .values  #.astype(float)\n",
    "X[0:5]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "What are our labels?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 4, 3, 1, 3])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y = df['custcat'].values\n",
    "y[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Normalize Data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Data Standardization gives the data zero mean and unit variance, it is good practice, especially for algorithms such as KNN which is based on the distance of data points:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.02696767, -1.055125  ,  0.18450456,  1.0100505 , -0.25303431,\n",
       "        -0.12650641,  1.0877526 , -0.5941226 , -0.22207644, -1.03459817,\n",
       "        -0.23065004],\n",
       "       [ 1.19883553, -1.14880563, -0.69181243,  1.0100505 , -0.4514148 ,\n",
       "         0.54644972,  1.9062271 , -0.5941226 , -0.22207644, -1.03459817,\n",
       "         2.55666158],\n",
       "       [ 1.19883553,  1.52109247,  0.82182601,  1.0100505 ,  1.23481934,\n",
       "         0.35951747, -1.36767088,  1.78752803, -0.22207644,  0.96655883,\n",
       "        -0.23065004],\n",
       "       [-0.02696767, -0.11831864, -0.69181243, -0.9900495 ,  0.04453642,\n",
       "        -0.41625141, -0.54919639, -1.09029981, -0.22207644,  0.96655883,\n",
       "        -0.92747794],\n",
       "       [-0.02696767, -0.58672182, -0.93080797,  1.0100505 , -0.25303431,\n",
       "        -0.44429125, -1.36767088, -0.89182893, -0.22207644, -1.03459817,\n",
       "         1.16300577]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = preprocessing.StandardScaler().fit(X).transform(X.astype(float))\n",
    "X[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Train Test Split\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set: (800, 11) (800,)\n",
      "Test set: (200, 11) (200,)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=4)\n",
    "print ('Train set:', X_train.shape,  y_train.shape)\n",
    "print ('Test set:', X_test.shape,  y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### **Classification**\n",
    "\n",
    "**K nearest neighbor (KNN)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Training\n",
    "\n",
    "Let's start the algorithm with k=4 for now:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KNeighborsClassifier(n_neighbors=4)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "k = 4\n",
    "#Train Model and Predict  \n",
    "neigh = KNeighborsClassifier(n_neighbors = k).fit(X_train,y_train)\n",
    "neigh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Predicting\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 1, 3, 2, 4])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "yhat = neigh.predict(X_test)\n",
    "yhat[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Accuracy Evaluation\n",
    "\n",
    "In multilabel classification, **accuracy classification score** is a function that computes subset accuracy. This function is equal to the jaccard_score function. Essentially, it calculates how closely the actual labels and predicted labels are matched in the test set.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train set Accuracy:  0.5475\n",
      "Test set Accuracy:  0.32\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "print(\"Train set Accuracy: \", metrics.accuracy_score(y_train, neigh.predict(X_train)))\n",
    "print(\"Test set Accuracy: \", metrics.accuracy_score(y_test, yhat))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "**Let's build the model again, but this time with k=6**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train set accuracy: 0.51625\n",
      "test set accuracy: 0.31\n"
     ]
    }
   ],
   "source": [
    "k=6\n",
    "neigh2=KNeighborsClassifier(k)\n",
    "neigh2.fit(X_train,y_train)\n",
    "\n",
    "y2 = neigh2.predict(X_test)\n",
    "from sklearn import metrics\n",
    "print(\"train set accuracy:\", metrics.accuracy_score(y_train,neigh2.predict(X_train)))\n",
    "print(\"test set accuracy:\", metrics.accuracy_score(y_test,y2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### With the other K\n",
    "\n",
    "K in KNN, is the number of nearest neighbors to examine. It is supposed to be specified by the user. So, how can we choose right value for K?\n",
    "The general solution is to reserve a part of our data for testing the accuracy of the model. Then choose k =1, use the training part for modeling and calculate the accuracy of prediction using all samples in our test set. Repeat this process by increasing the k and see which k is the best for our model.\n",
    "\n",
    "We can calculate the accuracy of KNN for different values of k.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.3  , 0.29 , 0.315, 0.32 , 0.315, 0.31 , 0.335, 0.325, 0.34 ])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Ks = 10\n",
    "mean_acc = np.zeros((Ks-1))\n",
    "std_acc = np.zeros((Ks-1))\n",
    "\n",
    "for n in range(1,Ks):\n",
    "    \n",
    "    #Train Model and Predict  \n",
    "    neigh = KNeighborsClassifier(n_neighbors = n).fit(X_train,y_train)\n",
    "    yhat=neigh.predict(X_test)\n",
    "    mean_acc[n-1] = metrics.accuracy_score(y_test, yhat)\n",
    "\n",
    "    \n",
    "    std_acc[n-1]=np.std(yhat==y_test)/np.sqrt(yhat.shape[0])\n",
    "\n",
    "mean_acc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "#### Plot the model accuracy for a different number of neighbors.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAABCaUlEQVR4nO3deZxbdb3/8dfnnCSzdrpOS+nOUimlLcu0VRYBWQStAoKyXRVBELRccbtyvffHvZd7vYr3KigXqGURXGhBdr1c9qIgIF2olJatdKGlpTPTfWYyk+Scz++Pk6SZPdMmTTLzeT4ew0xOzkm+mTJ557uLqmKMMcYUG6fQBTDGGGO6YgFljDGmKFlAGWOMKUoWUMYYY4qSBZQxxpiiZAFljDGmKOU1oETkDBF5W0RWi8i1PZw3U0Q8ETkv49g6EVkhIstFZEk+y2mMMab4hPL1wCLiArcApwEbgcUi8piqrurivBuAJ7t4mJNVtTHb5xwxYoROnDhx7wttjDFmv1u6dGmjqtZ2PJ63gAJmAatVdQ2AiCwEzgJWdTjvauBBYOa+PuHEiRNZssQqW8YYU0pEZH1Xx/PZxDcG2JBxe2PyWGahxgDnAPO6uF6Bp0RkqYhckbdSGmOMKUr5rEFJF8c6rqt0E/B9VfVEOp1+nKpuEpGRwNMi8paq/rnTkwThdQXA+PHj973UxhhjikI+a1AbgXEZt8cCmzqcUwcsFJF1wHnArSJyNoCqbkp+rwceJmgy7ERV56tqnarW1dZ2asI0xhhTovJZg1oMHCoik4APgAuAizJPUNVJqZ9F5G7gj6r6iIhUAY6q7k7+fDpwfR7LaowxncTjcTZu3Ehra2uhi9IvlJeXM3bsWMLhcFbn5y2gVDUhInMJRue5wF2qulJErkze31W/U8oo4OFks18IuFdVn8hXWY0xpisbN25k0KBBTJw4kS66IUwfqCpbt25l48aNTJo0qfcLyG8NClV9HHi8w7Eug0lVL8n4eQ0wI59lM8aY3rS2tlo45YiIMHz4cBoaGrK+xlaSMMaYHlg45U5ff5cWUMYYY4qSBZQxxhS5hx9+GBHhrbfeKnRR2lFV8rkruwWUMcYUuQULFnD88cezcOHCvD6P53m9nqOq+OoT9+LEvBgxL5a38lhAGWNMEWtqauIvf/kLd955Z7uA8jyP7373u0ybNo3p06dz8803A7B48WKOPfZYZsyYwaxZs9i9ezd33303c+fOTV87Z84cnn/+eQCqq6u57rrrmD17Ni+//DLXX389M2fO5IgjjuCKK65I15LeefcdTjnlFGbMmMExRx/Du6vf5Stf/gp/eOwP6ce9+OKLeeyxx3L22vM6is8YY/qLa564huUfLs/pYx55wJHcdMZNPZ7zyCOPcMYZZzB58mSGDRvGsmXLOProo5k/fz5r167ltddeIxQKsW3bNmKxGOeffz733XcfM2fOZNeuXVRUVPT4+M3NzRxxxBFcf30w1fTwww/nuuuuQ1X54pe+yCOPPsKZc87k4osv5rv/8F3OOfscWtta8X2fSy+7lF/8/Bec97nz2LlzJy+99BL33HNPrn49VoMyxphitmDBAi644AIALrjgAhYsWADAM888w5VXXkkoFNQzhg0bxttvv83o0aOZOTNYe7umpiZ9f3dc1+Xcc88Fgua7Z597llmzZjFt2jSee+453lj1Bs27m9m8aTPnnHMOSDDhtrKyko+f+HHee+896uvrWbBgAeeee26vz9cXVoMyxgwYqkpLvIWEn6AsVEbEjeBIdp/Te6vp5MPWrVuDkHjjDUQEz/MQEX7yk5+gqp2GbXd1DCAUCuH7fvp25soY5eXliCPEvTjN0WbmfmMuL77yIhPGTeDfr/93Yq0xtNMyqntcdPFF/O53v2PhwoXcddddOXjVe1gNyhgzILQmWtmwawMbdm2gvrme9TvWs3rratZtX0djcyPNsWbiXrzQxWzngQce4Etf+hLr169n3bp1bNiwgUmTJvHiiy9y+umnM2/ePBKJBADbtm3jsMMOY9OmTSxevBiA3bt3k0gkmDhxIsuXL8f3fTZs2MCrr76aHugAEEvE8NQj1hYMeBhZO5Km5iYeeughIKiJjRkzhkcffRSAtrY2WlpaAPjil77ITTfdBMDUqVNz+vqtBmWM6dfiXpzGlkZ2tu6kLFRGTVlNu/sTfoKdbTvZGt0KgOu4VIYrqQpXpQcIFGqy7oIFC7j22vabkZ977rnce++93HzzzbzzzjtMnz6dcDjM5Zdfzty5c7nvvvu4+uqriUajVFRU8Mwzz3DccccxadIkpk2bxuFTD+fIo44k4SfwNBi15zhBXWXIkCFcetmlHH3k0UyYMIG6urr08951z13MvWou1//r9YTDYe5deC8HHXQQo0aNYsqUKZx99tk5f/2SzzHs+1tdXZ3ahoXGGADP99ge3c7W6NZ06GQjVbNI+Amim6NM/shkRARHHBxx0j+XAlVFUTzfS4cRkNPyNzc3U3dUHcuWLWPw4MG9nv/mm28yZcqUdsdEZKmq1nU812pQxph+RVXZ1baLhuYGFKU6Ut2nGpAjDmWhMsooo03acBwHVcVTD89PvskL6cByxEGQolkSaX+EUsqzzzzLFZdfwXe+/Z2swqmvLKCMMf1Gc6yZLc1biHtxKsOVuI6bk8cVEQRptw2rqpLQRHob1kLWsvZnKGU65dRTWL1mNWWhsrw8vgWUMabktSZaaWhuoCXeQnmonPKy8rw/Z8fQ2t+1rEKF0v5kAWWMKVlxL8626Da2R7cTCUUYVDaoYGXZH7WsgRBKmSygjDElx/M9drbtpKG5AddxGVQ2qGj6gDLlopY10EIpkwWUMaZkqCq723ZT31yPrz5VkaqSeqPuay3LV3/AhVKmgfVqjTElqyXewrod69jctJmyUBnVZdX7/Q17y67WnH5BRiA5Tno+kqcecT+Op167GlZ3Xnn5Fa762lU9lv2FP7/A7JmzqSyr5KEHH9qr179jxw7m3dblpugAXHLJJTzwwAN79dhdsYAyxhS1tkQbG3du5P0d7yMiDCoblLPRecUoswaVGUp/ev5PfPXSr3Z5zVNPPsXpnzy9x8cdN34cd9x5BxdceMFel23Hjh38ct4v9/r6vrKAMsYUpYSfYEvTFtbtWEfMj1FTXkPEjRS6WEVp0XOL+MQpn+jxnIkTJzJt+rR0LS3l0Uce5YzTz0BV2bx5M1OnTOXDDz9k1cpVHPfR45h5zEyOOeoY3n33Xf75B//MmvfWMPOYmVz7D9eiqlzz99dw+OGH8+lPf5r6+vqcvi7rgzLGFBVffXa07qCxuRHHcfo80XagaWxsJBwO7/VE2bPOPouHH3qY2269jaeefIrr/uU6DjjgAH78ox8z9+/ncuFFFxKLxfA8j//4z/9g5cqVLF4arPX3yMOP8M4777BixQq2bNnC4YcfzqWXXpqz12YBZQaM1Cio1PeElyDuxxGEsBsm5IRwHRdXXFzHHXAd0oWWGgDR0NKArz6VkUr7NwCO/9jxtMXaaGpqYvu27cw8JthK44f/+UNO/+TpPPP0M5x62qn79Bw3/vxGjp5xNLNmz+L8C84H4KMf/Sg//tGP+WDjB5x1zlkceuihna574YUX+ML5X8B1XQ488EA+8Ymea3F9ZQFl+oWO4eP5Xno76pgXI+En8HXPdgOp0VSpN8D0SCklGGGlwQKaYSdM2A0TcSIWYnnUEm+hvqmeNq+NinAFIcfemlJefPlFIOiD+s2vf8Mdd93R7v4nn3iSb17zTQAuv+xyli9fzujRo3nsj9nvbLvpg004jkP9lnp838dxHC648AJmzprJ/z3+f8z51Bzm/XIekw6a1OnafNZu7f8CU/RSoeOrj+d7JPxEt+GTGsKbCh/XcQk5ISJupM9/SKk5K22JNqIaxVOP1OLKItIuxCJuhIgbIeSELMT6oC3RRmNLI02xJspCZQWdaFuKVJUVK1Yw48gZANx+5+19foxEIsHlX72ce35zD7/77e+46cab+PZ3vs2aNWs46KCDmHv1XNauXcuKFSuYPmM6Tbub0teecMIJ3D7/di77ymXU19ezaNEiLrroopy9PgsoU1Cp0EnVeroKH1VN12pyFT7ZEBFCEurx03wqxFoTrbTEW4KamJLe4E2QTiEWcSPp4BqoIZbwE2yLbmNbdBthJ1wywTSqJv9LKPXFsqXLOPLII7P6/3/J4iV84bwvsH37dv73j//L9f92PctfX84NP7qB4447juNPOJ4ZR87g2I8ey5mfOpM/PPoH7r33XsLhMKNGjeKf/vmfGDZsGB879mMcNeMoPvnJT/KjG37EoucWMW3aNCZPnsyJJ56Y09dn222YvOkpfOJenIQmgl0+uwifVAC54pZ8B3nm6gHpiZfJ15v6+3Mdt8sQywyw/hBivvrsbA1WgHDEoSJcUdT/vtve38bkwyYXuhjd+tEPf8TBhxzMF87/QsHKoKp9WizWttswBRX34mxt2cqutl09hk+FFPebU67sS00s8/fnSNc1sVSIFXOYqypNsSbqm+vx1LMBEDnyj//0j4UuQl5ZQJmc6bg+WnVZdaGLVDKyCTFffXz1ewyxkIQIuSEiToRIKELYCRc8xKLxKFuatqQHQFQ4Ffv1+U3psoAy+yxzeLDneyW3PlqpSNU+ewsxz/doSbTQFG/C930UTQ/qSIVY2G0/sCMfIRbzYjS2NLKrbRdlrg2AMH1nAWX2Sean48pwJW64/y5BUwoccXBchzDhbs9JhVhzvJndsd3pEEtJ1eYyQyzshtv1h/UUYgk/wfbodrZFtxFyQtSU1eT8dZqBwQLK7JWYF6OhucGGB5egvQ2xjv2JrridQizuxWlsaUQQWwHC7DMLKNMnHT8dWzD1T9mEWGqEZrsQAxsAYXLGAspkxVc/vQ+PfTo2EAyNdxlYTbobdm4gmojm7PEqQhWMGzxunx/nlZdf4Z677+G2X97W7TnzfzmfebfNw3VdqququXXerUw5fEq353dlx44dLFywkCuvurLL+y+55BLmzJnDeeed16fH7Y59zDE9UlWaY82s276OD5s+pCJcQWWk0sLJDEjRRJTqSHXOvvoSdvu63cYFF17AsuXLWLx0Md/+3rf53ne/16fXDv1suw0ROUNE3haR1SJybQ/nzRQRT0TO6+u1Jn9aE61s3LWRDbs2pLfVtqYbY4pPNttt1NTsGazS0tyS/pA5ILfbEBEXuAU4DdgILBaRx1R1VRfn3QA82ddrTX7EvTjbotvYHt1OJBSxUVjGFLG+bLdx26238fObfk48FueJp58ABu52G7OA1aq6BkBEFgJnAR1D5mrgQWDmXlxrcqjjPjyDygZZU54xBZbL7Tau+vpVXPX1q1i4YCE//s8fc+ev7gSKd7uNfLbXjAE2ZNzemDyWJiJjgHOAjpvc93qtyZ3URNu129fS2NxIZaSSyrD1MxlTDF58+UUWL13MvF/OY85n5rB46WIWL12c7nN68okn0z9fftnlzDxmJp+d89keH/ML53+Bxx7dsx1Hx+02IOizevDhBymvKGfOp+aw6LlFXT5WPt8n8hlQXZW648q0NwHfV01txtOna4MTRa4QkSUisqShoaHvpRzgovEo63es54NdHxB2w1SXVVs/kzEloqvtNhYvXdzlXlDvvvtu+ufH//dxDjn0EKD9dhuHTTmMm268CaDddhtzPjOHFStWMGjQoE7bbfz+/t/jeR6bN29m0aKuQ2xv5bOJbyOQOX5yLLCpwzl1wMJkAo8APiUiiSyvBUBV5wPzIVjNPCclHwAyl6EpD5VTU279TMb0piJUQVOsqfcT+/B4+6Iv223cduttPPfsc4TDYYYOGcqddwXNewNyuw0RCQHvAKcAHwCLgYtUdWU3598N/FFVH+jrtSn7ut1GY3MjLYkWqiPVlIfK0+uU9See77E9up2t0a2EnBAVYVu405ju2HYbvSvJ7TZUNSEicwlG57nAXaq6UkSuTN7fsd+p12vzVdaUuB8n5sXYFt2G5wetjmEnTFWkiqpIVXqrg1Lsm8mcaAvYRFtj+gHbbmMfqOrjwOMdjnUZTKp6SW/X7g+uuO0+DaTWI9vZuhMIOgQrQhVURarStSzXKe7Z9M2xZrY0byHuxYMFXYu8vMYYA7bUUa9cJ1jBOfWbUlXifrAgpq/BaJeIG6E6Uk1FuCJYNNMJF0XtpC3RRn1zPS3xFspD5ZSXFdd21caUAlUtir/n/qCvXUoWUH0kIunVm1M832NX2y62t24HgoU2K0IVVEeqKQuVEXEj+3VkXMJPsLVla3qirS3oaszecSMu27dtZ+iwoRZS+0hV2bp1K+Xl2X9QtoDKAddx2+0SqqrEvBhbmrcEn74QykJlVIWr9tSy3O5Xid5bvvrsbA12tLWJtsbsu+oR1Wxv3E5jQ2Ohi1KUUvuIZTuYrLy8nLFjx2b9+BZQeSASBFIZe/qyEn6CnW072RbdBgShVhmupCpcRVmojLAb3utalqrSFGuivrkeX33b7sCYHHFDLoMP6H0JoYHKV59YIsZBww7Ky+NbQO0nISfU7lOGrz7RRJTdbbvTW3KXh8r7PMQ9taNta6KVykhlvxsWb4wZuOzdrEAccSgPlbcbfJHwE2yLbksPvghJKD3EvePgi5gXY2vLVna27bSJtsaYfskCqkiICGE33K5vyvM9mmJN7GjdgSDpIe5hN8yO1h2EnJCtNG6M6bcsoIpYV4Mv4n6ctlibTbQ1xvR7FlAlJDXEfYDtsm2MGaBsqJcxxpiiZAFljDGmKFlAGWOMKUoWUMYYY4qSBZQxxpiiZAFljDGmKFlAGWOMKUoWUMYYY4qSBZQxxpg+83wlGvPy+hy2koQxxpisJDyftoRPW8LD8xXFp6Y8f/UcCyhjjDHdiiV8YgmPmOfj+X3bsn1fWUAZY4xppy3hJYPJx9f9G0qZLKCMMQOK7yuOYzsBZFJV2pKBFPM8CphJ7VhAGWMGBM9XmtsStCU8BAi5DiFHCLkOYdfBHWCh5ftKzAv6k+IJnyLJpHYsoIwx/ZrvK82xBG1xL/0mrEDc84l7EPwHHBFCrhB2HFxXiLhOv9tzzfOVWMKjLeGT8IozlDJZQBlj+iVVpSXmEY0lsnoj9lWJJZQYfvqY6wghxyHsSrrGVWqhlfCCprs2LwilUmIBZYzpV1SD+Tkt8cQ+96V4vuL5Hm2J4LYAbjKowhlNhMUmngql5HDwUmUBZYzpN6Ixj5ZYIm8jz5SgRpLwoDXZNChCspYV9GNFXKcggzBSw8HbCjzyLpcsoIwxJa81HgRTIWoLqqn+rD3NZ460r2GF3dw3DaomBznEi2vkXS5ZQBljSlZbwqO5rTDB1BNflbaER1vGsT39WU4wGGMvmgaLdTh4vlhAGWNKTizh0xJLtKu1FLs9/VnJpkGC/qxwL0PdfT8ZSl7xDgfPFwsoY0zJSHg+zbEEsUTpBFN3MvuzyOjPCpoGHRyBtoRfUiGcaxZQxpiilznJtj9TTQ52YOCGUiYLKGNM0epqkq0ZOCygjDFFp6+TbE3/lNcZZiJyhoi8LSKrReTaLu4/S0ReF5HlIrJERI7PuG+diKxI3ZfPchpjioOq0tKWYGtzGy0WTgNe3mpQIuICtwCnARuBxSLymKquyjjtWeAxVVURmQ7cDxyWcf/JqtqYrzKa/s/3FU812FxNg+G/nq/4GnypBh3TriM4IrgSzFdxHHBFcEtwaZtSle9Jtqb05LOJbxawWlXXAIjIQuAsIB1QqtqUcX4V2Acmkx1VxVfSYaPp4AFPFd8PjmX1P1TycbojgJMMsOCLdHC5ThBqtn3D3ivkJFtT3PIZUGOADRm3NwKzO54kIucAPwJGAp/OuEuBp0REgV+q6vw8ltUUET8ZOkHQkK7tZNZ+9ueHbCU5h6WXuHOSQeUI6drYnmDDamMdFOskW1M88hlQXf0ldvo/UVUfBh4WkY8D/w6cmrzrOFXdJCIjgadF5C1V/XOnJxG5ArgCYPz48TkrvMmPYLJi51pPOoT8LGs9RchXxfd6Lr0kwyv1lWpaTDUzDoTaWNzzaW4rrUm2pjDyGVAbgXEZt8cCm7o7WVX/LCIHi8gIVW1U1U3J4/Ui8jBBk2GngErWrOYD1NXVlep7W78TjXkkfD9d60n1/wx0mmyC7Kk2JpDsB0sGlmTWzlKhRsnVxvrTJFuzf+QzoBYDh4rIJOAD4ALgoswTROQQ4L3kIImjgQiwVUSqAEdVdyd/Ph24Po9lNTli/Qn7Tkn2sXlKoofzUrUxt13/2J4BHk6R1MYGyiRbk3t5CyhVTYjIXOBJwAXuUtWVInJl8v55wLnAl0QkDkSB85NhNYqg2S9VxntV9Yl8ldXsO+tP2P/StbG9HOCRCrd8bXXu+0pLLEGrTbI1eymvE3VV9XHg8Q7H5mX8fANwQxfXrQFm5LNsJjdKcdHOgaQvAzw6Drd3nOSxPg63t0m2JldsJQmzV6w/oX/xVXsfbt/lAA/Sw+0dEWIJj+bYvu9kawxYQJk+sv6EgSubAR7G5FJelzoy/YfvK02tcbY3t1k4GWNobGngobfu48nVT+btOXqtQYnIHOBxVbW2nAHI+hOMMSlrd7zH8+ueYdH6p1lRvxxF+eTBn+SqWVfl5fmyaeK7APi5iDwI/EpV38xLSUxRUVVa49afYMxA5vkeKxqWp0Np/c61AEwZMZWrjvkmJ044hWMOnJK35+81oFT170SkBrgQ+FVy6aFfAQtUdXfeSmYKpjUeDBm3ibXGDDytiVZe/eAlFq1/mj+9/yzbolsJSYhjDpzNhVO/zEkTTuGA6gMBUPy8ThjPapCEqu5K1qAqgGuAc4DvicgvVPXmvJXO7Fc2ydb0d6rKh82bGVExgrAbKXRxisaO1u288P4iFq1/mpc2vkBrIkpVuJrjx53ISRNO5bhxJ1FTVrPfy5VNH9RngEuBg4HfALOSyw9VAm8CFlAlLpYIhownbC6T6ceWbn6VW5b8jGUfLibiRpg8bApTa6cztXYah4+YxqQhB+M6bqGLud9s3PU+i9Y/w/Prn2H5h0vw1GNk1QF85tDPcfKEU6k7cDYRt6ygZcymBvV54MaOC7WqaouIXJqfYpn9Ie4Fk2xtLpPpz1bU/41bl9zIyx+8QG3lSK6e+V12tu1gZcMK/vDuQ9y36jcAVIarmDJ8KlNrpyWDazpjBo0ruTUPu6OqrGpcwfPJUHp329sAHDJ0Ml+Z8TVOmngah484AkeKZ3B3NgH1L8Dm1A0RqQBGqeo6VX02byUrAH+ANG3ZXCYzELy99U1uW3oTz69/hqHlw/jO7B/w+cMvpjxUnj7H8z3W7VzDqoYVrGx4nZWNK1i46jfEvBgAg8uGcPiIae1Ca2TVqEK9pD6LezEWb/4rz69/hj+tf4YtzR/iiMNRo+r4zuwfcNLEUxlXM6HQxeyWaC8d4cnt1o9V1VjydgT4i6rO3A/l65O6ujpdsmTvd4d/c8t6muMtVIYrCLsOriNEXKcoFtzMBd9XmmMJ2mxtNNOPrdm+mnnLfs5Tax5nUKSGL0+/nAunfomqSHVW18e9GKu3vxMEVsMKVjas4L3t7+Bp8IGutnJUEFgjgtA6vHYaQ8qH5vMl9cnu2G7+suF5nl//DC++/zxN8SbK3XI+NvYETpp4Kh8f/wmGlg/LyXMpPjXlDgcNO2ifHkdElqpqXcfj2dSgQqlwAlDVWDKk+iXVoOkrc205R4Sw6xByhJDrEHZLa+M5m8tkBoKNu95n3rJf8PjqRykPVXD5Ud/gi9O+2ufO/bAbYcqII5gy4gjOS46gjiaivL11VTq0VjUETWUpYweND/qykjWtKcOnZh2IubClaTPPv/8sz697hsWbXyHhxxlaPoxTJ53JyRNPY/aY49rVHEtFNgHVICKfVdXHAETkLKAxv8UqLr4qbQmPtuTt1ArRQWgFgRVyi6fdNkVVicY8WuI2l8n0Xx82beL2127h0bcfwHVc/m7apVwy/QqGVQzP2XNUhCo4ctQxHDnqmPSx3bHdvJmsYa1sfJ3X61/jyTX/C4AgTBpySLumwcnDDqMslJtBB6rK6u3v8Py6p1m0/mlWNb4BwPiaiVx8xCWcNOFUpo88quQHfWTTxHcw8DvgQIL35g3Al1R1df6L1ze5aOJrirXs1cgVAUIZtaxCNw1GY8GQcZvLZPqrxpYG7lx+Gw+8uQBFOe+wC7jsqK9TWzmyYGXaFm1MNwsGfVqvsy26FYCQE+bQYZOZOmJ6OrQOGnoIISe7JVETfoLlW5ayaN3TPL/+GT7YvQGAaSOP5OQJp3HShFOZNOTg/dq6k+8mvl4DKuMBqpPnF+3k3EIGVFccEUKuEHac/dY0aHOZTH+3o3U797x+OwtW/pq4F+Ozk8/l8qPmcuCgMYUuWiepeVcrG15vNxCjKRa8jZa75Rw2Ymq6afCIEdMZN3hCeiRdNN7CSxtfYNH6p3nh/UXsbNtB2Ikwe8yxnDThVE6ccEpBA7kY+qAQkU8DU4Hy1BusqtoOt73wVYkllBh7+rNcR5LNgk4QXjlqGmxLeLTEPJvLZPqt3bHd/HbFXfx2xV20xJs585DP8rWjr2bC4EmFLlq3RITR1QcyuvpATp10BgC++mzYuZ43GveE1kNvLuTeN+4GoDoyiKkjphF2Iyze9DJtXhs1ZYM5YdzJnDThVI4de8J+7d8qpGwm6s4DKoGTgTuA84BX81yufsvzFc/30kO8BXBdh3B6AIbTpx1O455Pc5ttGGj6r2i8hQUrf83dr89nV9tOTp10Blce/U0OGTa50EXbK444TBgyiQlDJvHpQ84Cgua7NdtXJwdhvM6qxhU0xZr43GEXcPKEUzlq9EzCTrjAJd//sqlBHauq00XkdVX9NxH5KfBQvgs2UCjB5n8JD4gnQ0sg7DqEHQc32UTYsT8r4fm0xDyby2T6rbZEG79/817u+tttbItu5YRxJ/P1umuYMuKIQhct50JOiMnDD2Py8MM457AvFLo4RSObgGpNfm8RkQOBrUDx1qn7AdVg+aGumwaFuKcWTKbfinsxHnn7AW5ffgv1zR8y+8Bj+fpp32LGqKMLXTSzn2UTUH8QkSHAfwHLCD70357PQpnO9jQNFrokpU9V2dG2nYbmenbHdjGsYji1lSOpjgwqdNEGtISf4H9XP8Ivl97MpqaNzBh1ND886b+ZeeDHCl00UyA9BpSIOMCzqroDeFBE/giUq+rO/VE4Y/qqJd5MQ3M99S0fUt9ST33zhzS01KePNTTX09BST9yPdbq2MlxFbeVIRlaOorZqVPBz1ShqK0elv9dW1hZ8Ac3+xlefp9c8zm1Lf866nWuYMmIqPzj+3zhu7IklNSHe5F6PAaWqfrLP6WPJ222Qnq9qzH4T92I0RhtpaN5CfcuWzt9b6mlo3kJTvKnTtVXhamorR1JbOZKjDqhrFzyDymrYFt2aDrL65i00tGzhb1uWUd+8pcsgG1o+LHi8qlHpMBtZOZKRlQdQWzWS2spRDKsYXlSLbhYjVeX59c9w69IbeXfb2xw89FB+duqtnDzxdAsmA2TXxPeUiJwLPKTZTpoyJku++uxo3Zas7QThkPoe1HqCn1OTHTOFnHA6eA4eOpmPjjmekanQqBzJyKoDqK0cuddDclWVnW07Opcro6xvN65ia7QR7bCIVEhCDK8ckS7DnlrYnnKNrDqA6nD1gHszVlVe3vgC/7PkZ6xqXMH4mon86OQbOf2gT5f8ygcmt7IJqG8DVUBCRFoJRkarqu7/3atMSWmK7U7XbOrTgdO+ptLQ0kDCj7e7TpBkv9AoRlUdwBG107t8gx9SPjSvtRQRYUj5UIaUD2Xy8MO6PS/hJ9ja0kB9S327IAu+17N+51oWb3qF3bFdna4tD1V0CK1kk2LG662tHJWzJXIKbcnmv3LL4p/x2pYljK4ew79+/AbmHHp21qspmIElmy3frefYZC2aiHL/qt/ym9fvpDHa0On+6sigdA3nmNGzM96I9/T5DK+sLak5HyEnxKjq0YyqHt3jedFElIbmVA0s2TeW8fMb9X+joWULbV7nVvTBZUPaBVb7vrEg4IaVDy/aGsjrW17jlqU38tcP/kJt5Sh+cNz1nPORz9uutqZH2UzU/XhXxztuYGgGtpjXxkNv3ccdr91KY7SBj405gb8bc2mHmsFIKsNVhS5qwVSEKhg/eCLjB0/s9hxVZVfbzqAG1kWzZ33zFt7d9g5bow342n5ytiMOwytqu2hKTNXKgmODIjX7rVnxra2ruHXJjfz5/ee63ZPJmO5kU6/+XsbP5cAsYCnwibyUyJSUuB/nD+88xPxlN/Nh82aOOWAWPznlZo4eXXTbhZUEEWFw+RAGlw/hkGEf6fY8z/fYGm1s15SYGWQbd73Pax8uYWfbjk7Xlrvl6cEcXQZZ8vu+hMia7au5benPeXptsCfT3LrvcNERXx7QH1BM32XTxPeZzNsiMg74Sd5KZEqC53s88d4fmLfs52zY9T5H1M7gX0+8gdkHHjvgOv0LwXXcYEBI1Sim1nZ/XmuilcaMpsT6lnoa0t+3sKrxDf60/llavdZO1w6K1HRqSuzYvDi8YkS7/qMNu9bzy6W/4PH3HqM8VMEVR83l76Zd1uc9mYyBLBeL7WAj0P/WGjFZ8dXn2bVPctvSm1izYzWTh03hF6ffzgnjT7ZgKkLloXLG1oxnbM34bs9RVXbHdnc5wCP189odq2lsaUjvKpsSNCuOoLZyJIMiNSzZ/FfCTpgvTruMS2ZckbOdW83AlE0f1M2QHkPrAEcCf8tjmUwRUlVe2PA8ty65kbe2rmTSkIP5ySk3c+qkM2y+T4kTEWrKaqgpq+HgoYd2e57ne2xv3dZlkNU3f0hjtJHPT7mo4Hsymf4jmxpU5gZLCWCBqv4lT+UxRUZVeXXTS9yy5EZer3+NsYPG8x8n/TdnHvzZoh0xZvLDdVxGVNYyorK2Xy7YaopPNgH1ANCqGtTtRcQVkUpVbclv0UyhvfbhEm5Z8jOWbP4ro6oO4P+d8EM+O/nckhoCbowpXdkE1LPAqUBqDZkK4Cng2HwVyhTWyobXuXXJjfxl458ZXjGC73/sOj532AX9ZrKoMWbfuI4QcR1CboigYS0/sgmoclVNL3Cmqk0iUpm3EpmCeXfb29y29CaeW/cUg8uGcM2s73P+1C9SEaoodNGMMQUkQMh1KAs5REJuelNVX31iedxhIZuAahaRo1V1GYCIHANEs3lwETkD+DngAneo6o873H8W8O+ATxDD16jqi9lca3Jn/Y613Lbs5zz53h+pCldx1THXcPERl9j2E8YMYCIQcV0iIYeI23nT1P0hm4C6Bvi9iGxK3h4NnN/bRSLiArcApxEMTV8sIo+p6qqM054FHlNVFZHpwP3AYVlea/bRB7s3Mn/Zzfzh3YeIuGV85cgr+fK0rzK4fEihi2aMKYBU010kFARToWUzUXexiBwGfISgpveWqsZ7uQyCFSdWq+oaABFZCJwFpEMms+mQYEFazfbaXNvVtovWROfJiv3RluYPueO1W3n47ftxRLho6iVceuTXGFYxotBFMyavXEeoCIfwVUn4PnHPZ6Dv0RByHcpch0jIIeQWPpQyZTMP6hvA71T1jeTtoSJyoare2sulY4ANGbc3ArO7ePxzgB8BI4FP9+XaXLp18a38v0X/j4OGHMLU2ulMrZ3OEbXTOXjY5H4zam1btJFf/W0+96/6LZ7vcc5h5/PVo77OqKoDCl00Y/LKdYTKSIjycOepEZ6vxD0fzx8YoSWQriEVqukuW9k08V2uqrekbqjqdhG5HOgtoLp61Z3+2VX1YeDh5KK0/04wYjCrawFE5ArgCoDx47ufLd+bkyeezKUzvsbftrzGs+ue5OG37wegzC3jI8MPZ2rtNKbWTufwEdOYOOSgkpqcuqttJ/e8fjv3vnEPbV4rcw49h68ddTVjasYVumjG5FVPwZR5Tsc5fZmhFfd8En5ph5YjEgRSMpRKZdWXbALKERFJbVaY7B/KZo38jUDmO+BYYFM356KqfxaRg0VkRF+uVdX5wHyAurq6vf5faPbY2dSED6Ap1kLYibBx9/usbHidlQ0rWNnwOo+8/QALVv4aCHZonTJiarqmNbV2OgdWjym6f/Sm2G7ufeNufr3iTppiuznj4DlcefQ3mTjkoEIXzZi8yiaYeru+u9BKeD6JZG2rmEOrmJvuspVNQD0J3C8i8whqMVcCT2Rx3WLgUBGZBHwAXABclHmCiBwCvJccJHE0QfBtBXb0dm0+iQjjaiYwrmYCZxwcrJXr+R5rd7zHysY9oXXvG/ektwQfWj6Mw0cc0S60RlT2sIpnHkUTUe5b+Rvu/tt8drRt5+QJp3HVMdf0uOmeMf3BvgZTb4/tOi5kPHYxhZYA4ZBDxHUpCxV30122sgmo7xM0oV1F8Dt4Cri9t4tUNSEicwkCzgXuUtWVInJl8v55wLnAl0QkTjB0/fxkTa3La/v86nLIdVwOGTaZQ4ZN5qzJ5wHBHkjvbnublQ0rWNWwgpWNr/Py8lvT+/SMqjqAw0dMywitadSUDc5bGWNeGw++uZA7lt/K1mgjx439OFfVfYsjaqfn7TmNKQb5DKbenreQoVWqTXfZEu3jb05EjgcuVNVv5KdIe6+urk6XLFnS+4ndeHPLeppiLUTcvV8xIRpv4a2tq3ij4fVkE+HrbNi1Pn3/uJoJewJrxDSmjJhKRXjf5j3H/TiPvfMgty/7n2BPptGzmVv3bY46oG6fHteYYleoYOqrXIaW6whlyUEO4QI33QUTdWMcNGzfug1EZKmqdnrDymq7DRE5EriQYP7TWuChfSpNP1YRruSoA+rahcOutp2sanwjHVivfbiYJ977AxBsV5A5cnBq7TQmDzssq62wO+7JNG3kkfzbiTcwy/ZkMv1cqQRTSlc1rXRY9RJamU13kZCTXsVhIOg2oERkMkHfz4UE/UL3EdS4Tt5PZes3asoG89Exx/HRMceljzW2NASB1Rj0Z/35/ed49J0HAAg7EQ4d9pGM4e7TmDTkkHSnbcc9mT4y/HB+8ck7OGHcSRZMpl8rtWDqSch1CLl0G1oKhJPLCw3Uv+tum/hExAdeAC5T1dXJY2tUtWiHgBVDE9/eUlU2NX2QrmWtaljBqsY3aI4Hc5krQpVMGTGVw0ZMZenmV3l76yoOGnIIVx1zDadM+mRJDXs3pq/6UzD1J4Vs4juXoAa1SESeABbS9fwkkwMiwphBYxkzaCynH/QpIPjHX79zbTq0Vjas4ME3FzCyahQ/POmnnHHwZ2xPJtOvWTANbN0GVMYE2irgbOBbwCgRuQ14WFWf2j9FHLgccZg05GAmDTmYOYeeAwT9To4M3Cq/GRgsmAwEW7j3SFWbVfV3qjqHYMLscuDafBfMdM11XAunHBGCFZtN8XAdYVB5mGFVZRZOJrtRfCmqug34ZfLLmKIUBI/gOIIjwVwRVyR5DFwRnOT9Kb6veKp4vqIKvmr6K/OYyQ+rMZmu9CmgjCmkvQmebDmO4CD09v7o+Z1Dy/MVxYJsb1gwmZ5YQJmC6y14UreLYekW1xHcHASZr1rU67jlmwWTyYYFlMmbroIn/VVkwZNrexdkiq/06yBL7cdUEbFgMr2zgDI5VxZyqYy4JbuC8v40UILMgsnsDQsokzMWTPlTqkHmSNCUZ8Fk9oYFlNlnFkzFoy9Btiew9gRZ5ujFfQkyCyaTCxZQZq9ZMJWuYPHSnvv+OgZXNkFmwWRyyQLK9JkF08AgIrhC1kHmqxJyxCaSm5yxgDJZs2AyXUkHmS3VaXLMAsr0yoLJGFMIFlCmWxZMxphCsoAynVgwGWOKgQWUSbNgMsYUEwsoY8FkjClKFlADmAWTMaaYWUBlCIcc3ET/HyprwWSMKQUWUBmqIiFcp4yIGyHuKQnPJ+ErCd/H84t4Jc4sWTAZY0qJBVQXRIRISIiE9ryRq2rJhpYFkzGmFFlAZakUQ8uCyRhTyiyg9kGxhpYFkzGmP7CAyrFChpYFkzGmP7GA2g96Ci3P94Pw2ofQsmAyxvRHFlAFkgotcKhIHutraFkwGWP6MwuoIpJtaIUcx4LJGNPvWUAVua5CyxhjBgL7CG6MMaYoWUAZY4wpSnkNKBE5Q0TeFpHVInJtF/dfLCKvJ79eEpEZGfetE5EVIrJcRJbks5zGGGOKT976oETEBW4BTgM2AotF5DFVXZVx2lrgRFXdLiJnAvOB2Rn3n6yqjfkqozHGmOKVzxrULGC1qq5R1RiwEDgr8wRVfUlVtydvvgKMzWN5jDHGlJB8BtQYYEPG7Y3JY925DPi/jNsKPCUiS0XkijyUzxhjTBHL5zDzrjZW6nLWqYicTBBQx2ccPk5VN4nISOBpEXlLVf/cxbVXAFcAjB8/ft9LbYwxpijkswa1ERiXcXsssKnjSSIyHbgDOEtVt6aOq+qm5Pd64GGCJsNOVHW+qtapal1tbW0Oi2+MMaaQ8hlQi4FDRWSSiESAC4DHMk8QkfHAQ8AXVfWdjONVIjIo9TNwOvBGHstqjDGmyOStiU9VEyIyF3gScIG7VHWliFyZvH8ecB0wHLhVRAASqloHjAIeTh4LAfeq6hP5Kqsxxpjik9eljlT1ceDxDsfmZfz8VeCrXVy3BpjR8bgxxpiBw1aSMMYYU5QsoIwxxhQlCyhjjDFFyQLKGGPMXlHdu13As2X7QRljjMma53u0eW34vg8CQ8qH5O25LKCMMcZ0S1WJeTHiXhxFCbthhpYPpTJcSVmoDEfy1xBnAWWMMaadVC3J8z1EhKpwFcMrhlMRriDshvdbOSygjDEDiq8+vvqEHHv7S+mqljSkbAhVkaq815J6Yv9CxpgBwfM9ookoDg4hJ0RToilYvlrAFZeQEyLkhEiuYNPvFUstqScWUMaYfi3hJ2iNt+I6LqOqRjGobBCOOPjqE/fixP04rfFWWhIttMRbUBQUHMdJh1ahahC51KmW5BRHLaknFlDGmH4p7sWJxqNE3AgHVB9AdVl1uzdhRxzKQmWUUUZ1pBoI3sQTfoK4H6ct0UZLvIVoIornewCICCEnRNgJ4zpuQV5XX3RXSyoPlxNxI4UuXq8soIwx/Upboo2YF6PMLWPc4HFUhiuzbrYTEcJumLAbpjJcydCKoUBQC4t7cWJeLAiteJSWeEtwDYLruITdcMH7tUqxltQTCyhjTL/Qmmgl7sWpCFcwrnocFaGKnPUnpZr6KsIVDC4fDAS1k7gfD2pqiSgtsZaC9Gt5vkfMi5HwEzjiUBmuLKlaUk8soIwxJS0ajxL34gwqG8To6tFUhCv2y/O6jovruJSHyhlUNgiqyLpfK+yE9zq0uqolDS4bXLK1pJ5YQBljSo6qEk1ESfgJBpcNZljNMMpCZYUuVp/6tXzfR9Gs+rU61pIqQhX9ppbUEwsoY0zJUFWi8Sg+PkPKhzCkfEjRv0HvS7+W53sgEJJQv60l9cQCyhhT9Hz1icajqCrDKoYxuHxw0czV2Vu99Wu1Jlopc8v6fS2pJxZQxpiilTm5dnjlcGrKago+Ui6fOvVrDXD991/aGFOyEn6CaDxKyAkxqmoU1ZHqkph3ZHLLAsoMOKqaXo/NUw9BcMTBddwB07ZfrFJNW2EnzOjq0Z0m15qBxQLK9Bvp0PG99M+qmh4ppaoIwdDe1KTKVNt+zIsRS8Tw1AuGBAOpb0gwOivzyxV3wKzZtj/EvBitiVbK3XLG1IyhKlxlv19jAWWKW6q24+me0PHVT96ZPCn5PpaaGBlxI0TcSDB01w2nAyWbWlLH2lUq8BJ+Ij3MN+bFiGksWD4m9eTpb9I5zKxpqluZk2vHDx6f08m1pvRZQJmC6K62k3qjT4VPao5IyAlREapIB1Bm2KTCJxdvbCKCKy4uLmF6HiWWGZip15HwE+3CLLUeXKoWl/naHMdJDyceaLWyaDyYw1QVrtqvk2tNabGAMnmR3hZaffa0mO15kw5JKD1aKRU6ruN2qukUc/9Du/L1UklKBVhmTTDuxduFWdSLpoNakHa/r1R4pX4/uQrk/UlVaU20kvATDCobxLCKYZSHygtdLFPELKBMTmV2cg+rGJaeHZ/Zd1OKb677ynWyq5WparumxdT3uB/vVDMDOvWviUi7psViCHlVpSXegq/B5NqhFUMH7Lwe0zcWUCYnUp3cZW4ZBw46kOpI9YALoVwQEUKS/LPspVbWLsQyQq1dX5kXI6EJgE41s46DPnL9waE/Tq41+5cFlNkn1sldOI44OG7vtaOOA00y+8syw8xTLz3wI7Nm1tWgj55qZZ7vEY1HEZEBMbnW5I/9X2P6LNWXUIgVpE3fZQ786E3HQR+p0Eqt0J3wE8QSQah1NXxfUUJOiNqqWmrKamwEo9knFlAma6kVpD3fY3DZYIbWDC2KFaRN7qRqSb3VeLobji8iVIYrC97vZfoHCyjTq8y+hKEVQxlSPsT6Ega4vgzHN2ZvWUCZbnm+R2uiFcD6Eowx+52925hOMhfqrK2qZVBkkPUlGGP2OwsokxbzYrQl2myhTmNMUbCAMrQl2oh5McrcMsbWjKUyXGlDxY0xBZfXj8cicoaIvC0iq0Xk2i7uv1hEXk9+vSQiM7K91uy7aDzKrrZdhJwQ4wePZ8KQCVRFbBVpY0xxyFsNSkRc4BbgNGAjsFhEHlPVVRmnrQVOVNXtInImMB+YneW1edGaaEXR9BI9/U3H9dDGVIyx9dCMMUUpn018s4DVqroGQEQWAmcB6ZBR1Zcyzn8FGJvttfkwvHI4leFKWuItRONRovFosLq2Bmuphd1wya44rapE41F8gvXQhpQPsfXQjDFFLZ8BNQbYkHF7IzC7h/MvA/5vL6/NidQ+QoPLBwOk9wGK+8GWCS3xFpoTzcHJGmyXkNoKolgHE/jq0xJrAbD10IwxJSWfAdVVNUO7OIaInEwQUMfvxbVXAFcAjB8/vu+l7IHruLiOSxllVEeqg0KoEvfjxL04MS8W1LaSqyskyxNslFfgJkLP94gmojg4tuyMMaYk5TOgNgLjMm6PBTZ1PElEpgN3AGeq6ta+XAugqvMJ+q6oq6vrMsRySUTSNa0qqhhaMRQgvQVCzIsFzYOJPU2EqpquaYWcUF6bCDO3uxhVNYpBZYOKtnZnjDE9yWdALQYOFZFJwAfABcBFmSeIyHjgIeCLqvpOX64tNuldX8MV6SbC1KZ0qSbCaCJoJlQUNKO2ldyWfF+k5jBF3AgHDjqQqkiVBZMxpqTlLaBUNSEic4EnCXa2uUtVV4rIlcn75wHXAcOBW5O1ioSq1nV3bb7Kmi+OOJSFyjo1Eab6tdoSbekmwtTOsyLBFuDZNhFmbncxbvA42+7CGNNviGreW8X2m7q6Ol2yZEmhi7FXUk2EcS+ermnFvBiwZ2HOVC0N9gRTdaSa4ZXDbbsLY0zJEpGlqlrX8bitJFEkMpsIa6gB2jcRtsZb9zQRqlJTVmPbXRhj+jULqCLWXROhr76NyDPG9HsWUCUm1dxnjDH9nQ3zMsYYU5QsoIwxxhQlCyhjjDFFyQLKGGNMUbKAMsYYU5QsoIwxxhQlCyhjjDFFyQLKGGNMUepXa/GJSAOwfh8eYgTQmKPi5JOVM3dKoYxg5cy1UihnKZQRclPOCapa2/FgvwqofSUiS7pasLDYWDlzpxTKCFbOXCuFcpZCGSG/5bQmPmOMMUXJAsoYY0xRsoBqb36hC5AlK2fulEIZwcqZa6VQzlIoI+SxnNYHZYwxpihZDcoYY0xRsoACROQuEakXkTcKXZaeiMg4EVkkIm+KyEoR+Wahy9SRiJSLyKsi8rdkGf+t0GXqiYi4IvKaiPyx0GXpjoisE5EVIrJcRJYUujxdEZEhIvKAiLyV/P/zY4UuU0ci8pHk7zD1tUtEril0uboiIt9K/v28ISILRKS80GXqSES+mSzfynz9Hq2JDxCRjwNNwK9V9YhCl6c7IjIaGK2qy0RkELAUOFtVVxW4aGkiIkCVqjaJSBh4Efimqr5S4KJ1SUS+DdQBNao6p9Dl6YqIrAPqVLVo58SIyD3AC6p6h4hEgEpV3VHgYnVLRFzgA2C2qu7L3MmcE5ExBH83h6tqVETuBx5X1bsLW7I9ROQIYCEwC4gBTwBXqeq7uXweq0EBqvpnYFuhy9EbVd2sqsuSP+8G3gTGFLZU7WmgKXkznPwqyk9BIjIW+DRwR6HLUspEpAb4OHAngKrGijmckk4B3iu2cMoQAipEJARUApsKXJ6OpgCvqGqLqiaAPwHn5PpJLKBKlIhMBI4C/lrgonSSbDZbDtQDT6tq0ZUx6SbgHwC/wOXojQJPichSEbmi0IXpwkFAA/CrZHPpHSJSVehC9eICYEGhC9EVVf0A+G/gfWAzsFNVnypsqTp5A/i4iAwXkUrgU8C4XD+JBVQJEpFq4EHgGlXdVejydKSqnqoeCYwFZiWbA4qKiMwB6lV1aaHLkoXjVPVo4EzgG8km6WISAo4GblPVo4Bm4NrCFql7ySbIzwK/L3RZuiIiQ4GzgEnAgUCViPxdYUvVnqq+CdwAPE3QvPc3IJHr57GAKjHJfp0Hgd+p6kOFLk9Pks08zwNnFLYkXToO+Gyyf2ch8AkR+W1hi9Q1Vd2U/F4PPEzQ7l9MNgIbM2rKDxAEVrE6E1imqlsKXZBunAqsVdUGVY0DDwHHFrhMnajqnap6tKp+nKCLJKf9T2ABVVKSAxDuBN5U1Z8VujxdEZFaERmS/LmC4I/trYIWqguq+o+qOlZVJxI09zynqkX1KRVARKqSA2JINpudTtC8UjRU9UNgg4h8JHnoFKBoBu504UKKtHkv6X3goyJSmfybP4Wgv7moiMjI5PfxwOfIw+80lOsHLEUisgA4CRghIhuBf1HVOwtbqi4dB3wRWJHs4wH4gao+XrgidTIauCc5SsoB7lfVoh3CXQJGAQ8H71OEgHtV9YnCFqlLVwO/SzafrQG+UuDydCnZX3Ia8LVCl6U7qvpXEXkAWEbQbPYaxbmqxIMiMhyIA99Q1e25fgIbZm6MMaYoWROfMcaYomQBZYwxpihZQBljjClKFlDGGGOKkgWUMcaYomQBZUqOiKiI/DTj9ndF5F9z9Nh3i8h5uXisXp7n88lVvxd1OD4x+fquzjj2PyJySS+Pd6WIfKmXcy4Rkf/p5r6mro7nkoiMTq0aLyInZa4gLyL/ISJPikiZiCwUkUPzXR5T/CygTClqAz4nIiMKXZBMyblf2boM+LqqntzFffXAN5NzirKiqvNU9dd9eP6cSS5omo1vA7d3cf0/EczxO1tV24DbCNZINAOcBZQpRQmCiYvf6nhHxxpQqmaQ/MT+JxG5X0TeEZEfi8jFEuxdtUJEDs54mFNF5IXkeXOS17si8l8islhEXheRr2U87iIRuRdY0UV5Lkw+/hsickPy2HXA8cA8EfmvLl5fA/As8OUuHu9gEXkiuXDsCyJyWPL4v4rId5M/z0yW8eVkmTNXnjgwef27IvKTDo/9UxFZJiLPikht8tiRIvJK8vEeTq4Th4g8LyL/KSJ/IgjTzydf499E5M9dvCaAcwnWbct8zu8QLDT6GVWNJg+/kPw3sIUEBjgLKFOqbgEuFpHBfbhmBvBNYBrBihyTVXUWwXYbV2ecNxE4kWArjnkSbBZ3GcGq0jOBmcDlIjIpef4s4J9U9fDMJxORAwkW1PwEcCQwU0TOVtXrgSXAxar6vW7K+mPgO13UyuYDV6vqMcB3gVu7uPZXwJWq+jHA63DfkcD5yd/B+SKSWoG6imB9uqMJtk74l+TxXwPfV9XpBAH8LxmPNURVT1TVnwLXAZ9U1RkEC7G2k/xdbU/WkFKOA64EzszYogVV9YHVBP9eZgCzgDIlKbmK+6+Bv+/DZYuTe2q1Ae8BqS0MVhCEUsr9quonN19bAxxGsAbel5JLTP0VGA6k+kleVdW1XTzfTOD55KKfCeB3BPsmZfP61gKvAheljkmwiv2xwO+T5fglwdJSZJwzBBikqi8lD93b4aGfVdWdqtpKsF7ehORxH7gv+fNvgeOT4T9EVf+UPH5Ph/Lfl/HzX4C7ReRyoKumztEENcNMqwEh+N12VE+wkrcZwKwKbUrZTQTrlf0q41iC5Aev5EKbmf04mZ/e/YzbPu3/Fjqu/6UEb6RXq+qTmXeIyEkE20t0RXopf2/+k2Bl8FSTmQPsSG5l0p3enjPzd+DR/XtANmugpV+3ql4pIrMJap3LReRIVd2acW4U6Lht+RbgYuBZEdmqqpkDRsqT15gBzGpQpmSp6jbgfoLmt5R1wDHJn88i2NG3rz4vIk6yX+og4G3gSeAqCbY7QUQmS++b8v0VOFFERiSb6i4kaD7Liqq+RVDLmZO8vQtYKyKfT5ZBRGRGh2u2A7tF5KPJQxdk+XQOkOq7uwh4UVV3AttF5ITk8S92V34ROVhV/6qq1wGNdN687h3a11JT5X2HYCXs34rIkRl3TQZWZll2009ZDcqUup8CczNu3w48KiKvEgw06K5205O3Cd6IRxH05bSKyB0Eb7DLkjWzBuDsnh5EVTeLyD8CiwhqNo+r6qN9LMsPCVazTrkYuE1E/pkgfBcSbBaX6TLgdhFpJtiPa2cWz9MMTBWRpcnzz08e/zJBP1wlPa9S/l/JoeFC8HtvVyZVbRaR90TkEFVd3eG+xSLyFeAxETkZaAKiqro5i3KbfsxWMzemnxGR6tSgAxG5Fhitqt8scLEQkXOAY1T1n3s571vAriLd8sbsR1aDMqb/+XSy5hYC1gOXFLY4AVV9WIL9g3qzA/hNnotjSoDVoIwxxhQlGyRhjDGmKFlAGWOMKUoWUMYYY4qSBZQxxpiiZAFljDGmKFlAGWOMKUr/H49jqQ+l901nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(range(1,Ks),mean_acc,'g')\n",
    "plt.fill_between(range(1,Ks),mean_acc - 1 * std_acc,mean_acc + 1 * std_acc, alpha=0.10)\n",
    "plt.fill_between(range(1,Ks),mean_acc - 3 * std_acc,mean_acc + 3 * std_acc, alpha=0.10,color=\"green\")\n",
    "plt.legend(('Accuracy ', '+/- 1xstd','+/- 3xstd'))\n",
    "plt.ylabel('Accuracy ')\n",
    "plt.xlabel('Number of Neighbors (K)')\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "button": false,
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best accuracy was with 0.34 with k= 9\n"
     ]
    }
   ],
   "source": [
    "print( \"The best accuracy was with\", mean_acc.max(), \"with k=\", mean_acc.argmax()+1) "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}