classification_metrics.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1 align='center'>  Performance Metrics for Classification Problems </h1>\n",
    "\n",
    "In this note book, I am going to illsutrate how we can evaluate the ML models deployed for classification task using different kind of metrics.\n",
    "\n",
    "For illsutareting purpose I have collected the data from kaggle. I will be doing all the analysis over them only. I have attach the link for each dataset, you can also download the same.\n",
    "\n",
    "\n",
    "For the implemetations of these metrics, I am using following tools and frameworks:\n",
    "- Python - as a primary language\n",
    "- Pandas - as an analytical engine for processing the data\n",
    "- numpy - for computation using numpy arrays\n",
    "- matplotlib - for plotting the figures\n",
    "- sklearn - for implememting the metrics\n",
    "- seaborn - for graph plotting \n",
    "\n",
    "Note that I will be implementing all the metrices from scratch.\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contents\n",
    "\n",
    "1. [Imports](#Imports)\n",
    "1. [Load the Data](#Load-the-Data)\n",
    "1. [Basic Exploratory Data Analysis (EDA)](#Basic-Exploratory-Data-Analysis-(EDA))\n",
    "1. [Features Selection](#Feature-Selection)\n",
    "1. [Features Extraction](#Feature-Extraction)\n",
    "1. [Modeling](#modeling)\n",
    "    1. [Train/Test SPlits](#traintest-split)\n",
    "    1. [Model Selection](#model-selection)\n",
    "    1. [Model Training](#model-training)\n",
    "1. [Metrics For Model Evaluation](#metrics-for-model-evaluation)\n",
    "    1. [Confusion matrix](#Confusion-matrix)\n",
    "    1. [Accuracy](#Accuracy)\n",
    "    1. [Precision](#Precision)\n",
    "    1. [Recall](#Recall)\n",
    "    1. [F scores](#F-scores)\n",
    "    1. [Macro-averaged F scores](#Macro-averaged-F-scores)\n",
    "    1. [Weighted F scores](#Weighted-F-scores)\n",
    "    1. [Micro-averaged F scores](#Micro-averaged-F-scores)\n",
    "    1. [Precision–recall curves](#Precision–recall-curves)\n",
    "    1. [Average precision](#Average-precision)\n",
    "    1. [Receiver Operating Characteristic (ROC) curve](#Receiver-Operating-Characteristic-(ROC)-curve)\n",
    "    "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's write some generic code which will be used throughout this notebook.\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn\n",
    "import sklearn"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Loader\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "#function for loading a csv file from specified path\n",
    "\n",
    "def data_loader(path):\n",
    "    \n",
    "    #use pandas to load the data from path\n",
    "    df = pd.read_csv(path)\n",
    "    \n",
    "    return df"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classfication Problem\n",
    "\n",
    "### Bank Customer Churn Prediction\n",
    "\n",
    "* Problem Statement : Given the attributes of members and the associated credit score build a predictive model to forcast if a member will churn or not in future.\n",
    "\n",
    "* About Data : The Data is from kaggle and same can be downloaded from <a href=\"https://www.kaggle.com/code/kmalit/bank-customer-churn-prediction\">here</a>. The label column is namesd as `Exited` and its values need to be forcasted using our models.\n",
    "\n",
    "* Algorithms to be used : Since this is a classification problem I will be using Logistic regression, Support Vector Machine(SVM), Decision Tree, Random Forest Classifier and XgBoost Classifier and will evaluate performance for each algorithms agains many performance metrics.\n",
    "\n",
    "* Evaluations metrics : Since this is a binary class classification problem. I will be using accuracy score, precision, recall, f1-score, log loss and AUC-ROC curve."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Always look at your data`\n",
    "## Load the data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I have downloaded the data from above link and have stored the same in my local file system. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>RowNumber</th>\n",
       "      <th>CustomerId</th>\n",
       "      <th>Surname</th>\n",
       "      <th>CreditScore</th>\n",
       "      <th>Geography</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Age</th>\n",
       "      <th>Tenure</th>\n",
       "      <th>Balance</th>\n",
       "      <th>NumOfProducts</th>\n",
       "      <th>HasCrCard</th>\n",
       "      <th>IsActiveMember</th>\n",
       "      <th>EstimatedSalary</th>\n",
       "      <th>Exited</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>15634602</td>\n",
       "      <td>Hargrave</td>\n",
       "      <td>619</td>\n",
       "      <td>France</td>\n",
       "      <td>Female</td>\n",
       "      <td>42</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>101348.88</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   RowNumber  CustomerId   Surname  CreditScore Geography  Gender  Age  \\\n",
       "0          1    15634602  Hargrave          619    France  Female   42   \n",
       "\n",
       "   Tenure  Balance  NumOfProducts  HasCrCard  IsActiveMember  EstimatedSalary  \\\n",
       "0       2      0.0              1          1               1        101348.88   \n",
       "\n",
       "   Exited  \n",
       "0       1  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#churn df contains all the data\n",
    "path= \"/Users/ajitkumarsingh/Desktop/Data-Science-Interview-Questions/performance-metrics/data/Churn_Modelling.csv\"\n",
    "churn_df = data_loader(path)\n",
    "\n",
    "# show first row\n",
    "churn_df.head(1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic Exploratory Data Analysis (EDA)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "columns name : RowNumber, CustomerId, Surname, CreditScore, Geography, Gender, Age, Tenure, Balance, NumOfProducts, HasCrCard, IsActiveMember, EstimatedSalary, Exited\n",
      "\n",
      "total number of columns : 14\n",
      "\n",
      "Number of rows in data : 10000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#columns name in churn df\n",
    "print(f\"columns name : {', '.join(churn_df.columns.tolist())}\\n\")\n",
    "\n",
    "#total number of columns\n",
    "print(f\"total number of columns : {len(churn_df.columns)}\\n\")\n",
    "\n",
    "#total count of the data\n",
    "print(f\"Number of rows in data : {len(churn_df)}\\n\")\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have 14 attributes and 10000 rows in the churn data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RowNumber          0\n",
       "CustomerId         0\n",
       "Surname            0\n",
       "CreditScore        0\n",
       "Geography          0\n",
       "Gender             0\n",
       "Age                0\n",
       "Tenure             0\n",
       "Balance            0\n",
       "NumOfProducts      0\n",
       "HasCrCard          0\n",
       "IsActiveMember     0\n",
       "EstimatedSalary    0\n",
       "Exited             0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#number of nulls per attributes\n",
    "churn_df.isnull().sum()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "No Null values are present in the data for any attribute. So that is good thing like we don't have to deal with null values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RowNumber          10000\n",
       "CustomerId         10000\n",
       "Surname             2932\n",
       "CreditScore          460\n",
       "Geography              3\n",
       "Gender                 2\n",
       "Age                   70\n",
       "Tenure                11\n",
       "Balance             6382\n",
       "NumOfProducts          4\n",
       "HasCrCard              2\n",
       "IsActiveMember         2\n",
       "EstimatedSalary     9999\n",
       "Exited                 2\n",
       "dtype: int64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# number of unique values\n",
    "churn_df.nunique()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the `RowNumber` and `CustomerId` seems like primary keys for this data. Because all the values in these columns seem unique or have no duplicates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RowNumber            int64\n",
       "CustomerId           int64\n",
       "Surname             object\n",
       "CreditScore          int64\n",
       "Geography           object\n",
       "Gender              object\n",
       "Age                  int64\n",
       "Tenure               int64\n",
       "Balance            float64\n",
       "NumOfProducts        int64\n",
       "HasCrCard            int64\n",
       "IsActiveMember       int64\n",
       "EstimatedSalary    float64\n",
       "Exited               int64\n",
       "dtype: object"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# data types\n",
    "churn_df.dtypes"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature Selection\n",
    "\n",
    "For time being I am manually selecting the features. If we look at the data and problem statement that we have, the following attributes seems redundant `RowNumber`, `CustomerId`, `Surname`. We can remove these columns from our analysis because they are not providing any value add and will unnecessary complicate the analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CreditScore</th>\n",
       "      <th>Geography</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Age</th>\n",
       "      <th>Tenure</th>\n",
       "      <th>Balance</th>\n",
       "      <th>NumOfProducts</th>\n",
       "      <th>HasCrCard</th>\n",
       "      <th>IsActiveMember</th>\n",
       "      <th>EstimatedSalary</th>\n",
       "      <th>Exited</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>619</td>\n",
       "      <td>France</td>\n",
       "      <td>Female</td>\n",
       "      <td>42</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>101348.88</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   CreditScore Geography  Gender  Age  Tenure  Balance  NumOfProducts  \\\n",
       "0          619    France  Female   42       2      0.0              1   \n",
       "\n",
       "   HasCrCard  IsActiveMember  EstimatedSalary  Exited  \n",
       "0          1               1        101348.88       1  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "redundant_features = [\"RowNumber\", \"CustomerId\", \"Surname\"]\n",
    "\n",
    "#drop redundant features\n",
    "churn_df_final = churn_df.drop(redundant_features, axis=1)\n",
    "\n",
    "#show 1 row\n",
    "churn_df_final.head(1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Features Extraction\n",
    "\n",
    "Most of machine learning algorithms expect input as some numbers and they can not deal with attributes of the non numeric type. \n",
    "\n",
    "We can transform those attrbutes using some techniques such as `Label Encoding` or `One Hot Encoding`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "non-numeric cols : Index(['Geography', 'Gender'], dtype='object')\n"
     ]
    }
   ],
   "source": [
    "#print the columns that are non-numeric\n",
    "non_numeric_df = churn_df_final.select_dtypes(exclude=['int','float'])\n",
    "print(f\"non-numeric cols : {non_numeric_df.columns}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have two categorical columns namely `Geography` and `Gender`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Geography    3\n",
       "Gender       2\n",
       "dtype: int64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#number of unique values\n",
    "non_numeric_df.nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Geography  Gender\n",
       "France     Male      2753\n",
       "           Female    2261\n",
       "Spain      Male      1388\n",
       "Germany    Male      1316\n",
       "           Female    1193\n",
       "Spain      Female    1089\n",
       "dtype: int64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#value frequncy of categorical columns\n",
    "non_numeric_df.value_counts()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have three ditsinct values namely `France`, `Spain` and `Germany` in `Geography` column and, two distinct values `Male` and `Female` in `Gender` column. We need to encode these variables with some numeric values then only we can feed the models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "#encode Gender column\n",
    "#replace Male with 1 and Female with 0\n",
    "\n",
    "gender_dict = {'Male':1,'Female':0}\n",
    "churn_df_final['Gender'] = churn_df_final['Gender'].replace(gender_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "#encode Geography column using one hot encoding\n",
    "#this will create some new columns (number of new columns will be equal to number of unique values in the data)\n",
    "final_df_encoded = pd.get_dummies(churn_df_final,columns=['Geography'])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have encoded both the categorical columns `Gender` and `Geography` using `lable encoding` and `one-hot encoding` repectively. The `final_df_encoded` now have all the attributes as numeric data type. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CreditScore            int64\n",
       "Gender                 int64\n",
       "Age                    int64\n",
       "Tenure                 int64\n",
       "Balance              float64\n",
       "NumOfProducts          int64\n",
       "HasCrCard              int64\n",
       "IsActiveMember         int64\n",
       "EstimatedSalary      float64\n",
       "Exited                 int64\n",
       "Geography_France       uint8\n",
       "Geography_Germany      uint8\n",
       "Geography_Spain        uint8\n",
       "dtype: object"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#data types of final df\n",
    "final_df_encoded.dtypes"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modeling\n",
    "\n",
    "### Train/Test Split\n",
    "\n",
    "For evaluating a model, we need some data to test it on, once training part is done. We usually split the data into two parts i.e train and test. \n",
    "On train data we update the models parameters and on test data we see how the trained model is performing. \n",
    "\n",
    "For spliting the data, I am using `train_test_split` function of module `sklearn.model_selection`. Also we need to segregate the label from the data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total count : 10000\n",
      "\n",
      "Train Data count : 8000\n",
      "\n",
      "Test Data count : 2000\n",
      "\n",
      "Train Labels count : 8000\n",
      "\n",
      "Test Labels count : 2000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "#split the data into test and train in 4:1\n",
    "\n",
    "train_data, test_data, train_labels, test_labels = train_test_split(final_df_encoded.drop(['Exited'], axis=1), final_df_encoded['Exited'], test_size=0.2, random_state=42)\n",
    "\n",
    "#count after split\n",
    "print(f\"Total count : {len(final_df_encoded)}\\n\")\n",
    "print(f\"Train Data count : {len(train_data)}\\n\")\n",
    "print(f\"Test Data count : {len(test_data)}\\n\")\n",
    "print(f\"Train Labels count : {len(train_labels)}\\n\")\n",
    "print(f\"Test Labels count : {len(test_labels)}\\n\")\n",
    "\n",
    "# number of rows and labels should match\n",
    "assert len(train_data)==len(train_labels) \n",
    "assert len(test_data)==len(test_labels)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have stored train and test data in `train_data`, `test_data` and train and test labels in `train_labels`, `test_labels`. I will be using them during training and testing time accordingly."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Selection\n",
    "\n",
    "We have numerous models out there to solve same type of poblems. But the interesting part is like we don't know which model will be best fitting our dataset and perform well on test dataset. We need to make choice here and to do that we need to evaluate these models one by one using some performance metrics.\n",
    "\n",
    "I will be training following models and evaluating their performance on test data\n",
    "\n",
    "- Logistic Regressor\n",
    "- Support Vector Machine(SVM) Classifier\n",
    "- RandomForest Classifier\n",
    "- Neighrest Neighbour Classifier\n",
    "\n",
    "Here, I will be using `sklearn` module of `scikit-learn` library to implement the above mentioned models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "#import above models\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.svm import SVC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "#create a list that contains one instances of each model type\n",
    "\n",
    "model_instances = {\n",
    "          'logistic_regressor':LogisticRegression(),\n",
    "          'random_forest': RandomForestClassifier(), \n",
    "          'knn':KNeighborsClassifier(), \n",
    "          'svm':SVC()\n",
    "          }\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Training\n",
    "\n",
    "In the above step we have created instances of each model and stored in `model_instances` dict type variable. Now we need to train these instances by feeding the train distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "#train the models on training set and save the trained model for evaluation\n",
    "\n",
    "for model_name, model in model_instances.items():\n",
    "\n",
    "    #fit the model with train_data and train_labels\n",
    "    model.fit(train_data,train_labels)\n",
    "    \n",
    "    #save the trained model\n",
    "    model_instances[model_name] = model\n",
    "    "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Metrics For Model Evaluation\n",
    "\n",
    "Now we have trained our models and they are ready to make predictions over test dataset. To make sure the model is predicting meaningfull values not random output we need some metrics to evaluate the output. This is where performance metrics come into picture. \n",
    "\n",
    "Let's implement some of the most widely used evaluation metrics for classification problems from scratch."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Performance Metrics Implementation"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Confusion Matrix\n",
    "\n",
    "This a `2D` matrix of size `n*n` where n is number of distinct tables we have. It stores information about the actual lable and corresponding predicted values. \n",
    "\n",
    "`ex1 = `\n",
    "<table>\n",
    "<tr>\n",
    "<th></th>\n",
    "<th></th>\n",
    "<th colspan=3 style=\"text-align:center\">predicted</th>\n",
    "</tr>\n",
    "<tr>\n",
    "<th></th>\n",
    "<th></th>\n",
    "<th>pos</th>\n",
    "<th>neg</th>\n",
    "<th>neutral</th>\n",
    "</tr>\n",
    "<tr>\n",
    "<th rowspan=3>gold</th>\n",
    "<th>pos</th>\n",
    "<td>15</td>\n",
    "<td>10</td>\n",
    "<td>100</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<th>neg</th>\n",
    "<td>10</td>\n",
    "<td>15</td>\n",
    "<td>10</td>\n",
    "</tr>\n",
    "<tr>\n",
    "<th>neutral</th>\n",
    "<td>10</td>\n",
    "<td>100</td>\n",
    "<td>1000</td>\n",
    "</tr>\n",
    "</table>\n",
    "\n",
    "For classifiers that predict real values (scores, probabilities), it is important to remember that __a threshold was imposed to create these categorical predictions__. \n",
    "\n",
    "The position of this threshold can have a large impact on the overall assessment that uses the confusion matrix as an input. The default is to choose the class with the highest probability. This is so deeply ingrained that it is often not even mentioned. However, it might be inappropriate:\n",
    "\n",
    "  1. We might care about the full distribution.\n",
    "  1. Where the important class is very small relative to the others, any significant amount of positive probability for it might be important.\n",
    "\n",
    "Metrics like [average precision](#Average-precision) explore this threshold as part of their evaluation procedure. \n",
    "\n",
    "We can use this matrix to derive various scores metrics like `accuracy score`, `precision score`, `recall score` and `f-1 score`.\n",
    "\n",
    "Note that in the below implementation of confusion matrix\n",
    "\n",
    " - `CM(i, j)` => Number of instances where actual label was `i` but the predicted label was `j`\n",
    "\n",
    " - `CM(i, i)` => Number of instances where actual label was `i` and the predicted label was also `i`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "function to get n*n confusion matrics\n",
    "\n",
    "Input :\n",
    "\n",
    "    :param y_pred - a list of predicated lable from the model\n",
    "    :param y_true - a list of actual labels\n",
    "    :param y_uniq - unique number of labels in the data\n",
    "    :return: Confusion matrix as a 2D numpy array\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "def confusion_matrix(y_true, y_pred, classes):\n",
    "\n",
    "    #metrics for storing TP, TN, FP and FN\n",
    "    \n",
    "    #note that here I am considering the labels start with 0 to n\n",
    "    n_label = len(classes)\n",
    "\n",
    "    conf_mat = np.zeros((n_label, n_label))\n",
    "\n",
    "    for i in range(len(y_true)):\n",
    "\n",
    "        y_true_i = y_true[i]\n",
    "        y_pred_i = y_pred[i]\n",
    "        \n",
    "        true_indx = classes.index(y_true_i)\n",
    "        pred_index =  classes.index(y_pred_i)\n",
    "        \n",
    "        conf_mat[true_indx][pred_index] += 1\n",
    "\n",
    "    return conf_mat\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Accuracy Score\n",
    "\n",
    "This score signifies the portion of true predictions among total cases examined. A higher accuracy score suggests that the model is making more correct predictions, while a lower accuracy score indicates that the model may be making more mistakes. It is mathematically expressed as:\n",
    "\n",
    "$$\n",
    "\n",
    "accuracy\\ score = \\frac{{TP + TN}}{{TP + FP + FN + TN}}\n",
    "\n",
    "$$\n",
    "Where:\n",
    "\n",
    "- `TP (True Positives)`  = Number of scenarios where the model's predicted labels match with the positive labels\n",
    "\n",
    "- `FP (False Positives)` = Number of scenarios where the actual label is negative, but the model predicts it as positive(Type I Error)\n",
    "\n",
    "- `FN (False Negatives)` = Number of scenarios where the actual label is positive, but the model predicts it as negative(Type I Error)\n",
    "\n",
    "- `TN (True Negative)` = Number of instances where both the actual label and the model's prediction are negative\n",
    "\n",
    "**Accuracy bounds**\n",
    "\n",
    "[0, 1], with 0 the worst and 1 the best.\n",
    "\n",
    "**Value encoded by accuracy**\n",
    "\n",
    "Accuracy seems to directly encode a core value we have for classifiers – how often they are correct. In addition, the accuracy of a classifier on a test set will be negatively correlated with the [negative log (logistic, cross-entropy) loss](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.log_loss.html#sklearn.metrics.log_loss), which is a common loss for classifiers. In this sense, these classifiers are optimizing for accuracy.\n",
    "\n",
    "\n",
    "In machine learning problems, the definition of `positive` and `negative` can vary depending on the specific problem and the context. In the `Bank Customer Churn Prediction` problem, the variable of interest is whether a customer churns (leaves) or not. in the context of this problem, `positive` refers to customers who churned, and `negative` refers to customers who did not churn. \n",
    "\n",
    "It's important to keep this definition in mind when interpreting performance metrics such as `accuracy`, `precision`, `recall`, and `F1-score`, as they are calculated based on the definitions of `positive` and `negative` classes in the specific problem context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\"\"\"\n",
    "    function to calculate accuracy score of each class using confusion matrix\n",
    "\n",
    "    :param a 2D confusion matrix\n",
    "\n",
    "    :return a list containing accuracy score for each class\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "def accuracy_score(confusion_matrix):\n",
    "\n",
    "    n_classes = confusion_matrix.shape[0]\n",
    "    accuracy_score_list = []\n",
    "\n",
    "    for i in range(n_classes):\n",
    "\n",
    "        tp = confusion_matrix[i][i]\n",
    "        fp = np.sum(confusion_matrix[i][:]) - tp\n",
    "        fn = np.sum(confusion_matrix[:][i]) - tp \n",
    "        tn = np.sum(confusion_matrix) - tp - fp- fn\n",
    "\n",
    "        accuracy_score_list.append(tp/(tp+fn+fp+tn) if tp+fn+fp+tn!=0 else 0)\n",
    "\n",
    "    return accuracy_score_list"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cautions / Limitations \n",
    "\n",
    "- It's important to note that accuracy score provides an `overall evaluation` of the model's performance across all classes, but it may be deceptive in cases where the label distribution is skewed or imbalanced. In such cases, the accuracy score may be high due to the majority class dominating the predictions, while the `minority class` may be misclassified.\n",
    "\n",
    "- To address this issue, it's recommended to use other performance metrics along with accuracy score, such as `precision`, `recall`, `F1-score`, or area under the Receiver Operating Characteristic (ROC) curve. These metrics provide insights on the model's performance at a class-level, allowing for a more comprehensive evaluation of the model's predictive accuracy, especially in `imbalanced` datasets.\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Related to accuracy\n",
    "\n",
    "* Accuracy is inversely proportional to the [negative log (logistic, cross-entropy) loss](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.log_loss.html#sklearn.metrics.log_loss) that many classifiers optimize:\n",
    "\n",
    "$$\n",
    "-\\frac{1}{N} \\sum_{i=1}^{N} \\sum_{k=1}^{K} y_{i,k} \\log(p_{i,k})\n",
    "$$\n",
    "\n",
    "* Accuracy can be related in a similar way to [KL divergence](https://en.wikipedia.org/wiki/Kullback–Leibler_divergence):    \n",
    "$$\n",
    "D_{\\text{KL}}(y \\parallel p) = \n",
    "    \\sum _{k=1}^{K} y_{k} \\log\\left(\\frac {y_{k}}{p_{k}}\\right)\n",
    "$$\n",
    "  Where $y$ is a \"one-hot vector\" (a classification label) with $1$ at position $k$, this reduces to \n",
    "  $$\n",
    "  \\log\\left(\\frac{1}{p_{k}}\\right) = -\\log(p_{k})\n",
    "  $$\n",
    "  Thus, KL-divergence is an analogue of accuracy for soft labels."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Precision Score\n",
    "\n",
    "It is the portion of true positive class among predicted postive classes. It is preferable choice when you want to be very sure of your predictions.\n",
    "\n",
    "It can be expressed as :\n",
    "\n",
    "$$\n",
    "\n",
    "precision\\ score = \\frac{{TP }}{{TP + FP}}\n",
    "\n",
    " $$ \n",
    "\n",
    " [Precision](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html#sklearn.metrics.precision_score) is the sum of the correct predictions divided by the sum of all guesses. This is a per-class notion; in our confusion matrices, it's the diagonal values divided by the column sums:\n",
    "\n",
    "**Precision bounds**\n",
    "\n",
    "[0, 1], with 0 the worst and 1 the best. (Caveat: undefined values resulting from dividing by 0 need to be mapped to 0.)\n",
    "\n",
    "**Value encoded by precision**\n",
    "\n",
    "Precision encodes a _conservative_ value in penalizing incorrect guesses.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "    function to calculate precision score of each class using confusion matrix\n",
    "\n",
    "    :param a 2D confusion matrix\n",
    "\n",
    "    :return a list containing precision for each class\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "def precision_score(confusion_matrix):\n",
    "\n",
    "    n_classes = confusion_matrix.shape[0]\n",
    "    precision_score_list = []\n",
    "    \n",
    "    for i in range(n_classes):\n",
    "\n",
    "        tp = confusion_matrix[i][i]\n",
    "        fp = np.sum(confusion_matrix, axis=0)[i] - tp\n",
    "        precision_score_list.append(tp/(tp+fp) if tp+fp!=0 else 0)\n",
    "\n",
    "    return precision_score_list\n",
    "\n",
    "     "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Weaknesses of precision\n",
    "\n",
    "Precision's dangerous edge case is that one can achieve very high precision for a category by rarely guessing it. Consider, for example, the following classifier's flawless predictions for __pos__ and __neg__. These predictions are at the expense of __neutral__, but that is such a big class that it hardly matters to the precision for that class either."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Recall Score\n",
    "\n",
    "[Recall](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html#sklearn.metrics.recall_score) is the sum of the correct predictions divided by the sum of all true instances. This is a per-class notion; in our confusion matrices, it's the diagonal values divided by the row sums. Recall is sometimes called the \"true positive rate\".\n",
    "\n",
    "It measures the proportion of actual positive cases that are correctly predicted by the model.\n",
    "\n",
    "Recall trades off against precision. \n",
    "\n",
    "It is calculated as:\n",
    "\n",
    "\n",
    "$$\n",
    "\n",
    "recall\\ score = \\frac{{TP }}{{TP + FN}}\n",
    "\n",
    "$$ \n",
    "\n",
    "\n",
    "\n",
    "\n",
    "A higher recall score indicates that the model is effectively capturing a larger proportion of the actual positive cases. It is a suitable metric when the goal is to minimize false negatives, i.e., correctly identifying as many positive cases as possible.\n",
    "\n",
    "**Recall bounds**\n",
    "\n",
    "[0, 1], with 0 the worst and 1 the best.\n",
    "\n",
    "**Value encoded by recall**\n",
    "\n",
    "Recall encodes a _permissive_ value in penalizing only missed true cases.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "    function to calculate recall score of each class using confusion matrix\n",
    "\n",
    "    :param a 2D confusion matrix\n",
    "\n",
    "    :return a list containing recall for each class\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "def recall_score(confusion_matrix):\n",
    "\n",
    "    n_classes = confusion_matrix.shape[0]\n",
    "    recall_score_list = []\n",
    "\n",
    "    for i in range(n_classes):\n",
    "\n",
    "        tp = confusion_matrix[i][i]\n",
    "        fn = np.sum(confusion_matrix, axis=1)[i] - tp\n",
    "\n",
    "        recall_score_list.append(tp/(tp+fn) if tp+fn!=0 else 0)\n",
    "\n",
    "    return recall_score_list\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Weaknesses of recall\n",
    "\n",
    "Recall's dangerous edge case is that one can achieve very high recall for a category by always guessing it. This could mean a lot of incorrect guesses, but recall sees only the correct ones. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### F Scores\n",
    "[F scores](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.fbeta_score.html#sklearn.metrics.fbeta_score) combine precision and recall via their harmonic mean, with a value $\\beta$ that can be used to emphasize one or the other. Like precision and recall, this is a per-category notion.\n",
    "\n",
    "It captures tradeoff between precision and recall.\n",
    "\n",
    "\n",
    "$$\n",
    "(\\beta^{2}+1) \\cdot \\frac{\\textbf{precision} \\cdot\n",
    "          \\textbf{recall}}{(\\beta^{2} \\cdot \\textbf{precision}) +\n",
    "          \\textbf{recall}}\n",
    "$$\n",
    "\n",
    "Where $\\beta=1$, we have F1:\n",
    "\n",
    "$$\n",
    "\n",
    "F-1\\ score = \\frac{{2*Precision * Recall}}{{Precision + Recall}}\n",
    "\n",
    "$$\n",
    "\n",
    "The `F1 score` is a suitable choice when both `precision` and `recall` are equally important, and we want to balance between the two. It is commonly used in classification problems where achieving a good balance between `precision` and `recall` is desired.\n",
    "\n",
    "**Bounds of F scores**\n",
    "\n",
    "[0, 1], with 0 the worst and 1 the best, and guaranteed to be between precision and recall.\n",
    "\n",
    "**Value encoded by F scores**\n",
    "\n",
    "The $F_{\\beta}$ score for a class $K$ is an attempt to summarize how well the classifier's $K$ predictions align with the true instances of $K$. Alignment brings in both missed cases and incorrect predictions. Intuitively, precision and recall keep each other in check in the calculation. This idea runs through almost all robust classification metrics.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "    function to calculate f-1 score of each class using confusion matrix\n",
    "\n",
    "    :param a 2D confusion matrix\n",
    "\n",
    "    :return a list containing f-1 score for each class\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "def f1_score(confusion_matrix):\n",
    "\n",
    "    #get precision score for each class\n",
    "    precision_score_list = precision_score(confusion_matrix)\n",
    "\n",
    "    #get recall score for each class\n",
    "    recall_score_list = recall_score(confusion_matrix)\n",
    "\n",
    "    f1_score_list = [2*p*r/(p+r) if p+r != 0 else 0 for p, r in zip(precision_score_list,recall_score_list)]\n",
    "\n",
    "    return f1_score_list\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Weaknesses of F scores**\n",
    "\n",
    "* There is no normalization for the size of the dataset within $K$ or outside of it.\n",
    "\n",
    "* For a given category $K$, the $F_{\\beta}$ score for $K$ ignores  all the values that are off the row and column for $K$, which might be the majority of the data. This means that the individual scores for a category can be very misleading about the overall performance of the system. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Macro-averaged F scores\n",
    "\n",
    "The [macro-averaged $F_{\\beta}$ score](http://scikit-learn.org/stable/modules/model_evaluation.html#multiclass-and-multilabel-classification) (macro $F_{\\beta}$) is the mean of the $F_{\\beta}$ score for each category:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def macro_f_score(cm, beta):\n",
    "    return f_score(cm, beta).mean(skipna=False)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Bounds of macro-averaged F scores**\n",
    "\n",
    "[0, 1], with 0 the worst and 1 the best, and guaranteed to be between precision and recall.\n",
    "\n",
    "**Value encoded by macro-averaged F scores**\n",
    "\n",
    "Macro $F_{\\beta}$ scores inherit the values of $F_{\\beta}$ scores, and they additionally say that we care about all the classes equally regardless of their size. \n",
    "\n",
    "**Weaknesses of macro-averaged F scores**\n",
    "\n",
    "In NLP, we typically care about modeling all of the classes well, so macro-$F_{\\beta}$ scores often seem appropriate. However, this is also the source of their primary weaknesses:\n",
    "\n",
    "* If a model is doing really well on a small class $K$, its high macro $F_{\\beta}$ score might mask the fact that it mostly makes incorrect predictions outside of $K$. So $F_{\\beta}$ scoring will make this kind of classifier look better than it is.\n",
    "\n",
    "* Conversely, if a model does well on a very large class, its overall performance might be high even if it stumbles on some small classes. So $F_{\\beta}$ scoring will make this kind of classifier look worse than it is, as measured by sheer number of good predictions.\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Weighted F scores\n",
    "\n",
    "[Weighted $F_{\\beta}$ scores](http://scikit-learn.org/stable/modules/model_evaluation.html#multiclass-and-multilabel-classification) average the per-category F$_{\\beta}$ scores, but it's a weighted average based on the size of the classes in the observed/gold data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def weighted_f_score(cm, beta):\n",
    "    scores = f_score(cm, beta=beta).values\n",
    "    weights = cm.sum(axis=1)\n",
    "    return np.average(scores, weights=weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'ex3' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m/Users/ajitkumarsingh/Desktop/All-About-Performance-Metrics/classification_metrics.ipynb Cell 62\u001b[0m in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> <a href='vscode-notebook-cell:/Users/ajitkumarsingh/Desktop/All-About-Performance-Metrics/classification_metrics.ipynb#Y212sZmlsZQ%3D%3D?line=0'>1</a>\u001b[0m weighted_f_score(ex3, beta\u001b[39m=\u001b[39m\u001b[39m1.0\u001b[39m)\n",
      "\u001b[0;31mNameError\u001b[0m: name 'ex3' is not defined"
     ]
    }
   ],
   "source": [
    "weighted_f_score(ex3, beta=1.0)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Bounds of weighted F scores**\n",
    "\n",
    "[0, 1], with 0 the worst and 1 the best, but without a guarantee that it will be between precision and recall.\n",
    "\n",
    "**Value encoded by weighted F scores**\n",
    "\n",
    "Weighted $F_{\\beta}$ scores inherit the values of $F_{\\beta}$ scores, and they additionally say that we want to weight the summary by the number of actual and predicted examples in each class. This will probably correspond well with how the classifier will perform, on a per example basis, on data with the same class distribution as the training data.\n",
    "\n",
    "**Weaknesses of weighted F scores**\n",
    "\n",
    "Large classes will dominate these calculations. Just like macro-averaging, this can make a classifier look artificially good or bad, depending on where its errors tend to occur."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Micro-averaged F scores\n",
    "\n",
    "[Micro-averaged $F_{\\beta}$ scores](http://scikit-learn.org/stable/modules/model_evaluation.html#multiclass-and-multilabel-classification) (micro $F_{\\beta}$ scores) add up the 2 $\\times$ 2 confusion matrices for each category versus the rest, and then they calculate the $F_{\\beta}$ scores, with the convention being that the positive class's $F_{\\beta}$ score is reported. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function creates the 2 $\\times$ 2 matrix for a category `cat` in a confusion matrix `cm`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cat_versus_rest(cm, cat):\n",
    "    yes = cm.loc[cat, cat]\n",
    "    yes_no = cm.loc[cat].sum() - yes\n",
    "    no_yes = cm[cat].sum() - yes\n",
    "    no = cm.values.sum() - yes - yes_no - no_yes\n",
    "    return pd.DataFrame(\n",
    "        [[yes,    yes_no],\n",
    "         [no_yes,    no]],\n",
    "        columns=['yes', 'no'],\n",
    "        index=['yes', 'no'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'ex1' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m/Users/ajitkumarsingh/Desktop/All-About-Performance-Metrics/classification_metrics.ipynb Cell 67\u001b[0m in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> <a href='vscode-notebook-cell:/Users/ajitkumarsingh/Desktop/All-About-Performance-Metrics/classification_metrics.ipynb#Y222sZmlsZQ%3D%3D?line=0'>1</a>\u001b[0m display(ex1)\n\u001b[1;32m      <a href='vscode-notebook-cell:/Users/ajitkumarsingh/Desktop/All-About-Performance-Metrics/classification_metrics.ipynb#Y222sZmlsZQ%3D%3D?line=1'>2</a>\u001b[0m display(cat_versus_rest(ex1, \u001b[39m'\u001b[39m\u001b[39mpos\u001b[39m\u001b[39m'\u001b[39m))\n\u001b[1;32m      <a href='vscode-notebook-cell:/Users/ajitkumarsingh/Desktop/All-About-Performance-Metrics/classification_metrics.ipynb#Y222sZmlsZQ%3D%3D?line=2'>3</a>\u001b[0m display(cat_versus_rest(ex1, \u001b[39m'\u001b[39m\u001b[39mneg\u001b[39m\u001b[39m'\u001b[39m))\n",
      "\u001b[0;31mNameError\u001b[0m: name 'ex1' is not defined"
     ]
    }
   ],
   "source": [
    "display(ex1)\n",
    "display(cat_versus_rest(ex1, 'pos'))\n",
    "display(cat_versus_rest(ex1, 'neg'))\n",
    "display(cat_versus_rest(ex1, 'neutral'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'ex1' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m/Users/ajitkumarsingh/Desktop/All-About-Performance-Metrics/classification_metrics.ipynb Cell 68\u001b[0m in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> <a href='vscode-notebook-cell:/Users/ajitkumarsingh/Desktop/All-About-Performance-Metrics/classification_metrics.ipynb#Y223sZmlsZQ%3D%3D?line=0'>1</a>\u001b[0m \u001b[39msum\u001b[39m([cat_versus_rest(ex1, cat) \u001b[39mfor\u001b[39;00m cat \u001b[39min\u001b[39;00m ex1\u001b[39m.\u001b[39mindex])\n",
      "\u001b[0;31mNameError\u001b[0m: name 'ex1' is not defined"
     ]
    }
   ],
   "source": [
    "sum([cat_versus_rest(ex1, cat) for cat in ex1.index])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the micro $F_{\\beta}$ score, we just add up these per-category confusion matrices and calculate the $F_{\\beta}$ score:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def micro_f_score(cm, beta):\n",
    "    c = sum([cat_versus_rest(cm, cat) for cat in cm.index])\n",
    "    return f_score(c, beta=beta).loc['yes']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'ex1' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m/Users/ajitkumarsingh/Desktop/All-About-Performance-Metrics/classification_metrics.ipynb Cell 71\u001b[0m in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> <a href='vscode-notebook-cell:/Users/ajitkumarsingh/Desktop/All-About-Performance-Metrics/classification_metrics.ipynb#Y226sZmlsZQ%3D%3D?line=0'>1</a>\u001b[0m micro_f_score(ex1, beta\u001b[39m=\u001b[39m\u001b[39m1.0\u001b[39m)\n",
      "\u001b[0;31mNameError\u001b[0m: name 'ex1' is not defined"
     ]
    }
   ],
   "source": [
    "micro_f_score(ex1, beta=1.0)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Bounds of micro-averaged F scores**\n",
    "\n",
    "[0, 1], with 0 the worst and 1 the best, and guaranteed to be between precision and recall.\n",
    "\n",
    "**Value encoded by micro-averaged F scores**\n",
    "\n",
    "* Micro $F_{\\beta}$ scores inherit the values of weighted $F_{\\beta}$ scores. (The resulting scores tend to be very similar.)\n",
    "\n",
    "* For two-class problems, this has an intuitive interpretation in which precision and recall are defined in terms of correct and incorrect guesses ignoring the class. \n",
    "\n",
    "**Weaknesses of micro-averaged F scores**\n",
    "\n",
    "The weaknesses too are the same as those of weighted F$_{\\beta}$ scores, with the additional drawback that we actually get two potentially very different values, for the positive and negative classes, and we have to choose one to meet our goal of having a single summary number. (See the `'yes'` in the final line of `micro_f_score`.)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Precision–recall curves\n",
    "\n",
    "Confusion matrices hide a threshold for turning probabilities/scores into predicted labels. With precision–recall curves, we finally address this.\n",
    "\n",
    "A precision–recall curve is a method for summarizing the relationship between precision and recall for a binary classifier. \n",
    "\n",
    "The basis for this calculation is not the confusion matrix, but rather the raw scores or probabilities returned by the classifier. Normally, we use 0.5 as the threshold for saying that a prediction is positive. However, each distinct real value in the set of predictions is a potential threshold. The precision–recall curve explores this space.\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's a basic implementation; [the sklearn version](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html) is more flexible and so recommended for real experimental frameworks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def precision_recall_curve(y, probs):\n",
    "    \"\"\"`y` is a list of labels, and `probs` is a list of predicted\n",
    "    probabilities or predicted scores -- likely a column of the\n",
    "    output of `predict_proba` using an `sklearn` classifier.\n",
    "    \"\"\"\n",
    "    thresholds = sorted(set(probs))\n",
    "    data = []\n",
    "    for t in thresholds:\n",
    "        # Use `t` to create labels:\n",
    "        pred = [1 if p >= t else 0 for p in probs]\n",
    "        # Precision/recall analysis as usual, focused on\n",
    "        # the positive class:\n",
    "        cm = pd.DataFrame(confusion_matrix(y, pred))\n",
    "        prec = precision_score(cm)[1]\n",
    "        rec = recall_score(cm)[1]\n",
    "        data.append((t, prec, rec))\n",
    "    # For intuitive graphs, always include this end-point:\n",
    "    data.append((None, 1, 0))\n",
    "    return pd.DataFrame(\n",
    "        data, columns=['threshold', 'precision', 'recall'])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'll illustrate with a hypothetical binary classification problem involving balanced classes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'np' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m/Users/ajitkumarsingh/Desktop/All-About-Performance-Metrics/classification_metrics.ipynb Cell 61\u001b[0m in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> <a href='vscode-notebook-cell:/Users/ajitkumarsingh/Desktop/All-About-Performance-Metrics/classification_metrics.ipynb#Y153sZmlsZQ%3D%3D?line=0'>1</a>\u001b[0m y \u001b[39m=\u001b[39m np\u001b[39m.\u001b[39mrandom\u001b[39m.\u001b[39mchoice((\u001b[39m0\u001b[39m, \u001b[39m1\u001b[39m), size\u001b[39m=\u001b[39m\u001b[39m1000\u001b[39m, p\u001b[39m=\u001b[39m(\u001b[39m0.5\u001b[39m, \u001b[39m0.5\u001b[39m))\n",
      "\u001b[0;31mNameError\u001b[0m: name 'np' is not defined"
     ]
    }
   ],
   "source": [
    "y = np.random.choice((0, 1), size=1000, p=(0.5, 0.5))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Suppose our classifier is generally able to distinguish the two classes, but it never predicts a value above 0.4, so our usual methods of thresholding at 0.5 would make the classifier look very bad:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = [np.random.uniform(0.0, 0.3) if x == 0 else np.random.uniform(0.1, 0.4)\n",
    "         for x in y]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The precision–recall curve can help us identify the optimal threshold given whatever our real-world goals happen to be:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prc = precision_recall_curve(y, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_precision_recall_curve(prc):\n",
    "    ax1 = prc.plot.scatter(x='recall', y='precision', legend=False)\n",
    "    ax1.set_xlim([0, 1])\n",
    "    ax1.set_ylim([0, 1.1])\n",
    "    ax1.set_ylabel(\"precision\")\n",
    "    ax2 = ax1.twiny()\n",
    "    ax2.set_xticklabels(prc['threshold'].values[::100].round(3))\n",
    "    _ = ax2.set_xlabel(\"threshold\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_precision_recall_curve(prc)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Value encoded by precision–recall curves**\n",
    "\n",
    "With precision–recall curves, we get a generalized perspective on F1 scores (and we could weight precision and recall differently to achieve the effects of `beta` for F scores more generally). These curves can be used, not only to assess a system, but also to identify an optimal decision boundary given external goals. \n",
    "\n",
    "**Weaknesses of precision–recall curves**\n",
    "\n",
    "* Most implementations are limited to binary problems. The basic concepts are defined for multi-class problems, but it's very difficult to understand the resulting hyperplanes.\n",
    "\n",
    "* There is no single statistic that does justice to the full curve, so this metric isn't useful on its own for guiding development and optimization. Indeed, opening up the decision threshold in this way really creates another hyperparameter that one has to worry about!"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Average precision\n",
    "\n",
    "Average precision is a method for summarizing the precision–recall curve. It does this by calculating the average precision weighted by the change in recall from step to step along the curve. \n",
    "\n",
    "Here is the calculation in terms of the data structures returned by `precision_recall_curve` above, in which (as in sklearn) the largest recall value is first:\n",
    "\n",
    "$$\\textbf{average-precision}(r, p) = \\sum_{i=1}^{n} (r_{i} - r_{i+1})p_{i}$$\n",
    "\n",
    "where $n$ is the increasing sequence of thresholds and the precision and recall vectors $p$ and $r$ are of length $n+1$. (We insert a final pair of values $p=1$ and $r=0$ in the precision–recall curve calculation, with no threshold for that point.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def average_precision(p, r):\n",
    "    total = 0.0\n",
    "    for i in range(len(p)-1):\n",
    "        total += (r[i] - r[i+1]) * p[i]\n",
    "    return total"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'plot_precision_recall_curve' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m/Users/ajitkumarsingh/Desktop/All-About-Performance-Metrics/classification_metrics.ipynb Cell 71\u001b[0m in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> <a href='vscode-notebook-cell:/Users/ajitkumarsingh/Desktop/All-About-Performance-Metrics/classification_metrics.ipynb#Y201sZmlsZQ%3D%3D?line=0'>1</a>\u001b[0m plot_precision_recall_curve(prc)\n",
      "\u001b[0;31mNameError\u001b[0m: name 'plot_precision_recall_curve' is not defined"
     ]
    }
   ],
   "source": [
    "plot_precision_recall_curve(prc)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Bounds of average precision**\n",
    "\n",
    "[0, 1], with 0 the worst and 1 the best.\n",
    "\n",
    "**Value encoded by average precision**\n",
    "\n",
    "This measure is very similar to the F1 score, in that it is seeking to balance precision and recall. Whereas the F1 score does this with the harmonic mean, average precision does it by making precision a function of recall.\n",
    "\n",
    "**Weaknesses of average precision**\n",
    "\n",
    "* An important weakness of this metric is cultural: it is often hard to tell whether a paper is reporting average precision or some interpolated variant thereof. The interpolated versions are meaningfully different and will tend to inflate scores. In any case, they are not comparable to the calculation defined above and implemented in `sklearn` as `sklearn.metrics.average_precision_score`.\n",
    "\n",
    "* Unlike for precision–recall curves, we aren't strictly speaking limited to binary classification here. Since we aren't trying to visualize anything, we can do these calculations for multi-class problems. However, then we have to decide on how the precision and recall values will be combined for each step: macro-averaged, weighted, or micro-averaged, just as with F$_{\\beta}$ scores. This introduces another meaningful design choice."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Log Loss or Binary Cross Entropy\n",
    "\n",
    "Log Loss, also known as Binary Cross Entropy, is a common metric used to evaluate the performance of a model when the output is in the form of `predicted probabilities`. It is calculated as the negative average of the sum of the logarithm of the predicted probabilities for the true labels and the logarithm of the complement of the predicted probabilities for the false labels.\n",
    "\n",
    "It can be expressed as:\n",
    "$$\n",
    "Log Loss = -\\frac{1}{N}\\sum_{i=1}^{N} \\left[ y_i \\cdot \\log(p_i) + (1 - y_i) \\cdot \\log(1 - p_i) \\right]\n",
    "$$\n",
    "\n",
    "The negative sign in the Log Loss formula is used to ensure that the overall value of the metric is minimized during optimization. In other words, it is used to represent the loss or cost associated with misclassification.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 282,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "    function to calculate log loss of a model's prediction in case of binary class classification\n",
    "\n",
    "    :param a y_true - actual label and y_pred - predicted probabilties\n",
    "\n",
    "    :return log loss a singular value\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "def log_loss(y_true, y_pred_prob):\n",
    "\n",
    "    #apply expression for calculating log loss\n",
    "\n",
    "    log_loss = np.average([-(y_ti*np.log(y_pi)+(1-y_ti)*np.log(1-y_pi)) for y_ti, y_pi in zip(y_true, y_pred_prob)])\n",
    "\n",
    "    return log_loss\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Area Under Receiver Operating Characterstics Curve (AUROC)\n",
    "\n",
    "The Receiver Operating Characteristic (ROC) curve for a class $k$ depicts recall the __false positive rate__ (FPR) for $k$ as a function of the __recall__ for $k$. For instance, suppose we focus on $k$ as the positive class $A$:\n",
    "\n",
    "$$\n",
    "\\begin{array}{r r r}\n",
    "\\hline\n",
    " & A & B \\\\\n",
    "\\hline\n",
    "A & \\text{TP}_{A} & \\text{FN}_{A}\\\\\n",
    "B & \\text{FP}_{A} & \\text{TN}_{A}\\\\\n",
    "\\hline\n",
    "\\end{array}\n",
    "$$\n",
    "\n",
    "The false positive rate is \n",
    "\n",
    "$$\n",
    "\\textbf{fpr}(A) = \\frac{\\text{FP}_{A}}{\\text{FP}_{A} + \\text{TN}_{A}}\n",
    "$$\n",
    "\n",
    "which is equivalent to 1 minus the recall for $B$ class. \n",
    "This Indicates how well probabilities from positive class are seperated from negative class.\n",
    "\n",
    "ROC summarizes models performance by evaluating the trade off TPR(sensitivity) and FPR(1-specificity).\n",
    "\n",
    "\n",
    "The ROC curve is a probability curve, and for an ideal model, the area under the ROC curve would be 1, indicating perfect classification performance. The interpretation of AUROC is as follows:\n",
    "\n",
    "- AUC is `0.x`, It means there is `10x%` chance of fair classification.\n",
    "\n",
    "It's important to note that the AUROC is commonly used for binary class classification problems. For multiclass classification, there would be `N` number of AUROC curves using the `one vs rest` methodology, where each class is compared against the rest of the classes separately.\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Bounds of ROC**\n",
    "\n",
    "* For individual ROC calculations of recall divided fpr: [0, $\\infty$), with larger better.\n",
    "* For ROC AUC: [0, 1], with 1 the best.\n",
    "\n",
    "**Weaknesses of ROC**\n",
    "\n",
    "Recall that, for two classes $A$ and $B$, \n",
    "\n",
    "$$\n",
    "\\begin{array}{r r r}\n",
    "\\hline\n",
    " & A & B \\\\\n",
    "\\hline\n",
    "A & \\text{TP}_{A} & \\text{FN}_{A}\\\\\n",
    "B & \\text{FP}_{A} & \\text{TN}_{B}\\\\\n",
    "\\hline\n",
    "\\end{array}\n",
    "$$\n",
    "\n",
    "we can express ROC as comparing $\\textbf{recall}(A)$ with $1.0 - \\textbf{recall}(B)$.\n",
    "\n",
    "This reveals a point of contrast with scores based in precision and recall: the entire table is used, whereas precision and recall for a class $k$ ignore the $\\text{TN}_{k}$ values. Thus, whereas precision and recall for a class $k$ will be insensitive to changes in $\\text{TN}_{k}$, ROC will be affected by such changes. The following individual ROC calculations help to bring this out:\n",
    "\n",
    "$$\n",
    "\\begin{array}{r r r r r}\n",
    "\\hline\n",
    " & A & B & \\textbf{F1} & \\textbf{ROC}\\\\\n",
    "\\hline\n",
    "A & 15 & 10 & 0.21 & 0.90 \\\\\n",
    "B & 100 & {\\color{blue}{50}} & 0.48 & 0.83 \\\\\n",
    "\\hline\n",
    "\\end{array}\n",
    "\\qquad\n",
    "\\begin{array}{r r r r r}\n",
    "\\hline\n",
    " & A & B & \\textbf{F1} & \\textbf{ROC} \\\\\n",
    "\\hline\n",
    "A & 15 & 10 & 0.21 & 3.6 \\\\\n",
    "B & 100 & {\\color{blue}{500}} & 0.90 & 2.08  \\\\\n",
    "\\hline\n",
    "\\end{array}\n",
    "$$\n",
    "\n",
    "One might worry that the model on the right isn't better at identifying class $A$, even though its ROC value for $A$ is larger."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate False positive and True positive rates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 274,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "    Calculate True Positive Rate (TPR) and False Positive Rate (FPR) for different thresholds in case of binary class problems\n",
    "\n",
    "    Args:\n",
    "        y_true (array-like): Actual labels (ground truth).\n",
    "        y_pred_prob (array-like): Predicted probabilities for positive class.\n",
    "\n",
    "    Returns:\n",
    "        tuple: A tuple containing two lists - TPR (True Positive Rate) and FPR (False Positive Rate)\n",
    "              for each threshold.\n",
    "\"\"\"\n",
    "\n",
    "def get_fpr_tpr(y_true, y_pred_prob):\n",
    "\n",
    "    #stores tpr and fpr at different thresholds\n",
    "    tprs = []\n",
    "    fprs = []\n",
    "    \n",
    "    #sort y_pred_prob in decreasing order\n",
    "    # First sort indices in descending order\n",
    "    sort_indices = np.argsort(y_pred_prob)[::-1]  \n",
    "    # Sort y_score array\n",
    "    y_score_sorted = y_pred_prob[sort_indices]  \n",
    "    # Rearrange y_true array to match sorted order of y_score\n",
    "    y_true_sorted = y_true[sort_indices]  \n",
    "\n",
    "    #derive classification threshold from y_pred_prob\n",
    "    thresholds = np.sort(list(set(y_pred_prob)))[::-1]\n",
    "    \n",
    "    for threshold in thresholds:\n",
    "        \n",
    "        y_pred = [y_p > threshold for y_p in y_score_sorted]\n",
    "\n",
    "        conf_mat = confusion_matrix(y_true_sorted, y_pred, list(set(y_true_sorted)))\n",
    "        \n",
    "        #using confusion matrix we can define tp, fp and hence tpr and fpr\n",
    "        tp = conf_mat[1][1]\n",
    "        fp = conf_mat[0][1]\n",
    "        fn = conf_mat[1][0]\n",
    "        tn = conf_mat[0][0]\n",
    "\n",
    "        tpr = tp/(tp+fn) if tp+fn != 0 else 1\n",
    "        fpr = fp/(fp+tn) if fp+tn != 0 else 1\n",
    "\n",
    "        #append the above values\n",
    "        tprs.append(tpr)\n",
    "        fprs.append(fpr)\n",
    "    \n",
    "    return fprs, tprs\n",
    "    "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find Area Under Curve (AUC)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 264,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "    Find Area Under ROC Curve\n",
    "\n",
    "    Args:\n",
    "        tprs (list of tpr at different thresholds)\n",
    "        fprs (list of fpr at different thresholds)\n",
    "\n",
    "    Returns:\n",
    "        a number - area under roc curve\n",
    "\"\"\"\n",
    "\n",
    "def auc_roc(fpr, tpr):\n",
    "    \n",
    "    #use trapezoidal rule to find area\n",
    "    auc = np.trapz(fpr, tpr)\n",
    "    \n",
    "    return auc\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot ROC Curve\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 265,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "    Plot ROC curve \n",
    "\n",
    "    Args:\n",
    "        tprs (list): true positive rates(recall/sensitivity)\n",
    "        fprs (list): false positive rates(1-specificity)\n",
    "\n",
    "    Returns:\n",
    "        None\n",
    "\"\"\"\n",
    "\n",
    "def plot_roc(fpr, tpr, model_name=\"Logistic Regressor\"):\n",
    "    \n",
    "    #get area under the curve\n",
    "    area_under_roc = auc_roc(fpr, tpr)\n",
    "    #define fig size\n",
    "    plt.figure(figsize=(8, 6))\n",
    "    plt.plot(fpr, tpr, color='darkorange', lw=2, label='ROC curve (area = %0.2f)' % area_under_roc)\n",
    "    plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')\n",
    "\n",
    "    #set x and y limit\n",
    "    plt.xlim([0.0, 1.0])\n",
    "    plt.ylim([0.0, 1.05])\n",
    "\n",
    "    #put headings\n",
    "    plt.xlabel('False Positive Rate (FPR)')\n",
    "    plt.ylabel('True Positive Rate (TPR)')\n",
    "    plt.title('Receiver Operating Characteristic (ROC) Curve for ' + model_name)\n",
    "    plt.legend(loc=\"lower right\")\n",
    "\n",
    "    plt.show()\n",
    "\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Prediction\n",
    "\n",
    "Now we have defined the performance metrics, we can use them to evaluate the model predctions.\n",
    "\n",
    "From Train/Test Split Section we have :\n",
    "\n",
    "- `train_data`, `test_data`, `train_labels` and `test_labels` we can use them for models performance evaluations.\n",
    "\n",
    "From Model Training section we have : \n",
    "\n",
    "- trained instances of `SVM classifier`, `Logestic Regressor`, `Random Forest Classifier` and `Nearest Neighbour Classifier` in `model_instances` dictionary. We need to feed the test data to these instances  to get the predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 246,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "    get predictions from test instances\n",
    "\n",
    "    Args:\n",
    "        trained_model () : trained instance\n",
    "        test_data (pandas dataframe) : testing data without label\n",
    "\n",
    "    Returns:\n",
    "        list of predictions against each row in test data\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "def make_predictions(trained_model, test_data):\n",
    "\n",
    "    return trained_model.predict(test_data)\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 247,
   "metadata": {},
   "outputs": [],
   "source": [
    "#store predictions against each model\n",
    "\n",
    "y_pred_per_model = {}\n",
    "\n",
    "for model_name, model_instance in model_instances.items():\n",
    "    \n",
    "    y_pred_per_model[model_name] = make_predictions(model_instance, test_data) \n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have stored model name and it's predictions over test data in `y_pred_per_model` and we have also `test_labels` which is like ground truths."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Evaluation\n",
    "\n",
    "This is like the last step we need to perform to compare which classification algorithm is performating relatively better over test distribution. In the above sctions, We are done with implementaion of performance metrics now we can use to evaluate the trained models."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plotting Confusion Matrix\n",
    "\n",
    "For each model, let's plot the confusion matrix separately."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 286,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "def plot_cm(conf_matrix, model_name, labels, ax):\n",
    "\n",
    "    # visualize the confusion matrix\n",
    "    ax.imshow(conf_matrix, cmap='Greens')\n",
    "    ax.set_title('Confusion Matrix for '+model_name)\n",
    "    ax.set_xticks(np.arange(2))\n",
    "    ax.set_yticks(np.arange(2))\n",
    "    ax.set_xticklabels(labels)\n",
    "    ax.set_yticklabels(labels)\n",
    "    ax.set_ylabel('True Label')\n",
    "    ax.set_xlabel('Predicted Label')\n",
    "    \n",
    "    #make sure score are align properly in graph\n",
    "    for i in range(2):\n",
    "        for j in range(2):\n",
    "            ax.text(j, i, int(conf_matrix[i, j]), ha='center', va='center', color='red')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 287,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqkAAAHwCAYAAACFT/ZlAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABTIElEQVR4nO3dd5wddbn48c+T3fSEhBp6r6FI7yWgImABCyoiig31Xi/X3q8UO4KiP2yoqCioiHgvCiJKkS69V5FOKIF0UrY8vz9mNjlZdjdnN7t7Zjef9+t1Xjt9npk9Z55nZr5zTmQmkiRJUpWMaHQAkiRJUmcWqZIkSaoci1RJkiRVjkWqJEmSKsciVZIkSZVjkSpJkqTKsUitU0SMjYg/RcTsiPj9Cizn6Ii4tD9ja4SI+EtEvLuP834lImZExDP9FEtGxOb9sJw+bVNE7BcRD6zo+iVJ0lLDrkiNiHdExM0RMS8ippeFx779sOi3AFOA1TPzyL4uJDPPycyD+yGeZUTEtLJYu6DT8FeUw6+sczknRsSvlzddZh6amb/sQ5wbAJ8Apmbm2r2dfyDVu02di+LMvDoztxrY6CStDLwgsqwqXRAZaBHxi4j4yiCv88MR8WxZM60+mOuux7AqUiPi48DpwNcoCsoNgR8Ah/fD4jcCHszM1n5Y1kB5Hti70xvt3cCD/bWCKKzI+2Yj4IXMfK4P625egfUOGf29nRHR1J/L68P6V4r/m1YuXhDxgshQFxEjgW8DB2fmhMx8YZDWOy0inqxn2mFTpEbEJOBk4D8z84LMnJ+ZLZn5p8z8VDnN6Ig4PSKeLl+nR8Tocty0iHgyIj4REc+VB533lONOAr4EvK08IL2v8wcsIjYuP6DNZf+xEfHviJgbEY9ExNE1w6+pmW/viLipPGu+KSL2rhl3ZUR8OSKuLZdzaUSs0cNuWAz8L/D2cv4m4K3AOZ321Xcj4omImBMRt0TEfuXwQ4DP12znHTVxfDUirgVeAjYth72/HP/DiDi/ZvnfjIjLIiI6rfdVwN+Adcvl/6Ic/oaIuCciZpXL3aZmnkcj4jMRcScwf3kFT0RMioizI+L5iHgsIr7YUVRHRFNEnFaeWT8SER/p9D+r3abNI+If5f9lRkT8rhx+VbmqO8pteFvnD1xEbBARF5QxvBARZywn5mPL//F3IuJF4MTyvXpqRDwexVnujyJibM08ny7fo09HxPuj5upuFGfjP4yIiyNiPnBgRKwbEX8oY3okIo6vWdbuUSTbOeW6vl0OHxMRvy63YVb5/pxSjls3Ii6MiBcj4l8R8YGa5Z0YEeeX884Bju1p+6WhJrwgstJfEKlnmiFgCjAGuKe3M/bD/6c+mTksXsAhQCvQ3MM0JwM3AGsBawLXAV8ux00r5z8ZGAkcRlGQrVqOPxH4dc2yOvdvDCTQDIwH5gBblePWAbYtu48Frim7VwNmAseU8x1V9q9ejr8SeBjYEhhb9n+jm22bBjwJ7A38sxx2GPBX4P3AlTXTvhNYvVznJ4BngDFdbVdNHI8D25bzjCyHvb8cP47i4HQssB8wA1i/pzhr+rcE5gOvLpf7aeBfwKhy/KPA7cAGwNhulpnA5mX32cD/ARPL/8mDwPvKcR8C7gXWB1YF/t7xP6vZzo5t+g3wBYoTuTHAvl2tr/M2AU3AHcB3yvfBMvN2E/+xFO+9/yr371iKBHhh+R6ZCPwJ+HrNe/2Z8v8xDvhVp33wC2A2sE8Z/zjgFooTrVHApsC/gdeU018PHFN2TwD2LLs/WK53XLlduwCrlOP+QZGUxwA7UiStV9a8h1qAI8r1d/l/8+VrKL6AScA84MgephldfoafLl+nA6PLcdMojtWfAJ4DpgPvKcedRHGxoaVcx/voIdeU/ceWn+e5wCPA0TXDr6mZb2/gpvLYcBOwd824K4EvA9eWy7kUWKObbeuI/0cUF4Uojw9PlseYK2um/S7wBEU+vAXYrxx+SKftvKMmjq+WcSwANmfZ4/IPgfNrlv9N4DIgOsX4qnL+9nL5vyiHv4GiIJtVLnebmnkeBT4D3Aksootaotzv/wk8BDzS0zaW404EzqPIS3PLde9aM34n4NZy3O+A3wJfqRn/AYp8+CJFPli3Uyz/UcYyt/z/bUZxPJ9TrndUD+/Rjtyb5T66vM73Sef/z9YUF59eBB4A3loz/WEUOXcu8BTwSYq8WPu/mVe7XS+Ls9Ef+H48cBwNPLOcaR4GDqvpfw3waM0Hb0HtG5PiANKRsE+kd0XqLODNdErQLFukHgPc2Gn89cCxNW+IL9aM+w/gkp4OHGX3Q8BW5Rv+aDoVqV3MOxN4RVfbVRPHyV0Me39N/+7lm/Qx4Kge1rUkzrL/f4DzavpHlG/maWX/o8B7l/N/zfLD0kRxcJlaM+6DHdsOXA58sGbcq+i+SD0bOJMuim16LlL3oijYuj1Z6mJ5xwKP1/QHxcFjs5phe7H0oHgWZcFa9m/Oy4vUs2vG71G7/HLY54Cfl91XUSTHNTpN816KE7kdOg3fAGgDJtYM+zpLE8GJwFW9/Qz78jUUXnhBZBor9wWRv5X7c2yd27iw3D9NFMfJG8pxoyjy5cfKeN5CUbR/pRx/ULl9O1Oc9Pw/ao6rZSwXAquU+2sRRcG+KcWJ1L3Au5fzXt6YZXNgPe+T2v/PJIoC/T1l/85lzB3vweksPTFZFdi5q/9NT69hc7sfeAFYYzmX4NeleFN0eKwctmQZuewtlpcoriz1SmbOB95GceVuekRcFBFb1xFPR0zr1fTXNviuN55fAR8BDgT+2HlkFE0a7itvZc+ieKP11IwAijditzLzRoqz+aA4g6vXMvsgM9vLddXugx7XXWMNln7wO9Tuz3U7Laun5X6aYltuLJsivLfOGDYAHsve36qrjWVNyquf5W32WcAl5XCobztqh21E0cRiVs3yPk9xqweKqzVbAveXt/RfVw7/FUXi+W3ZrOCUKNowrQu8mJlza9bR+X1b7/9MGmpWB2Ys5zN+NMWJ/XOZ+TzFSeAxNeNbyvEtmXkxxdWkvj582Q5sFxFjM3N6ZnZ16/a1wEOZ+avMbM3M3wD3A6+vmebnmflgZi6gOIbv2NNKM/M6YLWI2Ap4F8WJfedpfp2ZL5TrPI2i2Fredv4iM+8p52nptLyXKIrCbwO/Bv4rM+tq20iRky/KzL+Vyz2VoiDfu2aa72XmE+U+6M7XM/PFjmnq2MZrMvPizGyjOKa+ohy+J0Vxenr5Pjif4splh6OBszLz1sxcRHFhYa+I2Lhmmm9m5pzyf343cGlm/jszZwN/obhS2xv1vE+W/H8oTtgezcyfl9PfCvyBouCG4n0+NSJWycyZ5fheGU5F6vUUZyxH9DDN0xQJu8OG5bC+mE9RSHRYpmF2Zv41M19NcWZ7P/CTOuLpiOmpPsbU4VcUV10vLj/US0TR/vQzFG1VV83MyRSX9Tvaj2Y3y+xueMdy/5Piw/k0RYFXr2X2QUQERaFXuw96XHeNGRQfis7/445lTae41d9hg+4WlJnPZOYHMnNdiquxP4j6vubqCWDDPrRXqt3GGRRX9bfNzMnla1Jmdpyg1LMdtct7guIq7OSa18TMPAwgMx/KzKMorvp8Ezg/IsaXB86TMnMqxYH8dRTJ6GmK5DSxZh2d37f1/s+kocYLIkutjBdElpmmjm3svF/HlO+ddYGnsry0WKr9H3WOdx7Fe6823mdruhd00d/b91Q975POF0D26HQB5GiW1kNvpriK/Fj5jMdevYxn+BSp5ZnDl4DvR8QRETEuIkZGxKERcUo52W+AL0bEmlE8gPQlirOxvrgd2D8iNozioa3PdYyIiClRPAw0nuIS/DyK26OdXQxsGcVTos0R8TZgKvDnPsYEQGY+AhxA0aays4kUt5qeB5oj4ksUtws6PAts3JsG0RGxJfAVijPcY4BPR8SOdc5+HvDaiHhleZXuExT77Lp619+hPFM9D/hqREyMiI2Aj7P0f3we8N8RsV5ETKYo1rvbpiMjoqMQnElRdHX8D5+luKXSlRspishvRMT48uGjfXq5He0UJzXfiYi1ynjWi4jX1GzHeyJim4gYR/E+7smNwJwoHkAbG8UDZNtFxG7lst8ZEWuW651VztMWEQdGxPZRPIA3h+IEoC0zn6D4/3y93L4dKK7GnvOyNUvDjxdElloZL4gsmaaObezJdGC9Mo4OG/YQ73iKq/gr+j/rST3vk84XQP7R6QLIhMz8MEBm3pSZh1NcAPlflp5U1H0RY9gUqQCZ+W2KouSLFEXYExRnef9bTvIV4GaKhtF3UTRY7tN3kmXm3ygaOt9J0Vi6trAcQVFsPU3RTvMAig9y52W8QHF16hMUZ0ifBl6XmTP6ElOnZV+TmV0dFP9KcRvgQYozpIUse2bU8b18L0TEci/Nl2eEv6a47XBHZj5EcSv5V1F+c8Jy4nyAorj9fxRXEF8PvD4zFy9v3m78F8VB/d/ANcC5FG04oTh4X0rxP7uN4iShla5PIHYD/hkR8yja/fx3WfxD0c7ol+WZ41s7bU9buQ2bU7TdeZLiSkdvfYaivdQNUTwh/3fKW0iZ+Rfge8AV5TTXl/Ms6mpBNTHtSPFgxQzgpxRn/FDcsrmn3NbvAm/PzIUUyfB8igL1PoqHpToK/qMo2jM9TXEF5YTyMyENa14QWWplvSBSY3nb2JPry3mPL/8nb6J4tqPDuRQXI3Ysc+nXKNoAP7oC8S5Pb98nfy6nP6b8DIyMiN3KCyijoviu3kll84o5LHuhZ/Xy/dyzrEBDdF++GvECDqVoP9rwWFZwO7YpP/x1P6zly5evFXtR3Na8meKk+BngIsonoSm+9eJ7FFfLppfdHQ/TTKPTQyMUD+28quw+kZc/UPR9irsc/6J44rvjwal1KE4eZ7P0ifWp5TzHsuzT/ftSXFCZXf6t/caSK1n2Qdhl5u0Uy8virxm35MEpigeFflYWJ9MpLsLUbufqFBcSZgK3dhVH7bBye28EPlsz7sMUF5xG1xMn8EaKB4pml/tt267+Bz38z5c8oFrnNi7zv+TlDyrtSnHBpOPp/t+x7NP9H6J4oO1FioJw/R5iuYbyoeuy/yvAT5ezPcvE09v3STlsK4r3/vMUF9sup7ggMoriWYqZ5f65qdOyziqnn0UPT/dHObE07EXxPaMHUlxNnULRwPuGzPxoI+Pqi4h4I8WBYTzwS6A9M49oaFCSJPWjYXW7X1qOoHjKdibF2et9LL89Z/+suPgy/nldvH7Ux0V+kOLM9WGKq6gf7rdgJUmqAK+kSpIkDUMR8XmK50Q6uzozDx3seHrLIlWSJEmV4+1+SZIkVU5vv3BcNWLUiGSMu3Ag7Lzldo0OYdi69ZbbZmTmmsufUtJAMocMHHPIwBnMHOKnY0WMaYY91mp0FMPStZdc0+gQhq2xzeM7/6KIpEYwhwwYc8jAGcwc4u1+SZIkVY5FqiRJkirHIlWSJEmVY5EqSZKkyrFIlSRJUuVYpEqSJKlyLFIlSZJUORapkiRJqhyLVEmSJFWORaokSZIqxyJVkiRJlWORKkmSpMqxSJUkSVLlWKRKkiSpcixSJUmSVDkWqZIkSaoci1RJkiRVjkWqJEmSKsciVZIkSZVjkSpJkqTKsUiVJElS5VikSpIkqXIsUiVJklQ5FqmSJEmqHItUSZIkVY5FqiRJkirHIlWSJEmVY5EqSZKkyrFIlSRJUuVYpEqSJKlyLFIlSZJUORapkiRJqhyLVEmSJFWORaokSZIqxyJVkiRJlWORKkmSpMqxSJUkSVLlWKQOYT+7ZybP/mM6d13/7JJhJzw8hyevns5tNzzHbTc8x6EzFgLwjukvLRl22w3P0fb3p3jF3MUA/OW2Gdx+w7Pcff2z/PC+mYzIbMj2DAkLFzJqz/0ZtfMejNphV5pP/Moyo5tOO50xzeNhxowGBShJ9elNDumwwcJW5l7xNJ94bC4AE1rbl8ktz/9jOt95YNZgbsbQM2sWI996NKO23YlR2+1MXP9PePFFRr7mdYzaegdGvuZ1MHNmo6OshAErUiMiI+K0mv5PRsSJy5nniIiY2sP4d0XE3RFxT0TcGxGfLIdfGRG79lvwQ8Qv1h3HITut/rLh39lwAjvtuRY77bkWf1ljDADnrjNuybBjtluVR8c2ccfEUQC8dfvV2HHPKWy351qsubidI59dMKjbMaSMHs3iv1/M4lv/yeJbrmfEX/9G3HBjMe6JJxnx98vJDTdobIzSMGEeGVi9ySFLxj0wm7+sPnpJ/7zmEUum3WnPtXhsTBMXrDV2wGMfykZ+7FO0v+bVLL7nNhbfegO5zVY0f/M02g+axuL776T9oGk0f/O05S9oJTCQV1IXAW+KiDV6Mc8RQJcHl4g4FPgocHBmbgvsDMxewRg7lt3UH8sZbFevOpoXR/b+X3jUMy/xmylLDyJzm4tlNCeMSvA6ag8iYMKEorulBVpbimHAyE98htZvfGVJv6QVZh4ZQL3NIYc/t4B/j2vmnvEjuxy/+UutrLW4nasnj+qvEIefOXOIq6+l7b3vLvpHjYLJkxnxp4toe9fRALS962hGXPjnBgZZHQNZpLYCZwIf6zwiIjaKiMsi4s7y74YRsTfwBuBbEXF7RGzWabbPAZ/MzKcBMnNhZv6kZvyREXFjRDwYEfuV6zk2Is6oWe+fI2Ja2T0vIk6OiH8Ce5X9X42IOyLihoiY0n+7YnB95In53HHDs/zsnplMbml/2fi3PbuA36w9bplhl9w6g+eums7cpuD8KZ4F96itjVG77MnodTam/ZUHkXvsxog/XUSutw75ih0aHZ00nJhHGqCrHDKurZ3PPDaXkzaZ2O18Rz3zEr+bMtYT9R7Evx+BNdZg5Ps+yKhd96L5uP+A+fOJZ5+DddYpJlpnHeK55xsbaEUMdJvU7wNHR8SkTsPPAM7OzB2Ac4DvZeZ1wIXApzJzx8x8uNM82wG39LCu5szcneIs+YQ6YhsP3J2Ze2TmNWX/DZn5CuAq4ANdzRQRx0XEzRFxM10UgI32w/XHs9k+U9hxj7WYPnoEpz247EWC3Wcv5qURwT0Tlj0TPmTnNVhnv3UY3Z4c9OKiwQx56GlqYvEtN7DosQeJm24h7ryL5q+dQuuJ/9PoyKThaFjlkaGaQ056eC7f2XAC85u7Lxve/uwCfrO2Fzl61NpG3HY7rR/8AItvvh7Gj/PWfg8GtEjNzDnA2cDxnUbtBZxbdv8K2LcfVndB+fcWYOM6pm8D/lDTvxjouL7e7TIy88zM3DUzd6UPt9oH2nOjm2iPICP4yXrj2X3O4mXGv/2Zl152FbXDoqbgwjXHcPjzC7scr04mT6b9gP0YceFFxKOPMnrnPRm92Tbw5FOM3m0feOaZRkcoDXnDLY8M1Ryyx5zFnPLQHB655hk++sQ8Pv/IXP7ziXlL5tthbgvNmdy6irf6e5Lrrwvrr0fusRsAbW96I3Hb7eSUtWD69GKi6dPJtdZsYJTVMRifkNOB91GcYXannmaQ9wC79DC+4/JfG9Bcdrey7DbWtgBfmJltNf0tmUsea69dxpCy9qKlm/TG5xZyd80V08jkyOcW8Nua2/njW9uXzNPUnhz2wiLuHz8kN31wPP88zJpVdC9YQNNlV5A7voJF0x9j0cP3sejh+2D99Vh007Ww9toNDVUaRk7HPDIoussh+++6Jpvsuzab7Ls2p28wga9tMpHvbzBhybTFsw5dXwBRjbXXJtdfn3jgQQCaLr+S3GZr2l93GE1nn1MMO/sc2l//2kZGWRkD/gHKzBcj4jyKA8xZ5eDrgLdTnP0eDVxTDp8LdNfg5evAKRHxusx8JiJGAx/MzO/1sPpHgf+IiBHAesDuK7QxFXPuXS8ybeYi1mhp54mrp3PCpqswbeYidpzbQgY8OqaZD24zecn0+89czJOjm3hk3NJ/+/i25MI7XmB0e9KUcPmqo/nRej3lgZVbTH+Gke89DtraoL2dtre8mfbXHdrosKRhzTwyMHqbQ3ry1ucWcNiOL/+mAL1cy3dPZeS73guLF5ObbELLz34E7e2MfPsxNP38bHKD9Wn53a8bHWYlDNZZ3mnAR2r6jwfOiohPAc8D7ymH/xb4SUQcD7yltj1RZl5cNkL/e0QExVnzWfTsWuAR4C7gbuDW/tiYqnjH9qu9bNhZPRSY/1htNHvtvtYyw54b3cTunYape7nD9kU7oh4sevi+QYpGWqmYR/pZb3NIh5M2W+VlwzbbxztH9codX8Hif17zsuEtf7u4AdFUW6Rf3N5nscqoZA8LvIGw4JIHGx3CsDW2efwtmblSfR+kVEXmkIFjDhk4g5lDqtdqW5IkSSs9i1RJkiRVjkWqJEmSKsciVZIkSZVjkSpJkqTKsUiVJElS5VikSpIkqXIsUiVJklQ5FqmSJEmqHItUSZIkVY5FqiRJkirHIlWSJEmVY5EqSZKkyrFIlSRJUuU0dzciInbuacbMvLX/w5EkDRfmEUkrotsiFTith3EJHNTPsUiShhfziKQ+67ZIzcwDBzMQSdLwYh6RtCKW2yY1IsZFxBcj4syyf4uIeN3AhyZJGg7MI5L6op4Hp34OLAb2LvufBL4yYBFJkoYb84ikXqunSN0sM08BWgAycwEQAxqVJGk4MY9I6rV6itTFETGWopE7EbEZsGhAo5IkDSfmEUm91tPT/R1OAC4BNoiIc4B9gGMHMihJ0rBiHpHUa8stUjPzbxFxK7Anxe2Z/87MGQMemSRpWDCPSOqLeq6kAhwA7Etxq2Yk8McBi0iSNByZRyT1Sj1fQfUD4EPAXcDdwAcj4vsDHZgkaXgwj0jqi3qupB4AbJeZHQ3ef0lxoJEkqR7mEUm9Vs/T/Q8AG9b0bwDcOTDhSJKGIfOIpF7r9kpqRPyJou3QJOC+iLix7N8DuG5wwpMkDVXmEUkroqfb/acOWhSSpOHIPCKpz7otUjPzH4MZiCRpeDGPSFoR9Tzdv2dE3BQR8yJicUS0RcScwQhOkjT0mUck9UU9D06dARwFPASMBd5fDpMkqR7mEUm9VteX+WfmvyKiKTPbgJ9HhA3eJUl1M49I6q16itSXImIUcHtEnAJMB8YPbFiSpGHEPCKp1+q53X9MOd1HgPkU32/3poEMSpI0rJhHJPXacq+kZuZjZedC4CSAiPgd8LYBjEuSNEyYRyT1RT1XUruyV79GIUla2ZhHJPWor0WqJEmSNGB6+lnUnbsbBYwcmHCGlm023Yxzf/uLRochSZVkHunZDptvw6UXXtzoMIaleS1+De9w0FOb1NN6GHd/fwciSRp2zCOS+qynn0U9cDADkSQNL+YRSSvCNqmSJEmqHItUSZIkVY5FqiRJkipnuUVqFN4ZEV8q+zeMiN0HPjRJ0nBgHpHUF/VcSf0BxZcuH1X2zwW+P2ARSZKGG/OIpF5b7s+iAntk5s4RcRtAZs6MiFEDHJckafgwj0jqtXqupLZERBOQABGxJtA+oFFJkoYT84ikXqunSP0e8EdgrYj4KnAN8LUBjUqSNJyYRyT12nJv92fmORFxC/BKip+yOyIz7xvwyCRJw4J5RFJfLLdIjYgNgZeAP9UOy8zHBzIwSdLwYB6R1Bf1PDh1EUU7ogDGAJsADwDbDmBckqThwzwiqdfqud2/fW1/ROwMfHDAIpIkDSvmEUl90etfnMrMW4HdBiAWSdJKwDwiqR71tEn9eE3vCGBn4PkBi0iSNKyYRyT1RT1tUifWdLdStC36w8CEI0kahswjknqtxyK1/PLlCZn5qUGKR5I0jJhHJPVVt21SI6I5M9sobstIktQr5hFJK6KnK6k3UhxYbo+IC4HfA/M7RmbmBQMcmyRpaDOPSOqzetqkrga8ABzE0u+5S8CDiySpHuYRSb3WU5G6VvlE5t0sPah0yAGNSpI0HJhHJPVZT0VqEzCBZQ8qHTy4SJKWxzwiqc96KlKnZ+bJgxaJJGm4MY9I6rOefnGqqzNfSZLqZR6R1Gc9FamvHLQoJEnDkXlEUp91W6Rm5ouDGYgkaXgxj0haET1dSZUkSZIawiJVkiRJlWORKkmSpMqxSJUkSVLlWKRKkiSpcixSJUmSVDkWqZIkSaoci1RJkiRVjkWqJEmSKsciVZIkSZVjkSpJkqTKsUgdBmLhIrZ49XvZ6oB3stU+R7H2N36yZNwaPzmPrfd4K1vtcxTrnPj/ABj1+NPssP4BbDXtGLaadgzrf+KbjQp96HniSUa+8lBGbbczo3bYlabvfR+AuP0ORu09jVG77MmoPfYlbry5wYFKUv1GPPk0qx5yJKvvNI3VdzmIsd//KQCjL/gzq+9yEGuN34DmW+5YMn28MJNVDzmSNdfckokf+0KDoh4aJnzwY6y24XZM3mXakmFNd9zNpP1fy+Q9XsWkfV5D8023ATDysn8wee+DmbzrgUze+2BGXnlNg6KuhuZGB9AbEbE2cDqwG7AIeBT4X+ANmfm6hgXWYDl6FA//8QzaJ4yDlla2eO1xzHnVXoxYsIhJf7mKB676NTl6FM3Pv7hknkUbr8cDV/6qgVEPUc1NtH7ra+TOO8HcuYzafV/aX3UQzZ/9Iq3/8znaD30NIy6+hJGf/SKLL7+k0dFK6sQ80o2mJuZ+/Uu07rQ9MXceq+1zKIsP2p/WqVsx6zc/YZX/+swyk+eY0cz70qdovucBmu+9v0FBDw0Lj3krCz70Hia+//glw8Z/4cu89IWP0/KaVzLykssY/4UvM/vSC2hffTXmnH827euuTdM997PK649i5r9va2D0jTVkitSICOCPwC8z8+3lsB2B16/gcpszs3XFI2ygiKJABaKllWhphYDVf3EBz/73u8jRowBoXXO1RkY5PKyzDrnOOkX3xInk1lsRTz0NETB3bjF8zhxy3bUbF6OkLplHute+zhTa15kCQE6cQOtWW9D09DMsfuX+Xc8wfhwte+9O08OPDl6QQ1Trvnsx4rEnlh0YQcyZV3TOnkPbOkXOaNtx+yWTtE3dili0CBYtgtGjBy3eKhkyRSpwINCSmT/qGJCZt0fEZOCVEXE+sB1wC/DOzMyIeBTYNTNnRMSuwKmZOS0iTgTWBTYGZkTEg8CGwKbl39Mz83uDt2n9oK2NrV55LKMeeZIZ730zL+2yHWMefpwJ19/BOl/9ETl6NE+d9F8s2HkqUNzy3/LAd9E+YTzTP/9B5u+1Y2PjH4Li0ccYcfsdtOyxG/ntUxh12OHw6c9DezuLrr680eFJejnzSB1GPPYEI++4mzm77dToUIat+d86mVVefxTjP3cytLcz+4oLXzbNqD9eROsrtltpC1QYWm1SOw4cXdkJ+CgwleIAsU8dy9sFODwz31H2bw28BtgdOCEiRnY1U0QcFxE3R8TNs16YVX/0A62piQeu/BX33nkh4269lzH3PQytbTTNnsNDf/0ZT5/0ETZ+/xcgk5Ypa3Dv7f/Hg1eczVNf/m82+uCXGDF3fqO3YGiZN4+Rb30HLd8+BVZZhaYf/5SW077JokcfpOW0bzLyAx9udISSXq7heaQ2h7w444Vehj/wYt58Jh91HHNPOZFcZWKjwxm2xpx5NvNPOYmZ/7qF+aecxIQPf2KZ8U33PsD4L36FeWec0qAIq2EoFak9uTEzn8zMduB2ijPb5bkwMxfU9F+UmYsycwbwHDClq5ky88zM3DUzd528+uQVDLv/tU2ayLx9dmbiZTfQsu5azH7tNIjgpZ23hREjaHphFjl6FG2rTQJgwY5bs3jj9Rj9r8cbG/hQ0tLCyCPfQdtRb6P9jYcD0HT2OUu629/yJkbc1F0elFRRg5JHanPIamus3h9x95+WFia94zgWvv2NLDrisEZHM6yNPuc8Fh/xWgAWv/n1NN+8tN3piCefZpW3vZe5P/0e7Ztu3KAIq2EoFan3UJy1dmVRTXcbS5sxtLJ0G8d0mqfzpcPullF5TTNm0jS7aA8ZCxYy8aqbWLTFRsw+dH8mXF0US6P/9TixuIW21SfTNGMmtLUBMOrRpxj17ydZvPG6DYt/SMlk5Ac+TG6zFW0fW9oIPtddhxH/uBqAEZdfSW6xWaMilNQ980h3Mlnlw5+kdavNeen44xodzbDXvs4URl59PQAjr7yG9s03ASBmzWaVNx3D/JM/R+veuzcyxEoYOh8guBz4WkR8IDN/AhARuwEH9DDPoxQHpL8Abx7wCBtk5LMz2PAjXyba2qA9mXX4K5nzmn2JxS1scPxX2Grfd5Ajm3n8jC9BBBOuv634mqrmJnLECJ489dO0rTqp0ZsxJMS119P069/Qvv22jNplTwBav3wiLT86g5Ef/xS0tsLoMbT88IwGRyqpC+aRboy8/ibGnvsHWrbbmtX2OBiAeSd9hli0mImf+B9GzHiRyW9+N607bMusC88BYI2t9yTmzoXFLYz+01+Z+adzadtmy0ZuRiVNfNeHGXn1dcSMF1l1s5156X8+ybzvn8qET/0PtLaRo0cz94xvATDmR2fR9PAjjPvG6Yz7xukAzP7Tb8m11mjgFjTOkClSywbsbwROj4jPAgtZ+tUh3TkJ+FlEfB7454AH2SALt92CB684+2XDc9RIHv/RSS8bPvv1BzH79QcNRmjDTu67Nwtbu26/u/jGawc5Gkm9YR7pXsveu/PsS092OW7R4Yd2OXzG/TcMZEjDxtyzf9jl8FnXXfqyYQs++zEWfPZjAx3SkBGZ2egYhqypO26T5172i0aHMSxtPXn75U+kPhnbPP6WzNy10XFIK7tX7PyKvPTaixsdxrDUFE2NDmHYWnPsOoOWQ4ZSm1RJkiStJCxSJUmSVDkWqZIkSaoci1RJkiRVjkWqJEmSKsciVZIkSZVjkSpJkqTKsUiVJElS5VikSpIkqXIsUiVJklQ5FqmSJEmqHItUSZIkVY5FqiRJkirHIlWSJEmVY5EqSZKkyrFIlSRJUuVYpEqSJKlyLFIlSZJUORapkiRJqhyLVEmSJFWORaokSZIqxyJVkiRJlWORKkmSpMqxSJUkSVLlWKRKkiSpcixSJUmSVDkWqZIkSaoci1RJkiRVjkWqJEmSKsciVZIkSZVjkSpJkqTKsUiVJElS5VikSpIkqXIsUiVJklQ5FqmSJEmqHItUSZIkVY5FqiRJkirHIlWSJEmVE5nZ6BiGrIh4Hnis0XHUaQ1gRqODGKaG2r7dKDPXbHQQ0spuiOUQGHrHuqFkKO3bQcshFqkriYi4OTN3bXQcw5H7VtLKwGPdwHHfds3b/ZIkSaoci1RJkiRVjkXqyuPMRgcwjLlvJa0MPNYNHPdtF2yTKkmSpMrxSqokSZIqxyK1IiIiI+K0mv5PRsSJy5nniIiY2sP4d0XE3RFxT0TcGxGfLIdfGRE+RdiFiFg7In4bEQ+X++ziiDguIv7c6NgkqSfmkcYzh/Qvi9TqWAS8KSLW6MU8RwBdHlwi4lDgo8DBmbktsDMwewVj7Fh2U38sp2oiIoA/Aldm5maZORX4PDBlBZfb3B/xSdJymEcayBzS/yxSq6OVouH0xzqPiIiNIuKyiLiz/LthROwNvAH4VkTcHhGbdZrtc8AnM/NpgMxcmJk/qRl/ZETcGBEPRsR+5XqOjYgzatb754iYVnbPi4iTI+KfwF5l/1cj4o6IuCEiVuhDWBEHAi2Z+aOOAZl5O3A1MCEizo+I+yPinPJgREQ82pEQImLXiLiy7D4xIs6MiEuBs8v+s8qrD/+OiOMHe+MkDXvmkcYyh/Qzi9Rq+T5wdERM6jT8DODszNwBOAf4XmZeB1wIfCozd8zMhzvNsx1wSw/ras7M3SnOkk+oI7bxwN2ZuUdmXlP235CZrwCuAj5QxzKqrqd9thPFvpoKbArsU8fydgEOz8x3lP1bA68BdgdOiIiRKxStJL2ceaRxzCH9zCK1QjJzDnA20PkMaS/g3LL7V8C+/bC6C8q/twAb1zF9G/CHmv7FQEcbm3qXMZTdmJlPZmY7cDv1be+Fmbmgpv+izFyUmTOA51jBW0CS1Jl5pLLMIX1gkVo9pwPvozjD7E493xt2D8VZWHcWlX/bgI72Lq0s+54YU9O9MDPbavpbcun3l9UuYyjraZ8tqunubp+NYVnz61yGJPWn0zGPNII5pJ9ZpFZMZr4InEdxgOlwHfD2svto4Jqyey4wsZtFfR04JSLWBoiI0XW0YXkU2DEiRkTEBhS3FFYmlwOjI2LJLaeI2A04oId5HmXpQenNAxeaJNXHPNIw5pB+ZpFaTacBtU9nHg+8JyLuBI4B/rsc/lvgUxFxW+cG75l5MUXbpL9HxD0Ut1KWd9Z1LfAIcBdwKnDrim7IUFKe0b8ReHX59SH3ACcCT/cw20nAdyPiaoozW0mqAvPIIDOH9D9/cUqSJEmV45VUSZIkVY5FqiRJkirHIlWSJEmVY5EqSZKkyrFIlSRJUuVYpK7kIqKt/M3muyPi9xExbgWW9YuIeEvZ/dOImNrDtNPK343u7TqW/M5xPcO7WcYyvy29IuuVpJWdeWTF1qvuWaRqQfmbzdtR/ETdh2pHRkRTXxaame/PzHt7mGQa0OuDiySpcswjGhAWqap1NbB5eXZ6RUScC9wVEU0R8a2IuCki7oyIDwJE4YyIuDciLgLW6lhQRFwZEbuW3YdExK0RcUdEXBYRG1McxD5Wnn3vFxFrRsQfynXcFBH7lPOuHhGXll80/WMg6t2YiNg9Iq4r570uIraqGb1BRFwSEQ9ExAk187wzIm4s4/pxXw+ukrSSMo+YR/rNsP/dV9UnIpqBQ4FLykG7A9tl5iMRcRwwOzN3i4jRwLURcSmwE7AVsD0wBbgXOKvTctcEfgLsXy5rtcx8MSJ+BMzLzFPL6c4FvpOZ10TEhsBfgW2AE4BrMvPkiHgtcFwvNuv+cr2tEfEq4Gss/dm53YHtgJeAm8qD43zgbcA+mdkSET+g+PnAs3uxTklaKZlHzCP9zSJVYyPi9rL7auBnFLdPbszMR8rhBwM7RNlOCJgEbAHsD/wmM9uApyPi8i6WvydwVceyyt+U7sqrgKkRS05wV4mIieU63lTOe1FEzOzFtk0CfhkRWwAJjKwZ97fMfAEgIi4A9gVaKX5D+aYyjrHAc71YnyStjMwj5pEBYZGqBZm5Y+2A8oM1v3YQ8F+Z+ddO0x1G8aHtSdQxDRRNT/bKzAVdxNLX3+79MnBFZr6xvDV0Zc24zsvMMtZfZubn+rg+SVoZmUeW9ptH+pFtUlWPvwIfjoiRABGxZUSMB64C3l62NVoHOLCLea8HDoiITcp5VyuHzwUm1kx3KfCRjp6I2LHsvIriVgkRcSiwai/ingQ8VXYf22ncqyNitYgYCxwBXAtcBrwlItbqiDUiNurF+iRJXTOPqNcsUlWPn1K0E7o1Iu4GfkxxFf6PwEPAXcAPgX90njEzn6do/3NBRNwB/K4c9SfgjR0N3oHjgV2jaFB/L0ufDj0J2D8ibqW4XfR4D3HeGRFPlq9vA6cAX4+Ia4HODdevAX4F3A78ITNvLp8i/SJwaUTcCfwNWKe+XSRJ6oF5RL0WmX29Ai5JkiQNDK+kSpIkqXIsUiVJklQ5FqmSJEmqHItUSZIkVY5FqiRJkirHIlWSJEmVY5EqSZKkyrFIlSRJUuVYpEqSJKlyLFIlSZJUORapkiRJqhyLVEmSJFWORaokSZIqxyJVkiRJlWORKkmSpMqxSJUkSVLlWKRKkiSpcixSJUmSVDkWqZIkSaoci1RJkiRVjkWqJEmSKsciVZIkSZVjkSpJkqTKsUiVJElS5VikSpIkqXIsUiVJklQ5FqmSJEmqHItUSZIkVY5FqiRJkirHIlWSJEmVY5EqSZKkyrFIlSRJUuVYpEqSJKlyLFIlSZJUORapkiRJqhyLVEmSJFWORaokSZIqxyJVkiRJlWORKkmSpMqxSJUkSVLlWKRKkiSpcixSh5mIGBsRf4qI2RHx+xVYztERcWl/xtYIEfGXiHh3H+f9SkTMiIhn+imWjIjN+2NZktTfzB/LqlL+WFlZpDZIRLwjIm6OiHkRMb38MOzbD4t+CzAFWD0zj+zrQjLznMw8uB/iWUZETCuLtQs6DX9FOfzKOpdzYkT8ennTZeahmfnLPsS5AfAJYGpmrt3b+SVpoJg/zB8rC4vUBoiIjwOnA1+jOCBsCPwAOLwfFr8R8GBmtvbDsgbK88DeEbF6zbB3Aw/21wqisCLv742AFzLzuT6su3kF1itJ3TJ/mD9WKpnpaxBfwCRgHnBkD9OMpjgIPV2+TgdGl+OmAU9SnKU9B0wH3lOOOwlYDLSU63gfcCLw65plbwwk0Fz2Hwv8G5gLPAIcXTP8mpr59gZuAmaXf/euGXcl8GXg2nI5lwJrdLNtHfH/CPjPclhTOexLwJU1034XeAKYA9wC7FcOP6TTdt5RE8dXyzgWAJuXw95fjv8hcH7N8r8JXAZEpxhfVc7fXi7/F+XwNwD3ALPK5W5TM8+jwGeAO4FFHfu303IT2Lzs3rfctgNrxn0IeAiYCXy/I66O/wVwajnuEeDQRr+XffnyNbgvzB8d8a9U+QMI4Dvl/2x2Od12wJ7AM0BTzbRvBO4su08Efg/8uty3dwFbAp8rl/UEcHCj39c9vucbHcDK9io/IK2d34SdpjkZuAFYC1gTuA74cjluWjn/ycBI4DDgJWDVcvyJLHtQ6dy/MeVBBhhffoC3KsetA2xbdh9LeZABVqMojo4p5zuq7F+9HH8l8HD55h9b9n+jm22bRnFA2Rv4ZznsMOCvwPtZ9iDzTmD1cp2fKD+MY7rarpo4Hge2LecZybIHmXEUZ9vHAvsBM4D1e4qzpn9LYD7w6nK5nwb+BYwqxz8K3A5sAIztZplJceB7TXlw2L3TuD8DkymujDwPHFLzv2gBPkBxQP4wRfKJrtbjy5ev4fnC/DGNlTB/UOSMWyjyQwDbAOuU4x4GXl0z7e+Bz9Zs58Jy/mbgbIqTiS+UcXwAeKTR7+ueXt7uH3yrAzOy59spRwMnZ+Zzmfk8xRnuMTXjW8rxLZl5McXZ2lZ9jKcd2C4ixmbm9My8p4tpXgs8lJm/yszWzPwNcD/w+pppfp6ZD2bmAuA8YMeeVpqZ1wGrRcRWwLsoPjydp/l1Zr5QrvM0iisEy9vOX2TmPeU8LZ2W9xLFgevbFGeW/5WZTy5neR3eBlyUmX8rl3sqxQF175ppvpeZT5T7oDtHAmcCh2XmjZ3GfSMzZ2Xm48AVLLsPH8vMn2RmG/BLioQwpc7YJQ0P5g9WyvzRAkwEtqa4OHFfZk4vx/2GovAnIiZSFO2/qZn36sz8a/me+T3Fics3yjh+C2wcEZPr3I5BZ5E6+F4A1lhOu5N1gcdq+h8rhy1ZRqeD1EvAhN4GkpnzKT48HwKmR8RFEbF1HfF0xLReTX/tE4z1xvMr4CPAgcAfO4+MiE9ExH3lk6azKG51rbGcZT7R08iyMPw3xdnoeXXE2GGZfZCZ7eW6avdBj+sufRQ4LzPv6mJcT/twybjyYAl9+J9LGtLMH0utNPkjMy8HzqBoBvZsRJwZEauUo88F3hQRo4E3AbdmZu3+framewHFSU5bTT9UOJdYpA6+6ykuvx/RwzRPUzS87rBhOawv5lPcpuiwzJOG5RnWqymuzN0P/KSOeDpieqqPMXX4FfAfwMU1hRcAEbEfRRudt1LcippM0RYnOkLvZpndDe9Y7n9SnFE/TXHLpV7L7IOICIpbM7X7oMd1l44EjoiIj/Zi3ZIE5o9aK1X+yMzvZeYuFM0RtgQ+VQ6/l6IAPhR4B0XROmxYpA6yzJxN0cD7+xFxRESMi4iREXFoRJxSTvYb4IsRsWZErFFOv9yvy+jG7cD+EbFhREyiaDANQERMiYg3RMR4isba84C2LpZxMbBl+bUnzRHxNmAqRRvKPsvMR4ADKNrHdDaRou3U80BzRHwJWKVm/LMUtynqfg9HxJbAVyhu2RwDfDoidqxz9vOA10bEKyNiJEUbp0UU7b1642nglcDxEfEfvZxX0krM/LHUypQ/ImK3iNijnHc+xYlK7b4+Fzge2J/ilv6wYZHaAJn5beDjwBcpPkRPUNy2+N9ykq8AN1M8wXcXcGs5rC/r+hvwu3JZt7DsgWEExYflaeBFig/8ywqnzHwBeF057QsUZ5Cvy8wZfYmp07KvycyuzvL/CvyFoqH6YxQfytrbIR0fxBci4tblrae8PfZr4JuZeUdmPgR8HvhVeZtkeXE+QHFw+n8UDeZfD7w+Mxcvb94ulvU4RaH6mYh4f2/nl7TyMn8ss+yVJX+sQnGVema5PS9QtGvt8BuKh7Uu74/9WiUdX3EjSZIkVYZXUiVJklQ5FqmSJEmqHItUSZIkVY5FqiRJkirHIlWSJEmV09OvVmg5YtSIZIy7cCDsuMXURocwbN1+6x0zMnPNRschrezMIQNn5y23a3QIw9att9w2aDnET8eKGNMMe6zV6CiGpSsuuqzRIQxbq45eo/NPFEpqBHPIgLn2kmsaHcKwNbZ5/KDlEG/3S5IkqXIsUiVJklQ5FqmSJEmqHItUSZIkVY5FqiRJkirHIlWSJEmVY5EqSZKkyrFIlSRJUuVYpEqSJKlyLFIlSZJUORapkiRJqhyLVEmSJFWORaokSZIqxyJVkiRJlWORKkmSpMqxSJUkSVLlWKRKkiSpcixSJUmSVDkWqZIkSaoci1RJkiRVjkWqJEmSKsciVZIkSZVjkSpJkqTKsUiVJElS5VikSpIkqXIsUiVJklQ5FqmSJEmqHItUSZIkVY5FqiRJkirHIlWSJEmVY5EqSZKkyrFIlSRJUuVYpEqSJKlyLFIlSZJUORapkiRJqhyLVEmSJFWORaokSZIqxyJ1CPvZPTN59h/Tuev6Z1827hOPzSX//hSrL24DYGR7ctY9M7nz+me5/YZnOeDFRUum/cq/ZvP41c8w94qnBy32oWTsccezyvpbM3GnfZcMG/mH/2PijvswacyaNN1y2zLTj7jrHibsfwgTd9yHiTvvBwsXDnbIkrRc3eWQjzw+j/uve5a7r3+Wbz40e8nwzz4yl4eufYb7r3uWg18ojmsTWtu57Ybnlrye/8d0vvPArMHcjCFtxCWXMmrqjozaanuavnlqo8OpnAErUiMiI+K0mv5PRsSJy5nniIiY2sP4d0XE3RFxT0TcGxGfLIdfGRG79lvwQ8Qv1h3HITut/rLh6y9s5dUvLOKxMU1Lhn3gqfkA7LDXFF698xqc9tBsIhOAP605lt13X3Nwgh6CFh/zdub/6XfLDGubug3zf/cL2vbba9mJW1sZf+yHeemMU5l7+7XM+9v/wciRgxitNHyYRwZWVzlk2ouLOHzGQnbYcy2222sKp240AYBt5rXw9mdfYtu9pnDITqvzg/tnMSKTec0j2GnPtZa8HhvTxAVrjW3E5gw9bW00H/9xWv78RxbfdQtNv/s9ce99jY6qUgbySuoi4E0RsUYv5jkC6PLgEhGHAh8FDs7MbYGdgdldTdtbEdG0/Kmq5+pVR/PiyJf/C7/z4Gw+vcUksmbY1PmtXLbaaACeH9XErOYR7DqnBYB/ThrFM6OH5C4YFG377U2uuuoyw9q32ZL2rbZ42bTNf7uCtu2n0r7DdgDk6qtBk/tW6iPzyADqKod8+Mn5fGOjCSweEUCRLwAOf34hv50yjsUjgkfHNvOvsc3sPnvxMvNu/lIray1u5+rJowZnA4a4uPFmcrNNyU03gVGjaHvrWxhx4Z8bHValDGSR2gqcCXys84iI2CgiLouIO8u/G0bE3sAbgG9FxO0RsVmn2T4HfDIznwbIzIWZ+ZOa8UdGxI0R8WBE7Feu59iIOKNmvX+OiGll97yIODki/gnsVfZ/NSLuiIgbImJK/+2KwfP65xfw1Ogm7py47NW7OyaM5PDnF9DUnmy8oJVd5i5mg4VtDYpy+Gp66GGIYPxrj2TCHgcy+tTvNTokaSgzjwyyLV9qZb9Zi7nhxue48ubn2bUsRNdb1MYTNXfnnhzTxHqL2peZ96hnXuJ3U8ZCxKDGPFTF00+TG6y/pD/XX494enoDI6qegW6T+n3g6IiY1Gn4GcDZmbkDcA7wvcy8DrgQ+FRm7piZD3eaZzvglh7W1ZyZu1OcJZ9QR2zjgbszc4/MvKbsvyEzXwFcBXygq5ki4riIuDkibqalvatJGmZsWztfeGQuX9pslZeNO2vdcTw5uombb3ye0x+YzXWTRtHqcaT/tbbSdO0/eemXP2LeFRcx8sKLab78qkZHJQ1lwyqPVDmHADRnsmprO3vutiaf2mIS5931ImTSVbrITv1vf3YBv1nbW/11y857EAv8Tga0SM3MOcDZwPGdRu0FnFt2/wrYlxV3Qfn3FmDjOqZvA/5Q078Y6LjO3u0yMvPMzNw1M3eli1vtjbTZgjY2WdDGHTc8xyPXPMP6i9q49Z/PM2VRG20jgo9vNZmd9lyLI3ZcncmtyUPjmhsd8rDTvv66tO2/N7nG6jBuHC2HvIqm2+5odFjSkDXc8kiVcwgUV0gvWLO4GnrTpFG0B6zR0s6To5uWufu2/sI2nh69NP4d5rbQnMmtq3irv1653nrEE08u6Y8nnyLXWbuBEVXPYHxCTgfeR3GG2Z0uTide5h5glx7Gdzyu3gZ0VF+tLLuNY2q6F2Zm7f3ulswlpzW1yxgy7p4wkikHrMMm+67NJvuuzZOjm9h5jzV5dnQTY9vaGddWnLW/6oWFtAbcN8EHevpb66sPYsRd98BLL0FrK81XXUfbNls1OixpqDsd88ig+N81x3LQzGI3bDG/hVHtMGPkCC5ccwxvf/YlRpVNxrZY0MqNk5YWpEc98xK/mTKuUWEPSbnbLsS/HiYeeRQWL6bpvPNpf/1rGx1WpQx4kZqZLwLnURxgOlwHvL3sPhq4puyeC0zsZlFfB06JiLUBImJ0RHQ+s+7sUWDHiBgRERsAu/d+C6rr3Lte5Pqbnmerl1p54urpvLd8gr8ray1u59Z/Ps+91z3LZx6bxzHbLn0Q6JsPzeaJq6czri154urpnPDwnMEIf8gYd8wHmHDAIYx48F+ssun2jPr5rxn5fxexyqbb03TDzYw/4h2Mf+2RAOSqk1n03x9m4t6vZuJu02jbaQdaDzu4wVsgDW3mkYHRVQ45a91xbLqglbuuf5bf3j2Td2+7KkRw74SRnDdlHPde/yyX3PYC/7nVZNprbk2/9Tlv9fdaczOt3z2NkYcdzqjtdqbtLW8mt+32iylWSpFdtYnojwVHzMvMCWX3FOAR4JTMPDEiNgbOAtYAngfek5mPR8Q+wE8ozmbf0rk9UUS8B/gEEBRnzWdl5rcj4kqKxvA3l0+B3pyZG0dEAL8GdgTuBqYAJ2bmlbXxdRHvW4DXZeaxPW7jKqOSPdZagb2k7sy8yFvkA2XV0Wvckpkr1VftaGga7nnEHDJwFlzyYKNDGLbGNo8ftBwyYEXqysADzMCxSB04FqlSNZhDBo5F6sAZzCK1eq22JUmStNKzSJUkSVLlWKRKkiSpcixSJUmSVDkWqZIkSaoci1RJkiRVjkWqJEmSKsciVZIkSZVjkSpJkqTKsUiVJElS5VikSpIkqXIsUiVJklQ5FqmSJEmqHItUSZIkVU5zdyMiYueeZszMW/s/HEnScGEekbQiui1SgdN6GJfAQf0ciyRpeDGPSOqzbovUzDxwMAORJA0v5hFJK2K5bVIjYlxEfDEiziz7t4iI1w18aJKk4cA8Iqkv6nlw6ufAYmDvsv9J4CsDFpEkabgxj0jqtXqK1M0y8xSgBSAzFwAxoFFJkoYT84ikXqunSF0cEWMpGrkTEZsBiwY0KknScGIekdRrPT3d3+EE4BJgg4g4B9gHOHYgg5IkDSvmEUm9ttwiNTP/FhG3AntS3J7578ycMeCRSZKGBfOIpL6o50oqwAHAvhS3akYCfxywiCRJw5F5RFKv1PMVVD8APgTcBdwNfDAivj/QgUmShgfziKS+qOdK6gHAdpnZ0eD9lxQHGkmS6mEekdRr9Tzd/wCwYU3/BsCdAxOOJGkYMo9I6rVur6RGxJ8o2g5NAu6LiBvL/j2A6wYnPEnSUGUekbQierrdf+qgRSFJGo7MI5L6rNsiNTP/MZiBSJKGF/OIpBVRz9P9e0bETRExLyIWR0RbRMwZjOAkSUOfeURSX9Tz4NQZwFHAQ8BY4P3lMEmS6mEekdRrdX2Zf2b+KyKaMrMN+HlE2OBdklQ384ik3qqnSH0pIkYBt0fEKcB0YPzAhiVJGkbMI5J6rZ7b/ceU030EmE/x/XZvGsigJEnDinlEUq8t90pqZj5Wdi4ETgKIiN8BbxvAuCRJw4R5RFJf1HMltSt79WsUkqSVjXlEUo/6WqRKkiRJA6ann0XdubtRwMiBCWdo2WbTzTjnNz9vdBjD0pimsY0OQdIKMo/0bOtNN+UX55zZ6DCkyuqpTeppPYy7v78DkSQNO+YRSX3W08+iHjiYgUiShhfziKQVYZtUSZIkVY5FqiRJkirHIlWSJEmVs9wiNQrvjIgvlf0bRsTuAx+aJGk4MI9I6ot6rqT+gOJLl48q++cC3x+wiCRJw415RFKvLfdnUYE9MnPniLgNIDNnRsSoAY5LkjR8mEck9Vo9V1JbIqIJSICIWBNoH9CoJEnDiXlEUq/VU6R+D/gjsFZEfBW4BvjagEYlSRpOzCOSem25t/sz85yIuAV4JcVP2R2RmfcNeGSSpGHBPCKpL5ZbpEbEhsBLwJ9qh2Xm4wMZmCRpeDCPSOqLeh6cuoiiHVEAY4BNgAeAbQcwLknS8GEekdRr9dzu3762PyJ2Bj44YBFJkoYV84ikvuj1L05l5q3AbgMQiyRpJWAekVSPetqkfrymdwSwM/D8gEUkSRpWzCOS+qKeNqkTa7pbKdoW/WFgwpEkDUPmEUm91mORWn758oTM/NQgxSNJGkbMI5L6qts2qRHRnJltFLdlJEnqFfOIpBXR05XUGykOLLdHxIXA74H5HSMz84IBjk2SNLSZRyT1WT1tUlcDXgAOYun33CXgwUWSVA/ziKRe66lIXat8IvNulh5UOuSARiVJGg7MI5L6rKcitQmYwLIHlQ4eXCRJy2MekdRnPRWp0zPz5EGLRJI03JhHJPVZT7841dWZryRJ9TKPSOqznorUVw5aFJKk4cg8IqnPui1SM/PFwQxEkjS8mEckrYierqRKkiRJDWGRKkmSpMqxSJUkSVLlWKRKkiSpcixSJUmSVDkWqZIkSaoci1RJkiRVjkWqJEmSKsciVZIkSZVjkSpJkqTKsUiVJElS5TQ3OgCtuFi4iC3e8GFGLG6B1jZmvf5AnvnMB1j7lJ+y+q/+j9bVVwVg+hc+xJxX7w3AmHv+xYaf/CYj5s6HEcEDl55FjhndyM0YGhYuZNS0g2HxImhto/1NR9B64hcZedS7iAcfBCBmzSYnT2LxLTc0OFhJqk8sXMTUw48nFrcQbW28+LoDeOrT72XcPf9i40+dRtP8BSzaYG0e/uH/0DZxPONvvY9NPnlqMXMmT33qWGYetn9jN2IIGnHJpTR//NPQ1kbbe99N22c+2eiQKmVIFakRsTZwOrAbsAh4FPhf4A2Z+bqGBdZgOXoU/7rgDNonjIOWVrZ83QeZ88q9AHj+Q2/nuf88etkZWlvZ+D9O5LHvn8CC7bag6cXZ5Mgh9VZonNGjWfz3i2HCBGhpYdT+ryIOOZiW35y9ZJLmT36WnDSpgUFK6o55pGs5ehT3XfAd2sePI1pamfr6jzD7oD3Y6Avf5fET/oO5e+/ImudexDrf/y1PfvZ9LNh6E+6+9MfQ3MzIZ19g+wPfy8yD94Zmc0nd2tpoPv7jtFzyJ3L99Ri15360v/615NRtGh1ZZQyZ2/0REcAfgSszc7PMnAp8Hpiygssd+p+oiKJABaKllWhphYhuJ1/lihtZMHVzFmy3BQBtq02CpqZBCXXIiygKVICWFmhtWXZfZ9J0/gW0v/3IxsQnqVvmkR5E0D6+Jo+0tpIRjP3XE8zd6xUAzD5gN1a76B8AtI8bs6QgHbFwcY85R12LG28mN9uU3HQTGDWKtre+hREX/rnRYVXKkClSgQOBlsz8UceAzLwduBqYEBHnR8T9EXFOeSAiIh6NiDXK7l0j4sqy+8SIODMiLgXOLvvPiogrI+LfEXH8YG/cCmtrY6tp72L7bQ5j7rTdeWmXbQFY42fns/UB72TD479C06w5AIx++HGIYLMjP8pWB72btf7frxsZ+dDT1saoXfZk9Dob0/7Kg8g9dlsyKq6+lpyyFrnF5g0MUFI3zCM9aWtju4Pex87bHsHsA3Zl/i5TeWnrTVj1kmsBWO1PVzDqqeeWTD7+lnvZfv93s/209/DItz7uVdReiqefJjdYf0l/rr8e8fT0BkZUPUOpSN0OuKWbcTsBHwWmApsC+9SxvF2AwzPzHWX/1sBrgN2BEyJiZFczRcRxEXFzRNw884VZ9Uc/0JqaeODKs7nnzv9j3K33Mua+h5lx7Ju496bzuf+Ks2mZsgbrfel7AERbG+P/eQeP/uhEHvzzj5l88T+YcNVNDd6AIaSpicW33MCixx4kbrqFuPuepaN+93va3uZVVKmiGp5HanPIrCrlEICmJu6+/GfcdvvvmXDrfYy979/8+/TPMOXnf2S7V3+ApnkLaB+1dJPm7zKVu676JXf/9Ues+91ziIWLGhj8EJT58mFekV7GUCpSe3JjZj6Zme3A7cDGdcxzYWYuqOm/KDMXZeYM4Dm6uf2TmWdm5q6Zueuqq09ewbD7X9ukiczbZ2dWufwGWtdarbiNP2IELxxzOONuuw+Axeuuxby9dqJt9cnkuDHMftVejLvzgQZHPgRNnkz7Afsx4q9/K/pbW2n64//R9ta3NDYuSX0xKHmkNodMrmAOgSKPzNlnJyZdcSMLt9iI+887jbv/9hNeeOMrWbTRui+bfuGWG9M+bgzj7n+kAdEOXbneesQTTy7pjyefItdZu4ERVc9QKlLvoThr7Urt6VsbSx8Ia2XpNo7pNM/8OpdRec0zZtI0ey4AsWAhE/9xEwu32IjmZ2YsmWbSxVeycOtNAZh74B6MvfdfxEsLobWVidfdxsItN2lI7EPO88/DrFlF94IFNF12BbnVVgCM+PvlRff66zUuPkk9MY90o3nGrJo8sohVrrqZhZtvSPPzM4sJ2ttZ9ztn89y73wDA6MemQ2srAKOeeIYxDz/Bog0ssHojd9uF+NfDxCOPwuLFNJ13Pu2vf22jw6qUIfMBAi4HvhYRH8jMnwBExG7AAT3M8yjFAekvwJsHPMIGaX72BTb6yMlEezu0J7MOP4g5B+/LRv9xEmPvfhAiWLzBOjx+6mcAaJu8Cs99+Ci2Ovi9EMGcV+3FnIPrubOlmP4MI997HLS1QXs7bW95M+2vOxSApvPOp80HpqQqM490Y+SzL7DZ8V8j2oo88uLh05h18N5MOfN8pvz8jwDMPGx/nj/qMAAm3ngnW/6/c8nmZhgRPPqNj9Fa0SvDldXcTOt3T2PkYYcXX0F17LvIbac2OqpKGTJFamZmRLwROD0iPgssZOlXh3TnJOBnEfF54J8DHmSDLNx2cx644uyXDX/sByd0O8/MIw9h5pGHDGRYw1LusD2Lb76+y3EtZ505yNFI6g3zSPcWbLsZd1/2s5cNf/a4t/DscS9vwjTjyNcw48jXDEZow1r7YYew+DBzcXciu2q4q7pM3XGbPOfvP290GMPSNqvu0OgQhq2xzeNvycxdGx2HtLLbZset8xeXenI7EF6xuoe4gTKYOWQotUmVJEnSSsIiVZIkSZVjkSpJkqTKsUiVJElS5VikSpIkqXIsUiVJklQ5FqmSJEmqHItUSZIkVY5FqiRJkirHIlWSJEmVY5EqSZKkyrFIlSRJUuVYpEqSJKlyLFIlSZJUORapkiRJqhyLVEmSJFWORaokSZIqxyJVkiRJlWORKkmSpMqxSJUkSVLlWKRKkiSpcixSJUmSVDkWqZIkSaoci1RJkiRVjkWqJEmSKsciVZIkSZVjkSpJkqTKsUiVJElS5VikSpIkqXIsUiVJklQ5FqmSJEmqHItUSZIkVY5FqiRJkirHIlWSJEmVY5EqSZKkyrFIlSRJUuVYpEqSJKlyLFIlSZJUOZGZjY5hyIqI54HHGh1HndYAZjQ6iGFqqO3bjTJzzUYHIa3shlgOgaF3rBtKhtK+HbQcYpG6koiImzNz10bHMRy5byWtDDzWDRz3bde83S9JkqTKsUiVJElS5VikrjzObHQAw5j7VtLKwGPdwHHfdsE2qZIkSaocr6RKkiSpcixSKyIiMiJOq+n/ZEScuJx5joiIqT2Mf1dE3B0R90TEvRHxyXL4lRHhU4RdiIi1I+K3EfFwuc8ujojjIuLPjY5NknpiHmk8c0j/skitjkXAmyJijV7McwTQ5cElIg4FPgocnJnbAjsDs1cwxo5lN/XHcqomIgL4I3BlZm6WmVOBzwNTVnC5zf0RnyQth3mkgcwh/c8itTpaKRpOf6zziIjYKCIui4g7y78bRsTewBuAb0XE7RGxWafZPgd8MjOfBsjMhZn5k5rxR0bEjRHxYETsV67n2Ig4o2a9f46IaWX3vIg4OSL+CexV9n81Iu6IiBsiYoU+hBVxINCSmT/qGJCZtwNXAxMi4vyIuD8izikPRkTEox0JISJ2jYgry+4TI+LMiLgUOLvsP6u8+vDviDh+sDdO0rBnHmksc0g/s0itlu8DR0fEpE7DzwDOzswdgHOA72XmdcCFwKcyc8fMfLjTPNsBt/SwrubM3J3iLPmEOmIbD9ydmXtk5jVl/w2Z+QrgKuADdSyj6nraZztR7KupwKbAPnUsbxfg8Mx8R9m/NfAaYHfghIgYuULRStLLmUcaxxzSzyxSKyQz5wBnA53PkPYCzi27fwXs2w+ru6D8ewuwcR3TtwF/qOlfDHS0sal3GUPZjZn5ZGa2A7dT3/ZemJkLavovysxFmTkDeI4VvAUkSZ2ZRyrLHNIHFqnVczrwPoozzO7U871h91CchXVnUfm3Deho79LKsu+JMTXdCzOzraa/JZd+f1ntMoaynvbZopru7vbZGJY1v85lSFJ/Oh3zSCOYQ/qZRWrFZOaLwHkUB5gO1wFvL7uPBq4pu+cCE7tZ1NeBUyJibYCIGF1HG5ZHgR0jYkREbEBxS2FlcjkwOiKW3HKKiN2AA3qY51GWHpTePHChSVJ9zCMNYw7pZxap1XQaUPt05vHAeyLiTuAY4L/L4b8FPhURt3Vu8J6ZF1O0Tfp7RNxDcStleWdd1wKPAHcBpwK3ruiGDCXlGf0bgVeXXx9yD3Ai8HQPs50EfDcirqY4s5WkKjCPDDJzSP/zF6ckSZJUOV5JlSRJUuVYpEqSJKlyLFIlSZJUORapkiRJqhyLVEmSJFWORepKLiLayt9svjsifh8R41ZgWb+IiLeU3T+NiKk9TDut/N3o3q5jye8c1zO8m2Us89vSK7JeSVrZmUdWbL3qnkWqFpS/2bwdxU/Ufah2ZEQ09WWhmfn+zLy3h0mmAb0+uEiSKsc8ogFhkapaVwObl2enV0TEucBdEdEUEd+KiJsi4s6I+CBAFM6IiHsj4iJgrY4FRcSVEbFr2X1IRNwaEXdExGURsTHFQexj5dn3fhGxZkT8oVzHTRGxTznv6hFxaflF0z8Got6NiYjdI+K6ct7rImKrmtEbRMQlEfFARJxQM887I+LGMq4f9/XgKkkrKfOIeaTfDPvffVV9IqIZOBS4pBy0O7BdZj4SEccBszNzt4gYDVwbEZcCOwFbAdsDU4B7gbM6LXdN4CfA/uWyVsvMFyPiR8C8zDy1nO5c4DuZeU1EbAj8FdgGOAG4JjNPjojXAsf1YrPuL9fbGhGvAr7G0p+d2x3YDngJuKk8OM4H3gbsk5ktEfEDip8PPLsX65SklZJ5xDzS3yxSNTYibi+7rwZ+RnH75MbMfKQcfjCwQ5TthIBJwBbA/sBvMrMNeDoiLu9i+XsCV3Usq/xN6a68CpgaseQEd5WImFiu403lvBdFxMxebNsk4JcRsQWQwMiacX/LzBcAIuICYF+gleI3lG8q4xgLPNeL9UnSysg8Yh4ZEBapWpCZO9YOKD9Y82sHAf+VmX/tNN1hFB/ankQd00DR9GSvzFzQRSx9/e3eLwNXZOYby1tDV9aM67zMLGP9ZWZ+ro/rk6SVkXlkab95pB/ZJlX1+Cvw4YgYCRARW0bEeOAq4O1lW6N1gAO7mPd64ICI2KScd7Vy+FxgYs10lwIf6eiJiB3LzqsobpUQEYcCq/Yi7knAU2X3sZ3GvToiVouIscARwLXAZcBbImKtjlgjYqNerE+S1DXziHrNIlX1+ClFO6FbI+Ju4McUV+H/CDwE3AX8EPhH5xkz83mK9j8XRMQdwO/KUX8C3tjR4B04Htg1igb197L06dCTgP0j4laK20WP9xDnnRHxZPn6NnAK8PWIuBbo3HD9GuBXwO3AHzLz5vIp0i8Cl0bEncDfgHXq20WSpB6YR9RrkdnXK+CSJEnSwPBKqiRJkirHIlWSJEmVY5EqSZKkyrFIlSRJUuVYpEqSJKlyLFIlSZJUORapkiRJqhyLVEmSJFXO/wdz+CzTp67djQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 864x576 with 4 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#plot 2 graphs per row\n",
    "# Calculate number of rows and columns for subplots\n",
    "num_models = len(y_pred_per_model)\n",
    "num_rows = num_models // 2 + num_models % 2\n",
    "num_cols = 2\n",
    "\n",
    "# Create subplots with num_rows rows and num_cols columns\n",
    "fig, axes = plt.subplots(num_rows, num_cols, figsize=(12, 4*num_rows))\n",
    "\n",
    "# Iterate over models and plot confusion matrix\n",
    "for i, (model_name, y_pred) in enumerate(y_pred_per_model.items()):\n",
    "    row_idx = i // num_cols\n",
    "    col_idx = i % num_cols\n",
    "    cm = confusion_matrix(list(test_labels), y_pred, list(set(test_labels)))\n",
    "    plot_cm(cm, model_name, [\"Not Churn\", \"Churn\"], axes[row_idx, col_idx])\n",
    "\n",
    "# Adjust spacing between subplots\n",
    "plt.subplots_adjust(wspace=0.5, hspace=0.5)\n",
    "plt.show()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above plots are showing confusion matrix for different models. Using these plots we can deduce various scores against each model and then we can make comparision as well. \n",
    "\n",
    "**Note** that in this notebook, I am illustrating how can we implement several performance metrics for classification problems and how can we interpret the results. So, we are not concerned by how well a model is performing and about hyperparameters tuning."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Classification Report\n",
    "\n",
    "For each model, we can display a report kind of structure which consists of `class` and its `accuracy score`\n",
    ", `precision score`, `recall score`, `F-1 score` and `support`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 248,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification Report For logistic_regressor \n",
      "\n",
      "Class                Accuracy        Precision       Recall          F1-Score        Support        \n",
      "Not Churn            0.7865          0.8117          0.9788          0.8874          1607.0         \n",
      "Churn                0.0140          0.4516          0.0712          0.1231          393.0          \n",
      "\n",
      "\n",
      "Classification Report For random_forest \n",
      "\n",
      "Class                Accuracy        Precision       Recall          F1-Score        Support        \n",
      "Not Churn            0.7735          0.8800          0.9627          0.9195          1607.0         \n",
      "Churn                0.0910          0.7521          0.4631          0.5732          393.0          \n",
      "\n",
      "\n",
      "Classification Report For knn \n",
      "\n",
      "Class                Accuracy        Precision       Recall          F1-Score        Support        \n",
      "Not Churn            0.7455          0.8073          0.9278          0.8633          1607.0         \n",
      "Churn                0.0185          0.2418          0.0941          0.1355          393.0          \n",
      "\n",
      "\n",
      "Classification Report For svm \n",
      "\n",
      "Class                Accuracy        Precision       Recall          F1-Score        Support        \n",
      "Not Churn            0.8035          0.8035          1.0000          0.8910          1607.0         \n",
      "Churn                0.0000          0.0000          0.0000          0.0000          393.0          \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "\n",
    "#iterate over y_pred_per_model dict and each model let's build this classification report\n",
    "\n",
    "y_true = np.array(test_labels)\n",
    "labels_encoded = {0:'Not Churn', 1:'Churn'}\n",
    "\n",
    "for model_name, y_pred in y_pred_per_model.items():\n",
    "\n",
    "    print(f\"Classification Report For {model_name} \\n\")\n",
    "\n",
    "    cm = confusion_matrix(list(test_labels), y_pred, list(set(test_labels)))\n",
    "\n",
    "    #get number of distinct labels\n",
    "    n_labels = cm.shape[0]\n",
    "\n",
    "    #using confusion matrix get following score\n",
    "    precision = precision_score(cm)\n",
    "    recall = recall_score(cm)\n",
    "    accuracy = accuracy_score(cm)\n",
    "    f_1 = f1_score(cm)\n",
    "    support = np.sum(cm, axis=1)\n",
    "    \n",
    "    #print the values\n",
    "    print(\"{:<20} {:<15} {:<15} {:<15} {:<15} {:<15}\".format(\"Class\", \"Accuracy\", \"Precision\", \"Recall\", \"F1-Score\", \"Support\"))\n",
    "\n",
    "    #loop over each label and print the corresponding score\n",
    "    for label in range(n_labels):\n",
    "        print(\"{:<20} {:<15.4f} {:<15.4f} {:<15.4f} {:<15.4f} {:<15}\".format(labels_encoded[label], accuracy[label], precision[label], recall[label], f_1[label], support[label]))\n",
    "    \n",
    "    print(\"\\n\")\n",
    "\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given that the dataset has imbalanced distributions with `393` positive classes and `1607` negative classes, relying solely on accuracy score to evaluate model performance is inadequate. Other metrics, such as the `F-1` score, which considers both precision and recall, are more appropriate for performance measurement.\n",
    "\n",
    "From the Classification Report, it is evident that the Random Forest algorithm outperforms the other algorithms in terms of `F-1` score. \n",
    "\n",
    "**Note** that the choice of metrics depends on the specific problem and its business impact. What may be the best metric for one problem may not necessarily be suitable for another. Therefore, careful consideration of the problem and its implications is crucial in selecting the appropriate evaluation metric for the given scenario.\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plotting ROC Curve\n",
    "\n",
    "As we discussed earlier ROC curve is a probabilistic curve and it can be used when you have a predictive model which ouput probabilities against each label instead of the value of actual label.\n",
    "\n",
    "Among selected algorithms we have right now `Logistic Regressor` can be used to produce associated probabilities against each class. In `sklearn` we have `predict_proba` method we can use it to get assigned probabilities to each label.\n",
    "\n",
    "For getting `True Positive Rate(TPR)` and `False Positive Rate(FPR)` we have already defined `get_tpr_fpr` in the above section and, for plotting the ROC curve we have defined `plot_roc` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 267,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfkAAAGDCAYAAAAoD2lDAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAABmOElEQVR4nO3dd3gUVRfA4d9Jp4TQe5UuHUITRHoRpIlgoauIgopYUJFPRbEiNlBERZqKjaaiKCpFBCnSQYr03kNLCEnu98dM2E1I2UB2J+W8z5Mn0+fMbDl7Z+7cK8YYlFJKKZX1+DkdgFJKKaW8Q5O8UkoplUVpkldKKaWyKE3ySimlVBalSV4ppZTKojTJK6WUUlmUJnkfEJHNItLc6TgyChF5VkQ+cWjfU0TkZSf2nd5E5B4R+eUa173m96SILBOROtey7rUSkUdE5DVf7jM9iUgOEfleRCJE5Bun40mOiJwXkRuuYT3HPtMqZdkuyYvIHhGJtN/MR+wv/dze3KcxppoxZpE39xFPRIJF5FUR2Wcf5w4ReVJExBf7TyKe5iJywH2aMeYVY8x9Xtqf2Alhk4hcEJEDIvKNiNTwxv6ulYi8ICIzrmcbxpjPjTFtPdjXVT9srvU9KSK3AeeMMWvt8RdE5LL9eTojIn+JSONE6+QVkQ/tz9tFEdkoIgOS2PbdIrLa3tZhEflJRJrasycBvUWkcAqxZeTXvgdQBChgjLnjejeW1OcqPRhjchtjdqV139f6mRaRRSISZb/mJ0RklogUS+t2VPKyXZK33WaMyQ3UBuoAzzgbTtqJSEAys74BWgG3AqFAH2AQ8K4XYhARyWjvoXeBR4FHgPxAJWAO0DG9d5TCa+B1Du57MDA90bSv7M9TQeAPrPcgACISBCwEygCNgTDgSeA1ERnuttxw4B3gFaxkWBr4AOgCYIyJAn4C+qYQW7q99l44v2WA7caYmAwQS0Yz1H7/VAByA2PTewdOn0NH92+MyVZ/wB6gtdv4G8CPbuONgL+AM8B6oLnbvPzAZ8Ah4DQwx21eJ2Cdvd5fQM3E+wSKA5FAfrd5dYATQKA9PhDYam9/AVDGbVkDDAF2ALuTOLZWQBRQKtH0hkAsUMEeXwS8CqwEIoC5iWJK6RwsAsYAy+xjqQAMsGM+B+wCHrCXzWUvEwect/+KAy8AM+xlytrH1Q/YZ5+LkW77ywFMtc/HVuAp4EAyr21F+zgbpPD6TwEmAD/a8f4NlHeb/y6wHzgLrAFudpv3AvAtMMOefx/QAFhun6vDwHggyG2dasCvwCngKPAs0B6IBi7b52S9vWwY8Km9nYPAy4C/Pa+/fc7ftrf1sj3tT3u+2POO2a/pBqA61g+8y/b+zgPfJ/4cAP52XP/Z52QNid5D9nJB9utZMtE5meE2fqP9ehayx++1Y8qVaFu97Hjy2Md9Hrgjlc/uPcAf1/HaLwLucxu/cv6S+nwBE4GxibYxFxhuDxcHvgOO28s/ksx+X0z0et+LVcB6Dthrn59pQFiiz8S9WJ+JJUlssznJfw6q2sd6BtgMdHabVwD4Huv9u8p+HyU+B/HfE7cCW+z3xEHgCTz4TNvrNsX1HbIf6O/ha/IQsNltvAquz882oGcajyXB9yUpf0+PsI/znL2vVvb0BsBqez9HgXFu63S2z/EZ+1iqus3bY29zA3AJCEjp/e2tP5/v0Ok/En65lQQ2Au/a4yWAk/ab2w9oY4/Hf2H9CHwF5AMCgVvs6XWxPqgNsb4w+9n7CU5in78D97vF8yYw0R7uCuzE+pAGYH0J/JXoTfsr1o+NHEkc22vA4mSOey+u5LvIfjNXx/rQfocr6aZ2DhZhffFUs2MMxCoplcdKNLcAF4G69vLNSfRlRNJJ/mOshF7L/kBUdT8m+5yXtD8wyX25DQb2pvL6T8H6wmhgx/85MNNtfm+sL48A4HHgCBDiFvdl+3Xys+Oth/WjKMA+lq3AMHv5UKyE/TgQYo83THwO3PY9B/jIfk0KY/0Ii3/N+gMxwMP2vnKQMMm3w0rOee3XoSpQzO2YX07hc/Ak1uegsr1uLazLyonPXTXgQgqvZZD9ep3A/kIDZgJTk9hWgH087bB+9MSQypcg1ufs1HW89otIPclf+XwBzbASlNjz82EluOL2678G+J993Ddg/cBtl8y+E7zeWD/md9rr5QZmAdMTfSam2e+FpD7rzUnic4D1edyJ9aMtCGiJlbQqu70eM4GcWD/I9idxDuKT/GHsH7n2sXv6mS5t7/MuO54CQO3UXhN7uYXAXHs8lx3fAPv9Utd+b1VLw7G4v57Jfk9jvff3A8XdXoPy9vByoI89nBtoZA9XAi5gfUcGYhVAdmL/yLe3vQ4oldRr6Ks/R3bq5J994s/bb0ID/AbkteeNiP+guS2/wH4zFMP69ZoviW1+CLyUaNo2XD8C9uD6Qr0P+N0eFvuN1cwe/wm4120bflgJs4zbm7ZlCsf2CW4JK9G8FdglZPuD9ZrbvBuxShr+KZ0Dt3VHp3KO5wCP2sPN8SzJu5cOVwJ32sMJvjjt85dckh8JrEgltinAJ27jtwL/prD8aaCWW9xXlaoSLT8MmG0P3wWsTWa5K+fAHi+C9eMmh9u0u7BLrlgJaV+ibfTHleRbAtuxfnD4JXHMKSX5bUAXDz47TYAjSRxHNFZJJhbrB2Fzt/kL3d9ridY9glU6vyfxdpNZviIQex2v/SJST/It3cYF6wdt/Ofzflyf3YZJvB7PAJ95+Hr/BjzkNl4Z6wdk/I9FA9yQwrE0T+pzANxsn1c/t2lf2vv3t/dR2W1eSiX5fcADQJ7U9k3Cz/Qz2J8BD17TRVjfcRH2vtcBpe15vYCliZb/CHg+Dcfi/nom+z2NdUXyGNYV18BEyyzBuhpTMNH0UcDXbuN+WIWn5m6fsYGenAdv/mW0+6m+0tUYE4r1Zq2CdS8RrPtmd9gViM6IyBmsy07FsH6NnTLGnE5ie2WAxxOtVwrrF39i3wKNRaQ4VknBAEvdtvOu2zZOYX3RlHBbf38Kx3XCjjUpxez5SW1nL9Yv0YKkfA6SjEFEOojIChE5ZS9/K65z6qkjbsMXsX4xg3UO3feX0vGfJPnj92RfiMjjIrLVrgV9ButSsvuxJD72SiLyg12p7CzWPeX45UthXQL3RBms1+Cw23n/CKtEn+S+3Rljfse6VTABOCoik0Qkj4f79jTO01hXIxL72hiTF+uHyiasqxvxknxP2vcoC9rzTwIFPbhvGYqVDJLi6Wufmivn2Fjf1DOxfmwB3I115Qes16t4os/Js1jnwBPFsT538fZiJXj39VN6r6e03f3GmLhE2y4BFLL34enn6Xasz/JeEVmcuEJlCtLyvgfrNkcYUBPXFTuwznHDROf4HqAonh+L+7Rkv6eNMTuxfqC/ABwTkZn2dzRYt00qAf+KyCoR6WRPT/Aa2ud8P55/X/tEdk3yABhjFmOVcuIreuzHKsXmdfvLZYx5zZ6XX0TyJrGp/cCYROvlNMZ8mcQ+zwC/AD2xvjS+tL9M4rfzQKLt5DDG/OW+iRQOaSHWh6KU+0QRaYD1Zv7dbbL7MqWxfhWfSOUcXBWDiARjXe4fCxSxv+znY/04SS1eTxzG9aFPHHdivwElRST8WnYkIjdjXcnoiXXFJi9WUnF/MiHx8XwI/AtUNMbkwfqij19+P9ZtjKQk3s5+rJJ8QbfznscYUy2FdRJu0Jj3jDH1sC6rV8K6DJ/qeqnE6W4HVn3LEknNNMacwCr5veBWQ3oh0EFEciVa/Has412BdTk0Cus2SEqqYtURSYonr/0FrEu78YomsUzic/Ul0ENEymCV3r+zp+/Hus/r/jkJNcbcmsoxxDuElXTilca6ZXE0hVg83W6pRBViS2OVMI/b+/Do82SMWWWM6YL1Q3MO8LWHcXn6fkq8v41YpfEJIhJ/lXNxonOc2xjzYBqOxT3WFL+njTFfGGOaYr0uBnjdnr7DGHMX1nl4HfjWfj8neA3tmEthneuk9u+IbJ3kbe8AbUSkNlaFqttEpJ2I+ItIiP24SEljzGGsy+kfiEg+EQkUkWb2Nj4GBotIQ7vGeS4R6SgiSZV6AL7AqiV8uz0cbyLwjIhUAxCRMBHx+HEbY8xCrC+770Skmn0MjbBKHx8aY3a4Ld5bRG4UkZzAaOBbY0xsSucgmd0GYd3TOg7EiEgHwP2xrqNAAREJ8/Q4Evka65zks5PL0OQWtI/vA+BLO+YgO/47ReRpD/YVivXFcRwIEJH/YVUMS22ds8B5EakCPOg27wegqIgME+vRxlARaWjPOwqUjf8ytt9fvwBviUgeEfETkfIicosHcSMi9e33XyBWMovCunwev6+Unn3+BHhJRCra79+aIlIg8ULGmMtYSTvZmIwx/2Ld3nnKnjQdOAB8IyJl7c9NO+A94AVjTIQxJgLr3vYEEekqIjnt5TqIyBtum78F6zOY1H49ee3XAd3t7VfAKqGlyFiPCh63z9EC+0c6WLeUzorICLGegfcXkeoiUj+1bdq+BB4TkXJiPcL7CtZTCmmqfW8f45U/O64LwFP2OWwO3IZ1Gy8W697/C/Y5qEIyTyvY5+8eEQmzX/ezJHw/pfSZ/hxoLSI9RSRARArY36+emIqVTDtjfX4qiUgf+1gC7fd51bQci5tkv6dFpLKItBSr0BKFVfci1j4XvUWkkF1SP2NvKxbru6mjiLSyP3ePY/1w/Svxjh3l9P0CX/+RqHa9cd2r+c4ebohV0esU1of7R1z3iPJjvQmPYl26nOW2jfZYNTzPYJU+vwFCk9onViWQc7jVInWb1werEtRZrF+ek93mXblflsLxhWD92tyP9UbdCTxNwnt0i3DVrj+LVUO1oNv8lM7BItzua9rThtjn5AzWl/pM3O4BA5OxLqeeIfna9QGJ4ouvjJPL3uYZrEptzwH/pXD8gvUY1WasS/EHsSpLxlfWmZIotubY9xex7vN9ap+Tw1iJ6sprR9KV5ZphleTPY912GU3C+4LVsX54nca6TfC0Pb0A8Kc9/R97WhjWe/EA1hWEtbjqJvR3327iaVhPVmyw4ziB9UWb255XEVeN4jmJ35P2cT+HVUP8HNb7uGQy57cj8JPbeFLnpCFWoins9rn5COs9Emm/Nvclse17sGoxX7DP1Y/ATW7v6wNYV4uu9bUviPVD6hzWkwovkMz96ETbHWXPuyPR9OJYyfqI/TquINF3S3LnCauA9T+sz+lxrB/X+ZL7TCSxveb2Mon/KmBdyVlsv4e2AN3c1itkn9f4GumvA78lPgdYP95/to8rftmmnnym7fk3Yz25Ev891i+Z41iU+L2AdTVttT1c2Y73uL2/37Er8Xl6LIm2neT3NNatgpX2e+MU1g+M+Ep4M7Du15/Hem91ddteN/scR9jnvJrbvD3JvR98+Rdfa1RlIyKyCOsDmelaqBKRB7ESn0clXJX+RORP4GFjN4jjo30+jPVY31OpLqw8JiKvA0WNMf2cjuV6ZaVjSU9ZvZEFlcmJdW/3Bqz7thWxLomNdzSobM5Y9y19vc/3fb3PrMi+rB2EdbWwPtYtC6+0PultWelYvEmTvMrogrAu9ZbDusQ2E+veq1Iq7UKxbjEUx7oE/RZWAz+ZUVY6Fq/Ry/VKKaVUFqW165VSSqksSpO8UkoplUVlunvyBQsWNGXLlnU6DKWUUspn1qxZc8IYUyit62W6JF+2bFlWr17tdBhKKaWUz4jI3tSXupperldKKaWyKE3ySimlVBalSV4ppZTKojTJK6WUUlmUJnmllFIqi9Ikr5RSSmVRmuSVUkqpLEqTvFJKKZVFaZJXSimlsihN8koppVQW5bUkLyKTReSYiGxKZr6IyHsislNENohIXW/FopRSSmVH3izJTwHapzC/A1DR/hsEfOjFWJRSSqlsx2sd1BhjlohI2RQW6QJMM8YYYIWI5BWRYsaYw96KSSmllMpM5o95hLJmxTWv72QvdCWA/W7jB+xpVyV5ERmEVdqndOnSPglOKaWUcsqFC9E8cfdoJs4rQO3idYFV17QdJyveSRLTTFILGmMmGWPCjTHhhQqluTtdpZRSKlP56uOfmTgvkED/WO6qk2TVNo84WZI/AJRyGy8JHHIoFqWUUipjOLCU/pe7srZJe+5r+A+1nvmNEYWqX9OmnCzJzwP62rXsGwERej9eKaVUdrRt2wlat57Gnj1nYMXL+PkZ3u/2E7XufAIK3HjN2/VaSV5EvgSaAwVF5ADwPBAIYIyZCMwHbgV2AheBAd6KRSmllMqIjDF8+OFqnnjiFyIjY3j22d/4ot4v1swW70LdR65r+96sXX9XKvMNMMRb+1dKKaUyssOHzzFw4Dx+/nknAH361OT9d1rBZ/YCN/a97n04eU9eKaWUypa++24Lgwb9wKlTkeTPn4OJEztyxx3VYP9i10Ihea97P5rklVJKKR/au/cMd931HZcvx9GuXXkmT+5C8eKh1syze63/QXnSZV+a5JVSSikfKlMmL6+/3prg4AAefDAcEfuJ8gN/ws/9rOESTdJlX5rklVJKKS+6dCmGUaP+oEGDEvToYdWUf+yxxlcveGqLazgd7seDJnmllFLKazZsOErv3rPYuPEYhQvn4tZbK5IzZ2DSC8ffj69xP1S5M132r13NKqWUUuksNjaOsWP/on79j9m48RgVKuRn7tw7k0/wAP9+Yf2POpVucWhJXimllEpHe/eeoV+/OSxebFWiGzy4HmPHtiVXrqDkVzr8t2u48fPpFosmeaWUUiqdGGO4/favWbPmMEWK5OLTTzvTsWOl1FdcMNA1XKhGusWjl+uVUkqpdCIivP9+B3r0uJGNGx/0LMGvfANO2pXuWrybrvFokldKKaWuw08/7WDUqN+vjDduXIpvvrmDQoVypb7yznmwdIRrvHLPdI1NL9crpZRS1+DChWiefPJXPvxwNQBt2pSnWbMynm/AGJjbxTU++DDkKpquMWqSV0oppdLo778P0KfPbHbsOEVgoB8vvdSCJk1Kpb6iOxPnGu67Pt0TPGiSV0oppTx2+XIsY8Ys5eWXlxAba6hWrRAzZnSndu3rSNDiB4Vqpl+QbvSevFJKKeWhMWOW8uKLi4mNNQwf3ojVqwdde4I/vsH6b0z6BZiIluSVUkopDw0b1oiFC3cxenQLWrYsd30bm9XBHvBekteSvFJKKZWMw4fP8fDD84mKigEgb94Qli4dcP0JHuDiUet/qw+uf1vJ0JK8UkoplYTvvtvCAw/8wMmTkeTJE8yYMa0AXL3GXavoc/DZja7xSndc3/ZSoEleKaWUchMREcUjj/zMtGnrAWjbtjwPPVQ//XZwbD2cP2ANF6oFOQum37YT0SSvlFJK2RYv3kO/fnPYuzeCkJAAxo5tw0MP1b/+0ru74+us/3nLQ5+16bfdJGiSV0oppYDVqw/RosVUjIHw8OJMn96NKlW8UMr+/WHrf3BeSM8fD0nQJK+UUkoB9eoVo3v3qtx4YyFGjWpGYKB/+u7gciRMc+t8pvm49N1+EjTJK6WUypZiY+N4550VdOxYiSpVCiIifP31Hfj5eal0fWYHnPnPGi5cF0o2885+3OgjdEoppbKdvXvP0KrVNJ544lf69JlNXJz1rLrXEjzAtq+t/2HloPdq7+3HjZbklVJKZRvGGKZP38DDD//E2bOXKFw4F88/f4t3kztA1Bn4e4w1HFra6/fi42mSV0oplS2cOHGRwYN/4LvvtgLQtWsVJk3q5FmXsNfj2DqYXsc13v4z7+7PjSZ5pZRSWV50dCwNG37Crl2nyZ07iPfea0///rXT99G45BxZ6Rpu8rJ1ud5HNMkrpZTK8oKC/Bk2rCFffbWZadO6ccMN+Xy382PrrP817odGI323X7TinVJKqSxq1aqDzJ699cr4kCENWLy4v+8SvDHww52w/kNrPOqUb/brRkvySimlspSYmDheeWUpo0cvJmfOQOrUKUbZsnntynW+qfCGMfBRcbhwxDUt/Anf7NuNJnmllFJZxvbtJ+nTZzYrVx4E4P7761K0aG7fBnFiM8zv7UrwBW6EnosgZyHfxoEmeaWUUlmAMYaPPlrD44//wsWLlylZMg9Tp3ZNny5h02pqdbcRgf6bfR+DTZO8UkqpTO+xxxbw7rt/A3DPPTUYP/5W8uYN8X0g+xe7hus8DI2e830MbrTinVJKqUyvX79aFCmSi5kzb2fGjO7OJHiABQNcwy3fg5yFnYnDpkleKaVUpnP27CU+/njNlfE6dYqxe/ej9OpVPYW1vMwYiNhtDdce6lwcbvRyvVJKqUxlyZK99O07m717IwgLC6Fnz2oA5MgR6Gxg2791Dd/0onNxuNGSvFJKqUzh0qUYRoz4lebNp7B3bwT16hWjZs0iTofl8lMf13CO/M7F4UZL8koppTK8TZuO0bv3LNavP4qfnzByZFP+979b0r/P92u1cy7EXrKGbxrtbCxuNMkrpZTK0BYu3EXHjl8QHR1L+fL5mD69G40bl3I6LBdjYG5X13j9pxwLJTFN8koppTK0Ro1KUrp0GC1alGXcuHbkzh3kdEgJndnpGu61FAKCnYslEU3ySimlMhRjDN9+u4UOHSqSO3cQuXMHsXr1/YSFOfRYXGoOLHENl2zqXBxJ0Ip3SimlMoyTJy/Sq9e39Oz5LU888cuV6Rk2wQOc3GL9D87raBhJ0ZK8UkqpDGHBgp0MGDCXw4fPkzt3EA0alHA6JM/s+dn6X+VuZ+NIgiZ5pZRSjrp48TIjRvzK+PGrAGjSpJTv+3y/HrmKWaX5ovWdjuQqmuSVUko55syZKBo1+oRt204SGOjH6NEtePLJm/D3z4R3k0NLOh3BVTTJK6WUckzevCGEhxfH39+PGTO6UadOMadDSrsjK52OIFma5JVSSvnUzp2nuHQphmrVrM5bPvywIwEBfs43S3stTBxEn7OGA3M5G0sSMuH1EKWUUpmRMYZJk9ZQq9ZEevX6lqioGABCQ4MzZ4IHiI12DRdr6FwcydCSvFJKKa87cuQ89903jx9/3AFArVpFiY6OJSQkE6ehmCj4tKI1LH7WXwaTic+uUkqpzGDOnH+5//7vOXHiInnzhjBxYkdnu4RNLxePwvkD1vCN/ZyNJRma5JVSSnnNo4/+xHvvWRXTWre+gSlTulCiRB6Ho0pnoaWh/WSno0iSJnmllFJeU7t2UUJCAnjjjdYMGdIAPz9xOqT0c2qb0xGkSpO8UkqpdBMdHcvq1Ye46Sarl7j+/WvTsmU5ypTJ62xg3hCxy/p/bp+zcaQg49USUEoplSlt2nSMBg0+plWrafz77wkARCRrJviLx2H5i9ZwjfudjSUFXk3yItJeRLaJyE4ReTqJ+WEi8r2IrBeRzSIywJvxKKWUSn9xcYa3315OePgk1q8/SvHioVy4EJ36ipnZvO5w4Yg1HJJxm9/12uV6EfEHJgBtgAPAKhGZZ4zZ4rbYEGCLMeY2ESkEbBORz40xWfzdoZRSWcO+fRH07z+HP/7YA8B999Vh3Lh2hIZmnD7V092W6XDwT2s4JD/Uf8rZeFLgzXvyDYCdxphdACIyE+gCuCd5A4SKiAC5gVNAjBdjUkoplU5++mkHd931HRERlyhUKCeffNKZzp0rOx2W9/3U1zV8738QktexUFLjzSRfAtjvNn4ASNwc0HhgHnAICAV6GWPiEm9IRAYBgwBKly7tlWCVUkqlTbly+bh0KZbOnSvz8ce3UbhwxmvWNd3tnOca7rM2Qyd48O49+aSekzCJxtsB64DiQG1gvIhc9QClMWaSMSbcGBNeqFCh9I5TKaWUh/755zDGWF/lVaoU5J9/BjFnTq/skeAB5nZxDReu7VgYnvJmkj8AlHIbL4lVYnc3AJhlLDuB3UAVL8aklFLqGly8eJlHHvmJevUmMXXq+ivTq1YthHXHNRvY9o1r+I7fnYsjDbx5uX4VUFFEygEHgTuBuxMtsw9oBSwVkSJAZWCXF2NSSimVRmvWHKJ379n8++8JAgL8OHMmyumQfO/CEZjvlsJKt3AuljTwWpI3xsSIyFBgAeAPTDbGbBaRwfb8icBLwBQR2Yh1eX+EMeaEt2JSSinluZiYOF577U9efHExMTFxVK1akBkzulO3bibs8/16xMXCRLdjbvORc7GkkVdbvDPGzAfmJ5o20W34ENDWmzEopZRKu8OHz3H77V+zfLnVAcujjzbk1VdbZd4uYa+VMTCtpmu8dCuo3Mu5eNJIm7VVSil1lXz5cnD27CVKlAhlypSutG59g9MhOePsHjhpP/mdrzL0+BUyUR0ETfJKKaUAOHr0PMHBAeTNG0JISACzZ/eiYMGc5MuXw+nQnDPrVtfwgK2ZKsGDtl2vlFIKmDv3X2rU+JChQ113WCtWLJB9E/zlSFj4IJz61xpv/HymS/CgJXmllMrWzp27xGOPLeDTT9cCcOTIeSIjL2e/e+/u4mJhQj6IvWSNB+SE+k86G9M10iSvlFLZ1LJl++jTZza7d58hONif119vzcMPN8xafb5fi1P/uhJ8oZrQ6RsIzJyN/WiSV0qpbMYYw3PP/c5rry0jLs5Qp05RZszozo03aouiGANTq1vD4gd916e8fAan9+SVUiqbERFOnowE4Nlnm7JixX2a4ONt+8o1XHOQc3GkE4lvgzizCA8PN6tXr3Y6DKWUylTi4gyHD5+jRAmre5ALF6LZsOEojRuXSmXNbGTrFzD/Htf4YzHg5+9cPG5EZI0xJjyt62lJXimlsrj9+yNo02Y6t9wyhfPnowHIlStIE3xi7gm+y5wMk+CvhyZ5pZTKwr74YiM1anzI77/v5uzZS2zbpi2HJytXUet/r6VQoUvKy2YSWvFOKaWyoFOnIhkyZD4zZ24CoFOnSnzyyW0UKZLb4cgygbzlnY4g3WiSV0qpLOb333fTt+9sDh48R65cgbzzTnvuvbdO9ukSNq12zLF6mIuJdDqSdKdJXimlspiTJy9y8OA5GjcuybRp3ahQIb/TIWVsmz5xJfgi9SBn1nnSQJO8UkplASdPXqRAgZwA3HFHNWbP9qdTp0oEBGjVq1QdWGL9b/YmhD+eKZuvTY6++koplYnFxMTxyitLKVPmHdatO3JleteuVTTBeypnEet/iaZZKsGDluSVUirT+u+/U/TtO4e//toPwMKFu6hdu6jDUWUy0efhzE5rODivo6F4gyZ5pZTKZIwxfPrpWoYN+5kLFy5TvHgoU6Z0oU2brFMr3OtiL8PWGbBgoGta3huci8dLNMkrpVQmcuzYBe6//3vmzdsGQK9e1fjgg47kz59Nu4S9Vnt/TZjgq9wF/kHOxeMlmuSVUioTuXjxMn/8sZuwsGA+/LAjd91Vw+mQMqcd31n/81WGWg9A7aHOxuMlmuSVUiqDu3Ahmpw5AxERypbNy9df30G1aoUoVSrM6dAyp7P7XEk+T2mo95iz8XiRVr1USqkMbNmyfdSsOZEJE1Zdmda+fQVN8NfKGPi8PlyKsMar9XM2Hi/TJK+UUhlQdHQsI0f+RrNmU9i16zRffLGRuLjM1WtohrN2AnzXDi4es8arDYCyHZyNycs8ulwvIuHAzUBxIBLYBCw0xpzyYmxKKZUtbdlynN69Z7F27RFE4Omnm/Diiy3w88taz3D7jDHw1S1wcKlrWmgpaD/ZuZh8JMUkLyL9gUeA3cAaYBsQAjQFRojIJmCUMWafl+NUSqksLy7OMH78SkaMWEhUVAxly+Zl+vRuNG1a2unQMrcDSxIm+O7zoXBd5+LxodRK8rmAJsaYJFvtF5HaQEVAk7xSSl2n2Ng4pk/fQFRUDAMH1ubtt9uTJ0+w02Flfof+cg0/FpMl+on3VIpJ3hgzIbl5IpLLGLMu3SNSSqls5tKlGIKDAwgM9GfGjG5s3XqCrl2rOB1W1vDf9/Dns9Zw6VbZKsGDB/fkRaQEUAzYYIyJFpHCwDCgP9Y9eqWUUtfg9Gmrz/fIyBhmzeqJiFC5ckEqVy7odGiZ3/GN8G1rVyU7gJqDnIvHIandkx8GjAR2AsEi8i4wDpgG1PN6dEoplUX99tsu+vefy4EDZ8mVK5AdO05RqVIBp8PKOg79lTDB91oMJZs5F49DUivJDwIqG2NOiUhprGTfzBizwvuhKaVU1hMZeZlnn/2Nd975G4BGjUoyfbr2+Z7u4hu7qX4vtJ0Ekj2fGE8tyUfFPyZnjNknIts1wSul1LVZu/YwvXvPZsuW4wQE+PHCC7cwYkRT7RI2PV04Cj/eBfv/sMaD82bbBA+pJ/mSIvKe23hh93FjzCPeCUsppbKezz/fyJYtx6lSpSAzZnSjXj2t1pTu/hjmSvAAjZ5zLJSMILUk/2Si8TXeCkQppbKi2Ng4/P2tkuTLL7ckX74QHnusMTlzBjocWRZ1erv1P+wG6LsegnI7G4/DUnuEbqqIFALKADuNMWd8EpVSSmVyxhg++2wdb721nGXLBpI3bwghIQGMHJn9Kn/5jDFw7B9ruNGobJ/gIZW260XkPmAz8D7wr4h09klUSimViR07doFu3b7i3nvnsWXLcWbM2OB0SNnD8fWu4Rs6OhdHBpLa5fphQDVjzHERuQH4HJjn9aiUUiqT+v77bdx33/ccO3aBsLBgJky4lbvv1j7fvS5iD0yvYw37BUDOQo6Gk1GkluSjjTHHAYwxu0RE21dUSqkknD8fzfDhC/j4Y+tycYsWZZkypSulS2uXsD5xcotr+KYXnYsjg0lr7fqSWrteKaWutnz5fj7++B+Cgvx59dVWDBvWSHuN86Xt31r/y3WAhs86G0sGorXrlVLqGhljELESeZs25Xn99dbcemtFqlcv7HBk2czmabD5M3tEf1i5Sy3JVzbG6E8ipZRKZOvW4/TvP5d33mlH48alAHjqqSYOR5UNnfwXfu7nGm8/xbFQMqLUmgFq75MolFIqk4iLM7z//t/UrTuJlSsPMmrUH6mvpNJf1BlYNgqmVHVNG7hDK9wlklpJ3l9E8pHM9Y/4Jm+VUio7OHjwLAMGzOXXX3cB0L9/bd59V8tCPrXvd/j7Fdj3W8LpLd6FfBWciSkDSy3JV8G6D59UkjfADekekVJKZUBffbWJBx/8kdOnoyhQIAeTJt1G9+5VU19RXb/LF2Hlq/D3q2BiE84LLQVd5kCRuo6EltGlluS3GGPq+CQSpZTKoE6fjuShh+Zz+nQUt95akU8/7UzRotqamlcd/QeOrQMM/HLf1fPbfgqFakKReiBa2S45qSV5pZTK9vLly8HHH9/G8eMXGDSo3pUa9cpLYqJgZlOIiUw4vVhj6DAVwspZDd6oVKV2lt71SRRKKZWBREXF8Oyzv1G0aO4rNeb10rwPnd1nJXi/QKja25pWpg1UvcvZuDKh1JJ8ExFZY4zZmHiGiOQCegGXjDGfeyU6pZTysXXrjtC79yw2bz5OzpyBDBxYh4IFczodVvZwajtEnYIf77TG4y5D+8nOxpTJpZbkJwCjRKQGsAk4DoQAFYE8wGSs9uyVUipTi42NY+zYvxg16g8uX46jUqUCzJjRTRO8r6x8A5aOSDitXAdnYslCUutqdh3QU0RyA+FAMSAS2GqM2eb98JRSyvt27z5N375z+PPPfQAMGVKfN95oo32++5J7gi/WEHIUhLafOBdPFuFRzQVjzHlgkXdDUUopZwwd+hN//rmPYsVyM3lyF9q31+etfWLzVOuZd3f9N0OBG52JJwvS6olKqWxvwoRbef75RYwb15YCBfTyvE9E7Iaf+yecFlYO8lV2JJysSpO8Uirb+f77bXz11WamTeuGn59Qtmxepk7t6nRY2cu8Hq7hu/6CkPwQWhr8/J2LKQtKU5IXkVzGmAveCkYppbwpcZ/vnTtXpmfPag5HlQ3ERMGi4XD+kGvaMes1oHxnKN7YmbiyAY+SvIjcBHwC5AZKi0gt4AFjzEPeDE4ppdLL8uX76dNnNv/9d5qgIH9eeaUlPXrovV+vO/lvwk5kEmuhzbF4k6cl+beBdsA8AGPMehFpltpKItIeq0Edf+ATY8xrSSzTHHgHCAROGGNu8TAmpZRK1eXLsYwevZhXXvmTuDhDzZpFmDGjGzVqFHE6tOzBPcEXrAE3jXaN56sIYWV9HlJ24vHlemPM/kRNOcYmtyyAiPhjPWffBjgArBKRecaYLW7L5AU+ANobY/aJSOE0xK6UUqn6+ON/ePnlpYjAU0/dxOjRLQgO1upIPmGMa7j1h1BrsHOxZFOevtP325fsjYgEAY8AW1NZpwGw0xizC0BEZgJdgC1uy9wNzDLG7AMwxhxLS/BKKZWa+++vy++/7+bhhxtwyy1lnQ4n+zi8EuZ1c41XH+hcLNmYn4fLDQaGACWwSuW1gdTux5cA9ruNH7CnuasE5BORRSKyRkT6JrUhERkkIqtFZPXx48c9DFkplR0dPHiWu+/+juPHrTrCgYH+fPttT03wvnTgT/iioauiXWgp8A9yNqZsytOSfGVjzD3uE0SkCbAshXWS64M+8f7rAa2AHMByEVlhjNmeYCVjJgGTAMLDwxNvQymlAPj6680MHvwDp09HERTkz5QpXZ0OKfs5shq+utk13nwc1NTL9E7xNMm/D9T1YJq7A0Apt/GSwKEkljlhP5Z3QUSWALWA7SillIfOnIli6ND5fP651ZdWhw4VePXVVg5HlQ1FnbZK8PF6LYGSNye/vPK6FJO8iDQGbgIKichwt1l5sGrMp2QVUFFEygEHgTux7sG7mwuMF5EAIAhoiFWTXymlPPL777vp128OBw6cJWfOQN56qy0PPKB9vjviUgSYOGu4wzRN8BlAaiX5IKxn4wOAULfpZ4EeSa5hM8bEiMhQYAHWD4LJxpjNIjLYnj/RGLNVRH4GNgBxWI/Zbbq2Q1FKZTe7dp2mTZvpxMUZGjQowfTp3ahUqYDTYWVP276BH3paw3nKwI19nI1HASDGpH6LW0TKGGP2+iCeVIWHh5vVq1c7HYZSKoN4+umF5MgRwMiRzQgI8LQusUpXMVHwbg7XeM0HoM1E5+LJgkRkjTEmPK3reXpP/qKIvAlUw+pPHgBjTMu07lAppa5VfJ/v4eHFadXqBgBee621w1Flc8bAV25tmPXbBAW1qeCMwtMk/znwFdAJ63G6foA+y6aU8hn3Pt9LlcrD9u0PExKijdo47vR2OLLSGs5fVRN8BuPpta0CxphPgcvGmMXGmIFAIy/GpZRSABhj+OyztdSsOZE//9xH0aK5+eijTprgM4rlbs3U3vWXc3GoJHn6Kbls/z8sIh2xHoUr6Z2QlFLKcvz4BR544Admz/4XgNtvr8rEiZ0oWFD7fM8Qds2Hf7+whku1gJC8joajruZpkn9ZRMKAx7Gej88DDPNWUEopZYyhbdsZrFt3hDx5ghk/vgO9e9fUR+OcdnwDrHoDzh+E/Ytc05uOcSoilQKPkrwx5gd7MAJoAVdavFNKKa8QEcaMackbbyxj6tSulCmT1+mQ1E99Ycv0q6e3eBeKNbx6unJcao3h+AM9sdqc/9kYs0lEOgHPYjVDW8f7ISqlsosVKw6wevUhhg5tAMCtt1akQ4cKWnp3UlwsHFwKx9YlTPDhT0DRBlaDN7mKOhaeSllqJflPsZqmXQm8JyJ7gcbA08aYOV6OTSmVTVy+HMtLLy1hzJilADRsWIL69a3+rDTBO2jLDFj0GESeSDh9yCkIyedMTCpNUkvy4UBNY0yciIQAJ4AKxpgj3g9NKZUd/PvvCXr3nsWaNYcRgccfb0yNGkWcDit7Or0Dlj4Dl89DxC5r3F35LhA+XBN8JpJako82xmqI2BgTJSLbNcErpdKDMYYJE1bx5JO/EhUVQ+nSYUyd2pXmzcs6HVr25N4sbWLN34bqAyA4zLcxqeuWWpKvIiIb7GEBytvjAhhjTE2vRqeUyrJefHExL764GIC+fWvx3nvtCQsLSWUt5RWntidM8HUehnK3gggUvwmCQpNfV2VoqSX5qj6JQimV7QwaVI8vv9zEmDEt6dHjRqfDyb5MHHzZ2DXeaymUbOpcPCpdpZjkM0qnNEqpzC8iIor331/JM880xd/fj+LFQ9my5SH8/bVTGccYAwsfhKhT1vgtYzXBZzHaLqRSyusWLdpD376z2b//LIGBfowYYSUSTfAOOncQTm6CDZOs8bAbIPxxZ2NS6U6TvFLKa6KiYnjuud8ZN245xkD9+sXp1k3vAjpu1Zuw5KmE03qvcSYW5VUeJ3kRyQGUNsZs82I8SqksYv36I/TuPZtNm47h7y+MGtWMZ5+9mcBAf6dDy97O7k+Y4AvVhOoDtd35LMqjJC8itwFjgSCgnIjUBkYbYzp7MTalVCa1cuVBmjadzOXLcVSsmJ/p07vRsKH2aZUhLBjoGu6/BQrolZWszNOS/AtAA2ARgDFmnYiU9U5ISqnMrl69YjRuXIpq1Qrx5pttyJUryOmQVMQeOPgn7FtojVfsrgk+G/A0yccYYyK0eUmlVFKMMUyfvoGWLctRsmQe/P39+OWX3gQHa7WfDGHzNPi5X8Jpnb5yJhblU55Wbd0kIncD/iJSUUTeB/7yYlxKqUzi+PEL3H771/TrN4cBA+YSF2cANMFnFIdXJkzwJW+B238GP319sgNPX+WHgZHAJeALYAHwsreCUkplDj/+uJ17753H0aMXCA0Nok+fmugFP4cdWWV1CRsbbY1H7HLN67MWCtd2JCzlDE+TfGVjzEisRK+UyuYuXIjmiSd+YeJE67GrZs3KMHVqV8qWzetsYApmd4KLx66e3ukrTfDZkKdJfpyIFAO+AWYaYzZ7MSalVAZ26VIM9et/zNatJwgK8mfMmJY89lgjbdgmIzh/yJXg6w2H2g9Zw8F5IUcBx8JSzvEoyRtjWohIUaAnMElE8gBfGWP0kr1S2UxwcAA9e1Zj1qytzJjRnZo1tVvYDOHcAZhUyjVe4z7IW965eFSGIMaYtK0gUgN4CuhljPH5czHh4eFm9erVvt6tUtnatm0nOHz4/JVuYC9fjiU21hASopW3MoSLx+BDtx9bVe6GW2egFSSyDhFZY4wJT+t6Hl1fE5GqIvKCiGwCxmPVrNeWLZTK4owxfPDBKurU+Yhevb7l2LELAAQG+muCzwhObYPt3yZM8HUf1QSvrvD0U/oZ8CXQ1hhzyIvxKKUyiEOHzjFw4FwWLPgPgDvuqEZwsDZJmyFEnYE142DFSwmnhz8Jt7zhSEgqY/L0nnwjbweilMo4vv12Cw888AOnTkWSP38OPvqok/b5nhGseQf+mwv7FyWcXvF2KFANbnrB9zGpDC3FJC8iXxtjeorIRsD95r0AxhhT06vRKaV87umnF/L668sAaN++ApMnd6ZYsVCHo8rmjIHP68PRRD3F5S4JXedAkXqOhKUyvtRK8o/a/zt5OxClVMbQoUMFxo9fyZtvtmHw4HC0OesMYMXLCRN8j4UQFApFw0H00UWVvBSTvDHmsD34kDFmhPs8EXkdGHH1WkqpzOTSpRgWLPiPzp0rA3DLLWXZu3cYBQrkdDgyxdl9Vrew29zamR8eq4ldeczTd0qbJKZ1SM9AlFK+t2HDUerX/5guXWby22+u5k81wTto/2L4sgnMCIePyyRM8P02aoJXaZLaPfkHgYeAG0Rkg9usUGCZNwNTSnlPbGwcb7+9gpEjfyc6OpYKFfITGhrsdFgKYP2HcChR/183dIKW70NYWUdCUplXavfkvwB+Al4Fnnabfs4Yc8prUSmlvGbv3jP06zeHxYv3AjB4cD3Gjm2rfb5nBJcjXSX3Rs9B+S5Wk7T5Kjgalsq8UkvyxhizR0SGJJ4hIvk10SuVufzxx266dv2Ks2cvUaRILiZP7sKtt1Z0OiwV749HXMNV7oYCVZ2LRWUJnpTkOwFrsB6hc69ma4AbvBSXUsoLatQoQs6cgbRqVY5Jk26jYEG99+6Yo2tgwb0QE+Wadnqb9T8ghyZ4lS5Sq13fyf5fzjfhKKXS26JFe7jpplIEBflTsGBOVq++n+LFQ/XROKcYA0tGwOo3k1+m9z++i0dlaR61eCciTYB1xpgLItIbqAu8Y4zZ59XolFLX7MKFaJ566lc++GA1I0fezMsvtwSgRIk8DkeWjRkDP/WBrZ+7prV8H0q3do3nLAw58vs+NpUledp2/YdALRGphdUD3afAdOAWbwWmlLp2K1cepE+f2WzffpLAQD/y5g1xOiS19UtYNhIidrumDdoPodrXl/IeT5N8jDHGiEgX4F1jzKci0s+bgSml0u7y5VheeWUpL720hNhYQ/XqhZkxoxu1ahV1OrTsa9/v8OsDcGZnwukPHITcxZ2JSWUbnib5cyLyDNAHuFlE/IFA74WllEqrU6ci6dDhc1auPAjA8OGNGDOmlXYJ67SVryVM8Ld9A2XaQHCYczGpbMPTT38v4G5goDHmiIiUBlKoNaKU8rV8+ULInz8HpUrlYcqUrrRsqfVlHbVjDuxbCHt/tcabvgL1hkOANjqkfEeMMakvBYhIEaC+PbrSGHPMa1GlIDw83KxevdqJXSuV4Rw+fI7o6FjKlMkLwLFjFwgK8td78E4ycbBgIGyemnD64COQq4gzMalMT0TWGGPC07qeR40gi0hPYCVwB9AT+FtEeqR1Z0qp9DNr1lZq1PiQu++eRWxsHACFC+fSBO+kuFjYsyBhgm/yMvRZqwleOcLTy/UjgfrxpXcRKQQsBL71VmBKqaRFRETx6KM/M3XqegBy5w7i3LloTe5OO70TJidqPfChk/o4nHKUp0neL9Hl+ZN43oOdUiqdLFmyl759Z7N3bwQhIQGMHduGhx6qrw3bOM3EJUzw/kFwyzhN8Mpxnib5n0VkAfClPd4LmO+dkJRSSRk16nfGjFmKMRAeXpzp07tRpUpBp8PKvqLPwc/94fwhOOpWT+imF6Hx/xwLSyl3HiV5Y8yTItIdaIrVfv0kY8xsr0amlEogT55gRITnnruZUaOaERjo73RI2Y8xcHIzxF6CGfWxuvBwk7+qJniVoaTWn3xFYCxQHtgIPGGMOeiLwJTK7uLiDNu3n7xSWh8+vDFt25bXhm2ccv4wfNEIziVqzbtIOLR8D8QfCtdxJjalkpFaSX4yMA1YAtwGvA9093ZQSmV3e/eeoX//uaxbd4SNGx+kZMk8+Pv7aYJ3Smw0fJSodbrCda2e4jpMA9EqSipjSi3JhxpjPraHt4mIdo2klBcZY5gxYwNDh/7E2bOXKFw4F3v3nqFkSe1UxlEHlriGb7gNOn4BQbmdi0cpD6WW5ENEpA6ufuRzuI8bYzTpK5VOTp68yODBP/Ltt1sA6Nq1CpMmdaJQoVwOR6bYOcf67x8EXeeCPs2gMonUkvxhYJzb+BG3cQO09EZQSmU3ixbt4e67v+Pw4fPkzh3Ee++1p3//2vpoXEYQcwk2T7GGizbQBK8ylRSTvDGmxfVsXETaA+8C/sAnxpjXklmuPrAC6GWM0QZ2VLaTI0cAx45doGnT0kyb1pVy5fI5HZKKt3M2XL5gDVfr72goSqWV17qnsnuqmwC0AQ4Aq0RknjFmSxLLvQ4s8FYsSmVEe/eeudLmfMOGJVm0qD+NG5fE318rcTkq6jQsuBeO2Xcjo8+55lW6w5mYlLpG3vw2aQDsNMbsMsZEAzOBLkks9zDwHeBIhzdK+VpMTByjRy+mQoX3+f77bVemN21aWhO80/4YBhPyW6X3s3utv6hT1rxWEyBYK0CqzMWbHU2XAPa7jR8AGrovICIlgG5Y9/bro1QWt2PHSfr0mc3ff1vNTaxff5TbbqvscFTZkDGw6TMribv7513XcOnW0NZ+uCggBHLp44sq8/EoyYtV++ce4AZjzGi7P/mixpiVKa2WxLTE/dq+A4wwxsSmVMFIRAYBgwBKly7tSchKZSjGGCZNWsPw4b9w8eJlSpbMw9Sp2ue7Y46tg1/uTX7+kFMQovUiVObnaUn+AyAOq8Q9GjiHdYk9pdL3AaCU23hJ4FCiZcKBmXaCLwjcKiIxxpg57gsZYyYBk8DqT97DmJXKEI4fv0D//nOZP38HAPfcU4Px42/VXuOcdNm+zx5aCqonSvbFG2mCV1mGp0m+oTGmroisBTDGnBaRoFTWWQVUFJFywEHgTuBu9wWMMVeKMSIyBfghcYJXKrMLCvJn06Zj5MsXwocfdqRXr+pOh5Q9GQMH/7Q6lPnxTmtaaCm46Xln41LKizxN8pftWvAGrvQnH5fSCsaYGBEZilVr3h+YbIzZLCKD7fkTrz1spTK2s2cvERjoR44cgYSFhTBrVk+KFs1NiRJaccvros9b99bjK8zFO7o6Yct1AEW1KpDK2jxN8u8Bs4HCIjIG6AE8l9pKxpj5JOqSNrnkbozp72EsSmVoS5fupW/fOXTpUpl33mkPQL16xVNZS6WLmEvwfh6urv6TSKWekLc8NH3ZJ2Ep5RRPu5r9XETWAK2wKtR1NcZs9WpkSmUyly7F8Pzzi3jjjWUYA8uW7efSpRiCg735EItKYMVLXEnwIQWg4TMJ5/sFQMUeEFrC56Ep5QRPa9eXBi4C37tPM8bsS34tpbKPTZuO0bv3LNavP4qfnzByZFNGjbqFoCDt890njq2HQ3/B32Nc0x46pr3DqWzP0yLGj1g/jwUIAcoB24BqXopLqUzBGMPbb6/gmWd+Izo6lvLl8zF9ejcaNy6V+soqfcRcgum1E07rt0kTvFJ4frm+hvu4iNQFHvBKREplMn/9tZ/o6FgGDarLW2+1I3fu1B48UenGGPj0Btd49Xuh/G1QUMsfSsE1tnhnjPnH7lRGqWzHGMPZs5cICwtBRJg4sRMDBtSmY8dKToeWvRgDsztaj8QBlG4F7T5xNialMhhP78kPdxv1A+oCx70SkVIZ2KlTkQwe/AM7d55ixYr7CAryp2DBnJrgfWXPr3ByszW88RPXMECPX5yJSakMzNOSfKjbcAzWPfrv0j8cpTKuBQt2MmDA3Ct9vq9ff4T69bWWts/8867VgUxShp7Re/BKJSHVJG83gpPbGPOkD+JRKsO5ePEyI0b8yvjxqwBo0qQU06Z144YbtOlTn3JP8HUftf4H5bGGg8McCUmpjC7FJC8iAXbLdXV9FZBSGcnq1Yfo3XsW27adJCDAj9Gjm/PUU020S1hfijwFM5u6xgduh3wVnYtHqUwktZL8Sqz77+tEZB7wDXAhfqYxZpYXY1PKcatWHWTbtpPceGMhZszoRp06xZwOKXuJPg8fFHCN5y0PeSs4F49SmYyn9+TzAyexeqGLf17eAJrkVZYTGXmZHDkCARg8OBw/P6Fv31pXpikf2jLdNVzlbugwDVLolloplVBqSb6wXbN+E67kHk+7fFVZijGGjz/+h//97w+WLRtI+fL5EREeeCDc6dCyl5Wvw+qxIP5w+bw1LW956Pi5s3EplQmlluT9gdwkTO7xNMmrLOPIkfPcd988fvzR6vN95sxNjBzZzOGospkzu+DIKlj6dMLp4gdNtCMZpa5Fakn+sDFmtE8iUcohc+b8y/33f8+JExfJm9fq8/3OO7XPd5/Z8yvs/cUqvbu7bzcEhIB/CITkdSQ0pTK71JK83vxSWda5c5cYNuxnJk9eB0Dr1jfw2WddKFlS+3z3qfl3Q+QJ13jF7lChG4SVdSwkpbKK1JJ8K59EoZQD9uw5w4wZGwkJCeCNN1ozZEgD/Pz0d61PXb7gSvA3jbaapi1xk7MxKZWFpJjkjTGnfBWIUr4QExNHQID1jHuNGkWYPLkzdeoU48YbCzkcWTYUcwney+0aD38CAnM4F49SWZC26KGyjU2bjhEePomZMzddmXbPPTU1wTvBGJjVwTVe60FN8Ep5wTX1QqdUZhIXZ3j3XavP90uXYhk79i969aqG6PPWvnV8I/zxqPVY3OntcCnCmh52A7T+wNnYlMqiNMmrLG3//gj695/L77/vBuC+++owblw7TfC+EH0eTm6xhk0cfNk46eXu/tt3MSmVzWiSV1mSMYYvv9zEQw/9SETEJQoVysknn3Smc+fKToeW9cXFwo5Z8EPPpOff/DqUam61XFeoFvgH+TQ8pbITTfIqS4qOjuXFFxcTEXGJzp0r8/HHt1G4cC6nw8oe9v6SMMEXqgn+wdZwpZ5Q/wln4lIqG9Ikr7KUuDiDn58QHBzA9Ond2LDhKPfeW0cvz3tbXAzERFnDvz/smn7rF1D1LmdiUkppkldZQ2TkZZ5+eiGRkTFMmnQbAA0alKBBgxIOR5YNHN8A02pdPb3uME3wSjlMk7zK9NasOUTv3rP5998TBAb6MWJEE8qXz+90WFlXTBRsmWb18x57CZa/4JoXaN8SyV0cmr/lSHhKKRdN8irTiomJ4/XX/+SFFxYTExNH1aoFmTGjuyZ4b7l0FpaOgPUTk57f9Xso38m3MSmlUqRJXmVK//13ij59ZrN8+QEAHn20Ia++2kr7fPeW5C7J1x9h/S/X3qoxr5TKUDTJq0zp9deXsXz5AUqUCGXKlK60bn2D0yFlXae2wZpxrvFiDaHRKKud+YAQ5+JSSqVKk7zKNIwxV2rJv/lmG0JCAnjxxebky6fNoaa7mEuw4SOrZbp1E1zTq98L7T5xLi6lVJpokleZwpw5//L22yv4+ed7yJEjkLCwEN57r0PqK6q0M3EwuRKc25dweuU7ocEIZ2JSSl0TTfIqQ0vc5/unn65l6NAGzgaV1c27PWGCb/IS3NAJCtd2LCSl1LXRJK8yrD//3EffvrPZvfsMwcH+vP56ax56qL7TYWVtvz4AO+e4xodFg79WZlQqs9IkrzKc6OhYnn/+D15/fRnGQO3aRfn88+7aJay3nd0HGya5xh8+qwleqUxO+5NXGc78+Tt47bVliAjPPNOUv/++TxO8L3zdwjX84HEICnUuFqVUutCSvMpwunSpzGOPNaJ796o0bVra6XCyrhObYGoNEH9r3MRa/yvfCTkLOheXUirdaEleOW7//gg6dvyCLVuOAyAijBvXThO8N0WdthI8WMk9PsHnKAgdP3cuLqVUutKSvHJM4j7fY2LiWLCgt9NhZQ/bvnYNd/4Oyne2hsXf6uddKZUlaJJXjjh1KpKHHvqRr77aDECnTpX45JPbHI4qmzi6FhYOtobzV4GK3Z2NRynlNZrklc/9+ut/DBgwl4MHz5ErVyBvv92O++6rq32++0JcLMyo6xqvcb9zsSilvE6TvPKpkycv0q3bV1y4cJlGjUoyfXo3KlTQXuN8JvKEa7jdZKjW37FQlFLep0le+VSBAjl5++12HD16gaefbkpAgNb99KnfH7H+ix9UH+BsLEopr9Mkr7wqJiaON95YRtGiuRk4sA4A999fz+GosiFj4MAS2G5XuAvTXvuUyg40ySuv+e+/U/TtO4e//tpP7txBdO1ahfz5tcc4nzMGptwIp/51TbtnpXPxKKV8RpO8SnfGGD79dC3Dhv3MhQuXKV48lM8+66IJ3inrJiRM8N3nQ0g+5+JRSvmMJnmVro4ePc/993/P999vB6Bnz2p8+GFHTfBOiDoDix6DzVNc04bH6XPwSmUjmuRVuurdezYLF+4iLCyYDz7oyF13VddH43wp+hxEnoRj/1hdxrrrMlcTvFLZjCZ5la7eeqstTz+9kI8+6kSpUmFOh5O1RZ+DvQsh7rI1fv6QVXJPrHRraPYaFK579TylVJamSV5dl2XL9jFnzr+8+WZbAGrWLML8+fc4HFU2sfhJ2PBR0vPylLEek2s5Hm641bdxKaUyDE3y6ppER8fy4ouLeO21ZcTFGW6+uQydO1d2Oqzs5eJR63+xxhBa0hoWf6j1AJRq7lhYSqmMQ5O8SrMtW47Tu/cs1q49ggg8/XQT2rUr73RYWd+5A1bvcf9+Catetx6NA6j/JFTs5mxsSqkMSZO88lhcnOH99/9mxIiFXLoUS9myeZk2rSs331zG6dCyrgNL4PRO2Pcb/PvF1fND8kOhWr6PSymVKWiSVx4bP34lw4YtAGDgwNq8/XZ78uQJdjiqLOzsfvjqlqunF6wOQWHQYap1791PP8ZKqaTpt4Py2L331uHrrzfz+OON6datqtPhZH2HV1j/g/NChW4QEAL1hkO+Co6GpZTKPLzaO4iItBeRbSKyU0SeTmL+PSKywf77S0T0umMGcvp0JI899jPnzl0CIFeuIJYuHaAJ3hcuHIUfelrDsVHQfjK0/kATvFIqTbxWkhcRf2AC0AY4AKwSkXnGmC1ui+0GbjHGnBaRDsAkoKG3YlKeW7hwF/37z+HgwXNcuhTLBx90BNCGbXwh+hx8UtY13ulrx0JRSmVu3rxc3wDYaYzZBSAiM4EuwJUkb4z5y235FUBJL8ajPBAZeZlnnvmNd9/9G4BGjUoyfHhjh6PKJnbMsv6iz0NMlDWtyUtQ/jZn41JKZVreTPIlgP1u4wdIuZR+L/CTF+NRqfjnn8P07j2LrVtPEBDgxwsv3MKIEdrnu9dFnoJ53eHA4oTTi9aHRs85E5NSKkvwZpJP6rquSXJBkRZYSb5pMvMHAYMASpcunV7xKTc7dpykUaNPuHw5jsqVCzBjRnfCw4s7HVb2cHh5wgTf9hPwD9YGbZRS182bSf4AUMptvCRwKPFCIlIT+AToYIw5mdSGjDGTsO7XEx4enuQPBXV9KlYswD331CQ0NIjXXmtNzpyBToeUPZzdDytft4bzVYQ+ayEwl7MxKaWyDG8m+VVARREpBxwE7gTudl9AREoDs4A+xpjtXoxFJWKMYfLktdSuXZR69awS+6efdsbPTyvW+dSUqnD5gjVcuK4meKVUuvJakjfGxIjIUGAB4A9MNsZsFpHB9vyJwP+AAsAHdq3tGGNMuLdiUpZjxy4waND3zJ27jSpVCrJu3QMEBwdogvelqNNw4YgrwZdqDk1edjQkpVTW49XGcIwx84H5iaZNdBu+D7jPmzGohL7/fhv33fc9x45dICwsmOeeu5mgIH+nw8pedv8MszuCiXNNu+M3q9c4pZRKR9riXTZx/nw0w4cv4OOP/wGgRYuyTJnSldKltc93n9n7G+xbaLVkZ+KsS/NBeaDhSE3wSimv0CSfDcTFGW65ZQr//HOY4GB/Xn21FY8+2kgvz/vKoRWwYjTsTvSEaIOn9RE5pZRXaZLPBvz8hMcea8TYsX8xY0Z3qlcv7HRI2UNcLJzZCavfTJjgbxoNwWFwYx/nYlNKZQtiTOZ6Ii08PNysXr3a6TAyvK1bj7N+/VHuvLM6YNWmj4mJIzBQ7797VfQ52DUfYi/Bz/0Szqs/AuoNg1xFHQlNKZV5iciaa6mYriX5LCYuzjB+/EpGjFiIMYYaNQpTrVphREQTvLf89z3ssbrgZd2Eq+cHhECRcKgzVBO8UsqnNMlnIQcOnGXAgLksXLgLgAEDalOqlFasS1dn98OvgyDqlGvakZVXLxccBuU7Q4Fq0GCE7+JTSik3muSziK++2sTgwT9y5kwUBQvmZNKkTtolbHowBo6sgi8akUyrzC4t3gHxhxwFoVIP8NOPl1LKWfotlAW8/PISRo36A4COHSvyySedKVo0t8NRZQHHN8D3PeD0jqvn1XsMKvdyjeerBCH5fBebUkp5QJN8FtCrVzXef38lo0c3Z9Cgetrn+/UwBla+Cqe3w+apCee1mwzV+lvDeo6VUpmAJvlMKCoqhhkzNnDvvXUQESpWLMDu3Y9qpzLXIjYa4i67xr9rDwf/TLhM/aegwTMQktenoSml1PXSJJ/JrF17mN69Z7Nly3EA7ruvLoAm+Gux7w+YfSvERCU9v91nUKQuFKrp27iUUiqdaJLPJGJj43jzzb/43//+uNLne61aRZwOK3OIPAVbP4eYyITT9/9hJXi/APALsicaKHAj3P03+Okjh0qpzE2TfCawa9dp+vadzbJl+wEYOrQ+r7/eRkvvKTm2Hla9YTVKs+O7lJdtNAoa/883cSmllA9pks/gVq48SKtW0zh/PppixXLz2WddaNeugtNhZWwbPraeZU8sT1modEfCaYG5oNZgn4SllFK+pkk+g6tVqwg33JCPypUL8OGHHSlQIKfTIWVskScTJvj6T0HR+hCcz+qzXS/BK6WyEU3yGdD8+Tto1Kgk+fPnIDg4gMWL+xMWFqyPxqVm3x/wTUvX+P17IE8Zx8JRSimnaSfWGcj589EMGvQ9HTt+wUMP/Uh850F584Zogk/Nqe0JE3yj5zTBK6WyPS3JZxDLl++nT5/Z/PffaYKC/Klfv7jTIWVslyLg4nF7xMBnlV3zOn0FlXs6EpZSSmUkmuQddvlyLKNHL+aVV/4kLs5Qs2YRZszoRo0a+nhcss4dhMkVkn6+ve2nmuCVUsqmSd5BUVEx3HzzZ6xefQgReOqpmxg9ugXBwfqypOjQX1aC9w+G0JLWNGOgQleoMdDR0JRSKiPRbOKgkJAAGjYswfHjF5g2rRvNmuk95GRFn4MFA+H8ISvJg3XPfeA2Z+NSSqkMTJO8jx08eJbjxy9Su3ZRAN54ow2vvNKKPHmCHY4sAzIGTv0La9+D9ROvnh/fWYxSSqkkaZL3oa+/3szgwT+QN28I69cPJjQ0WFutS8nfY2DZqITTKnaHesMhKA8UrO5MXEoplUlokveBM2eiGDp0Pp9/vhGAxo1LERUVQ2iolt6TZYwrwYeWgnyVoePnkLOws3EppVQmokney/74Yzf9+s1h//6z5MwZyLhxbbXP93gRe2DBAIg6dfW8U/+6hlt9AOU7+SwspZTKKjTJe9FLLy3mf/9bBEDDhiWYPr0bFSsWcDaojOLiMVg3AfYvSnm5gJxQroMvIlJKqSxHk7wXVatWmIAAP/73v2Y888zNBARoA4MAbP8Ovu/hGq/cCxo8c/VyIla3r9revFJKXRNN8ukoNjaOlSsP0rhxKQC6d6/K9u1DKVcun8ORZQBHVsE/78Hp7XBkpWt66dYQ/jgUruVcbEoplUVpkk8ne/acoW/f2SxffoAVK+6lXj2rWdpsmeC3fgHLngPc6h1E7Lp6uR6/QpnWPgtLKaWyG03y18kYw9Sp63nkkZ84dy6aokVzc+5ctNNhOePUNji03KpMl5xGo6BANSjdQmvKK6WUl2mSvw7Hj1/ggQd+YPZsqyb47bdX5aOPOmW/Pt8vHoeVr8GacQmn91oKuYu5xkPyQ0g2vLKhlFIO0SR/jZYt28ftt3/N0aMXyJMnmPHjO9C7d83s+WjcmrcTJvgbOkKdh6FkU+diUkoppUn+WpUqFUZkZAy33FKGqVO7UqZMXqdDuj7nD0HcZdf4wgfhyGqrhntqos9a/0PyQc9FUKimV0JUSimVNprk02DjxqNUq1YYPz+hdOkw/vprIFWqFMTfP5M9GmeM1cnL+UPW+PIX4OSW69umXyB0maMJXimlMhBN8h64fDmWl19ewpgxSxk7ti3DhjUCrOfgM509v8DKV5NvhCa0tGu4UA2rf3ZPBOaEoNDrjU6pDOPy5cscOHCAqKgop0NR2UhISAglS5YkMDB9+jXRJJ+KbdtO0Lv37Ct9vp88edHpkNIuYjf8/qh1Wf3A4oTzKtmN0uQqBje/ZiVrpRQHDhwgNDSUsmXLZs+6NsrnjDGcPHmSAwcOUK5cuXTZpib5ZBhj+OCDVTz55K9ERsZQunQY06Z15ZZbyjodWtrsmAPzul09vdmbUOVOCC3p85CUygyioqI0wSufEhEKFCjA8ePH022bmuSTcPp0JHfd9R0LFvwHQN++tXjvvfaEhYU4HFkqzu6Dfb8DxjVtwUDXcN1hUKGL1VSsPqOuVKo0wStfS+/3nCb5JISGBnPmTBT58+fgo4860aPHjU6HlDITByvfgD+TaP893u0LoGxb38WklFLKcZmsWrj3REREceKEdb89IMCPmTN7sGnTgxkvwRtjJXX3v42fJkzw5W6Fav1df7eM1QSvVCbk7+9P7dq1qV69Orfddhtnzpy5Mm/z5s20bNmSSpUqUbFiRV566SWMcV3F++mnnwgPD6dq1apUqVKFJ554woEjSNnatWu57777nA4jWZcuXaJXr15UqFCBhg0bsmfPniSXi46OZtCgQVSqVIkqVarw3XffAfDYY49Ru3ZtateuTaVKlcibNy8Ax48fp3379j45Bi3JA4sW7aFfvznUrFmEefPuREQoWzav02ElFLHHeuTt21YQk0Jt355/QKnmvopKKeVFOXLkYN26dQD069ePCRMmMHLkSCIjI+ncuTMffvghbdu25eLFi9x+++188MEHDBkyhE2bNjF06FB+/PFHqlSpQkxMDJMmTUrX2GJiYggIuL4U8sorr/Dcc8/5dJ9p8emnn5IvXz527tzJzJkzGTFiBF999dVVy40ZM4bChQuzfft24uLiOHXqFABvv/32lWXef/991q5dC0ChQoUoVqwYy5Yto0mTJl49hmyd5KOiYnjuud8ZN245xkCRIrmIiLhE3rwZ7N77jtkwr3uiiYnv2xi443dN8Ep5w1teujf/uEl9GVvjxo3ZsGEDAF988QVNmjShbVvrCl3OnDkZP348zZs3Z8iQIbzxxhuMHDmSKlWqABAQEMBDDz101TbPnz/Pww8/zOrVqxERnn/+eW6//XZy587N+fPnAfj222/54YcfmDJlCv379yd//vysXbuW2rVrM3v2bNatW3elhFqhQgWWLVuGn58fgwcPZt++fQC88847VyWzc+fOsWHDBmrVsnqgXLlyJcOGDSMyMpIcOXLw2WefUblyZaZMmcKPP/5IVFQUFy5c4Pvvv+fhhx9m48aNxMTE8MILL9ClSxf27NlDnz59uHDhAgDjx4/npptu8vj8JmXu3Lm88MILAPTo0YOhQ4dijLnqvvnkyZP591+reXM/Pz8KFix41ba+/PJLXnzxxSvjXbt25fPPP9ck7y0bNhyld+9ZbNx4DH9/4bnnmjFy5M0EBmagvsvjYuGv5+HvMa5pxRpa/a/Xe8y5uJRSPhUbG8tvv/3GvffeC1iX6uvVq5dgmfLly3P+/HnOnj3Lpk2bePzxx1Pd7ksvvURYWBgbN24E4PTp06mus337dhYuXIi/vz9xcXHMnj2bAQMG8Pfff1O2bFmKFCnC3XffzWOPPUbTpk3Zt28f7dq1Y+vWrQm2s3r1aqpXr35lvEqVKixZsoSAgAAWLlzIs88+e+Wy9/Lly9mwYQP58+fn2WefpWXLlkyePJkzZ87QoEEDWrduTeHChfn1118JCQlhx44d3HXXXaxevfqq+G+++WbOnTt31fSxY8fSunXCXjEPHjxIqVJW1+EBAQGEhYVx8uTJBEk8/hbKqFGjWLRoEeXLl2f8+PEUKVLkyjJ79+5l9+7dtGzZ8sq08PDwNF3FuFbZMsm/9dZfPPvs70RHx1KxYn6mT+9Gw4YZ6FEyEwfR52BKdTh/wDW95yIodYtjYSmVbaWhxJ2eIiMjqV27Nnv27KFevXq0adMGIMnSZLy01M5euHAhM2fOvDKeL1/qHUjdcccd+PtbhaFevXoxevRoBgwYwMyZM+nVq9eV7W7Z4mpF8+zZs5w7d47QUFeDWYcPH6ZQoUJXxiMiIujXrx87duxARLh82dXMdps2bcifPz8Av/zyC/PmzWPs2LGA9ajjvn37KF68OEOHDmXdunX4+/uzffv2JONfunRpqscYz72OQ7zE5zcmJoYDBw7QpEkTxo0bx7hx43jiiSeYPn36lWVmzpxJjx49rpw3gMKFC3Po0CGPY7lW2TLJHzt2gejoWB58MJw332xDrlxBvtnxic1w4XDKyxgD3yWqJBcUCnf8BkXrey82pVSGE39PPiIigk6dOjFhwgQeeeQRqlWrxpIlSxIsu2vXLnLnzk1oaCjVqlVjzZo1Vy6FJye5Hwvu0xK3+JcrV64rw40bN2bnzp0cP36cOXPmXCmZxsXFsXz5cnLkyJHisblve9SoUbRo0YLZs2ezZ88emjdvnuQ+jTF89913VK5cOcH2XnjhBYoUKcL69euJi4sjJCTp265pKcmXLFmS/fv3U7JkSWJiYoiIiLjyYyNegQIFyJkzJ926We2R3HHHHXz6acKWQmfOnMmECRMSTIuKikrx/KSXbJHkjTEcOXKeYsWsX5GjR7egXbsKtGyZPi0KAXD+MGz9HOKS6Uv+2DrY/k3athmYG0o0ge4/edZRjFIqSwoLC+O9996jS5cuPPjgg9xzzz288sorLFy4kNatWxMZGckjjzzCU089BcCTTz5J9+7dadq0KZUqVSIuLo533nmH4cOHJ9hu27ZtGT9+PO+88w5gXa7Ply8fRYoUYevWrVSuXJnZs2cnKIG7ExG6devG8OHDqVq1KgUKFEiw3SeffBKAdevWUbt27QTrVq1albfeeuvKeEREBCVKlABgypQpyZ6Ldu3a8f777/P+++8jIqxdu5Y6deoQERFByZIl8fPzY+rUqcTGxia5flpK8p07d2bq1Kk0btyYb7/9lpYtW171o0hEuO2221i0aBEtW7bkt99+48YbXU9lbdu2jdOnT9O4ceME623fvj3B7QpvyfJJ/sSJizzwwA+sWHGAjRsfJH/+HAQHB6Rfgt8xC7bMgJ2zPV+ndKvUl7mho953V0pdUadOHWrVqsXMmTPp06cPc+fO5eGHH2bIkCHExsbSp08fhg4dCkDNmjV55513uOuuu7h48SIiQseOHa/a5nPPPceQIUOoXr06/v7+PP/883Tv3p3XXnuNTp06UapUKapXr36lEl5SevXqRf369RMk5vfee48hQ4ZQs2ZNYmJiaNasGRMnTkywXpUqVYiIiLhyGf+pp56iX79+jBs3LsG968RGjRrFsGHDqFmzJsYYypYtyw8//MBDDz3E7bffzjfffEOLFi0SlP6v1b333kufPn2oUKEC+fPnT3Bro3bt2leefHj99dfp06cPw4YNo1ChQnz22WdXlvvyyy+58847r/px8McffyT5mqQ3SeqeQ0YWHh5ukqpMkZT583cwcOBcjh69QGhoEPPm3UXz5mWvfecxUdZjbHM6wyW7gsr5RPdUitSDMsk8k+4XADf2hXwVrj0GpZRPbN26lapVqzodRpb29ttvExoamqGflfeWZs2aMXfu3CTrQST13hORNcaY8LTuJ0uW5C9ciOaJJ35h4sQ1ANx8c2mmTet27c++R52GPQvgx7uSX6bDdMhT1rq8rpfWlVIqVQ8++CDffJPG25hZwPHjxxk+fLhHFR2vV5ZL8qtWHeSee2axY8cpAgP9GDOmJcOHN76+Pt8nFoVYt3vtuYpClbuhnn1/K0cBCMhgz9YrpVQGFxISQp8+fZwOw+cKFSpE165dfbKvLJfkT52KZMeOU1SvXpgZM7pRq1bR69vgmrddCb5ANbjpBVf3rEqpLC2lR9WU8ob0voWeJZL86dOR5MtnPYrQrl0FZs3qSYcOFQkJSebwLkfCuX1XT9/2Nax8FfwCXdOiz7qG+29Kx6iVUhlZSEgIJ0+epECBAprolU/E9yef3ON/1yJTJ3ljDB9+uJoRIxby00/30LRpaQC6dasKcTGw60eIOpVopTj4uX8qW468etKDx9IlZqVU5lCyZEkOHDiQrn17K5WakJAQSpZMv8bZvJrkRaQ98C7gD3xijHkt0Xyx598KXAT6G2P+8WTbhw+fY+DAefz8804AfvzsG5pG73UtsP3b1BueyVfp6mmBueHW6ZC7RMJpfhmouVullNcFBgZSrlw6tqWhlAO8luRFxB+YALQBDgCrRGSeMWaL22IdgIr2X0PgQ/t/ir77bguDBv3AqVOR5M+fg4nvNOGOY01hbTIrVO199bRy7aHqPWk6JqWUUioz8WZJvgGw0xizC0BEZgJdAPck3wWYZqyaBitEJK+IFDPGJFsE37PrBD16WI9ctKu8k8k951L8mN1EYUh+aPy8a+GAHFDpDgjJm57HpZRSSmUK3kzyJYD9buMHuLqUntQyJYBkk3xERBQ5Ai/zZqdfeOimVQkfSa/cC+o+cp1hK6WUUlmDN5N8UtVREz8b4MkyiMggYJA9eikmbsymobNh6FUtyX5o/6nrVBA44XQQ2YCeZ+/Tc+x9eo59o3Lqi1zNm0n+AFDKbbwkkLhfPU+WwRgzCZgEICKrr6VpP+U5Pce+oefZ+/Qce5+eY98QEc/ac0/kOpqBS9UqoKKIlBORIOBOYF6iZeYBfcXSCIhI6X68UkoppTzntZK8MSZGRIYCC7AeoZtsjNksIoPt+ROB+ViPz+3EeoRugLfiUUoppbIbrz4nb4yZj5XI3adNdBs2wJA0bnZSOoSmUqbn2Df0PHufnmPv03PsG9d0njNdV7NKKaWU8ow378krpZRSykEZNsmLSHsR2SYiO0Xk6STmi4i8Z8/fICJ1nYgzM/PgHN9jn9sNIvKXiNRyIs7MLLVz7LZcfRGJFRHt4vAaeHKeRaS5iKwTkc0istjXMWZ2HnxfhInI9yKy3j7HWscqjURksogcE5Eke0O7prxnjMlwf1gV9f4DbgCCgPXAjYmWuRX4CetZ+0bA307HnZn+PDzHNwH57OEOeo7T/xy7Lfc7Vv2VHk7Hndn+PHwv58VqbbO0PV7Y6bgz05+H5/hZ4HV7uBBwCghyOvbM9Ac0A+oCm5KZn+a8l1FL8leaxDXGRAPxTeK6u9IkrjFmBZBXRIr5OtBMLNVzbIz5yxhz2h5dgdWOgfKcJ+9jgIeB7wDt6vDaeHKe7wZmGWP2ARhj9FynjSfn2AChdsdjubGSfIxvw8zcjDFLsM5bctKc9zJqkk+uudu0LqOSl9bzdy/WL0jluVTPsYiUALoBE1HXypP3ciUgn4gsEpE1ItLXZ9FlDZ6c4/FAVawGzTYCjxpj4nwTXraR5ryXUfuTT7cmcVWyPD5/ItICK8k39WpEWY8n5/gdYIQxJlYkqcWVBzw5zwFAPaAVkANYLiIrjDHbvR1cFuHJOW4HrANaAuWBX0VkqTHmrJdjy07SnPcyapJPtyZxVbI8On8iUhP4BOhgjDnpo9iyCk/OcTgw007wBYFbRSTGGDPHJxFmDZ5+X5wwxlwALojIEqAWoEneM56c4wHAa8a6ebxTRHYDVYCVvgkxW0hz3suol+u1SVzvS/Uci0hpYBbQR0s81yTVc2yMKWeMKWuMKQt8CzykCT7NPPm+mAvcLCIBIpITq0fMrT6OMzPz5Bzvw7pSgogUwepQZZdPo8z60pz3MmRJ3miTuF7n4Tn+H1AA+MAuacYY7YjCYx6eY3WdPDnPxpitIvIzsAGIAz4xxiT5mJK6mofv5ZeAKSKyEeuy8ghjjPZOlwYi8iXQHCgoIgeA54FAuPa8py3eKaWUUllURr1cr5RSSqnrpEleKaWUyqI0ySullFJZlCZ5pZRSKovSJK+UUkplUZrklVJKqSxKk7xS18juGnad21/ZFJY9nw77myIiu+19/SMija9hG5+IyI328LOJ5v11vTHa24k/L5vsrkfzprJ8bRG59Rr2U0xEfrCHm4tIhNtrsdCe/oKIHHSLp3MS07eIyF1u2x0rIi3TGo9SGZE+J6/UNRKR88aY3Om9bArbmAL8YIz5VkTaAmONMTWvY3vXHVNq2xWRqcB2Y8yYFJbvD4QbY4amcT9vAn8aY+aKSHPgCWNMp0TLvACcN8aMFZGqwFKgMFZDT/HTKwJrgALGmMsiUgb42BjTNi3xKJURaUleqXQiIrlF5De7lL1RRK7qVtYufS5xK1nebE9vKyLL7XW/EZHUku8SoIK97nB7W5tEZJg9LZeI/Cgi6+3pvezpi0QkXEReA3LYcXxuzztv///KvWRtX0G4XUT8ReRNEVklIhtE5AEPTsty7F6yRKSBiPwlImvt/5XtJlJHA73sWHrZsU+297M2qfNoux342YMYADDGbMXq+rRgouk7sFoPy2eP7wUKiEhRT7etVEalSV6paxefJNeJyGwgCuhmjKkLtADeErmqa7m7gQXGmNpYHaSsE5GCwHNAa3vd1cDwVPZ9G7BRROphNW3ZEGgE3C8idYD2wCFjTC1jTHUSJUNjzNNApDGmtjHmnkTbngnE/ygIwmqPfD5WT4QRxpj6QH17X+WSC1BE/O1149s4/xdoZoypg1WSfsXum/x/wFd2LF8BI4Hf7f20AN4UkVyJtl0OOG2MueQ2+Wa312NkEvE0xGrS9nii6XWBHYn6mP8HaJLcsSmVWWTItuuVyiQi7WQNgIgEAq+ISDOsZFICKAIccVtnFTDZXnaOMWadiNwC3Agss38TBGGVgJPypog8h5Wo7sVKorPt3tUQkVnAzVhJfayIvI51iX9pGo7rJ+A9EQnG+rGwxBgTad8iqCkiPezlwoCKwO5E6+cQkXVAWazL4L+6LT/VvjxusNvkTkJboLOIPGGPhwClSdihTDESJWtgaeLL9bbHRKQ3cA7oZYwx9nl+TETuB26wj9PdMaB4MvEplWloklcq/dwDFALq2fd292AlqCuMMUvsHwEdgen2feXTwK/GmLsSbzAJTxpjvo0fEZHWSS1kjNlul/JvBV4VkV+MMaM9OQhjTJSILMLqH7wX8GX87oCHjTELUtlEpDGmtoiEAT8AQ4D3sDow+cMY002sSoqLkllfgNuNMdtS2geJzm0K3jbGjE1uuoh0B6aJSHljTJQ9L8Teh1KZml6uVyr9hAHH7ATfAiiTeAG7UtcxY8zHwKdAXWAF0ERE4u+x5xSRSh7ucwnQ1V4nF9ANWCoixYGLxpgZwFh7P4ldtq8oJGUm1m2Am7F6HsP+/2D8OiJSKfFldHfGmAjgEeAJe50w4KA9u7/boueAULfxBcDD8bc67NsPiW3HulJw3Ywxs7BukfRzm1wJ0F7qVKanSV6p9PM5EC4iq7FK9f8msUxzrPvwa7Eqjr1rjDmOlfS+FJENWEm/iic7NMb8A0wBVgJ/Y3WhuhaoAay0L5uPBF5OYvVJwIb4ineJ/AI0Axba980BPgG2AP+IyCbgI1K5GmjHsh6r//E3sK4qLMPqrjTeH8CN8RXvsEr8gXZsm+zxxNu9APwX/8MoHYwGhouIn/2DpAJW4lcqU9NH6JRSmZKIdMO6NfKcF7Zb1xgzKj23q5QT9J68UipTMsbMFpECXth0APCWF7arlM9pSV4ppZTKovSevFJKKZVFaZJXSimlsihN8koppVQWpUleKaWUyqI0ySullFJZ1P8B+7CmNUwO3NMAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 576x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#get trained insatnce of LR\n",
    "\n",
    "lr = model_instances['logistic_regressor']\n",
    "\n",
    "#use predict proba to get assigned probabilities over test data\n",
    "y_pred_prob = lr.predict_proba(test_data)\n",
    "y_true = np.array(test_labels)\n",
    "\n",
    "#get fpr and tpr using y_pred_prob and y_true \n",
    "fpr, tpr = get_fpr_tpr(y_true, y_pred_prob[:, 0]) # for only positive class\n",
    "\n",
    "\n",
    "#plot the roc \n",
    "plot_roc(fpr, tpr)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Log Loss/ Binary cross entropy loss calculation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 283,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.469983074748027"
      ]
     },
     "execution_count": 283,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#get trained insatnce of LR\n",
    "\n",
    "lr = model_instances['logistic_regressor']\n",
    "\n",
    "#use predict proba to get assigned probabilities over test data\n",
    "y_pred_prob = lr.predict_proba(test_data)\n",
    "y_true = np.array(test_labels)\n",
    "log_loss(y_true, y_pred_prob[:,1])\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is all about performance evaluation metrics in case of classification related use cases. I have tried to implement everything related with metrics from scratch. The list of metrics mentions here is not exhaustive there a lot of metrics as well. I have mentioned most frquent metrics that we use across all type of problems.\n",
    "\n",
    "In this notebook we covered following topics:\n",
    "\n",
    " - Confusion Matrix \n",
    " - Accuracy Score\n",
    " - Precision Score\n",
    " - F-1 Score\n",
    " - Recall Score\n",
    " - Log Loss/ Binary Cross Entropy Loss\n",
    " - Area Under ROC curve\n",
    " -  Classification report"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}