clustering_metrics.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1 align='center'> Performance Metrics for Clustering Problems </h1>\n",
    "\n",
    "In this note book, I am going to illsutrate how we can evaluate the ML models deployed for clustering task using different kind of evaluation metrics.\n",
    "\n",
    "For illsutareting purpose I have collected the data from kaggle. I will be doing all the analysis over them only. I have attach the link for each dataset, you can also download the same.\n",
    "\n",
    "\n",
    "For the implemetations of these metrics, I am using following tools and frameworks:\n",
    "- Python - as a primary language\n",
    "- Pandas - as an analytical engine for processing the data\n",
    "- numpy - for computation using numpy arrays\n",
    "- matplotlib - for plotting the figures\n",
    "- sklearn - for implememting the metrics\n",
    "- seaborn - for graph plotting \n",
    "\n",
    "Note that I will be implementing all the metrices from scratch.\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's write some generic code which will be used throughout this notebook.\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn\n",
    "import sklearn"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Loader\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#function for loading a csv file from specified path\n",
    "\n",
    "def data_loader(path):\n",
    "    \n",
    "    #use pandas to load the data from path\n",
    "    df = pd.read_csv(path)\n",
    "    \n",
    "    return df"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "<h2 align='center'>Market Basket Analysis</h2>\n",
    "\n",
    "\n",
    "**Problem Statement :**\n",
    "\n",
    "You own the mall and want to understand the customers like who can be easily converge [Target Customers] so that the sense can be given to marketing team and plan the strategy accordingly.\n",
    "\n",
    "**About Data :** \n",
    "\n",
    "You are owing a supermarket mall and through membership cards , you have some basic data about your customers like Customer ID, age, gender, annual income and spending score. \n",
    "Spending Score is something you assign to the customer based on your defined parameters like customer behavior and purchasing data.\n",
    "\n",
    "**Columns Info:**\n",
    "\n",
    " The following describes the dataset columns:\n",
    "\n",
    "- **CustomerID:** Unique Id for each customer\n",
    "- **Gender:** Gender of a customer\n",
    "- **Age:** Age of a customer\n",
    "- **Annual Income (k$):** Anuual Income of each customer in 1000's\n",
    "- **Spending Score (1-100):** Score assigned by the mall based on customer behavior and spending nature\n",
    "\n",
    "**Variable of Interest Seggregation**\n",
    "\n",
    "This is a clustering problem we don't have any variable attach with data for prediction. \n",
    "\n",
    "**Algorithms to be used :**\n",
    "\n",
    "Since this is a regression problem I will be using `K-Means`, `Hierarchical clustering`, `DBSCAN`, `Gausian Mixture Model(GMM)` and `Agglomerative clustering` which are based on Regression Algorithms for modeling pourpose.\n",
    "\n",
    "**Evaluations metrics :**\n",
    "\n",
    "Since this is a clustering problem, I will be using following metrics for evaluating output of the predictive models.\n",
    "\n",
    "- Cluster Purity\n",
    "- Homogeneity, completeness, and V-measur\n",
    "- Rand Index\n",
    "- Adjusted Rand Index\n",
    "- Silhouette score\n",
    "- Dun Index\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Always look at your data`\n",
    "## Load the data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I have downloaded the data from above link and have stored the same in my local file system. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CustomerID</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Age</th>\n",
       "      <th>Annual Income (k$)</th>\n",
       "      <th>Spending Score (1-100)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Male</td>\n",
       "      <td>19</td>\n",
       "      <td>15</td>\n",
       "      <td>39</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   CustomerID Gender  Age  Annual Income (k$)  Spending Score (1-100)\n",
       "0           1   Male   19                  15                      39"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#df contains all the data\n",
    "path= \"/Users/ajitkumarsingh/Desktop/Data-Science-Interview-Questions/performance-metrics/data/Mall_Customers.csv\"\n",
    "df = data_loader(path)\n",
    "\n",
    "# show first row\n",
    "df.head(1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic Exploratory Data Analysis (EDA)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "columns name : CustomerID, Gender, Age, Annual Income (k$), Spending Score (1-100)\n",
      "\n",
      "total number of columns : 5\n",
      "\n",
      "Number of rows in data : 200\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#columns name in churn df\n",
    "print(f\"columns name : {', '.join(df.columns.tolist())}\\n\")\n",
    "\n",
    "#total number of columns\n",
    "print(f\"total number of columns : {len(df.columns)}\\n\")\n",
    "\n",
    "#total count of the data\n",
    "print(f\"Number of rows in data : {len(df)}\\n\")\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have 14 attributes and 10000 rows in the churn data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CustomerID                0\n",
       "Gender                    0\n",
       "Age                       0\n",
       "Annual Income (k$)        0\n",
       "Spending Score (1-100)    0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#number of nulls per attributes\n",
    "df.isnull().sum()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We don't have any null values in of the columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CustomerID                200\n",
       "Gender                      2\n",
       "Age                        51\n",
       "Annual Income (k$)         64\n",
       "Spending Score (1-100)     84\n",
       "dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# number of unique values\n",
    "df.nunique()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From unique values of `CustomerID` it seems like Primary key in the dataset. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CustomerID                 int64\n",
       "Gender                    object\n",
       "Age                        int64\n",
       "Annual Income (k$)         int64\n",
       "Spending Score (1-100)     int64\n",
       "dtype: object"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# data types\n",
    "df.dtypes"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All columns seems to have `numeric` data type except `Gender`.\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature Selection\n",
    "\n",
    "This is the crucial part of a ML modeling. \n",
    "\n",
    "For time being I am asssuming all the columns present in `df` are equaly important and are key driving factor in the target value prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CRIM</th>\n",
       "      <th>ZN</th>\n",
       "      <th>INDUS</th>\n",
       "      <th>CHAS</th>\n",
       "      <th>NOX</th>\n",
       "      <th>RM</th>\n",
       "      <th>AGE</th>\n",
       "      <th>DIS</th>\n",
       "      <th>RAD</th>\n",
       "      <th>TAX</th>\n",
       "      <th>PTRATIO</th>\n",
       "      <th>B</th>\n",
       "      <th>LSTAT</th>\n",
       "      <th>MEDV</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.00632</td>\n",
       "      <td>18.0</td>\n",
       "      <td>2.31</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.538</td>\n",
       "      <td>6.575</td>\n",
       "      <td>65.2</td>\n",
       "      <td>4.09</td>\n",
       "      <td>1</td>\n",
       "      <td>296</td>\n",
       "      <td>15.3</td>\n",
       "      <td>396.9</td>\n",
       "      <td>4.98</td>\n",
       "      <td>24.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      CRIM    ZN  INDUS  CHAS    NOX     RM   AGE   DIS  RAD  TAX  PTRATIO  \\\n",
       "0  0.00632  18.0   2.31   0.0  0.538  6.575  65.2  4.09    1  296     15.3   \n",
       "\n",
       "       B  LSTAT  MEDV  \n",
       "0  396.9   4.98  24.0  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "redundant_features = []\n",
    "\n",
    "#drop redundant features(in this case it is empty)\n",
    "df = df.drop(redundant_features, axis=1)\n",
    "\n",
    "#show 1 row\n",
    "df.head(1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Features Extraction\n",
    "\n",
    "Feature extraction is a process in machine learning and data analysis that involves transforming raw data or input features into a set of representative features that are more meaningful and informative for a particular task or problem. It is an important step in the feature engineering process, which involves selecting, transforming, and creating new features from raw data to improve the performance and interpretability of machine learning models.\n",
    "\n",
    "This this problem we noted that we do have some columns which have like high cardinality and we can reduce cardinality using `binning` process.\n",
    "\n",
    "In `binning` method, we convert `numeric` columns to `categorical` columns to reduce overall cardinality of an attribute.\n",
    "\n",
    "Let's say, if the number of distinct values in a column is greater than the `50%` of the total count it means that column has high cardinality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CRIM       485\n",
       "ZN          27\n",
       "INDUS       77\n",
       "CHAS         3\n",
       "NOX         81\n",
       "RM         446\n",
       "AGE        349\n",
       "DIS        412\n",
       "RAD          9\n",
       "TAX         66\n",
       "PTRATIO     46\n",
       "B          357\n",
       "LSTAT      439\n",
       "MEDV       229\n",
       "dtype: int64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# number of unique values\n",
    "df.nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Columns with high cardinalities are CRIM, RM, AGE, DIS, B, LSTAT\n"
     ]
    }
   ],
   "source": [
    "#columns with high cardinality\n",
    "total_count = df.shape[0]\n",
    "unique_counts = df.nunique()\n",
    "\n",
    "#filter for high cardinality\n",
    "high_cardinality_columns = unique_counts[unique_counts>0.5*total_count].index.tolist()\n",
    "\n",
    "print(\"Columns with high cardinalities are {}\".format(', '.join(high_cardinality_columns)))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Discreatize the categorical columns one by one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "#dicreatization based on quatiles\n",
    "#classify crimes(CRIM) into HIGH, Medium and Low\n",
    "df['CRIM_BINNED'] = pd.qcut(df['CRIM'],q=3,labels=['Low', 'Moderate', \"High\"])\n",
    "\n",
    "#bin the age coliumn into Young, Adult and Old\n",
    "df['AGE_BINNED'] = pd.qcut(df['AGE'],q=3,labels=['Young', 'Adult', \"Old\"])\n",
    "\n",
    "#bin the average number of room per dwelling(RM) into Low, Moderate and High\n",
    "df['RM_BINNED'] = pd.qcut(df['RM'], q=3, labels=['Low', 'Moderate', \"High\"])\n",
    "\n",
    "#bin distance column (DIS) into Very Near, Near, Far and Very Far\n",
    "df['DIS_BINNED'] = pd.qcut(df['DIS'], q=4, labels=['Very Near', 'Near', \"Far\", \"Very Far\"])\n",
    "\n",
    "#bin Black population proportion\n",
    "df['B_BINNED'] = pd.qcut(df['B'], q=2, labels=['Low', 'High'])\n",
    "\n",
    "#bin % of lower status of the popultion (LSTAT)\n",
    "df['LSTAT_BINNED'] = pd.qcut(df['LSTAT'], q=3, labels=['Low', 'Medium','High'])\n",
    "\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note** that there might be many permutation and combinations here like what is optimal number of categories a column should be binned. For time being I have used common intution for transforming the above high cardinality columns.\n",
    "\n",
    "Also note that while binning we are replacing the numeric values to some non-numeric data type mainly strings here. We need to encode them before we feed the data to our ML models.\n",
    "\n",
    "There are several techniques for encoding the values into numeric once. Here I am using `One Hot Encoding` method to label the categorical values. In pandas we have `pd.get_dummies(df, columns=column_list)` to get dummies numeric values for non-numeric values. \n",
    "\n",
    "This method in pandas append new columns and the number of these new columns are only dependent on the number of `unique` values in the original columns. For example suppose we have a column name `Gender` and it has two distinct values `Male` and `Female`, this methods will create two new columns namely `Gender_Male` and `Gender_Female` and values of `Gender_Male` will be `0` where `Gender` column is `Female` and `1` if the value is `Male`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "#remove the high catdinality columns now\n",
    "df_binned = df.drop(high_cardinality_columns, axis=1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how many unique values we have per attributes after binning high cardinality columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ZN               27\n",
       "INDUS            77\n",
       "CHAS              3\n",
       "NOX              81\n",
       "RAD               9\n",
       "TAX              66\n",
       "PTRATIO          46\n",
       "MEDV            229\n",
       "CRIM_BINNED       3\n",
       "AGE_BINNED        3\n",
       "RM_BINNED         3\n",
       "DIS_BINNED        4\n",
       "B_BINNED          2\n",
       "LSTAT_BINNED      3\n",
       "dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#number of unique values\n",
    "df_binned.nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Encode the categorica columns now using One Hot Encoding\n",
    "suffix = '_BINNED'\n",
    "binned_columns = list(filter(lambda x : x[-len(suffix):]==suffix, df.columns))\n",
    "\n",
    "#use pd.get_dummies to encode the above columns\n",
    "df_encoded = pd.get_dummies(df, columns=binned_columns)\n",
    "\n",
    "#now let's see if we have new columns "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX',\n",
       "       'PTRATIO', 'B', 'LSTAT', 'MEDV', 'CRIM_BINNED_Low',\n",
       "       'CRIM_BINNED_Moderate', 'CRIM_BINNED_High', 'AGE_BINNED_Young',\n",
       "       'AGE_BINNED_Adult', 'AGE_BINNED_Old', 'RM_BINNED_Low',\n",
       "       'RM_BINNED_Moderate', 'RM_BINNED_High', 'DIS_BINNED_Very Near',\n",
       "       'DIS_BINNED_Near', 'DIS_BINNED_Far', 'DIS_BINNED_Very Far',\n",
       "       'B_BINNED_Low', 'B_BINNED_High', 'LSTAT_BINNED_Low',\n",
       "       'LSTAT_BINNED_Medium', 'LSTAT_BINNED_High'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#now see the columns name\n",
    "\n",
    "df_encoded.columns"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clearly `pd.get_dummies` method has appended some new columns in the data.\n",
    "\n",
    "Just make sure all the columns are of `numeric` data type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CRIM                    float64\n",
       "ZN                      float64\n",
       "INDUS                   float64\n",
       "CHAS                    float64\n",
       "NOX                     float64\n",
       "RM                      float64\n",
       "AGE                     float64\n",
       "DIS                     float64\n",
       "RAD                       int64\n",
       "TAX                       int64\n",
       "PTRATIO                 float64\n",
       "B                       float64\n",
       "LSTAT                   float64\n",
       "MEDV                    float64\n",
       "CRIM_BINNED_Low           uint8\n",
       "CRIM_BINNED_Moderate      uint8\n",
       "CRIM_BINNED_High          uint8\n",
       "AGE_BINNED_Young          uint8\n",
       "AGE_BINNED_Adult          uint8\n",
       "AGE_BINNED_Old            uint8\n",
       "RM_BINNED_Low             uint8\n",
       "RM_BINNED_Moderate        uint8\n",
       "RM_BINNED_High            uint8\n",
       "DIS_BINNED_Very Near      uint8\n",
       "DIS_BINNED_Near           uint8\n",
       "DIS_BINNED_Far            uint8\n",
       "DIS_BINNED_Very Far       uint8\n",
       "B_BINNED_Low              uint8\n",
       "B_BINNED_High             uint8\n",
       "LSTAT_BINNED_Low          uint8\n",
       "LSTAT_BINNED_Medium       uint8\n",
       "LSTAT_BINNED_High         uint8\n",
       "dtype: object"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#data types of final df\n",
    "df_encoded.dtypes"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We don't have any non numeric data type columns."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modeling\n",
    "\n",
    "### Train/Test Split\n",
    "\n",
    "For evaluating a model, we need some data to test it on, once training part is done. We usually split the data into two parts i.e train and test. \n",
    "On train data we update the models parameters and on test data we see how the trained model is performing. \n",
    "\n",
    "For spliting the data, I am using `train_test_split` function of module `sklearn.model_selection`. \n",
    "\n",
    "Also we need to segregate the `label` from the rest of the data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total count : 506\n",
      "\n",
      "Train Data count : 430\n",
      "\n",
      "Test Data count : 76\n",
      "\n",
      "Train Labels count : 430\n",
      "\n",
      "Test Labels count : 76\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "#split the data into test and train in 85:15 ratio and drop the label from the rest of the data\n",
    "\n",
    "train_data, test_data, train_labels, test_labels = train_test_split(\n",
    "        df_encoded.drop(['MEDV'], axis=1), \n",
    "        df_encoded['MEDV'], \n",
    "        test_size=0.15, \n",
    "        random_state=42\n",
    "    )\n",
    "\n",
    "#count after split\n",
    "print(f\"Total count : {len(df_encoded)}\\n\")\n",
    "print(f\"Train Data count : {len(train_data)}\\n\")\n",
    "print(f\"Test Data count : {len(test_data)}\\n\")\n",
    "print(f\"Train Labels count : {len(train_labels)}\\n\")\n",
    "print(f\"Test Labels count : {len(test_labels)}\\n\")\n",
    "\n",
    "# number of rows and labels should match\n",
    "assert len(train_data)==len(train_labels)\n",
    "assert len(test_data)==len(test_labels)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have stored train and test data in `train_data`, `test_data` and train and test labels in `train_labels`, `test_labels`. I will be using them during training and testing time accordingly."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Selection\n",
    "\n",
    "We have numerous models out there to solve same type of poblems. But the interesting part is like we don't know which model will be best fitting our dataset and perform well on the test dataset. We need to make choice here and to do that we need to evaluate these models one by one using some performance metrics.\n",
    "\n",
    "I will be training following models and evaluating their performance on test data\n",
    "\n",
    "- Linear Regressor\n",
    "- Random Forest Regressor\n",
    "- Gradient Boosting Regressor\n",
    "- Neighrest Neighbour Regressor\n",
    "\n",
    "Here, I will be using `sklearn` module of `scikit-learn` library to implement the above mentioned models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "#import the above models\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.neighbors import KNeighborsRegressor\n",
    "from sklearn.ensemble import GradientBoostingRegressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "#create a list that contains one instances of each model type\n",
    "\n",
    "model_instances = {\n",
    "          'linear_regressor':LinearRegression(),\n",
    "          'random_forest_regressor': RandomForestRegressor(n_estimators=4, max_depth=4, max_features=4), \n",
    "          'knn_regressor':KNeighborsRegressor(), \n",
    "          'gradient_boosting_regressor':GradientBoostingRegressor(n_estimators=4, max_depth=4)\n",
    "          }\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Training\n",
    "\n",
    "In the above step we have created instances of each model and stored in `model_instances` dict type variable. Now we need to train these instances by feeding the train distribution. \n",
    "\n",
    "Once the training is done save the trained instances in the same dict `model_instances`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "#train the models on training set and save the trained model for evaluation\n",
    "\n",
    "for model_name, model in model_instances.items():\n",
    "\n",
    "    #fit the model with train_data and train_labels\n",
    "    model.fit(train_data,train_labels)\n",
    "    \n",
    "    #save the trained model\n",
    "    model_instances[model_name] = model\n",
    "    "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Metrics For Model Evaluation\n",
    "\n",
    "Now we have trained our models and they are ready to make predictions over test dataset. To make sure the model is predicting meaningfull values not random output we need some metrics to evaluate the output. This is where performance metrics come into picture. \n",
    "\n",
    "Let's implement some of the most widely used evaluation metrics for regresion problems from scratch."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Performance Metrics Implementation\n",
    "\n",
    "##### Mean Squared Error (MSE)\n",
    "\n",
    "It estimated as average squared differences of predicted values and the actual values. For an ideal model `mse` would be equal to `0`. Lower `mse` depicts more accurate predictions. It is expressed in squared units of the target variable\n",
    "\n",
    "Mathematically it is expressed as :\n",
    "\n",
    "$$\n",
    "    MSE = \\frac{1}{n} \\sum_{i=1}^{n} (y_{\\text{pred}, i} - y_{\\text{true}, i})^2\n",
    "$$\n",
    "\n",
    "Note that the above expression is `differential` and hence can we calculate `gradient` for optimizing purposes.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "    Calculate Mean Squared Error (MSE) between true and predicted values.\n",
    "\n",
    "    Parameters:\n",
    "        -- y_true (numpy array or list): True values\n",
    "        -- y_pred (numpy array or list): Predicted values\n",
    "\n",
    "    Returns:\n",
    "        -- mse (float): Mean Squared Error\n",
    "    \"\"\"\n",
    "\n",
    "def mean_squared_error(y_true, y_pred):\n",
    "\n",
    "    #convert then into umpy array if not already\n",
    "    y_true = np.array(y_true)\n",
    "    y_pred = np.array(y_pred)\n",
    "    #squared the differences\n",
    "    squared_diff = (y_true - y_pred)**2\n",
    "    #find the avg of squared_diff\n",
    "    mse = np.mean(squared_diff)\n",
    "    return mse\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Cautions**\n",
    "\n",
    "- **Sensitive to outliers:** MSE is sensitive to outliers because if a predicted value (`y_pred_i`) is an outlier, the squared difference with the corresponding true value (`y_true_i`) will be larger, potentially resulting in an inflated MSE.\n",
    "\n",
    "- **Imbalanced errors:** MSE treats all errors equally, regardless of their magnitude or direction. This means that overestimation and underestimation errors are weighted equally, even though they may have different implications or costs in practice.\n",
    "\n",
    "- **Not robust to non-Gaussian errors:** MSE assumes that the errors follow a Gaussian (normal) distribution, which may not always be the case in real-world scenarios. If the errors are not normally distributed, MSE may not accurately reflect the model's performance.\n",
    "\n",
    "- **Lack of sensitivity to small errors:** MSE may not capture small errors well, as it squares the differences between predicted and true values. This can lead to a model with good MSE but poor performance in capturing small errors, which may be important in certain applications.\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Root Mean Squared Error(RMSE)\n",
    "\n",
    "This is root of mean squared error we talked about. The good thing about this metric is like the unit is in sync with target variable and hence can be used for comparing multiple models at once. Since `RMSE` is expressed in the same units as the dependent variable, making it easy to interpret in the context of the original data\n",
    "\n",
    "The expression of `RMSE` is differentiable just like we have for `MSE`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "    Calculate Root Mean Squared Error (RMSE) between true and predicted values.\n",
    "\n",
    "    Parameters:\n",
    "        -- y_true (numpy array or list): True values\n",
    "        -- y_pred (numpy array or list): Predicted values\n",
    "\n",
    "    Returns:\n",
    "        -- mse (float): Mean Squared Error\n",
    "    \"\"\"\n",
    "\n",
    "def root_mean_squared_error(y_true, y_pred):\n",
    "\n",
    "    #convert then into umpy array if not already\n",
    "    y_true = np.array(y_true)\n",
    "    y_pred = np.array(y_pred)\n",
    "    #squared the differences\n",
    "    squared_diff = (y_true - y_pred)**2\n",
    "    #find the avg of squared_diff\n",
    "    mse = np.mean(squared_diff)\n",
    "    return np.sqrt(mse)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Cautions**\n",
    "\n",
    "- Not robust to outliers\n",
    "- Always positive so not usefull when negative errors make sense.\n",
    "- Bias towards large values\n",
    "- Can be affected by sample size"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Mean Absolute Error (MAE)\n",
    "\n",
    "It is the avearge of absolute difference of predicted and actual values. It is more robust to outliers unlike MSE and RMSE.\n",
    "\n",
    "Mathematically, It can be expressed as:\n",
    "\n",
    "$$\n",
    "\\text{MAE} = \\frac{1}{n} \\sum_{i=1}^{n} |y_{\\text{true}_i} - y_{\\text{pred}_i}|\n",
    "$$\n",
    "\n",
    "Note that it is not differential at zero because of the mod `|.|` function.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "    Calculate Mean Absolute Error (MAE) between true and predicted values.\n",
    "\n",
    "    Parameters:\n",
    "        -- y_true (numpy array or list): True values\n",
    "        -- y_pred (numpy array or list): Predicted values\n",
    "\n",
    "    Returns:\n",
    "        -- mae (float): Mean Absolute Error\n",
    "    \"\"\"\n",
    "\n",
    "def mean_absolute_error(y_true, y_pred):\n",
    "\n",
    "    #convert then into umpy array if not already\n",
    "    y_true = np.array(y_true)\n",
    "    y_pred = np.array(y_pred)\n",
    "    #squared the differences\n",
    "    absolute_diff = np.absolute(y_true - y_pred)\n",
    "    #find the avg of squared_diff\n",
    "    mae = np.mean(absolute_diff)\n",
    "    return mae"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Coeffiecient Of Determination (R^2)\n",
    "\n",
    "It is a statistical measure used to evaluate the `goodness of fit` of a regression model. It represents the proportion of the variance in the dependent variable that is explained by the independent variables in the model. It is more usefull when we have linear relationship between dependent and independent variables.\n",
    "\n",
    "R-squared ranges from `0` to `1`, with `1` indicating a perfect fit and `0` indicating no fit at all.\n",
    "\n",
    "Mathematially, It can be expressed as:\n",
    "\n",
    "$$\n",
    "\\text{R-squared} = 1 - \\frac{SSR}{SST}\n",
    "$$\n",
    "\n",
    "Where :\n",
    "\n",
    "- *SSR(Sum of Squared Residuals)* represents the sum of the squared differences between the predicted values and the actual values of the dependent variable.\n",
    "\n",
    "- *SST(Sum of Squared Total)* represents the sum of the squared differences between the actual values of the dependent variable and the mean of the dependent variable.\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "    Calculate R-squared value given predicted value and actual label\n",
    "\n",
    "    Parameters:\n",
    "        -- y_true (numpy array or list): True values\n",
    "        -- y_pred (numpy array or list): Predicted values\n",
    "\n",
    "    Returns:\n",
    "        -- r^2 (float): Coefficient of determination(R^2)\n",
    "    \"\"\"\n",
    "\n",
    "def r2_score(y_true, y_pred):\n",
    "\n",
    "    #convert then into umpy array if not already\n",
    "    y_true = np.array(y_true)\n",
    "    y_pred = np.array(y_pred)\n",
    "    #sum of squared residuals\n",
    "    SSR = np.sum((y_true - y_pred)**2)\n",
    "    #sum of squared total\n",
    "    y_avg = np.mean(y_true)\n",
    "    SST = np.sum((y_avg-y_true)**2)\n",
    "    return 1 - (float(SSR)/float(SST))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note :** `R-squared` value can be negative due to outliers i.e when `MSE(model) > MSE(Baseline)`"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Limitations:\n",
    "\n",
    "- **Lack of information on prediction accuracy:** R-squared does not provide information about the accuracy of the model in making predictions. A high R-squared value does not necessarily guarantee accurate predictions, as the model may still have residual errors or may be overfitting the data.\n",
    "\n",
    "- **Sensitivity to outliers:** R-squared can be affected by outliers in the data. Outliers can disproportionately influence the sum of squared residuals (SSR) component of R-squared, leading to an inflated or deflated R-squared value. \n",
    "\n",
    "- **Inability to determine causality:** R-squared does not provide information about causality. Even if a regression model has a high R-squared value, it does not necessarily mean that the independent variables are causing the observed variation in the dependent variable. There may be other confounding variables or omitted variables that are driving the relationship.\n",
    "\n",
    "- **Limited to linear relationships:** R-squared is a measure of the goodness of fit of a linear regression model, which assumes a linear relationship between the dependent and independent variables. If the relationship is non-linear, R-squared may not accurately reflect the goodness of fit."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Adjusted R-squared \n",
    "\n",
    "`R-squared` suffers from problem that the scores improve on increasing number of predictors even if the additional predictors do not significantly improve the model's ability to explain the variation in the dependent variable. This can result in overfitting, where the model appears to fit the data well but may not generalize well to new data.\n",
    "\n",
    "To overcome the above problem associated with `R-squared`, `Adjusted R-squared` adjusts for the number of predictors in the model, penalizing models with more predictors if the additional predictors do not contribute significantly to the model's ability to explain the variation in the dependent variable.\n",
    "\n",
    "Mathematically It can be expressed as:\n",
    "\n",
    "$$\n",
    "\\text{Adjusted R-squared} = 1 - \\left( \\frac{{(1 - R^2) \\cdot (n - 1)}}{{n - k - 1}} \\right)\n",
    "\n",
    "$$\n",
    "\n",
    "Where:\n",
    "\n",
    "- R-squared (Goodness of Fit): It represents the proportion of variance in the dependent variable explained by the regression model.\n",
    "\n",
    "- n: Total number of observations in the dataset.\n",
    "\n",
    "- k: number of predictors(independent variables) in the regression model.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "    Calculate Adjusted R-squared value given predicted and actual label and number of predictors\n",
    "\n",
    "    Parameters:\n",
    "        -- y_true (numpy array or list): True values\n",
    "        -- y_pred (numpy array or list): Predicted values\n",
    "        -- k (integer): Number of predictors\n",
    "\n",
    "    Returns:\n",
    "        -- adjusted r^2 (float): Coefficient of determination(R^2)\n",
    "    \"\"\"\n",
    "\n",
    "def adjusted_r2_score(y_true, y_pred, k):\n",
    "\n",
    "    #convert then into umpy array if not already\n",
    "    y_true = np.array(y_true)\n",
    "    y_pred = np.array(y_pred)\n",
    "\n",
    "    #number of observations\n",
    "    n = len(y_true)\n",
    "    #sum of squared residuals\n",
    "    SSR = np.sum((y_true - y_pred)**2)\n",
    "    #sum of squared total\n",
    "    y_avg = np.mean(y_true)\n",
    "    SST = np.sum((y_avg-y_true)**2)\n",
    "    \n",
    "    #r2_score\n",
    "    r2_score = 1 - (float(SSR)/float(SST))\n",
    "    #adjusted r2 square\n",
    "    r2_score_adj = 1 - float((1-r2_score**2)*(n-1))/float(n - k - 1)\n",
    "    \n",
    "    return r2_score_adj"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note** `Adjusted R_squared` value is sensitive to sample size. It means the value may decrease as we increase the sample size even if the model is performing better."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Prediction\n",
    "\n",
    "Now we have defined the performance metrics, we can use them to evaluate the model predctions.\n",
    "\n",
    "From Train/Test Split Section we have :\n",
    "\n",
    "- `train_data`, `test_data`, `train_labels` and `test_labels` we can use them for models performance evaluations.\n",
    "\n",
    "From Model Training section we have : \n",
    "\n",
    "- trained instances of `Linear Regressor`, `KNN Regressor`, `Random Forest Regressor` and `Gradient Booster Regressor` in `model_instances` dictionary. We need to feed the test data to these instances  to get the predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "    get predictions from test instances\n",
    "\n",
    "    Args:\n",
    "        trained_model () : trained instance\n",
    "        test_data (pandas dataframe) : test data without label\n",
    "\n",
    "    Returns:\n",
    "        list of predictions against each row in test data\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "def make_predictions(trained_model, test_data):\n",
    "\n",
    "    return trained_model.predict(test_data)\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "#store predictions against each model\n",
    "\n",
    "y_pred_per_model = {}\n",
    "\n",
    "for model_name, model_instance in model_instances.items():\n",
    "    \n",
    "    y_pred_per_model[model_name] = make_predictions(model_instance, test_data) \n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have stored model name and it's predictions over test data in `y_pred_per_model` and we have also `test_labels` which is like ground truths."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Evaluation\n",
    "\n",
    "This like the last step we need to perform to compare which classification algorithm is performating relatively better over test distribution. In the above sctions, We are done with implementaion of performance metrics now we can use them to evaluate the trained models.\n",
    "\n",
    "Let's create a report in form of a data frame where headers are `model_name` and the performance metrics we discussed above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_squared_error</th>\n",
       "      <th>root_mean_squared_error</th>\n",
       "      <th>mean_absolute_error</th>\n",
       "      <th>r2_score</th>\n",
       "      <th>adjusted_r2_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>linear_regressor</th>\n",
       "      <td>10.948576</td>\n",
       "      <td>3.308863</td>\n",
       "      <td>2.515675</td>\n",
       "      <td>0.832344</td>\n",
       "      <td>0.476358</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>random_forest_regressor</th>\n",
       "      <td>13.647620</td>\n",
       "      <td>3.694269</td>\n",
       "      <td>2.595294</td>\n",
       "      <td>0.791013</td>\n",
       "      <td>0.361992</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>knn_regressor</th>\n",
       "      <td>23.958647</td>\n",
       "      <td>4.894757</td>\n",
       "      <td>3.780789</td>\n",
       "      <td>0.633120</td>\n",
       "      <td>-0.021293</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>gradient_boosting_regressor</th>\n",
       "      <td>34.745582</td>\n",
       "      <td>5.894538</td>\n",
       "      <td>4.333064</td>\n",
       "      <td>0.467939</td>\n",
       "      <td>-0.331306</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                             mean_squared_error  root_mean_squared_error  \\\n",
       "linear_regressor                      10.948576                 3.308863   \n",
       "random_forest_regressor               13.647620                 3.694269   \n",
       "knn_regressor                         23.958647                 4.894757   \n",
       "gradient_boosting_regressor           34.745582                 5.894538   \n",
       "\n",
       "                             mean_absolute_error  r2_score  adjusted_r2_score  \n",
       "linear_regressor                        2.515675  0.832344           0.476358  \n",
       "random_forest_regressor                 2.595294  0.791013           0.361992  \n",
       "knn_regressor                           3.780789  0.633120          -0.021293  \n",
       "gradient_boosting_regressor             4.333064  0.467939          -0.331306  "
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#test labels\n",
    "y_true = np.array(test_labels)\n",
    "\n",
    "#lets create a dict which contains model_name and the key pair of metric name and its value\n",
    "result_dict = {}\n",
    "\n",
    "#iterate over y_pred_per_model to get model name and corresponding predicted label\n",
    "for model_name, y_pred in y_pred_per_model.items():\n",
    "\n",
    "    #find mse\n",
    "    mse = mean_squared_error(y_true, y_pred)\n",
    "    \n",
    "    #find rmse\n",
    "    rmse = root_mean_squared_error(y_true, y_pred)\n",
    "\n",
    "    #find mae\n",
    "    mae = mean_absolute_error(y_true, y_pred)\n",
    "\n",
    "    #find r2_score\n",
    "    r2_value = r2_score(y_true, y_pred)\n",
    "\n",
    "    #find adjusted r2_score\n",
    "    adjusted_r2_value = adjusted_r2_score(y_true, y_pred, 31)\n",
    "\n",
    "    result_dict[model_name] = {\"mean_squared_error\":mse, \"root_mean_squared_error\":rmse, \"mean_absolute_error\":mae, \"r2_score\":r2_value, \"adjusted_r2_score\":adjusted_r2_value}\n",
    "\n",
    "\n",
    "#convert result_dict into pandas data frame\n",
    "result_df = pd.DataFrame.from_dict(result_dict, orient=\"index\")\n",
    "\n",
    "#disply the result df\n",
    "result_df.head(20)\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Observations\n",
    "\n",
    "From above table, we can deduce that `linear regression` is performing best among all the models across all the metrics. 2nd best model is `random forest regressor`.\n",
    "\n",
    "Worst performance is given by `Gradient Boost Regressor` across all the metrics.\n",
    "\n",
    "\n",
    "Note that here I am using the default settings of each model and not doing any hyper parameters tuning. So the above observation is done on default setup of each model and hence can change if we do hyper parameter tunings or optimizations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}