08_Naive_Bayes.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "08 Naive Bayes.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyMCOD5asfe2dthJgHQECqs+",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/sandipanpaul21/KNN-Naive-Bayes-in-Python/blob/main/08_Naive_Bayes.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Naive Bayes** works both for Classification and Regression Algorithim and Bayes Theorem is used to find Conditional Probabilities \n",
        "\n",
        "**How does the Naive Bayes classifier work?**\n",
        "\n",
        "**Why is Naive Bayes naive?**\n",
        "\n",
        "- Naive Bayes' underlying assumption is that the predictors (attributes / independent variables) are independent of each other. \n",
        "- This is a big assumption because it is easy to show that there is often at least some correlation between variables in real life. \n",
        "- It is precisely this assumption of independence that makes Bayes classification “naive.”\n",
        "\n",
        "But Naive Bayes is a fast algorithm since it scales easily to include many predictors without having to handle multi-dimensional correlations.\n",
        "\n",
        "**Conditional probabilities**\n",
        "- To understand Naive Bayes, we first need to understand conditional probabilities. \n",
        "- For that, let’s use the below example.\n",
        "  - Assume we have a bucket filled with red and black balls. \n",
        "  - In total, there are 15 balls: 7 red and 8 black.\n",
        "  - The probability of randomly picking a red ball out of the bucket is 7/15. You can write it as P(red) = 7/15.\n",
        "  - If we were to draw balls one at a time without replacing them, what is the probability of getting a black ball on a second attempt after drawing a red one on the first attempt?\n",
        "  - You can see that the above question is worded to provide us with the condition that needs to be satisfied first before the second attempt is made.\n",
        "  - That condition says that a red ball must be drawn during the first attempt.\n",
        "  - As stated earlier, the probability of getting a red ball on the first attempt (P(red)) is 7/15. That leaves 14 balls inside a bucket with 6 red and 8 black. Hence, the probability of getting a black ball next is 8/14 = 4/7.\n",
        "  - We can write this as a conditional probability:\n",
        "P(black|red) = 4/7. (probability of black given red)\n",
        "  - Similarly, P(red and black) = P(red) * P(black|red) = 7/15 * 8/14 = 4/15.\n",
        "  - Similarly, P(black and red) = P(black) * P(red|black) = 8/15 * 7/14 = 4/15.\n",
        "\n",
        "**Bayes’ theorem**\n",
        "- The Bayes’ theorem helps us calculate conditional probabilities of an event when we know the likelihood of a reverse event. \n",
        "- Using the example above, we would write it as follows:\n",
        "\n",
        "  P(Black|Red) = [P(Black) * P(Red|Black)] / P(Red)\n",
        "  \n",
        "  This will also give result as 4/15\n",
        "\n",
        "**Naive Bayes classifier**\n",
        "  \n",
        "    P(A|B) = [P(B|A) * P(A)] / P(B)\n",
        "\n",
        "    where,\n",
        "\n",
        "    P(A) : prior probability, probability which we get before any additional information is obtained\n",
        "\n",
        "    P(B) : evidence\n",
        "\n",
        "    P(A|B) : posterior probability, probability which we get or revised after any additional information is obtained or also known as conditional probability of event A occuring for the event B which has already occured\n",
        "\n",
        "    p(B|A) : likelihood, \n",
        "\n",
        "    Posterior = Likelihood * Prior / Evidence\n",
        "\n",
        "\n",
        "  - Example,\n",
        "    - P(Fire) is the Prior\n",
        "    - P(Smoke|Fire) is the Likelihood\n",
        "    - P(Smoke) is the Evidence\n",
        "    - Then according to formula,\n",
        "      \n",
        "      P(Fire|Smoke) = P(Smoke|Fire)*P(Fire)/P(Smoke)\n",
        "\n",
        "  - Another Example,\n",
        "    - P(Cloud) is the Prior\n",
        "    - P(Rain|Cloud) is the Likelihood\n",
        "    - P(Rain) is the Evidence\n",
        "    - Then according to formula,\n",
        "      \n",
        "      P(Cloud|Rain) = P(Rain|Cloud)*P(Cloud)/P(Rain)\n",
        "\n",
        "  - Here the assumption is each input variable is independent from each other\n",
        "  - By above formula, it calculate conditional probability of each class for a given instance. In other word, conditional probablity of all the variable given the class label.\n",
        "\n",
        "**Why Naive Bayes is called Naive**\n",
        "\n",
        "- Naive means show lack of experience / wisdom / judegement\n",
        "- Naive Bayes is naive because it makes the assumption that features of the dataset are indepedent of each other\n",
        "- This is a strong assumption and unrealistic for real data and thats why Naive\n",
        "\n",
        "**Disadvantages**\n",
        "- The assumption of independent features. \n",
        "- In practice, it is almost impossible that model will get a set of predictors which are entirely independent.\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "Xw8Cwyq54aIs"
      }
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aaJy5SriCZDL",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "26378a0b-a715-4981-e64a-9f4d7535d5d3"
      },
      "source": [
        "#  Libraries\n",
        "\n",
        "from sklearn import datasets\n",
        "import pandas as pd\n",
        "import statsmodels.api as sm\n",
        "# Import Gaussian Naive Bayes model\n",
        "from sklearn.naive_bayes import GaussianNB\n",
        "from sklearn import metrics\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.metrics import accuracy_score,recall_score,precision_score,f1_score\n",
        "from sklearn.metrics import confusion_matrix, classification_report, roc_auc_score\n",
        "import warnings\n",
        "warnings.filterwarnings(\"ignore\")\n",
        "import numpy as np\n",
        "from sklearn.metrics import classification_report"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning: pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.\n",
            "  import pandas.util.testing as tm\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A-2LJd1SCiKS",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "b3703b5d-35ed-45b5-9884-98ed9b70d267"
      },
      "source": [
        "# IRIS Dataset\n",
        "iriss = datasets.load_iris()\n",
        "iris = pd.DataFrame(iriss.data)\n",
        "iris.columns = iriss.feature_names\n",
        "iris['species'] = iriss.target\n",
        "iris.columns = iris.columns.str.replace(\" \",\"\")\n",
        "iris.columns = iris.columns.str.replace(\"(\",\"\")\n",
        "iris.columns = iris.columns.str.replace(\")\",\"\")\n",
        "iris.head()"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sepallengthcm</th>\n",
              "      <th>sepalwidthcm</th>\n",
              "      <th>petallengthcm</th>\n",
              "      <th>petalwidthcm</th>\n",
              "      <th>species</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>5.1</td>\n",
              "      <td>3.5</td>\n",
              "      <td>1.4</td>\n",
              "      <td>0.2</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>4.9</td>\n",
              "      <td>3.0</td>\n",
              "      <td>1.4</td>\n",
              "      <td>0.2</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>4.7</td>\n",
              "      <td>3.2</td>\n",
              "      <td>1.3</td>\n",
              "      <td>0.2</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4.6</td>\n",
              "      <td>3.1</td>\n",
              "      <td>1.5</td>\n",
              "      <td>0.2</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5.0</td>\n",
              "      <td>3.6</td>\n",
              "      <td>1.4</td>\n",
              "      <td>0.2</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   sepallengthcm  sepalwidthcm  petallengthcm  petalwidthcm  species\n",
              "0            5.1           3.5            1.4           0.2        0\n",
              "1            4.9           3.0            1.4           0.2        0\n",
              "2            4.7           3.2            1.3           0.2        0\n",
              "3            4.6           3.1            1.5           0.2        0\n",
              "4            5.0           3.6            1.4           0.2        0"
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BSad3e-YCsDd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "732d946d-3cee-4997-b68c-a2cff35f8ad8"
      },
      "source": [
        "# Target Column Distribution\n",
        "iris['species'].value_counts()"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "2    50\n",
              "1    50\n",
              "0    50\n",
              "Name: species, dtype: int64"
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5GqxQzKCCypn",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 173
        },
        "outputId": "e625e626-d4ee-46ee-9b42-6f43877d6a90"
      },
      "source": [
        "# Distribution (mean) of Independent Columns respect to Dependent Column\n",
        "iris.groupby('species').mean().round(2)"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sepallengthcm</th>\n",
              "      <th>sepalwidthcm</th>\n",
              "      <th>petallengthcm</th>\n",
              "      <th>petalwidthcm</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>species</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>5.01</td>\n",
              "      <td>3.43</td>\n",
              "      <td>1.46</td>\n",
              "      <td>0.25</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>5.94</td>\n",
              "      <td>2.77</td>\n",
              "      <td>4.26</td>\n",
              "      <td>1.33</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>6.59</td>\n",
              "      <td>2.97</td>\n",
              "      <td>5.55</td>\n",
              "      <td>2.03</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         sepallengthcm  sepalwidthcm  petallengthcm  petalwidthcm\n",
              "species                                                          \n",
              "0                 5.01          3.43           1.46          0.25\n",
              "1                 5.94          2.77           4.26          1.33\n",
              "2                 6.59          2.97           5.55          2.03"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WsRAmcb1C3aE",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "b444e580-f010-4d5f-8954-8016515a40bf"
      },
      "source": [
        "# Independent Variables\n",
        "Independent_Variable_Base_Set = iris[iris.columns[0:4]]\n",
        "Independent_Variable_Base_Set.head()"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sepallengthcm</th>\n",
              "      <th>sepalwidthcm</th>\n",
              "      <th>petallengthcm</th>\n",
              "      <th>petalwidthcm</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>5.1</td>\n",
              "      <td>3.5</td>\n",
              "      <td>1.4</td>\n",
              "      <td>0.2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>4.9</td>\n",
              "      <td>3.0</td>\n",
              "      <td>1.4</td>\n",
              "      <td>0.2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>4.7</td>\n",
              "      <td>3.2</td>\n",
              "      <td>1.3</td>\n",
              "      <td>0.2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4.6</td>\n",
              "      <td>3.1</td>\n",
              "      <td>1.5</td>\n",
              "      <td>0.2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5.0</td>\n",
              "      <td>3.6</td>\n",
              "      <td>1.4</td>\n",
              "      <td>0.2</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   sepallengthcm  sepalwidthcm  petallengthcm  petalwidthcm\n",
              "0            5.1           3.5            1.4           0.2\n",
              "1            4.9           3.0            1.4           0.2\n",
              "2            4.7           3.2            1.3           0.2\n",
              "3            4.6           3.1            1.5           0.2\n",
              "4            5.0           3.6            1.4           0.2"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X25sAaAwC_mr",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "5d0ed25f-7311-4d23-e6ec-7b29f7d89abf"
      },
      "source": [
        "# Dependent Variable\n",
        "Dependent_Variable = iris[iris.columns[-1:iris.columns.size]]\n",
        "Dependent_Variable.head()"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>species</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   species\n",
              "0        0\n",
              "1        0\n",
              "2        0\n",
              "3        0\n",
              "4        0"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9vnglX0dDDW4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "86ab904b-9f6c-4d13-e37d-9785b66f33af"
      },
      "source": [
        "# Split the Dataset\n",
        "\n",
        "# Model 1\n",
        "# Lets start with selecting one variable\n",
        "Independent_Variable_Set_v1 = iris[iris.columns[0:1]]\n",
        "X_train, X_test, y_train, y_test = train_test_split(Independent_Variable_Set_v1,Dependent_Variable,test_size = 0.3,random_state = 21)\n",
        "\n",
        "# Create a Gaussian Classifier\n",
        "model = GaussianNB()\n",
        "\n",
        "# Train the model using the training sets\n",
        "result = model.fit(X_train,y_train)\n",
        "\n",
        "# Model Prediction\n",
        "print(\"Sample Prediction of Model 1\")\n",
        "pred = result.predict(X_test)\n",
        "model_prediction = pd.DataFrame(pred.round(2),columns = ['Prediction'])\n",
        "print(model_prediction['Prediction'].head())\n",
        "\n",
        "# Test Set Target Column Distribution\n",
        "print(\"\\nTest Set Distribution\")\n",
        "print(y_test['species'].value_counts())\n",
        "\n",
        "# Predicted Set Target Column Distribution\n",
        "print(\"\\nPredicted Set Distribution\")\n",
        "print(model_prediction['Prediction'].value_counts())"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Sample Prediction of Model 1\n",
            "0    1\n",
            "1    0\n",
            "2    0\n",
            "3    0\n",
            "4    1\n",
            "Name: Prediction, dtype: int64\n",
            "\n",
            "Test Set Distribution\n",
            "1    16\n",
            "2    15\n",
            "0    14\n",
            "Name: species, dtype: int64\n",
            "\n",
            "Predicted Set Distribution\n",
            "1    19\n",
            "0    14\n",
            "2    12\n",
            "Name: Prediction, dtype: int64\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L9WM2W7fDmfj",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "334acc35-7c98-4c17-a466-eae9d96c24d9"
      },
      "source": [
        "# Model Performance\n",
        "\n",
        "# Take the average of the f1-score for each class: that's the avg / total result above. \n",
        "# It's also called macro averaging.\n",
        "\n",
        "# Compute the f1-score using the global count of true positives / false negatives, etc. \n",
        "# (Sum the number of true positives / false negatives for each class). Aka micro averaging.\n",
        "\n",
        "y_pred = model_prediction[['Prediction']]\n",
        "model_1_accuracy = accuracy_score(y_test,y_pred).round(2)\n",
        "print(\"Model 1 Performance\")\n",
        "print(\"\\nModel 1, Accuracy :\",model_1_accuracy)\n",
        "model_1_precision = precision_score(y_test,y_pred, average=\"micro\").round(2)\n",
        "print(\"Model 1, Precision :\",model_1_precision)\n",
        "model_1_recall = recall_score(y_test,y_pred, average=\"micro\").round(2)\n",
        "print(\"Model 1, Recall :\",model_1_recall)\n",
        "model_1_fscore = f1_score(y_test,y_pred, average=\"micro\").round(2)\n",
        "print(\"Model 1, F1 Score :\",model_1_fscore)\n",
        "print(\"\\nConfusion Matrix, Model 1\")\n",
        "model_1_cm = confusion_matrix(y_test,y_pred)\n",
        "print(model_1_cm)\n",
        "print(\"\\nClassification Report, Model 1\")\n",
        "model_1_cr = classification_report(y_test, y_pred)\n",
        "print(model_1_cr)\n",
        "\n",
        "print(\"Inference : Good Fit, can we make it better ?\")"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model 1 Performance\n",
            "\n",
            "Model 1, Accuracy : 0.67\n",
            "Model 1, Precision : 0.67\n",
            "Model 1, Recall : 0.67\n",
            "Model 1, F1 Score : 0.67\n",
            "\n",
            "Confusion Matrix, Model 1\n",
            "[[13  1  0]\n",
            " [ 1 10  5]\n",
            " [ 0  8  7]]\n",
            "\n",
            "Classification Report, Model 1\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.93      0.93      0.93        14\n",
            "           1       0.53      0.62      0.57        16\n",
            "           2       0.58      0.47      0.52        15\n",
            "\n",
            "    accuracy                           0.67        45\n",
            "   macro avg       0.68      0.67      0.67        45\n",
            "weighted avg       0.67      0.67      0.66        45\n",
            "\n",
            "Inference : Good Fit, can we make it better ?\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6KhsgLWzREB3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0e21fe0f-5847-49dc-faba-f121638f77c4"
      },
      "source": [
        "# Model 2\n",
        "# Lets build model with all variables\n",
        "\n",
        "X_train, X_test, y_train, y_test = train_test_split(Independent_Variable_Base_Set,Dependent_Variable,test_size = 0.3,random_state = 21)\n",
        "# Train the model using the training sets\n",
        "result = model.fit(X_train,y_train)\n",
        "\n",
        "# Model Prediction\n",
        "print(\"Sample Prediction of Model 2\")\n",
        "pred = result.predict(X_test)\n",
        "model_prediction = pd.DataFrame(pred.round(2),columns = ['Prediction'])\n",
        "print(model_prediction['Prediction'].head())\n",
        "\n",
        "# Test Set Target Column Distribution\n",
        "print(\"\\nTest Set Distribution\")\n",
        "print(y_test['species'].value_counts())\n",
        "\n",
        "# Predicted Set Target Column Distribution\n",
        "print(\"\\nPredicted Set Distribution\")\n",
        "print(model_prediction['Prediction'].value_counts())"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Sample Prediction of Model 2\n",
            "0    1\n",
            "1    0\n",
            "2    0\n",
            "3    0\n",
            "4    1\n",
            "Name: Prediction, dtype: int64\n",
            "\n",
            "Test Set Distribution\n",
            "1    16\n",
            "2    15\n",
            "0    14\n",
            "Name: species, dtype: int64\n",
            "\n",
            "Predicted Set Distribution\n",
            "2    17\n",
            "1    14\n",
            "0    14\n",
            "Name: Prediction, dtype: int64\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CK1T-iljE1dK",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5d940520-5d43-4f17-9deb-113db07decb5"
      },
      "source": [
        "# Model Performance\n",
        "\n",
        "# Take the average of the f1-score for each class: that's the avg / total result above. \n",
        "# It's also called macro averaging.\n",
        "\n",
        "# Compute the f1-score using the global count of true positives / false negatives, etc. \n",
        "# (Sum the number of true positives / false negatives for each class). Aka micro averaging.\n",
        "\n",
        "y_pred = model_prediction[['Prediction']]\n",
        "model_2_accuracy = accuracy_score(y_test,y_pred).round(2)\n",
        "print(\"Model 2 Performance\")\n",
        "print(\"\\nModel 2, Accuracy :\",model_2_accuracy)\n",
        "model_2_precision = precision_score(y_test,y_pred, average=\"micro\").round(2)\n",
        "print(\"Model 2, Precision :\",model_2_precision)\n",
        "model_2_recall = recall_score(y_test,y_pred, average=\"micro\").round(2)\n",
        "print(\"Model 2, Recall :\",model_2_recall)\n",
        "model_2_fscore = f1_score(y_test,y_pred, average=\"micro\").round(2)\n",
        "print(\"Model 2, F1 Score :\",model_2_fscore)\n",
        "print(\"\\nConfusion Matrix, Model 2\")\n",
        "model_2_cm = confusion_matrix(y_test,y_pred)\n",
        "print(model_2_cm)\n",
        "print(\"\\nClassification Report, Model 2\")\n",
        "model_2_cr = classification_report(y_test, y_pred)\n",
        "print(model_2_cr)\n",
        "\n",
        "print(\"Inference : Model 2 is OVERFIT, so choose Model 1\")"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model 2 Performance\n",
            "\n",
            "Model 2, Accuracy : 0.96\n",
            "Model 2, Precision : 0.96\n",
            "Model 2, Recall : 0.96\n",
            "Model 2, F1 Score : 0.96\n",
            "\n",
            "Confusion Matrix, Model 2\n",
            "[[14  0  0]\n",
            " [ 0 14  2]\n",
            " [ 0  0 15]]\n",
            "\n",
            "Classification Report, Model 2\n",
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       1.00      1.00      1.00        14\n",
            "           1       1.00      0.88      0.93        16\n",
            "           2       0.88      1.00      0.94        15\n",
            "\n",
            "    accuracy                           0.96        45\n",
            "   macro avg       0.96      0.96      0.96        45\n",
            "weighted avg       0.96      0.96      0.96        45\n",
            "\n",
            "Inference : Model 2 is OVERFIT, so choose Model 1\n"
          ]
        }
      ]
    }
  ]
}