Cross_Validation.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Cross Validation",
      "provenance": [],
      "authorship_tag": "ABX9TyPNcczlzbzN0tnEgo+ZVJXT",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/MainakRepositor/ML-Algorithms/blob/master/Cross_Validation.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sr0xh0FZ1SDI"
      },
      "source": [
        "WHAT IS CROSS-VALIDATION?\n",
        "\n",
        "METHODS OF CROSS-VALIDATION\n",
        "\n",
        "1) VALIDATION\n",
        "\n",
        "2) LOOCV (LEAVE ONE OUT CROSS-VALIDATION)\n",
        "\n",
        "3) K-FOLD CROSS-VALIDATION\n",
        "\n",
        "4) STRATIFIED CROSS-VALIDATION\n",
        "\n",
        "5) ADVERSARIAL VALIDATION\n",
        "\n",
        "ADVANTAGES OF CROSS-VALIDATION\n",
        "\n",
        "IMPORTING LIBRARIES\n",
        "\n",
        "DATA INSPECTION\n",
        "\n",
        "HOLD-OUT VALIDATION APPROACH\n",
        "\n",
        "LEAVE ONE OUT VALIDATION\n",
        "\n",
        "K-FOLD CROSS VALIDATION APPROACH\n",
        "\n",
        "STRATIFIED K-FOLD CROSS VALIDATION APPROACH\n",
        "\n",
        "SHUFFLE SPLIT APPROACH\n",
        "\n",
        "CLOSING COMMENTS"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a8_GQHbo1gLS"
      },
      "source": [
        "### What is Cross Validation\n",
        "\n",
        "Cross-validation is a technique in which we train our model using the subset of the data-set and then evaluate using the complementary subset of the data-set.\n",
        "\n",
        "Does this definition sound familiar to what you do \"train_test_split\"? You must be thinking if we have a parameter already for evaluation why do we need Cross-validation?\n",
        "\n",
        "Well here is the catch! In train-test-split you keep aside (70%,80%) of data for training and remaining for testing the model, but what if the data points that can bring some pattern is in test dataset and you end up losing it? Also when you split the data with some random_state and calculate the accuracy or any other metric it changes with change in random_state, in this case how would you decide what's the maximum and minimum accuracy,error your model produce?\n",
        "\n",
        "There comes Cross-validation as our Saviour. Wonder how?\n",
        "\n",
        "Cross-validation starts by shuffling the data (to prevent any unintentional ordering errors) and splitting it into k folds. Then k models are fit on k−1 of the data (called the training split) and evaluated on 1/k of the data (called the test split). The results from each evaluation are averaged together for a final score, then the final model is fit on the entire dataset for operationalization.\n",
        "\n",
        "This way there's no leaving out any part of dataset covering all trends!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z31H5A9y1omE"
      },
      "source": [
        "## Methods of Cross Validation\n",
        "\n",
        "### 1) VALIDATION\n",
        "In this method, we perform training on the 50% of the given data-set and rest 50% is used for the testing purpose. The major drawback of this method is that we perform training on the 50% of the dataset, it may possible that the remaining 50% of the data contains some important information which we are leaving while training our model i.e higher bias.\n",
        "\n",
        "\n",
        "### 2) LOOCV (LEAVE ONE OUT CROSS-VALIDATION)\n",
        "In this method, we perform training on the whole data-set but leaves only one data-point of the available data-set and then iterates for each data-point. Suppose you have 1000 records in your dataset and in this method you leave out one data point and use the rest 999 records for validation. In this way the validation is run for 1000 times and what if you have millions of records? Would you run million times? That's practically impossible!\n",
        "\n",
        "An advantage of using this method is that we make use of all data points and hence it is low bias. The major drawback of this method is that it leads to higher variation in the testing model as we are testing against one data point. If the data point is an outlier it can lead to higher variation. Another drawback is it takes a lot of execution time as it iterates over ‘the number of data points’ times.\n",
        "\n",
        "From the above two validation methods, we’ve learnt:\n",
        "\n",
        "We should train the model on a large portion of the dataset. Otherwise we’ll fail to read and recognise the underlying trend in the data. This will eventually result in a higher bias.\n",
        "\n",
        "We also need a good ratio of testing data points. As we have seen above, less amount of data points can lead to a variance error while testing the effectiveness of the model.\n",
        "\n",
        "We should iterate on the training and testing process multiple times. We should change the train and test dataset distribution. This helps in validating the model effectiveness properly.\n",
        "\n",
        "Do we have a method which takes care of all these 3 requirements?\n",
        "\n",
        "\n",
        "### 3) K-FOLD CROSS-VALIDATION\n",
        "Yes! That method is known as “k-fold cross validation”.\n",
        "\n",
        "It’s easy to follow and implement. Below are the steps for it:\n",
        "\n",
        "Randomly split your entire dataset into \"kfolds\".\n",
        "For each k-fold in your dataset, build your model on k – 1 folds of the dataset. Then, test the model to check the effectiveness for kth fold.\n",
        "Record the error you see on each of the predictions.\n",
        "Repeat this until each of the k-folds has served as the test set.\n",
        "The average of your k recorded errors is called the cross-validation error and will serve as your performance metric for the model.\n",
        "Now you might wonder what could be the way to decide the value of k?\n",
        "\n",
        "Well let me tell you there's no thumb rule for choosing the value of k and it completel depends on size of your dataset. However k=10 is considered to be ideal value as more than 10 would really take longer to validate.\n",
        "\n",
        "Higher the value of K leads to LOOCV approach where you will incure high computation costs.\n",
        "\n",
        "Lower value of K leads to validation approach where you will lose the important data.\n",
        "\n",
        "\n",
        "### 4) STRATIFIED CROSS-VALIDATION\n",
        "Stratification is the process of rearranging the data so as to ensure that each fold is a good representative of the whole. For example, in a binary classification if our target labels are imbalanced,it is advisory to arrange the data such that in every fold, each class comprises of about half the instances. If this is not covered, the model learns only to predict the majority label and when unseen data is fed it will predict only majority label which might deteriorate model's predictions.\n",
        "\n",
        "\n",
        "### 5) ADVERSARIAL VALIDATION\n",
        "Many data science competitions suffer from a test set being markedly different from a training set (a violation of the “identically distributed” assumption). It is then difficult to make a representative validation set. As a result, the internal cross-validation techniques might give scores that are not even in the ballpark of the test score. In such cases, adversarial validation offers an ideal solution.\n",
        "\n",
        "The general idea is to check the degree of similarity between training and tests in terms of feature distribution. If It does not seem to be the case, we can suspect they are quite different. This intuition can be quantified by combining train and test sets, assigning 0/1 labels (0 – train, 1-test) and evaluating a binary classification task. Specifically, we’ll run the distinguishing classifier in cross-validation mode, to get predictions for all training examples. Then we’ll see which training examples are misclassified as test and use them for validation.\n",
        "\n",
        "To be more precise, we’ll choose a number of misclassified examples that the model was most certain about. It means that they look like test examples but in reality are training examples. Cross-validation provides predictions for all the training points. Now we’d like to sort the training points by their estimated probability of being test examples. We did the ascending sort, so for validation we take a desired number of examples from the end. We will note that if differences between models in validation are pretty slim."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "M5ROlHpz1c9J"
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np \n",
        "import matplotlib.pyplot as plt\n",
        "import sklearn\n",
        "import warnings\n",
        "warnings.filterwarnings('ignore')\n",
        "# Import necessary modules\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.metrics import mean_squared_error\n",
        "from math import sqrt\n",
        "from sklearn import model_selection\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.model_selection import KFold\n",
        "from sklearn.model_selection import cross_val_score\n",
        "from sklearn.model_selection import LeaveOneOut\n",
        "from sklearn.model_selection import LeavePOut\n",
        "from sklearn.model_selection import ShuffleSplit\n",
        "from sklearn.model_selection import StratifiedKFold\n",
        "from yellowbrick.model_selection import cv_scores\n",
        "from yellowbrick.model_selection import CVScores"
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "f6tHVfRb1ESk",
        "outputId": "55ece023-c912-40a9-e7d5-9a3cd8e11500"
      },
      "source": [
        "url = 'https://raw.githubusercontent.com/MainakRepositor/Datasets-/master/diabetes.csv'\n",
        "data = pd.read_csv(url,error_bad_lines=False)\n",
        "data.head()"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Pregnancies</th>\n",
              "      <th>Glucose</th>\n",
              "      <th>BloodPressure</th>\n",
              "      <th>SkinThickness</th>\n",
              "      <th>Insulin</th>\n",
              "      <th>BMI</th>\n",
              "      <th>DiabetesPedigreeFunction</th>\n",
              "      <th>Age</th>\n",
              "      <th>Outcome</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>6</td>\n",
              "      <td>148</td>\n",
              "      <td>72</td>\n",
              "      <td>35</td>\n",
              "      <td>0</td>\n",
              "      <td>33.6</td>\n",
              "      <td>0.627</td>\n",
              "      <td>50</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>85</td>\n",
              "      <td>66</td>\n",
              "      <td>29</td>\n",
              "      <td>0</td>\n",
              "      <td>26.6</td>\n",
              "      <td>0.351</td>\n",
              "      <td>31</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>8</td>\n",
              "      <td>183</td>\n",
              "      <td>64</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>23.3</td>\n",
              "      <td>0.672</td>\n",
              "      <td>32</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1</td>\n",
              "      <td>89</td>\n",
              "      <td>66</td>\n",
              "      <td>23</td>\n",
              "      <td>94</td>\n",
              "      <td>28.1</td>\n",
              "      <td>0.167</td>\n",
              "      <td>21</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0</td>\n",
              "      <td>137</td>\n",
              "      <td>40</td>\n",
              "      <td>35</td>\n",
              "      <td>168</td>\n",
              "      <td>43.1</td>\n",
              "      <td>2.288</td>\n",
              "      <td>33</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Pregnancies  Glucose  BloodPressure  ...  DiabetesPedigreeFunction  Age  Outcome\n",
              "0            6      148             72  ...                     0.627   50        1\n",
              "1            1       85             66  ...                     0.351   31        0\n",
              "2            8      183             64  ...                     0.672   32        1\n",
              "3            1       89             66  ...                     0.167   21        0\n",
              "4            0      137             40  ...                     2.288   33        1\n",
              "\n",
              "[5 rows x 9 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6M2WORAJ2dmG"
      },
      "source": [
        "### Hold-Out CV"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hhvYM1Ac2QkE"
      },
      "source": [
        "x1 = data.drop('Outcome', axis=1).values \n",
        "y1 = data['Outcome'].values"
      ],
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ONCs4BRR2TrA",
        "outputId": "53f61997-5e2b-4590-d019-3b272a103038"
      },
      "source": [
        "X_train, X_test, Y_train, Y_test = model_selection.train_test_split(x1, y1, test_size=0.30, random_state=100)\n",
        "model = LogisticRegression()\n",
        "model.fit(X_train, Y_train)\n",
        "result = model.score(X_test, Y_test)\n",
        "print(\"Accuracy: %.2f%%\" % (result*100.0))"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Accuracy: 75.32%\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RDNREPL-2WT9",
        "outputId": "fc11f16b-9440-4b3d-9a80-b03cf86ee60c"
      },
      "source": [
        "X_train, X_test, Y_train, Y_test = model_selection.train_test_split(x1, y1, test_size=0.30, random_state=22)\n",
        "model = LogisticRegression()\n",
        "model.fit(X_train, Y_train)\n",
        "result = model.score(X_test, Y_test)\n",
        "print(\"Accuracy: %.2f%%\" % (result*100.0))"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Accuracy: 77.06%\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_IY1apfr2gp0"
      },
      "source": [
        "### LOOCV"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MDKSc71z2YiD",
        "outputId": "5052f3e5-c25f-4341-9a32-ec99e55a0c24"
      },
      "source": [
        "loocv = LeaveOneOut()\n",
        "model_loocv = LogisticRegression()\n",
        "results_loocv = cross_val_score(model_loocv, x1, y1, cv=loocv)\n",
        "print(\"Accuracy: %.2f%%\" % (results_loocv.mean()*100.0))"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Accuracy: 77.86%\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DO3sPSNA2iqT",
        "outputId": "b7dd2d00-75cd-4b35-dffd-c3b79b022f13"
      },
      "source": [
        "X_train, X_test, Y_train, Y_test = model_selection.train_test_split(x1, y1, test_size=0.30, random_state=22)\n",
        "model = LogisticRegression()\n",
        "model.fit(X_train, Y_train)\n",
        "result = model.score(X_test, Y_test)\n",
        "print(\"Accuracy: %.2f%%\" % (result*100.0))"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Accuracy: 77.06%\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S_k8Vtyr2vBl"
      },
      "source": [
        "### KFold CV"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "P7lb8DpY2okN",
        "outputId": "7462aea1-fafc-47cb-fb6d-6ccb27e4d7bb"
      },
      "source": [
        "for k in range(2,11):\n",
        "\n",
        "    kfold = KFold(n_splits=k, random_state=100)\n",
        "    model_kfold = LogisticRegression()\n",
        "    results_kfold = model_selection.cross_val_score(model_kfold, x1, y1, cv=kfold)\n",
        "    print(\"Accuracy: %.2f%%\" % (results_kfold.mean()*100.0)) "
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Accuracy: 76.69%\n",
            "Accuracy: 77.21%\n",
            "Accuracy: 77.60%\n",
            "Accuracy: 77.22%\n",
            "Accuracy: 76.95%\n",
            "Accuracy: 76.96%\n",
            "Accuracy: 77.60%\n",
            "Accuracy: 77.60%\n",
            "Accuracy: 77.34%\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 294
        },
        "id": "XegJQ6IW2xD6",
        "outputId": "9c84fffc-8d48-4dfe-fe56-987c6e032d8c"
      },
      "source": [
        "# Create a cross-validation strategy\n",
        "cv = KFold(n_splits=10, random_state=100)\n",
        "\n",
        "# Instantiate the classification model and visualizer\n",
        "model = LogisticRegression()\n",
        "\n",
        "# Fit the data to the visualizer\n",
        "visualizer = cv_scores(model, x1, y1, cv=cv, scoring='accuracy')"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEVCAYAAADpbDJPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZwU1bn/8Q+LjAKyigJBYlR8XAZRRgyLiIg/96hRSIxRQ4IaIxr0xgXFfbl43Ygm5qKJhqBxiSiI4oIxLnhHFMZEGYHHqAFlEVF2kUFgfn+cmrGnmaVnmOqeob7v14sX3VXVdZ7q6qmn6pyqc5qUlpYiIiLJ0zTXAYiISG4oAYiIJJQSgIhIQikBiIgklBKAiEhCKQGIiCRU81wHsL0ysybAJcAvgB0I3/WLwJXuvjpLMbQBlgKHuPu8tHmPAB+7+9VVfHYC8KG732xm84FB7r4sbZkzgXPc/Yga4jgGmOfun5jZWGChu4+v63alrbsAuA34DuGE5kvgMnd/oz7Wv63MbE9gOrDO3Q/ahvUsAM6sj+0ys0OBm9z9mGqW2Q34vrtPzXD54cC9wKfRpCbAN8Bt7j5xW2OOQ1W/6yRRAojPrcARwDHuvtjMWgF3A8+a2eHuHvsDGO6+xswmA2cBV5VNN7OdgZOBXhmuZ99tDOUS4GbgE3e/chvXVS5Kss8A57r7tGjaqcDTZra7u6+vr7K2wQBgqbsPzHUgZdz9baDKg3lkMHAUMDXD5QHedPejyt6Y2T7ATDN7293n1zngmNTD77rRUwKIgZl1AH4NHOzuiwHc/SszuxD4f0ATM7uOcNbaC3gEuAe4CTgtWs1MYGT0uWHAdUAzwlnVr9391aqmp4UzAfiTmY1JSTqnAv909w/N7BrgTMJvYR7hLHNV2vaUArsDS6I4TwI+A15LWWY34C/AHkAe8Dt3v8vMbgKGAPuZ2eXAcXx7ZXEg8L9AR2ADcIW7v2hmRwBjgVeBU4AdgeHuXl5eZBegS/RdEX3PT0UHnPVRXFcAvwQ2Ac8Cv3H3UjP7NXA+4arBCVcyy6MrnxWEg99NwFTgduBYoAVwv7v/d7TuC4GRhLPdNcDP3f39lO+kH+HqpI2ZvevuvVL2WfPo+zzX3T8ys+tJ+T24+2/JgJk1perfTW/g8Wj6w9Eyv47e/8nd9zazfOCPQJto++4GCoHfA83NrDUwPmX5nYD7gIGEfXaLuz9cWWzu/oGZOdATmG9m+xP2dxegJPq+ZkfbcDcwDPiQkNSPc/cj6mN/VDO9FNjd3RfV8HtYCPQH9gE+AE5uICcX20xtAPHoCyxKP+tx9w3u/oy7b4kmHQ8cH/2x/4hwcCwADgDaEc6cAf4AnODu+wEXEA7A1U1P9Q/Cfk49Az0TmBBVn1wI9AF6EA7cF1azXccCRwP7A4OAw1PmXQ38JzqrGgKMjc7CrwEWAz9197KDUdmB6zHg99FnzgEeja5OAA4GZkbb9odo/em+AGYBr5jZCDP7HoC7L4rKOCxaby8gHzgMGGpmfYHLgCOisj8hJJwyQ4BD3f0J4PJoe3sS9stQMzsxivOmaLl9CQelE1KDc/c3gSsJZ8a9zKw74WB7SvSZaYSDaZnU30Omqvvd3A/c5e49gNWEA1i664Dx7n4A0I9woH2fkAAmufvpacv/Bmjh7t8jnMz83sy6VhaYmQ2IYpoV7e8pwER334dwsH3azJpH230csDfhNzw8bVV13h+Z7KcMfg/DgB8DewGdgB9Wtr2NkRJAPDoAmdQrvuXuX0SvTwD+4u5fuftm4M+Egy3A58D5ZvZdd3/D3f+rhunlomQzkVANRPTH2hf4m7sXEc6A1kTLFQJ7VhPv4cA0d1/n7l8Df0uZ92vgoqjMjwlXCN+rZl3fAzoTkgDuPptwptUnmr/W3Z+OXr8DdK9k20oJB6HJwCjgYzN7P6oGgnBgmebua919I6FK7inCdz3J3T+PlvsT337XAC+7+4bo9Q+AP7h7ibt/RfguTyWc/ZYCI8xsN3d/wt1vq2Z7iWJ9xd0/TCl3cHQQhIq/h0xV+ruJztQLgEej5e4lnAGn+xw4Lbpa+NLdT3H3kmrKO55v99kioJu7L4nm9TOz+dG/L6IyT3P3BcC+wK7Ag9Fn/w9YTjizHgg8G/2uVqTEXGZb9kcm+6mm38M0d1/h7puAOVTyW2yslADi8QXhcr4mK1JedwJWprxfSfiDgXBW1BkoMrN/mtmgGqanm0D4I88DzgCejtoHWgK/MzOPLtUvoPrfRAfCmWRqjGX6AC+a2b+jxrUuNayrE7AqrS0kdZtTy9lMqObairuvdvfr3P1AwncxEXjMzPYjVBGtSll2fXSQrO67hor7pR0wruzARkg0rdz9G8KZ6QDgAzObYWY9q9nesm0uL9fDzQBNojjTy81UVdvSHigtq86L4v18649zBVBMSOafmtkFNZSX/p2uS5n3prvvG51FjwHWuPtL0bx2QEtgXsp3uSuh+q89Fbd9cVqZdd4fGe6nmn4PGf0WGyO1AcRjJrCbmfV293fKJprZDsD1wC2VfGYZ4Y+hTMdoGu7+EfDz6DL6bEKbwXeqmp6+4qiufy7hMvsnhD96gIsJVT8F7r7OzG6p7PMpVgJtU953Snn9MDCOUJ1Qambpf8SVbW8HM2uSkgTKtzkTZtYN2MOjO2M83M3xP2b2I0L1wBd8e3DFzMq+3yq/60osAe5w92fTZ7j7P4FhZtaCUDUxnnCgqcoyQjVLWTztgS1RnHVV1basIbQ1tXT39dFVRqf0D0cH8KuAq8ysD/CCmf29mvLSv9NuVJ64HgAuNbMfuvtkwve4xitpeI2qilqnTOpSTfm13h8Z7Kfa/B62K7oCiEF01nUbMNHM9gaIzrbvJzQMV9aA9Cxwppm1jP5YRwDTzKyTmb1kZm2iapqZQGlV06sJawKhimYXQrsAhLOc+dHB/7uEy/vWlX8cgDeBY6IYWxLqRsvsChRFB/+fAa1S1vUN4cwt1QJgEaFuFTPrTziDf7ua8tPtDkyJ2jKI1tOHcIk+i9BgeJKZtY++0ymEu1mmAaemJIRfRtMq8zRwjpk1M7MmZna1mR1rZj3N7AkzaxFVL82m+u8f4CXgcAu3hkKoB58eVS3UVaW/m+jAPo/QRlC2jVvFZ2bPmNkB0dtiwtluKZXvMwjf6dnRd9EZ+CcpCaFMtE3XAbdGJz4LgUVmNjQqdxcze9TC3XFvAyea2U5m1i4l5srUan9kuJ9q83vYrigBxMTdrycc8KdG1StFhLOKU6v4yCTguWi5YsL91Pe4+3LgBUJD2lxC/euIqqZXE9LfCHX/D6U0Qo8HBkXx3Qn8FzDEzC6uYh3PAP9HuEvitSjeMtcAk83sPcKB/z7gj2a2V7Rtj5lZeRtFdNZ/OnChmc0j3F00LKrXzUjUyHoe8L9RNdaHhKuQH7v7QnefSWj0+xcwl9CW8KiH2xpvBWZE1QjtCFUWlbmXcPB6H5gP7Ae8QdhH/wHeN7P3CVd2o2qIdxGhUfrpqNzDCQebTP01pY59vpn9mCp+N9HyFwBjovhaEapW0g9+vwMeifbBO4T69X8Tnl040sxmpS0/jlCVtJBwl9al7v5JFfE+SqiDPz9tf88HXifU7X9FaMOZTfhdPUn4rVaVTGu7P2rcT7X8PWxXmmg8AJHtV2oVm5ktB45y93dzHNZW0uIcSYhzu7nbpqHSFYDIdsrMym6bxMyOJDQ4f5DToCphZgcB/0mpqjuVUN0oMYv1CsDMxhGqHUqBUe4+K2XeyYR7u0uAx9z997EFIpJA0Z1QfybcvbWR0EXG87mNqnJmdgPhVuXNhIP/+VW0lUk9ii0BWLgl8TJ3PzH6IT7o7v2ieU0J9Xi9CX23PE+o114USzAiIrKVOG8DHUK46wJ3nxdd3rVx9zVE9xJHDZmY2cuEJxAnVLaioqKiPMJ95ksJZwgiIlKzZoTbamcVFBRs9YBfnAmgM+HOhDLLo2lrotc7m1kPwu2Agwl3FFSlDzAjlihFRLZ/Awl3S1WQzQfByh9DT7lX/EHCfcf/ofLH1MssBdhnn31o0aJFvQRTXFxMfn5+vaxLMTTuGHJdvmJoGOVvjzFs3LiRDz74AKJjaLo4E8ASwhl/ma6pQXjo2XEggIU+4hdUs67NAC1atCAvL6/eAqzPdSmGxh1DrstXDA2j/O04hkqrzuO8DXQ6UPbUX29gibuvLZtpZs+b2a7Rk4A/AKp7/FxEROpZbFcA7l5oZkVmVkjo72SkhVGDVkd9g/yRkCRKgbF16AURgFWrVrFu3bqaF0zz+eefs2hRbm86UgyZx9C6dWvatausZwIRqatY2wDcfXTapHdT5j1F6Jq3zmbMmEHbtm3p0KFDrT+71157bUvR9UIxZB7DJ598wpw5cxg4sMEMrCXS6DXa3kBXrVpF27ZtOfDAA+v0+a+++opWrVrVc1SKIa4YunXrxnvvvceqVat0JSBSTxptVxDr1q2r05m/NF7t27evU3WfiFSu0SYASZ4mTaq7U1hEaksJQEQkoRptG0BDsGjRIoYMGcLjjz/OQQcdVD79tNNOo0ePHtx66605jC5+S5cu5fLLL2fz5s106tSJ22+/vcKDek888QRTp04tf19cXMzs2bMZPnx4+bTPP/+cE044gXPPPZfRo0fz5ZdfUlJSwgUXXMDgwYOzuTkiiaMEsI123313nn322fIEsHDhQtasWZPjqLLjnnvu4YwzzuC4447jrrvuYtKkSZxxxhnl84cNG8awYWHQsLfffpvnn3+eZs2a8dBDD5Uvc84553DCCSfwyiuvkJ+fz7nnnsvixYv5xS9+oQQQo2a/eajyGY/MrfB2851nZSEayRUlgG3Uq1cvCgsL2bx5M82aNWPatGkMGDCADRs2ADB79mzuuusumjdvTpcuXbjpppto2rQpV1xxBUuWLKGkpISLLrqIwYMHc9ZZZ9G/f39mzpzJypUrGT9+PF27di0va+7cudxwww20aNGCFi1aMG7cOAAuvfRS1q1bx84778xdd93Fli1bGD16NGvWrGHTpk1cffXVHHDAARx99NHsv//+DBgwgIMPPpgbb7yRLVu20KZNG2699VbatGlTXlb62TvABRdcQL9+5UPa8tZbb3HDDTcAMHjwYB588MEKCSDVvffeyx133FFhWmFhIXvssQedO3eucCvo0qVL2W233eqyO0SkFpQAttEOO+xAr169eOutt+jfvz8vv/wyF154IS+++CIAN998MxMmTKBdu3bcdtttvPDCCwwYMIDDDjuMo48+mhUrVjBq1Kjys93WrVvzl7/8hTvuuIPp06dXqC556qmn+MlPfsIpp5zCm2++yfLly5k6dSqHHXYYZ599NhMmTODNN99k/vz59OrVi/POO485c+YwduxYHn74YT799FPuvfdeevTowc9+9jNuvPFGOnXqxJQpU/jrX//Kr371q/KyUs/eq/L111+XV/l07NiR5cuXV7rce++9R5cuXejUqeKY5BMnTuSqq66qMO3000/ns88+Y/z48ZntABGps+0qAdzw4rvcOP298vdvX3w8AIf+9tuha689+kCuO6YX+9z+HJ+tDWfpvbt1YNYlJ/DLJ97kTzM/LF/202tPo2vbljWWe+yxx/Lss8+yyy67sNtuu9GyZfjMF198wcKFC7nooosAWL9+Pe3bt6dNmzbMmTOHRx99lObNm7Nq1arydR1yyCEAdO7cucJ0gCFDhnD99dezYMECjj/+ePbaay/mzp3LqFFhiNOyZDFp0qTyg3nPnj1ZuHAhADvttBM9evQAwkH5mmuuYfPmzWzevJmePXvWuJ3VqW5ciUmTJvHDH1Yc3W/ZsmWsX7+e7t2789VX3w4D/NhjjzFv3jwuu+wypk6dqjt/RGK0XSWA647pxXXH9NpqemX1mB9cdvxWDx/dN6wf9w3rt9WyNenXr1/52fQxxxxTPn2HHXZg1113rVDnDTB58mRWr17NAw88wDfffMPQoUPL5zVr1qz8dfpBtV+/fkyaNIlXXnmF0aNHc/nll9OsWTO2bNlSYbkmTZpU+GzZ/B122KF82k477cTEiRNZv359pQ9hZVIF1LJlSzZs2MCOO+7IsmXL2HXXXSv9ft566y2uvvrqCtNee+01+vbtW/6+uLiYjh070qVLF/bbbz82b97MihUr6NixY6XrFJFtt10lgFxp0aIFffr04cknn+T5559n7tzQkNa2bVsAPvzwQ/bee28eeugh+vTpw8qVK+nWrRtNmzblpZdeYuPGjRmV8/DDDzNo0CBOOukkSktLmTdvHvn5+cycOZMDDzyQxx57jLy8PHr27Mlbb73FQQcdxL/+9a/ys/5U++67L6+//jqHHHII06ZNo0OHDhUO7plUAfXv358XX3yRk08+menTp1faTcOyZcto1arVVt14z5kzp0Ij7+zZs1m8eDFjxozhiy++KL9aEpH4KAHUk2OPPZYVK1aw8847V5h+yy23cOWVV1KypQntOu7C3v2PokvPQ3nkmsuZ8dYsBh17Ijt32IWr/vt21m74ptoyunfvzqhRo9h5551p0aIFY8eOJS8vj8svv5yzzjqLVq1alTe0XnXVVZx99tmUlpZy7bXXbrWuMWPGcM011zB+/HhatmzJnXfeWettvuiii7jiiit4/PHH6dq1K6eccgoAl1xyCWPHjmXHHXdk+fLllT6xvXz58gpn96effjpjxozhjDPOYMOGDVx77bU0barHVETiFOug8PWlqKhoD+A/+fn55f1kl/Ue2a1btzqtM9t94Mz+9MuMljtk9+xWeTSWvoBg2/d5VYqKiigoKKjXdTb0GKq8DTRNNm8DTeJ+iDuGkpISiouLAb5XUFCwIH2+TrFERBJKVUDSaJSWluquoO1IpVchehAtq2JNAGY2DuhLGPRllLvPSpk3EjiTMFTZbHe/uDbrbt26NZ988km9VwdIw7Vy5Uq6d++e6zBEthuxJQAzGwT0cPd+ZrYfYQD4ftG8NsBlwN7uvsnMpptZX3efmen627Vrx5w5c3jvvfdo3759rc8M169fX36/fjZ8vnRlRsstavJ1zJFUlO3voS4xlJaWsnLlSlavXq2xAETqUZxXAEOAKQDuPs/M2ptZG3dfA2yM/rU2s3VAS2BFbQsYOHBgnYeE/Oijj7b54afaOOmBf2S03IJrTos5koqy/T3UJYYmTZrQvXt3HfxF6lmcCaAzUJTyfnk0bY27bzCzG4CPga+Bx9z9g7oU0q5duzodGJYtW5bV6qPS1pkNXpPtKq1sfw8NNQaRXMh1p3zZbAQur6OJqoCuAvYB1gD/MLNe7v5uVR8Gym5nqjdFRUU1L5RluYipIXwPuY4h1+U3lBjS5TqmpP49pIsrpjgTwBLCGX+ZrsDS6PV+wMfu/gWAmc0ACkgZNL4yqc8BbKus3++bltGrku17kLe3+55r0hDvf4eE/h4ziGG7/3uIeT+kPAdQqTifA5gODAUws97AEndfG81bAOxnZjtF7w8B/h1jLCIikia2KwB3LzSzIjMrBLYAI81sOLDa3Seb2e3AK2a2CSh09xlxxSIiIluLtQ3A3UenTXo3Zd59wH1xli8iIlVTVxAiIgnVqLqC2OuWySz9KvSYWd1gL91umMTSNeGBqqoGe5l2Sg+eef9TTnnw1fJp/zv0+5zXb58KDYUn7P8dpo44kpMe+AfT5i4un775zrO4/80P+NWkt8qnTfnFERR068juNz5ZPu2cvnvXaoyBok+/rPM2fXrtaRQt+rL22xQ1RNV2m/qMm8Y7i8LjG13a7MSi64bWalCeCtv0+mf1u02RyrYpE81+89C2b1Mt9tOVh3ahoIBYfnuV7afaiOu3l4klq9fH/9tL36bo7yGO315l+ykTZeXVdpuueu4dzujeYusVRhptb6D1sM7E974IuguoKtv7fmgI30MmMWg/BHX9HmrqDbRRXQHItsn1Qyci0rCoDUBEJKGUAEREEkoJQEQkoZQAREQSSglARCShdBeQSJbpbixpKJQARCSRlIhVBSQiklhKACIiCaUEICKSUEoAIiIJpQQgIpJQsd4FZGbjgL5AKTDK3WdF078D/DVl0T2B0e7+SJzxiIjIt2JLAGY2COjh7v3MbD/gQaAfgLsvBo6IlmsOvApMjSsWERHZWpxXAEOAKQDuPs/M2ptZG3dfk7bccOBJd18XVyC631dEZGtxJoDOQFHK++XRtPQEcA5wdCYrjAY2iE1RUVHNC8UsqTE0hO1O1RDiUQy5L397jyGbTwI3SZ9gZv2A+ZVcFVSqziOCpZ3pVyXWkYAUQ5WyOgpTQ/gOFEPGMeS6/MYeQ8qIYJWK8y6gJYQz/jJdgaVpy5wI/D3GGEREpApxXgFMB24A7jOz3sASd1+btkwf4LEYY5AGJpP2GLXFiGRHbFcA7l4IFJlZIXAPMNLMhpvZD1MW6wJ8HlcMIiJStVjbANx9dNqkd9Pm94yzfBERqZqeBBYRSSglABGRhFICEBFJKCUAEZGEUgIQEUkoJQARkYRSAhARSSglABGRhFICEBFJKCUAEZGEUgIQEUkoJQARkYRSAhARSSglABGRhFICEBFJqFjHAzCzcUBfoBQY5e6zUubtDjwKtADecffz44xFREQqiu0KwMwGAT3cvR8wgjAqWKo7gTvd/VBgs5l1jysWERHZWpxVQEOAKQDuPg9ob2ZtAMysKTAQmBrNH+nun8QYi4iIpImzCqgzUJTyfnk0bQ3QCVgLjIsGjJ/h7lfWtMLi4uI44ixXVFRU80IxUwy5L18xNJwYcl3+9h5DrG0AaZqkvf4OcDewAJhmZie4+7TqVpCfn09eXl7tS35kbkaLFRQU1H7diqHeY8h1+Yqh4cSQ6/IbewwlJSXVnjjHWQW0hHDGX6YrsDR6/QWw0N0/cvfNwMvAATHGIiIiaeJMANOBoQBRNc8Sd18L4O6bgI/NrEe0bAHgMcYiIiJpYqsCcvdCMysys0JgCzDSzIYDq919MnAxMCFqEJ4DPBNXLCIisrVY2wDcfXTapHdT5n0IHBZn+SIiUjU9CSwiklBKACIiCaUEICKSUEoAIiIJpQQgIpJQSgAiIgmlBCAiklBKACIiCaUEICKSUEoAIiIJpQQgIpJQSgAiIgmlBCAiklBKACIiCaUEICKSULGOB2Bm44C+QCkwyt1npcxbAHwKbI4m/dTdF8cZj4iIfCu2BGBmg4Ae7t7PzPYDHgT6pS12nLuviysGERGpWpxVQEOAKQDuPg9ob2ZtYixPRERqIeMrADPLB/Z29ylm1s7dV9Xwkc5AUcr75dG0NSnTxpvZHsAbwJXuXlrdCouLizMNt06KiopqXihmiiH35SuGhhNDrsvf3mPIKAGY2SXAT4A8wln9NWa20t1vrkVZTdLeXwu8AKyI1nkaMKm6FeTn55OXl1eLIiOPzM1osYKCgtqvWzHUewy5Ll8xNJwYcl1+Y4+hpKSk2hPnTKuAfkJozF0Rvb8MOLGGzywhnPGX6QosLXvj7hPd/XN33wQ8B/TMMBYREakHmSaAte6+pexN9HpLNcsDTAeGAphZb2CJu6+N3rc1sxfNrEW07CAg3vodERGpINM2gI/M7DpCQ+6pwI+Baq9d3L3QzIrMrJCQLEaa2XBgtbtPNrPngJlm9jXwT2qo/hERkfqVaQIYCYwCFgNnEhpt763pQ+4+Om3Suynz7gbuzrB8ERGpZ5kmgDPd/Q7gjjiDERGR7Mm0DeBUM2sbayQiIpJVmV4B7AQsMDMHNpZNdPfDY4lKRERil2kCuCnWKEREJOsyqgJy99cId/IUAL2BjdE0ERFppDJKAGZ2I3A70AX4DnCPmV0ZZ2AiIhKvTKuABgP9yx4GM7PmwOvA2LgCExGReGV6F1DTtCeBN1Hzk8AiItKAZXoFUGRmU4G/R+//HzCrmuVFRKSByzQBXAz8CPg+YXSviajrBhGRRq02zwFscfdLAMzsfKAVoNG8REQaqUzbACZSsWvnlsBD9R+OiIhkS6YJoIO731P2xt3vAtrFE5KIiGRDpgkgLxrYHQAzKwBaVLO8iIg0cJm2AVwCPB11CNcU+AI4K7aoREQkdtUmADNrA4xw93HAPmZ2NeFuoA+AT2tauZmNIwwlWQqMcvetbh01s7FAP3c/ovbhi4hIXdVUBXQfsCuAme0D/BfhamA6NQzmYmaDgB7u3g8YAdxTyTL7A+pRVEQkB2pKAHu6e1mfP0OBJ9z9ZXe/n4p3BVVmCDAFwN3nEYaTbJO2zJ3AmFrGLCIi9aCmNoDU+/yPAB5IeV9TVxCdgaKU98ujaWsAovGBXwMW1BxmUFwc77jxRUVFNS8UM8WQ+/IVQ8OJIdflb+8x1JQAmpvZrsDOQD/CYPCYWWvCg2C10aTshZl1AH4OHEXoXTQj+fn55OXl1bJY4JFqx68vV1BQUPt1K4Z6jyHX5SuGhhNDrstv7DGUlJRUe+JcUxXQrcBcYA5wk7uvNLOdCIPCT6zhs0uoWE3UFVgavT4S6ATMACYDvaMGYxERyZJqE4C7P08YA6Czu98WTfsauNzd761h3dMJ7QaYWW9gibuvjdYxyd33d/e+wA+Bd8q6mRARkeyo8TkAd/8G+CZt2vQMPldoZkVmVkhoLxgZ1fuvdvfJdYxXRETqSaYPgtWJu49Om/RuJcssIDQwi4hIFmXaFYSIiGxnlABERBJKCUBEJKGUAEREEkoJQEQkoZQAREQSSglARCShlABERBJKCUBEJKGUAEREEkoJQEQkoZQAREQSSglARCShlABERBJKCUBEJKFiHQ8gGuaxL1AKjHL3WSnzzgVGAJsJ4wSMdPfSOOMREZFvxXYFYGaDgB7u3o9woL8nZV5L4HRgoLsPAPYlDDovIiJZEmcV0BBgCoC7zwPam1mb6P16dx/i7t9EyaAt8FmMsYiISJo4q4A6A0Up75dH09aUTTCz0cAo4Lfu/nFNKywuLq7vGCsoKiqqeaGYKYbcl68YGk4MuS5/e48h1jaANE3SJ7j7rWZ2N/Ccmb3h7v9X3Qry8/PJy8urfcmPzM1osYKCgtqvWzHUewy5Ll8xNJwYcl1+Y4+hpKSk2hPnOKuAlhDO+Mt0BZYCmGXoXVEAAAyWSURBVFkHMzscwN2/Bp4HBsQYi4iIpIkzAUwHhgKYWW9gibuvjebtAEwws9bR+0MBjzEWERFJE1sVkLsXmlmRmRUCW4CRZjYcWO3uk83sRuAVM9tEuA10alyxiIjI1mJtA3D30WmT3k2ZNwGYEGf5IiJSNT0JLCKSUEoAIiIJpQQgIpJQSgAiIgmlBCAiklBKACIiCaUEICKSUEoAIiIJpQQgIpJQSgAiIgmlBCAiklBKACIiCaUEICKSUEoAIiIJpQQgIpJQsY4HYGbjgL5AKTDK3WelzBsMjAU2E0YDO8fdt8QZj4iIfCu2KwAzGwT0cPd+wAjgnrRF7geGuvsAYGfg2LhiERGRrcVZBTQEmALg7vOA9mbWJmV+gbsvil4vBzrGGIuIiKSJswqoM1CU8n55NG0NgLuvATCzLsDRwDU1rbC4uLj+o0xRVFRU80IxUwy5L18xNJwYcl3+9h5DrG0AaZqkTzCzXYFngAvc/cuaVpCfn09eXl7tS35kbkaLFRQU1H7diqHeY8h1+Yqh4cSQ6/IbewwlJSXVnjjHmQCWEM74y3QFlpa9iaqDngfGuPv0GOMQEZFKxNkGMB0YCmBmvYEl7r42Zf6dwDh3fyHGGEREpAqxXQG4e6GZFZlZIbAFGGlmw4HVwIvA2UAPMzsn+sgj7n5/XPGIiEhFsbYBuPvotEnvpryuQ2W+iIjUFz0JLCKSUEoAIiIJpQQgIpJQSgAiIgmlBCAiklBKACIiCaUEICKSUEoAIiIJpQQgIpJQSgAiIgmlBCAiklBKACIiCaUEICKSUEoAIiIJpQQgIpJQsY4HYGbjgL5AKTDK3WelzNsRuA84wN0PiTMOERHZWmxXAGY2COjh7v2AEcA9aYvcDvwrrvJFRKR6cVYBDQGmALj7PKB9NBB8mauAyTGWLyIi1YgzAXQGlqe8Xx5NAyBtgHgREcmyWNsA0jTZ1hUUFxfXRxxVKioqinX9iqFxlK8YGk4MuS5/e48hzgSwhJQzfqArsHRbVpifn09eXh3Gkn9kbkaLFRQU1H7diqHeY8h1+Yqh4cSQ6/IbewwlJSXVnjjHWQU0HRgKYGa9gSWq9hERaThiSwDuXggUmVkh4Q6gkWY23Mx+CGBmTwCPhZf2qpmdEVcsIiKytVjbANx9dNqkd1PmDYuzbBERqZ6eBBYRSSglABGRhFICEBFJKCUAEZGEUgIQEUkoJQARkYRSAhARSSglABGRhFICEBFJKCUAEZGEUgIQEUkoJQARkYRSAhARSSglABGRhFICEBFJKCUAEZGEinVAGDMbB/QFSoFR7j4rZd5RwH8Dm4Hn3P2mOGMREZGKYrsCMLNBQA937weMIAwLmeoe4DRgAHC0me0fVywiIrK1OK8AhgBTANx9npm1N7M27r7GzPYEVrj7pwBm9ly0/Nwq1tUMYOPGjXUKpEurHTJarqSkpE7rVwz1G0Ouy1cMDSeGXJff2GNIOWY2q2x+k9LS0jqtuCZmdj8wzd2fjt7PAEa4+wdm1h+4zN3LBogfAezl7ldVtq6ioqLDgBmxBCoisv0bWFBQ8Eb6xFjbANI0qeM8gFnAQGApoc1ARERq1gzoQjiGbiXOBLAE6JzyvivhAF7ZvO9E0ypVUFBQAmyVvUREpEYfVTUjzttApwNDAcysN7DE3dcCuPsCoI2Z7WFmzYETo+VFRCRLYmsDADCzW4HDgS3ASOBgYLW7Tzazw4H/iRZ90t3viC0QERHZSqwJQEREGi49CSwiklBKACIiCZXN20AbDDPLB54Gxrn773MUw22EW1ubA2Pd/akslt0SmADsBuwI3OTuz2ar/LRYdgKKoxgmZLnsI4AngPejSXPc/aJsxhDF8VPgcmATcK27T8ti2SOAs1ImHeLurbNVfhRDa2Ai0B7IA25w9xezHENTYDyQD2wEznf3+Vkqu8LxyMx2Bx4i3MK5FDjL3WN5Gi1xVwBm1gr4HfByDmMYDORH3WQcC/w2yyH8AJjt7oOAHwF3Zbn8VFcDK3JY/mvufkT0LxcH/47AdcBhhLvhTs5m+e7+QNn2R3H8JZvlR4aHUHww4c7Bu3MQw8lAW3fvT+i6Jis3pVRxPLoRuNfdBwIfAr+Iq/zEJQCgBDieap47yILXgWHR61VAKzOr9FHtOLj74+5+W/R2d2BRtspOZWb7AvsDWTvjbYCOAv7u7mvdfam7n5fDWK4FctEp4xdAx+h1++h9tvUA3gZw94+A72bpb7Ky49ERwNTo9TOE30gsElcF5O6bgE1mlssYNgNfRW9HEHpDzfoTzmZWCHQjnHnmwp3AhcDPclQ+wP5mNhXoQKh6eCnL5e8BtIxiaA9c7+5Zvzo1sz7Ap+7+WbbLdvfHzGy4mX1I+A5OyHYMwBzgEjP7LbA3sCewC7AszkKrOB61Sqny+ZzwJG8skngF0GCY2cmEBHBhLsqPLndPAh42s5q646hXZnY28Ka7/yeb5ab5N3AD4fL/Z8ADZtYiyzE0IZz9nkqoCvlztvdF5BxCu1DWmdmZwCfuvjdwJJD1djl3f55wBfA6cDEwj5q7qMmGWGNQAsgRMzsGGAMc5+6rs1x2QdTQhLv/i3Al2CmbMRDO8k42s5mEg8810RgRWePui6PqsNLosv8zQrck2bQMKHT3TVEMa8n+voBQ7VCYg3IhdAn/IoC7vwt0zWaVaBl3v9rdB7j7rwhXIp9nO4bIuujmCKihm5xtpQSQA2bWFrgdONHdc9EAejjwmyiW3YDWZLne1d1/7O593L0v8CfCXUB/z2YMZvZTM7s0et2ZcFfU4mzGQOgC5Ugzaxo1CGd9X5hZV2Cdu9etv/Vt9yHw/SiW70axZLVK1Mx6mdmD0etjgXfcfUs2Y0jxd8JYKUT/vxBXQYlrAzCzAkLd8x7AN2Y2FDg1ywfiHxPqF/+WUvd3trt/kqXyxxOqO2YAOwEjc/hjz6WpwCNRVVwL4FfZPgi6+2IzmwTMjCZdlIN90YXcne0C3Ac8aGavEY5J5+cghjlAUzN7G9gA/DQbhVZxPPopMMHMfgksJMY7s9QVhIhIQqkKSEQkoZQAREQSSglARCShlABERBJKCUBEJKESdxuoNHxRT6mHEnoqPRh4M5r1gLs/lOE6XgWGVHU/eXSvd4G737KNsR4B3Ozuh9Xhs/2Bz9z9422JQaSulACkwXH3ywHMbA/gjainytquo9rPuPsLxPiATYZ+DjwOKAFITigBSKNiZgsIB8093X2Ymd0IDIlmLwLOdPdvzKwU2IHQ3XRHQqd3PYBX3P0iMxsOHOXuZ0brvBs4DvgeoS/4l6OHdO4H1gHPEfoNah114FVZbNdXUVZ+tJ4SoCWhu98WhB5hDzWzSwh90P9PyjIXuPs7ZjaB0BVAT2AfwlXQbVFXAX8GukfFX+nur0VdjV9H6EPmG+Bcd/9PND73kdH6FwM/i6uPeWk81AYgjdG/o4N/c2A9MNDdBwDtgGMqWf5gQj/zfYCfm1n7Spb52t2PBm4Gfh1Nu5vQQ+ggQrfdeRnEVllZ5wJPR/3d/wDo6O6TgX8Bv3H3fxCeDP+Vux8ZlXtVyjr3dPcfAEcT+o8CuJTQe2d/Qkd250QD/YwnPNk+iNDP/B1RDCOBflEf808Rur2QhFMCkMaoEMq70t0MzIi6ETiIcCBN94a7b3b3rwn97HSoZJlXo/8Xpsw/KGX6pAxjq6ysJ4FfmtkfgEMIoz2l+4xwsH4dGJ22Ha8CuPtCoE3UUdr3U6b/293PIoxm1QV4KmoDuRTo5O4rCZ2tvWZmvyF0PpetbkekAVMCkMZoI4CZDSCMlnR0dMY7o4rl06tsKutid1Ml85sCZf3yZNo52VZlufvrhIPz84Qun/9ayeceAm5198P59iy/ynUCpWz991tC6Fa5bISzgdH6cPehhF5XISSCgzLcHtmOKQFIY7YbsMDdv4p6kexLZtU0mZoP9I9en1rXlZjZRUA3d3+GMP7D96NZWwjtFBC25f3o7H4YNW9HIWE4UcxsDzN7GfgA2CVqc8DMDjez88xsTzO7xN3nu/udhCqgXnXdHtl+qBFYGrPpwG/M7A3CwO7XA9ea2Sv1tP5Lgd+b2RLCsJWlfHtFUBvzgUfNbA1hoO/R0fSXgPvM7GJCA/A/CFVQtwMPRdOrcg/wx6hH12bAGHf/Ohpc5QEz2xAtdx6hcfzgqKfLtcBKQoO2JJx6AxWpQnRHzQp3f9fMegOPunvuxhIVqWe6AhCp2jfAn6Kz6RbAL3Mcj0i90hWAiEhCqRFYRCShlABERBJKCUBEJKGUAEREEkoJQEQkof4/5CFIMpze6k0AAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cEx3Z6vt20hI",
        "outputId": "a56cd4e2-ec4a-49ff-aded-da913a31e8a7"
      },
      "source": [
        "# Stratified K-Fold CV\n",
        "for k in range(2,11):\n",
        "\n",
        "    skfold = StratifiedKFold(n_splits=k, random_state=100)\n",
        "    model_skfold = LogisticRegression()\n",
        "    results_skfold = cross_val_score(model_skfold, x1, y1, cv=skfold)\n",
        "\n",
        "    print(\"Accuracy: %.2f%%\" % (results_skfold.mean()*100.0))"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Accuracy: 76.82%\n",
            "Accuracy: 77.34%\n",
            "Accuracy: 76.95%\n",
            "Accuracy: 76.96%\n",
            "Accuracy: 76.82%\n",
            "Accuracy: 77.87%\n",
            "Accuracy: 77.47%\n",
            "Accuracy: 77.22%\n",
            "Accuracy: 77.61%\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EV5UW8413jkp"
      },
      "source": [
        "### Shuffle Split Approach"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xHCnZr6q26qV",
        "outputId": "e389c41a-35f7-4de6-f25b-6bd490258270"
      },
      "source": [
        "for k in range(2,11):\n",
        "\n",
        "    kfold2 = ShuffleSplit(n_splits=k, test_size=0.30, random_state=100)\n",
        "    model_shufflecv = LogisticRegression()\n",
        "    results_4 = model_selection.cross_val_score(model_shufflecv, x1, y1, cv=kfold2)\n",
        "    print(\"Accuracy: %.2f%% (%.2f%%)\" % (results_4.mean()*100.0, results_4.std()*100.0))"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Accuracy: 76.84% (1.52%)\n",
            "Accuracy: 77.78% (1.81%)\n",
            "Accuracy: 78.14% (1.69%)\n",
            "Accuracy: 77.40% (2.11%)\n",
            "Accuracy: 77.20% (1.98%)\n",
            "Accuracy: 76.38% (2.73%)\n",
            "Accuracy: 76.19% (2.60%)\n",
            "Accuracy: 75.61% (2.94%)\n",
            "Accuracy: 75.45% (2.83%)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0A64bm_I3l8k"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}