Project 3 Cross Validation.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "HW3_Cross_Validation.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/HarryWei530/Frequentist_ML/blob/master/HW3%20Cross%20Validation.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XZpSUUJknAF4"
      },
      "source": [
        "#Assignment 3\n",
        "Re-implement the example in section 7.10.2 using any simple, out of the box classifier (like K nearest neighbors from sci-kit). Reproduce the results for the incorrect and correct way of doing cross-validation."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LkgDVwd73aRR"
      },
      "source": [
        "## Analysis 7.10.2 Wrong and Right way of cross validation\n",
        "\n",
        "Wrong way:\n",
        "1. Screen the predictors: find a subset of “good” predictors that show\n",
        "fairly strong (univariate) correlation with the class labels\n",
        "2. Using just this subset of predictors, build a multivariate classifier.\n",
        "3. Use cross-validation to estimate the unknown tuning parameters and\n",
        "to estimate the prediction error of the final model\n",
        "\n",
        "The problem is that the predictors have an unfair advantage, as they were chosen in step (1) on the basis of all of the samples. Leaving samples out after the variables have been selected does not correctly mimic the application of the classifier to a completely independent test set, since these predictors “have already seen” the left out samples.\n",
        "___\n",
        "\n",
        "Right way: K-folds cross validation\n",
        "\n",
        "1. Divide the samples into K cross-validation folds (groups) at random.\n",
        "2. For each fold k = 1, 2, . . . , K\n",
        "\n",
        "    a. Find a subset of “good” predictors that show fairly strong (univariate) correlation with the class labels, using all of the samples except those in fold k.\n",
        "\n",
        "    b. Using just this subset of predictors, build a multivariate classifier, using all of the samples except those in fold k.\n",
        "    \n",
        "    c. Use the classifier to predict the class labels for the samples in fold k.\n",
        "\n",
        "The error estimates from step 2(c) are then accumulated over all K folds, to\n",
        "produce the cross-validation estimate of prediction error\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bqbY5yDK5Pw6"
      },
      "source": [
        "## Generate random dataset\n",
        "Dataframe contains 50 observations and 5000 features. Features generated using standard gaussian distribution. True error accuracy is 50%."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "krmRO8SJm6_M"
      },
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "from sklearn.neighbors import KNeighborsClassifier\n",
        "from sklearn.metrics import accuracy_score\n",
        "from sklearn.model_selection import cross_val_score\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "\n",
        "mu, sigma = 0, 1 # mean and standard deviation\n",
        "s = np.random.normal(mu, sigma, [50,5000])\n",
        "df = pd.DataFrame(s)\n",
        "df['label'] = np.zeros([df.shape[0],1])\n",
        "df['label'][0:round(df.shape[0]/2)] = 1  # 50% correct\n",
        "# display(df.head())"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q_TmIjoq7-Dq"
      },
      "source": [
        "## Wrong Way of Cross Validation\n",
        "1. Screen the predictors: find a subset of “good” predictors that show\n",
        "fairly strong (univariate) correlation with the class labels\n",
        "2. Using just this subset of predictors, build a multivariate classifier.\n",
        "3. Use cross-validation to estimate the unknown tuning parameters and\n",
        "to estimate the prediction error of the final model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xo8sqJ6Z77x_",
        "outputId": "9d64d4c6-1cbe-420e-8c48-6b543a936281",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 296
        }
      },
      "source": [
        "def generate_df(): \n",
        "  mu, sigma = 0, 1 # mean and standard deviation\n",
        "  s = np.random.normal(mu, sigma, [50,5000])\n",
        "  df = pd.DataFrame(s)\n",
        "  df['label'] = np.zeros([df.shape[0],1])\n",
        "  df['label'][0:round(df.shape[0]/2)] = 1  # 50% correct\n",
        "  # display(df.head())\n",
        "  return df\n",
        "\n",
        "def select_feature(df,num_feat,plot = False):\n",
        "  local_df = df.copy()\n",
        "  # Feature correlation using Pearson Correlation\n",
        "  correlation = []\n",
        "  correlation_abs = []\n",
        "  for features in df.columns[:-1]:\n",
        "    correlation.append(df[features].corr(df['label']))\n",
        "    correlation_abs.append(abs(df[features].corr(df['label'])))\n",
        "  local_df = local_df.append(pd.Series(correlation,name=\"correlation\"),ignore_index = False)\n",
        "  local_df = local_df.append(pd.Series(correlation_abs,name=\"correlation abs\"),ignore_index = False)\n",
        "  local_df = local_df.sort_values(by=(\"correlation\"),axis = 1, ascending=False)\n",
        "  local_df = local_df[local_df.columns[0:num_feat]]\n",
        "  correlation = local_df.loc['correlation',:]\n",
        "  correlation_abs = local_df.loc['correlation abs',:]\n",
        "  local_df = local_df.drop([\"correlation\", \"correlation abs\"],axis=0)\n",
        "\n",
        "  # if(plot == True):\n",
        "  #   plt.figure()\n",
        "  #   plt.hist(correlation)\n",
        "  #   plt.xlabel(\"Correlations of Selected Predictors\")\n",
        "  #   plt.ylabel(\"Frequency\")\n",
        "  #   plt.title(\"Wrong way\")\n",
        "  return local_df\n",
        "\n",
        "def classify(features,labels):\n",
        "  neigh = KNeighborsClassifier(n_neighbors=1,metric='euclidean')\n",
        "  neigh.fit(features, labels)\n",
        "  return neigh\n",
        "\n",
        "def cross_validation(features,labels,clf):\n",
        "  pred = clf.predict(features)\n",
        "  score = accuracy_score(labels,pred,normalize=True)\n",
        "  print(score)\n",
        "  return score\n",
        "\n",
        "def wrong_cross_validation(plot=False, print_output=False):\n",
        "  df = generate_df()\n",
        "  df = df.sample(frac=1,random_state=1).reset_index(drop=True)\n",
        "  # Select 100 features\n",
        "  features = select_feature(df,100,plot=False)\n",
        "  labels = df['label']\n",
        "  selected_feats = features.columns\n",
        "  # Create classifier object, perform cross validation, and reprot score\n",
        "  neigh = KNeighborsClassifier(n_neighbors=1,metric='euclidean')\n",
        "  score = cross_val_score(neigh,X=features,y=labels,cv = 5)\n",
        "  score = sum(score)/score.shape[0]\n",
        "\n",
        "  if(print_output == True):\n",
        "    print(f\"CV score = {round(score,3)}\")\n",
        "\n",
        "  # Plotting correaltion histogram using randomly selected 10 samples\n",
        "  if(plot == True):\n",
        "    sample_df = df.copy()\n",
        "    selected_feats = selected_feats.append(pd.Index([\"label\"]))\n",
        "    sample_df = df.sample(frac=0.2,random_state=2).loc[:][selected_feats]    \n",
        "    sample_corr = []\n",
        "    for features in sample_df.columns[:-1]:\n",
        "      sample_corr.append(sample_df[features].corr(sample_df['label']))\n",
        "\n",
        "    plt.figure()\n",
        "    plt.hist(sample_corr)\n",
        "    plt.xlabel(\"Correlation Coefficient\")\n",
        "    plt.ylabel(\"Frequency\")\n",
        "    plt.show()\n",
        "  return score\n",
        "\n",
        "\n",
        "# Run CV 50 times, find average error rate\n",
        "error_rate = 0\n",
        "num_cycle = 50\n",
        "for i in range(num_cycle):\n",
        "  error_rate += (1- wrong_cross_validation(plot=False))\n",
        "print(f\"Incorrect Cross Validation Error Rate is {round(error_rate/num_cycle*100)}%.\")\n",
        "\n",
        "# Gnerate a correlation histogram\n",
        "_ = wrong_cross_validation(plot=True)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Incorrect Cross Validation Error Rate is 3.0%.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAXUUlEQVR4nO3de5gldX3n8ffHAeQiCIQWZ1EcVLzwqAzYoCYSEcUQSRBXE3XFxX1cx/WWaJJd8fJEEpNd1FU2akwYxWV0vQZFCeBlQLxFuQw4jMMQRRHdwZEZkyhiXBT47h9VHQ8z3dOnZ7rO6el6v57nPFNVp+rU53T3fLq6Tp3fSVUhSeqPe407gCRptCx+SeoZi1+Sesbil6SesfglqWd2G3eAYRx00EG1bNmycceQpF3KNddc86Oqmth6eWfFn2RP4EvAvdv9nF9Vb0xyHvAk4Cftqi+sqrXbe6xly5axZs2arqJK0qKU5HvTLe/yiP8O4ISquj3J7sBXkny6ve+/VtX5He5bkjSDzoq/mneG3d7O7t7efLeYJI1Zpy/uJlmSZC2wGVhdVVe2d/1lknVJzk5y7y4zSJLuqdPir6q7qmo58ADg2CSPAl4LPAI4BjgQeM102yZZkWRNkjVbtmzpMqYk9cpILuesqh8DlwMnVdWmatwB/G/g2Bm2WVlVk1U1OTGxzYvSkqQd1FnxJ5lIsn87vRdwIvCPSZa2ywKcCqzvKoMkaVtdXtWzFFiVZAnNL5iPVdVFST6fZAIIsBb4Lx1mkCRtpcuretYBR02z/ISu9ilJmp1DNkhSz+wSQzZIWjiWnXHx2PZ981knj23fi4lH/JLUMxa/JPWMxS9JPWPxS1LPWPyS1DMWvyT1jMUvST1j8UtSz1j8ktQzFr8k9YzFL0k9Y/FLUs9Y/JLUMxa/JPWMxS9JPWPxS1LPWPyS1DMWvyT1jMUvST1j8UtSz3RW/En2THJVkuuSXJ/kz9rlhyW5Msm3k3w0yR5dZZAkbavLI/47gBOq6khgOXBSkscDbwbOrqqHAv8CvKjDDJKkrXRW/NW4vZ3dvb0VcAJwfrt8FXBqVxkkSdvq9Bx/kiVJ1gKbgdXAd4AfV9Wd7SobgUNm2HZFkjVJ1mzZsqXLmJLUK50Wf1XdVVXLgQcAxwKPmMO2K6tqsqomJyYmOssoSX0zkqt6qurHwOXAE4D9k+zW3vUA4JZRZJAkNbq8qmciyf7t9F7AicANNL8Ant2udjrwqa4ySJK2tdvsq+ywpcCqJEtofsF8rKouSrIB+EiSvwC+DpzbYQZJ0lY6K/6qWgccNc3ym2jO90uSxsB37kpSz1j8ktQzFr8k9YzFL0k9Y/FLUs9Y/JLUMxa/JPWMxS9JPWPxS1LPWPyS1DMWvyT1jMUvST1j8UtSz1j8ktQzFr8k9YzFL0k9Y/FLUs9Y/JLUMxa/JPWMxS9JPWPxS1LPdFb8SR6Y5PIkG5Jcn+QP2+VnJrklydr29vSuMkiStrVbh499J/DHVXVtkn2Ba5Ksbu87u6r+Z4f7liTNoLPir6pNwKZ2+qdJbgAO6Wp/kqThjOQcf5JlwFHAle2iVyRZl+R9SQ4YRQZJUqPz4k9yH+DjwKuq6jbgb4CHAMtp/iJ42wzbrUiyJsmaLVu2dB1Tknqj0+JPsjtN6X+wqj4BUFW3VtVdVXU38B7g2Om2raqVVTVZVZMTExNdxpSkXunyqp4A5wI3VNXbB5YvHVjtmcD6rjJIkrbV5VU9vwG8APhGkrXtstcBz0uyHCjgZuAlHWaQJG2ly6t6vgJkmrsu6WqfkqTZ+c5dSeoZi1+Sesbil6SesfglqWcsfknqGYtfknrG4peknrH4JalnLH5J6hmLX5J6xuKXpJ6x+CWpZyx+SeoZi1+Sesbil6SeGar4kzy66yCSpNEY9oj/3UmuSvKyJPftNJEkqVNDFX9VHQc8H3ggcE2SDyU5sdNkkqRODH2Ov6puBN4AvAZ4EvCOJP+Y5N93FU6SNP+GPcf/mCRnAzcAJwC/W1WPbKfP7jCfJGmeDfth6+8E3gu8rqp+PrWwqn6Q5A2dJJMkdWLY4j8Z+HlV3QWQ5F7AnlX1r1X1gc7SSZLm3bDn+C8F9hqY37tdNqMkD0xyeZINSa5P8oft8gOTrE5yY/vvATsWXZK0I4Yt/j2r6vapmXZ671m2uRP446o6Ang88PIkRwBnAJdV1eHAZe28JGlEhi3+nyU5emomyWOBn29nfapqU1Vd207/lOaF4UOAZwCr2tVWAafONbQkaccNe47/VcDfJfkBEOD+wHOG3UmSZcBRwJXAwVW1qb3rh8DBM2yzAlgBcOihhw67K0nSLIYq/qq6OskjgIe3i75ZVb8cZtsk9wE+Dryqqm5LMvi4laRm2OdKYCXA5OTktOtIkuZu2CN+gGOAZe02Ryehqt6/vQ2S7E5T+h+sqk+0i29NsrSqNiVZCmzegdySpB00VPEn+QDwEGAtcFe7uIAZiz/Nof25wA1V9faBuy4ETgfOav/91NxjS5J21LBH/JPAEVU1l1MuvwG8APhGkrXtstfRFP7HkrwI+B7w+3N4TEnSThq2+NfTvKC7abYVp1TVV2heCJ7OU4Z9HEnS/Bq2+A8CNiS5CrhjamFVndJJKklSZ4Yt/jO7DCFJGp1hL+f8YpIHAYdX1aVJ9gaWdBtNktSFYYdlfjFwPnBOu+gQ4JNdhZIkdWfYIRteTnOVzm3wbx/Kcr+uQkmSujNs8d9RVb+YmkmyG811/JKkXcywL+5+McnrgL3az9p9GfD33cWSNJtlZ1w87gjaRQ17xH8GsAX4BvAS4BKaz9+VJO1ihr2q527gPe1NkrQLG3asnu8yzTn9qnrwvCeSJHVqLmP1TNkT+D3gwPmPI0nq2lDn+KvqnwZut1TV/6L5AHZJ0i5m2FM9Rw/M3ovmL4C5jOUvSVoghi3vtw1M3wncjMMpS9Iuadirep7cdRBJ0mgMe6rnj7Z3/1afsCVJWsDmclXPMTQfmwjwu8BVwI1dhJIkdWfY4n8AcHRV/RQgyZnAxVV1WlfBJEndGHbIhoOBXwzM/6JdJknaxQx7xP9+4KokF7TzpwKruokkSerSsFf1/GWSTwPHtYv+U1V9vbtYkqSuDHuqB2Bv4Laq+itgY5LDtrdykvcl2Zxk/cCyM5PckmRte3v6DuaWJO2gYT968Y3Aa4DXtot2B/7PLJudB5w0zfKzq2p5e7tk2KCSpPkx7BH/M4FTgJ8BVNUPgH23t0FVfQn4551KJ0mad8MW/y+qqmiHZk6yz07s8xVJ1rWngg6YaaUkK5KsSbJmy5YtO7E7SdKgYYv/Y0nOAfZP8mLgUnbsQ1n+BngIsBzYxD3HALqHqlpZVZNVNTkxMbEDu5IkTWfWq3qSBPgo8AjgNuDhwJ9W1eq57qyqbh143PcAF831MSRJO2fW4q+qSnJJVT0amHPZD0qytKo2tbPPBNZvb31J0vwb9g1c1yY5pqquHvaBk3wYOB44KMlG4I3A8UmW07xWcDPNB7dLkkZo2OJ/HHBakptpruwJzR8Dj5lpg6p63jSLz51zQknSvNpu8Sc5tKq+D/zWiPJIkjo22xH/J2lG5fxeko9X1bNGEUqS1J3ZLufMwPSDuwwiSRqN2Yq/ZpiWJO2iZjvVc2SS22iO/Pdqp+FXL+7u12k6SdK8227xV9WSUQWRJI3GXIZlliQtAsNexy9JY7fsjIvHst+bzzp5LPvtikf8ktQzFr8k9YzFL0k9Y/FLUs9Y/JLUMxa/JPWMxS9JPWPxS1LPWPyS1DMWvyT1jMUvST3jWD3SThjX2DHSzvCIX5J6xuKXpJ7prPiTvC/J5iTrB5YdmGR1khvbfw/oav+SpOl1ecR/HnDSVsvOAC6rqsOBy9p5SdIIdVb8VfUl4J+3WvwMYFU7vQo4tav9S5KmN+pz/AdX1aZ2+ofAwTOtmGRFkjVJ1mzZsmU06SSpB8b24m5VFVDbuX9lVU1W1eTExMQIk0nS4jbq4r81yVKA9t/NI96/JPXeqIv/QuD0dvp04FMj3r8k9V6Xl3N+GPga8PAkG5O8CDgLODHJjcBT23lJ0gh1NmRDVT1vhrue0tU+JUmzc6weLQqOmSMNzyEbJKlnLH5J6hmLX5J6xuKXpJ6x+CWpZyx+SeoZi1+Sesbil6SesfglqWcsfknqGYtfknrGsXokaRbjHAvq5rNOnvfH9IhfknrG4peknrH4JalnLH5J6hmLX5J6xuKXpJ6x+CWpZyx+SeqZsbyBK8nNwE+Bu4A7q2pyHDkkqY/G+c7dJ1fVj8a4f0nqJU/1SFLPjKv4C/hckmuSrJhuhSQrkqxJsmbLli0jjidJi9e4iv+JVXU08NvAy5P85tYrVNXKqpqsqsmJiYnRJ5SkRWosxV9Vt7T/bgYuAI4dRw5J6qORF3+SfZLsOzUNPA1YP+ocktRX47iq52DggiRT+/9QVX1mDDkkqZdGXvxVdRNw5Kj3K0lq+Alcmjfj/JQiScPzOn5J6hmLX5J6xuKXpJ6x+CWpZyx+SeoZi1+SesbLORchL6uUtD0e8UtSz1j8ktQzFr8k9YzFL0k9Y/FLUs9Y/JLUMxa/JPWMxS9JPWPxS1LPWPyS1DMWvyT1zKIfq2ec49bcfNbJY9u3JM3EI35J6hmLX5J6ZizFn+SkJN9M8u0kZ4wjgyT11ciLP8kS4K+B3waOAJ6X5IhR55CkvhrHEf+xwLer6qaq+gXwEeAZY8ghSb00jqt6DgH+78D8RuBxW6+UZAWwop29Pck357ifg4Af7VDCeZI3b7No7JmmsRAzwcLMZabhLcRcu2SmaXpkLh403cIFezlnVa0EVu7o9knWVNXkPEbaaWYa3kLMZabhLcRcZvqVcZzquQV44MD8A9plkqQRGEfxXw0cnuSwJHsAzwUuHEMOSeqlkZ/qqao7k7wC+CywBHhfVV3fwa52+DRRh8w0vIWYy0zDW4i5zNRKVY1jv5KkMfGdu5LUMxa/JPXMoin+JAcmWZ3kxvbfA2ZY7y1Jrk9yQ5J3JMkCyHRoks+1mTYkWTbuTO26+yXZmORdXeWZS64ky5N8rf3+rUvynI6ybHdIkST3TvLR9v4ru/x+zSHTH7U/O+uSXJZk2uu3R5lpYL1nJakkI7lscZhcSX6//Xpdn+RD487UdsDlSb7efg+f3mmgqloUN+AtwBnt9BnAm6dZ59eBf6B5UXkJ8DXg+HFmau/7AnBiO30fYO9xZ2rv/yvgQ8C7Fsj372HA4e30vwM2AfvPc44lwHeABwN7ANcBR2y1zsuAv22nnwt8tOOvzTCZnjz1cwO8dCFkatfbF/gScAUwOYKfo2G+VocDXwcOaOfvtwAyrQRe2k4fAdzcZaZFc8RPM+zDqnZ6FXDqNOsUsCfNF//ewO7ArePM1I5TtFtVrQaoqtur6l/HmanN9VjgYOBzHWaZU66q+lZV3dhO/wDYDEzMc45hhhQZzHo+8JQu/3IcJlNVXT7wc3MFzftjujTs0CtvAt4M/L+O88wl14uBv66qfwGoqs0LIFMB+7XT9wV+0GWgxVT8B1fVpnb6hzSldQ9V9TXgcpojxU3AZ6vqhnFmojmK/XGST7R/5r21HchubJmS3At4G/AnHeaYc65BSY6l+QX+nXnOMd2QIofMtE5V3Qn8BPi1ec4x10yDXgR8usM8MESmJEcDD6yqUX4a0jBfq4cBD0vyD0muSHLSAsh0JnBako3AJcAruwy0YIdsmE6SS4H7T3PX6wdnqqqSbHOdapKHAo/kV0dDq5McV1VfHlcmmu/BccBRwPeBjwIvBM4dY6aXAZdU1cb5PJCdh1xTj7MU+ABwelXdPW8BF4EkpwGTwJPGnONewNtpfpYXmt1oTvccT9MFX0ry6Kr68RgzPQ84r6reluQJwAeSPKqrn+9dqvir6qkz3Zfk1iRLq2pTWwzT/fn2TOCKqrq93ebTwBOAHS7+eci0EVhbVTe123wSeDw7UfzzkOkJwHFJXkbzmsMeSW6vqp367IR5yEWS/YCLgddX1RU7k2cGwwwpMrXOxiS70fxp/k8dZJlLJpI8leaX6JOq6o4O8wyTaV/gUcAX2oOH+wMXJjmlqtaMMRc0/+eurKpfAt9N8i2aXwRXjzHTi4CToDkzkWRPmgHcOjkNtZhO9VwInN5Onw58app1vg88KcluSXanOSrq8lTPMJmuBvZPMnWu+gRgwzgzVdXzq+rQqlpGc7rn/Ttb+vORK80QHxe0ec7vKMcwQ4oMZn028PlqX5UbV6YkRwHnAKeM4Jz1rJmq6idVdVBVLWt/jq5os3VZ+rPman2S5mifJAfRnPq5acyZvg88pc30SJrXIrd0lqjLV45HeaM5x3oZcCNwKXBgu3wSeG/96tX1c2jKfgPw9nFnaudPBNYB3wDOA/YYd6aB9V/IaK7qGeb7dxrwS2DtwG15B1meDnyL5vWD17fL/pymuKD5T/l3wLeBq4AHj+DrM1umS2kuVJj6ulw47kxbrfsFRnBVz5Bfq9CchtrQ/p977gLIdATNFYfXtd+/p3WZxyEbJKlnFtOpHknSECx+SeoZi1+Sesbil6SesfglqWcsfnUuyf2TfCTJd5Jck+SSJA/rYD/LkqwfYp3/MDA/meQd87T/+yQ5Z+B5fiHJ43bwsX4vzWitl7fzH25HbXx1kj9v36w107Y79ZySvG5Ht9Wuwcs51al28LKvAquq6m/bZUcC+9UQQ2Uk2a2a8XCmnd9q3WXARVX1qO083vHAn1TV78zleQwjyUeA79Jcp313ksNoRmGc81g1ST4D/EVVfSXJ/YGvVNVD5znyTPu+varuM4p9aTw84lfXngz8cqr0Aarquqr6chpvTbI+yTfSjq+f5PgkX05yIbBhmvkl7XZXt0fBL9l6p+2R/ZeTXNvefr296yya4SjWtkfPxye5qN3mwCSfbB/ziiSPaZefmeR97RH8TUn+YJr9PQR4HPCGasdXqarvTpV+mvHy17e3Vw1sd1qSq9o857TP7U+BJwLnJnkrzQiph7TrHJfkvCTPbrc/JslXk1zXPs6+Wz2nfdrsV6UZBPAZ7fIXphkY8DNpPgPhLe3ys4C92n19cEe/6VrgRvFOOm/9vQF/AJw9w33PAlbTvKP6YJq3rS+leTv9z4DD2vW2nl9BU7DQDK+9BjgMWAasb5fvDezZTh8OrBl4rIsGMvzbPPBO4I3t9Ak0YyhBM3LiV9t9HUQzLs/uWz2XU4ALZniej6V5h+g+NGMfXU8zKN8jgb+feizg3cB/bKe/QPtO18Hn1c6fRzNUxB40Qw0c0y7fj2b8rcHn9N+B09rp/WnePboPzTuyb6IZZ2hP4Hs0I2kC3D7unxtv3d52qUHatOg8EfhwVd0F3Jrki8AxwG3AVVX13YF1B+efBjxm6qiXprwOpym1KbsD70qyHLiLZjyWYfI8C6CqPp/k19IMCgdwcTUDn92RZDPNL6qNc3ieF1TVzwCSfIJmRNa7aX4pXN2cEWMv5jYo18OBTVV1dZv5tvbxB9d5GnBKkqkhtvcEDm2nL6uqn7TbbAAexD2HD9YiZfGra9fTHJ3O1c+2Mx/glVX12cEVcs+PQHw1zdg1R9Kc0tzZDwIZHO3yLrb9v3M9cGSSJe0vsmGE5rWP1+5kttn28ayq+uY9FjYvOs/2nLRIeY5fXfs8cO8kK6YWJHlMkuNohsN+TnteewL4TZpBz2bzWeClaUZYJcnDkuyz1Tr3pTkavht4Ac3pJICf0gwZPJ0vA89vH/N44EdTR9Gzqarv0Jxy+rP2Be2p1xlObh/31CR7tzmf2S67DHh2kvu16x+YuX1W7jeBpUmOabffN80w0YM+C7xyINNRQzzuL6e+tlqcLH51qqqKpuiemuYyx+uB/0HzKVsX0IxKeh3NL4j/VlU/HOJh30szsuK1aS7fPIdtj1bfDZye5DrgEfzqL4Z1wF3ti6Gv3mqbM4HHJllH8yLw6czNf6Y5BfTtNtd5wOaquradvgq4kma00a9X1QbgDcDn2n2upnmNYyjVfIzfc4B3ts9zNc2pnEFvojntta792r9piIde2a7vi7uLlJdzSlLPeMQvST1j8UtSz1j8ktQzFr8k9YzFL0k9Y/FLUs9Y/JLUM/8f6BFRTiJJKuMAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hEVXt0MsACjv"
      },
      "source": [
        "## Right Way of Cross Validation Using K folds\n",
        "1. Divide the samples into K folds at random\n",
        "2. For each fold k in K, use the other folds to determine \n",
        "good predictors based on highest correlation to create a classifier\n",
        "3. Use this classifier on fold k, and repeat the process\n",
        "\n",
        "In this example, we used K Nearest Neighbors with n = 1.\n",
        "Our random state of 2 resulted in 50% correct accuracy/50% \n",
        "error rate."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bhAcWWvKhpLY"
      },
      "source": [
        "from sklearn.neighbors import KNeighborsClassifier\n",
        "\n",
        "def score(test_feat,test_label,print_output = False):\n",
        "  \"\"\"\n",
        "  Accuracy of validation label to predicted validation label\n",
        "  Inputs are numpy arrays \n",
        "  \"\"\"\n",
        "  num_correct = 0\n",
        "  for i in range(len(test_label)):\n",
        "    if(test_feat[i] == test_label[i]):\n",
        "      num_correct += 1\n",
        "  accuracy = num_correct/len(test_label)\n",
        "  if(print_output == True):\n",
        "    print(f\"Accuracy of current model is {round(accuracy,2)}\\n\")\n",
        "  return accuracy\n",
        "\n",
        "\n",
        "def right_cross_validation(df,k):\n",
        "  \"\"\"\n",
        "  df is an input dataframe, and k is the number of folds\n",
        "  returns the predicted error\n",
        "  \"\"\"\n",
        "  df_split = np.array_split(df, k)\n",
        "  error = [None]*k\n",
        "  for i in range(len(df_split)):\n",
        "    train = df_split[0:i] + df_split[i+1:len(df_split)] #excludes the validation and creates a list of numpy arrays\n",
        "    train = pd.concat(train) #combines the list into 1 dataframe\n",
        "    y_train = train[\"label\"]\n",
        "    x_train = train.drop([\"label\"], axis=1)\n",
        "\n",
        "    validation = df_split[i] #pops the validation array\n",
        "    validation = pd.DataFrame(validation) #converts the validation array into a dataframe \n",
        "    y_validation = validation[\"label\"]\n",
        "    x_validation = validation.drop([\"label\"], axis=1)\n",
        "\n",
        "    knn = KNeighborsClassifier(n_neighbors=1, metric='euclidean')\n",
        "    knn.fit(x_train, y_train)\n",
        "    y_pred = knn.predict(x_validation)\n",
        "\n",
        "    error[i] = score(y_pred,y_validation.to_numpy())\n",
        "\n",
        "  return(np.mean(error)) #average of errors for each fold"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LEoggGFAhtDx",
        "outputId": "82066134-e2b1-4b00-b955-3d3534d01346",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "correct_df = df\n",
        "correct_df = correct_df.sample(frac = 1, random_state =2)\n",
        "success = right_cross_validation(correct_df,5)\n",
        "error = 1-success\n",
        "\n",
        "print(f\"Correct Cross Validation Error Rate is {round(error,5)*100}%.\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Correct Cross Validation Error Rate is 38.0%.\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}