06_Lab_3_SVM_Classification_Evaluation_solution.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introductory applied machine learning (INFR10069) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 3: Support Vector Machine (SVM) Classification and Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab we initially re-examine the spam filtering problem from Lab 2. This time, we train a Logistic Regression model and a linear Support Vector Machine for the spam or non-spam classification task. In the second part of the lab we examine classification evaluation by using a K-nearest neighbour classifier.\n",
    "\n",
    "\n",
    "All the datasets that you will need for this lab are located at the `./datasets` directory which is adjacent to this file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Import packages\n",
    "from __future__ import division, print_function # Imports from __future__ since we're running Python 2\n",
    "import os\n",
    "import numpy as np \n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.svm import LinearSVC, SVC\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "KNeighboursClassifier = KNeighborsClassifier\n",
    "from sklearn.metrics import confusion_matrix, log_loss\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Spam filtering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.1 ==========\n",
    "Load `spambase_binary.csv` into a pandas DataFrame structure called `spambase`. Display the number of instances and attributes and the first 5 samples. Remember that the attributes have been binarised. The instances have also been shuffled (i.e. their order has been randomised). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "data_path = os.path.join(os.getcwd(), 'datasets', 'spambase_binary.csv')\n",
    "spambase = pd.read_csv(data_path, delimiter = ',')\n",
    "print('Number of instances: {}, number of attributes: {}'.format(spambase.shape[0], spambase.shape[1]))\n",
    "spambase.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.2 ==========\n",
    "We are going to use hold-out validation to evaluate our models below. Split the dataset into training and testing subsets; there is a [built-in sklearn function](http://scikit-learn.org/0.17/modules/generated/sklearn.cross_validation.train_test_split.html) to do this. Call the resulting matrices `X_train`, `X_test`, `y_train`, `y_test`. Use 90% of the data for training and the remaining 10% for testing. Make sure you don't include the target variable `is_spam` in the input features (`X_train` / `X_test`)!\n",
    "\n",
    "If you want to be able to reproduce your results exactly, what argument must you remember to set?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "X_train, X_test, y_train, y_test = train_test_split(spambase.drop(\"is_spam\", axis=1), spambase[\"is_spam\"],\n",
    "                                                    train_size=0.9, test_size=0.1, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n",
    "You must set the `random_state` argument, otherwise, every time you run the `train_test_split` function, you will get a different random split of the data. This will give you different results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.3 ==========\n",
    "Train a [`LogisticRegression`](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) classifier by using training data. Use the `lbfgs` solver and default settings for the other parameters. Report the classification accuracy on both the training and test sets. Does your classifier generalise well on unseen data?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "lr = LogisticRegression(solver='lbfgs')\n",
    "lr.fit(X_train, y_train)\n",
    "print('Classification accuracy on training set: {:.3f}'.format(lr.score(X_train, y_train)))\n",
    "print('Classification accuracy on test set: {:.3f}'.format(lr.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n",
    "Indeed, the classifier generalises well since the classification accuracy on the test set is comparable to the training classification."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.4 ==========\n",
    "Print the coefficients for class 1 for the attributes `word_freq_hp_binarized` and `char_freq_$_binarized`. Generally, we would expect the string `$` to appear in spam, and the string `hp` to appear in non-spam e-mails, as the data was collected from HP Labs. Do the regression coefficients make sense given that class 1 is spam? *Hint: Consider the sigmoid function and how it transforms values into a probability between 0 and 1. Since our attributes are boolean, a positive coefficient can only increase the total sum fed through the sigmoid and thus move the output of the sigmoid towards 1. What can happen if we have continuous, real-valued attributes?*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "hp_ind = spambase.columns.get_loc('word_freq_hp_binarized')\n",
    "dollar_ind = spambase.columns.get_loc('char_freq_$_binarized')\n",
    "print('Coefficient for word_freq_hp_binarized: {:.3f}'.format(lr.coef_[0, hp_ind]))\n",
    "print('Coefficient for char_freq_$_binarized: {:.3f}'.format(lr.coef_[0, dollar_ind]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n",
    "The coefficients make sense, since the attribute `word_freq_hp_binarized` has a negative coefficient, meaning that it provides support for `non-spam` classification. On the other hand, `char_freq_$_binarized` has a positive coefficient, thus providing support for class 1 (i.e. `spam`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.5 ==========\n",
    "Train a [`LinearSVC`](http://scikit-learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html#sklearn.svm.LinearSVC) (i.e. Linear Support Vector classifier) by using default parameters. Report the classification accuracy on the training and test sets. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "svc_linear = LinearSVC()\n",
    "svc_linear.fit(X_train, y_train)\n",
    "print('Linear SVC classification accuracy on training set: {:.3f}'.format(svc_linear.score(X_train, y_train)))\n",
    "print('Linear SVC classification accuracy on test set: {:.3f}'.format(svc_linear.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.6 ==========\n",
    "What are the coefficients for the attributes `word_freq_hp_binarized` and `char_freq_`$`_binarized`? Compare these to the ones you found with Logistic Regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "print('Coefficient for word_freq_hp_binarized: {:.3f}'.format(svc_linear.coef_[0, hp_ind]))\n",
    "print('Coefficient for char_freq_$_binarized: {:.3f}'.format(svc_linear.coef_[0, dollar_ind]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n",
    "They are consistent. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.7 ==========\n",
    "How does a linear SVM relate to Logistic Regression? *Hint: Consider the classification boundary learnt in each model.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n",
    "They both have linear classification boundaries with respect to the inputs. The classification mechanism is, however, fundamentally different. Logistic regression applies a linear transformation to the input and the result is transformed to a probability by using a non-linear function. Support vector machine classification is based on the similiarty of a new data point to a few training instances known as support vectors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 1.8 ==========\n",
    "By using the [`SVC`](http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html#sklearn.svm.SVC) class train two new support vector classifiers with Gaussian (`rbf`) and polynomial (`poly`) kernels. Again, report classification accuracies on training and test sets and compare with your results from Question 1.5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "svc_rbf = SVC(kernel='rbf')\n",
    "svc_rbf.fit(X_train, y_train)\n",
    "svc_poly = SVC(kernel='poly', degree=2)\n",
    "svc_poly.fit(X_train, y_train)\n",
    "print('RBF SVC classification accuracy on training set: {:.3f}'.format(svc_rbf.score(X_train, y_train)))\n",
    "print('RBF SVC classification accuracy on test set: {:.3f}'.format(svc_rbf.score(X_test, y_test)))\n",
    "print('Poly SVC classification accuracy on training set: {:.3f}'.format(svc_poly.score(X_train, y_train)))\n",
    "print('Poly SVC classification accuracy on test set: {:.3f}'.format(svc_poly.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n",
    "It appears that there is a very slight benefit to using a radial basis kernel. However, before we can make any solid claims, we need to perform a cross-validation and try different parameters for these kernels. For an example, see [the sklearn doc examples](http://scikit-learn.org/stable/auto_examples/svm/plot_rbf_parameters.html#sphx-glr-auto-examples-svm-plot-rbf-parameters-py)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Performance assessment\n",
    "We will now look at a few ways of assessing the performance of a classifier. To do so we will introduce a new data set, the [Splice](https://archive.ics.uci.edu/ml/datasets/Molecular+Biology+%28Splice-junction+Gene+Sequences%29) data set. The classification task is to identify `intron` and `exon` boundaries on gene sequences. For more information, you can read the dataset description in the link. The class attribute can take on 3 values: `N`, `IE` and `EI`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 2.1 ==========\n",
    "Load the `splice_train.csv` and `splice_test.csv` into two separate dataframes. Display the shape and first 10 instances for each dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "data_path_train = os.path.join(os.getcwd(), 'datasets', 'splice_train.csv')\n",
    "splice_train = pd.read_csv(data_path_train, delimiter = ',')\n",
    "print('Number of instances: {}, number of attributes: {}'.format(splice_train.shape[0], splice_train.shape[1]))\n",
    "splice_train.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "data_path_test = os.path.join(os.getcwd(), 'datasets', 'splice_test.csv')\n",
    "splice_test = pd.read_csv(data_path_test, delimiter = ',')\n",
    "print('Number of instances: {}, number of attributes: {}'.format(splice_test.shape[0], splice_test.shape[1]))\n",
    "splice_test.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 2.2 ========== \n",
    "Convert the categorical attributes into numeric ones by using the [`LabelEncoder`](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html) tool. Make sure to not transform the target variable (`class`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "for column_train, column_test in zip(splice_train.drop(\"class\", axis=1).columns, splice_test.drop(\"class\", axis=1).columns):\n",
    "    le = LabelEncoder().fit(splice_train[column_train])\n",
    "    splice_train[column_train] = le.transform(splice_train[column_train])\n",
    "    splice_test[column_test] = le.transform(splice_test[column_test])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 2.3 ==========\n",
    "Store the training and testing data into numpy arrays `X_train`, `y_train`, `X_test` and `y_test`. Display the shapes of the four arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "X_train = splice_train.drop(\"class\", axis=1)\n",
    "X_test = splice_test.drop(\"class\", axis=1)\n",
    "y_train = splice_train[\"class\"]\n",
    "y_test = splice_test[\"class\"]\n",
    "print('X_train shape: {}'.format(X_train.shape))\n",
    "print('y_train shape: {}'.format(y_train.shape))\n",
    "print('X_train shape: {}'.format(X_test.shape))\n",
    "print('y_test shape: {}'.format(y_test.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 2.4 ==========\n",
    "Familiarise yourself with [Nearest Neighbours Classification](http://scikit-learn.org/stable/modules/neighbors.html#classification). Use a [`KNeighborsClassifier`](http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html#sklearn.neighbors.KNeighborsClassifier)\n",
    "by using a single neighbour. Report the classification accuracy on the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "knn = KNeighborsClassifier(n_neighbors=1)\n",
    "knn.fit(X_train, y_train)\n",
    "print('KNN classification (k=1) accuracy on training set: {:.3f}'.format( knn.score(X_train, y_train)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 2.5 ==========\n",
    "Is the above result meaningful? Why is testing on the training data a particularly bad idea for a 1-nearest neighbour classifier? Do you expect the performance of the classifier on a test set to be as good?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n",
    "The above result is not meaningful. The 1-nearest neighbour classifier will classify each instace to the class of the nearest insance in the training set. If we test the classifier on the training set, then each input will be classified correctly since the closest instance in the training set is the instance itself. The performance on unseen data is expected to be much lower."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 2.6 ==========\n",
    "Now report the classification accuracy on the test set and check your expectations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "print('KNN classification (k=1) accuracy on test set: {:.3f}'.format(knn.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 2.7 ==========\n",
    "Plot a histogram of the target variable (i.e. `class`) in the test set. *Hint: matplotlib won't allow you to plot a histogram for categorical values. Instead, you can use Pandas bulit-in bar plot tool in conjunction with the [`value_counts`](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.value_counts.html). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "class_counts = splice_test[\"class\"].value_counts()\n",
    "ax = class_counts.plot(kind=\"bar\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 2.8 ==========\n",
    "What would be the accuracy of the classifier, if all points were labelled as `N`? \n",
    "\n",
    "**Pro Tip** - You should always use a ['Dummy Model'](http://scikit-learn.org/stable/modules/model_evaluation.html#dummy-estimators) (a ridiculously simple model) like this to compare with your 'real' models. It's very common for complex models to be outperformed by a simple model, such as predicting the most common class. When complex models are outperformed by 'Dummies', you should investigate why: often there was an issue with the code, the data, or the way the model works was misunderstood."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "print('The baseline classifier (predict always N) would achieve a classification accuracy score of: {:.3f}'.\n",
    "      format(class_counts[\"N\"] / class_counts.values.sum()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 2.9 ==========\n",
    "Now we want to explore the effect of the `k` parameter. To do this, train the classifier multiple times, each time setting the KNN option to a different value. Try `5`, `10`, `50`, `100`, `200`, `500`, `1000`, `1500` and `2000` and test the classifier on the test set. How does the k parameter effect the results? *Hint: Consider how well the classifier is generalising to previously unseen data, and how it compares to the dumb prediction accuracy.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "k_list = [5, 10, 50, 100, 200, 500, 1000, 1500, 2000]\n",
    "ca = []\n",
    "for k in k_list:\n",
    "    knn = KNeighborsClassifier(n_neighbors=k).fit(X_train, y_train)\n",
    "    score = knn.score(X_test,y_test)\n",
    "    ca.append(score)\n",
    "    print('Performance on test with {} nearest neighbours: {:.3f}'.format(k, score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n",
    "The model generalisation appears to improve as k increases...but only up to a point. We assume this because accuracy increases and model complexity is decreasing (why?). However, as k gets too large, we end up predicting the mean class for every data point (why?!) and performance drops considerably."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 2.10 ==========\n",
    "Plot the results (k-value on the x-axis and classification accuracy on the y-axis), making sure to mark the axes. Can you conclude anything from observing the plot?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "plt.scatter(k_list, ca)\n",
    "plt.plot(k_list,ca, )\n",
    "plt.xlabel('k')\n",
    "plt.ylabel('Classification accuracy on test set')\n",
    "plt.title('K-nearest neighbours classification')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n",
    "Increasing K improves performance up to a certain point. Beyond that point, the classifier uses virtually all training samples to classify a new instance and as a results classifies all instances to the dominant class (identically to our dumb baseline model)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 2.11 ==========\n",
    "Select best value for `k` from Questions 2.9 and 2.10 and plot the normalised confusion matrix on the test set. Then plot the confusion matrix for a 5-nearest neighbour classifier. What do you observe?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "def plot_confusion_matrix(cm, classes=None, title='Confusion matrix'):\n",
    "    \"\"\"Plots a confusion matrix.\"\"\"\n",
    "    if classes is not None:\n",
    "        sns.heatmap(cm, xticklabels=classes, yticklabels=classes, vmin=0., vmax=1., annot=True)\n",
    "    else:\n",
    "        sns.heatmap(cm, vmin=0., vmax=1.)\n",
    "    plt.title(title)\n",
    "    plt.ylabel('True label')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "for i in [1000, 5]:\n",
    "    knn = KNeighborsClassifier(n_neighbors=i).fit(X_train, y_train)\n",
    "    pred = knn.predict(X_test)\n",
    "    cm = confusion_matrix(y_test, pred)\n",
    "    cm_norm = cm/cm.sum(axis=1)[:, np.newaxis]\n",
    "    plt.figure()\n",
    "    plot_confusion_matrix(cm_norm, classes=knn.classes_, title = str(i) + ' neighbours')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n",
    "With 1000 neighbours, the `N` class is almost always predicted correctly, but the other two classes are not. With fewer neightbors the `EI` class is predicted much better. The 5 neighbours solution is a more 'complex model' (why?) and the mean class accuracy seems to be better. The choice of model will depend on your goals. If you don't care about predicting `IE` correctly, the `1000` neighbour model gives greater overall accuracy and is a simpler model.  therefore I would err towards using it. However, your application may tolerate a drop in overall accuracy in return for better 'mean class accuracy' - in this case, you may consider the more complex 5 neighbour model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 2.12 ==========\n",
    "Read about the [logarithimic loss](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.log_loss.html) (or cross-entropy loss). It is often the error metric used when we are trying to optimise classification models.\n",
    "\n",
    "This metric takes as input the true labels and the estimated probability distributions (bernouli or multinomial). It makes sense to use this metric when we are interested not only in the predicted labels, but also in the confidence with which these labels are predicted.\n",
    "\n",
    "For instance, think of the situation where you have a single test point and two classifiers. Both classifiers predict the label correctly, however classifier A predicts tha the test point belongs to the class with probability 0.55, whereas classifier B predicts the correct class with probability 0.99. Classification accuracy would be the same for the two classifiers (why?) but the `log_loss` metric would indicate that classifier B should be favoured.\n",
    "\n",
    "Produce a scatter plot similar to the one in Question 2.10 but this time show `log_loss` on your y axis. Which value for `k` would you pick if `log_loss` was the error metric? Comment on whether accuracy or logarithmic loss is better for selecting a model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "\n",
    "k_list = [5, 10, 50, 100, 200, 500, 1000, 1500, 2000]\n",
    "logloss = []\n",
    "for k in k_list:\n",
    "    knn = KNeighborsClassifier(n_neighbors=k).fit(X_train, y_train)\n",
    "    pred_proba = knn.predict_proba(X_test)\n",
    "    this_logloss = log_loss(y_test, pred_proba)\n",
    "    logloss.append(this_logloss)\n",
    "    print('Performance on test with ', k, ' nearest neighbours: ', this_logloss)\n",
    "plt.scatter(k_list, logloss)\n",
    "plt.xlabel('k')\n",
    "plt.ylabel('Logarithmic loss on test set')\n",
    "plt.title('K-nearest neighbours classification')\n",
    "\n",
    "# Extension - plot baselines too\n",
    "baselines = ['prior', 'most_frequent']\n",
    "fig, ax = plt.subplots(len(baselines), 1, figsize=(5.5,6))\n",
    "from sklearn.dummy import DummyClassifier\n",
    "bl_loss = {}\n",
    "for ii, baseline in enumerate(baselines):\n",
    "    plt.sca(ax[ii])\n",
    "    dummy_classifier = DummyClassifier(strategy=baseline).fit(X_train, y_train)\n",
    "    pred_proba = dummy_classifier.predict_proba(X_test)\n",
    "    this_logloss = log_loss(y_test, pred_proba)\n",
    "    plt.scatter(k_list, logloss)\n",
    "    plt.axhline(this_logloss, label='{} baseline'.format(baseline), linestyle='--')\n",
    "    plt.xlabel('k')\n",
    "    plt.ylabel('Logarithmic loss on test set')\n",
    "    plt.legend()\n",
    "\n",
    "plt.suptitle('K-nearest neighbours classification')\n",
    "plt.tight_layout()\n",
    "plt.subplots_adjust(top=.9)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n",
    "We would pick the 50-neighbours classifier as it has the lowest generalisation log loss. \n",
    "\n",
    "One of the benefits of the log loss error metric is that it seems to be giving a smoother result: models with small k are more complex and are shown to be overfitting (the log loss is high). Similarly, models with large k are underfitting. Comparing this with the equivalent graph for accuracy, this is much less clear.\n",
    "\n",
    "The accuracy may change quite dramatically with a small change in predicted probability (observe the spikes in the acuracy graph). This is not the case for log loss and this makes it more suitable for gradient based optimisation.\n",
    "\n",
    "However, the point where the knn model = dumb baseline is note indicated as clearly on this log loss graph. There are a couple of different dummy classifier baselines we could use for log loss:\n",
    "\n",
    "1. Predict the dominant class with probability of 1 (sklearn.dummy.DummyClassifier(strategy='most_frequent').fit(X_train, y_train))\n",
    "2. Predict probabilites respecting the class distribution from the training dataset (i.e. use prior probabilities) (sklearn.dummy.DummyClassifier(strategy='prior').fit(X_train, y_train))\n",
    "\n",
    "I plotted these baselines on a second set of graphs with dotted lines. As you can see - predicting with 100% confidence comes at a very high cost (in terms of logarithmic loss)!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 2.13 ==========\n",
    "\n",
    "Could you use the `log_loss` metric to evaluate the performance of an SVM classifier? Explain."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n",
    "You can, but you probably shouldn't! SVMs are not probabilistic classifiers (i.e. they only yield a classification decision and not a posterior probability distribution) - they are trained to minimise a distance metric. Other classifiers, such as logistic regression, **explicitly try to minimise the logarithmic loss**. It would make sense that these models would report a better log loss than an SVM, but the SVM could still be producing 'better' predictions e.g. rankings.\n",
    "\n",
    "Nevertheless `sklearn` does provide a `predict_proba()` method which returns probability estimates on predictions by using cross-validation. You can also calibrate the predicted probabilities such that they perform better by the log_loss metric. See [here](http://scikit-learn.org/stable/modules/calibration.html) for more information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}