1a_ml_fundamentals.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Practical 1a: Machine Learning Fundamentals",
      "version": "0.3.2",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": [
        "x2P-s50pgj-N",
        "sFJBmN5Hivhv"
      ],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/deep-learning-indaba/indaba-pracs-2019/blob/master/1a_ml_fundamentals.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SB0EeXzyu_sz",
        "colab_type": "text"
      },
      "source": [
        "# Practical 1a: Machine Learning Fundamentals\n",
        "\n",
        "© Deep Learning Indaba. Apache License 2.0."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c9tLLZUyVwcV",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "## Introduction\n",
        "In this practical, we introduce the idea of classification (sorting things into categories) using a machine-learning model. We explore the relationship between a classifier's parameters and the decision boundary (a line that separates categories) and also introduce the idea of a loss function. Finally, we briefly introduce Tensorflow.\n",
        "\n",
        "## Learning Objectives \n",
        "* Understand the idea of **classification**\n",
        "* Understand the concept of (linear) **separability** of a dataset.\n",
        "* Understand what the **parameters** of a classifier are and how they relate to the **decision boundary**\n",
        "* Be able to briefly explain what **Tensorflow** is."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M6fLK2cibUOG",
        "colab_type": "text"
      },
      "source": [
        "**Imports**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mHlHxAdBu7Dy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip install -q moviepy > /dev/null 2>&1\n",
        "!pip install -q imageio > /dev/null 2>&1\n",
        "!pip install -q tensorflow==2.0.0-beta0 > /dev/null 2>&1\n",
        "\n",
        "import tensorflow as tf\n",
        "import numpy as np                 # Numpy is an efficient linear algebra library.\n",
        "import matplotlib.pyplot as plt    # Matplotlib is used to generate plots of data.\n",
        "from matplotlib import animation, rc\n",
        "\n",
        "from IPython import display\n",
        "\n",
        "try:\n",
        "  tf.executing_eagerly()\n",
        "  print(\"TensorFlow executing eagerly: {}\".format(tf.executing_eagerly()))\n",
        "except ValueError:\n",
        "  print('Already running eagerly')\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iGmJsXYFl8OF",
        "colab_type": "text"
      },
      "source": [
        "Make sure to run the imports cell above, otherwise the rest of the cells will fail when you try to run them. (To run a cell press `shift` + `enter` with your mouse cursor in the cell or press the play button in the top right of the cell.)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1xhQkS8A_KrJ",
        "colab_type": "text"
      },
      "source": [
        "## Outline\n",
        "In this practical, we tackle the task of **classification** of a simple, synthetic dataset. Classification in machine learning involves learning a labelling of examples into one (or more) discrete categories. This differs from another common task in machine learning called **regression**, which involves learning a mapping from inputs to a continuous-valued output. \n",
        "\n",
        "1. We begin by introducing a synthetic dataset of red and blue points which we want to separate\n",
        "2. We introduce and explore the idea of **linear separability**\n",
        "3. We define a **loss** as a measure of how good of a seperator a particular line is\n",
        "4. We briefly introduce TensorFlow and show how it can be used to automatically find the minimum of a loss function."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AQhAK-B7bYU4",
        "colab_type": "text"
      },
      "source": [
        "**Helper Functions (RUN ME)**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H020s1EsB_9p",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# this is a helper function to assist with plotting the dataset below\n",
        "def plot_dataset(inputs, labels):\n",
        "  # Plot the given 2D inputs and labels using Matplotlib.   \n",
        "  plt.scatter(\n",
        "      inputs[:, 0], inputs[:, 1], \n",
        "      c=['red' if label > 0 else 'blue' for label in labels])\n",
        "\n",
        "  plt.axis('equal')\n",
        "\n",
        "  plt.xlabel('x1')\n",
        "  plt.ylabel('x2')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RyJ1PgtEFpa7",
        "colab_type": "text"
      },
      "source": [
        "### Examples of Binary Classification Problems"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7LAhNs_GFwOZ",
        "colab_type": "text"
      },
      "source": [
        "In this practical, we are using a synthetic dataset where we have two classes of 2-D points that come from different distributions, distinguished by the colours, red and blue. To make this more concrete, here are some examples of more real-world binary classification problems.\n",
        "\n",
        "* Determine whether an email message (input) is SPAM or NOT SPAM (label)\n",
        "* Determine whether an image, represented by its encoded pixel values (input) is a picture of a DOG or a CAT (label)\n",
        "* Determine whether energy usage of a building will go UP or DOWN (label) next month, using a time series of past energy usage values (input)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3l1rLP3HufZv",
        "colab_type": "text"
      },
      "source": [
        "## The Data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FCu4YZy-uj0v",
        "colab_type": "text"
      },
      "source": [
        "Run the code in the cell below, and look at the resulting plot. It should produce a simple 2-D data set consisting of 2 classes of points, the classes are represented by colours blue and red. Our task is to build a **binary classifier** that can distinguish between red and blue points (red and blue are referred to as the **classes** of the points), using only the 2-D coordinates of a point. In other words, we want a function that takes as input a 2-D vector representing the coordinates of a point and returns a value of 1 or 0 indicating whether the point is red or blue. Here we have **encoded** the colours red and blue into the numbers 1 and 0 (which make it easier to work with in maths and code!)\n",
        "\n",
        "**Note:** we have arbitrarily encoded red as 1 and blue as 0, you could do it the other way around too as long as you're consistent!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2SrsrFSTtrl6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#@title Generate the Dataset  {run: \"auto\"}\n",
        "# Define the centre(s) of the points\n",
        "centre = 1.8    #@param {type:\"slider\", min:0, max:2, step:0.1}\n",
        "\n",
        "points_in_class = 20  # How many points we want per class\n",
        "\n",
        "# A fixed random seed is a common \"trick\" used in ML that allows us to recreate\n",
        "# the same data when there is a random element involved. \n",
        "np.random.seed(0)  \n",
        "\n",
        "# Generate random points in the \"red\" class\n",
        "red_inputs = np.random.normal(loc=centre, scale=1.0, size=[points_in_class, 2])     \n",
        "# Generate random points in the \"blue\" class\n",
        "blue_inputs = np.random.normal(loc=-centre, scale=1.0, size=[points_in_class, 2])    \n",
        "# Put these together\n",
        "inputs = np.concatenate((red_inputs, blue_inputs), axis=0) \n",
        "    \n",
        "# The class (label) is 1 for red or 0 for blue\n",
        "red_labels = np.ones(points_in_class)    \n",
        "blue_labels = np.zeros(points_in_class)\n",
        "labels = np.concatenate((red_labels, blue_labels), axis=0)\n",
        "\n",
        "# num_data_points is the total data set size\n",
        "num_data_points = 2 * points_in_class\n",
        "\n",
        "plot_dataset(inputs, labels)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "an9GF8nZWS4K",
        "colab_type": "text"
      },
      "source": [
        "###What does the data look like? \n",
        "The inputs are 2-dimensional vectors (points in a 2-D space). Here are the coordinates of 4 points, which we've deliberately chosen so that points 1 and 2 are \"red\" and points 3 and 4 are \"blue\". "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5f8M-vQYWUuI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print('Input 1:\\t', inputs[0])\n",
        "print('Input 2:\\t', inputs[1])\n",
        "\n",
        "print('Input 3:\\t', inputs[-1])\n",
        "print('Input 4:\\t', inputs[-2])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E14elwRYWoxB",
        "colab_type": "text"
      },
      "source": [
        "The labels are either 0 (blue) or 1(red). Here are the labels corresponding to the points above:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q4EJG8g4Wtih",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print('Label 1:\\t', labels[0])\n",
        "print('Label 2:\\t', labels[1])\n",
        "\n",
        "print('Label 3:\\t', labels[-1])\n",
        "print('Label 4:\\t', labels[-2])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qNuMy1XwIJ-z",
        "colab_type": "text"
      },
      "source": [
        "## Linear separability\n",
        "\n",
        "For a 2-dimensional dataset with 2 classes, we can say that the dataset is **linearly separable** if it is possible to draw a (1-D) line separating the examples of each class. In this case, we would want a line between the red and blue points such that all of the red points lie on one side of the line and all of the blue points on the other. Linear separability of a D-dimensional dataset with 2 classes means that there exists a single (D-1)-dimensional (hyper-)plane that separates the classes (a hyperplane is a generalisation of a straight line to many dimensions). \n",
        "\n",
        "### Exploratory Task\n",
        "In the code cell under the heading \"The Data\", change the slider for the ```centre``` value. This will automatically update the value in the code and will redraw the plot.\n",
        "\n",
        "* At what value of centre does the dataset become linearly separable?\n",
        "\n",
        "\n",
        "### Question for discussion\n",
        "Can you think of some 2-D, 2-class datasets, similar to the one above, that are separable (the points from the 2 classes don't overlap each other), but are not **linearly** separable? Draw some examples on paper or plot them using Matplotlib and discuss this with your neighbour and tutors. \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GV5plkAowLy8",
        "colab_type": "text"
      },
      "source": [
        "## Drawing the line\n",
        "\n",
        "As you may recall from school, a line in 2 dimensions, with coordinate axes $x_1$ and $x_2$, which passes through the origin (0, 0) can be represented by the equation $w_1x_1 + w_2x_2 = 0$. \n",
        "\n",
        "We can also write this in vector form as: $\\mathbf{w}^T\\mathbf{x} = 0$, where $\\mathbf{w}^T = [w_1, w_2]$ and $\\mathbf{x}^T = [x_1, x_2]$.\n",
        "\n",
        "**Note:** Our line above will go through the origin $(0,0$). If we want to describe a line that doesn't pass through the origin we need to add a bias term.\n",
        "\n",
        "When a line (or more generally, a hyperplane) is defined this way, we call the **parameters** $\\mathbf{w} = (w_1, w_2)$ a **normal vector** for the line. The normal vector is orthogonal (perpendicular) to the line. We want to construct such a line that separates red and blue  points, which we will call a **decision boundary**. The name \"decision boundary\" comes from the fact that the line describes the boundary in the input space where the model's output (decision) changes from one class to another.\n",
        "\n",
        "In the following cell, we plot our dataset along with a normal vector $\\mathbf{w}$ and decision boundary. You can adjust the values of $w_1$ and $w_2$ by using the sliders on the right. Observe the effect that the values have on the normal vector drawn in *red* and decision boundary in *black*. Adjust the values so that the black line separates the blue and red points (i.e. red points on one side and blue on the other). Your line should also have the normal vector pointing in the direction of the red points. The reason that direction is significant is that we want to eventually **classify** points on one side of the line (decision boundary) as being red and the other as being blue. \n",
        "\n",
        "Is it possible to find a line through the origin that perfectly separates the points?\n",
        "\n",
        "**Note**: Each of our inputs is a 2-D vector, made up of two coordinate values. We refer to these 2 coordinate axes as $x_1$ and $x_2$. For example, if we have an input $(1, 2)$, then we would say $x_1 = 1$ and $x_2 = 2$ for that point."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TAXUNshcvPsg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#@title Effect of parameters {run: \"auto\"}\n",
        "\n",
        "# Define the parameters\n",
        "w1 = -1 #@param { type: \"slider\", min: -5, max: 5, step: 0.1 }\n",
        "w2 = 1 #@param { type: \"slider\", min: -5, max: 5, step: 0.1 }\n",
        "\n",
        "plot_dataset(inputs, labels)\n",
        "\n",
        "# Add the weight vector to the plot. We plot it in red, as it has to \"point\"\n",
        "# in the direction of the red points.\n",
        "ax = plt.axes()\n",
        "ax.arrow(0, 0, w1, w2, head_width=0.3, head_length=0.3, fc='r', ec='r')\n",
        "\n",
        "# Plot part of the decision boundary in black. It is orthogonal to the weight\n",
        "# vector.\n",
        "t = 2 # this is how long the line should be\n",
        "plt.plot([-t * w2, t * w2], [t * w1, -t * w1], 'k-')\n",
        "\n",
        "plt.xlim([-4, 4])\n",
        "plt.ylim([-4, 4])\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "igAjsyMldbMr",
        "colab_type": "text"
      },
      "source": [
        "## Classification\n",
        "Given a normal vector $\\mathbf{w}$, we can evaluate which side of the decision boundary a particular point $\\mathbf{x_i} = (x_{i,1}, x_{i, 2})$ lies by evaluating $\\mathbf{w^Tx_i}$. If $\\mathbf{w^Tx_i} > 0$, the point $\\mathbf{x_i}$ lies to one side of the boundary (in the direction of the normal vector), and we can classify that point as belonging to class 1 (in our case, \"red\"). If $\\mathbf{w^Tx_i} < 0$ the point lies on the other side and can be classified as class 0 (in our case, \"blue\"). Finally if $\\mathbf{w^Tx_i} = 0$ the point lies on the decision boundary and we can decide whether to classify it as either 0 or 1, or ignore it. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jqW7RpSTaRZH",
        "colab_type": "text"
      },
      "source": [
        "## How \"good\" is the line?\n",
        "\n",
        "If you've played around with the above code, you may have developed some intuition around how different settings of the parameters influence the final placement of the decision boundary. The purpose of machine learning is to *automatically* adjust the values of $w_1$ and $w_2$ to find a suitable decision boundary! But to do this, we need to mathematically specify some **loss** or **objective** function. The loss is a function of the parameters $w_1$ and $w_2$ and tells us how good a certain configuration of the parameter values are at classifying the data. This function is defined such that it reaches its optimum setting when it is minimised, i.e. the *smaller* its value, the *better* the separation between the classes. An additional property a loss function can have that is often crucial for machine learning is being *differentiable*. A differentiable loss function allows us to use *gradient-based optimisation* to find its minimum and the corresponding optimal values of $w_1$ and $w_2$. \n",
        "\n",
        "For this classification problem, we consider the **binary cross-entropy** loss function to measure how good the model's predictions are.  This loss function compares the model's prediction for each example, $\\mathbf{x_i}$ to the true **target** $y_i$ (we often refer to the true label associated with an input as the \"target\"). It then applies the non-linear log function to penalise the model for being further from the true class.\n",
        "\n",
        "The binary cross entropy function uses an operation called a  $\\operatorname{sigmoid}$ function. This functions allows our classifier to output any real value. The binary cross entropy loss function, however, expects the predictions made by a classifier to be between $0$ and $1$. The sigmoid function \"squashes\" any real number inputs to lie in the interval $(0, 1)$.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BZjjZYLLk4VG",
        "colab_type": "text"
      },
      "source": [
        "For those of you who are more mathematically inclined, the equation for the binary cross entropy loss, on a dataset with $N$ points is defined as follows:\n",
        "\n",
        "\\begin{align}\n",
        "l(\\mathbf{w}; \\mathbf{\\hat{y}}, \\mathbf{y}) = -\\frac{1}{N}\\sum_{i=1}^N y_i log(\\hat{y}_i) + (1-y_i)log(1-\\hat{y}_i)\n",
        "\\end{align}\n",
        "\n",
        "where $\\hat{y}_i = \\operatorname{sigmoid}(\\mathbf{w}^T\\mathbf{x_i})$ and the $\\operatorname{sigmoid}$ function is defined as:\n",
        "\n",
        "$$\n",
        "\\mathrm{sigmoid}(a) = \\frac{1}{1 + e^{-a}} .\n",
        "$$\n",
        "\n",
        "If you don't understand all of the math, don't stress too much, but if you want like to learn more about the binary cross-entropy loss, check out this [blog](https://towardsdatascience.com/understanding-binary-cross-entropy-log-loss-a-visual-explanation-a3ac6025181a).\n",
        "\n",
        "Let's now wrap this in a Python function so that we can compute the loss for any values of $w_1$ and $w_2$:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wKkpBZ6ZWLoF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def compute_loss(w1, w2, inputs, labels):\n",
        "  \n",
        "  total_log_likelihood = 0\n",
        "  N = len(inputs)\n",
        "  \n",
        "  # Add the contribution of each datapoint to the loss\n",
        "  for (x1, x2), target in zip(inputs, labels):\n",
        "    # As our targets are 0 or 1, our prediction function must output a value between 0 and 1.\n",
        "    # The sigmoid function 'squashes' any value to lie between 0 and 1:\n",
        "    prediction = tf.sigmoid(w1*x1 + w2*x2)  \n",
        "    \n",
        "    # Compute the local loss term\n",
        "    # We add 1e-10 to make the log operations numerically stable (i.e. avoid taking the log of 0.)\n",
        "    log_likelihood = target * tf.math.log(prediction + 1e-10) + (1.-target)*tf.math.log(1.-prediction + 1e-10)\n",
        "    total_log_likelihood += log_likelihood\n",
        "  \n",
        "  loss = -total_log_likelihood\n",
        "  average_loss = loss / N\n",
        "  return average_loss"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x2P-s50pgj-N",
        "colab_type": "text"
      },
      "source": [
        "### Optional section: More on the sigmoid function"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "44ra3nJ-rLJe",
        "colab_type": "text"
      },
      "source": [
        "**Question:** Discuss with your neighbour what you think is important when designing a good loss function i.e. what should the loss function do when the model produces a good/bad prediction?\n",
        "\n",
        "If you are comfortable with the mathematics, show how the **sigmoid** function (and the binary cross-entropy function) is good for evaluating the quality of predictions by using the following questions:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "djpjQZ64RZW7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "ax = plt.axes()\n",
        "xs = np.arange(-10,10)\n",
        "ax.plot(xs, 1 / (1 + np.exp(-xs)))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AQqQ_quqwCFF",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "\n",
        "As noted earlier, the sigmoid function is defined as\n",
        "$$\n",
        "\\mathrm{sigmoid}(a) = \\frac{1}{1 + e^{-a}} .\n",
        "$$\n",
        "Can you show that\n",
        "$$\n",
        "1 - \\mathrm{sigmoid}(a) = \\frac{1}{1 + e^{a}} ,\n",
        "$$\n",
        "and draw both of these on a sheet of paper?\n",
        "\n",
        "* What is its value when $a = \\mathbf{w}^{T}\\mathbf{x}$ is positive? negative? and zero?\n",
        "* What happends to its value when  $a = \\mathbf{w}^{T}\\mathbf{x}$ becomes larger?\n",
        "* What is the value of $\\mathrm{sigmoid}(\\mathbf{w^Tx})$ when $\\mathbf{w}^T\\mathbf{x} = 0$? How does this change how we classify points on either side of the decision boundary?\n",
        "\n",
        "**HINT**: Remember the idea of the loss function is to return small values when the classifier makes good predictions and large values when the classifier makes bad predictions. \n",
        "\n",
        "*If you are not comfortable with the maths*, remember the main aim of this question to **highlight the importance of the binary cross-entropy loss** and **NOT** the math so focus on the concepts!\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x46fjqTUf4Dj",
        "colab_type": "text"
      },
      "source": [
        "### Bonus Question\n",
        "We derived the `compute_loss()` function above based on minimising the log-loss of the prediction error. This is related to a concept called 'cross-entropy'. But another way of deriving exactly the same loss function is by maximising the likelihood of the data under the model $P(y | x, w_1, w_2)$. If you are familiar with this concept (eg. from statistics), see if you can derive it this way as well.\n",
        "\n",
        "### Optional Further Reading\n",
        "More information on the [cross-entropy loss](http://ml-cheatsheet.readthedocs.io/en/latest/loss_functions.html) and another interesting connection to [information theory](https://rdipietro.github.io/friendly-intro-to-cross-entropy-loss/)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0tJJrBynf6ms",
        "colab_type": "text"
      },
      "source": [
        "## Loss value for your chosen $w_1$ and $w_2$\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "edKlqlACgFsE",
        "colab_type": "text"
      },
      "source": [
        "The following line of code computes the loss value for your chosen values of $w_1$ and $w_2$. Try changing the values of $w_1$ and $w_2$ using the sliders. Can you see how a better separation results in a lower loss? \n",
        "\n",
        "**Note:** If you've used an older version of TensorFlow before, it might be confusing how this code cell works! We explain more about this later... "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QyzwKx6ef_Vm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#@title Effect of parameters on the loss {run: \"auto\"}\n",
        "\n",
        "# Define the parameters\n",
        "w1 = -1 #@param { type: \"slider\", min: -5, max: 5, step: 0.1 }\n",
        "w2 = 1 #@param { type: \"slider\", min: -5, max: 5, step: 0.1 }\n",
        "\n",
        "plot_dataset(inputs, labels)\n",
        "\n",
        "# Add the weight vector to the plot. We plot it in red, as it has to \"point\"\n",
        "# in the direction of the red points.\n",
        "ax = plt.axes()\n",
        "ax.arrow(0, 0, w1, w2, head_width=0.3, head_length=0.3, fc='r', ec='r')\n",
        "\n",
        "# Plot part of the decision boundary in black. It is orthogonal to the weight\n",
        "# vector.\n",
        "t = 2 # this is how long the line should be\n",
        "plt.plot([-t * w2, t * w2], [t * w1, -t * w1], 'k-')\n",
        "\n",
        "plt.xlim([-4, 4])\n",
        "plt.ylim([-4, 4])\n",
        "\n",
        "plt.show()\n",
        "\n",
        "compute_loss(w1, w2, inputs, labels).numpy()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z9KAMYSUgmkM",
        "colab_type": "text"
      },
      "source": [
        "## Visualising the loss function"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ukphZS4_hMgN",
        "colab_type": "text"
      },
      "source": [
        "We can visualise the loss function for our dataset by plotting its value at every point in a whole grid of $w_1$ and $w_2$ parameter values. We do this using a **contour plot**, which is a technique for visualising a 3-D function on a 2-D plot by letting colour represent the third dimension. All of the points with the same colour have the same loss value. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y4HZS5zZt3Pu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# We define a function so we can re-use this code later\n",
        "def plot_contours():  \n",
        "  # Generate a whole bunch of (w1, w2) points in a grid\n",
        "  ind = np.linspace(-5, 5, 50)\n",
        "  w1grid, w2grid = np.meshgrid(ind, ind)\n",
        "\n",
        "  # Compute the loss for each point in the grid\n",
        "  losses = []\n",
        "  for w1s, w2s in zip(w1grid, w2grid):\n",
        "    loss = compute_loss(w1s, w2s, inputs, labels)\n",
        "    losses.append(loss)\n",
        "\n",
        "  # Pack the loss values for every value of w1 & w2 into one (50,50) array\n",
        "  losses_array = np.concatenate(losses).reshape(50,50)\n",
        "\n",
        "  # Now plot the resulting loss function as a contour plot over the whole grid of (w1, w2) values.\n",
        "  fig = plt.figure()\n",
        "  plt.contourf(w1grid, w2grid, losses_array, 20, cmap=plt.cm.jet)\n",
        "  cbar = plt.colorbar()\n",
        "  cbar.ax.set_ylabel('Binary cross-entropy loss value')\n",
        "  plt.xlabel('w1 value')\n",
        "  plt.ylabel('w2 value')\n",
        "  plt.title('Total loss for different values of w1 and w2')\n",
        "\n",
        "plot_contours()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IRRe9YecuRLB",
        "colab_type": "text"
      },
      "source": [
        "As you can see by the bar on the right, as the colour goes from red to blue the loss value gets lower and lower until it reaches the smallest value (dark blue). Since we want the loss to be as small as possible, we want our values of $w_1$ and $w_2$ to produce a loss in the dark blue area of the contour plot. This is achieved within a certain range of values for $w_1$ and $w_2$. \n",
        "\n",
        "**Question:** Can you read which values of $w_1$ and $w_2$ will give you the smallest loss? Are these values similar to the ones you found earlier to linearly separate the data? \n",
        "\n",
        "**Question:** Depending on the center point you chose, the smallest value for the loss may not be 0. Under what circumstances is this the case?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "676bWBwTiXXD",
        "colab_type": "text"
      },
      "source": [
        "## Optimising the loss using TensorFlow"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MMLA-wk5xB7p",
        "colab_type": "text"
      },
      "source": [
        "Now that we have a function that gives us the loss for different values of $w_1$ and $w_2$, we want an automated method to find the values that minimise the loss function. This is where optimisation by **gradient descent** comes in. The idea is that for each (batch of) data points, we compute the loss using the current values of $w_1$ and $w_2$ on the data. We then compute the **gradient** (or derivative) of the loss function at the current values of $w_1$ and $w_2$. The negative of the gradient points in the direction of *steepest descent* along the loss function. By adjusting the values of $w_1$ and $w_2$ in the direction of the negative gradient, we move closer towards the minimum of the loss function (provided the loss function is \"well behaved\"). How big of a step we take is mediated by the **learning rate**. To do this more easily, we will use TensorFlow.\n",
        "\n",
        "Don't worry if you don't understand what's going on here, you will see this in a lot more detail during the **Mathematics for Machine Learning** lectures!\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sFJBmN5Hivhv",
        "colab_type": "text"
      },
      "source": [
        "### Optional extra reading: TensorFlow"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DRtDggkBi0X3",
        "colab_type": "text"
      },
      "source": [
        "TensorFlow (TF) is an open source software library for numerical computation using the concept of Tensors. You can think of Tensors as being a generalisation of matrices to higher dimensions, or roughly equivalent to multi-dimensional arrays. Scalars are 0-dimensional tensors, vectors are 1-dimensional, standard matrices are 2-dimensional, and higher-dimensional tensors have 3 or more dimensions. You can think of dimensions as representing groups of numbers that mean the same thing. For example, for images, we often use 3-dimensional tensors where the first dimension represents the red, green, and blue color channels of the image, and the next two are the columns and rows of pixels of the image. \n",
        "\n",
        "**Note**: Don't be confused when people say \"2-D vector\" or \"3-D vector\", which refers to a 1-dimensional tensor that has size 2 or 3.\n",
        "\n",
        "The major advantage of using TensorFlow is that it can automatically derive the gradients of many mathematical expressions involving tensors. It achieves this through a process called \"automatic differentiation\". Tensorflow also supports multiple \"kernels\", allowing you to easily run your code on normal processors (CPUs), graphics cards (GPUs) and other more exotic hardware accelerators like Google's Tensor Processing Units (TPUs)\n",
        "\n",
        "Tensorflow actually provides **two modes of operation**, the first, called \"graph mode\", builds a computation graph upfront and then feeds data into the graph. By building the graph upfront, Tensorflow can apply optimisations to the graph that allow it to extract peak performance from the hardware you're running on. You will have encountered this mode if you used Tensorflow before or attended the Indaba last year! The second mode, called [\"Eager-mode\"](https://www.tensorflow.org/guide/eager), is a lot newer and evaluates Tensor operations imperatively (in the order you write them), similar to NumPy and PyTorch. Eager-mode is slightly less performant but a lot more intuitive, especially if you've never used a \"define-and-run\" programming style (like graph mode) before, and is therefore the mode we will use in these practicals. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eLVl2PhK2VpP",
        "colab_type": "text"
      },
      "source": [
        "### Using Tensorflow to optimise the loss\n",
        "We use TensorFlow to optimise the parameters of the model with gradient descent. We loop over the dataset multiple times (called \"epochs\") and plot the final decision boundary along with a plot showing how the parameters and loss changed over the epochs.\n",
        "\n",
        "**Note**: TensorFlow is probably overkill for this example, because the gradient is very easy to calculate, but we introduce it here because it will become essential to calculate the gradients of more complex models in later practicals! "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cKre-CI8IG9z",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "lr = 0.33  # The learning rate\n",
        "\n",
        "# Initialise Tensorflow variables representing our parameters.\n",
        "# We need to use TensorFlow variables here rather than Numpy or Python ones so \n",
        "# that TensorFlow is able to compute gradients.\n",
        "w1 = tf.Variable(-2.0, dtype=tf.float64)  \n",
        "w2 = tf.Variable(-4.0,  dtype=tf.float64)  \n",
        "print(w1.dtype)\n",
        "\n",
        "plot_contours()\n",
        "\n",
        "# Loop over the dataset multiple times\n",
        "parameter_values = []\n",
        "for epoch in range(100):\n",
        "  plt.scatter(w1.numpy(), w2.numpy(), marker='o', color='black')\n",
        "  \n",
        "  # The GradientTape is how TF keeps track of gradients in Eager-mode\n",
        "  with tf.GradientTape() as tape:\n",
        "    loss = compute_loss(w1, w2, inputs, labels)\n",
        "  \n",
        "  # Now we take a step in parameter space in the direction of the gradient to move the parameters closer (hopefully!) to their optimum\n",
        "  dw1, dw2 = tape.gradient(loss, [w1, w2])\n",
        "  \n",
        "  # Step 'lr units' in the direction of the negative gradient\n",
        "  # We achieve this by subtracting lr * dw1 and lr * dw2 from the w1 and w2 variables\n",
        "  w1.assign_sub(lr*dw1)\n",
        "  w2.assign_sub(lr*dw2)\n",
        "  \n",
        "print('Finished optimisation, the final values of w1 and w2 are:')\n",
        "print(w1.numpy(), w2.numpy(), loss)\n",
        "\n",
        "# Plot the final point on the loss surface.\n",
        "plt.scatter(w1.numpy(), w2.numpy(), marker='x', color='red')\n",
        "plt.show()\n",
        "\n",
        "# Plot the final decision boundary\n",
        "plot_dataset(inputs, labels)\n",
        "ax = plt.axes()\n",
        "ax.arrow(0, 0, w1.numpy(), w2.numpy(), head_width=0.3, head_length=0.3, fc='r', ec='r')\n",
        "plt.plot([-2 * w2.numpy(), 2 * w2.numpy()], [2 * w1.numpy(), -2 * w1.numpy()], 'k-')\n",
        "\n",
        "plt.xlim([-4, 4])\n",
        "plt.ylim([-4, 4])\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "meoIDXSQjKeE",
        "colab_type": "text"
      },
      "source": [
        "Can you see how the model steps across the contour plot until it minimises the loss? \n",
        "\n",
        "How did the final values of $w_1$ and $w_2$ found by Tensorflow correspond to the ones you found manually? If they aren't, can you explain why?\n",
        "\n",
        "## Optional Tasks\n",
        "If you've worked through this practical, answered all the questions and feel you have a good understanding of what's going on, try the following tasks:\n",
        "\n",
        "1. Add a **bias** parameter to the equation for the decision boundary and visualise how that changes the decision boundary, the loss and the ultimate solution found by Tensorflow.\n",
        "2. Add a **regulariser**, for example, the L2 regulariser (see the appendix below for more information) - how does it affect the contour plot of the parameters vs the loss? How does changing the strength of regularisation affect the loss? \n",
        "\n",
        "Note: The benefit of using regularisation will be discussed in the next practical! "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SVFq4xvBmGQq",
        "colab_type": "text"
      },
      "source": [
        "# Next Steps\n",
        "Have a look at [2017's practical,](https://github.com/deep-learning-indaba/practicals2017/blob/master/practical1.ipynb) which takes a more \"bottom-up\" approach, covers more detail on how gradients are computed and also looks at a multi-class classification problem with a non-linear decision boundary. \n",
        "\n",
        "Note: 2017's practicals use Tensorflow's \"graph mode\" as opposed to \"Eager mode\" that we use here."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4jpKVuEkQF46",
        "colab_type": "text"
      },
      "source": [
        "# Appendix"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EgLuuVNRQHqy",
        "colab_type": "text"
      },
      "source": [
        "### L1 and L2 Regularization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SlO75RlmQKQF",
        "colab_type": "text"
      },
      "source": [
        "Two of the most simple regularization methods are L1 and L2 regularization (you may have heard of them as _Lasso_ and _Ridge_ regression if you've used linear regression before). Both of these methods regularize the model by adding a term to the loss that penalizes the model if it becomes too complex.\n",
        "L1 regularization adds a term based on the L1 norm:\n",
        "\n",
        "$loss_{L1} = loss + \\lambda \\sum_i |w_i|$\n",
        "\n",
        "where $\\lambda$ is a parameter that controls the amount of regularization, and $w_i$ are the parameters of the model. L1 regularization has the effect of forcing some parameters to shrink to 0, effectively removing them from the model.\n",
        "\n",
        "L2 regularization similarly adds a term based on the L2 norm:\n",
        "\n",
        "$loss_{L2} = loss + \\lambda \\sum_i w_i^2$.\n",
        "\n",
        "L2 regularization has the effect of preventing any of the parameters from becoming too large and _overpowering_ the others. \n",
        "\n",
        "In some cases it can work well to use both L1 and L2 regularization. \n",
        "\n",
        "For more information see the articles [here](http://enhancedatascience.com/2017/07/04/machine-learning-explained-regularization/) and [here](https://towardsdatascience.com/l1-and-l2-regularization-methods-ce25e7fc831c)."
      ]
    }
  ]
}