10_Lab_5_Neural_Networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [Introductory applied machine learning (INFR10069)](https://www.learn.ed.ac.uk/webapps/blackboard/execute/content/blankPage?cmd=view&content_id=_2651677_1&course_id=_53633_1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 5: Neural Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*by [James Owers](https://jamesowers.github.io/), University of Edinburgh 2017*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. [Introduction](#Introduction)\n",
    "    * [Lab Outline](#Lab-Outline)\n",
    "    * [The Data](#The-Data)\n",
    "1. [Part 1 - Introducing the Neural Network Model](#Part-1---Introducing-the-Neural-Network-Model)\n",
    "    * [Resources to Watch and Read pt. 1](##Resources-to-Watch-and-Read-pt.-1)\n",
    "    * [Model Design](#Model-Design)\n",
    "    * [The Cost Space](#The-Cost-Space)\n",
    "1. [Part 2 - Fitting the Model & Optimisation](#Part-2---Fitting-the-Model-&-Optimisation)\n",
    "    * [Resources to Watch and Read pt. 2](#Resources-to-Watch-and-Read-pt.-2)\n",
    "    * [Finding the Best Parameters](#Finding-the-Best-Parameters)\n",
    "    * [Gradient Descent](#Gradient-Descent)\n",
    "    * [Backpropagation](#Backpropagation)\n",
    "1. [Part 3 - Implementation From Scratch](#Part-3---Implementation-From-Scratch!)\n",
    "1. [Part 4 - Implementation With Sklearn](#Part-4---Implementation-with-Sklearn)\n",
    "1. [Moar?!](#Please-sir...I-want-some-more)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# https://docs.python.org/2/library/__future__.html\n",
    "# make printing and division act like python 3\n",
    "from __future__ import division, print_function\n",
    "\n",
    "# General\n",
    "import sys\n",
    "import os\n",
    "import copy\n",
    "from IPython.display import Image, HTML\n",
    "\n",
    "# Data structures\n",
    "import numpy as np \n",
    "import pandas as pd\n",
    "\n",
    "# Modelling\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "from scipy.optimize import check_grad\n",
    "\n",
    "# Plotting\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "# Local module adjacent to this notebook\n",
    "import iaml\n",
    "from iaml.data import load_letters\n",
    "\n",
    "# http://ipython.readthedocs.io/en/stable/interactive/magics.html\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This lab:\n",
    "1. introduces a simple neural network model in a supervised learning setting\n",
    "1. provides impetus to understand the fitting procedure of that, and other networks\n",
    "1. encourages you to implement a model from scratch\n",
    "1. models the same problem with the sklearn package\n",
    "1. makes you think about what you've done!\n",
    "\n",
    "It does not discuss in detail:\n",
    "1. any of the plethora of different activation functions you can use e.g. RELUs, SELUs, Tanh, ...\n",
    "1. how to initialise the parameters and why that matters\n",
    "1. issues with the fitting process e.g. local optima, and how to avoid them e.g. learning rate schedulers, momentum, RMSProp, Adam, cyclic learning rates\n",
    "1. issues with model complexity e.g. overfitting, and solutions such as dropout, regularisation, or using [shedloads of data](https://what-if.xkcd.com/63/)\n",
    "1. other tricks for speeding up and stablising fitting such as batch sizes, weight norm, layer norm\n",
    "1. deep networks and their tricks like skip connections, pooling, convolutions\n",
    "1. nor other more complex architectures like CNNs, RNNs, LSTMs, GANs, etc. etc.\n",
    "1. many, many, MANY other things (that probably were published, like, [yesterday](https://arxiv.org/abs/1711.04340v1))\n",
    "\n",
    "However, if you understand what is in this notebook well, **you will have the ability to understand [all of these things](https://i.imgflip.com/1zn8p9.jpg)**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lab outline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I provide you with a function that creates data then link you to some excellent resources to learn the basics. These resources are superb, short, and free. I highly, highly recommend setting aside a couple of hours to give them a good watch/read and, at the very least, use them for reference. \n",
    "\n",
    "After you have had a crack at the problems, I'll release the solutions. The solutions, particularly to part 3, walk you through the process of coding a simple neural neural network in detail.\n",
    "\n",
    "Parts 3 & 4 are practical, parts 1 & 2 are links to external resources to read. Whilst I recommend you soak up some context first with 1 & 2, feel free to jump in at the deep end and get your hands dirty with part 3 or 4."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Throughout this lab we are going to be using a simple classification example: the TC classification problem (not to be confused with the real [TC](https://www.youtube.com/watch?v=NToYkBYezZA)). This is a small toy problem where we, initially, try to distinguish between 3x3 grids that look like Ts and Cs. Let's create the dataset and have a look..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I have written a function `load_letters()` to generate synthetic data. For now, you will use the data generated below, but later you have opportunity to play with generating different data if you like. The function is located in the `iaml` module adjacent to this notebook - feel free to check out the code but I advise you **do not edit it**. Run (and don't edit) the next few cells to create and observe the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bounds = [-1, 1]\n",
    "X, y, y_labels = load_letters(categories=['T', 'C'], \n",
    "                              num_obs=[50, 50],\n",
    "                              bounds=bounds,\n",
    "                              beta_params=[[1, 8], [8, 1]],\n",
    "                              shuffle=True, \n",
    "                              random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's print the data (I'm just creating a Pandas DataFrame for display, I probably wont use this object again)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.set_option(\"max_rows\", 10)\n",
    "df = pd.DataFrame(\n",
    "    np.hstack(\n",
    "        [np.around(X,2), \n",
    "         y[:, np.newaxis], \n",
    "         np.array([y_labels[ii] for ii in y])[:, np.newaxis]\n",
    "        ]\n",
    "    ),\n",
    "    columns = ['x{}'.format(ii) for ii in range(9)] + ['Class (numeric)', 'Class Label']\n",
    ")\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pd.reset_option(\"max_rows\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data are arranged as vectors for your convenience, but they're really `3 x 3` images. Here's a function to plot them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plot_grid(x, shape=None, **heatmap_params):\n",
    "    \"\"\"Function for reshaping and plotting vector data.\n",
    "    If shape not given, assumed square.\n",
    "    \"\"\"\n",
    "    if shape is None:\n",
    "        width = int(np.sqrt(len(x)))\n",
    "        if width == np.sqrt(len(x)):\n",
    "            shape = (width, width)\n",
    "        else:\n",
    "            print('Data not square, supply shape argument')\n",
    "    sns.heatmap(x.reshape(shape), annot=True, **heatmap_params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for ii in range(3):\n",
    "    plt.figure()\n",
    "    plot_grid(X[ii], vmin=bounds[0], vmax=bounds[1], cmap='Greys')\n",
    "    plt.title('Observation {}: Class = {} (numeric label {})'.format(ii, y_labels[y[ii]], y[ii]))\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's make the train and test split. Again, don't alter this code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.5, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train, X_valid, y_train, y_valid = train_test_split(\n",
    "    X_train, y_train, test_size=0.33, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[dd.shape for dd in [X_train, X_valid, X_test, y_train, y_valid, y_test]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1 - Introducing the Neural Network Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Resources to Watch and Read pt. 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Reading/watching time:** 30 minutes\n",
    "\n",
    "First, watch this video from 3 Blue 1 Brown: [But what *is* a Neural Network? | Deep learning, chapter 1](https://www.youtube.com/watch?v=aircAruvnKk)\n",
    "\n",
    "If you prefer reading, try 2 sections of Nielsen's Book Chapter 1:\n",
    "* [Sigmoid Neurons](http://neuralnetworksanddeeplearning.com/chap1.html#sigmoid_neurons)\n",
    "* and [The Architecture of Neural Networks](http://neuralnetworksanddeeplearning.com/chap1.html#the_architecture_of_neural_networks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Design"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just so as there's something in this notebook to quickly reference - here's a nice illustration of what's going on in a neural net. Within the calculation of the $z$'s you'll see the learned **parameters**: $w$'s and $b$'s - these are the weights and biases respectively. *N.B. I omit the bias $b$ parameters in the Part 3 implementation.* The functions $g$ are the activation functions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/neural-net.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Cost Space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "When we talk about the cost space, loss$^*$ space, or cost surface, we are talking about a function that changes with respect to the parameters. This function determines how well the network is performing - a low cost is good, a high cost is bad. A simple example for two parameters is shown below. **Our goal is to update the parameters such that we find the global minimum of the cost function.**\n",
    "\n",
    "$^*$ 'loss' and 'cost' are interchangeable terms - you'll see them both around but I try to stick to 'cost'!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/cost_space.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "N.B. The cost function is often referred to with different letters e.g. $J(w)$, $C(\\theta)$, $\\mathcal{L}(x)$, and $E(w)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2 - Fitting the Model & Optimisation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Resources to Watch and Read pt. 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Watching/reading time:** ~1 hour\n",
    "\n",
    "First, watch these two videos from 3 Blue 1 Brown:\n",
    "1. [Gradient descent, how neural networks learn | Deep learning, chapter 2](https://www.youtube.com/watch?v=IHZwWFHWa-w)\n",
    "2. [What is backpropagation and what is it actually doing? | Deep learning, chapter 3](https://www.youtube.com/watch?v=Ilg3gGewQ5U)\n",
    "\n",
    "This will take you just over half an hour (if you watch at 1x speed). They are really excellent and well worth the time investment.\n",
    "\n",
    "Again, if you prefer reading try Nielsen's section [Learning with Gradient Descent](http://neuralnetworksanddeeplearning.com/chap1.html#learning_with_gradient_descent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finding the Best Parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "So, we've got a function, let's call it $C(\\theta)$ that puts a number on how well the neural network is doing. We provide the function with the parameters $\\theta$ and it spits out the cost$^*$. We could just randomly chose values for $\\theta$ and select the ones that result in the best cost...but that might take a long time. We'd also need to define a way to randomly select parameters as well. What if the best parameter setting is very unlikely to be selected?\n",
    "\n",
    "**Calculus to the rescue!** The cost $C(\\theta)$ is a function and, whilst we can't see the surface without evaluating it everywhere (expensive!), we can calculate the derivative with respect to the parameters $\\frac{\\partial C(\\theta)}{\\partial \\theta}$. The derivative **tells you how the function value changes if you change $\\theta$**. \n",
    "\n",
    "For example, imagine $\\theta$ is 1D and I tell you that $\\frac{\\partial C(\\theta)}{\\partial \\theta} = 10\\theta$. This means that if I increase $theta$ by 2, the cost function will go up by 20. Which way will you update $\\theta$? You want to *decrease* the cost, so you would want to *decrease* $\\theta$ by some amount.\n",
    "\n",
    "The only thing we need to do is choose a cost function $C(\\theta)$ that has a derivative function $\\frac{\\partial C(\\theta)}{\\partial \\theta}$...and that is easy!\n",
    "\n",
    "$^*$It's much easier if you imagine $\\theta$ as just one number to start with, but the maths is basically the same as $\\theta$ becomes a vector (or matrix) of numbers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gradient Descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So how do we actually update the parameters?! All update the parameters in the opposite direction to the gradient; you always try to take a step 'downhill'. Here's the formula:\n",
    "\n",
    "$$\n",
    "\\theta \\leftarrow \\theta - \\eta \\frac{\\partial C(\\theta)}{\\partial \\theta}\n",
    "$$\n",
    "\n",
    "where \"$\\leftarrow$\" means \"update from\", and $\\eta$ is the \"learning rate\" - a hyperparameter you can choose. If you increase $\\eta$ you make bigger updates to $\\theta$, and vice versa. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many more complicated ways to update the parameters using the gradient of the cost function, but they all have this same starting point.\n",
    "\n",
    "Below is an example cost surface. A few things to note:\n",
    "\n",
    "* The axes should be labelled $\\theta_0$ (1, -1.5) and $\\theta_1$ (-1, 1) on the 'flat' axes, and $C(\\theta)$ (-4, 4) on the vertical axis\n",
    "* The surface is shown - we don't have direct access to this in reality. To show it, the creator has queried the cost function *at every [$\\theta_0$, $\\theta_1$] location* and plotted it\n",
    "* The animated balls rolling along the surface are different gradient descent algorithms - each frame of the GIF shows one update. The equation shown above is SGD - the GIF highlights a potential issue with the algorithm!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<img src=\"https://i.imgur.com/2dKCQHh.gif\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Visualisation by [Alec Radford](https://blog.openai.com/tag/alec-radford/), summarised excellently in [this blog post](http://ruder.io/optimizing-gradient-descent/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Backpropagation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Reading/watching time:** 1 hour\n",
    "\n",
    "Right...it's time for some derivatives. If you've been liking the videos - go ahead and watch the next in the series:\n",
    "\n",
    "1. [Backpropagation calculus | Appendix to deep learning chapter 3](https://www.youtube.com/watch?v=tIeHLnjs5U8)\n",
    "\n",
    "If you have time, I recommend now having a crack at reading half of [Nielsen Chapter 2](http://neuralnetworksanddeeplearning.com/chap2.html), up to and including the section entitled [The Backpropagation Algorithm](http://neuralnetworksanddeeplearning.com/chap2.html#the_backpropagation_algorithm).\n",
    "\n",
    "I'm just going to write out some derivatives you're going to find useful for Part 3 below:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "$$\n",
    "\\begin{align}\n",
    "z^{(L)}                             &= W^{(L)}a^{(L-1)}  \\\\\n",
    "\\frac{\\partial z^{(L)}}{\\partial W} &= a^{(L-1)}\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\text{linear}[z]                             &= z \\\\\n",
    "\\frac{\\partial \\text{linear}[z]}{\\partial z} &= 1 \\\\\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\text{sigmoid}[z] = \\sigma[z]                &= \\frac{1}{1 + e^{-z}} = \\frac{e^{z}}{e^{z} + 1}\\\\\n",
    "\\frac{\\partial \\sigma[z]}{\\partial z}        &= \\frac{e^{z}}{e^{z} + 1} - (\\frac{e^{z}}{e^{z} + 1})^2 \\\\\n",
    "                                             &= \\frac{e^{z}}{e^{z} + 1} ( 1 - \\frac{e^{z}}{e^{z} + 1} ) \\\\\n",
    "                                             &= \\sigma[z] (1 - \\sigma[z])\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\text{crossentropy}[y, a] = C[y, a]       &= - \\frac{1}{N} \\sum_{i=1}^N y_i \\log a_i + (1-y_i)\\log(1-a_i) \\\\\n",
    "\\frac{\\partial C[y_i, a_i]}{\\partial a_i} &=  \\frac{1 - y_i}{1 - a_i} + \\frac{y_i}{a_i}\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "And finally, this is all backpropagation really is...\n",
    "$$\n",
    "\\begin{align}\n",
    "\\frac{\\partial C[y_i, a_i]}{\\partial w_j} &=  \\frac{\\partial a_i}{\\partial w_j}\\frac{\\partial C[y_i, a_i]}{\\partial a_i}\\\\\n",
    "                                          &=  \\frac{\\partial z_k}{\\partial w_j}\\frac{\\partial a_i}{\\partial z_k}\\frac{\\partial C[y_i, a_i]}{\\partial a_i}\\\\\n",
    "\\end{align}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Challenge: derive these yourself."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Reading extension"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more on gradient based optimisers [check out this blog post](http://ruder.io/optimizing-gradient-descent/)\n",
    "\n",
    "For another look at backpropagation - try [Christopher Olah's blog](http://colah.github.io/posts/2015-08-Backprop/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 3 - Implementation From Scratch!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 3.1 =========="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First thing is first: **don't get stuck on this**. I recommend you attempt this question for an hour and, if you don't get anywhere, move on to Question 3.2. You can even move straight on to Part 4. It's exactly the same problem addressed here in 3.1, but using sklearn instead of coding it yourself."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model Specification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/network_design.png\" width=\"50%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to fit a very small neural network to classify the TC data. Here is the specification of the model:\n",
    "\n",
    "1. Input of size 9\n",
    "1. Hidden layer of size 3\n",
    "    * Linear activation function\n",
    "1. Output layer of size 1\n",
    "    * Logistic activation function\n",
    "\n",
    "As for the **cost function**: use Cross-Entropy. However, if you're getting bogged down with derivatives, feel free to try squared error to start with (this is what Nielsen and 3 Blue 1 Brown start with in their tutorials). Squared error is [not necessarily the right cost function to use](https://jamesmccaffrey.wordpress.com/2013/11/05/why-you-should-use-cross-entropy-error-instead-of-classification-error-or-mean-squared-error-for-neural-network-classifier-training/) but it will still work!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For a given input vector $x$, we can predict an output probability $a^{(2)}$ (were the $^{(2)}$ indicates the layer number, *not a power* - I'm following 3 Blue 1 Brown notation as best I can) using the following formula:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "a^{(2)} &= f^{(2)}[z^{(2)}] \\\\\n",
    "        &= f^{(2)}[W^{(2)}a^{(1)}] \\\\\n",
    "        &= f^{(2)}[W^{(2)}f^{(1)}[z^{(1)}]] \\\\\n",
    "        &= f^{(2)}[W^{(2)}f^{(1)}[W^{(1)}a^{(0)}]] \\\\\n",
    "        &= f^{(2)}[W^{(2)}f^{(1)}[W^{(1)}x]] \\\\\n",
    "        &= \\sigma[W^{(2)}(W^{(1)}x)]\n",
    "\\end{align}\n",
    "$$ \n",
    "\n",
    "where:\n",
    "\n",
    "* $f^{(2)}$ is the activation function of the output layer (a sigmoid function $\\sigma[]$)\n",
    "* $f^{(1)}$ is the activation function of the hidden layer (the identity - 'linear activation')\n",
    "* $W^{(2)}$ and $W^{(1)}$ are the parameters to learn\n",
    "* $a^{(L)} = f^{(L)}[z^{(L)}]$ are the activations **exiting** layer $^{(L)}$\n",
    "* $z^{(L)} = W^{(L)}a^{(L-1)}$ is the pre-activation weighted sum calculated **within** layer $^{(L)}$\n",
    "\n",
    "The formula for the Cross-Entropy cost function is:\n",
    "\n",
    "$$\n",
    "C(a) = - \\frac{1}{N} \\sum_{i=1}^N y_i \\log a_i + (1-y_i)\\log(1-a_i)\n",
    "$$\n",
    "\n",
    "Notice how only one term in the sum is ever non-zero because $y_i$ is only ever 0 or 1. In our case, $N$ is the number of data observations in the dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The parameters of the model are two matrices:\n",
    "\n",
    "1. $W^{(2)}$ - $3 \\times 9$ matrix\n",
    "    * used within the hidden layer (the $1^{st}$ layer) to get $z^{(1)} = W^{(1)}x$ for some $9 \\times 1$ input vector $x$. $z^{(1)}$ is thus $3 \\times 1$.\n",
    "1. $W^{(1)}$ - $1 \\times 3$ matrix\n",
    "    * used within the output layer (the $2^{nd}$ layer) to get $z^{(2)} = W^{(2)}a^{(1)}$ for some $3 \\times 1$ input vector $a^{(1)}$. $z^{(2)}$ is thus $1 \\times 1$.\n",
    "\n",
    "**Note that I'm not asking you to fit *bias parameters*.**\n",
    "\n",
    "You'll often see parameters referred to as $\\theta$, it's a catch all term. In our case it's just a list of all the weights, $\\theta = [W^{(1)}, W^{(2)}]$. **We have 3 x 9 + 3 x 1 = 30 parameters to learn in total.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Advice"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use any of the equations and code I've given you or linked you to in this lab but **you do not have to!** You're free to code as you please. Personally, since this is a simple example, I did not do anything fancy (I didn't create any objects with methods and attributes). I simply:\n",
    "* created a list containing the two parameter matrices `theta = [W1, W2]`\n",
    "* created a function to do prediction (the forward pass)\n",
    "* created a function to do the backward pass (updating the weights)\n",
    "    * This is the tricky bit - I coded functions that are the [relevant derivatives](#http://localhost:8888/notebooks/10_Lab_5_Neural_Networks.ipynb#Backpropagation), and wrote code to iteratively pass back the 'deltas' - (I think Nielsen's equations [here](http://neuralnetworksanddeeplearning.com/chap2.html#the_backpropagation_algorithm) are very useful)  \n",
    "* wrote a training loop which called these two main functions\n",
    "    * each epoch calls the forward pass to predict, then the backward pass to update the parameters.\n",
    "\n",
    "When the training was finished, my \"model\" was simply the parameters I had fitted, along with the 'forward pass' function - a function which uses those weights to predict a probability for any input data.\n",
    "\n",
    "**You do not have to code it up like me**, you can do it however you like! The point of this part is for you to explore, code up all the equations, understand how to calculate the loss, and how to use that loss to update the parameters of the model by backpropagation.\n",
    "\n",
    "**Debugging**: You're probably going to have issues particularly in the backprop section. You are welcome to make use of the `scipy.optimize.check_grad()` function. This takes as input a function f, g: a function that is (supposed to be) the function's derivative. \n",
    "\n",
    "If you didn't watch it already, now is a great time to take 10 minutes and watch [Backpropagation calculus | Appendix to deep learning chapter 3](https://www.youtube.com/watch?v=tIeHLnjs5U8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ===== What you actually need to do for this question! ====="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Write a training loop which uses gradient descent to learn the parameters. Each iteration of the loop is called an **epoch**. Run your code for *no more than 100 epochs*. You should be able to achieve 100% accuracy on this problem. \n",
    "\n",
    "In this case, for simplicity, you may initialise the weights to be samples from a normal distribution mean 0 variance 1, but please note that this [is not necessarily good practice](https://intoli.com/blog/neural-network-initialization/)!\n",
    "\n",
    "**Do not code up a grid search for the learning rate hyperparameter**. You may instead play with the learning rate manually until you are happy. Try small values first like 0.0001 (if your backprop code is correct you **should** see your cost decreasing every epoch). Since this problem is so simple, a range of values should work. Again, with real data, you *must* do a search over hyperparameters, but here we are focussed on *coding* a working model.\n",
    "\n",
    "To test whether or not what you have written has worked, please output the following:\n",
    "1. After the training loop:\n",
    "    1. plot a graph of training and validation loss against epoch number\n",
    "    1. print or plot the final parameters you have learned using a Hinton diagram - feel free to use [code you can find online](http://bfy.tw/F74s)\n",
    "    1. pick one weight parameter and produce a plot of its value against epoch number\n",
    "        * Extension: do that for all the weights **leaving one specific input node** (i.e. the weights for one pixel of the input data)\n",
    "    1. use your model to:\n",
    "        1. print a few of the validation data examples and their predicted probabilities\n",
    "        1. print the output for a T and C with no noise (you can make that input data yourself)\n",
    "        1. print the output of a few random binary vectors i.e. 9x1 vectors of only 0s and 1s (again, you can make that input data yourself)\n",
    "\n",
    "1. Within the training loop:\n",
    "    1. print the training and validation crossentropy loss **and** percentage accuracy every epoch\n",
    "    1. save the value of the training and validation losses for every epoch [for the plot after the loop]\n",
    "    1. save the value of a weight parameter of your choice [for the plot after the loop]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ===== Example outputs ====="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below I give you some examples of what I'd like you to produce. **I produced these using a learning rate of 0.003, 100 epochs, and weights initialised with N(0,1) with a random seed of 42**. I found that you could learn faster i.e. you can use a larger learning rate, but I wanted to make smooth plots for you. \n",
    "\n",
    "You don't need to produce plots exactly like this, you can do them how you like, but try and display the same information. You can also use my plots for checking (if you use the same settings as me)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/cost_per_epoch.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1B"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/hinton_W1.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/hinton_W2.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1C"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/W1_x4__per_epoch.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1D"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/predict_valid_0.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/predict_valid_No noise T.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/predict_valid_No noise C.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/predict_valid_N(0, 1) sample 1.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/predict_valid_N(0, 1) sample 2.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 2A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/training_log.png\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 3.2 =========="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Did you need a network this large to do this classification task? Give the values for the parameters of a network with no hidden layers, one output node, and an output activation function of a sigmoid that would get 100% accuracy. This network only has 9 parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 3.3 =========="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should recognise the model described in question 3.2. What is it?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 3.4 =========="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why did I create input data, `X`, that was between [-1, 1] i.e. why wasn't it between [0, 1] like normal?! Would the model specified in question 3.1 above have worked if `X` was in [0, 1]? Explain why or why not.\n",
    "\n",
    "*Hint: if you're stuck, you can try it out by generating some new data and trying to fit it.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 3.5 [EXTENSION] =========="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a dataset which makes the problem harder. Have a look at the dataset generation code. You can use the arguments to create data with:\n",
    "* more letters (make the problem a multiclass classification)\n",
    "    * You'll need to implement the multiclass version of the sigmoid for the output activation function - [the softmax](https://en.wikipedia.org/wiki/Softmax_function) (and of course it's derivative) \n",
    "* increase the noise on the data\n",
    "\n",
    "Some other things you could implement:\n",
    "* include rotated letters in the data\n",
    "* make larger data (bigger than 3x3)\n",
    "* make the letters non-centred e.g. 5x5 data with 3x3 letters in 1 of 9 different places\n",
    "\n",
    "You'll probably need to adapt the code you wrote in 3.1, but you can probably copy and paste most of it. For an additional challenge: introduce [bias parameters](http://neuralnetworksanddeeplearning.com/chap1.html) and create your `X` data in range [0, 1] (i.e. set the bounds argument to [0, 1])...\n",
    "\n",
    "Some other things to try if you get code happy:\n",
    "* Implement stochastic gradient descent updates (updating parameters every training example, as opposed to every epoch) - tip: randomise data order each epoch\n",
    "* Implement batch gradient descent updates - tip: randomise data order each epoch\n",
    "\n",
    "**Requirements**:\n",
    "1. Describe the modelling problem and your input data. Plot some examples of the data\n",
    "1. Write down the model specification (I should be able to reproduce your model with this description):\n",
    "    * number of nodes in each layer\n",
    "    * a description of the parameters to learn (and a total number of parameters)\n",
    "    * the activation functions used for each layer\n",
    "    * cost function used\n",
    "1. All the outputs asked for in Question 3.1: loss per epoch plot, final parameters, a weight against epoch plot, and example predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 4 - Implementation with Sklearn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 4.1 =========="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you did Question 3.1, this should be a breeze! Use the same data and perform the same modelling task. This time you can use Sklearn's Neural Network object [MLPClassifier](http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html#sklearn.neural_network.MLPClassifier).\n",
    "\n",
    "\n",
    "Before you begin, read the [introduction](http://scikit-learn.org/stable/modules/neural_networks_supervised.html) (sections 1.17.1 and 1.17.2 at a minimum, 1.17.5, 1.17.6, 1.17.7 are recommended)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 4.2 =========="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The learned parameters are stored in the fitted sklearn [MLPClassifier](http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html#sklearn.neural_network.MLPClassifier) object **as two separate attributes**.\n",
    "\n",
    "1. Print the parameters learned by your fitted model\n",
    "1. Print the total number of parameters learned\n",
    "\n",
    "Look at the number of parameters described in question 3.1 (you do not need to have done this question 3.1 - just read its description). Below the code:\n",
    "\n",
    "1. Explain why the number of parameters learned by sklearn is different from the number specified in 3.1?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [Please sir...I want some more](https://www.youtube.com/watch?v=Ex2r86G0sdc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well done, you successfully covered the basics of Neural Networks!\n",
    "\n",
    "If you enjoyed this lab, you'll love another course @ Edinburgh: [Machine Learning Practical](https://github.com/CSTR-Edinburgh/mlpractical). Check it out."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Next steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first thing to do, if you haven't already, is do the extension question 3.5. **In particular, you should implement bias parameters in your model code**.\n",
    "\n",
    "Next, go back to the very top of the notebook where I detail things I will not cover. Pick some words you don't understand (perhaps along with the keyword 'example' or 'introduction') and have fun reading/watching some tutorials about them online. Code up what you have learned; if you can code it up without peeking, you know you have understood it very well indeed. Another good \"starter for 10\" google is \"a review of neural networks for [images|text|music|bat detection|captioning images|generation|...]\".\n",
    "\n",
    "Here are some things that you might find fun to read:\n",
    "* [Visualising networks learning](http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=circle&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=5&networkShape=3&seed=0.42978&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&hideText=false)\n",
    "* [Trying to understand what features are learned by Deep Nets](https://distill.pub/2017/feature-visualization/)\n",
    "* [Modelling sound waves](https://deepmind.com/blog/wavenet-generative-model-raw-audio/)\n",
    "   * ...and using that to [encode instruments](https://magenta.tensorflow.org/nsynth)\n",
    "* An [Introduction to LSTMs](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) and their [unreasonable effectiveness](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)\n",
    "* How to encode the entire meaning of a word [in a few numbers](http://colah.github.io/posts/2014-07-NLP-RNNs-Representations/)\n",
    "* [Convolutions for text data?!](http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning resources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also:\n",
    "* [there](http://ufldl.stanford.edu/tutorial/supervised/MultiLayerNeuralNetworks/)\n",
    "* [are](http://neuralnetworksanddeeplearning.com/chap1.html)\n",
    "* [literally](https://www.coursera.org/learn/machine-learning)\n",
    "* [so](https://www.coursera.org/learn/neural-networks)\n",
    "* [many](http://deeplearning.net/)\n",
    "* [learning](http://datasciencemasters.org/)\n",
    "* [resources](https://metacademy.org/graphs/concepts/backpropagation)\n",
    "* [online!](http://www.deeplearningbook.org/)\n",
    "\n",
    "(about neural nets etc.)\n",
    "\n",
    "In all seriousness, make sure you check out [metacademy](https://metacademy.org/). You can search for a topic and it gives you a list of free resources, an estimated time you need to understand it, and prerequisite topics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Attributions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Parts of this lab were inspired by  D. E. Rumelhart, G. E. Hinton, and R. J. Williams, Parallel distributed processing: Explorations\n",
    "in the microstructure of cognition, vol. 1, MIT Press, Cambridge, MA, USA, 1986,\n",
    "pp. 318–362.\n",
    "\n",
    "\n",
    "Thanks also to:\n",
    "* [3 Blue 1 Brown](https://www.youtube.com/channel/UCYO_jab_esuFRV4b17AJtAw)\n",
    "* [Michael Nielsen](http://neuralnetworksanddeeplearning.com)\n",
    "* [Christopher Olah](http://colah.github.io/)\n",
    "\n",
    "for producing some excellent visualisations and learning resources and providing them free of charge.\n",
    "\n",
    "Additionally, many thanks to the developers of open source software, in particular:\n",
    "* [Numpy](http://www.numpy.org/)\n",
    "* [Scipy](https://www.scipy.org/)\n",
    "* [Sklearn](http://scikit-learn.org/stable/)\n",
    "* [Matplotlib](https://matplotlib.org/)\n",
    "* [Jupyter](http://jupyter.org/)\n",
    "* and of course [Python](https://www.python.org/) itself!\n",
    "\n",
    "your work is invaluable and appreciated."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Credits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This lab was created by [James Owers](https://jamesowers.github.io/) in November 2017 and reviewed by [Patric Fulop](https://www.inf.ed.ac.uk/people/students/Patric_Fulop.html)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}