10_Lab_5_Neural_Networks_solution.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [Introductory applied machine learning (INFR10069)](https://www.learn.ed.ac.uk/webapps/blackboard/execute/content/blankPage?cmd=view&content_id=_2651677_1&course_id=_53633_1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 5: Neural Networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*by [James Owers](https://jamesowers.github.io/), University of Edinburgh 2017*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. [Introduction](#Introduction)\n",
    "    * [Lab Outline](#Lab-Outline)\n",
    "    * [The Data](#The-Data)\n",
    "1. [Part 1 - Introducing the Neural Network Model](#Part-1---Introducing-the-Neural-Network-Model)\n",
    "    * [Resources to Watch and Read pt. 1](##Resources-to-Watch-and-Read-pt.-1)\n",
    "    * [Model Design](#Model-Design)\n",
    "    * [The Cost Space](#The-Cost-Space)\n",
    "1. [Part 2 - Fitting the Model & Optimisation](#Part-2---Fitting-the-Model-&-Optimisation)\n",
    "    * [Resources to Watch and Read pt. 2](#Resources-to-Watch-and-Read-pt.-2)\n",
    "    * [Finding the Best Parameters](#Finding-the-Best-Parameters)\n",
    "    * [Gradient Descent](#Gradient-Descent)\n",
    "    * [Backpropagation](#Backpropagation)\n",
    "1. [Part 3 - Implementation From Scratch](#Part-3---Implementation-From-Scratch!)\n",
    "1. [Part 4 - Implementation With Sklearn](#Part-4---Implementation-with-Sklearn)\n",
    "1. [Moar?!](#Please-sir...I-want-some-more)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# https://docs.python.org/2/library/__future__.html\n",
    "# make printing and division act like python 3\n",
    "from __future__ import division, print_function\n",
    "\n",
    "# General\n",
    "import sys\n",
    "import os\n",
    "import copy\n",
    "from IPython.display import Image, HTML\n",
    "\n",
    "# Data structures\n",
    "import numpy as np \n",
    "import pandas as pd\n",
    "\n",
    "# Modelling\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "from scipy.optimize import check_grad\n",
    "\n",
    "# Plotting\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "# Local module adjacent to this notebook\n",
    "import iaml\n",
    "from iaml.data import load_letters\n",
    "\n",
    "# http://ipython.readthedocs.io/en/stable/interactive/magics.html\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This lab:\n",
    "1. introduces a simple neural network model in a supervised learning setting\n",
    "1. provides impetus to understand the fitting procedure of that, and other networks\n",
    "1. encourages you to implement a model from scratch\n",
    "1. models the same problem with the sklearn package\n",
    "1. makes you think about what you've done!\n",
    "\n",
    "It does not discuss in detail:\n",
    "1. any of the plethora of different activation functions you can use e.g. RELUs, SELUs, Tanh, ...\n",
    "1. how to initialise the parameters and why that matters\n",
    "1. issues with the fitting process e.g. local optima, and how to avoid them e.g. learning rate schedulers, momentum, RMSProp, Adam, cyclic learning rates\n",
    "1. issues with model complexity e.g. overfitting, and solutions such as dropout, regularisation, or using [shedloads of data](https://what-if.xkcd.com/63/)\n",
    "1. other tricks for speeding up and stablising fitting such as batch sizes, weight norm, layer norm\n",
    "1. deep networks and their tricks like skip connections, pooling, convolutions\n",
    "1. nor other more complex architectures like CNNs, RNNs, LSTMs, GANs, etc. etc.\n",
    "1. many, many, MANY other things (that probably were published, like, [yesterday](https://arxiv.org/abs/1711.04340v1))\n",
    "\n",
    "However, if you understand what is in this notebook well, **you will have the ability to understand [all of these things](https://i.imgflip.com/1zn8p9.jpg)**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lab outline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I provide you with a function that creates data then link you to some excellent resources to learn the basics. These resources are superb, short, and free. I highly, highly recommend setting aside a couple of hours to give them a good watch/read and, at the very least, use them for reference. \n",
    "\n",
    "After you have had a crack at the problems, I'll release the solutions. The solutions, particularly to part 3, walk you through the process of coding a simple neural neural network in detail.\n",
    "\n",
    "Parts 3 & 4 are practical, parts 1 & 2 are links to external resources to read. Whilst I recommend you soak up some context first with 1 & 2, feel free to jump in at the deep end and get your hands dirty with part 3 or 4."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Throughout this lab we are going to be using a simple classification example: the TC classification problem (not to be confused with the real [TC](https://www.youtube.com/watch?v=NToYkBYezZA)). This is a small toy problem where we, initially, try to distinguish between 3x3 grids that look like Ts and Cs. Let's create the dataset and have a look..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I have written a function `load_letters()` to generate synthetic data. For now, you will use the data generated below, but later you have opportunity to play with generating different data if you like. The function is located in the `iaml` module adjacent to this notebook - feel free to check out the code but I advise you **do not edit it**. Run (and don't edit) the next few cells to create and observe the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bounds = [-1, 1]\n",
    "X, y, y_labels = load_letters(categories=['T', 'C'], \n",
    "                              num_obs=[50, 50],\n",
    "                              bounds=bounds,\n",
    "                              beta_params=[[1, 8], [8, 1]],\n",
    "                              shuffle=True, \n",
    "                              random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's print the data (I'm just creating a Pandas DataFrame for display, I probably wont use this object again)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.set_option(\"max_rows\", 10)\n",
    "df = pd.DataFrame(\n",
    "    np.hstack(\n",
    "        [np.around(X,2), \n",
    "         y[:, np.newaxis], \n",
    "         np.array([y_labels[ii] for ii in y])[:, np.newaxis]\n",
    "        ]\n",
    "    ),\n",
    "    columns = ['x{}'.format(ii) for ii in range(9)] + ['Class (numeric)', 'Class Label']\n",
    ")\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pd.reset_option(\"max_rows\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data are arranged as vectors for your convenience, but they're really `3 x 3` images. Here's a function to plot them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plot_grid(x, shape=None, **heatmap_params):\n",
    "    \"\"\"Function for reshaping and plotting vector data.\n",
    "    If shape not given, assumed square.\n",
    "    \"\"\"\n",
    "    if shape is None:\n",
    "        width = int(np.sqrt(len(x)))\n",
    "        if width == np.sqrt(len(x)):\n",
    "            shape = (width, width)\n",
    "        else:\n",
    "            print('Data not square, supply shape argument')\n",
    "    sns.heatmap(x.reshape(shape), annot=True, **heatmap_params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for ii in range(3):\n",
    "    plt.figure()\n",
    "    plot_grid(X[ii], vmin=bounds[0], vmax=bounds[1], cmap='Greys')\n",
    "    plt.title('Observation {}: Class = {} (numeric label {})'.format(ii, y_labels[y[ii]], y[ii]))\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's make the train and test split. Again, don't alter this code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.5, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train, X_valid, y_train, y_valid = train_test_split(\n",
    "    X_train, y_train, test_size=0.33, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[dd.shape for dd in [X_train, X_valid, X_test, y_train, y_valid, y_test]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1 - Introducing the Neural Network Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Resources to Watch and Read pt. 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Reading/watching time:** 30 minutes\n",
    "\n",
    "First, watch this video from 3 Blue 1 Brown: [But what *is* a Neural Network? | Deep learning, chapter 1](https://www.youtube.com/watch?v=aircAruvnKk)\n",
    "\n",
    "If you prefer reading, try 2 sections of Nielsen's Book Chapter 1:\n",
    "* [Sigmoid Neurons](http://neuralnetworksanddeeplearning.com/chap1.html#sigmoid_neurons)\n",
    "* and [The Architecture of Neural Networks](http://neuralnetworksanddeeplearning.com/chap1.html#the_architecture_of_neural_networks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Design"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just so as there's something in this notebook to quickly reference - here's a nice illustration of what's going on in a neural net. Within the calculation of the $z$'s you'll see the learned **parameters**: $w$'s and $b$'s - these are the weights and biases respectively. *N.B. I omit the bias $b$ parameters in the Part 3 implementation.* The functions $g$ are the activation functions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/neural-net.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Cost Space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "When we talk about the cost space, loss$^*$ space, or cost surface, we are talking about a function that changes with respect to the parameters. This function determines how well the network is performing - a low cost is good, a high cost is bad. A simple example for two parameters is shown below. **Our goal is to update the parameters such that we find the global minimum of the cost function.**\n",
    "\n",
    "$^*$ 'loss' and 'cost' are interchangeable terms - you'll see them both around but I try to stick to 'cost'!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/cost_space.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "N.B. The cost function is often referred to with different letters e.g. $J(w)$, $C(\\theta)$, $\\mathcal{L}(x)$, and $E(w)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2 - Fitting the Model & Optimisation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Resources to Watch and Read pt. 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Watching/reading time:** ~1 hour\n",
    "\n",
    "First, watch these two videos from 3 Blue 1 Brown:\n",
    "1. [Gradient descent, how neural networks learn | Deep learning, chapter 2](https://www.youtube.com/watch?v=IHZwWFHWa-w)\n",
    "2. [What is backpropagation and what is it actually doing? | Deep learning, chapter 3](https://www.youtube.com/watch?v=Ilg3gGewQ5U)\n",
    "\n",
    "This will take you just over half an hour (if you watch at 1x speed). They are really excellent and well worth the time investment.\n",
    "\n",
    "Again, if you prefer reading try Nielsen's section [Learning with Gradient Descent](http://neuralnetworksanddeeplearning.com/chap1.html#learning_with_gradient_descent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finding the Best Parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "So, we've got a function, let's call it $C(\\theta)$ that puts a number on how well the neural network is doing. We provide the function with the parameters $\\theta$ and it spits out the cost$^*$. We could just randomly chose values for $\\theta$ and select the ones that result in the best cost...but that might take a long time. We'd also need to define a way to randomly select parameters as well. What if the best parameter setting is very unlikely to be selected?\n",
    "\n",
    "**Calculus to the rescue!** The cost $C(\\theta)$ is a function and, whilst we can't see the surface without evaluating it everywhere (expensive!), we can calculate the derivative with respect to the parameters $\\frac{\\partial C(\\theta)}{\\partial \\theta}$. The derivative **tells you how the function value changes if you change $\\theta$**. \n",
    "\n",
    "For example, imagine $\\theta$ is 1D and I tell you that $\\frac{\\partial C(\\theta)}{\\partial \\theta} = 10\\theta$. This means that if I increase $theta$ by 2, the cost function will go up by 20. Which way will you update $\\theta$? You want to *decrease* the cost, so you would want to *decrease* $\\theta$ by some amount.\n",
    "\n",
    "The only thing we need to do is choose a cost function $C(\\theta)$ that has a derivative function $\\frac{\\partial C(\\theta)}{\\partial \\theta}$...and that is easy!\n",
    "\n",
    "$^*$It's much easier if you imagine $\\theta$ as just one number to start with, but the maths is basically the same as $\\theta$ becomes a vector (or matrix) of numbers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gradient Descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So how do we actually update the parameters?! All update the parameters in the opposite direction to the gradient; you always try to take a step 'downhill'. Here's the formula:\n",
    "\n",
    "$$\n",
    "\\theta \\leftarrow \\theta - \\eta \\frac{\\partial C(\\theta)}{\\partial \\theta}\n",
    "$$\n",
    "\n",
    "where \"$\\leftarrow$\" means \"update from\", and $\\eta$ is the \"learning rate\" - a hyperparameter you can choose. If you increase $\\eta$ you make bigger updates to $\\theta$, and vice versa. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are many more complicated ways to update the parameters using the gradient of the cost function, but they all have this same starting point.\n",
    "\n",
    "Below is an example cost surface. A few things to note:\n",
    "\n",
    "* The axes should be labelled $\\theta_0$ (1, -1.5) and $\\theta_1$ (-1, 1) on the 'flat' axes, and $C(\\theta)$ (-4, 4) on the vertical axis\n",
    "* The surface is shown - we don't have direct access to this in reality. To show it, the creator has queried the cost function *at every [$\\theta_0$, $\\theta_1$] location* and plotted it\n",
    "* The animated balls rolling along the surface are different gradient descent algorithms - each frame of the GIF shows one update. The equation shown above is SGD - the GIF highlights a potential issue with the algorithm!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<img src=\"https://i.imgur.com/2dKCQHh.gif\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Visualisation by [Alec Radford](https://blog.openai.com/tag/alec-radford/), summarised excellently in [this blog post](http://ruder.io/optimizing-gradient-descent/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Backpropagation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Reading/watching time:** 1 hour\n",
    "\n",
    "Right...it's time for some derivatives. If you've been liking the videos - go ahead and watch the next in the series:\n",
    "\n",
    "1. [Backpropagation calculus | Appendix to deep learning chapter 3](https://www.youtube.com/watch?v=tIeHLnjs5U8)\n",
    "\n",
    "If you have time, I recommend now having a crack at reading half of [Nielsen Chapter 2](http://neuralnetworksanddeeplearning.com/chap2.html), up to and including the section entitled [The Backpropagation Algorithm](http://neuralnetworksanddeeplearning.com/chap2.html#the_backpropagation_algorithm).\n",
    "\n",
    "I'm just going to write out some derivatives you're going to find useful for Part 3 below:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "$$\n",
    "\\begin{align}\n",
    "z^{(L)}                             &= W^{(L)}a^{(L-1)}  \\\\\n",
    "\\frac{\\partial z^{(L)}}{\\partial W} &= a^{(L-1)}\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\text{linear}[z]                             &= z \\\\\n",
    "\\frac{\\partial \\text{linear}[z]}{\\partial z} &= 1 \\\\\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\text{sigmoid}[z] = \\sigma[z]                &= \\frac{1}{1 + e^{-z}} = \\frac{e^{z}}{e^{z} + 1}\\\\\n",
    "\\frac{\\partial \\sigma[z]}{\\partial z}        &= \\frac{e^{z}}{e^{z} + 1} - (\\frac{e^{z}}{e^{z} + 1})^2 \\\\\n",
    "                                             &= \\frac{e^{z}}{e^{z} + 1} ( 1 - \\frac{e^{z}}{e^{z} + 1} ) \\\\\n",
    "                                             &= \\sigma[z] (1 - \\sigma[z])\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\text{crossentropy}[y, a] = C[y, a]       &= - \\frac{1}{N} \\sum_{i=1}^N y_i \\log a_i + (1-y_i)\\log(1-a_i) \\\\\n",
    "\\frac{\\partial C[y_i, a_i]}{\\partial a_i} &=  \\frac{1 - y_i}{1 - a_i} + \\frac{y_i}{a_i}\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "And finally, this is all backpropagation really is...\n",
    "$$\n",
    "\\begin{align}\n",
    "\\frac{\\partial C[y_i, a_i]}{\\partial w_j} &=  \\frac{\\partial a_i}{\\partial w_j}\\frac{\\partial C[y_i, a_i]}{\\partial a_i}\\\\\n",
    "                                          &=  \\frac{\\partial z_k}{\\partial w_j}\\frac{\\partial a_i}{\\partial z_k}\\frac{\\partial C[y_i, a_i]}{\\partial a_i}\\\\\n",
    "\\end{align}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Challenge: derive these yourself."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Reading extension"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more on gradient based optimisers [check out this blog post](http://ruder.io/optimizing-gradient-descent/)\n",
    "\n",
    "For another look at backpropagation - try [Christopher Olah's blog](http://colah.github.io/posts/2015-08-Backprop/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 3 - Implementation From Scratch!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 3.1 =========="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First thing is first: **don't get stuck on this**. I recommend you attempt this question for an hour and, if you don't get anywhere, move on to Question 3.2. You can even move straight on to Part 4. It's exactly the same problem addressed here in 3.1, but using sklearn instead of coding it yourself."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model Specification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/network_design.png\" width=\"50%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to fit a very small neural network to classify the TC data. Here is the specification of the model:\n",
    "\n",
    "1. Input of size 9\n",
    "1. Hidden layer of size 3\n",
    "    * Linear activation function\n",
    "1. Output layer of size 1\n",
    "    * Logistic activation function\n",
    "\n",
    "As for the **cost function**: use Cross-Entropy. However, if you're getting bogged down with derivatives, feel free to try squared error to start with (this is what Nielsen and 3 Blue 1 Brown start with in their tutorials). Squared error is [not necessarily the right cost function to use](https://jamesmccaffrey.wordpress.com/2013/11/05/why-you-should-use-cross-entropy-error-instead-of-classification-error-or-mean-squared-error-for-neural-network-classifier-training/) but it will still work!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For a given input vector $x$, we can predict an output probability $a^{(2)}$ (were the $^{(2)}$ indicates the layer number, *not a power* - I'm following 3 Blue 1 Brown notation as best I can) using the following formula:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "a^{(2)} &= f^{(2)}[z^{(2)}] \\\\\n",
    "        &= f^{(2)}[W^{(2)}a^{(1)}] \\\\\n",
    "        &= f^{(2)}[W^{(2)}f^{(1)}[z^{(1)}]] \\\\\n",
    "        &= f^{(2)}[W^{(2)}f^{(1)}[W^{(1)}a^{(0)}]] \\\\\n",
    "        &= f^{(2)}[W^{(2)}f^{(1)}[W^{(1)}x]] \\\\\n",
    "        &= \\sigma[W^{(2)}(W^{(1)}x)]\n",
    "\\end{align}\n",
    "$$ \n",
    "\n",
    "where:\n",
    "\n",
    "* $f^{(2)}$ is the activation function of the output layer (a sigmoid function $\\sigma[]$)\n",
    "* $f^{(1)}$ is the activation function of the hidden layer (the identity - 'linear activation')\n",
    "* $W^{(2)}$ and $W^{(1)}$ are the parameters to learn\n",
    "* $a^{(L)} = f^{(L)}[z^{(L)}]$ are the activations **exiting** layer $^{(L)}$\n",
    "* $z^{(L)} = W^{(L)}a^{(L-1)}$ is the pre-activation weighted sum calculated **within** layer $^{(L)}$\n",
    "\n",
    "The formula for the Cross-Entropy cost function is:\n",
    "\n",
    "$$\n",
    "C(a) = - \\frac{1}{N} \\sum_{i=1}^N y_i \\log a_i + (1-y_i)\\log(1-a_i)\n",
    "$$\n",
    "\n",
    "Notice how only one term in the sum is ever non-zero because $y_i$ is only ever 0 or 1. In our case, $N$ is the number of data observations in the dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The parameters of the model are two matrices:\n",
    "\n",
    "1. $W^{(2)}$ - $3 \\times 9$ matrix\n",
    "    * used within the hidden layer (the $1^{st}$ layer) to get $z^{(1)} = W^{(1)}x$ for some $9 \\times 1$ input vector $x$. $z^{(1)}$ is thus $3 \\times 1$.\n",
    "1. $W^{(1)}$ - $1 \\times 3$ matrix\n",
    "    * used within the output layer (the $2^{nd}$ layer) to get $z^{(2)} = W^{(2)}a^{(1)}$ for some $3 \\times 1$ input vector $a^{(1)}$. $z^{(2)}$ is thus $1 \\times 1$.\n",
    "\n",
    "**Note that I'm not asking you to fit *bias parameters*.**\n",
    "\n",
    "You'll often see parameters referred to as $\\theta$, it's a catch all term. In our case it's just a list of all the weights, $\\theta = [W^{(1)}, W^{(2)}]$. **We have 3 x 9 + 3 x 1 = 30 parameters to learn in total.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Advice"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use any of the equations and code I've given you or linked you to in this lab but **you do not have to!** You're free to code as you please. Personally, since this is a simple example, I did not do anything fancy (I didn't create any objects with methods and attributes). I simply:\n",
    "* created a list containing the two parameter matrices `theta = [W1, W2]`\n",
    "* created a function to do prediction (the forward pass)\n",
    "* created a function to do the backward pass (updating the weights)\n",
    "    * This is the tricky bit - I coded functions that are the [relevant derivatives](#http://localhost:8888/notebooks/10_Lab_5_Neural_Networks.ipynb#Backpropagation), and wrote code to iteratively pass back the 'deltas' - (I think Nielsen's equations [here](http://neuralnetworksanddeeplearning.com/chap2.html#the_backpropagation_algorithm) are very useful)  \n",
    "* wrote a training loop which called these two main functions\n",
    "    * each epoch calls the forward pass to predict, then the backward pass to update the parameters.\n",
    "\n",
    "When the training was finished, my \"model\" was simply the parameters I had fitted, along with the 'forward pass' function - a function which uses those weights to predict a probability for any input data.\n",
    "\n",
    "**You do not have to code it up like me**, you can do it however you like! The point of this part is for you to explore, code up all the equations, understand how to calculate the loss, and how to use that loss to update the parameters of the model by backpropagation.\n",
    "\n",
    "**Debugging**: You're probably going to have issues particularly in the backprop section. You are welcome to make use of the `scipy.optimize.check_grad()` function. This takes as input a function f, g: a function that is (supposed to be) the function's derivative. \n",
    "\n",
    "If you didn't watch it already, now is a great time to take 10 minutes and watch [Backpropagation calculus | Appendix to deep learning chapter 3](https://www.youtube.com/watch?v=tIeHLnjs5U8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ===== What you actually need to do for this question! ====="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Write a training loop which uses gradient descent to learn the parameters. Each iteration of the loop is called an **epoch**. Run your code for *no more than 100 epochs*. You should be able to achieve 100% accuracy on this problem. \n",
    "\n",
    "In this case, for simplicity, you may initialise the weights to be samples from a normal distribution mean 0 variance 1, but please note that this [is not necessarily good practice](https://intoli.com/blog/neural-network-initialization/)!\n",
    "\n",
    "**Do not code up a grid search for the learning rate hyperparameter**. You may instead play with the learning rate manually until you are happy. Try small values first like 0.0001 (if your backprop code is correct you **should** see your cost decreasing every epoch). Since this problem is so simple, a range of values should work. Again, with real data, you *must* do a search over hyperparameters, but here we are focussed on *coding* a working model.\n",
    "\n",
    "To test whether or not what you have written has worked, please output the following:\n",
    "1. After the training loop:\n",
    "    1. plot a graph of training and validation loss against epoch number\n",
    "    1. print or plot the final parameters you have learned using a Hinton diagram - feel free to use [code you can find online](http://bfy.tw/F74s)\n",
    "    1. pick one weight parameter and produce a plot of its value against epoch number\n",
    "        * Extension: do that for all the weights **leaving one specific input node** (i.e. the weights for one pixel of the input data)\n",
    "    1. use your model to:\n",
    "        1. print a few of the validation data examples and their predicted probabilities\n",
    "        1. print the output for a T and C with no noise (you can make that input data yourself)\n",
    "        1. print the output of a few random binary vectors i.e. 9x1 vectors of only 0s and 1s (again, you can make that input data yourself)\n",
    "\n",
    "1. Within the training loop:\n",
    "    1. print the training and validation crossentropy loss **and** percentage accuracy every epoch\n",
    "    1. save the value of the training and validation losses for every epoch [for the plot after the loop]\n",
    "    1. save the value of a weight parameter of your choice [for the plot after the loop]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ===== Example outputs ====="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below I give you some examples of what I'd like you to produce. **I produced these using a learning rate of 0.003, 100 epochs, and weights initialised with N(0,1) with a random seed of 42**. I found that you could learn faster i.e. you can use a larger learning rate, but I wanted to make smooth plots for you. \n",
    "\n",
    "You don't need to produce plots exactly like this, you can do them how you like, but try and display the same information. You can also use my plots for checking (if you use the same settings as me)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/cost_per_epoch.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1B"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/hinton_W1.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/hinton_W2.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1C"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/W1_x4__per_epoch.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1D"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/predict_valid_0.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/predict_valid_No noise T.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/predict_valid_No noise C.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/predict_valid_N(0, 1) sample 1.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/predict_valid_N(0, 1) sample 2.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 2A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/training_log.png\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Forward pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def sigmoid(z):\n",
    "    \"\"\"Sigmoid activation function\n",
    "    \"\"\"\n",
    "    return 1 / (1 + np.exp(-z))\n",
    "\n",
    "\n",
    "def linear(z):\n",
    "    \"\"\"Linear activation function i.e. identity\n",
    "    \"\"\"\n",
    "    return z\n",
    "\n",
    "\n",
    "def predict(X, params, activation_funs, return_all=False):\n",
    "    \"\"\"Performs the 'forward pass' of the network which\n",
    "    calculates the output $a^{(L)}$, the predicted probability for y i.e.\n",
    "    the activation of the final layer\n",
    "    \n",
    "    Arguments\n",
    "    ---------\n",
    "    X : numpy array, The data, assumed N x D where N is the number of \n",
    "        observations and D the dimensionality\n",
    "    params : list, each element is the weight parameters for successive layers\n",
    "    activation_funs : list, each element is the activation function\n",
    "        corresponding to each layer\n",
    "    return_all : bool, whether to return a list of all the activations\n",
    "        from each layer (set to True), or just to give the final predictions\n",
    "        (Default: False)\n",
    "    Returns\n",
    "    -------\n",
    "    if return_all is True:\n",
    "        [activations, zs] : list, activations and zs are lists of numpy arrays\n",
    "    if return_all is False:\n",
    "    a : numpy array, the predicted probabilities i.e. the activations\n",
    "        that are output from the final layer\n",
    "    \"\"\"\n",
    "    activations = [X.T]\n",
    "    zs = []\n",
    "    for ii, (W, f) in enumerate(zip(params, activation_funs)):\n",
    "        a_incoming = activations[ii]\n",
    "        z = W.dot(a_incoming)\n",
    "        a_outgoing = f(z)\n",
    "        zs.append(copy.deepcopy(z))\n",
    "        activations.append(copy.deepcopy(a_outgoing))\n",
    "    if return_all:\n",
    "        return [activations, zs]\n",
    "    else:\n",
    "        return activations[-1]\n",
    "\n",
    "\n",
    "def crossentropy_cost(y, a):\n",
    "    \"\"\"Calculates the crossentropy cost (aka loss) for predictions a (the \n",
    "    activation output from the final layer) against true labels y. Assumes y is \n",
    "    binary data in [0, 1].\n",
    "    \n",
    "    See https://goo.gl/Aw7Q3i FMI.\n",
    "    \"\"\"\n",
    "    nr_obs = len(y)\n",
    "    # np.nan_to_num v useful for avoiding log(0) issues\n",
    "    costs = (-y * np.log(a)) - ((1 - y) * np.log(1 - a)) \n",
    "    # if the true label y = 1:\n",
    "    #     the first bracket gets very large as a goes to 1\n",
    "    #     and the second bracket is 0\n",
    "    # if the true label y = 0:\n",
    "    #     the second bracket gets very large as a goes to 0\n",
    "    #     and the first bracket is 0\n",
    "    return np.nan_to_num(np.sum(costs)) / nr_obs\n",
    "\n",
    "\n",
    "def squared_cost(y, a):\n",
    "    \"\"\"Calculates the sum of squared errors cost (aka loss) for predictions\n",
    "    a (the activation output from the final layer) against true labels y.\n",
    "    \"\"\"\n",
    "    nr_obs = len(y)\n",
    "    costs = (a - y)**2 \n",
    "    return np.sum(costs) / nr_obs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Backward pass (calculating the gradients and updating the parameters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def crossentropy_cost_derivative(y, a):\n",
    "    \"\"\"Derivative of the crossentropy cost function w.r.t activation a.\n",
    "    \"\"\"\n",
    "    # np.nan_to_num will help avoid division by 0 issues (generally as a -> y)\n",
    "    cost_derivs = np.nan_to_num((1 - y) / (1 - a)) - np.nan_to_num(y / a)\n",
    "    return np.nan_to_num(cost_derivs)\n",
    "\n",
    "\n",
    "def squared_cost_derivative(y, a):\n",
    "    \"\"\"Derivative of the MSE cost function w.r.t activation a.\n",
    "    \"\"\"\n",
    "    nr_obs = len(y)\n",
    "    return 2*(a - y) / nr_obs\n",
    "\n",
    "\n",
    "def sigmoid_derivative(z):\n",
    "    \"\"\"Derivative of sigmoid activation function w.r.t. z\n",
    "    \"\"\"\n",
    "    return sigmoid(z) * (1 - sigmoid(z))\n",
    "\n",
    "\n",
    "def linear_derivative(z):\n",
    "    \"\"\"Derivative of linear activation function i.e. identity w.r.t. z\n",
    "    \"\"\"\n",
    "    return 1\n",
    "\n",
    "\n",
    "def backward_pass(y, theta, fwd_pass_data, cost_fun_derivative, \n",
    "                  activation_funs_derivatives, learning_rate=0.001):\n",
    "    \"\"\"Performs the backward pass through the network.\n",
    "    Returns updated parameters\n",
    "    \n",
    "    Arguments\n",
    "    ---------\n",
    "    y : array, the outcome data\n",
    "    theta : list, each element is the weight parameters for successive layers\n",
    "    fwd_pass_data : list, the output data from the forward pass [activations, zs]\n",
    "        activations and zs are lists of numpy arrays corresponding to the activations\n",
    "        and intermediatry zs for each neuron in each layer. These are used for \n",
    "        *evaluating* the derivatives. **Create these with the predict function**\n",
    "    cost_fun_derivative : function, the function which returns the value of\n",
    "        the cost function's derivative when supplied with the true labels y, and\n",
    "        final layer activations a\n",
    "    activation_funs_derivatives : list, list of functions. Each function returns the\n",
    "        value of the derivative of the activation function of the layer (the index in the\n",
    "        list indicates the layer number excluding the input layer)\n",
    "    learning_rate : float, hyperparameter which determines the size of the step in the \n",
    "        direction of the negative gradient you should take when updating the parameters \n",
    "    \n",
    "    Returns\n",
    "    -------\n",
    "    theta: list, same as input but updated\n",
    "    \n",
    "    \"\"\"\n",
    "    nr_layers = len(theta)\n",
    "    activations, zs = fwd_pass_data\n",
    "    \n",
    "    ## Get the initial delta from the final layer (needs cost function derivative)\n",
    "    delta = cost_fun_derivative(y, activations[-1]) * activation_funs_derivatives[-1](zs[-1])\n",
    "    nablas = [None, None]  # https://en.wikipedia.org/wiki/Nabla_symbol\n",
    "                           # These are the gradient values used to update the parameters\n",
    "    nablas[-1] = np.dot(delta, activations[-2].transpose())\n",
    "    \n",
    "    ## Iterate *backwards* through the layers, passing deltas back and evaluating\n",
    "    ## the derivatives w.r.t. the parameters (uses the forward pass activations and z's)\n",
    "    ## N.B. for our initial model with 2 layers, this loop only has 1 iteration!\n",
    "    for ii in range(2, nr_layers+1):\n",
    "        W = theta[-ii + 1]                 # weights applied to the activations leaving this layer\n",
    "        a_incoming = activations[-ii - 1]  # activations coming into this layer\n",
    "        z = zs[-ii]                        # the z's calculated within this layer\n",
    "        act_fun_deriv = activation_funs_derivatives[-ii]\n",
    "        \n",
    "        delta = np.dot(W.transpose(), delta) * act_fun_deriv(z)\n",
    "        nablas[-2] = np.dot(delta, a_incoming.transpose())\n",
    "        \n",
    "    ## Update the parameters for each layer by gradient descent\n",
    "    for ii in range(len(theta)):\n",
    "        nabla = nablas[ii]\n",
    "        W = theta[ii]\n",
    "        theta[ii] = W - (learning_rate * nabla)\n",
    "    return theta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_accuracy(y, a, decision_boundary=.5):\n",
    "    \"\"\"Convenience function for getting accuracy\"\"\"\n",
    "    y_hat = (a > decision_boundary).astype(int)\n",
    "    correct_predictions = np.sum(y == y_hat)\n",
    "    accuracy = correct_predictions / len(y)\n",
    "    return accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training Loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Settings\n",
    "random_seed = 42\n",
    "nr_epochs = 101  # The way I've written code means 11 epochs amounts to 10 parameter updates\n",
    "activation_funs = [linear, sigmoid]\n",
    "activation_funs_derivatives = [linear_derivative, sigmoid_derivative]\n",
    "cost_fun = crossentropy_cost\n",
    "cost_fun_derivative = crossentropy_cost_derivative\n",
    "decision_boundary = .5\n",
    "learning_rate = 0.003 # You can actually go a lot faster than this\n",
    "                      # I chose to go slower for clearer graphs!\n",
    "\n",
    "# Weight initialisation - N(0, 1): quick and dirty\n",
    "## a better approach: http://neuralnetworksanddeeplearning.com/chap3.html#weight_initialization\n",
    "np.random.seed(random_seed)\n",
    "# W1 = 1/np.sqrt(9) * np.random.randn(3, 9)\n",
    "# W2 = 1/np.sqrt(3) * np.random.randn(1, 3)\n",
    "W1 = np.random.randn(3, 9)\n",
    "W2 = np.random.randn(1, 3)\n",
    "params = [W1, W2]\n",
    "\n",
    "# Things to store\n",
    "cost_per_epoch = dict(train=[], valid=[])\n",
    "accuracy_per_epoch = dict(train=[], valid=[])\n",
    "params_per_epoch = []\n",
    "\n",
    "\n",
    "# Training loop\n",
    "print('Beginning Training!\\n{}\\n'.format('-'*len('Beginning Training!')))\n",
    "print(' {:^10} | {:^10} | {:^10} | {:^10} | {:^10} '.\\\n",
    "      format('Epoch', 'Train Cost', 'Valid Cost', 'Train Acc', 'Valid Acc'))\n",
    "print(' {0}   {0}   {0}   {0}   {0} '.format(10*'-'))\n",
    "\n",
    "for ii in range(nr_epochs):\n",
    "    # WARNING: \n",
    "    # If your network is large and/or you are doing many epochs, it's not \n",
    "    # sensible to save the parameters every epoch - for this toy problem it's ok\n",
    "    params_per_epoch.append(copy.deepcopy(params))  # without deepcopy, python will\n",
    "                                                    # use a pointer only, resulting\n",
    "                                                    # in a final parameter list with\n",
    "                                                    # nr_epochs identical sets of params!\n",
    "    # Forward pass\n",
    "    fwd_pass_data = predict(X_train, params, activation_funs, return_all=True)\n",
    "    activations = fwd_pass_data[0]\n",
    "\n",
    "    a_out = dict(train=activations[-1],  # this is just to avoid calculating train output activations twice\n",
    "                 valid=predict(X_valid, params, activation_funs, return_all=False))\n",
    "    for dataset_name, (XX, yy) in dict(train=(X_train, y_train), \n",
    "                                       valid=(X_valid, y_valid)).iteritems():\n",
    "        aa = a_out[dataset_name]\n",
    "        accuracy_per_epoch[dataset_name].append(get_accuracy(yy, aa, decision_boundary))\n",
    "        cost_per_epoch[dataset_name].append(cost_fun(yy, aa))\n",
    "    \n",
    "    print(' {:>10d} | {:<10.7f} | {:<10.7f} | {:<10.7f} | {:<10.7f} '.\\\n",
    "          format(ii, cost_per_epoch['train'][ii], cost_per_epoch['valid'][ii], \n",
    "                 accuracy_per_epoch['train'][ii], accuracy_per_epoch['valid'][ii]))\n",
    "    \n",
    "    # Don't do backward pass on the final epoch (just to avoid repeating code) \n",
    "    if ii == nr_epochs - 1:\n",
    "        break\n",
    "    \n",
    "    # Backward pass\n",
    "    params = backward_pass(y_train, params, fwd_pass_data, cost_fun_derivative, \n",
    "                           activation_funs_derivatives, learning_rate=learning_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1A ---------------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure()\n",
    "for dataset_name in ('train', 'valid'):\n",
    "    plt.plot(cost_per_epoch[dataset_name], label=dataset_name)\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Training cost')\n",
    "plt.title('Training and Validation Cost')\n",
    "plt.legend() \n",
    "plt.savefig('./img/cost_per_epoch.png', dpi=100, bbox_inches='tight')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1B ---------------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def hinton(matrix, max_weight=None, ax=None):\n",
    "    \"\"\"Draw Hinton diagram for visualizing a weight matrix.\n",
    "    Source: https://matplotlib.org/examples/specialty_plots/hinton_demo.html\n",
    "    \"\"\"\n",
    "    ax = ax if ax is not None else plt.gca()\n",
    "\n",
    "    if not max_weight:\n",
    "        max_weight = 2 ** np.ceil(np.log(np.abs(matrix).max()) / np.log(2))\n",
    "\n",
    "    ax.patch.set_facecolor('gray')\n",
    "    ax.set_aspect('equal', 'box')\n",
    "    ax.xaxis.set_major_locator(plt.NullLocator())\n",
    "    ax.yaxis.set_major_locator(plt.NullLocator())\n",
    "\n",
    "    for (x, y), w in np.ndenumerate(matrix):\n",
    "        color = 'white' if w > 0 else 'black'\n",
    "        size = np.sqrt(np.abs(w) / max_weight)\n",
    "        rect = plt.Rectangle([x - size / 2, y - size / 2], size, size,\n",
    "                             facecolor=color, edgecolor=color)\n",
    "        ax.add_patch(rect)\n",
    "\n",
    "    ax.autoscale_view()\n",
    "    ax.invert_yaxis()\n",
    "\n",
    "final_params = params_per_epoch[-1]\n",
    "\n",
    "for ii, W in enumerate(final_params):\n",
    "    plt.figure()\n",
    "    hinton(W.T)\n",
    "    plt.title('$W^{{({})}}$'.format(ii + 1))\n",
    "    plt.tight_layout()\n",
    "    plt.savefig('img/hinton_W{}.png'.format(ii + 1), dpi=100, bbox_inches='tight')\n",
    "    plt.show()\n",
    "\n",
    "W1 = final_params[0]\n",
    "for ii in range(3):\n",
    "    plt.figure()\n",
    "    hinton(W1[ii, :].reshape(3, 3).T)\n",
    "    plt.title('$W^{{({})}}_{}$'.format(1, ii))\n",
    "    plt.savefig('img/hinton_W{}_{}.png'.format(1, ii), dpi=100, bbox_inches='tight')\n",
    "    plt.tight_layout()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1C ---------------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# I disregard my instructions, and show the plots for weights leaving every input data pixel (seeing as there aren't many)\n",
    "# You need only plot a single weight over time, I'm just showing off.\n",
    "W1_per_epoch = [W1 for W1, W2 in params_per_epoch]\n",
    "for ii in range(3):\n",
    "    for jj in range(3):\n",
    "        plt.figure()\n",
    "        plt.title('Weights multiplying pixel at ({}, {})'.format(ii, jj))\n",
    "        idx = ii + 3*jj\n",
    "        weights_over_time = [W1[:, idx] for W1 in W1_per_epoch]\n",
    "        lines = plt.plot(weights_over_time)\n",
    "        plt.legend(handles=lines, labels=['Weight for hidden unit {}'.format(kk) for kk in range(3)],\n",
    "                   bbox_to_anchor=[1.5, .6])\n",
    "        plt.savefig('img/W1_x{}__per_epoch.png'.format(idx), dpi=100, bbox_inches='tight')\n",
    "        plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1D ---------------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "data = [X_valid, y_valid]\n",
    "# nr_examples = y_valid.shape[0]\n",
    "nr_examples = 3\n",
    "for ii in range(nr_examples):\n",
    "    xx, yy = data[0][ii, :], data[1][ii]\n",
    "    yy = np.array([yy])\n",
    "    plt.figure()\n",
    "    plot_grid(xx, vmin=bounds[0], vmax=bounds[1], cmap='Greys')\n",
    "    aa = predict(xx, params, activation_funs)[0]\n",
    "    y_hat = (aa > decision_boundary).astype(int)\n",
    "    cost = cost_fun(yy, aa)\n",
    "    plt.title('Validation Example {}\\n'\n",
    "              'y={}, a2={:.5f}, $\\hat{{y}}$={}, cost={:.5f}'.format(ii, yy[0], aa, y_hat, cost))\n",
    "    plt.savefig('img/predict_valid_{}.png'.format(ii), dpi=100, bbox_inches='tight')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "XX = np.array([2*iaml.data.LETTERMATS['T']-1,\n",
    "               2*iaml.data.LETTERMATS['C']-1,\n",
    "               np.random.randn(9),\n",
    "               np.random.randn(9),\n",
    "               np.random.randn(9),\n",
    "               10*np.random.randn(9)])\n",
    "yy = np.array([0, 1, 0, 1, 0, 1])\n",
    "names = ['No noise T', 'No noise C', 'N(0, 1) sample 1', 'N(0, 1) sample 2', 'N(0, 1) sample 3', 'N(0, 10) sample']\n",
    "data = [XX, yy]\n",
    "nr_examples = len(yy)\n",
    "for ii in range(nr_examples):\n",
    "    xx, yy = data[0][ii, :], data[1][ii]\n",
    "    yy = np.array([yy])\n",
    "    plt.figure()\n",
    "    plot_grid(xx, vmin=bounds[0], vmax=bounds[1], cmap='Greys')\n",
    "    aa = predict(xx, params, activation_funs)[0]\n",
    "    y_hat = (aa > decision_boundary).astype(int)\n",
    "    cost = cost_fun(yy, aa)\n",
    "    plt.title('{}\\n'\n",
    "              'y={}, a2={:.5f}, $\\hat{{y}}$={}, cost={:.5f}'.format(names[ii], yy[0], aa, y_hat, cost))# \n",
    "    plt.savefig('img/predict_valid_{}.png'.format(names[ii]), dpi=100, bbox_inches='tight')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### ------------ Below here is some extra stuff, I didn't ask you for it ------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### A quick forward pass test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here I test a few random parameter initialisations to see if the forward pass is working. **I'm not doing any learning here**, I'm just randomly picking values from a normal distribution for all the weights. The reason I'm doing this is to see if the cost function values look sensible, and to check if my code works!\n",
    "\n",
    "Ultimately, as you'll see, I end up concluding that learning is going to be pretty easy for any initialisation we start on!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "activation_funs = [linear, sigmoid]\n",
    "cost_fun = crossentropy_cost\n",
    "costs = []\n",
    "param_list = []\n",
    "for ii in range(50):\n",
    "    np.random.seed(ii)\n",
    "    W1 = np.random.randn(3, 9)\n",
    "    W2 = np.random.randn(1, 3)\n",
    "    params = [W1, W2]\n",
    "    a2 = predict(X_train, params, activation_funs)\n",
    "    cost = cost_fun(y_train, a2)\n",
    "    costs.append(cost)\n",
    "    param_list.append(params)\n",
    "plt.figure()\n",
    "plt.plot(costs, '.')\n",
    "plt.xlabel('Random initialisation nr')\n",
    "plt.ylabel('Crossentropy cost')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color=\"red\">Hmm...that's interesting. We seem to have some random weight initialisations with really low costs! Let's check out the predictions we get using the best performing parameters from our random search of weights...</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "idx_min = np.argmin(costs)\n",
    "params_min = param_list[idx_min]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for ii in range(3):\n",
    "    plt.figure()\n",
    "    plot_grid(X_train[ii], vmin=bounds[0], vmax=bounds[1], cmap='Greys')\n",
    "    aa2 = predict(X_train[ii], params_min, activation_funs)\n",
    "    plt.title('y = {}, a2 = {:.5f}'.format(y_train[ii], aa2[0]))\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color=\"red\">Hey that looks pretty good! I wonder if it's that good for all the letters. Let's check out the worst cases.</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a2 = predict(X_train, params_min, activation_funs)\n",
    "losses = (-y_train * np.log(a2)) - ((1-y_train) * np.log(1-a2))  # crossentropy for each data item\n",
    "losses = losses.flatten()\n",
    "plt.figure()\n",
    "plt.plot(losses.flatten(), '.')\n",
    "plt.xlabel('Data item')\n",
    "plt.ylabel('Cost for data item')\n",
    "plt.title('Cost for each data item using best params')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_order = np.argsort(losses.flatten())[::-1]\n",
    "print('Worst predictions:')\n",
    "for ii in range(3):\n",
    "    idx = loss_order[ii]\n",
    "    xx, yy = X_train[idx, :], y_train[idx]\n",
    "    plt.figure()\n",
    "    plot_grid(xx, vmin=bounds[0], vmax=bounds[1], cmap='Greys')\n",
    "    aa2 = predict(xx, params_min, activation_funs)\n",
    "    plt.title('y={}, a2={:.5f} loss={:.5f}'.format(yy, aa2[0], losses[idx]))\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color=\"red\">Not bad! I reckon the accuracy will be 100% since a decision boundary at .5 (as is normal) would result in the correct classification for the very worst cases...let's check just in case!</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "decision_boundary = .5\n",
    "y_hat = (a2 > decision_boundary).astype(int)\n",
    "correct_predictions = np.sum(y_train == y_hat)\n",
    "accuracy = correct_predictions / len(y_train)\n",
    "print('Training Accuracy = {}'.format(accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font color=\"red\">The predictions look good - losses appear to be small and we have 100% accuracy! Apparently we have found a good parameter setting from a random search...so this problem *should* be easy to optimise! I don't expect you to have done this quick check, but it's a good example for how checking your code can lead to some informative results in itself. \n",
    "<br/>\n",
    "<br/>\n",
    "The above check turned out to be invaluable for me: my solution contained bugs in the backprop function (my cost function derivative was wrong) and, because I had a good idea about reasonable cost values from this analysis, I was able to better diagnose my issues.</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Backward pass tests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_vals = np.linspace(-10, 10, 100)[:, np.newaxis]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sum([check_grad(linear, linear_derivative, xx) for xx in test_vals])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sum([check_grad(sigmoid, sigmoid_derivative, xx) for xx in test_vals])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_vals = np.linspace(0.01, .99, 100)[:, np.newaxis]\n",
    "c_1 = lambda x: crossentropy_cost(np.array([1]), x)\n",
    "c_0 = lambda x: crossentropy_cost(np.array([0]), x)\n",
    "c_deriv1 = lambda x: crossentropy_cost_derivative(np.array([1]), x)\n",
    "c_deriv0 = lambda x: crossentropy_cost_derivative(np.array([0]), x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sum([check_grad(c_1, c_deriv1, xx) for xx in test_vals])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sum([check_grad(c_0, c_deriv0, xx) for xx in test_vals])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 3.2 =========="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Did you need a network this large to do this classification task? Give the values for the parameters of a network with no hidden layers, one output node, and an output activation function of a sigmoid that would get 100% accuracy. This network only has 9 parameters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n",
    "The 'C' character data should be close to -1 in the centre square, whereas the 'T' data is close to 1. This means that all we need to do is return that value (passed through a sigmoid) to get 100% accuracy! Any weight vector with a positive value in index 4 (the $5^{th}$ entry) and zeros everywhere else will produce the desired result. \n",
    "\n",
    "In fact, there don't even need to be zeros everywhere else, so long as the sum is always >.5 for T and <.5 for C."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 3.3 =========="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should recognise the model described in question 3.2. What is it?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n",
    "It's a logistic regression!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 3.4 =========="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why did I create input data, `X`, that was between [-1, 1] i.e. why wasn't it between [0, 1] like normal?! Would the model specified in question 3.1 above have worked if `X` was in [0, 1]? Explain why or why not.\n",
    "\n",
    "*Hint: if you're stuck, you can try it out by generating some new data and trying to fit it.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n",
    "No, it wouldn't have worked. The reason is that, regardless of how negative the learned parameters are, if you input a 0, multiplying it will never make it negative. Therefore, as the calculations are passed forward to the final sigmoid activation, the output activations will always be >= 0.5. The cost would still probably push us in the correct direction but your network would get very frustrated trying to predict outputs of `0` (`T` data).\n",
    "\n",
    "This could be solved by including a bias parameter at each layer. Now you can make your hidden layer activation negative despite the input being 0. To simplify things, I chose to change the data instead!!! Below is some code that shows empirically that it doesn't work:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bounds = [0, 1]\n",
    "X_, y_, y_labels_ = load_letters(categories=['T', 'C'], \n",
    "                              num_obs=[25, 25],\n",
    "                              bounds=bounds,\n",
    "                              beta_params=[[1, 8], [8, 1]],\n",
    "                              shuffle=True, \n",
    "                              random_state=42)\n",
    "\n",
    "X_tr, X_val, y_tr, y_val = train_test_split(\n",
    "    X_, y_, test_size=0.33, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "np.random.seed(random_seed)\n",
    "W1 = np.random.randn(3, 9)\n",
    "W2 = np.random.randn(1, 3)\n",
    "params2 = [W1, W2]\n",
    "\n",
    "# Things to store\n",
    "cost_per_epoch = dict(train=[], valid=[])\n",
    "accuracy_per_epoch = dict(train=[], valid=[])\n",
    "params2_per_epoch = []\n",
    "\n",
    "# Training loop\n",
    "print('Beginning Training!\\n{}\\n'.format('-'*len('Beginning Training!')))\n",
    "print(' {:^10} | {:^10} | {:^10} | {:^10} | {:^10} '.\\\n",
    "      format('Epoch', 'Train Cost', 'Valid Cost', 'Train Acc', 'Valid Acc'))\n",
    "print(' {0}   {0}   {0}   {0}   {0} '.format(10*'-'))\n",
    "\n",
    "for ii in range(nr_epochs):\n",
    "    # WARNING: \n",
    "    # If your network is large and/or you are doing many epochs, it's not \n",
    "    # sensible to save the parameters every epoch - for this toy problem it's ok\n",
    "    params2_per_epoch.append(copy.deepcopy(params2))  # without deepcopy, python will\n",
    "                                                    # use a pointer only, resulting\n",
    "                                                    # in a final parameter list with\n",
    "                                                    # nr_epochs identical sets of params!\n",
    "    # Forward pass\n",
    "    fwd_pass_data = predict(X_tr, params2, activation_funs, return_all=True)\n",
    "    activations = fwd_pass_data[0]\n",
    "\n",
    "    a_out = dict(train=activations[-1],  # this is just to avoid calculating train output activations twice\n",
    "                 valid=predict(X_valid, params2, activation_funs, return_all=False))\n",
    "    for dataset_name, (XX, yy) in dict(train=(X_tr, y_tr), \n",
    "                                       valid=(X_val, y_val)).iteritems():\n",
    "        aa = a_out[dataset_name]\n",
    "        accuracy_per_epoch[dataset_name].append(get_accuracy(yy, aa, decision_boundary))\n",
    "        cost_per_epoch[dataset_name].append(cost_fun(yy, aa))\n",
    "    \n",
    "    print(' {:>10d} | {:<10.7f} | {:<10.7f} | {:<10.7f} | {:<10.7f} '.\\\n",
    "          format(ii, cost_per_epoch['train'][ii], cost_per_epoch['valid'][ii], \n",
    "                 accuracy_per_epoch['train'][ii], accuracy_per_epoch['valid'][ii]))\n",
    "    \n",
    "    # Don't do backward pass on the final epoch (just to avoid repeating code) \n",
    "    if ii == nr_epochs - 1:\n",
    "        break\n",
    "    \n",
    "    # Backward pass\n",
    "    params2 = backward_pass(y_train, params2, fwd_pass_data, cost_fun_derivative, \n",
    "                           activation_funs_derivatives, learning_rate=learning_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 3.5 [EXTENSION] =========="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a dataset which makes the problem harder. Have a look at the dataset generation code. You can use the arguments to create data with:\n",
    "* more letters (make the problem a multiclass classification)\n",
    "    * You'll need to implement the multiclass version of the sigmoid for the output activation function - [the softmax](https://en.wikipedia.org/wiki/Softmax_function) (and of course it's derivative) \n",
    "* increase the noise on the data\n",
    "\n",
    "Some other things you could implement:\n",
    "* include rotated letters in the data\n",
    "* make larger data (bigger than 3x3)\n",
    "* make the letters non-centred e.g. 5x5 data with 3x3 letters in 1 of 9 different places\n",
    "\n",
    "You'll probably need to adapt the code you wrote in 3.1, but you can probably copy and paste most of it. For an additional challenge: introduce [bias parameters](http://neuralnetworksanddeeplearning.com/chap1.html) and create your `X` data in range [0, 1] (i.e. set the bounds argument to [0, 1])...\n",
    "\n",
    "Some other things to try if you get code happy:\n",
    "* Implement stochastic gradient descent updates (updating parameters every training example, as opposed to every epoch) - tip: randomise data order each epoch\n",
    "* Implement batch gradient descent updates - tip: randomise data order each epoch\n",
    "\n",
    "**Requirements**:\n",
    "1. Describe the modelling problem and your input data. Plot some examples of the data\n",
    "1. Write down the model specification (I should be able to reproduce your model with this description):\n",
    "    * number of nodes in each layer\n",
    "    * a description of the parameters to learn (and a total number of parameters)\n",
    "    * the activation functions used for each layer\n",
    "    * cost function used\n",
    "1. All the outputs asked for in Question 3.1: loss per epoch plot, final parameters, a weight against epoch plot, and example predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your code goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 4 - Implementation with Sklearn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 4.1 =========="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you did Question 3.1, this should be a breeze! Use the same data and perform the same modelling task. This time you can use Sklearn's Neural Network object [MLPClassifier](http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html#sklearn.neural_network.MLPClassifier).\n",
    "\n",
    "\n",
    "Before you begin, read the [introduction](http://scikit-learn.org/stable/modules/neural_networks_supervised.html) (sections 1.17.1 and 1.17.2 at a minimum, 1.17.5, 1.17.6, 1.17.7 are recommended)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "mlp = MLPClassifier(hidden_layer_sizes=(3, ), \n",
    "                    activation='identity',\n",
    "                    solver='sgd',\n",
    "                    alpha=0, \n",
    "                    batch_size=X_train.shape[0],\n",
    "                    learning_rate='constant',\n",
    "                    learning_rate_init=0.3, \n",
    "                    max_iter=100,\n",
    "                    shuffle=False, \n",
    "                    random_state=None,\n",
    "                    tol=0.0001, \n",
    "                    verbose=True, \n",
    "                    warm_start=False, \n",
    "                    momentum=0, \n",
    "                    nesterovs_momentum=False, \n",
    "                    early_stopping=False, \n",
    "                    validation_fraction=0)\n",
    "mlp.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ========== Question 4.2 =========="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The learned parameters are stored in the fitted sklearn [MLPClassifier](http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html#sklearn.neural_network.MLPClassifier) object **as two separate attributes**.\n",
    "\n",
    "1. Print the parameters learned by your fitted model\n",
    "1. Print the total number of parameters learned\n",
    "\n",
    "Look at the number of parameters described in question 3.1 (you do not need to have done this question 3.1 - just read its description). Below the code:\n",
    "\n",
    "1. Explain why the number of parameters learned by sklearn is different from the number specified in 3.1?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Your code goes here\n",
    "final_Ws = [mat.T for mat in mlp.coefs_]  # Just transposing so we can use same code as above\n",
    "final_bees = mlp.intercepts_\n",
    "\n",
    "for ii, W in enumerate(final_Ws):\n",
    "    plt.figure()\n",
    "    hinton(W.T)\n",
    "    plt.title('$W^{{({})}}$'.format(ii + 1))\n",
    "    plt.tight_layout()\n",
    "    plt.show()\n",
    "\n",
    "for ii, b in enumerate(final_bees):\n",
    "    plt.figure()\n",
    "    hinton(b[np.newaxis, :])\n",
    "    plt.title('$b^{{({})}}$'.format(ii + 1))\n",
    "    plt.tight_layout()\n",
    "    plt.show()\n",
    "    \n",
    "W1 = final_Ws[0]\n",
    "for ii in range(3):\n",
    "    plt.figure()\n",
    "    hinton(W1[ii, :].reshape(3, 3).T)\n",
    "    plt.title('$W^{{({})}}_{}$'.format(1, ii))\n",
    "    plt.savefig('img/hinton_W{}_{}.png'.format(1, ii), dpi=100, bbox_inches='tight')\n",
    "    plt.tight_layout()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Your answer goes here*\n",
    "\n",
    "The sklearn object will fit bias parameters by default (it calls them intercepts). We did not fit them in our code in 3.1. You can read more about the bias parameter and how to fit it [here](http://neuralnetworksanddeeplearning.com/chap1.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [Please sir...I want some more](https://www.youtube.com/watch?v=Ex2r86G0sdc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well done, you successfully covered the basics of Neural Networks!\n",
    "\n",
    "If you enjoyed this lab, you'll love another course @ Edinburgh: [Machine Learning Practical](https://github.com/CSTR-Edinburgh/mlpractical). Check it out."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Next steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first thing to do, if you haven't already, is do the extension question 3.5. **In particular, you should implement bias parameters in your model code**.\n",
    "\n",
    "Next, go back to the very top of the notebook where I detail things I will not cover. Pick some words you don't understand (perhaps along with the keyword 'example' or 'introduction') and have fun reading/watching some tutorials about them online. Code up what you have learned; if you can code it up without peeking, you know you have understood it very well indeed. Another good \"starter for 10\" google is \"a review of neural networks for [images|text|music|bat detection|captioning images|generation|...]\".\n",
    "\n",
    "Here are some things that you might find fun to read:\n",
    "* [Visualising networks learning](http://playground.tensorflow.org/#activation=tanh&batchSize=10&dataset=circle&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=5&networkShape=3&seed=0.42978&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&hideText=false)\n",
    "* [Trying to understand what features are learned by Deep Nets](https://distill.pub/2017/feature-visualization/)\n",
    "* [Modelling sound waves](https://deepmind.com/blog/wavenet-generative-model-raw-audio/)\n",
    "   * ...and using that to [encode instruments](https://magenta.tensorflow.org/nsynth)\n",
    "* An [Introduction to LSTMs](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) and their [unreasonable effectiveness](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)\n",
    "* How to encode the entire meaning of a word [in a few numbers](http://colah.github.io/posts/2014-07-NLP-RNNs-Representations/)\n",
    "* [Convolutions for text data?!](http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning resources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also:\n",
    "* [there](http://ufldl.stanford.edu/tutorial/supervised/MultiLayerNeuralNetworks/)\n",
    "* [are](http://neuralnetworksanddeeplearning.com/chap1.html)\n",
    "* [literally](https://www.coursera.org/learn/machine-learning)\n",
    "* [so](https://www.coursera.org/learn/neural-networks)\n",
    "* [many](http://deeplearning.net/)\n",
    "* [learning](http://datasciencemasters.org/)\n",
    "* [resources](https://metacademy.org/graphs/concepts/backpropagation)\n",
    "* [online!](http://www.deeplearningbook.org/)\n",
    "\n",
    "(about neural nets etc.)\n",
    "\n",
    "In all seriousness, make sure you check out [metacademy](https://metacademy.org/). You can search for a topic and it gives you a list of free resources, an estimated time you need to understand it, and prerequisite topics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Attributions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Parts of this lab were inspired by  D. E. Rumelhart, G. E. Hinton, and R. J. Williams, Parallel distributed processing: Explorations\n",
    "in the microstructure of cognition, vol. 1, MIT Press, Cambridge, MA, USA, 1986,\n",
    "pp. 318–362.\n",
    "\n",
    "\n",
    "Thanks also to:\n",
    "* [3 Blue 1 Brown](https://www.youtube.com/channel/UCYO_jab_esuFRV4b17AJtAw)\n",
    "* [Michael Nielsen](http://neuralnetworksanddeeplearning.com)\n",
    "* [Christopher Olah](http://colah.github.io/)\n",
    "\n",
    "for producing some excellent visualisations and learning resources and providing them free of charge.\n",
    "\n",
    "Additionally, many thanks to the developers of open source software, in particular:\n",
    "* [Numpy](http://www.numpy.org/)\n",
    "* [Scipy](https://www.scipy.org/)\n",
    "* [Sklearn](http://scikit-learn.org/stable/)\n",
    "* [Matplotlib](https://matplotlib.org/)\n",
    "* [Jupyter](http://jupyter.org/)\n",
    "* and of course [Python](https://www.python.org/) itself!\n",
    "\n",
    "your work is invaluable and appreciated."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Credits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This lab was created by [James Owers](https://jamesowers.github.io/) in November 2017 and reviewed by [Patric Fulop](https://www.inf.ed.ac.uk/people/students/Patric_Fulop.html)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}