1b_build_tensorflow.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Practical 1b: Build Your Own TF.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/deep-learning-indaba/indaba-pracs-2019/blob/master/1b_build_tensorflow.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mEB4b33cMG-V",
        "colab_type": "text"
      },
      "source": [
        "# Practical 1b: Build your Own Tensorflow\n",
        "\n",
        "© Deep Learning Indaba. Apache License 2.0."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WOxz_rxOMNr_",
        "colab_type": "text"
      },
      "source": [
        "## Introduction"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SBNjby3_MPtI",
        "colab_type": "text"
      },
      "source": [
        "In this practical, we will build a small Python framework that allows us to train our own neural networks, like Tensorflow does. Our framework will depend only on `numpy`. By working through this practical you'll get a chance to understand in more detail how modern deep learning frameworks work, and the basic idea behind Automatic Differentiation, which is a powerful software technique that allows us to quickly and easily compute gradients for all kinds of numerical programs."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SuEFKaqiqe3q",
        "colab_type": "text"
      },
      "source": [
        "$$ \n",
        "\\newcommand{\\vec}[1]{\\mathbf{#1}}\n",
        "\\newcommand{\\vechat}[1]{\\hat{\\mathbf{#1}}}\n",
        "\\newcommand{\\x}{\\vec{x}}\n",
        "\\newcommand{\\utheta}{θ}\n",
        "\\newcommand{\\th}{\\vec{\\utheta}}\n",
        "\\newcommand{\\y}{\\vec{y}}\n",
        "\\newcommand{\\b}{\\vec{b}}\n",
        "\\newcommand{\\W}{\\textrm{W}}\n",
        "\\newcommand{\\L}{\\mathcal{L}}\n",
        "\\newcommand{\\xhat}{\\vechat{x}}\n",
        "\\newcommand{\\yhat}{\\vechat{y}}\n",
        "\\newcommand{\\bhat}{\\vechat{b}}\n",
        "\\newcommand{\\What}{\\hat{\\W}}\n",
        "\\newcommand{\\partialfrac}[2]{\\frac{\\partial{#1}}{\\partial{#2}}}\n",
        "\\newcommand{\\ipartialfrac}[2]{{\\partial{#1}}/{\\partial{#2}}}\n",
        "\\newcommand{\\dydx}{\\partialfrac{\\y}{\\x}}\n",
        "\\newcommand{\\dld}[1]{\\partialfrac{\\L}{#1}}\n",
        "\\newcommand{\\dldx}{\\dld{\\x}}\n",
        "\\newcommand{\\dldy}{\\dld{\\y}}\n",
        "\\newcommand{\\dldw}{\\dld{W}}\n",
        "\\newcommand{\\idld}[1]{\\ipartialfrac{\\L}{#1}}\n",
        "\\newcommand{\\idldx}{\\idld{\\x}}\n",
        "\\newcommand{\\idldy}{\\idld{\\y}}\n",
        "\\newcommand{\\idydx}{\\ipartialfrac{\\y}{\\x}}\n",
        "\\newcommand{\\red}[1]{\\color{red}{#1}}\n",
        "\\newcommand{\\green}[1]{\\color{green}{#1}}\n",
        "\\newcommand{\\blue}[1]{\\color{blue}{#1}}\n",
        "\\newcommand{\\because}[1]{&& \\triangleright \\textrm{#1}}\n",
        "\\newcommand{\\relu}[1]{\\textrm{relu}({#1})}\n",
        "\\newcommand{\\step}[1]{\\textrm{step}({#1})}\n",
        "\\newcommand{\\gap}{\\hspace{0.5mm}}\n",
        "\\newcommand{\\gapp}{\\hspace{1mm}}\n",
        "\\newcommand{\\ngap}{\\hspace{-0.5mm}}\n",
        "\\newcommand{\\ngapp}{\\hspace{-1mm}}\n",
        "$$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rnyKdnXqv2pr",
        "colab_type": "text"
      },
      "source": [
        "## Learning Objectives"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ECHxDVDGv5ks",
        "colab_type": "text"
      },
      "source": [
        "* Understand the the term **automatic differentiation**.\n",
        "* Understand **forward** and **backward passes** and their application.\n",
        "* Understand the **vector-Jacobian products** involved in backpropagation.\n",
        "* Implement all the steps involved in training a simple **multi-layer perceptron (MLP)** to solve the MNIST digit prediction task.\n",
        "* Optionally, understand how to mathematically **derive the gradient calculations** for the backward pass."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b6uKH8Cnqqgw",
        "colab_type": "text"
      },
      "source": [
        "## Background"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TLJgrNIqURqO",
        "colab_type": "text"
      },
      "source": [
        "This practical will show how to create your *own* deep learning framework from scratch, using only `numpy`. This sounds like a difficult task, but the core ideas are actually suprisingly simple. Understanding how they work will advance your mathematical intuition and put you in a better position to follow (or create) the deep learning innovations of tomorrow! "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b-gp40kWV6xy",
        "colab_type": "text"
      },
      "source": [
        "Practically all the popular deep learning frameworks (Tensorflow, PyTorch, etc) are based on an algorithm called reverse-mode automatic differentiation (**RAD**). To understand RAD, let's start with a computer program that computes the output of numerical function $y= f(x;\\theta)$ when given the input $x$ and parameters $\\theta$. RAD gives us a recipe to efficiently compute the **gradient** of the output $y$ with respect to its parameters $\\theta$ for this input $x$. Don't worry if you aren't familiar with what a gradient is.\n",
        "\n",
        "> *Note:* The numerical functions that we will be optimising are usually referred to \"neural networks\", but there isn't really a fundamental distinction between these two concepts. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8eLpOOmVjXXF",
        "colab_type": "text"
      },
      "source": [
        "More precisely, these gradients tell us how we should make **small** changes to the network's parameters $\\theta$ in order to optimize the final output of the network on some input data $x$. Essentially, if we use the output of our network to calculate a *loss* (which we'll write as $\\mathcal{L}$) that measures how badly our network performs on a task, then changing the parameters $\\theta$ of the network to reduce $\\mathcal{L}$ is the same as making the network perform the task better! This is known as **training** a model. If we base these parameter updates on the gradient of the loss wrt the model parameters, and take small steps along the gradient based on randomly selected sets of examples, then we are performing **stochastic gradient descent (SGD)**, which is the primary method used to train neural networks.\n",
        "\n",
        "Throughout this practical, you should try to keep these equivalances in mind:\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VCsbf2y6V6s2",
        "colab_type": "text"
      },
      "source": [
        "<center>stochastic \n",
        "  gradient descent of $\\L$<br>$\\Updownarrow$<br>\n",
        "  changing parameters $\\theta$ to reduce $\\L$ on batches of examples\n",
        " <br>$\\Updownarrow$<br>\n",
        "  training our net to perform a task by showing it examples</center>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RHnMMzmUV6qU",
        "colab_type": "text"
      },
      "source": [
        "So just what is reverse-mode automatic differentiation (RAD)? The core idea is fairly simple: it is a recipe for computing gradients of a large, complex function composed of simpler functions by computing gradients for the simpler functions that it is composed of. \n",
        "\n",
        "> *Note*: the reason we use the term \"reverse-mode\" is that the order in which we compute the gradients is reversed from the order we compute the output, but this will become more obvious later. \n",
        "\n",
        "> *Note*: We think of these smaller functions as taking vectors (often written as $\\mathbf x$) as inputs and producing vectors (often written $\\mathbf y$) as outputs. Sometimes $\\mathbf x$ and $\\mathbf y$ are arrays with more dimensions, like images, but we'll ignore this for now (if you're curious, we effectively treat them as vectors for the purposes of automatic differentiation)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4eZ0hZHRysx0",
        "colab_type": "text"
      },
      "source": [
        "### What are gradient vectors?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XK4syJWp0SgP",
        "colab_type": "text"
      },
      "source": [
        "> *Note: feel free to skip this if you already understand what a gradient vector represents.*\n",
        "\n",
        "Let's start with a scalar function $f$, which maps the input $x$ to the output $y$: $$y = \\green{f(x)}$$\n",
        "\n",
        "Here, $f$ could be a polynomial, exponential, or whatever your favorite kind of mathematical function is. \n",
        "\n",
        "Let's now  _approximate_ the function $\\green f$ around a particular point $x_0$ with a straight-line function $\\blue {d\\ngap f}$:\n",
        "\n",
        "$$\\blue{d\\ngap f(x)} = f(x_0) + \\red{f'(x_0)}(x - x_0)$$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6n0es69z0Lrs",
        "colab_type": "text"
      },
      "source": [
        "<center><img src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAArgAAAH0CAMAAADhbK6XAAACuFBMVEUAAAD///+/Hi7DLDvBNUPHOkjLSFXPVmLTZG/Ub3nXcnzbgInejpbinKPmqrDquL3uxsry1Nf24uT68PG7L0LNpavPvL+3P1XQx8qoeZmmgaOhkrcUFBliYmRYWFp3d3lsbG6WlpiBgYPi4uTz8/Tw8PHl5ebKysu1tbarq6yhoaKMjI3a2+vIyuHg4e7j5O+jp82Pk7SkqMuprdCssNHLzeHR0+Tm5/Gdoso7PUyTmL2gpcuLj6uvs9OyttS1ude4vNi7v9q3us/N0OTs7fXBxd3T1ufa3OnS1N7y8/jv8PXHydL4+fvZ2tzW19nT1NbR0tROT1BbXF3q6+zn6Onf4OHc3d7k6u/D1uTS3+kLWI0PdbwGLEcEHS8be74efcAthsQ8jshAjMJCj8RLl8xOlcVaoNFpqNV4sdl5rM+Gud2FsdGVwuGTutWky+Wjw9mz0+nC3O7g7fbe5erR5fISebVQV1vP1trv9voUfa4jhrJBl7oXgacahaCMwc/F4Odhq7sciZhKnKY6mqArk5M0NTXt7u48nZkhkYoklYPI5N8nmXy739QpnXWTzblEqoQ4pXhgt5MsoW4AplEwtnFQwYdgx5KP17K/6NPf8+nv+fQQq1sgsWYvpWdAvHxwzZ1/0qef3b2v48jP7t5JsnW+480xqV/I0Mtlv4Wl2rbY4ds0rVg/sWON0aB0yInD1Mc2sVFAs1pCtVxPuWeFxI+Yx6CiyamhwKY1qkU5tUoynkFFuVVMuVtRvmBewmxfvG1qx3eD0I58woaP1Zmb2aTA58bm9egZTyALIg53zIJefWKo3q+047rN7NHZ8dzg8OLy+vP8/Pz5+fn29vb09PTv7+/q6urf39/V1dXPz8/AwMC/v7+vr6+fn5+Pj4+Hh4d/f39wcHBgYGBQUFBAQEAwMDAgICAQEBAewcC9AAAACXBIWXMAADAkAAAwJAELuPDgAAAgAElEQVR4nO2d+X8c513Hd3aRZGTZli3KHZGUND3SlkCBmqOE0gNSckEOjoGuFhZJLEJoxQY20AIFWlrKUbKKLymybEM4CiXcCCGxQiuBkFbISVzXlR0UH9G/wWvumX1m5pnZfWbmeWY/75+s2dGx9ttffZ/v9zvPk5EAEBCIC4QE4gIhgbhASCAuEBKIC4QE4gIhgbhASCAuEBKIC4QE4gIhgbhASCAuEBKIC4QE4gIhgbhASCAuEBKIC4QE4gIhgbhASCAuEBKIC4QE4gIhgbhASCAuEBKIC4QE4gIhgbhASCAuEBKIC4QE4gIhgbhASCAuEBKI271Mjni89bHRKnGNNyBu9zBVmrQLOSmXPd56tVDg3lyIm2Kmi0VZNvSsFPLjxXzFfLeT8qTnO6/mi8Q1zoC4KWZ6vJiX5Wn1DVYLcrkyIptCVuRRnzdelkvENb6AuOmmKOuJ7IScl8ZlS9xC3jcbGPPMIzgB4qabvDymvb+CPCqVZHlKf7clSkidlgvENa6AuKmmLOuZ7LSsmFqe1t9sNe8fcJWQ650C8wDETTUlI8Wdkh2/+yeNQOxJmfOQC3FTjZnimgZrFMycwZMRueL1Eg9A3FQjG5G1KOdt73Pa8ZE7Y5QsOGEgbkqZLpenrRRXysv2yuyE7KzTlstqHqF+jsEk37kCxE0llaJcKMqTWoIwIRtM6O911BFNyyMjxXy+Ik3KxaKV+1Zkmee/GYibRiblfFlRL6+muNVyeUKWS+Vy2agkOFLcKcXnaTk/ka9IY7KV2cpcl3IhbgqZ1PUbN1PcCdmx1LI7OZ1Xs4miklVUZNsrI1wXxCBu+ijLeiZgpbhjjt/70/YSw5iW7haU9VrVngoXuV6dQdz0MSLLWk4wYQpacKzGynaN9RCrBeeqLS5DXBArk2aCMGpUca2ymIpD3NbgbAJxQawUTAdNXctWQUFqFVejJBP9BogL4qQqG5mCFUUnnRUCF3GLZEsC4oI4KctG58Bq8447TXURVybHcyEuiJOyrc07IknVKfVPji5YlcgLzFxiyhoag7ggTsximJbiToy6xFOit2CkuNO2OtkIfRAnQSBu2qgY4k6qKW5xwnbJoGCt1UrySFVd0Gkf2apmRFjmCoibOkY0Syt5Ja6qIXSqNcKOyeP6n6bVXpmRF1fy1n0uiTBPQNzUManG0HJ+XFFyvKglAs7nHazJr4qibLUwllcEr+THrXsmyOUaT0Dc9DEuF0rF/KSi55j6PHqxdUKxapk8Jo+VRsakyfxIyTmBO+oo/XIHxE0h06XShCJmuVRS11p5ebzlTY5afbJyqaTkB9MTpQn7MxJSXqY8lZYsEDf9TMtEfWCSmgfQn0pLFoibbsqlqjRFdsWkEccjaC4UOd9YAeKmmkmlgjtOZApKocE/oJb5XppB3JRTVJoQ+bxLdC36V2mLbp/DExA31YzJxWqJGFiU1C1B/J6FnHD9HJ6AuKmmWswX8u4OVvLeyUKFkkhwAMRNOZWyV1VrMu9VqPVzmhcgbvdS8Vp/lXjfYxTiAlGBuEBIIC4QEogLhATiAiGBuKlmgv9zn9oE4qaYSqH1mZ30AHHTS0ndXNR3JEFcIG5aqRa0TXHJybBUAHFTy6iirWdbV3QgbmqZzstykfPhxPaBuOllwmMuLBVA3BST1lKYBHGBqEBcICQQN1VMc37mOTsgbpqYoJ4tnRogbnqoFGWX/ZlTCsRNDVqHl9i0JqVA3JRQ0Tu8hZTOJrQCcVNBdVzTNrUdXgKImwbKI5q36e3wEkDcFDDZbeEW4qYDZZxGlke7pRKmAnHTwIQs57ukmmAAcVNBYayrwi3EBaICcYGQQFwgJBBXVCa6Lat1AnHFpFJI6+O7AYG4IqJ1eLtm9tYNiCsgeod3pJv/DiCucHTfQI0bEFc0pvSBmu7q8BJAXLGojurhtss6vAQQVyzymrdd1+ElgLhioT6fM9LV9QQNiCsYBVke7/pwC3HFo4xwqwJxgZBAXCAkEBcICcTln3JaDyDpBIjLO0qHt0s2+QgDxOUctcNb6Pa/BRKIyzVGh7er52lcgbg8M6V3eLt8oMYNiMsv00VNW7QcXIC43DKhh1t0eN2AuLwyiXDrB8Tllao6MI4SrgcQl1vK6dul+RxxhfaCJxCXX0oswu1u0wbxarxcmD3r8f1mF4hLFCBu6thtNrcajc16fWXRlXq9vtbYisPiswsLF2yx9ELtAnGLzqXZU8Q1fyBuemiuN9a8bHWlvtlY34nu7S/U5hdmZkxzL9Z8wurF2mnimi8QNwXsNrfW6ktuagZhZXUrkuh7sTYnna+ZUfbs7Dxxi43ztTPENT8gLk9Ux8KuxnbWG+0ra2NlbXuD+OKdMVdbkGqWuPM1rwRX5dzsXKgVGsTliKm8XAz+02ysN+qEgBpL9fpqo9FQV2X2z1A+3mo0Vuuusi+vbu8S36VtztRqZ6SZmpEqXKxRstgLfpkECcTlhunR4OM0O1uby4R4i4srdcXWYPbtNLcaZEa80mCV9C7Uauck6ZLx4Zx/wFVC7myYkAtxeUHv8NIP4226BNql+tpWs63f9TvbrbnG8hoTd+drc7aPztR8M1xJzXLDrM8gLh8EHKjZ2SKkXVltBIyx3uxsr66wdnfWoeop71KYwRmH6TQgLheU9IGakk+83djebMlMl1nWA3bX1+zZx/JWZ4u1szVHzjpLyxQC3mMCcTkgwDm8zbWWdLTeeZwl2djatH2LzXXihuBcrNUuWje3RtOLC6cVSc+eXrDdNB8mV4C4HGCEW4+fZHd71RFqlza3ImwbrNu+2XKjvf8c8zUD/cJC7Xnby5fm5hbma+elC7VT52etDoXzJgoQlwOmfM7h3WjJaiOVVmd91fp+q+1kDJfOnJmp1c6cOWMUFZ635w2XZk+rZYbztUtKh8IMsxdqM8RX8gTi8sCoxy7NG1uOBGFlLa4xmd1t67/LZlvfdMaxNpuxZQHn5tQ/n1eT4JpN3DNmgA4AxOWB6qhLuHVau8S0OxCAjTUzZai3oW6tdt7xkdXQXdDS3edrylpsvjZvpgqX1MpvQCAunzitXY4t1DrYXmlb3TM1R/3LLu6c9sIcmRjUQswrQFwO2XVYu9JhYaoTmmaVoR7uhzjtlND+0Wk1rJ5zhmTiLhoQlzd2tx0lKeazLyHZMBdqoZZpp5z5Kqmks1zmdZc3EDchJkZdv2/TXvnajDmtdcdUdylEcWzemQiQSp53SWjJu7yBuImgdHjJOsJGw9a7qnNhrYqp7nLglkTNOQxGKjlDprhSrXaJuOYFxE0CtcPbOk5jq0Almte6YaobMNW9VHN2wWZb84KzRop70XIV5TDOMTq89sN4d9Zs/ao1vqxVadaNfIF4yYULLSF2vnXY9oKR4s5aRkNcvinJRIfXqjstLq0m/SyuF9t6HrMSoHXXmsGeskoImtOn9Bsu2GYYTtNHHy0gbszo5/DaOry2Sj9PiS3JbkP/KelBd75lqMbm5KySRZzVxxjO2QKu3W46EDdWyHN4bZntcoPDFMHBRj1g0J1tmZc5a2UBs7W5c+dmnp9RGhSXZuxLuBmyQOYNxI2VSWe43bWVEToaIoyNLe23w9KW7zc8W2udUJwzc94Ltfnzc/Pnzs7UTjkz33MuBTJvIG68FG3n8O6sChRsDXb19simX05zkShsLVhpwNmLC+qLZxYuOAbHL9Aep3QAceNlOm/s0myfvxIi2Bps60HXZxG5QDyFc7Y2S9zVwjxR6/UD4sbMpBpubTnCkjDB1sDIdF3ThbMLpxUHiWUW9aGzs2FqChA3ETasvu7KtohvoOGdLswp1a4a+fTY2VnKo5DPh9vKBuLGTtNKbbmt2dLYWfaqLtRqtUuu2epp/yfKzpBB2heIGzPrdXFzBBv6Gm2JyM5naxfPzc24lQfmPTcZVR+LcP0cbyBu5FQKU+a32DZT22UhcwQbW+6J7qW5mu2hBjvnZnzcfH42+HyNCsSNmpI5TrO7ZWrbzsMwvNHUEvXVwD/WuRlyIEznVFhvIW7ElNWBmjG1kGCuyNp6cpY/NrQJi7rLEs2dc6e8kgXPFzyBuFFidninLW2FTm2d7GrLzJUk5isgboSYAzW/bxYSwjxFIABr2psiiwuRA3Ejozqmh9vfMLUVfkVGsJ2UuRA3KoxzeH/9C+nV1lyixW8uxI0Kbd/QZz5nFhKIkmc62NGS97j/U0LcqKgo3n76pRTVvzzYWUnCXIgbGSX52RfSr61SXEjCXIgbFRurn32pG7RNyFyIGw1W3Tb12iZjLsSNgu7SNhFzIS57uk5by9z43i/EZUh1fKRqmwDrHm1Nc+Or50Jcdigd3nFT25Vu0jZ+cyEuK/QO7wsp7pL5o5sb0wgRxGWE0eH9TJdqa5ob06wYxGVCdVTT9pnnAu8Ll0K07m885kJcFkzq4Vbp8KZscDEUmrnBn4noAIjbOcY5vGqHd7V7tVXMDfk0T/tA3M6pWOF2MzVPN7TJdlyNCIjbOTu/I8vyx17sssKtB1sxNSIgbqdsrC6+9KxSTAh+QEKqWY2nnAtxO0Nr7774Im3jzS6iHktpAeJ2hNkn6+JSQitaOXeTuM4WiNsBxoEeadkogRFaUSziYjbEbZMfe2LXPEIJazIn6+pfS7QpP8Rti8u/9NHfXuraqQQqjeinFiBuO/zyz8my/GIKN/hgxWbkCzSIG56Xf/GjSsPhY0huPdEWaFF20JiJm8AuPAnxEz+rd3g/320jtyHQFmgRZlGsxN2OZ7QieV75hY/qHd6/+6enu+Mtt8V2xGkuI3G34xoKSpof/xlN24+98J9//vjjT3TDW24TteayEtmXZyNufENByXLvz+vn8H6m/r+PP/7444+9nPq33DZamrsW1ZdnFHFXu8Hcy/f98+9q4fZP1yXpScXcp4ibgIGW5ka1DGCV46bV3Guv7e/vX1f/eO/f/ocyTiM/81mtBPaIYu5biM8ABlqaG1FNjFlVQTPX7eQrgdm7nVE52Jcu/9U/Km/wOfnjf6GvOF5Rk4XLqXrDbNmMcGiBXR1X25w6kW3Vo2I/Y3LnX/5da5T9sdXJRLJAYXc5upoYwwbEdtrMtXmbyfz3vxJTYJcfVcy9l/g8YNCMribGsnOWMnP3Mg7+Sxmncf4b3KuI+yiSBW/UX8N1z5c7gGnLN7EDASLhtlPczL8tEb/07lPMvS8dbzca1JpYFDP2bGcV0mTu1RZvM/9D/ipBskBjJ6pkgfGQjX6UBRGaBOR6q7i3Xd7DW9RkgbgMTBoRJQusp8MSOsoiAvaDiCs9pZj7JHEZmESULDAfa9QPhI+s1RcbwcS9/Jhi7ivEdWAQUbLAfh5X3+JX+B1dAqUKerLwCHEZmESTLEQwSL5bT0VZ7Fdbxd0nblF5GskChZUonkCL5AkIrf27LHRx4TdHfrpF3KvEPSovP4YxMX92ophZiObRnYZWFhN3a5fdtZee+Umnt7eIm3SeUEIuZsp9WItg1RPRM2daQTeSynMcrC8r4zQ/Zff24Irn91WTBcyUe6PNLLAdcIzqYUm9LCbkEm1XnWpa/Hj+Vyxvv/F7HybuM0CyQKPJ/mmIyJ7y3dCKCyviPQa7pW+Y8IWqdOMN3dvv+u6THyBuNHjlUSQLFDaZ//6N7vF0PW4tCfYk7I6+rdKy9nNf/9FPfvJH/u9bT548efKDxL0ql9WBBYRcXzaWWK/PotxXYU3fD454gV92G/q2SmvGX7I6jPCDirjvc/2p731U8/ZpeOvHFusHZCLdEGR7SbDHIv7gHzRtbRsmaFM036+Y65IsmOH2U8RLwMEK4/VZtDvZ6P1fQSq6G78lf2Kx9dQcTdyPfM/JkycfeKj1M97ymObtU5jJpcF6fRbxFkx6oitEXeyPfk2W5edap8X1ucXvVELug85PuPyUHm7xxGQAVtl6EPneYXrSWOc9Xfiz39MOKvv7lq6JMXD7fYq5H7a/8ik93H4zwm0Qdtmuz6Lf9E4b0eW8jbbxJ89qO30Uqy2vGOJ+RBH3AauY+/LTmraPYow8IA2m/bMYdmvUh26shTp37P7NpzVt8xPEj2Y+4vAdirnvNy4/oYfb+xBuA7PMsrAQyzajW47SKHdsfU4Pt6PT5I9minv52xVz361efEUPt48g3IagyXLv9nj2x9WrC1yWdNeXjXA7RbxmF1d6ryLuex42NlTAMGNoWGaLMW3sbJyXwF3QVc4f+awWbluzWw3b05DvUsx9l/TKI3q4xXMPCRLbjuTrSxxmuhva/6ePy/JImXhRwybuw+9RzP0hvQaGcbBEiW8rfaOky88BjBvGsTl/LY+7h1unuNK3KOJ+Gzq8PBDnGRBG0OXjpGZzLGGxseuyKDNwbJzwfsXcH0C45YBYDy8xgu5S8os07SjTRfr5Iw5xH35AMfeH0eFNnphP3Vlf5mKRZmlbp41ROLeqebci7rcTN4HYifu4qN01wxj/QBfpz6Bo+4eLwc6EbNlj6UGi8wsSIf5zznYSPgBXXZK98Kz8wmKg055axH3oAdcxMRA3SRzQp4/pJnIq446i7UtKy+HZvyRedKN1V7sPu4yJgfhJ5GRJM1+IW911Ndq/oHV4x4mX3SC2Y3yf1fkFyZHQkagbm/Gru7utrgxf+oRfh5eAEPeDLWNiIBESO8u3WY9X3Y01LUF57hnZr8NLQIgrfcAxJgaSIcFDqI3SWBzLtG39v8nn9XA7EizcuoorqZ3f9xJ3gjhJ9PT0bVPdepR94J01o2r7oh5uvTu8BC7iWmNiIDESPvbfUnd5K5qMYWNrZdH6HgU13HoN1LjhIq4xJgYSJGFxjYW+ljEwD7u725vmV1/cXJekiizLpeDh1kPch5EsJE7i4kpS05JreY3hg+wOa5cbWh5dKlSIG31xE1fr/LpvEALigQNxrRU/Q3c3tqxIrgXbNnEVVxsTc9kgBMQFF+Iq0XHFFh07zBl219eWbdaudJQ9u4urjYmh85scnIjrWPurZYat9gLv7vqa7b+AlSK0jbu46PwmDT/iKgu1Vbtzi/VGM1SsbG6tOqRdXFHTjnLIrNaJh7gYE0sYrsRtWVCpEXOzsU6PvTvNRouzSsxWY211VB4JVUZowUtcjIklC2fiSmrcXW5xcHG5vtrYajZbDd5pNrcaq3XidiVL1mP1VF4pgBHfIzhe4vp3fr/ocdIJYAWH4ipGbm0SMloJQL3u4qpN2m0zrZ0e1TplPs+U0fAU13dM7MvDXyauAZbwKa6kyeujpzv1tXX7WmwiH7rDS+Atru+YGMyNGH7FVdhoNjZbc1d3luuN7ZZMYrqoD9SE6fASeIurJQsfIi5r3DN8D3ENsINvcTV2mo1Gvb7kIWy90SCSX4WSHm7DdXgJfMT16/y++o5hmBslIohrstNUaDQaykqt2Wz6lWj17DZsh5fAR1yfMbG33j08fNdbicuAGUKJG4ayFm47/jp+4kofcu/8vnr/8PDwO14lrgN2pFZcaVyWix1UEwx8xdU6v63nSL3truHhu79E3AxYkl5xqwVyl+Y28BXXbUzs6puHh4ffTtwJ2JJecRnhL659TOx+Nad9+/Dw8Ju/SNwHGANxKVDEtXV+337XF6UvKYuytxF3AeZAXAoUcW1jYq/e/U33DA8P349FWRykStxqsD0+QkET1zYm9qVhLMpiI03iTuQZlL9aoYr7kNX5vX/4HoTbmEiPuFqHl0EBzAlVXNuY2Kt3YUAhLlIjrj5QUyRe6BC6uLYxsS/fjYgbEykRt1Jg1OElCCDue61k4c1o88ZEOsQtyTKjDi9BAHFtG4Qg4MZFGsQt6+G2yDzcBhTXb0wMREMKxJ32PIeXBUHE1cfEiMsgOtIQcdUJRhYDNW4EEhcbhMROGsSt5uX8JHGVEcHEdR8TA9GRisXZVNBdmtsgmLhuY2IgSjCrQCGguNggJGYgLoWg4mKDkHiBuBSCiovdxOJFVHEnIluNtRBYXK3z+07iMogEMcWtFOR8ROWvVoKL67tBCGCMkOKWIhmncSe4uP67iQG2CCiuMVDT0QY1gQkhLs6RihHhxK2OR9rhJQgjrvcGIYA1oolbHtEHamJKcUOJi3Ok4kMscc1wG1dNIaS4GBOLDbHEHQt3Di8LQomLzm9siCXudD74seeMCCcuxsTiQrAcd0KWx2IMt+HFxZhYTIi2OBuLpwhmEVJc6Z3o/MYCZhUohBUXY2LxAHEphBYXY2KxAHEphBYXnd9Y4F/cSpzFL5Lw4vqeIwUYwbu4SsthlLgaI22IizGxGOBcXK3DG2vhtoU2xPU/RwowgWtxjQ5vgXglPtoRF53f6OFZ3Kl8/B1egnbExZhY9PArblU/qSzeDi9BW+JiTCxyuBV3Ug+3MXd4CdoTF53fqOFV3Ok8g3N4WdCeuBgTixpuI+5Ep8eeM6JNcTEmFjH85rgFuZB4uO1AXHR+o4VfcSsR7NLcBu2Kiw1CogWzChTaFhdjYpECcSm0L+5D6PxGCMSl0L64GBOLEp7ErY5HcDJkp3QgLsbEIoQjcadG4tqdJgydiIsxsejgRlytwzuSfOG2hU7ERec3OngR1xioiW+nj4B0JC7GxCKDD3G1c3h56PASdCQuzpGKDC7E1c/h5aHDS9CZuOj8RgUH4nIcbjsXF2NiEcGBuKXIzuFlQYfiYkwsInhIFQqRHHvOiE7FRec3GngQtyznOQ23LMTFmFgkcLE4m4xrl+Y26FhcjIlFAmYVKHQurpYs4BwptkBcCgzERec3AiAuhTbFHTxh+wBjYuxJSNypPK9VhFbaFHcgOzBkfYTOL3MSEVdpOSS5O00Y2hY3mztqfoQNQpiThLhah5ffCpiD9sXNZnsHjQ8xJsaaBMTVd6gZIV7gkk7EzWb79VQXY2KsSUDcCZ47vARf+6Y3fXX74mZzeqqrdn6Vei7KYmxIIlUoctzhbWGo3/krPzCGuNlszzH1c96vm4vtR9mQhLjTI6KE26Fe3b6jxEsULHGz2b7jxvrsJFJdVqCO60efKd8Jn7vcsIubzfYPWRH35EnMLTAA4vpw3FKvpy8cPQ5xs7kBy1uMirEA4vpwJMuOr/p6U1w8EMEAiOvDAENxs9mvg7gMgbg+sIy4X/MNZsTFFiEMgLg+nLC8OzQQjj6ntj2DHzC9fcD7G4LAQFw/+g3xckM+d7nhyDJyA5L08PsQcFkCcf0w6ri54z43uWIXt1+1/uEHtXiL1hkTIK4/A1+RzWa/MmwV1y5unyn9u9/14Ic+jBExNkBcCp0N2RgNX8AaiEuhI3FzA2FzYxAQiEuhE3H7w6cYICAQl0L74rYxUwYCA3EptCtuLvRAGQgDxKXQprjHkdxGC8SlwGBfBRABEJcCxOUTiEsB4vIJxKUAcfkE4lKAuHwCcSlAXD6BuBSiEvd43wBxDQQH4lKIStzebBadtQ6AuBQiEvdE6yPvRwPONRyD7ioQl0JE4h7NZvvtH/fnAop7NPzmJKkE4lJgKu5AT7bniPqnQ85MoT/4MxYwVwXiUmApbl/2yFCf6t1QNttne2EgG2LePITkKQbiUmAo7kC2V3n8UqkmHMvaVT2ePUTc7M1QTw8meCAuDYbi9mQPK7uRHVUfH+6xvdAbNMHVOJpFJQ3i0mAn7nHF2WNa+TZnT1SPZg8TN/vSE3oPvvQBcSmwE/dINmskp8ezOdsLoT084qxIdCUQlwI7cfuzWeOPh+2/7I9le4l7/RkKv0FJ6oC4FNiJ22sVEnrs4vVnjxD3UuhDSQziUmAnbtZMZYccv+pz2Tb2yen6XAHiUmAj7uDAQH822zcwoG20YH8izZnvSoOHcj3KY+1HenN9nmF10Pk53QjEpcBG3AF1j/Levr4+Yh12xNmKyA0MHsvlTvT3Hjve45kQDIXf2z9tQFwKzFKFw9barOW67df+UbX1ezjbmxsyehWuYLQM4lJgJm6fR/Ggz6bnUE7tpw2oyvZlvVdtfd5OdwkQlwIzcbMefV27gwNa1nBYzQSOH9K9Pdbf1zdwwvOTuhOIS4GVuCe8fvHbm2j6cqzX3hAe6sseHjzWUgCDuBCXAitxj3mlpS7XHbO6h7WEoc9xG8SFuBRYiTuQzbp3u0hxB+0nWZ7Qo++gI0WGuBCXAitx+xzzYDZ6iSXYgL3YNWCkxll7nwLiQlwKrMTNOcq1NkgHHSmu+bKjxkB+UrcBcSkwEnfIa21GOjhkpLiHj6jC6y/32u/r9exNdAsQlwIjcQezXk/nDNjKZEeUjvAxPcXVirqm8H32FRuZGHcbEJcCI3EHPJu0R61V12HV0n7dygE1YbCJa+UaJ7xWet0DxKXASNxDnmMxJ6xWsNorO57NqcH5uKavq7jhR3hTB8SlwEjcHq+1mfKS8Wv/ULZPGswNHM32njgxoGexNnGtB3wOh33YJ31AXAqMxPVcm9ktHOrPqfsuHOvNZg/pNvdYVQXrK/SGH+FNGxCXAqN53Ky3ase9Krwqh4w1mW11d8L3M7oDiEuBjbhH/FTr8ysRHNVzDPt67HDXF8MgLpWOxR3M5QalQ35J6aB3/qsUxbRyhO1pnaEcdgSBuDQ6FrdHSU5zXsUwlUP+Ibd3SJKO56znK0Nt2JRWIC6FjsXNZXMnjngvzSQ1hPpVt4715Pr7stYzP+E2bEorEJdCx+Iezh4+mqM8lDvol0lI0uDgoBWwh3q7PlGQIC6dzhdnRw95P65r3hN8uYXNGlUgLoWYDi856l0uc3K4B95KEJdOXKfunAj4+z/ofWkH4lLAcVF8AnEpQFw+gbgUIC6fQFwKEJdPIC4FiMsnEJcCxOUTiEsB4vIJxKUAcfkE4lKAuHwCcSlAXD6BuBQgLp9AXAoQl08gLgWIyycQlwLE5ROISwHi8gnEpQBx+QTiUoC4fAJxKUBcPoG4FCAun0BcChCXTyAuBYjLJxCXAsTlE4hLAeLyCcSlAHH5BOJSgLh8Av4NOPwAAAMBSURBVHEpQFw+gbgUIC6fQFwKEJdPIC4FiMsnEJcCxOUTiEsB4vIJxKUAcfkE4lKAuHwCcSlAXD6BuBQgLp9AXAoQl08gLgWIyycQlwLE5ROISwHi8gnEpQBx+QTiUoC4fAJxKTz55JNPvux/C0gAiAuEBOICIYG4QEggLhASiAuEBOICIYG4QEggLhASiAuEBOICIYG4QEggLhASiAuEBOICIYG4rlx5/fbtW3uStHfr9u39K253gGSBuG5cO7i1t3cnc/36wWt7tzI3Xe4ACQNxXbh6cEOSpOuZg4Mb0o1MBn9HHIJ/FBdu31Ku7WUydyRpP5O5Td4BkgbiklzLXJU0cV+TpCs3b2o57tX927df3yNuBskAcUn21YCrxNpr1ms3Mrev791EvssLENeT25kD66XrGdXm1zKve90OYgXienKguarxhh59D7Q0AiQNxPXimpri6lwzSgu3bRdBgkBcLxwp7mtKgUHhpj0Mg+SAuF4YKe6esh573aiJ7RsGg2SBuCR7+0qozeix9eZN1WJTXPyNcQH+GQj21JbDjUxmXzKLuhCXN/DPQLCvxNorbxyola8rd9T6F8TlDfwzENzIHFy7euvOtYODa9L1N7SOg03cA+ITQAJAXJL9NzKZm1eka7cymTduaK/uG+K+jskFPoC4gbhh1XHR9OUCiBuIq5mMNmpzJ3OD+x+2K4C4wbitCXsVKS4nQNxgXFWWatKVO5nrIvy0XQDEDci1O5lbNw/egLecAHEDc21vD3Pk3ABxgZBAXCAkEBcICcQFQgJxgZBAXCAkEBcICcQFQgJxgZBAXCAkEBcICcQFQgJxgZBAXCAkEBcICcQFQgJxgZBAXCAkEBcICcQFQgJxgZBAXCAkEBcICcQFQgJxgZBAXCAkEBcICcQFQgJxgZBAXCAkEBcICcQFQgJxgZBAXCAkEBcICcQFQgJxgZBAXCAkEBcICcQFQgJxgZBAXCAkEBcICcQFQgJxgZBAXCAkEBcICcQFQgJxgZBAXCAekiT9P/ngMusBXbenAAAAAElFTkSuQmCC\"\n",
        "height=\"250\"></center>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IgxZ4df33XZ3",
        "colab_type": "text"
      },
      "source": [
        "As the picture above shows, we can use the derivative $f' = \\frac{dy}{dx}$ to construct the best linear approximation to a function $f$ around a specific point $x_0$. Specifically, the derivative gives us the _slope_. If the function above was horizontal at $x_0$, you can see that the gradient $\\red{f'(x_0)}$ would be zero.\n",
        "\n",
        "Notice the _crucial_ fact that the gradient $\\red{f'(x_0)}$ points in the _direction_ in which $\\green{f(x)}$ _increases_ at $x_0$, and the magnitude tells us _how quickly_ it increases. Here, the gradient is positive, which tells us the function is increasing to the *right*, and the large magnitdue tells us it is increasing relatively fast."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8Kg42dIw26Eh",
        "colab_type": "text"
      },
      "source": [
        "**Exercise**: does the gradient *always* point in the direction of the nearest maximum of a function? If not, can you provide an example of a point $x$ on a simple function for which the gradient $f'(x)$ does not point toward the nearest maximum of $f$?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "86E1GNVPwIC7",
        "colab_type": "text"
      },
      "source": [
        "### Moving to higher dimensions\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b1ojWHK2wLi7",
        "colab_type": "text"
      },
      "source": [
        "We saw how to interpret the gradient of the output of a scalar function with respect to its input. In fact, a similar thing happens if we are working with a function whose input is a *vector* rather than a scalar. However, we must replace the multiplication between the *scalar* derivative $\\red{ f'(x_0) }$ and the *scalar* difference $(x - x_0)$, with a dot product between the *vector* gradient $\\red{\\nabla f(\\vec x_0)}$ and the _vector_ difference $(\\vec x - \\vec x_0)$:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rkqEe17x32eQ",
        "colab_type": "text"
      },
      "source": [
        "$$\\blue{d\\hspace{-0.3ex}f(\\x)} = f(\\mathbf x_0) + \\red{\\nabla f(\\mathbf x_0)}\\gap\\cdot \\gap (\\mathbf x - \\mathbf x_0)$$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r8WVIvRx01vJ",
        "colab_type": "text"
      },
      "source": [
        "<center><img src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAArgAAAH0CAMAAADhbK6XAAADAFBMVEUAAAD////GqKrCLjq+Hi68Jja0NkLIQE/RYm3hmKDtw8f03eGoU2OMi4yihrLm4/CalsavrNJ/fK2SjsWWksaem8pua6t3daSMisBubJRfXp+GhbiSksZeXoD6+v7Y2NlARIh9gbhQU2tLWZNAUIx0grJCR1hrfaxgdqUSExVKbqJEaZpRcJtUeKJihq1ffJtCaI5HbZLy9vpagqZiiq5ehqhLaYJRcIlZeZQ5i8lWhqpLc5FiiqpehKGfxuQ4PUEOdr4efsJGcpJijq6Bttzb6vRWoNFejapijqpDYHI7aYNBcIxGdpJLepVHdI1Of5pUgp1XhqFci6Zikq692+xGepY0TlxikqofJytGepJYiqJekqpCd44qNTlARkdLUVL2+vrd4eHKzMzAwsK4urqvsbF0dXXs7e2nqKiYmZlmj4nu9vRWW1ml4MPH7NoFp1RoyZeE1KyfoaAptGtGv4Dh8+jR1NIeezslikIvi0pka2Y1lE0aMCArlkQ1m00xo0Y2o0o8m091oH02sko2rko1nEY0lEUzjEEzhEA6sko2o0Y6rko6qko6pkk6okovezo6kkdFn1I+qko6nkY+pko6mkY+oko6lkYtajVAiUlGkk8+tko+sko8rEY+rko3l0E6kkI+mkY6jkJLplRSnVl+p4KQuJSkxac+okY+nkZQjVS60rxEtko+jkJQslY+ikI3dztWqVpEd0Y8gz4tUS5Mgk5fmWBurG9KukpUulVxtHF+xn5+wX5rmms+Vz6Knor6/vr2+vbk5eSBgoFgs19rrWl6v3h2uXRtu2mCyn4pPChevVZouGN+xnl+wnmCxn53vnCCynqGxYB1wmxzvGp8xHKCxnqGyn5rwV2GynqGxnqKyn5zwmB9xWyDxnSGtnuGynSKynpIZz9+xmReiVCGvnKOynqMynVUeEaIyWySynpsmVZ1pl6SznaSynaSznKIv2p/smWSzm6Pym2SynKWznaRxXKWznKWzm7Wzs768vL+/v76+vr19fWSkpICAgLnkulhAAAACXBIWXMAADAkAAAwJAELuPDgAAAgAElEQVR4nOy9CVhUV5733zxv///zzsyb7h41rcZ2YqMhiUsAlQIXAoJaUotVlFIjFODrbtzXRIl2Ok6zxEhIQCMx4gZGJUZBEbUgdEeqAAGBsUCRWNUsBmRQo4GkSEcz73PuVufec+6tlTX1O88JRVFUVL5+/dzv+Z1zf+XhLncNwHIL110DstzCddeALLdw3TUgyy1cdw3IcgvXXQOy3MJ114Ast3DdNSDLLVx3DchyC9ddA7LcwnXXgCy3cN01IMstXHcNyHIL110DstzCddeALLdw3TUgyy1cdw3IcgvXXQOy3MJ114Ast3DdNSDLLVx3DchyC9ddA7LcwnXXgCy3cN01IMstXHcNyHIL110DstzCddeALLdw3TUgyy1cdw3IcgvXXQOy3MJ114Ast3DdNSDLLVx3DchyC9ddA7LcwnXXgCy3cN01IMstXHcNyHIL110DstzC/UXWnNk8v+uFryNP9c9yC/eXWHNmzeX5XfN/pZ+VW7iDtF6fPXvutIXU7232rODguZOZ3+icWdN4f9MLgweGct3CHZw1JxgUJdzZwbM9ZlsEOXnarMn8v+nX6e/q3+UW7qCtucHB5G9tYXDw2teDgxmXnS0szWmz5iDP9b9yC3fQ1jTaY+cGT/OYHRxMX4/NsQIDc4L5rtz6U7mFO1hrMiPVWcFzPSbPfp3Gg2nChguEbuUF/aHcwh2s9TqNuHOCg+GMa2Ew/5UZ/QphS+4X5RbuYK3ZNOIyCiZrdrDVqHZWcP+nXLdwB2tNo52VUTBZNqjSBm33ebmFO0hrLYO404JnQb/FhazP8PW6VZro+3ILd9DVwmnBwXPnLCQAYfLcudOCg2dNmzuXWYtgZQaTqaWJyXODg5mXACru938obuEOspozLXjuHI/Zs6YR4pszdxoQ7txpjCqnwcKdDJ5/PXjanFmzJ78ebGGI4P6fK7iFO7hqzixSmMBByd/YwmAW1LIQdy7xmmnBoEFhFvS6af0fct3CHVQ1mb4kW8gIdzZbuLCZvk4u/c4lXhEcbFkxm9v/1yDcwh1UNZvW5UJGoOxrs8mwjKeRvjqLEPvrr0NfcAvXXb1Za4Nplc4ODl5LPprGWk5YCF93kYY7JxiR6ez+vwThFu5gqrlQBkYlWpPZslyIBgYL0Uux2f0/D3MLdzDVLFqDFrlyZIkR7tzgYG6Xo9tx3dWbtTYY6mSkcgHOtdkcVKVzUXud6xauu3qxLFGCxUXnciwW4YK1KOK6UwV39WrNgRB3rofHwrkY8wzmtiowPTjTLIp2C9ddvVkLaeHOIax39lxCqGzhQknXnLmzSUsm8oeF0H6eWcjlWr8rt3AHU1HLZh7TCBedtRCGXaqgXgVirWzyLIolplleNxmx5f5XbuEOpiJjrMlzCeG+PgttxoU7vxYGA33Onkbi8FzoCs2WDrK+LrdwB1NNnhY8e+HCWXMnzw2evXDW62gzLuGm9MNZwbPnzJ41Zzb4nmnQ7nWP1wfArjO3cAdXzZ42F/ybP3n2tLlUsw032LJsOQM9j7PneHi8Pm3utNlrWS9xN5K7q2+Li7jATq1ltHMGACm4hTuIazJ5pALnNzh5ltBxIB4DZOeOW7iDtojOXJy9vm5Fl5NnTbOi7P5QbuEO1poLzq7BnqEgeALTADFct3BdUU+f9cNf1Nzg4IX4ExKEj7LBf0+/K7dwXVDLFgUVL+lvv6g5s+bO5vlHXwgWJk/Df09/K7dwXVErh4/0jF35Yv/6RU2e/TqfPufyr4zNFuaIflNu4bqilnoOGzpspKfnytXXBsYveDZf4PX6NF5J969yC9clVTxkGKjhoz1XvtHPjBdffLbK93y/K7dwXVI/ev1h2NBhYA4B2l06CH5L/bzcwnVNxZGWS9So0V5xP785GH5X/bjcwnVNLYkZNhQew71in1vWH1OywVJu4bqofh41jFNDvIKeczNDT5VbuC6qVTEk47Lm0NExQSvcvtsT5Rauq2rlEK7lkjUyJmjFqsHxW+xP5Rauq+ofi9iUC42RQTE/u7Xr2nIL12X183DEbS01ZHRQ3LJB8hvtF+UWrsvq2SKEcVlz+GivlW7tuqrcwnVdbRmJGC3iuwNlZa3fl1u4rqunQUNRvuUOoF13SuZ8uYXrwnrOquWSK2sjvVYWL306aH7bfVJu4bqwnnmNwLAtbo4Y6RW30r2y5kS5hevKsk65cI0JWrncnZI5WG7hurS8RqBUKzBGeMWMdmvXoXIL16W1ZTTiq9ZqZFD5zwOk/bw/lVu4Lq1VcaOwTCs8h3sFPecOyewrt3BdWw5YLlFDxsT0tz1r/bvcwnVtPfUcIsC0gmPI6KDFWwbTn0WPllu4Lq4Vnoib2l6jRnqtWz2o/jh6rNzCdXE9XWlrloufYN9PsXtlzWq5hevqWuaM5RI1YqTnymL36oRwuYXr8lpsX5aLHWBlrdid8AqUW7gur6VeiIc6VMM941aucCe8POUWrutr3QgefrV/jg6KGyiH4/RyuYXr+nrTacqFa3TQYrd20XILtwcq1gWUC48hXouK3QeMsMst3B6oJUGIbzpbQzxjVv446P6gnCi3cHui1o1yEePCc5TnonXuPWt0uYVrX21ff+rUqc0lWqLKcnNPn0k7k3bmzMdgZOpytdpNWq325mcxL7804g8ut12QksUWuwNeD7dwbaqnL57668k1+VlZKSkpSYkJCQcTwIxPOJAQD8bB+IQDByjtgvHB6TMf54WN9fX1Hes78ZWJr7z88ksuZN5RIOF1X6y5hStUfzt16rPLuW/vSNqxOzGB0OnBY4mJKUkHDpxJS0sDE9QHaR+kpR04EJ8WH58QHx8fn5a2//SZGF9OvTLp5ZdfGop4qCM16gWvoJX/EPh1/xLKLVxMbdhwavPJy/mJiYk7EhMPHj1w/MzxM2doTz2ddjAhMTHt49Mfn/54/2l4JIP5wQd798bvjd+l9vXx9fH1DSGGpSYC/S4YgWNY2+eQoUNH/tJXJ9zCZdXaN9dv1urfBTxw8EBCQsKBNEqt5KC1+vGxxKNp+0+nJpMzY//+5IxkemYkZ+xPPUpYrpQYhHZnQHPixFdeeemlESMcRl3imLJf9sqaW7h0bf/81MnLWQmACQ4cP5B2/GBCwgHIZwG7fnya8tfk0/sTEhM/ID02eT8Yp4n/JqcSMzU5ddc42m2lvj7EpD4P8Q2Z4EtM4L6jRiAUa8MYQn3XcM/Ro3+hB4y4hQuuvt489dmJxMSE4weBYtOOg6QgYTfjtB9bFEsMyltPJ8Xvz9ifuj8jNTU5Y1/q/ox9xEd6xswICfGFJ61d5jmKIohrN/tqKHQy5Egvr1/i0f2/eOFeW/pZflJCwoGDQLXHgWoBzx7YfQBKCT5mqRZ47GnCV9N2p6UmU2Nf6r7UVOK/+1JTM/alpuapQ3yRQbov8rzvxJde+oM9nDtkFPz5SK+gn39p2v0lC/en7Z9vPpSQACv2Y4JpTx9NIJwWvvIiWNbCsMBn92WkJiRkpJLjSOqR1Pcz3k99n/54VO07MSQENwntzmC+NoOYM3wjXraDerknQw4ZE/vzL2p14pcq3J+WnMpOSjx48MDR4weOE0x7hhiEx+5MO3P6A4hn4UkzLOWzyTuSU1NTM1LfZ+b7GUfIx3lqxFmh4ePjg3HekFdesuH8MXAiwxDkuVGjFy3+5Wj3FyncVacuf3ICsAHNtGcA1QK3BUz78W7Ia5msIBXwbDIxSYbdR3lt4l7gr2C+f+T9I0csc2+Mb8iMkBD+CaiX/Dr4QEzwWP6KLcw7BPeaESMXxa74ZRxK9osT7qpTX2aDbJZkWtJr084cZ9Ztz6Tt/uBjyGsJpk1NTt0PsWwyybGUx6bF73s/9UgGOfbtfSf+nfj43Qm7E3anx/ggRaS7kFJ9pVIf+jGUQkx8+SWEa9lz6BD0OWIO7383Z+2J+kUJ96cln+WfOHGQ0mwam2upBOGDBJppMyxeS/As6bEMyzLzyN7d8Xvj4xPjwX/3/iV5396/7Mt4//CRw3vzQkJCJkITCJVSpo8vqWLwvNRXCr5Opr7UeoWPr2/AK4LIO4T/q8M9vVYO9jbIX45wn35+MvvEsYPHDhw8cPQA4bXHKa5lstoPTn+ctpvFtckQ0xJ5gYVlj2RkEM4aH7/36O74jIwjzDh8hBrp6hCEbeEBlAoseZyPj49EKkWI12fiC39E6JYZv0eegcbwMYP86P5fhnCfbgBUe+w4xbUgQyDJluZaIq09vf/0B/Ew1+7fDzEt4baExyYng5aE+Hfik/dmHDny/uH338uI3/se+AjU+h4xiMrYFSLEuPSUSgn/BQW/nqSHiRNf4rHWUcid1Ti+Ozpu5ZZBCw2/AOE+XXLq8iHgtQcPHgdsa0kR0iwrY2Ramxa/n+Ha/RauJZkWjL17Cc3u3bsv472MI5a5Lz7tvfcpn4UqJSaEViBf+fpOkNK8S6mXSBss3Osz8ZWXsGc1EGfmYJ63zCGenqPfGJz954NduE+XnDqZfezgwaPHGbIlFJuGWxc7HZ8M8lraayGuzUh9n9JsRvKR9w8feY+ewGHfBzM+mfTZT+GZERMiD5GFyHlEGwL4lq1pX18f0n+Bbik+9p3hM/GVUajvjrBiuUT9ZrRn3F8H4erEoBbuv19bnZ90jPDaA0fJxDYNw7V0hvBBPM22rKw2NTVtb3xCYsJeS3bAoVni4zt7D2cAPGDme5++92mKGpEqS7YTkKdCQsi+Bov3krw7zucVpK9huI19vsO9Yp5bPshSskEs3Gef7yFUi3LtGW63F8G1yfHJ5JpYMtN3sC8j9TS4AksDbEtmtATLkjwL0SyoPyfTjz49/Ol71Hg1JCQiZAYxQ6iPlin1hT9n+a4UvJpIHnwp5vX1mRDyMifhFbqzGrvAfstlg0m7g1W4z5Z+lvThh1iuhd2WWdFNTt7/JxbXpqYmp+6N352YkEzmCITDWrgW5dlPD8fvO3wYaPlTaOwgLFdGDE75IF7L8l3KjX1J9YZQXTozXoJ7eUfZs7cN7LccPIfjDErhPn3xs08Yr4Xc9uMzrBThY6gPITkj/jSLa5PjE5OSU5msFlx5WdiW8loLzxJKPfzOPkqxUO2SykIiQixTTs0QqeUxflr4d6Kvj88EwLxSH9+JIQEvW3azYdfP+GvoyJjYQXJTbFcLd8N65Cmy1m9Anuqhevb5pRMnjn1x8BhYH2O5LcS2H0NsS8z4DyxUuzd+R0LaPiKvJa7L2GRLc62FZxmHjd/36XucsSOG8Fp6WgwV8VhcSX1pCvYlr9iIqzbfSQvo/oRRKNFaGcNjggbD0f0uFu76jXz6XL+RT9IuradLv/zixMFjx48dxGS2eLYlMtsP/kxxbXJiQvyfk6l1sQy694Di2ve5XHv4MNtj4//yKbd2iUJgzyV41tcX4V2+GSJlUt0QXybrDZn48ghHLJdCY8/Fcct746fRg+WkcNdv3bh1E/SpgDqFvuaq+sdfs8gUwRrbfgyxLZHZxhNe+8Hu3fEf7LOsjbG5lkoR4OL6a8Y7lsdUnY6RhbAH0CL3OXrMQG1XynJnsM5G5ryT/viHYUOHDkdY1qY5ZLTXwD6kwTnhbt647attGxnlbt+4GXmJpTZt3I4858r6aemTT44dO34QZdszMNue4bAtmPvfSc5I3ZuQcDQZrI4xvV6WvPbwe1Rm+x4rp+Xw7Keffpqx9zD7ibOf7pIwfiuTg0HoNsLKlLN42BfOg8HVGpHxKl7+wzBMvmtjjRrtNYCP7ndKuBs2bly7eePGbdSnazduXYu8xlJrt24U+rKTteRU/vkTlNsKsW0ah22J3q8P4uMTj8YngyyB7kXA5bVHkJyWPUjlclkhMYZ0U7qkMpmMxb1Ck6kJvrDrgjUKwne9XxpuhWmFxqjRnqN/HpgpmVPC3bRxm8emjYzjbrICA+sFDdmZerrhs+xEIkfAsy2nI4HcgcP0I+yNT4xPS03OIPRqYVsW13LqU5zbkrV376dnPz1LeC01dklkclkENaNDZOCjFb9F5wwi26W415foMiMzXv+XHfbcYYTvesY9NwChwSnhAs1u2LaN8tHtG7cir2DX1p6Bha+WnfzkxAmUbY/TbJuGJrd0r23qB/EJCbuTk1P3ZVh6v2C2RfJaDNmyNHx2bwalW3oejWHcVhYtpR6QHoyf2BFCsDF9mUZwL7nGNnbGy86d0zBipOfo4oG2suaMcDdsZFmsNcMFlrsJec7pWnUqG6iWn20Rt7Ww7emjiUc/SI0HiHCE8VvAtgzfonntYTSrtfAsMeLPfsoeb0nkJN3KZFL6kS0jQsb13miCj4Ff+0rplMxXNDHE+4+IldpXQz1jilcMpNUJZ4S7mSXctRutIqwNL7Gzftq++uKJD2m3hXttabql+r/YKw4k36Ylkrt0/wyEy8kSILbF5LV4r6Xr8G5axdTHxBjGbyNlETJ62lhsBp5A264vvfLmIxoLVipecuiEBnh4Llq3esBo1xnhbtq4EfpsPXORxl/bXByJXfsMMMKHLLY9QO0w581tyez2g92J8WSv7Tsk3Vr2jeEzWx6VsryWGofjz5L1KfUxT0LwLfDbSBl7Rtg4Lb5Ls26ID8G7ISG+InClFiJ/zfnzIUfGxK1egvwx98dyRrjbWFC7yYZLr82uZIWf3sy/eOIYoVt7c9vTCYlH6Q6wP+8le23hLIHgWzbdok4L6ZjNtGfPvp9MqZaaR2OAx8pYfktOO4ukX6mUZF7ac8GlGqHdQGzfrn1zeEzMQFhZc1S42zdv3rxx49bNmzfTS2UbWYtm2zdt27Ztu4fHhk3btlkEvYHl0U4VkC1Q7XGoJyHNNrZN252QvI/eR5Ycn5F6hNo/9pe97xwFYwdZb721Y8eO3e/sjd/7zjvvZPC6Lcdvifrz4Stnz569QhnulbNvqe3lW/4BfBd0OsgI5RL5rohkB9+QkFdccSrvEK/+f2NhR4W7fuu2rRs3bt26bSv1jz9bkxs2btuwfuvG9Zs3bt6w1WKzaze6KFd4ujwuEbDthwzbwkkCnNsySQKV26btTkyD9zXs3pGSkv5W+ls7srOzsnJ1elA1ZDU1GWoMerJ0WVnZWYd2vAXEnLgbkS6i4L2HWZ67Nw/rtw54Lp1ISKVSGeO5M0TkGhvoI5v0R5Re7R9gZa1f35zVCVTYwPLYDTA3bCDkvH7j1q0bPNZvhOB3K28vgz31dMWl9Jycizk5KRcvIkkCN7dlKAHkCPGJCafJfWTvpOxIf+ut7OysB1qt6XbT7aY7TV833WlqYk2mWunRaqqtBSomFLybw7ZnLXXkz2dZlSfB8C09ac7FPcc3wXpaNKFcwLsiKcW7Pj7SGZOcCnaZGuXVn+NdJ4S7iZURbIaFu43w2A0bwXOb2cJ1fg1i1epLSUkfAro9djDxYlJKEuO2BzC5LbROtjcxIX7//lTgsCnZWXr9rYY7DXcavm74uqEJHZxqRUajVp+b9SHQ72GGbSHt7j3CEu7pRajXghmJec5mTwZhrg/BDL4iS8Tr4+P7ykuuuO/EqCDkz77/lBPCZV+bbYbkuYFU9HoiLtuwadNX0Pc4K9xrf81OAkkC02977ERSIkO3BNti18mS4xNTdqQkfpKdrdffun375te3ydFEjq+bvuZqlSxIr60PWy2Trrayyqysd9/e8c7eTymepeZe+jEYZ/8CKJc1IiItg5dnrY1oabQEeK5MNJ/uayCz3Ul/dMHp5yPfQP74+005IdyNLOFCJOuxiXy4CSXarU7GCqs+u3DxBKHbLw7Se8kOHkhKTIToFrPD4XRiSvqFT7Lz9SW3G27fvN1AjjsNtxu+brjT0IR6Lqrdr6HJHq2tTa3GJ/lZHx7csfsdC9cejof8971PT+/i+Cx34LwWnUhJpWGE0Y5ldZT5+PhMesl50o3tv/GC48Jdu5Hlnhgv3YauAWNeZUf931P3Uki3Pc7e33AwKYkIE3DZ7dHElJScS1n6gts3mXH769u3LW6L8i2kWUq3razJFOnB1Girzc16d/fujAxSrBmpV2i/fe/s2SvpEpJfI7AD4V6hyeLeEKlEJosI8RdZ+n2pjOGVUc5GDEPWOfHT6tlyXLicBd9tqJduRJ9yhnGffX4p5QRNCQePAa89fpSm2wMpx1jZLfDb/aeP7khJ/+SSVnu7+no15bU3abflpVuc3+JHa2sbNaFq/OjtnbvfOXz2ytl9RyhueA9EDFcW8XltpI2sy8e90rBomSzEX8rZ10a6LoZd7ZieK5AfQz8px4W7mQ0CmxCVcpRNlOOO+9PS/HQL21LrZHB2ezHhOCu7PZr4Vs4nube+0ZZU3755u/r2beC0xLSbbnFs28r1W9bIfXv37r0ZeynH/U/ivylqrNcyg+LZSHkEM60QLj2kkmiZ3N+Xta+Ncl1nE4aY/rpDzXHhbmMvJmxCuACDuI4z7ptxOSnEOhnFtkTP7VFWdpuQxPjtAcC0+derb5Vqa65Xo34rwLc49eL5lkMN9MUaPduz3n4rcS/w3P+krtd2IR7LjMjIiMhI5Fkb/JgskShEKhPNh/oa6N0TPj6vONXDMHI08qPoH+W4cLeyexM2I60KNOJu2AQ/h5iwLbXky6QTSSc+JNcbjlvoFkoTzqR9fCAxDbjt0fT0C3qg1QKtvrr6NjlYfHsH9luB7BbhWxv9lhmVuUk73vlPynevpKgRl42MjIyUWyb6dcSVcewbIZNIZVKpCO5pYM5q8PEVPvXRSnn2081pjguXQ7DrIQNeD9aB19Iv2LYJ/iYHhPtsz8WLJ1LOnzh24iDac0v7bRo4TTzpaGJeTnZJ9fXq69W39NoCwmsRvxXgW9Rt8QPDtuDzR6z5qLWt7FFr7c4duw+Tnvsq1mfhifgsbuB8VyKVhYhEzLoa7Lq+Pj4vO8G5Qf2zYcxh4XIJdrtFuORa2XoqddgAAcPajRu/8rCzfjqVc+LEiaQTx1g9t+yuWzK7PZiYnp5/g3TYgsvaapJtbzJse5OmWzv4Fpvd2uS3j8BsNIGPxo927Dhy9sqVoxDlyiNxA/VYvsHlXLDLQiSV0X2/0L41wnVfEvZVgRpe7KhEerQcFu7mjZzVW8vnxFrZ2q2k467dCl2ObUBA2GqtvpQEdJvI9CWQ2e3xA5w9ZQnp6Vn66wX6QuC2JfqaQsprEb8FbHu7ockq37ay+RbNbhG/hSfwWzD11MfH+W/v2Ht213zCQbk+a7fnYrg3RCKTSUVSZmsbm3V9fcYKnLQrPPrnyq/Dwt3EbfSytDVu3rh1+/ZtW7dvBJ0K8O51+9saN+QT6w2JiWTPLeS3aXBvwsGU9CwtUOz1W/qq2yXa0ls027Lo1h6+beXlWxvYFsp2DbT/PmrPTdiljmAxLW7yOCzvoFlXKpJFiqKjQ5iMl826vj6v2HK4I6ZGreuP9w12WLhbuea53vLEZuC2az02bNu4cet69jfZhbif7wGUcOJDAhQs2S17T9nxlJQL2XpCtWAU6PWlBderGb9t4PItD93i/RY3MHzLZVto1pnAR2rUxEgkUhzbOuy5kPdKRDLZ6AhZiJQJeNm0C7IxDMNanyP7Iyw4LFz0Msv6Vkjr2ynhWvXZpYsniE7xxBNEdnsUTRPOJKYkQaqtvl1aq9XeopwWy7dNDvCt/WwLzcpHjx4y44E6QCqVRGPoFhp83iowANdGSqJl/vMB30Zb2JfFur6+fKebC9ZQz37YnOuocDegIa11DrBlkwRTy7Mvngd+e+xEYuJBzJ6y48fPJKbkXMqvLqhmdFui1RZcL9HeYvz2pov5FmXbVjzbMrO1sY7y2/9++N8P/7t8hneA/ziJBPFZpz03UiYLi44WAe8FOyrxrOszEb0/mvUxZDHy0+nzckC420EYuxk1z6/Qp9i1dqvtmcKbe1JOkOu7Hx5LgvaUMWlC2oHEtz7JL6m2uO31qlLtDaDXklJevm3qTb6l2NbQZnHchw/U4tDAwImiMIkUw7f0FPBWoREtkfvPJ5g3OsTSz0DuFaZZ12eiA6br2f96yh0Q7jYQGuBWEjZjnrPn65Za+9ccym2PnTi2A8oSmPtAHk1Jz64uAIPW7Q29llawtobw29ssvxVIb/F+iw6EbXHZLdtvwccnlNsSo0g5VSwOnRoYIBonika8FvZcBzw4OixCJCN7GaSsdgYW6xJrafZxblC/20HpmHDXb0fWycivCO0+x38Ppv72Zg5FCceOfXg88ZglSzhIuu3xpLdy9LDXVl+vqtHeqGbYtrTKGt/ylEv5lh51jRDldqhDxeQQTxSJZsgQwo0kC32eGgKOGxkhksp9abqNhnNeuIfB18fuUxj6Hyw4INz1G7et34o9JUz44svWo8N+zE45b/HbE0l0bwLdmZCYnpdfUF0A++2ty9pC6Aqtulp7HfHbHuBbVnb7CO+3wG0NrbTfPnr4qMg/lPBcMFUzwsb5z4e8lb0aLDQRtyWHZL5oPp3thkRD/bss1vX1tfHuEcwY3d/axBy5ONuwadtmvAY3bOW/Ptu01abtZn/bk3ORzMDITrBEmm+pLCEx71DpzQKW21bX6KuY/IscN/S8fGvj3jJUqXbyLeSyxlrosw61OBQainESkWh+JKYw7GudhWUSmT+peuKcMlb/Ltyv6/OKnTsk+lubmIsPdt7AmxvYqNvPcy7SuiXWyhISqd4Equs2PbuAcVvKb0v0JbCKSf2WlPYu3/L5Lem0xkbSbYlRpBJbPFcsDp0ikcz3EfkjGhTyW37+lUpF85lsF5BuJId1GdP9I4Zl+efwOORn1afl6qP01/JludvxHs150Zc5J85TaQLZm5AEZQmJZNcXe5ToS+BVMia/ra3C8u3XvZTfskejEfr8SQzsuJrQeWJVhEQa6S8SzeDYLsK3fANWuyRaBHX4skmXxbq+9uULnv3rZmn96uYlq3OSSN1Se3iPHUxMYLLbpLxsfXVBNYGtcW8AACAASURBVNtvC7XagmrUb4F69X3Ht61sv3343w9rTZByiyZCfksOP/k4aUSkVCTyj3TAc8lJe656vghy4OgQ9p4JmHUBL9g8hvavPb/9R7h/+5xacSDTW7LzNoHy22MpF7JLCq4XXK+G+faGtrQK6qVh9ydU1aB8y5vftrqSb7l+++jho2/rLI+/fxJD6XUeNMQKqSQyMlLq7ztfCnuuXZNQu0ziL4Wpl8x0IzE9DCG+E+3oGutfK7/9RrjXTqZcPM/x24OJCUR2m5iSk1/CdtvqgusFpfoSDjfchvPb0qoe41ur2S2zUkZnCSYTpOPm+VPF4nmEcuGp8JGAZFfq7+8vinbEcynflUr82emuVMY+HxJKdX1fsb1/Iag/wUI/Ee7fPs+n6RY6M+FYIvhvYnpSfgHhtqxRoi29DvvsTaQ/QXsdzW9dz7fc3gTEbYnRZgBeS43HMSTbIlMhlRCbIqLni+b7R9vHuZYcOCJaNJ99lRci43Tv0v264P49Np/AMGpxP2oT6x/C3X4yJ+m8hRPofWU7Pzx4ICn9gr6A67bVBTe0cHKL8duG2zertD3UnwDzbSveb2G3JYYJvj5rDggV44dCSrhuBNCuv2h+NJI02OLDUomIkzWERHPP5LWwru8rtp42NrofHRDSL4T7+QVACV+QfnvsIHMvh4SDiekXSq4XUMPCtwUl+gJIqwjfUvltbVWv9yegXksPA8G35PxmF4tvWUMplchoxJX6+8+PRhNeqwwsQtI1WTQ4O0TiI5FQUyqNplg3JMT394i74qsftYn1A+Fu/zL94kWGE45Z9jkkJqXnlCJeW11QTVGCsN8Cry271cv5LZ/fPnr4qA1ehcgLwDAuPadIpBanlfqLiLU1oYF6bpiI3TcmlUgkkpAQ5VTx1NCp4tfEflNCQiSSceA5oiaOwDAtOoeM7jd3iuh74Z7KufjF+RMnviB7E6B9ZYl52ZDbWkaVvvQWrVVeviXz2xtlDvHtQ3v81jrfUnRrZBj3+0cPFyF8C88oiQT21Rk8a2v8iW+kej6TMkRLJD5KkktCQ2fOE4fOnDdTPFUMpp9SKpFIAev6vmQT6Y7+Gfn59VH1tXDX5udcPGHxW8u+sqS8SyWo1xZUF5QS67t4v0X6b7VVfZDf4vwWXJ/BlHtehGdceijVUpbHzvcRiQQ8F/Hd6DAZ+XGcRK6C1+mmiokx02/ma+T0k0vU0pCQCRNtOn/BC/kJ9lH1sXA/v5BDZ2Af0mcrEXlC3icl1QXXCxC+rbpcytEqD9+S+e2d0n7BtyTbGhjGffT9oxiUbtlDIZFybNVfJJIjXsvHvBLwWolEruL+nZg6byaY4qnQkAPftWkncL9Z+e1T4W7fA+j2Iiu7JdbL0j8pKShAvLa6oFqv5aZiAn4LEtySGizh8tECd7iIb8nxqAz6rEiEsC2S60qiua463wbipfJciUQiDSfeR8zujSA5d2ro1Jmvif3o6TdRMs53og2Zbn85IKQvhft5Nshuz39BZbc03x5Lv6AlVxi4fFsCrTjcZjstyrdkgttgMDl+Ppjr+JYc7W3M467vYzBsy5mq+RI0U5g/H2hXsADlRoeJ/Czvh7AIwbpimnVniqfOnKqQSGww3d/0k5XfvhPu9pM5oO/2PMdvL6Zf0BcW4Py2RFtaLeS3yPkJRI9CXY0VvrXjfDDH+Rah3O8f3QtD1ISOKTLUdSMi58/3hdbWcEMiCYny92YRAtt3KdJ9zcK6U/1mTvQZZz3THd4/jh7tM+F+np0DrsnOU2tlFN8eS7+YX02olsu3BciKAy/fss8Hq2kQ5Fv+88FcyrfkrG0DH7vIkacU+4n9/KaoAgOn+E0BHy0zMEDprVSCOUMikUZzaRcYr/98//ky5GmioiUyv3lisUjMomZrrDtz6sypSh8fq6br+SPyw+yD6iPhPjt14SKlW2qnA6HblPT8kgKs35ZoSwsF/RY5P4FOcE1aYcJFuJYegucnOOC3jx4+6jI8bPvBZCTrfhjQq5+YHCjjqlSBUeF+gYFSSZhUIpKAKZFGWwg3Mhp05KDEG02zrTKA+5441rVwLvDd1yb6vGAl0x0V2x/C3L4R7rU9OYnnub0JB1NysgsL6Q4aFt8WlCLdi0iGi/ItlSfUVFkjXBzftrqYb7saG43txspao7Hd1NbYCXw3LwrDtZgZJSNQVzYf3PEBDJEkmqLcaOl8//ls4pVKFPOo9wgI5L4f23WJTFds4VzCdf1mjLVy5E2/CHP7RLincnLOXzxvoVvCbxPzLhQWFGL9tvRyCbyfF+O3eL6l8tsya35r6/m3fOcnCPtt+w9Go6G2xmgEnxkhdrinRpiWZ6gkEnbXAnDfMIlECvoXo/0lUNIgCbG8R7i/WKFWx6jDwvlYlxx+bNZ9bdwLwpluTD+4NUQfCPcfl3NyTsB+C1YdTqQ3lxQUFGL5tlSPaPY2mivg+JbcY1Zj6Knzb63wbZvJZLxrAIplONfQ3kUz7vdtRUokveUbaKoLShIdJpFER8si54v8/Um6VcHvIFr0K6L+Rx3Ow7ozSdZlk+5r8nGC+4BH9IPrs94X7obzrLUy8tTb9PTLBTx+W/VlYYH9fsvqUNAJ+i1+CPCtDX7b1mg03DUY21ByMFro4ft7aiS75Z9yiQyXH0RERkdLJGqJOsjLUyQJYX2/+tVf0fWqvzDrwpnuzNem+o0dOxTHt9QcvQX5sfZ29bpwT+WQHTVfQHx7Ii+/8Cbht9Ucv71e8HctqlZ+vkUJF/CtqQbBW5gIePhWwG8F+bbNaDAY2tsgvoXS26729u8tnluEZVqeqcKaLl3qMIlaHRsW97KY+f6wX0H1ajgv67L7F4hMd+Zrr4wR2pHW97DQy8IlO8EuQuntiYMn0i+UFPL4LVhycMBvuT24ZVi/dbRHAZspEG7bajKUoT7LysXajFA+1mEz5RJjkoSvPzdCFBQplfgFhnkGLfb09iMJ91VYuL9axEnDUNZlZbrTp4z5PUq39BjS5yu/vSvczy/lnKdUy/Bteo6WYFsM31ZrSwtu8usV5dubX+PPUDAZsIzrkh4F5hrMaKg1trVxdApP0mWNjRbH7crjY1r8mCmTIFZLlleYREZ+T0Cc12LPAO954phfsUvEeU9WvoBmuq9NnTiGv2Osz08T61XhfpaTcp7Dt0l5+YWFPH5beJnpD0PUi/Vb/j1mehPGcbFs69AeszaDwWAyIXkCNhnrgim3Q83LtJgZqgmdjltJi4gMC5IoLK8P9F8c6yX6H45wY7jviWa6oaxMd+qkMfxnL3j18WlivSjc7V/mMGsO5Llgx5LSL5UUVuP5tkB7GdUrhm+vW3oUMHxLJbgNeh7N2t+Dy+HbNlNtnRGhA4RuH1lc1tQIU66Sl2mhOU8Tygy5FLMnQq2ePo/9PWvUHN3+6lX0fYX6F4DrThnLuzdi+MqfkJ9wb1bvCffzSzlJ51l8eyI9R1/A57dEZ4INfmvjGWH6Wzx+i/JtkxDfsnsUGuuMT4zf8Ca4+HXfNijL7XoQg5AsOkI18FBKJVzHlQRFga+w/v0XcYX7K+R9OayL618Y+8JQlHCJ0cc3newt4f7jsxzSb79gTk64mJ5dWl3Iw7el2pLr1fbxreAZCg0GV/coANXWNbahGhXgW4py26DP0oWzXLZmySGWc5rGooPioK/TGS4iXPTduakumum+NnEsT7owqm/D3F4S7oaTZApm4dvE9JzLhWy/tThugR7aa4aol99vkQTXMsq4kQLisnw9Cji+bTTWGusacQkudrCpocsA0cMToSwX0Sw1wiXw/ghpWGwgV9sYx30VfX9iCmS6M/1m+k0cOwrHuH8YNvI55Mfci9U7wqU6bxm/PXYsJV1fUHgT8G0hyreF2N05NuwxQxJcS7XVoprF8W2r9R6FRpOhthGf4fIQbhd7tLdBn+cFouxJTMGhsJhudMi6MNxrucJd5D0T+X+QUzjTnTKGp3chqC/3/PaKcD+jehOY7tsTeRcIr2X8FibcwtLLnBPCEP3y+a3APXobyqoQv3WkR8FUV1bXxDn/1orfIuzQZiD49vsuMJ/wUC6qRNaYLqFIVyILDFKhXw/VcOMwtWK0f0DgTPT/FCoWznSnjMHfI21ULPKT7r3qBeH+48sctt9+mJJXSmm2mtFsAe23JZdLbnLOCEP0iiVca/cwq2E0y8+3VhjXVFvW3oYkuEIa5fItRbkPoc+KAlDyRFSIGUoi05XKNWFhmK9qQpUcUgDvG+jv7++tQv9/sOuime5r4/DhwsiVyA97EAl3+yfNF8/DOVhS3qVCqjB8W3q5UNhvIQ0jfMtLuABwtSaEFTCDP8M1GSpNDvTgYsjhUVsdpOLHSJaLaBA/VJLoCIlMowxS4r++CNbt/8yn3l8JtBuIsi43050JZ7rjfo8y7tBhf/Dqu/v89rhwN1BpAuO3Kela2m1vEqplEe7lUkSlDvAtQrigyI4FmAnwfIv12zZDvalNuEeBT6WI33Z932WAP8sLYDMnokDeIZPINKFhnviv+wdBsPA/csv7qxQzAvyt9OoC1oV7F8Zi28VG9N2tIXpauJfTcy5SfEtWXnYpx28hvi35sop7Ai5mYNbMrPAtqVzDLQd7FL4xGIwN+DUzWwgXYQeQ5XbRjPuw6zGLcu3QbahGrAkNDApEnwcjNmwC0x72qpxNtYFKeYB3QECgrZmu38xx2M4Fzz5rE+tZ4W7X51xk+S1ht7x8q9VWV9vMt5weXGyPArsP94bBoR4Fo8HYiDgwr+NyNYq6LTEM0GNzkb99dEtqViEDm3Q0nuvQr4ER4DV+/Pjx6kWvLlqkFnn5azhMKw5XBgQS2sWTLpLp4hfRYvoKFnpUuJ9np19k8a2FbjF8W3i5tJrlt7gEV8hvBQm3yUK5GLKFM1yW35rKDN/w9ygwhIuoV4AcCC03Mt2N5kfmx3n28K0cnF4nlckUynCNSirnI9zF48aPHz/BKywiYnqoxt8rUINkuFHeAUqV0tvfW6B/wRrnjuorWOhB4X516hLbb9MvXLa4LcK3YK0McVqkQ8HmHlxcmWpQvhXqUWg01Bnb8H0KPAOnUfxoYfzW3PX9BZG17Hb6dCVxrq1MLlMqoyy9C+J1auS1xFgTCwxXFBQYIQfKVseBd5/H7VUIDAgMnBfo7e0dOJMn07WcvYD33NF9dEBIzwl31cmclIsQ3yamXzoJ+20hi2/B7nMH/VawR4HdE1ZrR4bbVlNnaOTpUbA9w8Xw7SOCbY3tBN+aH5m7zI8688Q82a1KoZTJZshkshkRwGHRrwfND8GmuF6eE8ZPGB8rmk44bmhgkHcouaIWyqXdgEAV+K+3twqb6UK9C1jO7aNkoceEu+rLnJzzkN/mpOdDjMDl2wJtacH1AjzfYhmXvwcXm+FSVVdna4ZrMhhMKNdaJVxUo3zjoYHxW7O5634Ypy9hulIpp45gViqjMIqmhihocVhYFPp8QNCE8ePHj4tVRZGOGyrypN4fSXDFSqVSJZ6nAsarEsx0sZ7bR7eG6CnhvnmI6b0Ffnsx/UJhIeK3DN+W6AuRkxRQ9fKvmQnyLdykUCZAuJYehSZjWV1bq1AfrpOEC4ax/XvKb81d5id5KpIUwpVK4jRbqUyuQtSIG0r/xbFBi8MCuISrHj9h/HivME1UZEQ48YzXSJou0AxXNUMZDj6KAwICAv3QTFeQc4OWIT/9XqgeEu6Gizm0aoHfJjF2S/FtNYtvSy5Xo/d4wBEupgfXdsIFmq2rs77PrNFgwKcI9me4PG5LKZfx2+/N3zZLFHJZtEQSIldg/+XHjAUaeqwRqWODFsf5/wfzqoCgiQThzgxVyUjH1QR4Qp04aL8CyBeIxwEBAS/7cTNdpncBl+d69sVNJ3tGuJ/lgN5bhm/T0y8X8vBtAS5NsNNvbclwLZZrhW/rDAYmWuDbZ2ZHlwLKDkx+a+h6ZO5seVCZlfnBzp2HYmQyhVypVIarposxKrUyZLJA/zCvcq8wf1Kdi8MA4XqNDtWER8imk9+r9rcQB8q64lBvhZK671q4t793IP7shdfGjEIod3RftIn1hHC/+jIn5yLEt3n5kGbZfFtdcL1QX4Xc44FPryjh3saco/A1lm/JHoXaRl7Gfdja2lRX1oh6LToQfeI1yj8e1tbq3k7avfPt3NrabwnPvUd1eImjVAqFXKFQRqnwZKuxOC00oiSqBZp5AXGLg2Lj/APDYwHhTohVaUKjGOaI8lLC74Ow7jyxSq5kHgcGBHhTOmZxrh9m969nH8BCDwh31b0cIr2l+DYp73Ihlm9J9ZZeRtwWnyjw+60g4XKrDGFbZp/ZHYOhgb1mhuVbZzLcNkOlLmHn7p07dZUtnUYL4z4y/xDD1qdYNV0hl8sV4SrbWFclVVCUELY4KAgQ7gQ1+LsQHiGj2UMUxvJyDOvO8/MPEDN9un4BASP9LKxLca7fmKFczh0S1/uw4Hrhfn6p+SLEt29dgDVL8y1DuJdLkVNwrfCt3T0K7HXemkaeDLfOUNeGeiuOcXkGolLIb9t0uR/t3PlOVm5lreFbOr9tb2cY19z1/T18j5dYFaWcThgwTLWYIZbJmGeVgHB9g9ZoFgDHpd8rbHEA991R1hXLA1SWPl2xtzfZCQllulPGIJbbB7DgcuF+folYLaP4Fmzi/TuWbwn1Fn6Jvc8Dol/eDJfnHAU+x21tam0wYBm3rswksNcM57c2ZbjtuVkf7dz5p6ysSj3cg/uw6/tH5q6u2m8hz+2KQVVrGarp4UqlXKHEJF/MUEjpR9MJwiX+JigiZNSzgUEif657o6QrDlUpA+jz9wmvneLt7+0H9y5MGsOl3D4Ic10t3M8uWlZ5z3+Rnl5aKMC3JZcLWWfg2rDHDOnBtaFHAWbcptYyDN/WGExojwLvQBSKMm5bbua7b+/8M3BYI550gc+2t38PeW4HzxoYOQgnFauipisUiulRGMfVLNBMl1AEDBbNxnv5E89FyDXkO8Qt1gR4I++Lku48scpboWL1LgQGBnh7W3oXJiJxbu9vQHOtcNeezIH49mL6pb8XCvBtqRY5l5E3URDyW9sJF3hrYw2Hbk11tQ2CZ+FiMwWc39bW6nKzMrOyMjOzKiuNguxA+qzRwrhd5rY822g2VBw1XRmuJOmB48tS4vprCiDc6aJY8FgppxxXFRQQGuWP6STDsW6gQh7I7tP1A9r1o3oXxv6e27PQ6zeddKlwwWrZCYZvL1rCW8pvWXxbqC29Kci3VnsUuOco2EC4wGfLmmC+NRnqTLg+BR6+xfltW8uTyszcTKBZwLCoSrED+KzxWyjLNXfgKZcnSYhSKaPAYLmvRqrQLNAoxo8fHxG6INZfs0ATFSEnvxa2GFy5IZTLx7ozlQGBnLMXZhJNDYAaZiKnOY7obVhwpXCX3CP9luTb9PTLJwX4tgSX3trlt9Z7FFDCBVxb942Fb28Yahr4+3BxGS7Uh9v25IkuMwuMrKzKB0ZE0Wg+ZmFc0mfbas2w5y6yde2BGSqVUqFUKiDvnSEXE4Q7CaxCqEI1ikjKcWPnA84N8EbeA0+64lA/hnWhdTS/SQEB3lNmThnHvVfE8F7exuNC4W64CPyW7mJ86xKXEVhnKJRcLmTf4wHNcDF862iPgmWfGUm5lFK/KTXw9IUJjEcP22r1lVmZf9q586MsXTufRq0Mymdr4VzB/ARLuYjXokMZrlAqVSrysZQgXODSi8NIxgXP+6sJRw4IwKfDCOcSXQwBShXauyAO9B7pHfgCN8716t0w13XC/TyH2MtLksJFIk3A8S3pt/j0ljdRcE2GSyu30kS7rSXDRf0W5du22jJdZlZ80s6PsnIbUcK1luHChEt7rImV5XbtwvR/2TyUihBluEqsmTQOEC54JiA2SqOQRxDvEOtPki7m+gwMDOcSuW4AketyehdeE/t5j35hFCfPjfkK0UQPlsuEe4rsTiD9NiX98mcYvmXOCNOWQqcooDpFCNdFGS7Vh3unrrW19Y6hBO0Jw4824xN9VuZuwmFNqEaxOrXJb783mw1wrvD9E5RyMf4qMMRKhTJiQvCE8WKSa8M00yMA9S7w96L7GgJ49gZxe3WpqQwInAnvR2POXRj7wpAhQ+DjHIf0apuYi4T79LOc5osM3+ZdAGzLx7eFBfj01gG/FexRwBEuybZlTTfK2NkClnHbnpRl7nxrR9LbuU8qjbU1BmNtfcXVc1fPnvvo6rlzV69ePXfu3JWrmVev0J+c+5R4dJX47GpF/V2j0dhiNLZ3UeeEwRku7bHt35hhz90llNPaNvwmTBg3bgL52sCgQEWknCBcEfXdKqU/3/sglEsOZYB3KO7chbEjhg4dMuT3IyyduUsRXfRcuUa4RAzG8G1efiG3qqEMt1BfCPstlm95GJf/HAXbMlxKoXW5dRydsjy3rVZ/bufOt956a/fuvwA9nruaWVlhqDUajSZ7zgrrbG8xglFfAaRccbXi6pX6FmNL+7ewx3ZVwv7bdZdDuULuyjMIwlVIyK/GLVYQjusfy7xaHhi4QIOqlp9054kDZ6hg0gW9C0TXAuG3o4YMGUUa79DevOmkS4S7PT9v1668cpJvc/Iu/12Ib0sv49NbvkTBol/cOQr2E25rXVmdAelTAOo1ZmW9vTNpx1s7MnNzK8uMtUYjT4Zr9SwFhBwodmgDSq6vr68/d/VqfX19i/mR2dgF5wpdeTx5lc1jOpHhhkZJiC7cqCCJXGEhXDDEfNdnoTyZLrHyG6Cah/ToThlD8e2IUb8fMmrY0GEjezHMdYVwlxaVX4jLj7uUtyv9xPmc5tLCzziqvQntMdNqC1zAtwjhfm0b4ba23imra3pY8w1st4asrENJOxOysvT6WgPSryCc4eJ1ilItZ83M3PVte7vRUF9fX5FZUX/37nfEc2az+YnaOb/VTAeGKwaJbhSZ3gLHDfCCXhEQGEhmvqhuiYFyLjiLQenP3Y8mnvrKCwzgDh0xZMgfe/M0MRcI9+eiuJN7Tu45+deTxet25a0rFeRb7Ul+v8UTLjbDtd6jgCPcr1vvGGqbmlq/JlfPjB8Bh333oyxdZW0bh28F+xSs+C2eHbgZLj0NXe3AhSuuXq2ob+ky58kRJdkzxCDDHU98RwjxTkFhCk3oYhH0HlGBIwXyYpRxqREoV4q5PbpjWWnuiCEjRyLq6KlyWrjP1l06efKvJ8n6snnlhZMCfFtyuQRzHzPbehRckOE2Guoa2rS1uqysTw5lHQIOa3Dsfg/IsOksBTRTIHrEjNTj7vb2u3frr+RNV4ZHqRz02wWTCMMlHyuixQs0oiCFJiDoP+DXyFXezEobqlxe0p2nVAZCrgt6dP3GcdLc3rvppLPCXVX0JfDaPSdPntxzMm7xjx4rmr9ELs1ozyV0y+u4iHp5/Za1zwxxWozfNjW1NtQ+yD106MMPP3z7w6zcSn0tbq8Zpk8B47c29YXxswPbb81d5kqab8nRLFKFhyvCleHKqChEU9YGQbiT6FcpQsI1Cq8wjVrE+j6l0pt7ni4nXcBxLsjB/Kaw9wDPnDR2KPs+v569BQtOCvfZ4vyTTMXFgfXqNxdfhhUL7TGjzrOzi3CxPQr2EG5jbZkuKyvrww/fSvkkM7eyxkR6LLM/x6rf2s+4qMfy+y3oEWujH5vBfEBRrgp0IiiV01Vi1Fd5B0241IiSKhWiWP9YFfv1a1TeltcguuXPdDWh4ikqNudyG8VGruylO1Q7J9ynsZTfAlaIpbqJrwXtwfptqR65r44dPQr2ZrimWr3uUNaOpJS3P0nQZeaeO2Ri9eFSuyZ5M1xsn4LNOx+Qic1w6WmC/NbcZW6WsP7dDlcoFOHhPPt4OAMiXGrIQiJjg7jnPoeHe2O6xGwi3VA/bz/W+WLjOGfn9tZpYs4J9zmL3+4JYu58tapcy3Zcgm9BnIDjW0Sxt3GMi7nXAz7DvaOvzM06lJiz81CWrrGmXnfuXGVdU1WZgbPPrMHAyNTZe/YiOkVdVsBvzV3fGx5a/NZs7iI6FjieGqVUKKLCVda4FyZccogVEfOD5nFfp9B4w88huuXp0yVH+BQxxLmhnD1oQ3vpTCanhPtmEMm24LIsCLrv1ZJyJFMovFwq6Ld4wuX3WyTBrSvT52Zn70y6kH1IZ2pqulNTpsvMrawjaKGGk9uCYWjj6VPg7cO1OVNACBfSMuq35kftj80sz23GnoyvCVUpFXJluMDpIGIW4ZLfI5NHcM9c0IQGhqtwXWKI62I4F/TeTPKDOHfSGHZv7vDeuemkM8JdG8T4bX4QC21WlVOaZfaYXa4q4PAt4rQYwsVmuF+zCNek13+0Mycp50NdJX2XhzpD7rlcYx2dKRjK2jBnKdwx2OC3NvAtRqeoz1rx3BbIb81m8+MiAWfVhCsViunYDl2ScP1Yz6nkEUqJivvKecpQ75mc98UoF2VcavpNmWnhXC7m9s6eX2eE+1wclSecbObu3FiezvLbkvwSzn117PNb7prZ7YY7dXp9ljon5cJOnb4Wcl6DLvPjyhooDgObyeA+BWafWZlVvsVmuI6cF8ab4VLT2NbFptz5qIpYilIpZPJwZP9D6Hgu4QLHjVRqpAruOwQoVagPIwMlXOa6bIolX5jKTnOHjeiVlV8nhPviYhpv89D0rjib4Vuwt4xznwf7zgljEW5ZdtYnF3IOZWfptbXsDPdOpS63stbEynBN5B5I3FkKdSb7Mlxez+XoFPVYK377vflbo5nluT8sQt0UGSqCe2FvJgiX464qeUSUZoFcyvVtpYZaP4MGqlwB0hXTpDt13pQX2Pf8Hd4bbWJOCDcun+LbPNxJk0QoRvpt6eVCYb/F6peT4ZZmZSflJH2SnaXX16CE21CWe05XVsfJcBtqS5t4z8NtK7OW4brsvDCLbnF+C/aetbEclzkdxNoQg43r0/kJV0OfwKSQcvxZGaAZacP/AeFbW8weOAAAIABJREFUuH9hCu25Y19gn7MQ0wv3+XVcuMspw91T/ibyNQ8Pj39dR/fgarWc+zzYxbeG7KwLO3MOZefr9TXwPjMowzXoMnVlRiTBbaqtbbDoFD1LoQ7iXp7h6j4FxGup2dYCM25X9w+7UIflG6oohUKpoQl3ygKc44JEl53lqtZoZnqj74gol693Adwzwm+KH8W5nBPFftMLsOCwcFfFFRN+GxfH89dryzrSby9rMXfqtUK4N/T6/OzsQ9mHDmVn6/WGBt4M90aNTqerrMGtmoH7kbH6FLhnKZjqHGFcp/oUUK+lptHMptx7IkQ/QkMF9u4gGS7DuODRdJmc9RWlKjQAoWTcmIcSLpOG+U0RE547ZSx7B1ovHD3qsHBXk4Ybt45vd+ezlfmAcYn4lsO3vD0KBZdBpnUo+1A2QIJSlHBZGW5dmU6nM5iwfQoNhpo2pA+XczVWZsVve75PgZnt7TDjdps7d6FuKDw04ydMmDBhOvc1UZTjahZoIuVwFjEvYIHYG/eOqHYRvrXcMUI8xY9QMNQnRm5A43Ez15XDwo37EvhtUfFa5Ct0/dhcWlB4uZQ+lZHXcauvV5fo9VnZlw5lZ18CgjXYkuHWVOp0lXXoihnptzUG5qZmKN0yp9c02rbXrAf7FOjZbmQxLjjpGdGPlQEIdwLYM8l6XZRMzjCwgnV2uVKl8RboEoMGyrfQvdGmTCE8d+wQds9Cj4e5jgp3CTDcJ0WCncOrLxWS3WC8hFuiBw6bA1ICPV+PAibD/a+aylxdHX+XAsgSOJrFnRfWVNvnfQpMj4LxW8hvwdglRryQGGtU6HNMhjt9gUZDES81wuURls9UcKKrUmoWBKDvgyddHONS90abSZDu1LHsZMFzBSIH15ajwo1bCS7LcHEC/KIvkTuZkaMkHzhsTlHOpex89ApN2G9vl+oydXUCfbimWi3SiYucFUbtUxdkXIzf2rD3AZmCfQr0mpnxCey3Zv7TQcK84N5ay4AzXIXldOhwWSRMvBLopNEATWigYJcYTAsI3zKcKxYTme6kF9g9C4v4ENJF5ahwg0B3grUlkhfzoHNwiTvr6PX5ly41NzdnZ+dikwXhDPf27a9rKjMrbwn24ZaWmhDVonxLVGOdo4zb2d4O9kFiZouxsw1xWma9F+e3hOd2mWjFUo7bvQi/OuYZq1YrkWcXTCEMF8ppKd9VyiNYryNOuqH8V7lAE8Dj64hyBRLdUDLT5bTm9vQBIQ4Kd2X2yfxY6wBefI88J6wmX38hvSg9PSk/67yuRDhR4KyZNUB+W5ObWYbJcOHRyGQJ1giXOI3Jxgy3zWg0GivBTt6rVzPBTl6whdcAnmT9l3yigtgcebXi6tVzV6/eNbYQ99fh61NgPNfYBfttl9n8AHs6iCpocVBsUBiSBwDCHc9+bnqEQhUaLuPsqFBImV4HeahGhT9lATdwjEueoysmTZfTs9DDsOCgcINO5i+24TDfJbu+/CQ9Pb05J1dvAI6bm6UXWm3A8K2lD7dE91FZA5Lhcgi3rqwNo1kM35J9CugOM3iYjMb6c+eAXCuAMNswO3yFrsTAeAic+W7F1atXKq5erW9vaUe8lu4L6zRSGS7tuOZyHM3OV0cp1epYdRyGcCchvhkul3McV7MgXEK7bFSUZkEA0sfAmy7gGJe5N5pfoHgSO1kYwZs3uaQcE+6Sdfnr/oE8a6lry5Z5xsbGxsYVX9IyiUKV/pAWyRRQ/SJrZsBrb+hO66rwfbiQ997i2C1/hkvRQ1sdhm9NNYar566eu3Ku3mA0ueC+ZgzjEhrOvHruXH0LZ28vGAY243aZsSc9h6kJzg2K9WL1GhCEi7srZbgskns6jkpC34vSnyRdWwfCtxDnhk6d4jduFItyR/domOuYcOOKiv+GPAn2+z63cl3s4pXFbyxbRv51+6mcznCrsrJK7T5HgSJcfaa+xPr9Hu6U1vHsNcMzLvBfA8th6yt05yoq6msx9+5FFMqjU4RrcZlCJ7E7sqKisr7Tsl5mbIcyBWKUY1hWHQ3+G7gu1is2jnDkqECUcKGhlEeolAoF+1mpnPwYqFqgUQYi38NPugjfwvcA9ps0ltMmZuXa3alyTLivcv4yLSsuXhcbWwwEe439ldUXyDs9dGSVIm5rg99W375Zkplb1oD04aKMW6M34faa8Q/gr+DekeBDRS44esZgbMQzrrPnKeD7FLpa2u8C/da3AKftMrAZ19yFo1zaZwPUXkFEvrAYrLFNAoSL7dMlGFc8Xc6+a5o8hNRhQKhG7I3wssDAMS7FuaBn4fe9d59fh4T7jODuZ0uXbSkm6rkttMOiVVRaXWDI1ZUiGS6iVQzjXq/T6XU3sH24nNPCqgw1d/gUi8twKc81VdZX6nT1ZUYTxmPt78VFXdaGTKGzBZwRcre+3sjyW3O3OV2BuiAzROpY9WJl3GJwYhiWcImhkJPuqpqujIISBCV56oJSpVkQiPF1ftJF+BbiXCRZGNmDt4ZwzHEpwRZvQRwWqWXN2iys2/IlCpDfanW5ZdgeBSTDrSmrwu7tFWBcY22lrrIit9bYxmFc/j4Fq46LYwcrvbg04z5uqagAh9s8hjz3nprf+0I1gWFBsUGxciHC1YQqLalCuBK699904uzyeYB2bVw/IwfCt2zOfYFzn9+e2/PrmHAXLVsmdG0G17PybK095ygw3nurTKevEbjfA5Th3qmpaxA4TwHh27aael1lZSW4W6/JiKYJuIEoFKtTxGXxjItmCsQ0mNvb7wLzbaeyhW+LAlAPhIYyVh22QLNgOi/hAseFXBucYU6nw2Ii0VWKF2C7xIQ8FyVcC+dydkMM6TlYcEy4q23F7mfLli9fhzitIOGS2jXodLobDbadpVBXZuI7TQEdJgPQbFldE9WnUNOjfQrwbjOc17L2mhk7iauzlpb6u/V3azu7zOaOmBmI58EjLDZKiHDBqfoRrN0P4nDFdJppI2RiTSA4YcwuyuXnXHDOzZSxI1g9C6N77PrMwRx3nU13ZPtp6ZZrHh6LrfItkuGW6LJu8NzvASHchpoarl45a2YM3zYaKjN11KIDvWZWhyYIjvYpCDMuxmu/Z+8162y38G1X/dWKivbOB0XlahFPR8ECzTwvEfBRfsLlOC5Ju3LadRVS8QJwZx6xsK8jnsub6YaKQydw2sQW9dSt+xwU7pbVyFNoLVlNHJi64hLitfyEC/y2Jje3BHd2DTbDrSu7xXd6DYtxGwyVunNlnBwXtJNDUS5fn0JP9uLC4y57B8939RUVd1vM9/N2qUVYDg0jGBgQ7gT4npGw6shTRtn5llihID12uiQqCqS83ravn9Geix9icehY9jkLPZYsONqrEGv1b9Kq1UuprDe2xJ59ZmWZuluYLgV8hnunstbKeWGEOo2V564auC2MlA/XIV6LDkSfeJ0iLmun5za2d3Nyhe7OlqtX73Y+7ijfpUZ9MVC5QLPAT4BwNQsUkbhkQhxFdumGSpTEu/J6OlvxsOPiR6h4CifMHd1DK7+OCncZuj+SVU+XL2ekvewC4ra8hGs4XlmAPysMR7gmbQOy8wHxW4Mus7IOYV1mrxl5tyj+PgX7MgVk2MG4xOoZmx6I2V5fUdFuftK8K2Y+hkaJe+xMR70W57jQv/3hCuJkMplERdwZAn1fwYHwrYVzJ3LugRbDF5Q6Vw43kq8UzMGWvAH/cotKbCTcmky9UB8ul3ANZQgccAm3VnfO0IjPcKm+MFMd6rB9w7hd3ZWI41Kjk9Bud0feq2EzOF4oRLg4xmVGFLEjQk7cjZKvS4zfc3GMS3DuVM6tfnvo1hAOC3eFwK9n1Rb2/snlKOWi6r1ebTikLxDow+VmuFVUdsvruHdyz+qM+F5cOB2znI3Lx7g93ItrGZ0m1G/p+V1L/dW75u4HHXm7YqIt+oEIF1Ubj+PSSiRYVy6ZTq6huYpzJ3I6c4N65NYQju/yXckbLkOUQNW6Kus9Cobc3FtIl4JAhltjaEB6cWHGrTyXa8DvNePs7cWsmqHuixmITlGXta0XF9r70IJ4LWu01Fe0mLs7H9wvigmjaJUwXESVzIjEMi7DDOFyVZRCOl2zwJu3S4zXc/GUOw9ZPxsW1BMHODou3KX4TuGnL65AFf1GnLVEoSZXZ+A9u4bJcC2duKayKtxeM6pMlRUVhgaEay0DPk+BOv/OecZFyMG2/WbwXjNjJ8ZrIWV3dbeABYrurscdzWp1WCSZ4U5HT8mzOK7Qii5IGJRylUwWatMpC5yBY1wwmXtDULNHVn6dOBCkGLcBYtWyZT8hT3pcu1CCOi3EtyX6XANfHy4uw22oq2lAdprR1VZWqSszWe9TYDy3FuOwvXJmGIdxAcsaEZdFRufd+ooWsLb24F5zs1oyYfx4VJHMiJArkefYQxUwXyOXiecJr59RDo26Lm5O4NzptydOE3NCuG9iLPfN5ajdgirOFyJcXa4WWfdFMgUow72lrcHsNSOrrlKnq2uy0qfAJgeTqYfODLObcc3mWozPQldvwHPN3V3t9fV3O8HjPf7qMWq1iM9vF2giI8KR5ziuq5qhCFdKw207ZYHrudgBbiTFus9vnOuTBacOveMeYfNs+VKeePfFZv4MtySXvibjI1xOhltXU4XZa0bKVldZdkewTwHx29aHdwwo1fZYLy6v35K9uMZOxGGxo6ulvr6lu2vPmjVrvrzXXKQO4yGCyEhrjgt6xMLDlVIF9pQFK56LEi4xkAMcBXeDO1TOCHcJZ1XkxTf4/2KtLOQh3Cpd1i3cOgQmw6UIt7QW6cUl/dZUdk6HScTQ0co5M8wgyLiu6VRAvRbjt2DvGeqzLMaFZv3Vy2vWrFnzd/D4XnmeWoTZ6xPJnAfCP0KngPtCSPGrc1bGPAzjhornjRnBvs/vYpcnC04d7PwG65ezVOgQ9WXpiNMSSi3LNGD6cAXuadZQVoXZa0ZckOUauIq1zrfEaOTmCn3GuN3mlu+wDosd5vysyDV7us3dRLU0l6vDohHGDedXLD0ATWgUUn9+4uD3XBzjakKncFoWhrj8gBCnhPssFnq8BU+3dAXhCPdGVi6ai2H81pLh1mmxZyk01OVe5ew3EzhPgUsPTYYeZ1zEa1HPJdXb3o74LN5vu83dJ9esWROta4eee3Jh16IwmGqtMy7I1KYAPSpnSBzzXBznjhvC7swd7WpYcO4eEKsZy12yhYdumZdeQP22LKsO7cPlEi5zTzPguLU16F4zgLbndCZIscJ7zTCea0C4tq8Y19xtQ65Ad5ABUNjTWV/R2Q3Xg7xdMfPtYVzNAvKMR5W/xDpXYDwXZVzxvJkcyx0WY0Uf9paTd91ZRD1YbrXv8lk5l3ALcnWYdQjuWQpwn8KtshsI24JM9yPdLd41X9xAzsT9ps7mc3EFHRcZ9jOuuaUN9Vp2pkBPLUm4Xd9VXG1nf+3x/fJFYaBzJsoWxl2gERMrvprAiRLk7HJHORfcjQfm3OHcQ+v7VLgeq4l/AZ5h1hyQ+rmD7bdloHcRy7d89zSrqcGdplB3rpJvt5kNGS41DP2AcSkfbW9HvRU7CMNdQxDud3crurn14F55jFoZIY9CdYqOKeQz3oHSaORruIFqF5mcw8SGjXbtNh5n7ywZRHTUCDbcUHWtnEW4uboqHr/ly3DL6tC9Zg0mcPod0qUgeJ4CyritrTrePoXe2G/GPk+BN1fgMO7f14xcs+bv1Oed9RXtCAM/7mjeRa8NC48p5HWZ2FsTIUEc1THO9eXsPxu6CJGEM+WscJcWe7yItCbgayW0elaSWca9HuMhXJpvq0pNyF6zpppKXR2uT4HLt9YYlzhFjGcg+uTRKeqx9vaGUb7a3ojzV3SADHePmfHZlvr6TsR27145fb+5SC3hSXmZoaFOMg8MXKCU2kLFXM9FKddv7Ci25br2PlJO34R68c/LbeyhWHaJIdyyLANypSacKdSUNqBZQmVuLdqnYJVxsfd9MOD51kX7zWxdNyNzBSPitTjG/TsAhVL4uRZyJRie9VeutHd1dtwvUqvnY1Jey5hC9papAjQalVSBeqoDnDvxBXaWO9TLlSu/Tgv3RZv/Hv1UTl+V6XNvcV1WOMO9VVNz5zb3vLA6XZmwYm3pU2ByhTY7GVdApZiBei3quYyn2pYrWAiXqe/u1rdzHfdKC/HgSUdzszpMjiqWGvQdg8G9eEKj5bYkuqh2Off95VruqCBbt4bbUE4L12OdLYBLVPEDgnBLsnRct+X1W7Iv7EZZCbLXzFSZW8daN8OoF8+4GHIAJ45aYVyn9pvZxbjEYUxWM4Una9asGXmSy70t9XdZn9dfyWihH3c+uA86ynj6F+m7nozUaBaEKqRixFPt59xJnDtDDHNlmOu8cHnaGzF1LQ9otCa3BnM9xkO4RIZbUobsNWsqPVeD68Ntwme4vL24vLmC1RQX0SnGZ+1fNyP3mdmSKwDCXdOFQG13y7lOnOPS9bi5PE8tR3WrUVF7IOYR+8+ipLakaNY4l2u5rryPlPPC9Vhn868mtqr6emkmpsNRMMPV1iB7ze7orqJ9YTjt4gaWcVsNbf2FcXlyBba3EoR7EskRwKy4+x3zmGBc7ms6isqDwrhXYKG0E3sTLKyShCOeajfnKjn3P/vDiMUu6yl3gXCf2my5S+9dL82tgvpwUb5FMtzr+v+C/ZZgXFMucpwoTrP2MG5r2zd9u98MPjesHd3tiyVcsxkxXMJ0r3byOi5dD5p3xYj+g8UKNNeSu4k1UpwvW/FcJMvlNOa6EBZcIFyPYpuPKynS4U7QF8pwq/S3kPPCanNN7HQBKTsZl/DgWjzj2uy4CDvYd6YC62QQXK6AZriliJdSs6WihXpcceVKJ/p1cj7u2LUI2vMuplMH6l6pGqn9iS6Xc/3GcCh3aIyrliFcIdwXbbbcdYeweuUj3Nu3a7TIXrM7Oh2+F7fJsT4FfK5gN+MiLus445q7DYjDcgZor9mDIVyqOuvvkg/qr1Sg4S5UT9LLY0QUNTDdOAGU98olVvf+Ip7LodyJXMsd5apbQ7hCuB7P2XKuDUhyV69D/VYgU7hVWoOeppDJvY8k4rgYroXVjGfcVtMTIcbt+TMVYBW3G4UZ9weScHE+Ss27FSTvXrnyHf0c05vDnp0d93fFqAErUImYhtnFo5DYnehyONdvDCfLHeYp1PxqR7lEuKtW2sTcby73KKrGEC5fhnur9gZymkLDFX68dbBPgZMr2NSngNEp6rIOM273d1Z2+xKRwg94wqVNlwDdiitoEwOmHnc0L1LLp9Mu6k1Tg0IqtEcY77lszkUo9zeerkkWXCJcjzesnGtD1IvLPDx+voe4Le/e3prSW9zzwu7U5t5A1s+QEuBbAcYlcwWeXlwbHNeFjIvNFSCf/Ds2w+XMzrMtBONCz3MH63segOU1sq9RxZCvQmp3osvh3DFDOZQ7fDSiDEfKNcJdZQO5XAOXcE+LEK3yEW5pSQNyXpg+14TwrasYt7XRiD6HqpNPp4jLOsO45k7BXOEkYbjdgo7b3f1dRaetjktW+12wNhw2Hz5lQS63IdFFaQGiXM72s2EuOnrUNcL1WG2VXK6tJlpx1mkF/BbKFBq0Jdy9Zg1NNbkmhG/5HJdvIHzbyuw3q+s3jGvu4uYKsDcShLsH9Vju/K6i/eqVCtbzuMF8vbO2u6v7wb2iIrVoNJ2NaSKVUnsTXXbPwpgR7Cx32ChbbjRmtVwk3GfW2oRXrSY3Uq64h+YImAz3VlkVel5YzTkTZq8ZXrF2ZbisXMGxXlxhx8V4rSDjCucKBOG2ma05bnd3Z4VdjtvdTnY6EGvDYf6UciOiZHynOPF5LotzJw7hWu5IgdO7bC4XCddjmXCvzVdM6+NiAcZl/LZKz91r9nXDnZpMtD8M47iCGa4Q47Ya63gZ1wbCdSnjgkOe+TKFH0CGexLxV+y8Isi43Jyh02h5/KB5lzpMQeyi1EREIq4qPIQpd5gX91yDPhSulTPKl/5IP1r5JW+GyzBuTSl8XhjluCU6dK8Zpk/B/l5cZjRizm1E1MmjUWHHRbwW47kcNRlRfdlFuGTZ57jdRvan9/LK1VJVlHKBQirYE4l6rjDljopFBGJ3uUy4PwoFCy9aTvddcZ/Pb+kMt7q0FD0Tt+lWLprfYvzWKuGijGvZa1bb5tS6GTLsYVwuORi+42HcH+h+RrzHsueVKxXfWWNcyHPbjdz3eNxcHqP210yXOHGfiDEjOFnuH0Y6v/LrMuEKnVH+0wpLzvs0rpqPcUm+rS6rwp0Xpqth3dcMW04ybquprt8wLn+uQGe4NjquXZb7XTvyFFheyylfpJZYS3QR9TKcOwmx3GFBNvfC8pXrhLuE//psC/zLLL6H91uKcQu1VbjzFHS5+PMUUP0KMi6GHKC1tIYankzBoTMVnGJcc5cBz7gNIMPdg3orbgLHrb/bgmgfN8jvN2Lfp+Vc54Ny/HnovAOm3LEjuJTrPCy4Trj8B+a+yTrwZlkzH+MCzy0prWbvNSP7FEy6O+ReMwHG5clwbejFFehXQBTKp1PEZZ1k3G7jtzh9kYT7924bHffKlbvf1X+HPM1b3xqxX7nb0t1t1uU171KLEKfl9VyIcrmHLAwbNtrGLgHecqFwr/HdYGUFe0E4TsBvS0sb2H5L9SkcN6FeK+y4+MHTp0BPUx3Wbx3rxnWOcYkoF2Vcak86zhdx8+yV+u72q4j6sYP4fiP+fc6Bjy27Wx7cy4tRR/G5LB/nzhz7x2HcnoUg/nPmbCoXCtejGG+5qzmL0z9fxvAtmeHq67DnhTVVlt22j3AdY9yHbXV9dW4Y5soJmyucZAjXVsft7m7Bt+Riqx3fTNYJSNnc/m69ubuzozmGb887ol46y0Updwg/Wfa6cJ95Ik95eHj8g7vC92IRlnFv37ylLeGcQU4x7q1c5DwFq37rCOO2tRrwjOvAmQrw2bgOMa7Z2IUyLpHh7sF5Im4SjAt6xL5D/wbgRle3+TsD5n26us31j4m1tcxc4vMn98DuNaun3ljuCzF2FJdyh3o6dx8pVwoXf0b5aiRtKMLwbfXtm1XaW9jzwppuV9ZgzgtzaZ8Ckytw7zOJKJRHp6jL2um5qI5wucLfLYRru+N2d9YjX+CtdnyXL2G53eZuXSb1qwI3oihSR3PzXY5yacqdhFLuMC+nkgWXCnfVOrS9cRXaUvHGPdRvr98s0d/CnRfW8HXDrUyEbB0jXKuM+6jBgPFbZzsVHGNcdq5A+l6XPRkuzbjmbnNFJ/p3gGd0onvUiFlPPV/50WPmuccdzUVBYf6I06KcO3PsUG6WO8y5W0O4VLgeb6D2jznF8cU8lHFvlpTizgsjHPcjE3piGFKwPoX5VtBzDX3FuBjPRXMFMlKgP7NendSWM3ss14A8QxaZB5u729NqWb+mjuYYtSiQ13PpLBdnuain2V6uFa5HHFemq1Ape3jEIn57q1SLO7+GzHDfhc9T6JleXEu/wjc9wLg4r7XOuOY2I4dxwaLZyDU4P+SZLQAVzFQqYONo6cS/Vz29j838pweW54lf3+P7YHkN9Vu4awHtWBg6arETyYKLhfsGN577GYUHD48tl7iMW1vCf1+zujLsmbiu7VPA9ysg+uTVKeqyzua4YLcvnnAfP+l40NFx715zc/P95uaiC0XNRUWxRIHHzffBuH+v40HHgw56k+9d7JIYtjp5MohOahubuducm4n+Wp80l6slSpzn8lPuaCfaxFws3Gcr2X+JVmH7dK+Vs/22QFuF81uKcTPvoGwr7Lh8A+FbhHHbHpWhfttXjGs2dloY9/GDjnux6hi1V2xs3BvLli1btkTw2mbVEvCa4vJFQXnl5eU5904/sPGuKGRHJeK33ebuq5bHd3d0sj2XnA+aF8WIkB0TFOeO4zLu0GF/cGIbj4uF67H6r6xP38T/6a5j8W1V2S3WXjNOn8I5GwjX+T4FetS29RvGNbe1d5vvdtwrLy8vio1bsWzF755//vnfIn+YArWsuPhfwVevLVu5fF1sUHn5Fx0PrHpvJ88r7tJWDP4t2M2zK+5xR3mMeg2OcychfblOtYm5WrhP2Qu/WMP18FiRDzFuqfYWel6YhXFrKnkyXH6/dYJxH5nqUMb9775g3AfNzeXlccWrly2laevffvf87/4N+bMUqmXF9KLQctJBXly24rniol15zfcEcgaDGeO3IMuFP39XRz1G/p0wP7kPtl4inIucsQBOE3M4WXC1cD2Ww9vPsJdm4Pk8C9+W1iKnKbD6FMrq6DNxe4Nx2feDQPTJq1PUZR1n3AfNeeXrit9Y9iP7+uC3zz///PMLkT9LoVpWXEz9Y7yM1bu9ZNnq4rjy8uYOxFZBGXmaGyxHNJi7zd/lZiKKt4zvOu7HxITR90Lh68sFyYKjsOBy4XrAwP0jX6t7ucVvSzHn18ApbiVun5mw4/INWxi3tQ5lXJscFxkOMG7n/ebyopVv4G/P+fzzv3v++X9HnhYqi3C/wqwNrVq2pTivvPkel36/I9cYEM9taWF9/uBPvJ5L/I6e3EtfFDODUi7Rl4sw7rChDsOC64X7I7SJh8dwPTzeyCf5trqsBnNeGKtPQddkX5+CM4wLuLaxsS8Y9/GD+/fjvixe/iMuhSFqCTBcrKL5i2Zc8JDnRU+Xrij+Mq753gOoSYEnV+iGWnuBsmv3VaJeSw+aGnKK1JIoMsvFWu5IXo0Il+uFC7U3/jtvo/uqPMJvq7UlvH5LMy5vnwK/4zrMuMBnW8sc2/+ATFsZF+xNjFtZvIJPWlQBwn3+/yJPC5aFcfmuNsh6umzLyrh19+89JlX3hPJgruce4Xz++COdoOcSPRo/dNxXq9XzQccCJssdNtTBNrEeEK7lTqnX+P82lQO+vaWtQs8L4/QpNH3Ek+Ha3Ytra65gaO09xu3suN98r3jLMl6fZWoOMNwfkaeFa0Xxc7QsrF8FvbhoHqTSAAAgAElEQVTiuZX3mzueoHvPUMilPLczNxf1WmowCv7ebH5yr6hZLVJiLdfBW0P0gHA9mJvwvMmHuB4eW/JvV1dpb6DnhXH6FL5uyESdVthv+QamTwHHuI9MJj7GRfQrwA7WGfdxx/2i2GLbLk5++9t/A4Rr7+myK4p/poVr42FzX/28rry5o5bussT3K0Cz4t1Oq55LTbC8NnIoyrleDt3ntyeEy3SUC6xFr7pws1pbjZ4XhjJupr0prpOM++hhZc8z7rcd94vWMfxpreb82/NEcTLcf/lnK9/nsbr4DXoNXhAVOD+b5xYX3X+AaczlNPaSq3t/4j0/HfZccjYvjvUajeS5ArsV+asnhOsRR4eHyFcsFVuirUbOU0D7FJoIx7WjT4GfcfH0gDDuo1ZDDzPug3vNsTxN99j6LaBbMCzC/dd/+d//9Otf/xPu1XBtKWb0auchiW+uLErveIL0K6Dc29W5o8VWz31kXufhsWJd7JiRI2DKdeiAkB4R7t+CyI9ClxvF97AnhnH6FCjH7ZVeXGaYGnuOcR93lAc9Z1828Ox5usjVh3/95//9v35NlFXHhYQrZCI89WNcefMDlsdym8xIZ/0oF3VbHs9tJiTxdNm6RZ7DoTDXAVjoEeF6bCH/mIT+sK5dEPRby4lhOpRuXce4bRjGbWur7SHGfdK8KM6G6zBO/dvztOP+7rf//M//9GtL/QvyWu7PwSJcIRPhryU/l5d3dOLXziwzN7XTRs9tY67EXtwSFDN6FMW5MfbfGqJnhOtBWq7gH1Ys1atwHXNfM/jEMN1/2UO4rF5cBxm31WCFcRGN2sK49/PWrcZ3blippaTf/uY3/+f//zWrrJICJNxrgj8LoXq2fGXefcph7yLYSzpr7bt8K8hczy2C/rV5tvSNxTGjiTBX6DQZfPWQcJcT7Y1CjuuxMh9zBjmGcWtqMXzL77h4vsVnuDyM+8jQ5vzZuCzGfdxcHrfC0Z0qYMXsd7/5/36NlFVSgIS73JnjlJ8tX1nezMe45F6Idytt89zHnHRj1YriGK+RDpwm1kPC9SDuCyT4t3zpfSzhss9TaLrddENnT5+CKxi3td0oTLiIRgUd19icV7zMiXsq/kgaLqJb66QACfe5n5Av2lWkdtFTcWhn7czkSXTNHM+9jwL+0jfiFtm957enhLscmL/woQ+xyBnkN+ndvdCZuA1NZxCvxfqt6xiXmyvY5rjIAKr9oaP5/kr7sZZVP5J8iyj3fyGvRGpLMf0jEPzHz7Z6tmJl8/0nOL8lZm5mpy2eew8bIazqN47rEXdNoFWBqJWXcfc1Q+76cFtX5YpeXHsYlzjRBtGm3Yz7w4Pme8VLnb4l3VMqU/g/HOFaJwWPn4upH8Eqh0J+pFavXgkaIrGe+//aO5feNrIzDfNAnOYtRCOLLGcKyM7b+QP8G9q4mKpTN6K5qgURME6A/gmexaD3ynIgRIbaEzdgps2qntbQzECApJm20yIkNnVpSO1IcSNq0yQzqCpequqcKtaNxSrrPIQ70dWS/Orle77zne/cfetQ0bV67tt+ku6AwMCVlwkXfu5YU7Dc+/BfO7iMi+Baw9VexqQHh4x7cXwcPuNea16LfNNB+NdfGJ6btyk3V1i2gzEXLh9ybswUDgB2IvSvUL/V/nz36Q3Ob22eOwh1uHfOyoQLhNKSqAD6aMK19SlMK7mvXGu4p9H14s4fxyEz7s1lS4hGKxqjn//8F7/4+ceggMTcXN5Vu8r07sSxu4V4ZvrZlMrgBue5P2mTbjAPq+deRvKlrE64UFjiuED4Al9TsN9sdqQifuvsuE4Px/vN7PlW/7N3HiLj/n3QqkcwcxtDDlFuNruRd16kKdMRLTD0VE+dxbkAvt66fot67k+Pd946+u3Mcy8jeR5anXCB0GDdk1VtsKxPYdqJ+0eMTh0065hxPcxUMPXhHp+h2sSr1J5x/3rdr/qvSnpkaE8Lc+3iqxaKERVGEX1B5gZ3UWhd3qCeqz7GzN+xeu7fI5hHvlLhsgLlbrlU/b8xfmu598HoUTjqrqYX15ZxTZ57cRws4972O5Mo7pRxYrjw3GLBouI8JvBSU+FGZLhAsqYf5peda8Rzb353s8xzL6O4L2qFwgUCt6Szo/m1U8Y1JVythrv1yiXjOtRwEZX6ybg/HAXJuFePhIgk4shwY5ZutRfmXQvTxZotNDBKWXtaHkezHMKUN8eTzrTIsHDX73+HPRdhquXeOM2j9cMqhdsW3B0XTJ5i/dZSxdUf+zuhq7hIvnXNuBfHPb8Zt9up86FLX8sp2hpsigWzdjcs2mWMjCtFtUzE/XPC5q69truzs8Rz+0m6dQfHZIJuk5hh+27zFBYZ9/R0+xARKl6xSzLukseiR6F3gmjTNeP2d8vR1CeXUjKEagq1JUvnzUK7w6KibUBQkdzkqOkWrzdKaF1ZPPfum99gHNfkuTchZ+OCVQu3XUGeXKw0XTOu+ayZeuTuuEt6cfEZ1z5TweS5Fwc+Mu71bhVnRiuiOEsKJqzazeYLQwAK2Y8U5WcbxSV5zTuS0xHjj+XKwOK5h789dvJcvZbbQj6Fb1YqXCAvSTPKmXufwmKewpHaw+XblWXc8+MLRJ/4jHs7qPtssA2LVs7NI5+jVLQs1nL5bPYjpfwzLy0N3mi7OPeIF/q3Jn99+1h189zr8Kl7tcIdL7kWCPZxNVxrFXf6OHwZporrO+O+Of7ek+Pe9puy+/e4AgrZLFpC0LOBZbGmO66nngZPyO5ZiGv2TTWGvz1W37p4boAGXBurFS7gsS0Vc4Yd9xqu+bRZt4uoFdFsmF5ca8Y9vzhCVYpk3Ku+wMWwIEMoOLfiFk2hwRBuVJa7NA3B5/1rU0V3x17RNdVyB6Gj1YqFCzruK9o6Lt+a+hTMZ812DmPMuG8ujpfNsTkbVDwenY0e1wabUj5nEa7rO3sGM1oeRVunzebbdP/91tlzK8iH+mTVwoXumzb852gN1+a4i17crYPAvbj+M+4PR28QpzV77s1lNbJVT/QM9SLZP330UTYy4WJGy2MQldbcdW8+PXT03H7YZcGqhQs2Xb9fqoLJuJiEqz9e/f7IwW8jmalwbp1FfnLk4ri3l9VoOr9WhmmlFklU8LyLMWq2pln37u2n106eaz8J4ZuVC5dy/wrrHjLunIudA1fFhjpvZsu4+skzh8dtZ+U7ZKEpLgq7+D4Gn5S9f5bRpDNzXdVW0V0oeDfkT3DlwgW0a8rVp99harh/wd/7YORcDEi+9T1TAakrHDg47ledpuu3lBDma7RIkoK4pCJvRZOukQxuf/sW77lX7hlyKasXbtu1sMD2kYxr61Ow9ijsHIY4b+Yr456f9HB+e9VR1lFH8M+7qXLRim8Qaj6/6bGya0zQ2/71W7zndsL9GFcvXFDF7xROqSzrU7A9uirGcZFMGyTj2u+U7B2gjnvZinGLLBQSB4r5fM6lWdcP0PXfEIuo7F7f/Xj37G+/6WI998pxlKcnYhAu49p/Wf4fh3kKTjc/HG73HDTrmG8DZdw3+p1nlsegxbsuNZPD2H6BsoFcrdRlzQ7piq8DGv6Cwoz2893vv/vu7qetHaznhtv3jUG4QHBbkHIDbMY9xWdc7bGvIh034XtxMRn3vNczO+5ZvymlIyQA0OBwshSrD8qwmqmLmxVuknmAexc8/xL0yFpD2NXqCtf/dotk3B/v+qGaf+IQbnsT++tvMOo49ingMq72p6d2F4qN/rzZ/PH9sWm34UkzjqbFaID4A/F1fTfoQeYhDcYPMhnv9bxG8DE4SrN/c/fT7eND1HPfuteblhCHcIHitgyvYO59QFzW2qdw0t3p4RzX+REk455P6wq62y5rLU4QbR5/YkoyPJbOZBjAZTIZz1sAwYKCDisCvj74+93NVhf13FCbELEId1RxcavJf1pquH9xT7hT9tVvUMVGnXHfTB337LIZ/J8ubt6zvMPz26ZR3nmQeajFN3r6mygrCvbS+wUjbO7whH66eDy5vPrp+23TpJupggd/Cvp54xIumNSdDYtqoX0K/+d878OspnC4vY84bhRzwyx1hZPzN+cXl/WoOrFjoO3Yw8VkdP1RmYypPsk+/IfACf9wvYeBD26MDeNDRaF1+2P38Vu759bd/tolxCPctuDS+1fx0qeAnjd79bL7yqmGi6g0UMZ9c3Hww5tBJUWyHbHSl8grZxgFEdkcbsXNB1qdq5HpIO89B7r7sRujecWLqg7efvfr722eOwjxRBaPcIE8URx/u6pfI724iMuijnt6enq0dWB1XOdHsIx7fjCoJLwlwUKj5mwPM+oZ0z+5kjEWSM2MY8OQywU0S4Em96cqg7e/+dbquTfB7i3RiUm4o+bYccOJHfio4lrpbvdWmXGvNtOzJNMCpZdfskcZU1l9M2N8g+WMU5uhiL3+3usXZHlRevTdp6q1T6zv6GZLiUm4QJqMBadJl31sn8KSjGvQU1UPvbi+5obN/1x1XDerkwbvqe+QMUdccRYbYMZBB+NaiB0Xzna71fiX/d9vWTz3OrjlOnzB0VMHotM+RMVXxrXVwXrP1JVk3OuO47NnEuHwW2UI84hbZXS9Gi9wmQx2T/cTb024eBh0cdDo/+HTG7PnBt89i024nAAYBwcTrlxPmzn77aw01nXMtjNF4zKuw2xc489u8hsXTXiRragPB2lOvZV/YOjVeBvMZFCVaYurMDvcHO6DpUe/vjZ5bvDRjbEJFzQZQOH3TN99HizhzpV5qHaPzpf24i7xXFO+ve0L8Z7bDYXIeWqBeai56tio4gJQ0UxkYhIuJh+L/CfI67zj8EW1ld3fL/bP/hrYcuMTbqMOAIN/6qngMi62TwHnuFpNbE9Vj50zLjY9OGbcs8smzn4Sysect+sdjL0yJZPZ1F5SNkV9TebmuO1QQ3VF7gXyuulXUu3Pe3R/DHxqMj7hAqWtfTu4H0b9a0+9uNiMO/PdY7V7EEXGPR9cys610KTBcF4H2jGZTBmUM7CTkQHTfKDvCkNt9xcYqkYctx1ucpPLXS2i0vpu5rmBh9rEKFxGW0KKNcyPQ5sh5i3hOjiu/jjqqt+i+dZnxr2qlzFfYEJpSJyzPOzwDzKZTQ6IzUwmUzWSEGeqKtj7G6hwQ8wdgsIUtjWY1XL72Ofg5cQoXP1aCHynaCtcxp2fN9vb7p6Eybi3/RAl8ZgRWTlslXk8Ey6PlMNguHkR4rIvjd69NTz3NuARnjiFK+pV7rGENi9VvPbiOjvuNNse7aj7FwEz7lm/in5pyWTE8DgD8MvD6Y6ZkrHNymqgkdcPo+UnfYR/XP5N91yX3WY34hQumBi6gMivo/CV115cRLNor8Lx45e9IBlXraRFtiInR1P1kDNGxNy0bfnWQv4kJA/P/3DzkTYw5G6ApGtPxCrc0TSJM/ap3fCJxyqus+OaH73u9rdoxrVhzbfn+50Qm5txMuIm2AppIAzFytYd3xeTkDVsb08GDN35w493PwYcahOrcMHs/qSxveXmSYCEi8m4iz8H29vHp9799qJVTcWa7D0lRzamGehTCTNVpZKpmn8TGL8neu0wHv1arAud27u7YBMW4hUumEWpEWcNUc0z55lh3vwWmalwsd/d6r72lnEvW2lICSNODnUlLxZ2olhzhxS2iE15blYclwXhs7vbQMd9YxZubZ6lKMteRPmpr15cNONi+xQuDrpGmcHdc7/YTcEBhxGUIHa2aLSIodd8op8eD7nKdW4CLc9iFi5YNL23JfZX81dz/egSruVxsdd9pn574pJxb+tKTEPwA/MJxXEcE8PJ+BEMfQVlG7vF5EijzgmPgvhG3MLlTGU7uPgex/VwVVxrn4K1jnux/436TfcYdVr9MWgmu1X8PcNyHGRiWTiKXOjE5HufWKzzgerRcQvX2IWYwkjz/RXl2ncv7pKMa6mS9Q7UbvfgDMm415eTJC/KxHc8B6m4vkLWZZvWI23/n2IopCHjAsCbG9nG7GyNXBugGTdADddhFaZ78HG32/3m4MRcS3iS4C4wkYIcZONrrmzD8Be9fyL5/XpHDVh+EMTnYxcuUCxaEaemy7ZWk3Ft581Ous+evTw4NjLuVSWpJ3PEBic7TEdYESMYwXSpdtmPbiFXFqq0oATzjviFC23t5KzxWzoIVcV1yrfYOu7ey2fdveOz/toG4bsxYqAkc3E/ETBRbMWNXY5yW96P5cpKlRYmYaok8QsXIOPWeK3RXvDQp+DmuPh8e+qUIHpPOmU5pjWPV0YUV5MlJv7QjWsf8Y+4/LyECOUqTdd5Nry7r0G4FNKCKU7giHsSRZ8CqlB8b9jZ7kSPkZIsw3YCJjCKFC+vRbNa3QLKUfy9DP54yxQRSjRNN/nwOXrKGoQLBLRu2pD51moSLnamwufVxZOa2OBlWWbjKJPi+BUF5do6f3satUhyicOxLN1labpShxFH9nUIt425b/I909mLK+Ned9BdTbHEQkmuOcw5XAVtyEk1iWdjq3ZhoSTKaWqAL9COPwDelQW6Xi7Dd5H8DTbWIVwwQYWj7fpu7S/JuBjmyvWecftNxzWEyDSgxMsyx7Erk9OYhcbfwKw/o4i1RjRfg22+GKs06bpS41yPQYRjLcIdYywXwP7/drf2V55xr7yMsGszFAshB3UaTNjsKTL655O0T8lx1HqiLMJ7iofRyFac3atOQVlRyoJSgysvi6xFuNgOJLHzl9d/VtXuK9Rpo8u4Z33Bz7/VqM0wFAchx0Fe1zHHUgzDUKKr9ESRYXSp6h/D6R+nyX9dMRrLiOKg63fhHe14WgPWFA0ZoguYlbAe4Q5x10I0Na89+ko9eO0/4XrMuNf94H1gQ5EpMUyDhRDyHD+zYw1e/+/sFRxv/H+WYiiGEZN5acToXWSyfSdVNcWWpbgka7Ae4YIaRkC1r/R8e6Sqe5h8G0HGPas3k94HFhMNPgLZjhvcRHdZ3sdZ48hYk3BBBV29s5/NagrdrcNXSL4NmXF/OP+qhQko95D3bNgjZV+ysKxUq8I6+0HXJdwGxnI7C2/t7nRfR5xxB/U0TQNbGSIbZndX260VqrSirP1nuS7hAhq9G7ZpruIebHXPEJk6KXZ5xt1vYX5T7h8i57t/a8oI8mWhWhUmCUlbaxMuhw6C4J9aenH/rG4dLPFbzxn3Mh1nIVfL+3e1IGM+RFai6UqVj29vxgtrEy7mcupG33be7ER9rL6KIOOedTD7OveNMT/x2ygwnDUYIG9ZP+sTLoVOO+ogqfb10c5O19lxEXfFPS4G1ZRcZLpC2InX6Xg6DaVZrSrlEBeXrJr1CRc0kUUCjetTeL2387KHSNaiUOfzZtqew26qRouvAlHy3pHQ0HdrJ5BK+HiUNQq3jTRy88/xfQr76lbXzXGdz5udnj9tIb8f94sxJ3sr2kLll4ZkkbckkTUKFyA3Gootpz6Fiz1V3Tuy59vlGfesn6KpoStAZOHS3QEm7t3aSFincBuC/emogmTcxaOnafc11nEdH/d7z+EFy3ENt2n4LJwogqIoMe/WRsI6hQv+ZF/sV3toxjVxvNNVD05QxTpl3ME93uIdUdC5N3Pc4AyXjWLrdz2sVbhIYQE+xWZcc1Whu71z2LM5Lj7jnrXubRFs3KjJ+BrWCMKyQtNNBf/m9LBW4c4G5s4RWw4Z1+y7F93tZ3snS/PtVeue2u24USvjZMlyE6FOK4kfOOWN9QpXtLc3Yiq52Cquur3dPXLLuKcpGoofJSIrSfY17wvIKU2aTsxubSSsV7iAt2381s/cMq5lVXbU3VL3L07xc8PONnGe84HzvsRPrKO7RFauV+gqF9nZ2uSwZuGObJYLr5wyLo7X+zv/ofYwGfeKvnedYCKUZdMBMpHTGgyaUUwwSCZrFi7grHNtRn2/5816e6oxA9eUcVvK/drj/RJKMmSme2NUWaCryiTq4+BJY93CBbbpILuo1zo77izP7ne3t9WLWb59/ijZg0MjhpVlidJ/UVlF0M7WwsaHFwxQ1i5c1mq51TMPCRczU+H44OX29jdaxr1PDeMNTua0gY7QOA7ufiveh8XahWuaUa4xGWDyrYPj2ntxTw67L9VdZDTZB8mIgjzP8ZKsb33JKWkwiJD1C5ey9NqIT0LNVLit89pJcvaDLuFqk3MVpSkoilBOV4NBhKxfuMD61N5CvNZHN+5T/XO9YPSJBuudbbQaRArKQp2uC1qDwb1uH0qAcEXLVkHza58zFRZ13LMn5rw8ZnmZY6kPpr4wZPhyhaZpoXzP+zQNEiBcUDf/S0yeom7rLeN+gelNEDlJ5mEj5cs1EUrKwwot2K81vM8kQbhWy+0HzLjPq05ONGY5eQIbSZiD65cxa+zW+ppRfy9IgnCBYC6WX6Jui/VbW2LoL9t0EN9JNW0ObkqKnJ9Afbf2nidZZxIhXGDe+BVuEYW6KlbPuBcdb2MT3rdLmnw5Jrn6HTFwQtN0nYtoAugHSjKEq5jOM04GiN8unalwW/VXFBoyDQglWeL8XYO4WkRWatKVijIJf03ePSAZwgWmcgDso/l2Sca9rAczpzFFQchJsgTXWcAXIScL9cqDijK5T1tfIUmIcCcLyx01Eb9d4rj9MvL5/DFmmAavzQmX+NjOsogcx0m1iaRUNjcr92u3NhISItyxaaNW+U9Eog6K1R+3fbQKFvjLYCjGPO/2HUOVGNHtvKFXPmlTTKOhD3rmIa/9lkz0va84hnd/kCREuEBaFAUmAz+O+7y6mkQ41ubfNziOg8ZD0v7Lc5BtNBh9KjnFiBjaFEUZb2eNKebax2gfq3UWsBTFfDyk9LO1wn1sMIiQpAh32JQ46kv9/8K+e761ZNyBEFOFcyRqY/UZqsGysznkhqLn6DPz9YHkUxqagBlxUb9oQGMSsuRrHFIUFD1+jiI6QzOhJEW4gBMAK3Pa4J9x07vjho638TDSJhhU6aaypiRbyCOvwlPIpUW5iREuqGrOxJY55n3TvZK7yLdnyZ/3MZ4O7y6Hubd2SqlULBaxf2zvOMzncgWTAj3rVnvXlCg3OcJtTDd+odT8HLl5B8P56W0nySFxqN8OXo2wwaCYdSBnfb/SRh6UcguxFjZ8iDHvXeRrJTnCBfO0Cj/b+uNhzzHjzuaGXTUTuvs1hny1QlejP6iYw+s2W7S/W2m4kc3OXizZ3+7KcKPg9ubEkCDhzi+nftE8fX2ws60euTruZwkcnDC9HZxb0dnaQjabKyLYPbKQ3dDMea6/DX8eWsxGkGpWT4KEC5TZT+y5rsyjrrrTdZqM29tN1olIpizQtLZbu9JngVwWo6qczVAL2TwAheLiRfRDXMmlIiwkSbji7KbUyRdTp+3tqY/VfUS6p6df7CZmP1+fhCxMVnlv7ZwCkme1QoDtFRtZi/J867DgK1qsiyQJFyjTxZbWrjDPt/t7O9vqK4vjnj5NxFwwdja8G3nLqtCiq11VGzZDHVrfpYh8wDKG2TSk3EQJd2a5YsdaUzg5UFW1O3feV5/7upA3crRJyGvarc1ns/ZEazfUojVO5LO+61u5DeRVySNRwgWTaV2WRrKBNrFG7epDyc+e/GldtUYKymXt3tq17dailmtPuPrazASaLZaST0NWSJZwx9ObUsv72Cru0Z7aVdXqOq7am01Cltdxb60Je8otWnVZKORz2Vy+MF+bFa3P+wVta8L437yjOotpyArJEi6QDMvlLh17Fb7alTnIxqgeVrtquZyU4+B2y7Ua7rBQKGxk84WFcAsWEeYLQ1DcyIN8CYC847ZEKYBLx07ChAtoPb2KHazjnp6eXuv3+jGcLMFVnzwfNzhteLegJGp/rmBJuUjCBcAqbMvTfqFgvCpf0CtrjpabTUHITZpwJSMHtJCMa/B8cTptDCWZo9pe7+/yw2jWYJDAg4qa5S4WX0jCBSXr2ixneqlkGGkhq63XNrLO/TQbWeRViSNpwp3elKr0sI6LjBlva7MKmej0pe3W0nSVh8k9qGhOuQXUGgtZi+rMGswbqSGnW3axMFN0qWhXfy4Fm2eJEy7UGxXhFS7j9rHLsvcM1CZthlTvCErabi1M/CRkc8rdQAzX2DdbYJbx9A05S+wtbhQKBZtx54lwA6D32jAt1HHPWi53m75gYK0mUUEWbW0o0/q8+RfImxLJopaLbJrN/XROFnnWL1nCrbE/UbJ21uSc429iSJ5wef3EbweJt18sv9t0RFEcL3Oel/8NbRKyMOHeIW9JMgvLzWHKVrY1Fypca513uiFcsGxTpKGQmzzhAkHb9aftfvu06rUZoM1ATpbc212g8s90Vamlc978LOUW0YRr81OccPOWqsT0haLFp4njBuKdZrn8tTXjfu5zXvOoTRlHwBqMObRSUJ7eW5vig4ozy0VLCvqGr+XlHCLcDXPEnTfUWEq3AbaJYyeBwgVlTgu5Fsd9EvAcgTaxhtP1y+pXLX8YZ2uNWi4u4YKCbfMAedafW7J+tmfurRa9ozadPJIoXH1GuSXktmpuz/tLaMBys1oXFAmmokPaA0YtF5dw7Wszi3BLQ5MlD3WBz4u+loRBds4Coh2GrSwc93W1FuzzsPpxcCGVt4O7oqVcTA3XHgSsW76FrNYBmZ+q0ijqZheOuxBu0d6BlkQSKVztplT5epZxzzq+l1DjBqcdBxc+kHtrETTLxdRwkWZcLRnMRajbdClnCHe6i4YVboE02QSlBkFjNif3bNOP+rR7a5VqtVn+sO/0yGOOQgDM2gyAjfn75TaKoJQv6uux2Sl0rHDTsP+QUOGOaQAeTbvBKh63soYsT9MVmof34Xq6oT0STCkgz/KLCu0wn89rbWHFXD4/m7qQX2TcRSEhh/2dSBjJFC6QIKjofntFL9ftiOPplRwHTzBFfAzNI3p2b66dvf/QtCuRioibVOGCh0C5PT19dVVH3mJGux2c/jBvBw8Gpj3G9XT6TNYlk7wLKajiJle4/KQxOBOwJeEAAAH+SURBVH39xPFiB0Zpalctk3nzc4q5ErYF3PW05HAagU1Vs5RMBEmqcEFl1D/tT5BXaxtrxtnadDUYrJyStlzD1gPyqJgXGJZb2gg4sGl9JFa4rFDpW7rBmOluLZmEjCWv1QWwGi25Wmghly/kTXsZJZz2E0hihQvqzdmkcQrW9LO1RLIuDDeKTkNCi+4eOrRMeyRD78LCKNoV4bOemHt61bIPSosilx0HQeNIi24TLFxQX8/w7g8RzJlKPD4m6a6ZBAuXEB1IkcyB9LQhEeESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOESUgkRLiGVEOES0gcA4P8BNckzHBrWdmoAAAAASUVORK5CYII=\"\n",
        "height=\"250\"></center>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xvlDMc6ocKuz",
        "colab_type": "text"
      },
      "source": [
        "As the picture above shows, this vector gradient represents the slope of the *plane* $\\blue{d\\ngap f}$ that approximates the surface of $f$ near the point $x_0$. The gradient points in the direction of steepest ascent (\"the uphill direction\"), and its magnitude tells us the steepness of the plane (so zero gradient means the plane is flat, and a large magnitude gradient indicates a function that is increasing quickly at that point).\n",
        "\n",
        "> *Note*: You should remember the important fact that the gradient of a function $f(\\x)$ with respect to $\\x$ is a vector that lives in the same space as $\\x$. When we use these gradients to train neural networks, we will take the gradient of a function $f(\\x; \\th)$ with respect to $\\th$, and hence these gradients are in the same space as $\\th$. \n",
        "\n",
        "> *Note*: the gradient points in the direction of *increase* of the function. For SGD, we will instead be travelling in the direction of *decrease* of the loss $\\L$. This means that our updates will simply be based on the negative of the gradient $-\\idld{\\th}$. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8akunSUu8n8J",
        "colab_type": "text"
      },
      "source": [
        "### Gradient Descent on the Loss Landscape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PDzPc8T5FWhj",
        "colab_type": "text"
      },
      "source": [
        "What do we do in practice with gradients? The illustrations above talked about $f(\\x)$, but for training networks we are actually looking at functions $f(\\x;\\th)$, and considering the gradients with respect to $\\th$. \n",
        "\n",
        "We will use these gradients to make changes to the $\\th$. We measure the gradient $\\idld{\\th}$ on a *batch* or group of examples, and then change the parameters $\\th$ by a small amount in the negative direction of the gradient. Repeating this process again and again, the average loss $\\L$ of our network on the examples will hopefully decrease. Eventually, the parameters converge to a local minima of the loss and the network is fully trained (in practice we usually stop before this happens, and various things can prevent it from happening even if we do train forever).\n",
        "\n",
        "The diagram below shows an idealized picture of the situation: the two dimensional space of the image represents the space of possible settings for the parameters $\\th$, so that each point represents a particular network. The green contours show the loss $\\L$ as a function of $\\th$, pale green showing high levels of loss, darker green showing lower levels. The red arrows show the (negative) gradient vector as measured for each network (blue dot). By updating each network using the gradient we obtain a new, lower-loss network.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cd7nvz2fMfB9",
        "colab_type": "text"
      },
      "source": [
        "<center><img src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAu4AAAH0CAMAAABVZliYAAAAwFBMVEUAAAD///+0Rkq/Hy9/fX4Sd7vH3+8+ksFfqtKIutIVhYIOhFsMg0ELlEQZiUcfm1AarE8srFpBs2m35MfF6tLV7N0wk1JbwH3i9Ojw+fN8y5WO0aFGn12s2rZwtn2g1apdq2t/wou53Ll/uX6Yx5SJvX/G38GnzJyEuHOEtnOUwYRaaFWdp5mBtWpqklmbxYmx0aOMvHY0Pi/R5Mi8wrmNnIWWsXattqPL1r2dnm/Nu6i+ooupfWLBbmnv7+/Pz8+04C5lAAAACXBIWXMAADddAAA3XQEZgEZdAAAgAElEQVR4nO1dC3faOLd1UrhpUx6xk9qJcYxNCKQQKCXNtDPtzPf//9Vd1sOWLRv8kI5k8O7qmmlCHtibzdY+R0fGRYcOZ4OO7h3OCB3dpeIO4XESw3Ucy0n+OslnHvFjT/hiaICO7iLx5e7uNqJuROSbBoi+3p1MJrd3d19O5+pogI7uTXF/d/dpMnGsRvw+DMuxJpPp3d19u6+UBujoXhPIojTT8DqIVH/aWZ666OheEZFbgWc5j8j233a0r4iO7iVxf/c4cSvT3LKc6E/o0j/u43Q6Tf66yWcc9EiL+xZH4DiTaedyyqKj+1HcPU7KstC0LCt0XGc6nU4Xb7WxiL4++j6WVZb+jjN57LT+KDq6F+PhbjpxTI5ZHNMi9Z5Mp7P6/D6M2XQ6Cd0yym867uPdQ+ET6tDRPReRonNkygp5pOLSOJ6P2XQauZ7Dr0GzU/oidHTP4OHu8ErUstwQnOU8ZtFK4uBbj+VOOqHPoqN7gvvbQyYd8Vw5zbOYTd1DYm86k9tuHZugoztCpOlFTLcs191px/M0ZlM3LNT6TudjdHS/+DItYrppOZMpRy2NMXXdIqk33UnXj3DmdH+4LViRmpbbLqKzmE7C/Gd140xuz1vmz5fu949urqhbltskM9cFi2nBStZyHs/XzZ8n3e9vc6luOfPWSno+dm6uUTPdM6X8+dH9/tHNUT3Tck+M6Qnydf4sVf686P6Qq+qWOzkB93IYi2men7fcx/Py8mdE97scqptO+HiQJieFnZsj85Z7RhXYM6H7/dTl77Mz0TxNl4HZPOdVfza+5hzonuNgTP/0/UsxFjmVBnNyy12408Op0/3+lrOslnMKQWNTLCa8CJx+XnPSdL+bZO+o6bhnaGCKMOP7m63Trr2eLt1vOa53VM/BjNuiZbmna2tOlO632Wzdck42Vm+OaZi5WqZzoow/Qbo/cHWks4xgqoHzNaZziu01p0b3h2zPl+lM2kQ7lZhmnbxzckWo06L7bSZdt+ZnVEQSgVk2rjkxH39CdM/6dauzMHWQXbuelI8/Fbp/maS57pxzGakpFhlHaLqnkk6eBN3vp2ZajzquN8Uio/HWp5OoQLWf7g+ZuqnVLU0FIaPxp7BwbTvd79JpgtXpukhkNN5sffNkq+l+P7U6rktGxseb7Z7j0WK632aUp8thJGEWplSlzUlNW+l+n05i3C5fl4pZyjNak7ZKfDvpnhb2rm4KgekpSHwL6f6QuvCW2xl2ICzmqQvfxqCmdXS/Y/sETKczMaDYpd9WW1d9ahndUy6mS2IUYDFnb4HzyN0irdEmuj+kNmw4XRKjCKl1qzlpk6dpD93v2Q4wa94Ju0IsWOEx3fbkNG2h+136LfRsiaYNUi7eaUu1tR10f3RYMemEXQss2OpTS4LJNtD9MeVizp1lOoH1NFYbVq3a0/2BLZ92LkY3PPqtWrVqTveHCeMQu64YHcHmNNoTXmu6s2FMVz3VFguXJbzWMY3GdL9n6qdm1xajNdgd3Y7GhNeW7izZu5FI+uORIby+QbymdGfJbnVkbwWmjv4KryXdH1hl79anrcGMIbyei1YN6c5Gjx3Z2wWG8FqmNNrRnSW735G9dVhoTXjd6P7Y2Zi2g1V43SqtetGdWd53ZG8vGMJbevXS6ER3puuxI3u7wRBeq25JfejOZI8d2duPBfNGrU8qqQvdmeyxy9lPA8wOeleXNasmdE96fDuynw6SSbXmlLvlSqAF3e9iHeh6Y04LSfOYqYWF14Du9/Gyxgy7rscTA9MtqYOFV053pqPd6ch+gmDqTurLTqrpfhv7GKuLY04USSppqk7h1dL9PjHt3Qr1hDHVxdGopHviY0z33Alx6kgOKlbqaBTSPcljOtN++kgsvMqMRhndH+Kn3yXt54FdLG/qqk6q6J64uc7HnA3iUFJZp6QauidRe+djzgmJo1G0ZFVC90/xq7zzMWeG5F1dSVuBArp/sTofc8YIY4FXcBgCPN3j9PH060qL6WQ6Dx3f8U0L/RkPeeDPWE70uMlkujx5dzeL5W7CkUM2oOn+hbq3U43aZ9Np6PqWZ+VRuyTGnuW5vjudnqggzKmjsaAFHpjusbSf2g6O2U/X8a+9+hQvwrXn+K+nxvtE4IEdPCjdk0DmZJaoi+ncDZooeVmYnuNPTsfpxPMmYCMaSLrHezisU7hrj3M/sOTTPIOx6TiTUxCLOJMEzeDh6B5vz2v9Fo7dxA9Mjoh5MD3TC5zAD1bhPFyF8/mOwzLEn/Ojx0WPz/k2PEzLnbR91v00NrZwRVYwut/F715tlvZJaB3jo2lbQeCHYQ61y2Iehn4QeKbNffM0rgO/zUKf7N4G66KBonv7pX3pHpR0z3SCMFzW53g+5qsnJ/AO/WDTCVvL+TiigYokYege97W3M2ufTqxCwo09KwhXuVQVinAVBLm5PYIdhK30NnFEY8GsWEHofktfxC3M2qd+0XrUs/wmjqWm2s8LWT/23BbK/Bx0xQpA93iNau24J6s1Fj+dfPdsW4F421IFy9AvMDhm0LaT8+O8DmLFKp/ucR21VYcrLSZO3pp0bDpPSonOYh4GXp7Qm36rzhSPI0mAGqt0uscv3ha90S7DHFUfm4olPR/LVZD3ujTbdMAPrTnJNzSy6R4bmbbozSwMOO4MzUAfTc9DpPPcLz20WnMyZ7xidTkCiYVcuseJjMM9Qy0xcXhz4AVhDsE0ROjza9ix/7MdVz6WRbkJjVS609JSK3ZxLCZc68vYg49emmHuBxzlg1YsmmiNVe7GbZl0p8+gBWH7I+/WLb9lVKeY+9wC1mzBsT8ziJKTPLrH+aP2YfvOzYZ6XlupTjF3stUCz9G+DhVHePISSWl0j2275l0DP7J23bZa4tWPIcz6mnGgOeNpT4E8Ay+L7tS2W1pf4ZmbYYTla53AVMU8zAQ2mrsaejaXNAMvie70AD2d88esXx9bT+0iczmEGSevNePjkpOkBF4O3altD7nnowsWk7S5HTstd+uHMM/YGp2zmnjFx5FKBGTQnc7DM+fck9EE03TmOA7cA2Q5CcyD9FuZo20eT7czS2mhkUB3ukjV1bbPwlQQM257ClMWSz/9Gte15kpLrDJ2sYqn+xe9bfs81SMwDgBa1fXBPP2uZumZmtFdTqb4ljHhdKe97T73LDTALEzd7uBMdJ1FmvGarlupGxZ+2IdoutNIRkfdmKZWp95J5jBlkDZzgY63ii5YRQc0gulO1xn6NcksUjfZPK18vSqWPrtwHWu4I4T2BAsOaMTSPX5Rcr++Yjz67O31ztDEZLFi3+rG+g11o/skHI5kTSCS7rRLRrtF6pRdnto6btJQgaXPvt9ptx2ELliFBpIC6f5AF9SasX3O6tipNMSIwZytuOo2E4Xy3RLId3F0p8UlzXZyTJgbOj5vx56HVBhvalYFd4TzXRjdaXFJK7YvwtTNzLndHXZsG9lYr/YCyndhBSdRdP9CV9Lcr6wOC7bf8Rwz9rJYMln82NeJ8K7ggpMgumvI9gXTyD4OOhdzEEu261+n0tNELN/F0J2yXZ+esEXIWvaDt7oDgq+nwovluxC6E7brswObVfbzrZ5WBVuI04fwU5F8F0F3ynZtiktMGmN2lr0CmFWrPmdnPQrkuwC6U7brMgGSSWO6MKYqmB1eY12sqUC+N6e7Zto+Td6RzbNq7hUFVuGX3OVVAnF8b0x3vdj+mFRQvU7Za4Lx8KYeIY0wvjelu1ZsXyS9MZ1nbwJG4fU4W0gU3xvSXSvfnnQ9jjtlb4hkATTWok4uiO/N6H6vkbYv47fgLmcXgSSHH+sQMFO+N+snaER30gOpA9tnsWkfB+2nmh5IrKEO48ceRfSLNaE76YHUge1JDdXp2gWEYRkkb5jcJQeHCL43oDvVdvW+/Wf8vmt3K1ShmJsaORoBfG9Ad1eTjXqLxMd0K1ThWMVSEijPaB4b7+erT3dXk66wSeIwT4xpmiCeuDdWvvtj0pTvtek+0YPtyRLV60y7JCS7Hz3VVadJw/kEden+qEd/uz/ufAwAkhRetcATvtedP1OT7rda7NSbxcU/5+QZpxhxRqP65CGyn6/mfLF6dP+iBdvj9LHrGJCPOZUW1Zmk06S8WovuD7iYanG/CiRi1z7upB0EcY+GYgdP+F4rjqxDdxq4Kw2m4qy9k3YozONeeKUTacj8mVrxex26kwhSJduT3seuPwYQ8Z5IpX2SC7N2PFOD7nRxzP0acHikl72rosIirrIqLbKS+ZE1DmCtTncSQap8um68ajonqumB2MGrjCQndeOZynQnJ0gqDNzjpoHOtavAPF40KTQ0xE9Xjmeq0p0sUxWGUbvYP54f1fQAXTeNFfrZmvFMVbo7qiNIGraPu33XyhAXWRW+x5N4hiPoYVSk+0RxBBkbmRZ3yCzDebiK/rb4KdDboK5LclFruVqN7qR3QFmH+yNNftvT/PgaPgdB4Jm2bQ8KEH3OC4IgfG1P4w9dsZo/uJsExYU6y9VKdCdbU5UVGSatMTLLMAx82xsNB4NBtb+jiPlP+is/XbGqG740qbF5tRLdLbWdMmELjMzy2Q+KdbwKTDvwdT5XJ94Kryy2cKrb9yp0d5UuUxeO5onMKgjsUUUtP/p35AW+rhYnUG3grcr2vQLdiXFX1B80o2Ki4c1fhoH3ubxw2wTcJwoxsoNXDYWeJjTXivg+M6va9/J0J22QiqqpO8J2W7e7Pg+C0SGNRstQL/A3/tfVt1ysos8FgRfY9kGvb3u+bmsW2lOgKoGfVE3fy9PdUVlfoiOsTa3Y/hwUKrRte17gP+XR+zCeNkHg2SPuG9LvG2j17rakXfCKVNCpuHe1NN0nKo07bZLRKH+cBx6nv0iD7SD4+nyQ0aXw7AeeOea//3AwsHVy8zSRVBTXVbTvZemO9y8pGghLLqk2+zjmfp78jmyvjpwfxhNa/3IYedocrUb5rqZlbFateaYk3UmrjJqMNaAJL3epVeApx6uPPG9T4MxFYOUH9pjTeV18zZwmZtyNgwB2HWWP1i5Jd1dh4k4K1jqcxLH0PU5qoxWkPKYniHJOXuR1OBiZVpwC7tZBwKmy16Mc3e/UtcosPF0WqUvfyvpoOxDuXg7ja+Bl31s0YPyS9HZYSgiCjccdR9o8lKI7sTIqVt+La3IlFd/Tpe9xGQkw1SmeOJW/Vs14GtAoEcRphTSyFN0nynZ0LEwtIpmsXx8FPsdCUHz17JTGj1QfzUMcp6eC7xXsTBm63ynLIBe2BmyfB6OMrAvIGZvjOTBTv9bYU7qUtxTqe3k7U4LupJyqoHBGtV2lcj3ZrK6PpCYwVbEK0j2Xnsq9u746fZ+VnsRRgu4TVakM9e3qbuI8FcSMbMUWJg9Bak0xUhjH++r0PSxbbDpO9y+qrIxybX9K1U3tTQ7ZtICf/j2VXS+F+m6VLDYdp7uqVIb6dkXavvTZHkc70MjD8FilwppxoCioUafv05Kt70fpfqsolVGr7csglcNosTY9jGc2q1HladTpO05nphx9K9L9QVWBSaVvn3spYT9IM43gsxJ/rYTwVN+52ykbeKf20fD9GN1dRbtTA3XavvTYPLsFwp7gma0PKFF4X1X/zLRU+H6E7qoid0uZts8tNufgCKU9WBevIon3VfXPOGXC9yN0d9TM+nVUafvSSzy7p2HsWAZf7eQ5KFB4wnfwfUB4J9+R1ephuk/VrFNfFWk769ntVrmYNJ49lR6e+FBw1mBp/sSRuDTdcT0V3MpM1HQOLAPG92qdOx7HyktmIoD3j1mK9jeVWK0epPtESeQ+xe3TFncZ5cJnushbTvZvKIpnliDAl5L0R0K3nUyP11YP0f1eyTp1gdnucRdRKkI7yaxPgOzfkMInOTzwCANSM4Fe82E7c2is2CG6uyrmypB5MrC7OeZJnHEKyk7xnCg87Cz8JZYs6Pkzs6NzCQ7Q/U7JOhWHMmNQtgdJT2GLF6h5eE5qCKCOZqkmjjwaRh6gu6WinornQIKegf08jtMYRfuTZOIpfuP6DHlR50riyMUxeS+mOxZ34HEKU/AIMqmh6tjeKwJf4554yNYxMr8WOOdwj8h7Md0dBeI+A48gw9jdtrCCWhZx1/4IUEdw/D6Gnf++OFJrKqT7rYpmGRM4goykncwUOKEVKo9V/DwBh4UHKrrFJoflvZDuOLPnvp1UQC9Tn6nqjU/UxyTwY4EHc/BkHAewfT8s70V0f1Qg7qSaChWZJT3tJ+xjEsQ1YzAHT8YtAZPo4AzsIrpb8OI+G4P2hc1pIDM6wTwmD3FGY79yF0MOSLcYbOXmoLwX0P1OgbhboMtUn+YVXg4zThS0vgC2YsW31ONutUzMD7n3ArorEHecXNncJZOCJe0ZPMWovRhP1MFDrVjHCjZ7mAey93y638HP+33Ey1QY4z6n+zrPwrWzoH00NsyFXipoFjsUzuTT3QEXd9IY9sRdMBl4pyIHFsg8b3x0Go2dOpIJ/dPzAs/3wZoX4ogG5lKTKjlo+eaAe8+l+z28c3cA2yADuKw9OoPDK3ken6hzP45gRXs/YQx8AJ9GToobI3PpDi/uE7jEPbbtcteoq43ncfPYy8C2A8kD42mfJIiBJ82RoM0EZuEu7Ty634N3y5ChMhB+cklnPko0Ml+DOudlp8/rkzlj+CuZVwByjKECO+MWynse3Sfg+7F9sAxy/lnuVtRVwJ/vURfyzgWh5WSQEqsFns7gzpm8bU15dDeh+9wfwazMnOianE0cG8/mdJquE2x763nBJtgEL98JXqJ/bbyt59mFXzeWo/Ir0gc6AnhDJXZmx912ecBunCN2Lt0fwTcxmVDlVLofVUL+uMo9IW8wRuerfi+B4vNUPRnbCT24HklsZyA3geJtTY8ctfPo7kDPtw6hrMwT0Uzhs3xXvs3p8sj2NpsyPE9jE+RpvQTG03wKgO8O+CQOpyCL5On+BXr8AI7cAawM1XbBhdQVd1bSwPY2LxyRK+AlR+iFDyGmJVb5fCd2BnA1OC0oNfF0d6HHDzhAVsYn83xfuBvfBBsvrcMju5x3KaXzWY0X6+OfxlD6/gS+c7Ugi+To/gC9UMXrVJO7RKJBtH0kUiNXmRjG9mrYl0MItul3DrFTQVY2lL7b0OG7mz9iiaP7LXQK6cFE7sS3i2S776W8uhc0MjBFeAnS6wLvK/eL1MbKBtL3JfRqdZG/WOXobgEvVH/CrFPFa3vasXvbAraKEfmUkxc4cR5M3x3onR75i9Us3e+hF6omyDrVF63twTiVm3AElcB49jwRYdkSlL7j1eqYu/3SMM2trGbp7gK3y7gggzbmgrWdPWnV9qR4GB4v7E8VNnqe6rvsqAAXzgE7U3IXq1m6m8CN7mOIdSqppYpie2AzOYzgtelhbJisZiQopqH6Lnv1ZAPPJXDz3EyG7nfAFVWQEHIpVNv9RGNHWyBhT/DCZEGCFJ7o+0iyoQyBO4FneWdPZugOHLovIMSd9ECKYbufnG9nAzj2PDDnL9lCUhqi77L7Iz3gzkgnx81k6A68ryOEEHdiTkW0QK6SNAbWxaQRCB5YTBokJW+uCYE7I92cPrE03aFD9zHAFqZAXC010VVVyk6xSXIaEY7maQSxv8mEDWcWORNn0nQH9jKvAOL+ToJl7hZXxrOdmGaOf+DYxh5eROs+2eAkd/9qCNwp5vAjCdJ0N2G9jCnfuc9HouaExb0xOpD9e8rDC3h25D1Qbjzjwcr7hHczKbrfwXoZnLlLPUdliRW5+bbURNqhYvbjePEECjz+XnLrfSFs9r7g2yJTdJ/ANhCY8p07nptlcze3KuKZYypXqDw2NCcSkEmOAc74MGE7Z/hsJkV3WC8zke/cn7DVbqp98TmlI49jnGLEpy81niKywt/nnbuIAhHCNkbylSaW7l9gvQwqMY1lXt3lSMjMgSeqobY2PibBC81oGmdPX/H7hFQ7Mwbte19wlSaW7lPQXGYmv1vGE2Lc41YVTZaoWWxFzUQL5KfvPuxIYCs7kYCluwPaL+NLb4XE89vH3G2tTAI8S0Ar185iQ3t4mhp4/H1kKtASNot0s1EkQ/cHJP1g8xHGss+lWY5EVFNpIqOda2chaDAarq5KtTMBaBaJh2o85NL9FrT3Fy9UZV5ZT0AmvbL1ytqLsKFzwZotWPH7oUw7gw+gBLMQVqawytDdBY0hA9kpJD5Tr1kGuSK22Ba021oeNvQ3bcZ3/F72zF1McTBBG2eyUSRDd1DrvpC+UEV61yyDXI3JzDENE5ksXmwR+v6MsiyZZ0qEoDM4JpkoMqH7A2irO35T4y6GOOCYoZGVodqutW1PYIvQd3LZJN6YMWRldZYx7wndb0FjSFPyhuwl6pUZcbezCtuJtmtu2xMEIvQd1RhkrlY90OOaMuY9oTtoB8Gj7HYZnFU02cFMtb01bP/+fStA333Z4fscNHr30+Y9obsFOYLAlVxRXWLPzd3MCmxvm7Z/F6TveDawxNZIUDczTyfvCd1Brbst2ct4jSN3u3Xa/j0J4Ju8zp9ly7sFuUcbm3ee7neQqftC8uSwOc6PuVtZHl4Ltf17ou9NlugBqlfIc+9z0GzGSjUBx3QHte4TyV7GQgrV4C09kKLt/saLDiaL/0Sz3wPBvQnEvzfgO160SJR3UDfjpNpmYrq7kJNQLbleZtm0nkr2bopj+0uwtbmx2BTRJPiv3JfUhde4P9KTPHcmgGyLTJv3mO4m5EpVcqf79aDZWBm6M18M/74G69FgsB8c/jvyRA2tIWNj6j/9lWT3jitNHCnkYJoqNFG64yIT1K8gt8aEM/cGzh3P6rI5ItXAZpt7/kw+BM3Lxv0zDZ5/IDl7R25mydFCDlKFJkr3O8gikyO3GdJv6Nzx14+ai+3XoPIpfCIGk700te9Y3uWZTQ9ynlhqrUrpjlaqUL+BJ7dfZtQsc1/hPVCNhXbDnTczGEXn7222mwD/DbztiHf0XuN+tA3el1T/BY/8v7zOmRAyikytVSndHcCTgyXHkFic62futhDjHh27wfj10cjbbvIRBHb6sc3bjZtu48LZu7TFFWgUmdriQekOWVOVHEOi3Lh+4y/u+W5o3DdrVq+jM/iOINhmTizgvmUl4H1J9XsobLl972PAsw2meXSH3JXtSG11XzYzrlHzQNRt04RsPuNQilWdQ+qMjlGz3wDvS6ptZ3DdQdpi1QKMIhdsXZXQ/QtkMONJjSF95Fu5G1jlRg8bJe7JtKPBwC7NdQz2BJxtkyzeazg/Eq1fpM3Q8yHnzZjMOAJDQTAj17p7jWwrDiXGHH3KIxjFHtzm6HwcWy/x+03Oe2oo7+gqXnMXVxDmkMk7G80Y8C0Ej1JTd9wLWbumiGuSDcQ9lvZRUIPsCMmJBQ3Wy5tmvTNP6DpIczPIvANNAXCZaMaAbyHwpQ5CRblM7WEbOJKov079GpuROsoeW5rkNVM/hW/YFIpectLCYhuwbYbdr2owOSRQMONLbZixGvVC4pmStVeJdIN0I7Ij0JdN/RXrplltNZCazTiAG7TZaMZgOmYeuUdKgSl1pYpIUnee1qqZhQhodl7bxyTwaJ9Nbb6jV0ztvenoXXLEXV5BeAJcq+6YrhmDySG5B8qB1JVq2CiXCRrVU+PhFxx36xGeCHztX6dZXyh6n5O1vXIJuVZlkkhM93tAus+krlRRYFy7xjRu4tzpMN7jNaWS2DbkO3q91H7po6+W5jkhN6yayYHCRpJDQlkpqUUmNA+3bjURz8CtyS6i7fUDmRw047vfaP4x2hcl7T6ZgJOvmSQS0x1y6IYvle6N4gi7Qeb+lXjtinWlwwiIf6+Zz4yalCBwSMVdYEGwANeqzPANI4ndgXJIvCiXdBVX6Dxp7taVBOqTr1nbscVr+ybW95r+CudM9a/GYCAtUggAe4DDJHhn6A71WoueqKzyNPLPJnfnygG3UtZTUuyzR0K1fYP1vX5Y9NKofciWmbxD9gC7WbpDVplMmQOVgiZZM+ozqUcsosKitX1D9b1m/G43yWbwteQusRiEgLPE3KTOZIBXmaQGM0iR6q5UR/V1FPfJCMtkWOAemnp2JmiSzaA3O2nVb8AkktmuytAdqIVBKt1RFlFzpfqMcplaXiYQmbdnYddPZ16aXw/uEou6U8B0dxi6A27uWMrsmJkjZnA3rhyC+rkMYaQc4NfSiPuhZTBqYt7Rz5VVDxzDJZGcugNWmaYy6Y5qqnWLTNir1iEV9tfCl6kUuLxaq0vTa7KWQTUMWXT3AIP3pKwKTvefMunuN6mp2rVJFe3U2484mgoDyt5rufdGRWapdVUbcP+eQrqHMvshgyY93rX7Zb6K7R3gsa2dkOK+Ge6plqb7UBrdHcDgXSHdpVaZrAZWFVcROcKUQCDTuSPUz4yaVJnR85KVRAaAHe8Zut8DTv91Zao7ypm/cvetFNAe11qOwZbq3Dc0nKlFd+S/616RoUS6+4DqbsU9Ygb0TlVHMt1rN7sHtav1I2mZO8U2yt5rZTN2g/c7tAKXNVwJsosg6REDp7slc3NHEy1DNdU6DTMvlb2MV3nfNvoRdSYTeA3qqr7MnsgVYFnVUUd3qT0EjWP3OoYBly45hhZiO9oP0J+iB+RgVDc1ajSXQWZPZAi4n+lU1b1xDlGb7hW02qsxlwMvD7gfXYLujbMq7iKLwfw81P1aMt1rq3vt2N2LuFvBuhNtH+wrGCAb/QzuRx/HpknwLl3dgeh+sureoL/brjuDYJufutv2fm/kCXgy+5T7VBF87/Pn2k6rfp0ZzTLgLrIYdN5dAN3VqHsu3Tf7vWHk+fNE3fl+4e16zyWawVUfo77T4p5sWbp33r3pj9bVu9t196lui7z71jDW3Ac3m3imHq/ue8MwjAzffcr2fv+q+uR3VGeuu9+l8+6Noa139+omM8i759F9k6/uNlX37NfYxnodET71wR7i+dXHK8T6d+6nH4HeybI/e2cAACAASURBVEzn3RuhiXevnbtvCoNII9e7b4h3z37JFr0X7NMvkSAi+0eMq37/+on78YfRJHfvvHtzSPXuZtOqah11/1pYZspXd6Lv3CsBy3rGAF0lbEd8v+J+/GE0UXdfZlX1TLy7tj0zQe2emVHRNtV8716APX5w2rn7/X7/Y4J+v19R3pvM3TmZnpmMukO2iGnbM+PX3syE6kZ5xI7VfbvfG3v0iPV+v1+jj24jYu/Xe/zxaF3Lff1mc51ie8T3HvfjD6LJFTmZnhkz3SJ2Mv3uTWqIzyiHOESdImyLzDv17hHXt3vE52gtirLGLXopRB/dksfkfYcrxspgukdu5q/Ly8u/C36ZDFBWVb8BeDiQdR5o1+/eHEHjHKJOEvlStHePqPue/sdg0kn8r83WiB+b9/7Q59T9+vvmzweEyz/cb8LBb9xFJG8njrJ+d0i6TyRv3hvW35pZ+9yOInnHFN5SAcfsJ0njHn+SVpXyvQyn7lf9qz+XmO0f/pTokET7b2vvZhxKnKvkncfmPal7VcPGWzNr7aHYoroRX1jF/N5TJhM7g9hvG4T8W/q53BCH8+49wvUPl39xv0YOGsXuyPe/chdZDGyFW7MtuDkzaBKBrPXPHPlv7r6VA9rX2WC+BS/v2LvHTMY+HbuZ/RqRn/EyuXQP+il5v6JsL2vd7Sbn7yHfL+t4JhRILzl6SKFcds7MyUwRa7I187n+iMhN/lillHtBur4m/45oHn2WxDJFXiaqqabo3o/FvRThX5pfD+4SC4LKsUonQ3c0V6XuRHN0lmi9Q/fs3BGRlN4pdY9Yv4myyEj1qXW388UduZlUmWn94UPC+KPr6kb9kF+R7+cu8QnQHXwkqqwuAq/J1Cyv9m7V737umUwZdSd0j9xM9D9rw9hTTS+w7rjOdHUVs73/9UMKR9IZu4l192QOIkB59DVHDilws3SHnO8u9czsKHgb1pUzlOvUHHhNDlLK0j3yN6MkmTHof/fkvzbvZbY2y/zgM2oR+3iFyN73v//Bws6IfLGreRk0OcvEQ2d/cJdYDCDPIlM6313qOZPPTUbefqs/nI7ambSfwbSmSXscSK4NXFKKS0yRuO/Xa/r/TIK/HQw+9xP4379vEMW/b7Ys4/9wtubpKuhfXV83vhpSjzMAYhw33/32VOiOeyLrHlbj1c9mvr+MOH3fEhXH0eNmbwzSWh5r+tbA2O+33pbtogyib/r5+hp3u+P2MCTvEb03f1KMZ4PJp4C+Qq4/NzisRl4/JD6sBshPcIfVQPaISU0icU9kXfPuN5H3TfZMVbRXgy5O93umbEqNukHbZfbRNibC+bS2o+8ZPF35Pdob9heWd4S/8zX+/Zp9R+CeZjkEMjtmcA75kyOHFHBHkUH2iEk/aLJ+XRVnM3VPEabn5OX1Rm5jr5KDLa2srvc224pA1gPZkzv+jigeG/a//vxh1q5/oo/7fdokj/aEuNzTLAWpNVW1B02e1DHCDU7fCmr3zbB8t/MIXx1kNTDKebf5c8nuQ/mLdTUf/vx9xXTaRP/PPc1yL30U2XMXWAzmSo8RxmVVoEPipZ5Ftmt0lugKnb1X9xzhmO9Czt8bMdq+J+B+YvICYDT+stdjK1P9j9zzLAF0xqy0Q+Iht+49IuvC0x2oziS1J3J33SRqxlnzoPoOaMr3Edl3zRVYKyKwyf5trO3E1hvcD2TB+vgPMeOjHX/c0yx7JaQdfwvZ7c5UmSjdURIJ1I7pymwS2/mRPtcO3/B5RHXd+/fvL2si8DkNYxVgj6gvwl2PxhaD+3kZpHz8vzHfQ+55HscY1ae5yysIHmD7L3PwXoruUK+26KmOZV3IJaLJU12+e43ce1JvGgxGtQm/HcWvGfw9twdcTBZ/M4zvkR75d+5pHsUT+vmy+sPwShXIPDtJ7E7pDplE4rWqtCvpNcpmVsMG2TtCEJM1b8fHcXjx18eRTBW6R7lMj9oaou41shm0L0zWTibYlWqyUzWm+xfAJBKvVaWZ90YbeKi817cz379/jWee7iuHNJ6dfK0XtzOsK9L96mMPdQpf1ld39JqT5mV8wDOzcTDzJUV3nEQuuIdKgdTd2bvlsEmP97cVyt5rnlJNsEkEfjAqv2oNPObrbOY3WO/XUc213BCcr7SH8t/ex9rqvpHa646tO5B3XjA5ZEx3yGhmItW8467Ium1itLRaO4zECOwBA3t73NZ4qa9Ih+17Y4+qrmvu5+Qhs7+VBO9//X15+Yd7skWwpeYy+FDVOUcMKWBOVU3ojgwO0C+wkFpo2oWN+mYiO1O/7z3BJkXfwcguXrlut/Yo9WA788PtLSF9qfccJ7UlJBrF9O1vUoi6/Jt7trnA/TLSaiPoKOkhkJdgg5mY7pA9kXhOpDRjuBs3k/fVqGk6gxEwHp78XY8829sGXkD+ejYievpxdsFP3paU96/slpCoqsrm8Zd/cc83B+gNUlqNCQ/dgLLubDAT0x00mkErFXlvlbhvfcXfxZLAX1+z8Z3FyzYt8SUwCornCxhGuSUr6plJ2B4ydP/z53eZF/xAar8MboeE0lamQSyh+wNk18xSrnlf4il23F0sDZzONEkjY2Q8+WGMtrn13M2W0r2ceR+TFvkr0hFJk/g/pZQ9FndpC1WcugMtFHEw85ChO+haFSfvsuai4ulKw1F9eY/sTJPemTRegu3oIMuxX/cKdX2PVX1jGKWyGW8w/Bx3AF+732K6X5ZR9rj2IO/dN4RM3R/ZlWpCdwdw/x5O3qUVMXZznJ1z97E0cE2xUfqextdN5NI5L0/6YtZecMg5Ebrvj/TMEOAT5T/3Im2/wol74mVK6TvuC5UWJeCGGSjjHDIdMwzdQdeqoVw3g2dFNpB3kjvXGvd+AJsArU7X9oj89aKEsvjxBGtE97VhlDnMAM0Ny9QdEvNeJovE723yxAjHkEADxNIr1YTud5B1VclRJJH3Bu6d2HeB+t4E272Bznkqxfacd7Z/EdHLRjOeZHEPIWNIvLfjjqM7rqsC7S/B0zekFVYj994seyf7VpvH74IQ7Xgq852Itmd83O+I5t9+E5E/Es7gzF2ec8deBkpYZ2xNlaU7WqtCvcX4kt0Mlvf62fu3b8+2TvpeEkTbiY1br+mTiVj+V7JmPehoZIs79jJAzb9vk9RKlaE75GglsmFV4jVFfZG154lFWNlS/LtMEG0fE7Ybe/pcoqIq4jglfLHAb3BNmbugwrCC3Kaase4M3W8hzTt2MxLfMXH23kTeMd/bpO9pbY+mecTP5a/LD5f0fw5b+JHkzB3XmGyOEJLADN1I0/0B1LzLzmbwPLFGq9VY3wXl77LhpbR9YzDqHvl3TPdv3/6+PJTRBDjX4S6nOAwhc5lZqsjE0h3WvONsRl6derfDpZ0mq9XYv9uHUnFN8GKntP2r8b5n1D0F6mhy+sWe0VX7zF1McXBAcxk3bd1ZuoMm77jpXdqOVZR3NZkXiUH1vVn7OwQ2dkrbV8b2W1rdU+AdzcfwY7jCjb9DaZPydrTVPeDoIAk+2w6ZpjuseZ/LXqzu7Mbhe+LfdQkki0B3DNqktLZff/tWqO6Jo/lwiR7/Ts6hv8buX+KSimzbg/IQOHW/zaU7rHl/G0uO3ndLtC9p2MjOxPm73nzHOVSSt6/36WQmB0xGc5VM2LseDKWuU2FDd9www1h3lu64bQYqEMXRu7wd2vGu1WZ2hnaQDAZ2breiDoj3CiZsj1zKIXWP3riowPfwvFU8Ye9aZuPvbrccQ/b+ctY9RfcpZM87id5lXtsd2ZfE3elq2OD+yNzpdbqwHf1+tMrwvt9+O6rujKNZx7tBIr7LtDKkgQDMQeDheAV0/wK5P5u0RcrMIkn43qjYFOEpHnKkYUJDE5mBHc/WWRt79Cc6HYR7MilQR9NjNra63GUUiDHoQnXBDiHg6A4bReLxSlIXq7snnKs0aI1EWFFvrJ/A45w8MjLJk1yTiZLGfn9E32PCX/Zivl9xV1EcQtCNHdkOgizdXdAoEsu7tAniCGRfEneXqyKelaSXg9+M6e6QHMd2xLsT0J2sl/Fuv5/cVRQGE3Shir2MW0j3O1g3g4ZFSjumCWFp4/4+7i5XxTPJtQfMtCPVePHo72TnxU9HvXuEsN+/Yh3NVb8/4a6iKKxgU0jsZe4K6X5hQraJkSxS6tJoNx8ViV9V0IRGm0wyGcKU/2oupe545lgyYu9jvy/rnGwi7mDtMtjLmBfFdAd2MxPJe1Z35HyDhr2RBM829cnZOTAqQPOYYb60l1X3j3hmQY/amat+/xN3DUXdiiHggUy0G9I9QPdbWDeD5V2ue99hVR7VngnMINHTseKuAmZKWeEb1/r4QhWbGaTql5fUzEhTdxNySzb1MrcH6I7dDJi5IqUmqe4dL1eHg1GBBlbCitZYh0oVfhO/zwy9pqlT+hT6KInkLqAg4P4BsDJmnpfJ0t0BrTSRvkiZjWJkucq0kzSDn8iqqlTSi+OYgd3co11xI/bQRfv2m7uOTWGjOgucdeBzGY7uwJUm3PYuN3unM4EF8T0ifOzh4VOalyD27INR3fOwWcxTE1T7/V7vzyXayi2a7/hWA439jVvdvxykO640wb3hYPcutbS62+1esecuWtNVBXNgwcADlfgNMzt1VGjaq4E5p+8qNU/yX+46NgIWd44A0uDmeBmO7i6smyHZu9TOGRT4itT3b98Ce5Bo7MGRSALxEiTvK8ORgKgJYTW6ZjrE+mt2fCp3HRvAh83cSe/v5Ajd0YHCYMPzYnmX2Ri5i5sjxfE9pfAgy9bAY37gWJCy4xlKzBnbV/HJTpeXl3/+4q5jbeAZ13CZOx7rTg4PLqY7XqzCRe8ke5d2dE3Md8H6jhQ+1trojA6OoGK5nvysgS3Cs2Pgs0roSEl/svubjpPkrmAzBNDi7nP9Mrl0nwAvVt88iNVqou+C/HuE9IkFI1uOq3lJ6fpg4ImyMcko+4G/m3x0Ud7+m6r739wVbAIcQl5zN18acOj+eJTuDybgOR4RdhBhpBR9//ZtxeQkSHcDwR1k0bxs5vsP7MY5e+rXx/u92A0dP+WsVD3IkyXf4oXqw1G648UqXNcaaYyUd2p8DKLvYxH11QR+WnsHA88T5GuCIHOGzcAT52K+MX38qaAAMR0bGnELKhxCgvW504Wqw5Gbp/sX6MUqrjVJX63G+i6kn4DBKvAGKY0fDO3DE6yP4mXjeaPs9/RFCvu3b99eGG3frPf79Xt0kaIk8jceJyksmcFb9gArTGSh+oUjN093HL0DLlZJrUnmLm0C/1ibSV2sgqzGo8PHyo0xTcMPvKyoo+xH4JoDg3Z4RmzfG+tedPQTofvfOzI+VdCC1QKuMJGKKrdQzaX7LexEgtjOSF+t7nb+aCCo/53DKuD0GOXjthcE5Yb3bqLZ79mvR6fNi9b1b4jtJL/f7XZbRPR3w3jf7aIZNL93u+W/4viO5czjbrs8zPIXqrl0x31ikPI+hbIzuzAzj0Us/IDXZSrPtud53iYIgk1sc16C6N8ePX8vF7Zgv06wou9GSGP2hhH9B8v77hKvUv+IWrBiKwO5TsVHdtxwxM6nO24lA3RapDMSwM7s4rlgws0B4VE2S2nwdxTIkPVvzN6sEX5HXWOeE7rv/v2D/vNb0II1AG5zf3tb5FZUi+iOK6uAjTNvizFEJzDCkoblAuPrLHwuU6mKkScyXud+P/LbfU5R+d0wyE4bYmFIKNnQ0GArM4Y0x05uRbWI7vBZJKCd2S0DOheMo4FIrKI1Zx1ttz1vI0nUCeKZY+nLvTe22WuFF6x/uGtY5XKPYacPvBW0/h6gO5Z3yLcfPCFV4mF8LOK5YHJZFeHZDwK73MmqI9v0Al+Sx2KQjL1kr8l2H2s7i8vGvWIeeOSO93Xkins+3aF3eSC7BWZn8IIV6ZtEw5DGs/8Vn7pnm3ai5Wb0by/wAv+rfJoTfKXPPV3Xe1+v98b6G3ep/jRM4J+G0JF7YYmpmO54AgdgQ8/b248hVBoZtXDQDUESEkm9QRMZO2dH6t7Ycx+jCXzN4TNz+FQG9w+k520coTuWd1D3TtIZyfu0KZbUv45fzonsTzTXD/JWST3D4OV9t2xScTKB96e+UeeeL+5FdMfyDtgoRo9rArLvpOI0kL5i1QpU2kf522neDaPHfTAKJmsvWHEGCeqKiXPPF/ciuuNOAtjfE48EBrLvu92c5ia24B4aXUF7ZAZ2xjF+22NVf89drTZI4BVkkAedezHdH+HdO9npMZY6gpbBMp4LdhYOPnm2Wdq+GziCXOPiag5+19rzMSfLMe5Gy4Rb1D9wkO5Y3mHdO7HvJkT6jhBSvRt95dhxYvDHdN9VzrunYUSXfIV6ZgqAA5pKhmZpD0HPLsCw8nZkH6f7HXz2Dm3fmRLrwJSfwStEnLXz0h7hfW+s1+tDbKcJfJUWGg96e+rbMed+gO4kewcNTN9mY5DBBAye4nrmCS9Z4z1XedKOsFqv1/m+PcafihUnvEyFO6kDYXHQuR+iuxJ5x/Ydku/LuFNdyJQiDRFnUIPrPGkvjd+VWmhwmRy2eYB2yxSK+wG6k+wdVt7JVg+gahPGKpmle4IZzUsyt7hI2suiSgJP6kvAanlM3A/RHcs78ErjjbwHgi1Xd0wTjdhBBTrgOdllJaK9unQCTxrDoMkTHhH3Q3THjZGg25qS5hkblO/zZI5LcEJr1lWQ9FmKuZ7Y0Px77Jst8U30gK0BnraR2wpZgu64MRK21kR3akN1E1CEn2MLL3S2hUKskjet0TP3hGuiXAJvqqgvkRAyvxWyBN3JiKUd923l4hG/Eco9xIZHMIr3l56Cwq+85PmIXPljA3+Y79iQjqGJg8cP5G1iKkd3PGIJuNYUxzPQfF8yath2hWeUPb8drAGOJvCWimoqbR/gRymVpjs+Rxt0Gx8C4bv0wZFZxH2SUT2mxYvWZ+Z5BOJDrn8P8x3XxqFDGdo+cMuRuDzdSSsBtAcje5vg+R6tWZOUpqU5PDvXzJOS6JK5qfnvGoTtoPuX3uJhGzmzZSrQXU0YGfMdsNxEsfSSfaN2Cyut/ij5/S1Z5YsDU5dUsZ2sUw+EkCXoTsJI4NpYEr8r4HtK4Qfip3fJxLM3kq3sGIV8J2yHjvNos8yBClMpupPVKnCAGs/BVsL3lIcftsfT+OysSgmePQV26tLko//uo92ApCgOnm68Lazj69QSdL/4pMjOzExV/j3CnEk2BqM2LFtTwj6STXY2gX+np398/Em0/RpeHXE9dcrRtyrd8WpVgZ1ZqNT3KJZk5zXaekfxq82YnUvp5y8iBeMbTuB7zFE3n7G2w7N9V8rKlKH7FyW11aSdQBXfd7un1IAYT9t1q5/6PU2w60VaxsjBrOgks4jvNjzbceSeM+C6Ot1JbRU8RlWu78jTMDlHNIqXo5pybLwRe0aU6JrSYZBzD5ITt6+VaDuxMgfrqaXp/oDtDOisEAy1/h0jLZ0DTydX8xx4qVmUNrgw9HACz/BdgW8nVuboOrUc3Un4rsDOxPoOt50vB8sgPevR0yOcfA6YeWRDmOUpB5+cO5zwXQHby0XupelOwnf4dCbx79D9MxmEaRUd2Kptje9lj22Sf7ZVHvr0nO0ePXb7E3cPpcMva2VK0v1BVTrD6DuoJ+WxfPJSWjoY2oGivU8vgZmZG2zBRDE8JtEylbTQELrDkwQ3dlklrExJuhM7o2IR8jYjfAcZhn0QS/96kIEdANegcibHe6q4vtvtXJTK9JL1ar//kbuDkoE37JWyMmXpTtIZFXYm7p/Jzr5SgaWfykHoWaogq9e8c0FGgUKu73a7VxJCUjejwsw4R7cwVac7sTOwU8UoSP8M6H7tYoTeOCvyg7Fczq82nskfB/I5ELZHqTaiWmqEnjLvPq9gZUrTnRSb4FuBEdwh7PjIY4hEnj93w7YDX/g84adNYOf8rJHna/HivyJ0T5IZ7ubJxaxsgaka3Ymdge/9QaB8hzisrCReg/xDOaKTJUWcwYHO/eAlHdVNA11e97t34mY+UnGHtu5W+VSmEt3x2BlF9v1tOtYjoEkj5M/Lpn/H6DzVGtnNi190virJg7SheoRln5X3fr//P+7WSYVbslemOt0fbhTad7pfW4sFawqhn+PlU2pve14Q+P6myN2vos8FXhAdZcN9OevVPRWFpIOYjz/3Y32PMvjLf//9/fuf3/9w908OpqXLqZXpTk7TVmTf44LTWCt1w1iGjleox/xfG53JZHM5S/HfkRfMtXpfw4gmWuPu36urqEPs+gPFv9z9k8OJEttT69KdFFdVNBOg52bpZ+BTePavC8x2A4zsQC/7wgAnxJ+TBuBeTPcPv7n7JwFmlQyyMt2JffehiU5BtspkzwPVCsvQ922vWKernK8a+KHOz5TKj/vpHQn8p8U/H0D57pTZjN2A7vemSvv+9kbmbA7HunlYDsvwKTpQtYbYj8xIz5815jnGityLIcMGMiP4ErXRSPfv2GyYh4aGcahEd2LfocdGJngkBl5hC3xVzMNVEGWKXrHmR5/z/MAJ57raFh6k1D002b5wQvcPv97e3n794u6eYC7cVDXulelO0vcbFc0zCLGB19nQHMEyfA2fo7+HH6YzllR20sdd/PuB4btsJpjVEvdadCf2XdVylak46ZjQnAtiI5OZLxepO97Ayt024bAqJu716I4HcaiqNkV4HGqe0Jw8aA/TONvsG6n7v8jRSA8inUqtMrXpfvHFVLV1lYK2wAPPgO+AMc83Mm9E3f/BlkYy38kytWyrTH26kxNXgU/UTsOnAt+eFevJIMgzMq/r/X79iuT99xsA38kSsqJxr0V32iymYKt2jB1VGJmD4TrwiNeoY/b2r43ea3RY5duvyyh//CU7iHysy/Y6dCfVVSV7mygWsX9UOaXg7BC/raaMTM/oRcmjYfTefl1GLJfMd3wmTeVlal26P5B1gkq+v03omyrcIdvnjjldNI3Thca9sY//gzn+T8T3S0lxJB4HWX2ZWpfuNJ5RGEcyQ2g6Bw+E2LVbP9J3omes3xLWI6B4RhLfMdtv6rC9Ht1pPKMwjowQ0uvfOXj5iF174bEce2Ob/ENeHOnUDGUa0J12Eyjm+4zWWLsMXjboWmlY2AC+Nvbsp2TFM06d3oGmdKdxpLJuMQJaY21B11ibEb+PjovO6fqxRutVBv9K6YucN2J7bbqTeEY532fXw87RSEZcWBp6BdL+a71fc0b9UgLfCeuqtLiLobsufE8c/LhzNFLgxO+gRff6nXXtCcTHkfOGbG9Ad9ItpmIycBpxl+RwvDpBtilGrCY5TQMEa2O97qE/6YOD/xHdHflYP3BvTncyaklpeRVjOu4cjRwkeYxZPPzRiPGa/sQvsd2RjyQPrBVBNqY75bt6fX97cxIN6qpOwrCM85jhoR2bvXWk7tHfTCBP4nfuC+phZ9YvL4mgu0b6/jZL7kzXViAIQaIhBUvUoxAYvwvQ9oZ0v7i/0UbfGUcz7sqsAuAn13PJXezSEBa/E7ZX25rKoRndaXlVB31/e2NuULdmbYhkhVoYtZeDoPidsr1eMTVGQ7qTUal66DvraMxub18DhHZ8IQvzmJIQ0x25E8P2xnTXS9/jww8iwnchTU2EyUWsbdoTIL5fNuO7IG0XQHe99P3tLYnOOoWvhVWi7HZx+FgB/zTujhTGdgF010zfmU744dDsPHxFMMpuNjPtCZp2A4tjuwi666bvb2+v407ha4Hx7OOiPt8aaBbPiPLtouge67uKo/lysXA7wlfHU6LsY1/oVrUm8cxUINvF0D3W96IeIngskjprl8OXQhLjDoeh6H2Z9flOusLEsF0Q3am+a8T3t0WY3L6x07UWHMQyGMtSdoy6caRYtouie6zvCsctcUgpfNc8Voy5lSjDUAbZSbdYdb67YtkujO4X95YW+/nSWDApzdDqTHwu5ollH5qurPESuBuY+/BhkB7zZn0yLITRPe6P1Irvb29M2DAcqz10V0csWcteuH9DBGrE74TtjXog0xBH95jvaufP8Jgw6jW2Ok/DYJ50XQyHluSFV9Vu4IUlWtuF0p2OW9KO728/2btqdxKPkRL2oSU/Ra7GdzLAXSjbhdI93r+qTYE1xix1a4Ou2Lqbe+wVcUEOZKlSbnqk2QdHsiYQS/eY7xoFkgSpVeu5u/hlYLMXQ3jMXoTL0nxfSmG7aLrT+TNK52EXYeow93jonW1Q88TmjsMAsBZeujuSTLSuPU+mCKLpfnFLLRf3FDTAzElJ/Dlm8WGK62M/u71ULv4pF787ktgunu5xgVW7BSvGJH27tTt3XSpWAftyHwbw78FlZgMvHMHFpQTi6R4XnPRbsGI8huOzZPw8tVwf2r6S80KPxzM7Gmg325aaCwl0v3ggC9YbbToks0i7+KF5+oyf++zidDiGdOxpHBtOMJERQFLIoHsc0NyI2iAgHovXlKk57fpTmPYwQ0taq0AZHO6OpLa9waiwA5BDdzoQ+8bR08AjzEIzw/iTzGqeMlw3YTL2AzjAd2rbbx45SgmBJLpf3NEFq6YGHiOd1ERc8E9K5Oe+l35+YzWGPY3i4QRkm96NeccRSm+6JwtW/SpOKTw6aY0f2sGJiPwqMLNc10V7CuLIKbXtEhapGNLoHnfQ6JnAs3jM8mLotX3t6vrZp3TtaxQc5M8GDmklVcYiFUMe3S8upuTXFzCsRDayPj6qQYUtpfzSDzLPRQO/nkFON/DMkmvbEWTSPTbw+iaSDBbpeiNevLbNyoe+N+aehNIcpgA4jrz9eNXv968+3iZGRpptR5BK94t7amh02tN3ABM/S5bh2PNb4uVXDvd6HZqOrksnxPd1n+CVGhkZtSUGcumeJPAtMDQYP3iRHw5NS29nM39KtTgSWDrEMIVAcWSk7Zjw1xLT9gSy6R63jOme0LCYZvM7BCtYadg1vAwdzr+g6ToNplSDSNRCwgAAC/BJREFUAMeRvY8RCN1NmbYdQTrdL77QJYjOJScOi0zVPaZRoI/Oz58cO4fpkYNpwaV+7Wf5bolvCctCPt2TFprWGBqCxSSbyRN4lh8qFfrlyuHCU0J1vw1Uf3t7u+v3Ed8vP1K+S8wfYwDQ/eKCdv20ZcXK4qdv5inocDi2rSB0OSZKxjwMvHyiD4eW04YEjOBjv3/V+xDz/arf/8TxRjxA6J4kNE7LBB7jMSyQ+Yj1ZhCsALKbVegEeSadvN8Ekx33a2uN937/40eG76dE9yShMXXc1VcKUz8v/UhchBcETyvhvn4ehkFg5q2c4x9sab8qzcNVRHfMd0L3K4404gFF96Tk1KoVaxbT0DlEPcx7yw/CcN6A+ctw5ftHWD5EWZHbIvuSxlX/KmL5JV2tXvU/cpxpMd2THppW1FgP4fGnbxWamzT1bdMOAicIQoycl8CcfCqIHhc9nvsmebAtZ9Luy4jV/ePHy8vLU1T3aErBSQg8xW7i5ybecjG2AneidVN1SXwkdKc4Le+OEK9Y21RzOozFchKJMgTPLX8+PQWdQFhe97GbIbjq9/+P44t4gNI9GUNzGgLPYDZ1Q8c6ZrZrwPJ8fzJtZaBVjJlzc9NPyTuMl4Gme1JjbWMGXwY/ppPQ8U3ruonP8UzL8f3pzxNjOUUYudprlu99GHEHp/vFxSQW+BO9mQkWy+l04rt+pPvoD8frqOMSfy6MHjefTk/HrxSBNrZf96mfibrEIJy7CrozAq/9PqcO4kErMDfWR9QKiTsiYdiugu5Jl6RGZ/V1gMHUim/9xcX/kebf/hWIk1FF93jf9uktWTscQjxXg+6+/r9PH68+foIiuyq6Mxm8eaJL1g48Yh9jCp91WhKq6M4UWTtHcx6YJm/pAK2++VBGd6aL5kbO2YYddELiY6z05uvXHscMeVBI94uHOJM0u4zmxBH7mJsJI+3//XjdG3uOGGXx48ePil+hku5MV0HnaE4ajI9JTRr433q/N9YcLcrgv9e1EWFdifFq6c5kkq3b2tehLOKBSTcWv0Tt1VP3V8T1PSI898liqKY742i6UPIkkZj2lI+h6NVR9//WVNd71fiunO6so7kBOwKuAxAWTvL2nTsxqY66/y8S9Vf8/5HMl1/sakB3NqORfXJzB1hMkjtbMAyvhrr/jDzM/+i/Uv84Bi3ozswq6NasJ4RpfFdvptwtJ6iu7j+MFMFfq9gZTeiesvAd4U8CSRxzaIR1ZXX/3z5xMhG+ROz/j3tYPnShe8rCdyFN+5HEMYdPJ6iq7v9FbE99iVHBvetD94uLu4TwWg/z7HAUs+RWOgWmnaCquq+NrFdfV3AzOtGdaRw7h80fpwuG7NbBKaf//YiCxN6nsl6E5O1pLUfhO/fAfOhF947wJwCG7Mc6H7/0MErPQv3P4J36mv9QIXSj+8XDpCN8m8GSPa+s1AxrXtyxupdsJdCO7mnCdx6+XZBLdlxfyqbs63bTPZVKtlfhX9e9/b7HfuTX+jXzmF30mDX3pW0FQ/ZD2WN9IGZnkxyj7XS/uLh3GcK3M4f/8b43jBTd94aRGdP763VvGHvuS9nvsuFeI7piypL9UPZYG//lMTv3g0XQlO5pwlvtJPzOMFLKbRjGT+5BB+n+C/nSd+7jGoIpKh0O2hsAt/xmvv7HSdA9Q/hW9tJk1L2Xx+x93gcJ3o1t1NVd/ABtMLFkKzutMHERey/PzxdBY7pfXNwzHt7U8XTQI8ioey4O0P0dvVoOPEATLFyTuU3yTop8zdXxXMkvgtZ0T6c0ZuvagzPqnotiNv/Cn9nv9TYzM+ZNWEYak2CdE7qTHLJsI4LmdE8T/sZp14kszdR9j18s71pHU48OFNnJmtT4kQbZ1sQ9Oh/a0z2qtDLGsFUmPlL31z2R515vvcfM/rXd79fvr6vXV0L33Xq/32ZZ/cMwuG+oG+Ys2Y9UUBsDr0nz0cYWsWKwhDfbs8fPMNZrtJ8ykumf0Xtx9NGfe6P3it6CDUz3d/SYrMivS7w1KAVr2W+cg70xQrDOJbqRa+iL0A66X1zcMQ7xxm/J8RWGEdWQojQR/XON/7tHPN4b2zf8r/3+F/rcz+wXax24syWlY12PYkAmD+zTf4wqK9XW0D3qh2fExGzFYblEst8Jc3uI7j/xvwj338hrIZvRRw/jvp82WLjs263EMIZFrknHhr51u5nKILVqNVvQXUCSmVfDWL3FDH/FdO/FdN+zj42hsZdJCbsld32aAPfLZE06Tt1fuUcXoE10j0x86krPNZd4otivKXX/ZZA4PUv3dd7XaocFW1E60s4uFHil+jPzLSul7q2j+8XFF9bEm47WLj5Rd0R34l/ejf0vamkK1X3G9ddogUfWUd64pfvUBeA1r3r6X67kF6N1dL+4eJiyl1znamuuuqN/7+OFaIG6v3NJjXoswpSwQ7kYgl6ejue+Bg6ghXSPRu2xEq9vy2Sud397W69fV3EMU6Du+xIFWVhMnNQ1B5/Q3stbk67zPngA7aR71E7DKo2mpiZf3XupFsd8dWdrTLOeehe/81mumxOYLCaFXk4wU6n596LFdI8kPiU3lqtfUpPv3bfGft1bb98Pefe1sV733n+RRyh28bM5Ky03rpqjN3o5Lj1X8Q+hxXS/uLhPufgbSzMbv0tyd0RlUkqakdKIsf+Jm+B/MY/F+EkfsX5/3artd08nMTfWJwXCjvCDp/t/FZ17y+mOqq0pxjsalZ9wRvb+E7fsveH/RlXW3j4C+uA7rpK84VcANfQ7Y/22eydF8/2c+85gWKQNu+lCVE8LkEP3PME/jLbT/eLiIZXF39w4mp9t9tPY/yD/k7O5CeGVpDY/3te9godAIK3rN9ajsiOVEDhu/5fj5o+g/XRHpiZ9Y3TSeA6G8YN8aK9xV8xikkrYb8xbVSYmxj7r06sMmCE4Cbqj6pOZZry2cXySueyNX9xntUDGrwPXk4rQy2j574qpzMUJ0R2F8RnGT7TsqokT9Xc92wRmab9+YypKYjj8lxbz7CTgUjghunPZZJTV6JfH/zSM/fvra0/LHrCdm9V1Tbh+QcxLzO//OG9TBqdF94uLB47x2tVcf0XbmvbrnnY9MZPM2+ONq3hxmkGk59TNILZXP5H11OiOo5r0bTMtPW2NTpi5GZm4cfTi+kXqoI7fldp+E5wg3S9yfPyN5XZnghRi6mQsjDZ+PYOI7/vXH2jr475aJoNxonRHjM/cwxtHwz4D5Zhxl0lXrkf4EVeka0j7SdM9Sicn2XdoS9O4Rg2y4Tpq69UhczyAH731ft2r0jjA4qTpjjons4y/MUPdd0FBYDHJOpjIriuvJUnGqdM9wu0kq2FRXnPGlF9MOAdzY2lsYcThHOiO2gw4kb8x/fkZGptZjqrfOCoa2FXgTOh+gZonecpbzrwlM2tEYOZyXj1y6wrbHKFxRnRHkXw2oMScb8XUmkZYzFMbTeNnfqtduC4V50X3CPd8Qom2/4XtGrdaAVPXynvG7skvTHmcH90j3Oeq/I3lnFoxaurmOPXoiZ4h1S/Olu4XWOV5L491/vEEvM2igOlR29x5Uv3irOl+gRrKsp3dif5NWiv0Uze08t68ogTmzLx6FudNd4Qv01xnE3HecttF+ukkzMleqKhrXi+FQEd3hIc7vp4eu5vI0mue0M+KFR29U92dt6jH6Oie4P6W6/hOkSac68f62aObm7skTFe/yVQjdHTPINL5YvpgrdeA9rPpxCnW807TC9DRPRd3j8XmhrDJckJ3ugPm/Ww6DV3nyK9mOpPbM6qUVkFH92I83E3zqu488S3XmUzlKf5sOnVdxzngWWKiW24n6YfQ0f0oIqXPTeh5mE6k+aE7nU6bZPeL6Ouj72M5B/0KA8ftFL0EOrqXxP3dpDTrWblFCB3XIX+nKbjxxyP1tqyy7E4QFQjuutVoSXR0r4i72zq0Fw/HmXR6Xhkd3Wvi7m4yKSjTy2b59K6jeU10dG+K+zss+BIV3yEk7zxLU3R0F4kviPqR6jfT/ejr3UnkVjohF4qO7lLx5S5C9BIgcJ0U3OQzt+ihHbuloqN7hzNCR/cO54KLi4v/B1hwWxSXlnQ3AAAAAElFTkSuQmCC\"\n",
        "height=\"250\"></center>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YoO4S5SD6lUI",
        "colab_type": "text"
      },
      "source": [
        "This is often referred to as a \"loss landscape\". This term makes an analogy with a physical landscape, where \"hills\" represent high loss and \"valleys\" represent low loss. Gradient descent is like a traveller always walking downhill, until they are at the lowest point in their local area of the landscape."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HL6hccy30NWz",
        "colab_type": "text"
      },
      "source": [
        "**Optional section**: The loss landscape for real neural networks on real tasks is usually very high dimensional, as the the number of parameters is large (typically in the millions). However, it is actually possible to make simplified plots of this loss landscape, by considering a *subspace* of the full space of parameters. Here is a plot of the loss landscape for ResNet-56 on the CIFAR-10 task in the vicinity of a trained network, where the $x$ and $y$ axes are (random) directions in $\\theta$ space and $z$ is the loss on the task. The left image is for a resnet without skip connections, the right one is the normal resnet (taken from [Li et al](https://arxiv.org/pdf/1712.09913.pdf)):"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ypswcHx3Ckv7",
        "colab_type": "text"
      },
      "source": [
        "<center>\n",
        "<image src=\"https://i.imgur.com/8HrLf4j.png\" alt=\"image of loss landscapes of ResNet-56\"/>\n",
        "</center>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yqs-aF3xqd9l",
        "colab_type": "text"
      },
      "source": [
        "### Vector-Jacobian Products and the Chain Rule"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S2qJc9Ej_ZNX",
        "colab_type": "text"
      },
      "source": [
        "We've seen how gradients are useful. This next section will describe how to calculate them in practice. \n",
        "\n",
        "We start with the fact that the neural networks are composed of smaller components, often called *layers*, which can be thought of as functions. Now let's imagine that the whole neural network is made of the composition of the functions, e.g. $f = f_1 \\circ f_2 \\circ f_3$ that are applied in sequence to an input $x$, e.g. $y = f_3(f_2(f_1(x)))$. \n",
        "\n",
        "> *Note*: RAD works with more complicated networks than simple chains of functions, but we'll stick with the simple case for explaining the ideas.\n",
        "\n",
        "Let's visualize this situation in a diagram:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RtaJZQhO4I7w",
        "colab_type": "text"
      },
      "source": [
        "<center><img src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAArwAAADICAMAAAA9SMKPAAAApVBMVEUAAAD///8aGhuOj5KfoKOSk5awsbObnJ6nqayWmJpOT1DExsjAwsSrra+nqatbXF2dnp+QkZKDhIWqq6y8vr+4uru3ubrR09TNz9DJy8zExscnKCg0NTVBQkJ2d3dpamr7+/v4+Pj19fXv7+/r6+vf39/V1dXS0tLPz8/MzMy/v7+vr6+fn5+Pj49/f39wcHBgYGBQUFBAQEAwMDAgICAQEBANDQ1fm0Q9AAAACXBIWXMAAAsSAAALEgHS3X78AAAM2UlEQVR4nO2dbXvaNhuGBV67ZywbBlZoMYStbfaQjJGUBP//n7ZDsrEt6wXJkprcR6/zU7FJr0icCL0SNgKAKJAXkAXyArJAXkAWyAvIAnkBWSAvIAvkBWSBvIAskBeQBfICskBeQBbIC8gCeQFZIC8gC+QFZIG8gCyQF5AF8gKyQF5AFsgLyAJ5AVkgLyAL5AVkgbyALJAXkAXyArJAXkAWyAvIAnkBWSAvIAvkBWSBvIAskBeQBfICskBeQBbIC8gCeQFZIC8gC+QFZIG8gCyQF5AF8gKyQF5AFsgLyAJ5AVkgLyAL5AVkgbyALJAXkAXyArJAXkAWyAvIAnkBWSAvIAvkBWSBvIAskBeQBfICskBeQBbIC8gCeQFZIC8gyxuT93+xUP5nI68Q+cNkJgbyKj+aPhLyxgHyKj+aPhLyxgHyKj+aPhLyxuENyrsKxa+GXyHyh8lMDOSFvOkyEwN5IW+6zMRAXsibLjMxkBfypstMDOSFvOkyEwN5IW+6zMRAXsibLjMxVOVd5Gvl2rAadozMc+XS8BfVMdNWSMhLU97ltGCMsfHO8NJGlzefjHlgMVfuDHxRHTKvFRLyUpR3PeGv6WS3YaxQbg6o4auRiw0Xd7obMzZVbg57Ua9mXi8k5CUob14yVt6uVouSMXar3Pav4WuRt1yjfLWa84Zwodwe9KJey3QoJOSlJy9XqOQKbblLG+W+fw1fieRNYME/urlHbKfcH/SiXsl0KSTkJSdv3jR//B9sqzzBv4btkTve7q4vyd+nFXQqJOSlJu+6bFo/MYTSj6Aiyis8EvMMSyHSUnnGoBfVmulWSMhLTd7ppR2sPlonyhMG1LA1smhbvonFo6giuRUS8hKTt9MmrVbLPM7oyRbZNrxi1tXU7kYVybGQkJeYvHPbJ/fQGrZFigkr5aqGiCI5FhLyEpOXu1QqVxXiyTu2jJdCXlRbpmMhIS8xeTeWiaPBNWyLtM2OhbyotkzHQkJeYvI6uhRNXqnLayWiSK/xhnEG8lqw1vDSNnE0+FW1RDp3PyOK5FpIyEtLXteGMJq8fIWCKVd1xBPpVVp7ZyCvBWsNu7oUTV7n7mdEkV7jDeMO5LVgreGp48RVNHkL2xpBQKQt07WQkJeWvK4NYTR5ncdOEUV6jdbeHchrwVDDyx2HLz0VuwrzwlMUeXMeMhXbYupA457waCL5FRLyUpF3xvrYhjQR5J0qgcpTQl5UbaZfISHvG5D3L+mRoYbnm81mUx2L2VTYGsLrNdwN1UZOeUh19qfKu9L19YvUZ/oVMk6mL56ZiRkk7+H0eKx5Op0Ol8un5vK39uIVPu/v/nSs4VvXSaSrNSyFmiO3zgN/z0hLpnMhI2Z64JlZ89j68ng6PWpuPDoL02WQvEfpw+14uay9eIX/39//7VjD02g7VqRQc2ThPPD3jLRkOhcyYqYHnpk1J8mXk/aGqzBdXlver/fONbxxbQiv1rAUao60ncEJirRkOhcyYqYHnpk1b0pezpfjuUrdt5eqK98Oe+XZZj7vXWtYjMOVqxqud8y6ocbIBS+M8bywp0hyOY2Z7oWMmOmBZ2aHQy3HF+nqwwu/+HK4U57vQsCAbV/9Ov+0V/hv8uxj7qgu7f5z/cBSw2sWcX9iJ9QYKUb/pkNrQZHmTPdCxsv0wTOzy0HY8qi5+qQ815GQ2YbK3nNj60HzyzngWMNi0T/adqs21Bi5cx87eUaaM90LGS/TB89MCdHInuWWlztzUp7pStBUmWwvfzRkzDj662+XGhZbvGbKZQ1Ok5FNqDGS7wpn9tkqL5E65TRmuhcySuanG+nhx5uP0uObT4GZEg+6vu2JsWFdhlHwPG/X3oeh7o5Gf/LSirekroZrRENoX3PyqeEm1Bi5cT3S4B1pznQvZIzM37Ps3a/tww9Zln1oH/76Lst+D8uUOalN70NApyF4kaK199DtQPS4d+KrvoYvuI/Dqxp2DTVGekw2eEaaMz0mG8Izb7Isy35qHn58l2XZu7bt/Ynf/iMoU0bT9AY1vMErbBd7be46FlZfww1j53G4Tw3ffzVFLj0mGzwjzcV0L2SEzPfczqx5KFzO2o6EePg+KLOH0vQ+DBokNQQvD9f2SnNmfZRi6bHLaxmHL+SeaRx5c8NkwzrP1W8ejSWvqZCLXD0DH5zJuwnZz83DT8LWtpv7SyZ3IwZk9njoTzi8KAM4L8L3NtQzZrb+rlIsLfZug3kcvtz2HIvTbdBONiynYsMDG0/lVbBI3QZtIde3G77DgrGtPJILz/xZ7hf8lmXZb+3DPyS1B2X2qRYlmo7CQRm/fW95lRkzf0QHX5RVU8M1YhyunO1a57eF4pjHqEL/ogommoXaHZvMxOZFxkrpd/GLNGbqCjkrRab41lNpq06EzA/vpfmEm/fS7MOn9x/6P+CZ2edOyNsM0QIb3ii7yqrV4uH2Ok2V6RpC3gRup8oQJ85UmTp2WhfF5bN73pMs0lSZppDzNmcsn+uINT3nQ8hUGeep2/SGNrwx5D3Uq9ND7XVbpNBt8RJftb9ThjhxFilKZbJhU7QNX8FY2WkGIy0YqIVcdjreG/kmtUWKUb/pDW14I8jL5xkOAT0Hx+Vh4xaviXLOLMrysFiolf7jORu3feud3PRGWqpVC7nrjOAKedGE2PJwRafpPVrHSS4Ey1vNkQ3v97puzDHOuhbKnECUjTnq2Knozj7s5LuRNsmohczrr5muG+HuZwytjTk1d83Gsi9n9qLc9iNU3sv87lB7XbdELhSXOi94ryscZUukuitcWrmdera8TtsTdYVcNvNyu94vRGdLZJdqgPQQo+ENlbddm6jt9V0vcd2MPtOMwwW52hW+WsMum9HVXeHz8bgVayzf9Ys0ZRoLuarNltZM4mT64Zmpodo3e4rR8AbK+9RpbCt7nz274K7HgDTj8IpbtSt8tYZdjgGpkw1dcuY58nc6kmMspJjq6K/3xcn0wzNTR930Rmh4w+R9ktbVhtnrdACznnVVrlY3+ktS1ztmVw9gVk2rto8tTBrLkw2RDkMaC7lazUu28e0dvc0DmFXT+3wO2Ap5IWQz+nNvI/FevKe8d6N3MdawOg5vb/S7wm4z6Rf0kepkQ5dJO4waEmnINBQyF2eKC8PycHjV+uCZqeVyiOxBd9OLofJ++eebujP+uZrvDfitTDUsdslo9ydqPmhjyDs37TJYVe6OZZeiiGQoJN9MMZuyctLbUUFV3vq0WHjDO0ze0+mFXWg7Lvvny7XzSTrg7IGuhvM8n1c7CrbzPO/9YYhcs2E8UF4ui/jiGsamM90unP46bQyRrIVcVeO1MsEbxpMY8tarWuEN7zB5WYd2ge+he3ngG0tTwwsmI3cSNOO1UHlve4G9hn09EX+cMq689kIKpoyVktNk5RUHgiI0vEPlfT5VnGV5z6cL8eSdMbZpGPenkjTjtVB5p6xsA8v+5pxlwbZquxgqkr2Qgv4EB2F5D9YNtO5Q/KvvXTTjtXjfEqlhVpa6I2bfQaS83yGmK+9d0OGfFuryqh/rSeWdsm3d25W7wmlEWmw37Ttl1p9HoyvvU9Dhnxbi8urGa+nkXRZNs7tg6eVdSot8u/7EM1l5YzW81OXVjdeSyTsvt42xE7mnnUQkseDWJI77S8dk5Y3V8FKXVzdeSyTvetLp7S6Y7khO5Mw5Y+MmZqas+BGS99D5LtHQU5cdiMs71ozX0si7KC5f1SvmA3rtfRKR1uW082/GCt0502RVq2WYvP+y7pzqKXQPegNZeZd5nud8M0DBFxIC1modW8HSOhGbRqQ5u6yqLfi2HLIrbNWR9/rBkbF/lWcMg6y8W7NLCeTtLyL0T2YmEmmxKSe3eT7jvaPASZXXlLdaeq3+vWfsm/KEgVCfKtORcp7XQDKRlrf8LwxM5+q6CCF5xdmfStn92XvfoRnI+7bltUBHXr7d8CSUPfrvmbUAeSFvuswL/MDN0/H4dFa+XToIyAt502U27Osv8D9F2EvWAnkhb7rMDneH43Hgt/cbgbyQN11mYiAv5E2XmRjIC3nTZSbmDcobBeV/NvIKkT9MZmIgr/Kj6SMhbxwgr/Kj6SMhbxwgr/Kj6SMhbxzemLwAuAN5AVkgLyAL5AVkgbyALJAXkAXyArJAXkAWyAvIAnkBWSAvIAvkBWSBvIAskBeQBfICskBeQBbIC8gCeQFZIC8gC+QFZIG8gCyQF5AF8gKyQF5AFsgLyAJ5AVkgLyAL5AVkgbyALJAXkAXyArJAXkAWyAvIAnkBWSAvIAvkBWSBvIAskBeQBfICskBeQBbIC8gCeQFZIC8gC+QFZIG8gCyQF5AF8gKyQF5AFsgLyAJ5AVkgLyAL5AVkgbyALJAXkAXyArJAXkAWyAvIAnkBTUaj0X92437zva3hpwAAAABJRU5ErkJggg==\"\n",
        "height=\"100\"></center>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h8TBGfzX4UWs",
        "colab_type": "text"
      },
      "source": [
        "What we're going to do now is consider *one part* of this chain of functions, shown below in the rectangle. We'll talk about this isolated function as $f_i$, and we'll write its input as $\\x$ and output as $\\y$ (remember this $\\x$ and $\\y$ are not the same as the input and output of the entire function $f$ that $f_i$ is part of). We also assume that the entire function (which $f_i$ is a part of) will produce a scalar output $\\L$ that represents the *loss* of the net on a particular example."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zoOMr0HPRZiF",
        "colab_type": "text"
      },
      "source": [
        "<center><img src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAArwAAADICAMAAAA9SMKPAAAAYFBMVEUAAAD///++vr+enp8jJy2nqaw2Wopqhqx+oMxedpdBRkwPEBHExsjAwsSnqauNkJKeoaPR09TNz9Cws7T8/Pz5+fn19fXt7e3f39/V1dXS0tLOzs7Ly8uvr6+AgIBoaGgoUZRCAAAACXBIWXMAAAsSAAALEgHS3X78AAALP0lEQVR4nO2daXubOhBG2a4di7S0SRu78cL//5f3YTHbjEASsmHMe/qhMXbA0hzLw0iQIARAKJAXiAXyArFAXiAWyAvEAnmBWCAvEAvkBWKBvEAskBeIBfICsUBeIBbIC8QCeYFYIC8QC+QFYoG8QCyQF4gF8gKxQF4gFsgLxAJ5gVggLxCLWHkzUEO6ZjNAXvGQrtkMkBfyigXyiod0zWYQLS/ZuDkw8ooE8oab7wXIKxrIKxLIG26+FyCvaCCvSCBvuPlegLyigbwigbzh5nsB8ooG8ooE8oab7wXIKxrIKxLIG26+F7AYHYgF8gKxQF4gFsgLxAJ5gVggLxAL5AVigbxALJAXiAXyArFAXiAWyAvEAnmBWCAvEAvkBWKBvEAskBeIBfICsUBeIBbIC8QCeYFYIC8QC+Rl2PmC7nph/vPFKlrjRd7X6hLIKyVSkJeBSAh5VxkpyMtAJIS8q4wU5GUovHuby2rl/T2X15P3dboE8k4AeQmQ9wlAXgLkZYC8DwfyMkDeUSAvAfI+AchLgLwMkPfhQF4GyDsK5CVA3icAeQlblvdw2JNtkPc5QF4GY3lVEgRBkObkCcj7DCAvg5m8e1WImxQCJ+RJyPsEIC+DkbyHNAiC/O1tX/8PeZ8O5GUwkXdXJAxFuluMv9zQC3mNOUVRdHT4PcjLYCDv4e7uW/FDoMgLIK8Zp6z89AdBElv/LuRlmJa3zBWq1xQ/BczLIa8B59rcktz2tyEvw7S8uzZV2PFZA+Sd5pQHPW6Wvw95Gabl7aYK+8OBPA95DcjSYMDJbgeQl2FS3nLEIFshrw3HohOT7D2KsuSusOXQC3kZJuUtK7xkK+S14Jy0We7xnj4ou31AXoZJeRNNngt5TTkXKUPWvFhBXl9MyqspMEBeU05pv7pwRdrgCyN5mUk1yGvIUQ2H2dLdFCds85mSt5yhYEsMkNeIMks4d19ZTlOcLXcDeRmm5N0ZFBsgr56YpAhfQZDerGeIIS/DlLwmxQbIq6VMeNO+qtnVYXED5GWYktek2AB5tZRJg/1KBgrkZdDLu98ppcqsIVWq/Fmb+kJeDVGZ3/LP2QF5GfTyHoYTmvrzNsirwdvA6yrvV+/Rk7rkL9nike7O9fLukoJy5E0qNFewGcn70AbVrC5S5dlaSja74Cbv1/n05+ldMjhopztUfqvJlWo+00q1Wyc/6L2d6+WttGzW8o4xLa+mQbfmfRfN6ZyU5007p9ujOcbCkSpISKnBGTd5vy+X787be06XfF8u/9g+ufHr6tiNOno7n5BXeZJX06Ck15xOcqgs2tM5xvMjdb5c/ukG32ouzXI2QoObvMdLr0+e0yXFQblge5G3t/MJeROTMq+BvJoGqV5zFPuEsbyDYzwxUhp7lbfTNfec99y190mZVHFQJtglx1u9OLSdpTlWW/LYaOKmu/MJeVOTSplJzqtvUFy/937xM6qy7dhm3FouUqy9p7JdGdnuhGu1oXh75/vJwLPOYXsHHXKu4n1tN5fl2G/yQg2dnU/IG/iSd6RBcSUvs9X2Ypl1RSr3mDW4l8q6b4/rkh+X3sP39/7DHy5dou+TsLE3bcZZ20i3Ox+XtyyXjTxvIe9Ig6rGDKadYpdv3FVFKvGYNcyo8/7917w92iXvWZb9bB9+/Mqy+KN9/DPL4n4fGVYPOwel1PbWY23xyK6a2OzcQF5teddOXn2DIi63VU6DllWkYl+R+uYa9u0za5gzSVH2STk00C75lWVZ1n5mP4uH1+bhj+Jh7NIl4Z/moAzdsTeydrfd+bi8ZsUG00kKbYMUHXojhytsw8Ui9c00LB+cl8xjRN6LEUe2S4o2Z5/9HvrV76GM7RLzg3K0Y2/cTSBsWzQur1mx4S4v2T0L0yBm6NUNvNweKWuIVOJvhsKHvMXb03RJ1Dz8mfW+nD71n2eyex4m2CX3sZd113jn4/KmJmvKLOXlGpQMh17twEv2xrNIpHqftjN7GurMg+SNep/f35c4y7JOJhX3OsyjvHd7+e8mshueCXkDeoecfa5IHjFbXjL0JkFKkkgvkerY+dhIZYG3dQ3h49KG35drr8kfn58f3cefn8OzAC9pQ1sxY3uI7IdlIm04UHnL++fMkpdtkOqXlWJy/mbXrtPykapaxDbWhRF5p/gzchrggIcTtpJ+zcGpRaPy5rTYkHOncHNP2O4F/eZLNiGVM9t2LR2pL79Zw5xS2fdIAeZRXaKtLHWoZouZlNd456PyKirvgZu1mFkqC4dDr37gnTzGEpH6Yhp21X4nuvGwSYqHdImu9N0lrqf/rcdew0kKrtiwZ+7tP3OSIhwOvaku47U5xrKRyr3WGl5perimqDPEqcPYazo9nJgVG2ZOD1d0JlNvrkPWiiKV+h145yzM+bemhTl3qhqZQ95rvDDHbGXDzIU5NadmYdkxdZxUXVGkYp9Tw+GsJZGdVUMLL4nscK/vDtc5mLXIZEkkU2zgcV4S2aXK3yP3gXeRSJ36K2bvJIOFUwyRYmucGpwXo3dXvC28GL2lnZuo7WUnpDiMF6NrluW45LwGDfpKq6H3K3XMFQfHWDJScTB5N7LcLid2vgyou1pzBReXlOSdwbayNzEtLhlfBkSKDdU1mTSRcL4MqEc99M7IeNcSqXJ5dXqNIv2AosiE+DiOOW9/ofFKLsDMe/Nqtb3krEGD2QWYzN1y9ocdm0j4uQCzWlGfpBNDlp7VRKp7F/RE5VlEImMwNPd5nUvfyxu+dpteTaRb3wBrQl5uZUPKLZH0dOn7/RqniDzjwnKRugaEJLldu8OwsvmqDF9H3q9rThdhVVcypvZhH5F3HzDy0sKvP3mrrNd54B2wWKTus/aEVOW3OCooIqis5hBfQV6l2gtu28zwo9mYKpVYzUlq5D0cDrtapd2h8zeHD1RnjzcdiT0OvItFKta52ye3m/9+BXm77W+H3qjXLVYjFy/vftDTzUt2XMrr7445w3RoDgtFqqwixNF7FN1uKgk0pONlNMpryJskqvyX9uVNVb1d+ZB3VxznTtqRVz1W3phf3+nEMpEq/3RKJ7uNMsbgBLc49QIvr5aEO1/zJ+/J40KsRSJ1ZP+8ZRTfVJNMJLntqBtCXh5LeZn1kD7l1V3848IykYo+yKb2uQKy1QzIy2An7549X/Mmr8+BF3/1nbJteXfc/Jo/eXOPAy/kpWxeXuZ8bYa8cedWl/qrLp2AvIRty8vOr82QN+uV/JTzxT8ckJewYXkP+3Jlg+uVFBzV3UbqJ27+7i8TQl6O7cq7a2o99ClXeasSaPXz2d/8RAnkJWy8VMbjLG/ezgieU8vZ/ikgLwHyMjjLe24WV90sV1lNA3kJkJdhRrUhCNL8dstT65Uqk0BeAuRlmFHnPdc38E88rSVrgbwEyMswa5LiFN9uVnfvNwTyEiAvA/6I4MOBvAyQd5TXk9cLZM/LsPPFStrT8lqRgrwMkFdGpCAvA5EQ8q4yUpCXgUgIeVcZKS/yArAEkBeIBfICsUBeIBbIC8QCeYFYIC8QC+QFYoG8QCyQF4gF8gKxQF4gFsgLxAJ5gVggLxAL5AVigbxALJAXiAXyArFAXiAWyAvEAnmBWCAvEAvkBWIRK2+WZT7/0ohQtt0LkFc0kFckkDfcfC9AXtFAXpFA3nDzvQB5RQN5RQJ5w833AuQVDeQVCeQNN98LkFc0kFckkDfcfC9AXtFAXpFA3nDzvSBaXgB5RQJv70iN4Hwgr3hI12wGyAt5xYIrKYBYIC8QC+QFYoG8QCyQF4gF8gKxQF4gFsgLxAJ5gVggLxAL5AVigbxALJAXiAXyArFAXiAWyAvEAnmBWCAvEAvkBTIJw/B/6qdFTPT7rFMAAAAASUVORK5CYII=\"\n",
        "height=\"100\"></center>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bHcBEY5L4HDY",
        "colab_type": "text"
      },
      "source": [
        "The RAD recipe gives us simple rules that tell us how to deal with each individual $f_i$; these rules then combine naturally to give us the gradient for the entire function. The gradient is \"back-propagated\" through all the $f_i$, starting at the last $f_i$ and moving backwards towards $f_1$, by calculating what are called vector-Jacobian products. The following diagram depicts one step in this process, corresponding to back-propogation through our selected $f_i$:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AHPYJU1_-0ta",
        "colab_type": "text"
      },
      "source": [
        "<center><img src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAArwAAADICAMAAAA9SMKPAAAAYFBMVEUAAAD////sHyfyXGL2lJj94+T6yMsUGB6nqaw3Wol+oMxjfZ8wNj6NkJLR09Ti8OodjFhZqoSMxaqu1sPJ5Nfx+PT+8fH8/Pz4+Pj19fXs7Ozf39/V1dXR0dHOzs7Ly8smd5szAAAACXBIWXMAAAsSAAALEgHS3X78AAANAklEQVR4nO2daXubOhBGzWJsX0xYjJ02beP//y/vIxYBHjBCAsSQ9/RDXeKYMDpRpdF2cABgCuQFbIG8gC2QF7AF8gK2QF7AFsgL2AJ5AVsgL2AL5AVsgbyALZAXsAXyArZAXsAWyAvYAnkBWyAvYAvkBWyBvIAtkBewhbW8EaggofkRQN5dQELzI4C8kJctkHcXkND8CNjLSy7+OFDzsgTyOpCXK5DXgbxcgbwO5OUK5HUgL1cgrwN5uQJ5HcjLFcjrQF6uQF4H8nIF8jqQlyuQ14G8XIG8DuQFgCOQF7AF8gK2QF7AFsgL2AJ5AVsgL2AL5AVsgbyALZAXsAXyArZAXsAWyAvYAnkBWyAvYAvkBWyBvIAtkBewBfICtkBewBbIC9gCeQFbIC9gC+Qd5DIXQzewxXkubD/IfPLuJiQ1kHfrJQV5ByESQt6NlRTkHYRICHk3VlKQdxDh3X+mbFXeD1P2J+8uQlIDed8BeQmQdwUgLwHy9gF5FwTyDgJ53wF5CZB3BSAvAfL2AXkXZE/yBkEQkIv6bEDeOE2SJMnur9fzNE1z8m5VIC/BdkgCz/U8z33V9+h7rus9yNvHsS9vnGSx49ySF1HzQulE217IS7AcklLbh+f6ncu+W+CT94+jLm8YhuTaDPLGya36O2nXvVmSZkLflHyDIpCXYDckQV3ldurewuXAdV2PfMM4ivJer4fD4XAaeKuJvGntZ5Zkr1dTyLsfeT3XLV/4LVGrethbTt5QmHu6Xk6Hw4l80VTeJEnKF3mr6s2SJC7/ysg3KAJ5CXZD4tbyHl33WF+sRPZd0hJWQUHe8HQ4HMLmb4pZzVv72TRw79XFrKcbpwrkJdgNie/VfjamBlVbt/naJMblLRoMYf2it+qdJ1XWVLNZVQnH2upC3h62EpKmdyabEnqMyhvW7v4nXhyu5A3m8t7zPG/Lezdo6kogL8F2SI6+7/tH5yHlPeolGSRj8hZthfIdp2XkzdMkTdPknkt5M4MMmQTyEuyG5OG7nh/4XhBIZTWbupIxeS9NU+Ey5K5ptkH0zW7pTcqb1n04EyAvwWpIgiq/G7ieVNY1azWMytuubQczvfryNi2EVI5IxHO0GiAvxWZIAq9J89byBoathjF5RW17IFfJu7TlTWV+LEvql7ekGrcwAvISbIbEk4lc320SvsvKex1KMMwjb8vTm0z4pnM0eSEvxWJIWq1bXw5IeK6rM6OhQUHe/nbuLPI2Fa+oeSuPkzmavJCXYjEkrdZtU9/qDau1GJH3IHMN79CVt926TctBtWKkTXtYrQXkJdgLSdDyVNa3xk1eu/I2CYZWz22WRBnk7cFeSFqt2ya367WGifV4L28zQvEWXXlbnjYv52k1bEXecxSRa9PYibxNk7eeuut1Wg0PjYzve3kjpWSDibxx9VJOccg7uYY81c2abUNeEUBycRo7kbcythleCzojFL5O+/e9vNdl5Y2lvFmSVj23NEmy7FZdNsiabUHe7yJ+5PI09iBvM5FMVrcPMYfXE8PFDjFZlXF5x5MNJtmG0s1crqPIk5I0y+/tKWeTsS7v76hQ93AiX5nGLrINtbNNdsxza7wg0JzkMCRveLlerpdyBvq1+POm6astb160dO+ZdFfkH9KkxsBdq/J+X6/XYiZIMT5JvjyNXcj78Fzv+BCL2Cp3fdcLfL8lMPkWBQblPbyyhLzlrJxm9eW9tDjPSoEN1l9alffcDt1aHbZf5Ar5sZYIyfhtCx5ilaXr144+KosD3yuu9w9WjH32kLyX6+l6KmsP8ep6uk6QV/GBCuJb3ih6b17mt7zuzPUydg+bJSVqXsGKNe/nn+/f5OILC4RE5ba6jH/2kLyllGqZshd5l3wg9XtsoaTWk/fX3+fzH7n6wvwh+fxQuK0mn+OP9Fbeq468Cjdd48EWKKl/Ux9sNXlFPJ5f5PILs4dE7bZ6qHz2W3lPapmyjrxLPtCUe2yhpNZq84p6V+E/hblDIn6bl/pP9lPlkUblHZ9T1pFX6aZrPNgWSmqlmvf3t9qPNnNIVG+rg9pnv5V3cN3PoLxLPlCN2j22UFLryKv8o/WF5Pl8+0/y/iYk9t19K+9FbVpOS97tuKtSUuelS2oVecWP9lcpDUJD8jxHUftSFEXn79bjR1HUjVETEvXbTkf1s9/JG02Vt7jpJ7nH/A+mcI+ekorWLqk12rxf38/nP7WY05CIZ27F5OWfImBR1B+SrwXd/VLV6J28qsmGWl7lm67xYFsoqRVq3q+nEsV7aUiKZ74NheQ8HJIpt52K+mePykuu9lDKO/mBslSFTK+sBkrqOVQ0S5TU8vIq/mhv5dX4fZ502/Y8hnd4kx/pnbyDO+T01rwTH6haMDxOqldWtKT66pUFSoqTvOfuMz+7v97DLSkm8r4mG8K+7IOmvHGuQqxXVrSknrSkzguU1Ku8y7Z5zZoNH8/ule9zuxcgvvwakeo/oz/kDr2QH1eFWZoNIZU37G9IaDYbFn0wlZK6dUrqNlRSG242TOok94RkOq1uwMdC/Zs5OmxFpiyi8tIuXNVh21KyYfaS0nmwzafK9ENS3PbfdlNlPWnekNTFbXkXzf1NfbDZS+pDM1W27UEK7ZBsfJCiL9kQRrTe5TNIsXZJrTS3AcPDFNWZDTMMD9/y95N3NR7s5wwPK08omX26x+R5LEff83ylRROmE3OUM2WmE3PyYo/TVHmppZ2JORoPtt6UyA8rUyJ/TZxoJzY59cWCIPIVitmUyJA2eYcwmxKZFTs23CcsWLMyJXJqSa06GV1pEvUCU5wnzd2uNu8/qh26ZjQZfWBmw9s2b33Tv+Q+w+TNbiPKS9bsTEafPMt+xTVslpYB/ZqwuERKq7hHg8kyIDKzIRzc199oGZDcSvo2YVteK8uAppRUwaqrh7e9ALOzbY7qPk9j2ckReTt1bji0N7rJAkxHDgA3W5CosOUFmDVY+t7Gb+08YrjDXsWwvH3JhlPfEAWVdxKZ7KnNsql0zQZ2zPlTrn03zCbtRt5HUNe3nunephWD8oZ9yQaa+DWXt2GevU0rNiBvNMuuI/s7ysr0BCtJr7zhJbxUezZcwktzEkVIdZ5D3viWZVksdjnN7iLhm+dFDZzn8X1C+vcFuyX1fT5X2zaIKiCKDJak7kjewPNcz38UNa9fUmw8Uh5xNZ1eeV/2ypFvuPQ3ec3kFZs93fIsycWu0vVWZU49Y5LlXmVPst2QQdJhP80GT+zqdPTdYkv0ah6knDA51/GtF7lLTrlrjhT2uoC8fypB/yRiY/84y4pd9pzidCthM/kGRSAvwa688jwgsfGT7wSB2KnMr7ZNDwKd0yn65B1koL9mIm8se2nC2riqcMuty9Ip2YdXNtPAM2Un8h7bG/OWL6tDrfSPppgkb+98SDN5m33Qs+plVjcWcqP+G+QlWA1JK8PgVjmzh7D3aJB6mChvb3/NQN7Wxv63KuF7T6qq16jihbwUmyEJOqdSuPKia3IK5hR5h5q8BvK2DqCQZ1lVVW9udg4m5CXYDEnr9JTW0UCFvfpnsU2R99I318FI3rx7llXZ1L2X6YbM7FAgyEuwGZLuWVbH1muDAYsp8g711/TlbU4OFD23rLma3GLDA4ghL8FiSI4deeXLst2gPVasLG8YFgdU9J6dbSJv6yyr+jDMOCk29zcbK4a8BIshaTV5g04D4ugZ2Ksq76XOV/a1ek3klRv6t2TN5ECFPpCXYLfZUBn6aPXcROZXpBz0TqSYmG0YQlveWNa8nanoouo1naQDeQl2O2zNuZfVpUc5anHU77TZlVc6m8pz2Kp/GrZ4IW8PllNlRWOhPg/o6NXthaLh4PlzHmW1krxi8U9+v6XyPKCS2Hx2JOQlWA2JOH7Y9zvDbGUnLnC1s72W5RXTcnrOrLo3vTddIC/Bbkgeged5geFR2V1sy/tKnMVFj814Zi/kJewmJDUbkzcvZufMUPFCXgrk7WM+eUUjIp+j4oW8lPNckE+2xGUu5vnxRZIsTo1TDXsqKcg7yMbkLccn5ljKRiLOtaQg7yBEQsvyirOIjdu7DuTtgzwa15DUEAltyzsXJOJcS2o+eQFYGcgL2AJ5AVsgL2AL5AVsgbyALZAXsAXyArZAXsAWyAvYAnkBWyAvYAvkBWyBvIAtkBewBfICtkBewBbIC9gCeQFbIC9gC+QFbIG8gC2QF7CFtbxRFGkf/bkffm4UIC97IC9LIK8DebkCeR3IyxXI60BerkBeB/JyBfI6kJcrkNeBvFyBvA7k5QrkdSAvVyCvA3m5AnkdyMsVyOtAXq5EoIJzKeoDeXcBCc2PAPJCXrZA3l1AQvMjwEoKwBbIC9gCeQFbIC9gC+QFbIG8gC2QF7AF8gK2QF7AFsgL2AJ5AVsgL2AL5AVsgbyALZAXsAXyArZAXsAWyAvYAnkBWyAvYAvkBWyBvIAnjuP8D/weQ1amYuKvAAAAAElFTkSuQmCC\"\n",
        "height=\"100\"></center>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FK1Jgc_0V6Xm",
        "colab_type": "text"
      },
      "source": [
        "What back-propogation actually does is to compute the gradient of the loss with respect to the input vector $\\red{\\ipartialfrac{\\L}{\\x}}$ in terms of the $\\green{\\textrm{vector}}\\textrm{-}\\blue{\\textrm{Jacobian}}$ product, which is a vector form of the familiar **chain rule** from calculus:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9nP6fLLPmLn7",
        "colab_type": "text"
      },
      "source": [
        "$$\\red{\\dldx} = \\blue \\dydx^\\mathrm{T}\\ngapp   \\cdot {\\green \\dldy}$$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3RWnZ11wWSMe",
        "colab_type": "text"
      },
      "source": [
        "Let's look at all the elements of this equation. We must think carefully about the terms in this equation, as they are *not scalars*, they are arrays. This table summarizes the situation:\n",
        "\n",
        "|  | Shape | Array Components  | Meaning |\n",
        "| :--- | :--- | :--- | :--- |\n",
        "| $\\red{\\idldx}$ | column vector of size $m,1$ | $\\idld{x_i}$ (where $i \\le m$) | summarizes how changing input $\\x$ will change $\\L$\n",
        "| $\\green{\\idldy}$ |  column vector of size $n,1$ | $\\idld{y_i}$ (where $i \\le n$) | summarizes how changing output $\\y$ will change $\\L$\n",
        "| $\\blue{\\idydx}^\\mathrm{T} $ | matrix of size $m,n$ | $\\ipartialfrac{y_j}{x_i}$ (where $i \\le m$, $j \\le n$) | summarizes how changing input $\\x$ will change output $\\y$\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SfvlGitHLTCT",
        "colab_type": "text"
      },
      "source": [
        "Let's write out the individual components of this equation, to make the situation more explicit:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7bwZr8Ou91gu",
        "colab_type": "text"
      },
      "source": [
        "$$\\red{\\begin{pmatrix}\n",
        "\\ipartialfrac{\\L}{x_1}\\\\\n",
        "\\ipartialfrac{\\L}{x_2}\\\\\n",
        "\\vdots \\\\\n",
        "\\ipartialfrac{\\L}{x_m} \\\\\n",
        "\\end{pmatrix}}\n",
        "= \n",
        "\\blue{\\begin{pmatrix}\n",
        "\\ipartialfrac{y_1}{x_1} & \\ipartialfrac{y_2}{x_1} & \\dots & \\ipartialfrac{y_n}{x_1}\\\\\n",
        "\\ipartialfrac{y_1}{x_2} & \\ipartialfrac{y_2}{x_2} & \\dots & \\ipartialfrac{y_n}{x_2}\\\\\n",
        "\\vdots & \\vdots & \\ddots & \\vdots \\\\\n",
        "\\ipartialfrac{y_1}{x_m} & \\ipartialfrac{y_2}{x_m} & \\dots & \\ipartialfrac{y_n}{x_m}\\\\\n",
        "\\end{pmatrix}}\n",
        "\\cdot\n",
        "\\green{\\begin{pmatrix}\n",
        "\\ipartialfrac{\\L}{y_1}\\\\\n",
        "\\ipartialfrac{\\L}{y_2}\\\\\n",
        "\\vdots \\\\\n",
        "\\ipartialfrac{\\L}{y_n}\\\\\n",
        "\\end{pmatrix}}\n",
        "$$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zBscm-PYMYwa",
        "colab_type": "text"
      },
      "source": [
        "The matrix $\\blue{\\ipartialfrac{\\y}{\\x}}$ is called a *Jacobian matrix*. It summarizes all the ways in which changing the vector $\\x$ by a small amount will influence the vector $\\y$. It allows us to transform the gradient vector $\\green{\\ipartialfrac{\\L}{\\y}}$ (which summarizes how changing $\\y$ will change $\\L$), into $\\red{\\ipartialfrac{\\L}{\\x}}$ (which summarizes how changing $\\x$ will change $\\L$). \n",
        "\n",
        "The way this transformation works is intuitive: since $\\x$ influences $\\L$ _through_ $\\y$, we can combine all the ways that this influence happens using a matrix multiplication, which linearly sums all the influences through independent components of $\\y$. This summing of influences is clear when we write out the matrix multiplication above in component form:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "foD6qYTBAv-r",
        "colab_type": "text"
      },
      "source": [
        "$$\\red{\\dld {x_i} } = \\sum_{j\\le n} \\green{\\dld{y_j}} \\blue{\\partialfrac{y_j}{x_i}}$$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LSuTfsehAx0A",
        "colab_type": "text"
      },
      "source": [
        "This equation says: the $\\red{\\textrm{influence of $x_i$ on $\\L$}}$ is the combination\\* of all the ways that $\\blue{\\textrm{$x_i$ influences $y_j$ }} \\green{\\textrm{which goes on to influence $\\L$}}$. This is very intuitive: $\\x$ influences $\\L$ *through* its influence on $\\y$.\n",
        "\n",
        "> <small>\\* Remember that we are assuming these influences are *infinitesimal*, in other words, so small that we can model them one at a time and combine them linearly. Since neural nets usually contain non-linearities, this assumption is *not* true in practice, but if the parameter updates are small enough it is a sufficiently good approximation that we can still use these gradients to optimize the loss $\\L$ of a network.</small>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m3iw8xtZWSKb",
        "colab_type": "text"
      },
      "source": [
        "A very important point, though, is that we never actually *compute* the Jacobian matrix $\\blue{\\ipartialfrac{\\y}{\\x}}$ directly, or store it as a matrix. Instead, we compute the Jacobian-vector product $\\blue{\\ipartialfrac{\\y}{\\x}} \\cdot \\green{\\ipartialfrac{\\L}{y}}$ *implicitly*, giving us the vector $\\red{\\ipartialfrac{\\L}{\\x}}$. The exact calculation depends on the function/layer, as we will see later, but it is much more efficient than calculating and using the full Jacobian matrix *explicitly*."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HsGXiJbnWSHx",
        "colab_type": "text"
      },
      "source": [
        "All of the above may seem abstract, but try to get comfortable with the basic idea, which is that we are using our knowledge of how the _output_ of a layer influences the final loss $\\L$, to compute how the _input_ of a layer influences the final loss! This measurement of influence is summarized by a gradient vector such as $\\idldx$. This propogation of gradients from the output of a function to its input is called _backpropogation_, and is the what RAD tells us how to do. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j8IQAC-zKRzY",
        "colab_type": "text"
      },
      "source": [
        "### A special notation for partial derivatives of the loss "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sIp9HLGyKUDh",
        "colab_type": "text"
      },
      "source": [
        "In this practical we'll be working a lot with partial derivatives of the scalar loss $\\L$ with respect to arrays (vectors, matrices, etc.). Normally this would be written as ${\\partial \\L}/{\\partial \\mathbf x}$, ${\\partial \\L}/{\\partial \\mathbf W}$, and so on. But these quantities are so important and frequent that we're going to introduce a \"shortcut\" notation for this that is faster to read:\n",
        "\n",
        "$$\\red \\xhat := \\red{\\dld \\x}$$\n",
        "\n",
        "This notation will make it easier to perceive the elegant ideas that are behind automatic differentiation, but it isn't a standardized notation (for example, some other texts use $\\tilde x$).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "59BmN54wql45",
        "colab_type": "text"
      },
      "source": [
        "## Implementing the Linear layer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mQM5hcqC_0vO",
        "colab_type": "text"
      },
      "source": [
        "Now that we've seen how we can compute a gradient for a single function using a vector-Jacobian product, we can begin implementing our deep learning framework! We will start with implementing the workhorse layer of all all deep neural nets, which is called (by different people) either the \"fully-connected\", \"linear\", \"affine\", or \"dense\" layer. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7TkltNwhWSFX",
        "colab_type": "text"
      },
      "source": [
        "This layer just computes the function $\\y = \\W \\x + \\b$, where $\\y$ and $\\b$ are vectors and $W$ is a matrix. The input vector $\\x$ either comes from the previous layer, or this is the first layer, from the input to the entire network. The $\\W$ and $\\b$ are the parameters (summarised before as $\\th$) which will be learned by gradient descent. The matrix $\\W$ is often called the \"weight matrix\", and the vector $\\b$ is called the \"bias vector\". The general formula for computing the components of the matrix equation $\\y = \\W \\x + \\b$ is: "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mEpDd5z7XXAg",
        "colab_type": "text"
      },
      "source": [
        "$$ y_i = b_i + \\sum_j W_{ij} \\gapp x_j$$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zMjgSUAsDFEz",
        "colab_type": "text"
      },
      "source": [
        "> *Note*: If it isn't clear, the index $i$ and $j$ refer to particular components of the input and output vectors $\\x$ and $\\y$ respectively. So if, for example, $\\x = (x_1, x_2, x_3, x_4)$, then $i$ takes possible values $i = 1, \\dots, 4$. We usually don't bother writing these ranges explicitly in summations, etc."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EgeUvHKYWiTW",
        "colab_type": "text"
      },
      "source": [
        "The way you write this in numpy is pretty simple, it is just ```b + W.dot(x)```, so that part is easy!\n",
        "\n",
        "Let's summarize what we've figured out so far in the form of a Python class. Note that this definition won't be complete, so we're calling it `Linear1` for now... as we add features we'll obtain `Linear2`, `Linear3`, etc."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gR3XXLBMWnrR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "class Linear1:\n",
        "    def __init__(self, num_in, num_out):\n",
        "        self.W = np.random.randn(num_out, num_in)\n",
        "        self.b = np.random.randn(num_out)\n",
        "    \n",
        "    def forward(self, x):\n",
        "        y = self.W.dot(x) + self.b\n",
        "        return y"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EU4fwW9sWqMe",
        "colab_type": "text"
      },
      "source": [
        "The `__init__()` method sets up the weight matrix and bias vector with random values, and the `forward()` method computes the output `y` from `x` and the current weights and biases. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tG1xmKfWFeDJ",
        "colab_type": "text"
      },
      "source": [
        "### The forward pass"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U1zA2tK0FnFr",
        "colab_type": "text"
      },
      "source": [
        "As you can see, the `Linear` class has one other method, `forward`, which is what computes the output $y$ from the input $x$.\n",
        "\n",
        "Let's create an instance of this class and apply it to a vector to see that we've got it right. \n",
        "\n",
        "**Exercise**: Run the following code, but before you do, try to predict what the output will look like. Hint: try answer if it's possible to know the exact numbers it will contain, given the use of `randn` above?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dx6-BA6mmS2V",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "linear = Linear1(2, 3)\n",
        "x = np.array([-1, 1])\n",
        "linear.forward(x)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TmjTqUCEmKCM",
        "colab_type": "text"
      },
      "source": [
        "Below is a diagram that depicts the **computational graph** that we've just implemented, which is the term that is often used for the way that data flows through a network to turn the input $\\x$ into the output $\\y$ (the parameters $\\W$ and $\\b$ behave a little like inputs here, as you can see). When we combine many layers together into a multi-layer perceptron (or more complex network), these graphs will \"lock together\" into a larger graph. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GK2hdgNDm_iD",
        "colab_type": "text"
      },
      "source": [
        "<center><img src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAArwAAAEsCAMAAAAW6pCvAAAArlBMVEUAAAD///8ZilUOFw1dl1NWjE0dLxoyUS0HDAZBaTkrRiZlo1lzumZPgEY6XTNsrmBIdEAkOiAWIxP4+Pjy8vLv7+/r6+vl5eXf39/Y2NjPz8/MzMzLy8vJycnFxcXCwsK/v7+8vLy5ubmzs7OysrKvr6+srKypqammpqalpaWjo6Ofn5+cnJyZmZmPj49/f39wcHBmZmZgYGBQUFBAQEAwMDAmJiYgICAQEBANDQ1cATUUAAAACXBIWXMAAAsSAAALEgHS3X78AAAQDUlEQVR4nO3dfV/TSBfG8eTGQim0RUAEBDai2xVqF8Wy4vt/Y/cnT81JJk/ATM45nev3z+6mFWft1+nkoWkQIqQ04EVqA16kNuBFagNepDbgRWoDXqQ24EVqA16kNuBFagNepDbgRWoDXqQ24EVqA16kNuBFagNepDbgRWoDXqQ24EVqA16kNuBFagNepDbgRWoDXqS2rcb7vwEyflPF49IW8DIgMX6IkHFpC3gZkBg/RMi4tAW8DEiMHyJkXNraeryHDnsLXonj0hbwMiABXjsBLwMS4LUT8DIgAV47AS8DEuC1E/AyIAFeOwEvAxLgtRPwMiABXjsBLwMS4LUT8DIgAV47AS8DEuC1E/AyIAFeOwEvAxLgtRPwMiABXjsBLwMS4LUT8DIgAV47AS8DEuC1E/AyIAFeO3mFdzcY7246DoJglD8SBDvFI+Mg2C1+0Yg8Ng5mwCsnz/BWInjLlfGSgFdQXuF9NxpNJxnE8d50NJpvhI5GR8fZIzv7o9G74hfNR9PJOH3k/f7oAHjl5N+ad5ZCfGc8cLiT2jW2Hx4exLDHo+pmZ3hHo9F0Npvt7RqP9Ap4t6BaJPMUr7H98PAoeWBibI/11nJ3hbdYpRgPAS/Jw6MN7xMWU2N7xnpsbI87rkMNvLx5iLd5gm1kfXg4Dg6Mbc7wzmZ7x8DbnYd4myfYZtbT4NjY5naHbR94O/PxJEXjBDttZD0J9o1tbvHOgbczH/E2TrDZYbQa1rWrBreHyoC3Mx/xNq4bssO5Juv6VQPwMufltQ2T+gl2Ghw3sK5fNQAvc17indZPsJNgv2E5XL9qAF7m/LyqLFkfjOfG1oP65fBR/aoBeJnzE2+6bjgqb4wXtvXL4ff1qwbgZc5PvOm64X15Y7KwrVs3zIP6VQPwMufpxejpcYV5ZdtB/WG0plUD8DLnKd6adUNKNLtsp8S6adUAvMx5irdm3ZARNVk3rhoGw3sw2p/NZlPjksymgHcLakFirBtyoibrxlXDMHgP9nY2F5m9P6oeIakNeLegFiTGBLsharBuXDUMgfcg/ujSeDe5Mj05EDLrwRd4t6AWJMYEuyFaZd28ahgAb/yxj0l+Ffx8P/6LtdO9egDeLagNSWWCLYhWWTevGgbAGwS79G/OfNfYz6wLeLegNiSVCZYQrbBuXjUMgbd6sq+XXuDdgtqQVCZYQrTMumXVMABe4/qL+dg8iWIEvFtQK5LSBEuJllm3rBoGwGuub49qL8uwMy5t+Yu3NMGWiJZYt6waWPCmo9vzHm/oNd7SBFsiSlm3rRp48E7MU4DAu4W1IyETbJkoZX1UvXyHHe+RcQrQCHi3oHYkZIKtLGwJ6/dtTFjwvjNOARoB7xbUjoRMsJWFbcF63voGzYI3faDuplRvHpe2fL4/72aCrS5sC9atqwZOvK3XmwHvFtSBZDPBGkQ3rFtXDcDLnM94NxOsQTRn3b5qwLKBOa9v659NsCbRnLUxJdtB8ia8B+klD8Z2C+PSltd4swm2hmjG2piS7SB5E970Tu3NZ06AdzvqQpJNsDVEU9b7xpRsB8mb8O4lDzSfOQHe7agTSXZ/J5Nodss9c0q2guRNeHdqL9ixMi5t+Y03u7NeDdGMtTElW0Hy9gtzWide4N2GOpFMG4lOGqZkK0j64zX2y+Y7uCRyk+dfImh8YC3P+JhQXbgYnTfP8U4aiY57KHGOdycIJvRvVvKtRPgYUJ7neKeNGCYNU7INJH3x7sz3gmBnM7z5DB/ALOX7dw+Pm4hOm6ZkC0j64j06PJzuBMHO3tFoNJolt1HDR99JvuOdNBLdaZiSLSDpiTfZW5vvF/ccCY73cdMRku94Dxp33WddnxQb7Otb3+3v7e7uTmbT9gNkbx+XtnzHe3hgfrFl1qjTykB4XxzwbkFSkQCvnYCXAQnw2gl4GZAAr52AlwEJ8NoJeBmQAK+dgJcBCfDaCXgZkACvnYCXAQnw2gl4GZAAr52AlwEJ8NoJeBmQAK+dgJcBCfDaCXgZkACvnYCXAQnw2gl4GZAAr52AlwEJ8NoJeBmQAK+dth6v64zfVPG4tAW8DEiMHyJkXNoCXgYkxg8RMi5tAS8DEuOHCBmXtrYa70tbrVYrjEtPwEsCXl0BLwl4dQW8JODVFfCSgFdXwEsCXl0BLwl4dQW8JODVFfCSgFdXwEsCXl0BLwl4dQW8JODVFfCSgFdXwEsCXl0BLwl4dQW8JODVFfCSgFdXwEsCXl0BLwl4dQW8JODVFfCSgFdXwEsCXl0BLwl4dQW8JODVFfCSgFdXwEsCXl0BLwl4dQW8JODVFfCSgFdXwEsCXl0BLwl4dQW8JODVFfCSgFdXwEsCXl0BLwl4dQW8JODVFfCSgFdXwEsCXl0BLwl4dQW8JODVFfCSgFdXwEsCXl0BLwl4dQW8JODVFfCSgFdXwEsCXl0BLwl4dQW8JODVFfCSgFdXwEsCXl0BLwl4dQW8JODVFfCSgFdXwEsCXl0BLwl4dQW8JODVFfCSgFdXwEsCXl0BLwl4dQW8JODVFfCSgFdXwEsCXl0BLwl4dQW8JODVFfCSgFdXwEsCXl0BLwl4dQW8JODVFfCSgFdXwEsCXl0BLwl4dQW8JODVFfCGYXh2d5H8c4Pk4u7MeBJDUsclJeANw9PlapWgyJGcrVbLU+Npgyd1XGIC3jAMo9VqeVYgOVuuVpHxJIakjktKwBuG4ckiVZIiiY0sTownMSR1XFIC3jBXcpoiORVkROq4hAS8SbGSxUmMJP034wlMSR2XjIA37SSe12Ik8VwnyIjUcYkIeLPiFWXaUtThKKnjkhDw5uVKpBmROi4BAe+mVIk8I1LHxR/wFl3ESM6NzexJHRd7wEu6WK0ujI0Ckjou7oCXdiHUiNRxMQe8cafnV18X+V794uvVuZArCKSOS0jAG57d3K2M7m7Yd5CkjktOvuM9uaoRkjm5YjwpIHVcovIb72m0NGyQlhHT27TUcQnLZ7wnVwWR+8WXz1HS5y+L+4IJxywndVzi8hjvx/yN+fvi9vpTqevbxff8Tfqj8Qs9HZe8vMV7cpPPbVUhuZN8nrsZdJKTOi6J+Yr3NDsC9S0yeGyKvqXPWQy4wpQ6LpF5ije72uX7rSGj1G36Jj3cdQVSxyUzP/FmRv6pfWMuvUn/M6gSqeMSmpd4UyNd0xud5IZRInVcUvMRb2bkLwNEbX8NpkTquMTmId7TxMh951tz3nWye+/+hglSxyU3//CeLF5mJFfi+tOPUsclOP/wJsdRv7/AyKdP18k79I3xo7wYl+C8w/vxJevKvHR96fScltRxSc43vCfJudde+/O02+SMrMM3aKnjEp1veK+S46gGgs6S46pXxo/b+nGJzjO8yR79yxaWacny0t2evdRxyc4zvNGr3pw/5W/Qzu7RKHVcsvMLb3zzpNU3A0Cv4qthXN1wSeq4hOcX3mRl2XK9VluRw9Wl1HEJzy+88S79vfHy9yw+JXBn/MghxhUEQWBs3ORwXMLzCu/Zq1eWn/LVpZNLCbrG1Y7X3bikNxze5frHQ9bjer3MN683m38VGy1Gd8RvXrlLn5bs2Fs7nfWScbXjdTeuSj+KV/DHev2jeHDzwA8nL2Fjw+F9CGgP+ebajfY6W0bFzkz87rwwXvzeLSy+P79oXO143Y2r0rr0Cq6LB9dOX8Lmth3vHbly5bTj3bmr5P25ZWZyNq4OvK7GVc1jvHEfHp7T/8fLYlO65dfy0ni2jS7IdVfn8Yvc8u7c1fXK3t0aXzSuDryuxlXXMnu5PpQei37HG38vB76V5dA7bJfp//zfxZb4//vJjdwwLH2HzlX7Pn139xYPSr1kXB14XY2rtmXy+v2oPBRvfax7utMGP9qQ6n3eaF3W/FFYLX410s8bfO1YWnYWLy6/2hrbC8bVhdfRuOpLJtnn8swbv4rr2mc7bfhDZWW98X853kPdvBrxa/zFeOWLfv4Jgj8/jc1FX2Jlxs93Pq5OvI7GVV9Ut7ZdBwHD3a8ZjvNSvZF7u+m9EPLv4vtsvPJ50X/pcvw/45FNn1dWv8G637h64HU0robW5tQbcSwaeE5SFHqXdAHhruS7+M7D9nOwP/Md5ua5N7KLpN+4euB1NK6GaqZenomX5wxbrrfV7sp2F+1I/uR4/xgPlZAMNa6gMeOpjsZlvCRZxtQbOd5taYrn9HCmt3TMrJrxh2klE4mJpfE5bpAkmb9nE93B8DbqjaoHHH4bO3DDxHRtQ3bErG29a/xZWslEYmJpfA7wpqUnJTYLhaWx/zZQXBfmVI+YmRl/lm9+LVa1SPKwbKiOy3hJ8s6TYWx20bgmXr6rytKzxW167ZXsgFx07Bhx7bB1jYtrh63Zbhg+0qmXbeJlw7vMpAyht+chKRwqq4yrudLUyzbxcuGNjzMsO1cOdsJJileOqyUy9T607rk4jQdveoyse91rI5wefu24Wkqn3nV6YdXv5ue5jQVvfnx3CL24MOfV42or3WWJWCdeFrzFuYlMr8OzM7gk8vXjaiu9knXNOvFy4H0kk22q98ndgh8Xo79+XK1lUy/nxMuA97F0Xs21XnwM6A3jaiudep+eOS6FzBv8YvSnymXLl8nfYHdXo+MDmK8fV2v5x7oY79YzKN4Pf/+qnhYPw6f0eO8Qfwb46LvFss9vMU68A+Jdr39vTnAWy6TLp3zb87r0cWonddzcoz3cdKTcknviHRAvPTtfnE6MSmftXf81xu2ebPabeeIdFO/TOu25jPd5nef8jwI32rPZsvWS1gHCLU57hlucGp3zfPinCDeX7hduLm32yPPhnyLc1r9fuK2/EfvEiy9U6Re+UMWMfeLFV1n1Cl9llbQkd/dk+9QlCV8i2CN8iWDSv6WjnGu2a9A34etbexjB17cmpR95z/7jIQj+NZ4xcPji7J5G8MXZ2Yn89N8vg+CX8YSh8xBvcr11//Vluq7s/GzBFo8rL/nsT0r28tnldax98xFvrqTXvv3tgEakjisrvgBwnZB9cHoNdu+8xJspWf3T+RZ9nRxHHcyI1HFlxR+BeXx4eHw27i7Nk594cyVdk1w6vQ1oROq4si6zG/ivZXzlpqd4k3sTxH1ruZYr+pY+ZzHg2Vep48o7Xz48NN9AcuB8xRue3GR3Nbq/rX2Tvr69z55wM+gJLKnjkpi3eMPw412m4Pui6uT6dvE9e/BusPNX0sclL4/xhidXy81d5e4XXz5HUfRXFH3+srjfbF5eMUxvUsclLp/xhuFpVDCpaRnxXGsodlzC8htvPMvdGTbyN2bO2U3quETlO9748NRNjZO7G/avopY6LjkBb9zp+dXXRe5j8fXqXMjbstRxCQl4kdqAF6kNeJHagBepDXiR2oAXqQ14kdqAF6kNeJHagBepDXiR2oAXqQ14kdqAF6kNeJHagBepDXiR2oAXqQ14kdqAF6kNeJHagBepDXiR2oAXqQ14kdqAF6kNeJHagBepDXiR2oAX6SwMw/8DVKG4HThUmaoAAAAASUVORK5CYII=\"\n",
        "height=\"100\"></center>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7R4nYHEWpxVU",
        "colab_type": "text"
      },
      "source": [
        "**Exercise**: Make sure you understand how this diagram shows the operation of the `forward(x)` function that we wrote in the definition of the `Linear1` class above. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N_6rXtJVquwN",
        "colab_type": "text"
      },
      "source": [
        "### Gradients and the backward pass"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vY9Lg2-nWsG3",
        "colab_type": "text"
      },
      "source": [
        "Now, we haven't yet tackled the gradients, which involve code to compute the vector-Jacobian product. This is more complex, because there is not just one vector-Jacobian product, there are actually *three*. That is because there are three ingredients of a Linear layer that go into computing $\\y$! They are $\\W$, $\\b$, and $\\x$, and each gets its own gradient. Here's a picture that illustrates the computational graph that computes the gradients:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "92v2uVyYiNqM",
        "colab_type": "text"
      },
      "source": [
        "<center><img src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAArwAAAEsCAMAAAAW6pCvAAAArlBMVEUAAAD///+6LS0QCAhfMjLddXWOS0v8hobsfn7NbW29ZWWtXFyeVFR+Q0NPKiovGRkgERFuOzs/IiL4+Pjy8vLv7+/r6+vl5eXf39/Y2NjPz8/MzMzLy8vJycnFxcXCwsK/v7+8vLy5ubmzs7OysrKvr6+srKypqammpqalpaWjo6Ofn5+cnJyZmZmPj49/f39wcHBmZmZgYGBQUFBAQEAwMDAmJiYgICAQEBANDQ2WZ9bGAAAACXBIWXMAAAsSAAALEgHS3X78AAARHUlEQVR4nO2dbV+bWBOHD3eixsQ0PletWmq72SrN2rXJ1n7/L3b/gIEAw1MUzjnD+V+vdgHjWC6GOY9RHgBCgbxALJAXiAXyArFAXiAWyAvEAnmBWCAvEAvkBWKBvEAskBeIBfICsUBeIBbIC8QCeYFYIC8QC+QFYoG8QCyQF4gF8gKxQF4gFsgLxAJ5gVggLxAL5AVigbxALJAXiGWw8v5PA+yXCo5LIpBXsyTsQyyJSyKQV7Mk7EMsiUsikFezJOxDLIlLIoOW96BH3iOvjXFJBPJqlgTydgfk1SwJ5O0OyKtZEsjbHZBXsySQtzsgb8KEHakF8poH8hKT0ZQdqwPymgfyxkwWanTIjtYAec0DeSMmC6V2sxfymgfyRozVaLpQix3qXshrHsh7ELt7GGbfHezVKO9OTUnIOwB2kWQeVwyThTpl56rQJ+9uTUnIOwB2kOQoqXYnCzVmZyvQJu+OTUnIOwDaS3K0lWMHe3XJu2tTEvIOgNaSHGbVmCzUPruiFF3y7tqUhLwDoK0kh6NcWpss1BG7pgxN8u7clIS8A6ClJHujwit5slAzdlUJeuTdvSkJeQdAS0mOWDk5WbQqe7XI+4amJOQdAG0l4a9jfqQMHfK+pSkJeQfAAKZEvqkpCXkHgHx539aUhLwDQLy8b2xKQt4BIF7eNzYlIe8AkF828IYjP8KBvAMAa9iGD+TVLAnk7Q5n5D1Vo9OUcLJLOs9QqePtmZHKjmVNM+dGag557cIheQtk5M2TlzcD5LUMZ+Q9nE5nYxJxdDKbTtPWz3Q6PVrQmeP96TTTyJ9MZ+NRfObD/nQP8tqFWzXvPBaxZHbscewuO35wsBdNqGWLGXqTdzqdzubz+Un7RR2dxCURt+SdxPKy4+E0gpDSftS9Ut37kndbpbBTrYC8A6Bckg+RFiUDVbHWI3Y8pHRODOQ1j2PyVifYSq0PDkZqjx3rTd75/GQBedvhmLzVCbZa65lasGP9Ntj2IW8rXBukqEyws0qtx6VzEfuUdwJ5W+GavJUJlrrRSrQurRr67SqDvK1wTd7KuoG6c7nW5VUD5LUA5+Y2jMsT7EwtKrQurxogrwU4J++sPMGO1X5FOVxeNUBeC3BvVllUH4yKU2NHaq+8HD4qrxogrwW4J29cNxSWg4WFbXk5/KG8aoC8FuCevHHd8CF/MCpsy+qGiSqvGiCvBTg4GT3uV5gUju2Vd6NVVQ2Q1wIclLekbogVpWk7Oa2rqgbIawEOyltSN5CiXOvKqkGbvHvT/fl8PmNTMquAvAOgRhJWNySKcq0rqwY98u6dHKeTzD4ctdqHCvIOgBpJWIJNFWVaV1YNOuTdC5cujU6jmelRR8i8hb6QdwDUSMISbKpoUevqqkGDvOGyj3G60d5++GAdN1cPkHcA1ElSSLBbRYtaV1cNGuRV6jT75ExOWTuzDMg7AOokKSTYjKIFraurBh3yFgf7WtkLeQdAnSSFBJtRNK91TdWgQV42/2Iy4oMoDMg7AGolySXYrKJ5rWuqBg3y8vr2qHRaRjdxScRNeXMJNqdoTuuaqsGIvHF0J+ww5B0YtZLkEmxO0azWdVWDGXnHfAgQ8g6QekkyCTavaFbro+L0HePyHrEhQAbkHQD1kmQSbKGwzWj9oU4TI/IesiFABuQdAPWSZBJsobDdaj2pfUEbkTc+UbYp1bvjkoir+/OmCbZY2G61rq0aTMpbO98M8g6ABknSBMsUTbWurRogrwW4Km+aYJmiidb1VQPKBgtwdlt/SrBc0URrlpK7keRd8u7FUx7Y8Q7ikoiz8lKCLVGUtGYpuRtJ3iVvvFN79cgJ5B0GTZJQgi1RNNZ6n6XkbiR5l7wn0YnqkRPIOwwaJaH9nbiitOUeT8mdSPIueY9LJ+x0EpdE3JWXdtYrUZS0Zim5E0nePzGnNvFC3iHQKMmsUtFxRUruRJL28rJ22eQYUyJzOPwlgmzBWgJbJlQGJqObx2F5x5WKjlpY0ru8x0qNs09W9K1EWAaUxWF5Z5UyjCtScheStJX3eHKi1HEa3mSOBZgMl797eFSl6KwqJXcgSVt5jw4OZsdKHZ8cTafTebSNGpa+F3BZ3nGloscVKbkDSVrKG7XWJvvbPUfUYh+bjhRwWd69yqb7vGmlmLavbz3cPzk9PR3PZ/UdZO+PSyIuy3uwx7/Ykpg2uqJJ3p2BvAPAVkkgb3dAXs2SQN7ugLyaJYG83QF5NUsCebsD8mqWBPJ2B+TVLAnk7Q7Iq1kSyNsdkFezJJC3OyCvZkkgb3cMWt6+Yb9UcFwSgbyaJWEfYklcEoG8miVhH2JJXBKBvJolYR9iSVwSGay8O3N9jbiEAXmJ69XKSktsjcsGIG/M9Wq1Wl2xw8axNS4rgLwRl0EoSXDJThjG1rjsAPJ6qSP2WWJrXJYAeTOO2GaJrXHZAuT1vLNgtVqGiixXq+CMnTaGrXFZA+T1zpar1fIslCT+L3aBIWyNyx4gb2hGcO6FknjngT2W2BqXRTgvb+TIpRdLElWZdlhia1w24by8PrWGYkkiS3x2kQFsjcsmXJc3fCFHLXmSxLuMXtbGsTUuq3BdXu/yMR59TSTxrh+t6JayNS6bcF7ehFQSy7A1LhuAvATklQfkJSCvPCAvAXnlAXkJyCsPyEtAXnlAXgLyygPyEpBXHpCXgLzygLwE5JUH5CUgrzwgLwF55QF5CcgrD8hLQF55QF4C8soD8hKQVx6Ql4C88oC8BOSVB+QlIK88IC8BeeUBeQnIKw/IS0BeeUBeAvLKA/ISkFcekJeAvPKAvATklQfkJSCvPCAvAXnlAXkJyCsPyEtAXnlAXgLyygPyEpBXHpCXgLzygLwE5JUH5CUgrzwgLwF55QF5CcgrD8hLQF55QF4C8soD8hKQVx6Ql4C88oC8BOSVB+QlIK88IC8BeeUBeQnIKw/IS0BeeUBeAvLKA/ISkFcekJeAvPKAvATklQfkJSCvPCAvAXnlAXkJyCsPyEtAXnlAXgLyygPyEpBXHpCXgLzygLwE5JUH5CUgrzwgLwF55QF5CcgrD8hLQF55QF4C8soD8hKQVx6Ql4C88oC8BOSVB+QlIK88IC8BeeUBeQnIKw/IS0BeeUBeAvLKA/ISkFcekPf86vbbckUsv91enbNLjGBrXAkX7Ih2HJf38v5xxXi8v2QXIq4CF68+O6YbU/Ja8Nx6Z7clhpAnt2fscufjynKxUa837KhmDMlrwXN77gfMjQyBb+g1bWtcOS42Spm314y85p/bs9utIk/Lr1/8iC9fl09bTUxkOVvjKvCiXv2N2hh+fxqR1/xz+zF5Mf9YPtx9ynH3sPyRvKQ/sh90NK4CL+HduzBurxF5TT+3Z/dJbisakniS5Ll7rUnO1riKPMeZ52Kj1uycTkzIa/q5PaceqO8+0yPF/x5fs9RYYdoaV5EgeWtebNQLO6sRA/Kafm4v46ryxwMzI8dD/JIOtHVP2RpXkWBb8Rm2V7+8pp9bcuTv0hdz7iX9t1ZLbI2ryE22tXKxUf+wK7ShXV7Tz23sSFN6yyY5PZbYGleRm9dcS/tiowJ2jS50y2v6uSVHPjMhSvmszRJb4ypy9VroJbrYqL/YVZrQLK/p5/Y8cuSp8dWccBc174PeW0e2xsUIWA/nxcZY2atXXtPP7dlyN0cSS5Y990zZGlcJvIeIH9GFXnlNP7dRP+qPHRz59OkuekPfs49yIi7L0Vw28KeUH+mPj7vUlQlxfdnrmJatcdmOS1Miz6Kx11bt+SwP0Yhsjy9oW+OyHpfkvY36UZkEjUT9qrfs4wYfl/U4JG/Uot+tsIyJysv+Wva2xmU/Dsnrv+nl/Cl5Qfc2AdnWuOzHHXnPwgT3nQnQinA2TNBTdWlrXAJwR96osqyZr1WH32N1aWtcAnBH3rBJ/8Ruf0vCIYFH9pE64lJKKXYwpce4BOCMvJdvriw/JdVlL1MJmuKql7e/uCSgR95g/fOZeFmv0+kM6/Twr3XFHIfO2tL3b2zSx0QN++xwlra46uXtL64CP7d38Od6/XN7Mj3xs+oW9oceeZ9VlufkcOnBLGd+ZzOnwrfzkt381ixz72eNcdXL219cBda5O5hZRbBuuIV9YrW84XyVjiq684a3cxPR+zlJazrjapC3r7iKOCxvyMXza/w3Zubzxkd+BeULiaO5Vtfs8Ju4Cm9yzdu5ibvw568MxNUgb19xlRHQ7cpPR/F/hwd/B1clP9EzOhtsN/Efn5kDGf7dm3Jz43vR2Vvwtr5N38xT2imlN64GefuKq5Qgun8/C6fCo2am9GrtbYjt3U6LDEr+KVLCtQXd3YtvDaVlI2FW++bpj6tJ3p7iKidKsq+FiYCBMrUCXm9XWd7e8P8qW6gd34vwHn9ld37Lv3+U+vMvO7zla2iZ/rga5e0prnL8stp2rZSBksEzsQxoa6/f5G6XexOEpeEXducT/P/icvw/diblyyraKFdvXC3k7SmuCtY89fqmigYDCzBTe4O6HZ+ugo7XuNSPwf6bNJirc280Eqs7rhby9hRXBSWp11zi1T/Clthb6+71qg+qJfmTyPuHncpJoisuVQm7tKe42C0hWOr1a5otfaN/eJjszfWZFejHkRJJuCyV1/js43qMq0rdKnm7h90Twi92OPxmDTh9GJjbQD1m1fUu5GXOWiMvDUqkhULA2m8aMTExp9hjxkHZUAW7VHPZ4F1FgaVNNJOJ18yssni0uNZeSxtsmuNq3WDrPK5qXrKp12jiNSJvQKbU2au5SwpdZa3JpV6jideEvGE/Q9BYOWCQop28WgcpIjKp97mu5dI/JrY4DaVtrnsxPNxKXq3DwxFx6l3HE6t+s9MaMbDFaaxss72YmNNGXp0Tc4i4yeIbT7za5d2OTZC9NaMzmBJp15TIhHgm69p44tUt70sm2cb21n0xBSajWzMZPQelXtOJV7O8L7lxtRb2YhlQg7y6lgHliFPv5tXwlwHpnYy+KUxbvome4MrZ6B4WYFqzADNPsqzL8G492uS9+OtXcVjc8zZxf6+OfwMsfe8QWr9lOPFqkne9/p0Od27LpJtNcux1nVtO3QsNm3vUg01H8gQ2JF5N8mYH6rfDiX5u/L7vxxjbPXXJbwsSrzZ5N+uY17y8r+uE3v8psNFelwR1U1p1gS1OW4AtThlX5hb/bMHm0s1gc2nOi7nFP1uwrX8z2NafYUXixReqNIMvVOFYkXidkhdfZfUOgszunkZXXWZwSl58ieCb+SfXy7k2Ogc9xS158fWtbyVe8k4//Ky0f+F5KW7Jiy/OfivxYGj8wzdK/dL2i+twTN5ovUH7+jKuK3XM1LI1roRo7U+s7M1r7UxAjbgmb2JJq7b9g0ZHbI2LCCcAriNln+tnserEOXnJktXfja/ou6gfVZsjtsZFhEtgXp6fX17Z7tLmcE/exJKmJBenN42O2BoXcUMb+K/t+cpNB+X1zpfxrjDfa+Zy+d/ja/TshWB3XAlXwfOzid37K3FRXu/snjY1enoofUnfPTzRBfdaB7BsjctWnJTX8z4+kgU/lkVP7h6WP+jko7bxK9vjshNH5fXOboN0U7mn5dcvvu9/9v0vX5dP6eHg1kB6szUuK3FVXs8797ealBD4ZuYaWhuXhbgrb5jlHpkbyYvZZHazNS7rcFnesHvqvsSTx3vjX0Vta1x24bi84Wv66vbbMvFj+e32ypLXsq1xWQTkBWKBvEAskBeIBfICsUBeIBbIC8QCeYFYIC8QC+QFYoG8QCyQF4gF8gKxQF4gFsgLxAJ5gVggLxAL5AVigbxALJAXiAXyArFAXiAWyAvEAnmBWCAvEAvkBWKBvEAskBeIBfICmXie939PNcCMWTFqfAAAAABJRU5ErkJggg==\"\n",
        "height=\"100\"></center>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c9Q-7FiKrbME",
        "colab_type": "text"
      },
      "source": [
        "Note that the direction of arrows is reversed. That is because, for example, computing $\\hat{\\mathbf{x}}$ requires us to know $\\hat{\\mathbf{y}}$. This is the mirror situation of the forward pass, whereas computing $\\mathbf y$ required us to know $\\mathbf x$. For this reason, this is called the \"backward pass\". \n",
        "\n",
        "Let's summarize how the backward and forward passes are related:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IrU0n9nDVpRX",
        "colab_type": "text"
      },
      "source": [
        "| Pass | Layer Class Method | Mathematical Notation  | How Chaining Works |\n",
        "| :--- | :--- | :--- | :--- |\n",
        "| forward pass | ```.forward(x)``` | $\\mathbf y = f(\\mathbf x)$ | $\\mathbf y$ becomes the $\\mathbf x$ for next layer |\n",
        "| backward pass |``` .backward(yhat)``` | $\\hat{\\mathbf x} = \\hat{f}(\\hat{\\mathbf y})$ | $\\mathbf{\\hat x}$ becomes the $\\mathbf{\\hat y}$ for previous layer |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jm3Qa8cCVxrs",
        "colab_type": "text"
      },
      "source": [
        "You can see that there is an interesting symmetry here. The function $\\hat f$, which computes the gradients of $f$, is sometimes called the **dual** of $f$.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CYHOY3GQWiQl",
        "colab_type": "text"
      },
      "source": [
        "In practice, the three gradients $\\xhat$, $\\What$, $\\bhat$ that are computed by $\\hat f$ are not treated equally. The gradients $\\What$ and $\\bhat$ will be used to optimize $\\W$ and $\\b$ using gradient descent, whereas the gradient $\\xhat$ will be needed to compute the gradients of the *previous* layer, effectively by becoming the $\\yhat$ for _that_ layer. We can summarize this situation in one table, that shows all the things being computed during the forward and backward passes for our `Linear` layer:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "slb1buXjw3Bs",
        "colab_type": "text"
      },
      "source": [
        "| Array  |  Role |  Gradient | Role of Gradient | Shape (of both) |  \n",
        "| :--- | :---- | :--- | :--- | :--- | \n",
        "| $\\mathbf x$ | input to $f$ | $\\hat{\\mathbf{x}}$ |  output from $\\hat f$ | vector of size $m$ |\n",
        "|  $\\mathbf W$ | parameter of $f$ |  $\\hat{\\mathbf{W}}$ | optimize $\\mathbf W$  | matrix with $n$ rows, $m$ cols | \n",
        "|  $\\mathbf b$ | parameter of $f$ |  $\\hat{\\mathbf{b}}$ | optimize $\\mathbf b$ | vector of size $n$ | \n",
        "|  $\\mathbf y$ | output from $f$ |  $\\hat{\\mathbf{y}}$  | input to $\\hat{f}$ | vector of size $n$ | "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A03hCLQBuhz8",
        "colab_type": "text"
      },
      "source": [
        "**Exercise**: Look carefully at this table and make sure that you see the symmetry between $f$ and $\\hat f$, which reflects the symmetry between the forward and backward passes. Also, note the important fact that every array and its gradient are the same shape -- is it straightforward to see why this *must* be true?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O9oRyVoPF8eF",
        "colab_type": "text"
      },
      "source": [
        "### Implementing the backward pass"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x581zCvLF-iX",
        "colab_type": "text"
      },
      "source": [
        "The forward pass (the $f$) for `Linear` was very simple, being essentially `W.dot(x) + b`. What does the backwards pass (the $\\hat f$) for `Linear` look like? What should it even do? Looking at the table above, we can see it needs to compute the quantities $\\What$, $\\bhat$, and $\\xhat$. However, of these three, only $\\xhat$ will be needed by the previous layer. $\\What$ and $\\bhat$ need to be stored somewhere so that we can use them to optimize the values of $\\W$ and $\\b$. Therefore, we can implement $\\hat f$ as a Python method that returns $\\xhat$, but _stores_ $\\What$ and  $\\bhat$ somewhere.\n",
        "\n",
        "In fact, the layer itself will contain these arrays $\\What$ and $\\bhat$, and the `.backward(y_hat)` method will modify them. Let's see how this might work (note: this code uses 0 as a placeholder, since we haven't figured out how to derive the gradient computations yet!):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AieVxi6gWntZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class Linear2(Linear1): # <- inherit __init__ and forward from Linear1\n",
        "    \n",
        "    def backward(self, x, y_hat):\n",
        "        raise NotImplementedError\n",
        "        self.W_hat = 0 # TODO: compute and store W_hat\n",
        "        self.b_hat = 0 # TODO: compute and store B_hat\n",
        "        return 0       # TODO: compute and return x_hat"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yi2ryhgsuUit",
        "colab_type": "text"
      },
      "source": [
        "Using vector calculus we can compute the mathematical expressions for the vector-Jacobian products that will give us expressions for $\\hat x$, $\\hat W$, and $\\bhat$. To do this is straightforward but requires some time-consuming mathematical derivations, and so we have left the full derivations to an appendix at the end. We encourage you to read these if you are interested, you will learn a lot! \n",
        "\n",
        "Here, we will summarize show a simple derivation for the gradient of the bias term ($\\bhat$), however, because it has a simple result and an intuitive explanation:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_PY-Hz3jyb1u",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "\\begin{align}\n",
        "\\hat{b}_i :=& \\gapp \\dld{b_i} \\\\\n",
        "=& \\sum_j {\\dld{y_j} \\partialfrac{y_j}{b_i} } \\because{vector-Jacobian product} \\\\\n",
        "=& \\sum_j {\\dld{y_j} \\partialfrac{\\left(  b_j  + \\sum_k W_{jk} \\cdot x_k \\right)}{b_i}} \\because{substitute the definition of $y_j$} \\\\\n",
        "=& \\sum_j {\\dld{y_j} \\partialfrac{b_j }{b_i}} \\because {other terms are constant w.r.t. $b_i$}\\\\\n",
        "=& \\gapp \\dld{y_i} \\because{only the term with $i = j$ is non-zero} \\\\\n",
        ":=& \\gapp \\hat{y}_i \n",
        "\\end{align}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "A0gIP_IrzFiK"
      },
      "source": [
        "In other words, the gradient $\\bhat$ for the bias vector $\\mathbf b$ is simply the gradient $\\yhat$ for the output $\\y$ of the linear layer. This is intuitive if you think about it: the biases are  simply added in the expression $\\y = \\W \\x + \\b$ to the other term to give $\\y$, so increasing each component of the bias directly increases the corresponding component of $\\y$ by _exactly the same_ amount.\n",
        "\n",
        "**Exercise**: make sure you are happy with the reasoning given above. If necessary, remind yourself what a gradient represents in terms of \"influence\"."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1tBTyQdbyFnl",
        "colab_type": "text"
      },
      "source": [
        "Here's a summary of the gradients for *all* the parameters of the Linear layer, and the corresponding numpy code that computes them:\n",
        "\n",
        "| Gradient | Mathematical expression | Numpy computation |  \n",
        "| :---: | :--- | :--- |\n",
        "| $\\xhat$ | $W^T \\yhat$ | `W.T.dot(y_hat)` |\n",
        "| $\\What$ | $\\yhat \\otimes \\x$ | `np.outer(y_hat, x)` |\n",
        "| $\\bhat$ | $\\yhat$ | `y_hat` |"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YFul7-GSzBq0",
        "colab_type": "text"
      },
      "source": [
        "We're now ready to implement the `backward` method properly that we started in `Linear2`. Remember that our `backward` method is supposed to *store* the parameter gradients $\\What$ and $\\bhat$, and _return_ the input gradient $\\xhat$."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J8Ej1ap-zP-n",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class Linear3(Linear1): # <- inherit __init__ and forward from Linear1\n",
        "    \n",
        "    def backward(self, x, y_hat):\n",
        "        self.W_hat = np.outer(y_hat, x)\n",
        "        self.b_hat = y_hat\n",
        "        return self.W.T.dot(y_hat)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AXIHaGr4kTjZ",
        "colab_type": "text"
      },
      "source": [
        "### Connecting the forward and backward pass"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CmWgmZ9tz6aT",
        "colab_type": "text"
      },
      "source": [
        "Ok, so we have a definition of the `forward` and `backward` method of our `Linear` class. But how do these fit together, exactly? And how do we use them to train a network? \n",
        "\n",
        "Before we go any further, one thing should be clear from our definition of `backward`: it needs to know both `y_hat` *and* the the original input `x` to the layer in order to compute the three gradients. But if the `forward` and `backward` passes are going to be run at different times, how will the value of `x` used in the forward pass be available to the backward pass? \n",
        "\n",
        "One simple answer is if the `forward` method simply _memorizes_ the value of $x$ so that the `backward` method has access to it. There is a very simple and elegant way of doing this, which is to have the `forward` method return two things: the output $y$, but *also* a **lambda function** that applies the backward pass using the _current_ value of $x$:\n",
        "\n",
        "> *Note*: You can think of a lambda function as a simple function that doesn't have a name, just a definition\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Qsfbsat8z51y",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class Linear4(Linear3): # <- inherit backward method from Linear3\n",
        "    \n",
        "    def forward(self, x):\n",
        "        y = self.W.dot(x) + self.b\n",
        "        dual = lambda y_hat: self.backward(x, y_hat)\n",
        "        return (y, dual)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rrqSrP8Z1iiv",
        "colab_type": "text"
      },
      "source": [
        "This lambda function simply stores the value of `x` inside it, so that it can be passed to `backward` when needed.\n",
        "\n",
        "> *Note*: in software programming, there is a special name for a temporary function that \"owns\" or stores data inside it like this: it is called a *closure*. When the temporary function is no longer needed, the stored data goes away as well, which is what we want (this process of automatically forgetting unneeded data is called *garbage collection*)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CuHLviDLm0j4",
        "colab_type": "text"
      },
      "source": [
        "### The final Linear class: SGD and batching"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aGqN4PHSkeEN",
        "colab_type": "text"
      },
      "source": [
        "To make everything clearer, let's define a final version of the `Linear` class that is self contained and puts all of the definitions together. \n",
        "\n",
        "We are also going to use this opportunity to implement basic SGD optimization, as well. We do this by defining a `descend()` method that adjusts the parameters to reduce the loss. It takes a `step_size` argument that determines how large a step should be taken. \n",
        "\n",
        "Note we are now using `+=` in `backward(..)` to set `W_hat` and `b_hat` so that the gradients can easily be summed across many examples, and we reset these gradients to zero after taking one descent step. *This allows us to gather the gradients for many examples together into a batch, before doing one step of SGD.*"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "D89zxRxW3nDM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "class Linear:\n",
        "  \n",
        "    def __init__(self, num_in, num_out):\n",
        "        self.W = np.random.randn(num_out, num_in)\n",
        "        self.W *= 2 / np.sqrt(num_in + num_out) \n",
        "        # ^ we use a variant of He initialization\n",
        "        self.b = np.zeros(num_out)\n",
        "        self.W_hat = np.zeros_like(self.W)\n",
        "        self.b_hat = np.zeros_like(self.b)\n",
        "    \n",
        "    def forward(self, x):\n",
        "        y = self.W.dot(x) + self.b\n",
        "        dual = lambda y_hat: self.backward(x, y_hat)\n",
        "        return (y, dual)\n",
        "      \n",
        "    def backward(self, x, y_hat):\n",
        "        self.W_hat += np.outer(y_hat, x)\n",
        "        self.b_hat += y_hat\n",
        "        return self.W.T.dot(y_hat)\n",
        "      \n",
        "    def descend(self, step_size):\n",
        "        self.W -= step_size * self.W_hat\n",
        "        self.b -= step_size * self.b_hat\n",
        "        self.W_hat.fill(0) # reset the gradients to zero\n",
        "        self.b_hat.fill(0)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5_hqoiUO9pW2",
        "colab_type": "text"
      },
      "source": [
        "**Exercise**: what would happen if we did NOT set $\\What$ and $\\bhat$ to zero after doing one step of SGD?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vY-R5oIG1iew",
        "colab_type": "text"
      },
      "source": [
        "We can now use this final definition of the linear layer to compute the output $\\y$ *and* the three gradients $\\xhat$, $\\bhat$ and $\\What$. Here is some code that calculates the gradients for a single Linear layer on a particular input, and prints them:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mOhvsoYv9tlc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# make a linear layer that maps 2-vectors to 2-vectors\n",
        "layer = Linear(2, 2)\n",
        "\n",
        "# feed it the input vector [2, -3]\n",
        "x = np.array([2, -3])\n",
        "y, dual = layer.forward(x)\n",
        "\n",
        "# print the output\n",
        "print(\"output = \", y)\n",
        "\n",
        "# compute the gradients for y_hat = [1, 1]\n",
        "y_hat = np.array([1, 1])\n",
        "x_hat = dual(y_hat)\n",
        "\n",
        "# print all the gradients\n",
        "print(\"x_hat =\", x_hat)\n",
        "print(\"b_hat =\", layer.b_hat)\n",
        "print(\"W_hat =\", layer.W_hat)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9_qIF0YHFSRo",
        "colab_type": "text"
      },
      "source": [
        "Remember the interpretation of these gradients: they represent the influence that $\\x$, $\\b$ and $\\W$ have on the output of the net. In this case, we *imposed* a value for $\\yhat$ to \"jumpstart\" our calculation, but normally this would depend (via backpropagation) on the way that we define the loss $\\L$ of the network. We will go into more later on."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Cs6JkewmAayn",
        "colab_type": "text"
      },
      "source": [
        "## Implementing a non-linearity"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FV2iuyr2Aelx",
        "colab_type": "text"
      },
      "source": [
        "Our next to ask is to implement a non-linearity such as **relu** (which is short for rectified linear unit). Relu is applied to each scalar element of a vector individually (called *element-wise*), returning either 0 if the element is negative, or the element itself if it is positive. Therefore the derivative of **relu** is **step**, which is a function that is $0$ for $x < 0$ and $1$ for $x > 1$."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bevt9LtvD0oH",
        "colab_type": "text"
      },
      "source": [
        "<center><img src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAyAAAAEsCAMAAAAM8ycIAAAAMFBMVEUAAAD////BLDHekJPW1Na+v8Csra4BkUZWnniMzaq+48/P6tvz9vT5+fnr6+s/Pz85mHckAAAACXBIWXMAAAsSAAALEgHS3X78AAAS2UlEQVR4nO2di5qjIAyFvSBi6+j7v+18gPZi1VJNJOj555udbneXbUmPIRzSZjkAYBEIBIAVIBAAVoBAAFgBAgFgBQgEgBUgEABWgEAAWAECAWAFCASAFSAQAFaAQABYAQIBYAUIBIAVIBBmOt0bfepneBY6XZSfkYJAeNF9Wek+U2d+judA92YuUhAIK1XWu1+z9sRP8hQ8ItW9P5sUBdKl82rTbtrzLKviP5YInCBSCQrkr9J/Ah7GEve6ru/jn1V9aV8n2TXXWMIj9UbVm9lILQtEV7rSEm9pras/sY+vqS2TqfRXJy4QqX23ViM1JxD/zzqLvFtKO/4IH59SdM/yVn8KpMz6ycKWCkSK4tZqpJYFQgjlArzy0045pKJb/9zrD4FUfc9WgFwuUveAv7SZ2UgdIhDCAYfLEum80wlk1Mczjm1vGMsPRIqOhUgllkGU1pXSWlWU866oFkCf+lBZb7T94vEKhUfqRh4pPoEsRSqtDGKz9p8VyB/lvFNN+6c+8j4b4KnS5UbK5g8bJJmRmmEpUnMCIa8oqSaoc7siViC5m3yiYVua3fqnPp7Pt6sGeCIrOVKaI1LUu8bfI3WID0IUx7ubdT/tpPNOwkv+SPfsFVGkOtGReuF7pFIyClWl23yc9rzVmmnzdBOv66tr2uYvKO0idRsiVYmK1CvfI5WUk+73rQeB5B+72DF5qz9wetfHppIYqTfOlUEGlMCM/V6fXz6DDEiM1DvbMojwCyD5tO/fG5nsXx12wUSkdvI9UuJ9kMpUkzHop33vC3pmf/cYhEfqJl4g35Hug+heuwPIZfZ49Ym7LkXTh9BIPa435NtXZxXI9utS1+cqs+eQ+6d/I00g8fRxvUgdP8fCM4iuhitSJlYgEfUhLFJGeKQ2Id9Jd5cllT1DRz7tu/zZOX0ctoslPFLkNQijk76EeCe9ctNuMsYifQ+z+eOaTrqPVP8SKcEW+sAJfBBf+ZmXxylJIPPrq2v6ID5ShdBIzXMCJ720065ez1gKmvaF+uOaTrqRHKkFTpBBWv92ReZ5j5xpX6rPr5lB2l7bSJXPe8hrEHLO4KR3ptTm9d2KxOyNLO5fXdRJb43RL35VCj5I+k66KlSet29dLFIEMuoj4ns/iYqUOSBS8EGm2IVt27+914QQgdzi60NUpEqxkdqF+AzSa/0+60LOYknQh7RIVZNI4SxWIHsGVKafzIqI65IIfQiLVDmN1FnPYontdPbQO+m/v8rX9XFRJ/0DWWce5jiBk/6JgG3eL/nj8j3pA/BBohB/2r+tr9BR6DmrDyKc6AL5Wn+gJ92Ds1hRiC2Q7/U5MogHPelRiLyLFbB/hZ50D3rSAxG+N/KTDyJjf3dAeKTggwRCO2DU65IofUiPFHrSA5GeQcKHk6WP60UKZ7ECiCgQYfpApPiBk/6DPxuqDzjpHvSkB3ISfzY4f8BJ98AHiUIsgYSvr+CDeNCTHoVI0/5D/QEn3YOzWFGIM+2/1OfIIB70pEchyt7IT/tXcNI96EkPJH0fRNr+7gB8EHbgg4QIRKg+ECl+kEECzmJJ1QfOYvGDDPJ92sXqA2ex+IGT/rUn/Xd9wEn3oCc9kKSd9A35A066Bz5IFI6d9i3rK/ggHvSkR+FQgWyqP+Cke3AWKwpHCmRbfY4M4kFPehQO3MXauH8FJ92DnvRAUvVB5O7vDsAHYQc+yPK0i9cHfBB+kEEWp12+PnAWix9kkCWBJKAPRIofOOkL/uwefcBJ96AnPZAE/dld+QNOugc+SBSOEMi+9RV8EA960qNwwLTvrD/gpHtwFisK/NO+tz5HBvGgJz0K7Hsjgz7M5v0rOOke9KQHkpYP0gz6IP0/eIAPwg58kKlAEtIHfBB+kEEmZ7FS0gfOYvGDDPI+7SYlfeAsFj9w0t960mnyB5x0D3rSA0nGSSdaX8FJ98AHiQLftFPVH/BBPOhJjwKbQMjqDzjpHpzFigKXQOj2r5BBPOhJjwLTLhbh/i6cdA960gNJwQdJyv8YgQ/CDnwQP+1p+R8j8EHYQQZx086aPzRbTYKzWLTMRAoZxAqEUR+V7rOCYVwHzmIRYiPVfwwHJz3PW2p9vD5fbfqMrZZGT/pOvkcKTjpD/fE+zXPXJalcuyc9NIMIh1og9Our9+uw5ssgwknNB5mLFATCUH9Mr0tsNYhwigR4fSkVoRmkU2rokVCqGwojOfcprSu6xzfqg/L5HpdBJMbneV8KAim/RGp2F0tZhict75YXCNHjG+sP2mf5Np2MNYjwSAl4/X/nS6Qu7oMc4Z8zZhDhkZKuDcfL4w3OIKSzJHl3/ZDzJdf1QQS8/L+ypQa5TAZ5rz940H2fZT1TDkkhg1AOyGoUzkfqyhlkUn/wUHl4Bk8hg1AOGCFSF3bSH+urL5+T/jvoSfeQCwQ96SEQCYSx/kBPuodcIOSgo3ARzvocHYUe+QJBT/oSrP0f6En3IINEgUIgvPu7yCCes2aQ83c6v+sjQqczEbIjpeTvYqEnfZZJ/lj8nHTxyI6UE0gZ8BfDR0RPegC7BTKtPyJMOxGyI5VABvnOBTPIR/0BgTyQnkHw+SAB7BTIZ30OgfAMeNoMcmonfWb/KoI/S4TsSNFnEDjpIewSyCHvfwUn3UGfQciBDzLhmPdPhA/ioM8g5MBJf+eg9xeFk+5ABonDdoEc9f67yCCO02aQszrpQ/3RfAwIJ50FOOmhiNjFapf0gW3eJ/BBplzGB1nWBwTCNCCc9FAEZJAVfeAs1hOcxZpykQyypg9kEKYB4aSHEj2DrOoDPelP4KRPuYSTvq4PeuCkO+CDxOFngRytD/ggHjjpcfhVIIfrA066BxkkDj8K5Hh9IIN44KTH4TeBBOgDTjoPpfxtXjjpIfkDPsgD0kglIJDvnNwHCVpfwQfhGRACCSVaBgmrPyCQB9IzCM5iBRAukMD6HALhGfC0GeQsTnro/hV60h8IzyBw0kMIFUiE/d0BOOkOeoGQc2UfJJ4+4IN4EhDIhZ30iPqAk+5BBolDkEBi6gMZxHPaDHICJ/0nfcBJ5wFOeiiH72L9lj+wzfsAPsiUU/ogP66vIBCeAeGkh3JwBvm1/sBZrAc4izXlhBnk5/ocGYRnQDjpoRyaQX7fv0JP+gM46VNO56RH3d8dgJPugA8Sh1WBSNAHfBAPnPQ4rAlEhD7gpHuQQeKwIhAZ+kAG8cBJj8OyQDbqA046D04gpHMLJz2ERYFszR/wQR6QRqqQL5DvnMgH2by+gg/CMyCMwlAOySDb6w8I5IH0DIKzWAHMC2RHfQ6B8AyYQA3ynbM46Xv2r9CT/kB4BoGTHsKcQITs7w7ASXfAB4nDjEBk6QM+iIc+g5BzESddmD7gpHvoaxByrpFBBn0YKfpABvGcNoMk5qSP+WPzFiCcdB7Qkx4K6y7Wfa8+sM37BD7IlOR9kHu9Vx8QCNOAp/VBUsogQ/4wey4tOIv1AGexpiSeQQjyBzII14DoSQ+FLYPsrz9y9KS/Aid9StJOOkn+YABOugM+SBweAqGoP1iAD+KAkx6HUSBS8wec9AGcxYrDIBCa+oMFZBAHnPQ4eIEQ5g846Tw4gZCODCc9BCcQyvqDzAfpjr5cJuCDUA5IJ5DwSCXqg5DWH1TT3v3wAe40yI6UXIH8EKk0M4gmrT8Ip11/PNW21IZtJXS5DEJUcM5HSs9FKskMYmj3r8gSd1t9zLvO+sp+E/0PE5BBNhIeqRSddEPsf9D5s39+3p/Pt8p6+x9kPU/dnoCTTjlgjEgl6KQ3Yv2PYd7/ntfhLHO3TWZiPqpwSCNFLxBCQiMVLpD7LYwmEBNGvYREfQzz/rguqSxz57y0uzwdRafCKAMpwr7miRWIr/yp10JkMVJzAmkWX5SC0IGvgsPRWutxJVBl2dtPapZelYKQGqbQSM0E7paCPoyWTNUeIhAlXx+F6DgFRGomcMurGjnI1oeuhrX8PcvcvgvTEiuBBCJeIEOkuqVIpZlBaNdXVUU5mp31x2ZL70u/nqdITyGDUE7tIZGaVE1zqX8phdiS+ft3Y2wF/vY9d18TWPN/1uLfPyf919cZ4XB+c+TxW39B6oYKkJylFBJYV38yd19p6/mA789ZPEOkzvg56T8PSLe5OZn1PNd9X+k+YzqCIjxSN8ECCY3UCT8n/fcByYb7+zzC0Om+TOeoCe2AlexITQbrdB941CTtz0n/HbKe9HaYdfSke+gjReWkt1VopE73OekR6carEnrSPWIj1YZH6myfkx6TrtJ+SwEdhR7yGoSKTmt/Kbjc56THZVwioyfdQ16DkDHuyyCDRAEZxHOGSJ3qc9K3DoiedB7OECn4IHt8EK11nlcmWsaADxLKEKnfJww+yPZpb3tTmazsq2jdHvBBwuj6siq2RQoZZPu098o3oqlDuz1euVykNjYB7YgUMsh2gRh3hEdHLMoRqTDK7ZGCk77Pny2ymdmCk+4hr0H2RMpsixSc9H30CW4DrkAbKVE+yMZIwQfZjmrz1lV908cDH8RzhkhBIJuxb4BhbH9N209OO8JJ90iJVJmZvLT1+YZIQSCb0VmlTWGbCKYPBxnEI+UsVpXZN+PcFik46dv3RlTvzCfzcVoeTrpHjA/iI1Waj8DASQ8aEB/iOXJSH2QH8EEgELYBzxApZBB8TvoL1+lJDwUZBBmEbUDBPenBwEnH56S/cJWe9BE46VGAk+45Q6TggzAAH8Qjtif9AZz0KMBJ98jtSR9BBokCMogHPelRQE/6A0RqJ3DSgwaEDzICH2QKfBD4IGwDntUHwVmsVMBZLHaQQSAQtgHPehYL/uxO4KR7RPWkzwInPQpw0j3wQaIAHyQVzuqDCAcZJBVwFisKyCCpcNazWPBndwIn3SPfB4GTHjQgtnlH4INMgQ8CgbANiJ70QHAWiwmcxWIHGQQZhG1A9KQHgp50JnDmYSdw0qMAJ90DHyQK8EFSAT3pUUAGSQWcxYoCMkgqoCc9CnDSHyBSO4GTHjQgfJAR+CBT4IPAB2EbED3pgeAsFhM4i8UOMggEwjYgetIDgT/LBHrSdwInPQpw0j3wQaIAHyQV0JMeBWSQVMBZrCggg6QCetKjACf9AXrSdwInPWhAbPOOwAeZAh8EAmEbED3pgeAsFhM4i8UOMggyCNuA6EkPBD3pTODMw07gpEcBTroHPkgU4IOkAnrSo4AMkgo4ixWFJDNId0XRnCFScNIP8GdVZXqWKxEitZPvkUrKB2nd8xmnvaWqKNl9kMroLA2B0EbqRh4p7oT0GamUfJBO6/YpkJbs8nTE7noiAiEaUGndPWuQpCOVUgZRWld/o0D+tA8CxbgQyAhLpCq6SB1wFivhDDLMu5t2qw+q1zWbQMre4l5018ogyUXKLEYqLSe9svNup/2PcgeRzZ/Vbtrd4+QRiPBI3U4QqcScdDvvViD2J9mg5Mxch3kEQg5tpCofKcFbvd8jlZoPUukByRvsM9pNRCCEnCRSycVtmHXRpzlmr0vUJyLFc45IJSeQP3dlqqgXo6RMXxPWfcoyk+yp+m0kGanCRuo16c0KxClf6q2/4cpE9/ha6uf76c86GHIIIsUSqZe7ZwXSWcTe+qsq0Y/vSIRHSqcfqRRrx8NfhmAjZGdM4nG5zRUAfgECAWAFCASAFSAQAFaAQABYAQIBYAUIBIAV0hTI/XZrmuPfyBhcjzQF0pi6rm8CHgg4O4kuse4QiGRUme77HU9ItQaBQCRTFEV5kqcCgQB6SgjkKFpbjt/yW+Pk0NrfuFPjTiCuVs9z+2sj/Hlcgq4sy67MO2UziHJvN9apslR53ipVlmWuyjK144vSBdLVdV0bW5M3eW7q+mbq+j4KpLF/mLsfRsBjvThdWZSlKorc6qMoykLlbVEU9netKouRxFKL/CWWFUHTWAk0LmvUdd2OS6zGCSRvIBABlEWh8lzZX0YdWG04vbg/LZS7nZZC5AvE2OTR1HXTeTX4ZOIFcoNA5ODWVXnuvr0M/A8vjeddRVLPKhGB5LYOcamkgUCE4ldRqn3uYtl7yvIhkCIf1ZIQSQjEjDfqm+M+EYiBQASghiLjmUFcKaLsVzfepSAQanwG8XmifrzvATKIPFqfQ8q3DDI+yuE2BELOQyD3erjVTjMIBCKBoh2yyGsN4tXwzCCoQch5CMStpO553lppeIF07kdrvE5AVMYlVPkQiN3lLVu/szUoA7tYxLTO62jc0qqtnSlSN3nnrA+vk8EmMXirk8i4Pd2yKLqxHCkfZcmwl1Wo9Cx26QK5N46h9mgaY0yb5zd/r9WPMY3z12+Xe2tPaZTKOoX+TddUWbrFVVva+9phbVUVzlZPCjRMgUNI9XgWBAIOIbnqfAACAUfQFqnt7w5AIOAAuqK0XwkusiAQAFaAQABYAQIBYAUIBIAVIBAAVoBAAFgBAgFgBQgEgBUgEABWgEAAWCLP83+tmJqibdsbSgAAAABJRU5ErkJggg==\"\n",
        "height=\"150\"></center>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ioWxiXxTBIyW",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "\n",
        "Since Relu has no parameters, we only need to compute the value of $\\xhat$ from $\\yhat$ (see the appendix for our derivation of this code):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0hxlrvrtAq7q",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class Relu:\n",
        "    def forward(self, x):\n",
        "        positive_mask = x > 0 \n",
        "        y = x * positive_mask\n",
        "        dual = lambda y_hat: y_hat * positive_mask\n",
        "        return (y, dual)\n",
        "      \n",
        "    def descend(self, step_size):\n",
        "        pass # no parameters to optimize"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-hcTiwvh1ZoZ",
        "colab_type": "text"
      },
      "source": [
        "## How functions chain\n",
        "\n",
        "So far, we've defined just a single layer, and how it computes the parameter gradients $\\What$, $\\bhat$ as well as the input gradient $\\xhat$. \n",
        "\n",
        "But a single layer can't do much on its own. How can we \"compose\" several of these layers together into a deeper net? \n",
        "\n",
        "The answer is that we can define a new class (called `Chain`) that represents a **composition of layers**. It will also have a `forward` method that computes the output $\\y$ of the whole chain, as well as returning a dual function that computes the gradients when given a $\\yhat$.\n",
        "\n",
        "If you have a list of layers `functions = [f1, f2, ..]`, then it is easy to apply them in sequence using a simple for loop:\n",
        "\n",
        "```\n",
        "for f in functions:\n",
        "    x = f(x)\n",
        "```\n",
        "\n",
        "(After this for loop has executed, the variable `x` will hold the output)\n",
        "\n",
        "We will do a similar thing for the `forward` method of `Chain`, except we will *also* compose the dual functions. Remember, the dual functions are not *run* during the forward pass, they are merely composed (in reverse order):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nBdPOu1c5MiK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# return a function that applies f and then g\n",
        "def compose(g, f):\n",
        "    return lambda x: g(f(x))\n",
        "\n",
        "class Chain:\n",
        "    def __init__(self, *layers):\n",
        "        # create a chain using Chain(layer1, layer2, ...)\n",
        "        self.layers = layers\n",
        "\n",
        "    def forward(self, x):\n",
        "        dual = lambda y_hat: y_hat # <- empty chain acts like identity\n",
        "        for layer in self.layers: \n",
        "            # apply each layer to previous input\n",
        "            x, d = layer.forward(x)\n",
        "            # compose dual function of layer with previous dual functions\n",
        "            dual = compose(dual, d) \n",
        "        # return final output, and composition of all duals\n",
        "        return x, dual\n",
        "      \n",
        "    def descend(self, step_size):\n",
        "        for layer in self.layers:\n",
        "            layer.descend(step_size)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jf4EO5pZq8GP",
        "colab_type": "text"
      },
      "source": [
        "Let's try out our new `Chain` layer. We start by making a new chain that contains a linear layer, a non-linearity, and then another linear layer. The input to the whole chain is a vector of size 2, the output is a also a vector of size 2:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TcW3GuXu0I4i",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "chain = Chain(Linear(2, 3), Relu(), Linear(3, 2))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ouIx_wpR1aW6",
        "colab_type": "text"
      },
      "source": [
        "This network corresponds to the following multi-layer perceptron (MLP). Remember that each \"layer\" like a Linear layer corresponds to a pair of neighboring stacks of \"units\" or \"neurons\". The lines represent all the individual components of the weight matrix."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UJC18lX7GDzm",
        "colab_type": "text"
      },
      "source": [
        "<center><img src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAfQAAAH0CAMAAAD8CC+4AAAAYFBMVEUAAAD///9mLZFtOpV7VaCJbqmXhraJhZWloMEnJyjy8vOPj5C1v8ugoaL6+/yyub6qra/e4eOOmZ+9zNS7yM/U3uPS1tju8vT2+PlDRER2d3fu7+/q6+v4+PjBwcGZmZmGXYFvAAAACXBIWXMAABYlAAAWJQFJUiTwAAAgAElEQVR4nO2dDXfqOq6GQ1soHOBAC6d7CLvw///lrHw7eSV/JJIJJM+su+5MNyDLbywrjiMni5nJMYs+QWbRJ8gs+gSZRZ8gs+gTZBZ9gsyiT5BZ9Akyiz5BZtEnyCz6BJlFnyAg+v/+/Pnv75D//PnzvzF04+wH70dL9H///pxE+O+/f6H1EZn9sPthiP6//4RM5HzB9RWL2Q8K049a9H+/iE8O4jGjZPaDo/GjEv3Pf8xHh/CAQTL7YaHyoxRdw8TpdPoLzigz+2Gl9KMQ/a/tk0P4A+6oMvvhoPAjUbURubdmP/z8yET/H/yTIBHnw9kPTz8SvQmkBHxSY/bD049ksfgDfxclWhI0++HrR7L4F/4sTKTb3NkPbz8S5WASLwea/fD2I1lIrvXRgF8qzH54+5GopooFUYbI7Ie/H4l6NImUAs1++PsRQ/Qf8Ow5O+tl/EjgTwqAZwrMfvgziz5BP2bRJ+jHLPoE/ZhFn6Afs+gT9GMWfYJ+zKJP0I+Jir5er1fL9XoH/2AFjD7cj35MUPT18uOt5mO1hg+wgNGH+tGfqYm+W76/dVn66g5GH+fHIKYl+m4Jiucs/eI8GH2UHwOZlOiM5Bkr+DABGH2QH0OZkOi7D5Da4MMjxoPRh/gxnOmIvgad27y7BzsYfYQfAkxGdJfmPiEejD7ADwmmIvoKJCZwqQ5G4/shwkRE9xjnGY55HYxG90OGaYi+A3lp3u2qg9HYfggxDdGtebvJB3x1VH4IMQnRvSb0Auu0DkYj+yHFFETf4coryzt8ezx+iDEF0QMG+tvbEr4+Gj/EmILoAQP97e3dsgwPRuP6IcYERA8a6NbbNjAa1Q85JiC6d+peYEngwWhUP+R4fdF979E9hjoYjemHIK8vemB0n0UXAjyL2VmWh+g0fHwHozH9EOT1RQ+c0mfRhQDPYnYWiOoEfqICjMb0Q5BZdAR+YhR+CDKLjsBPjMIPQV5e9OA7trc3dk0OjEb0Q5J5pCPwE6PwQ5BZdAR+YhR+CDKLjsBPjMIPQV5f9KBnbG/zfboU4FnMzpoXZ4B5GRbgd0yB0Zh+CPLiou+O909Q1cHyfrzBDz3YD1leWfTzYX+/3++hk3r2nf0Bfm0WPQzwLEZnFYqHi/5Rfu2IuoPRGH4o8KKin4+V5Pd74AP1z/qL9+Pu0X7o8Iqim4oHD/X31lf3h/Pj/FDj9UQ/HO9t9ntQ1sJ63/m6MdzBqKYfs+i+nbU7fncUX5+DbtWXecbf/Y1dbD90eSXRzwdmlPq/4lLueodocczDPBjV8WMW3buzdl3FjfnY801lY2GGuIAOs+ghgGfi/L3tWcVP/styrXeauvng/f4T4UCPGHq8gui/N4zGnRacvQL8+7nzNQzzP79gXhboPAWeX/SvGyje1e50Pvrct73f99B0DPOHL2iCJNACBZ5c9L9dxfdHUPx0Ou3v960zhc/u0FF1Kpu/KYZ5MK/AM4v++wPRl97eVnwsBZlbFMuvR/jyKW6YB9sKPK/oXzAAMawXVB+0rce+V6uv1Fgnw/xNJ8yDZQWeVHTM1smwnnOoRufpzCbxq92p+kF8zlJyPnYuM5UwD2YVeEbRMazffvjOqjTfn/Oy36B3VQ/47FT9tPg6dAzLh3kwqsDziY7Zet7xYLTkXH2qigSrTkb3saoSgUp1OjEo/PiFWUU4mwejCjyZ6Jit3/6xdtauXIs3d0Wc16vlR85yaR7uQH2W8OMfmFokwzwYVeCZRCfCejPKwGhBpQs7ek2qqLBn8oPaGBNtJACjCjyP6BBX2x0NRk9+83SLnV11+wUoc/4uGFXgSUTHbP3nH4/OCtTcyPrgX9CPf7qrv3uJtXkwqsAziI5r68Q9Mhg1NKdXXEgOtq+AUYUwD0YVGL/onh0LRptFGVJADtuXwKhCNg9GFRi56Hy27u4se6hmqUQkpgQwmiObzYNRBcYsOpEs8aMIjPbU3Jb8gdEKwTAPRhUYr+gQN+0d2bXZWogL4swu0oDRhqAL1EYMPUYqOmbrXFhnRK83TwVrbvkuGG2B2XyfMA+tUWCMohOjxn0P3LZYLbPgaPWBUx2MdhFYm1eUoWZ8ovecH1sG+QjtB5MPgFFkcDavqUPFyET3z9atncXnYp7QqoNRkmHZvKYOFWMSnQjr/qPEtGe56/KFvF0HoxwDsnlVIUrGIzrExbD50DBHChYK9SNglIe4gP3W5mPoMRLRMVvvrq37d5Z1JdUfIlyAUSuYzfuszStLkTMG0b3W1r07q/8NehtikQaMuugR5qEdCjxedKHVrMqW65l4AHALAEbdBGfz6mI8XvT+2TrTWdUT8b43aybwW2DUi7BsHlqhwCNFJ5Id+yiwUZqq+lVAc4waFvN2AqIZNEKBx4kOcY/vCG/RiXl4EJ38AIz6432Bx9DjQaJjtt43rLc6S1jz7p0AGA0Cs3kqzEMTFHiE6MRVP3x/2Ym5tx5IS3UwGorH2nwMPeKLrrSTVEfz9o+C0XCc2Tw0QIHIostl6yg6vV4+GGORBoz2wp7Nx9AjUPTder1crdf80WUkpT9EWKeTmV6iK2mevdteqQ5G+2KJdmDfSj89AkRfLY0X+z+WAYZybyCuyb4H5tiyPoDmOS0Y7Q8xAH4CRe+vh6/ouxWUcnhf+t4MU9l66Nq6g7/MvgcJ6j0VV3sbAsFsPlubj6GHn+g78mXPtze/y+sssbZu5xeWTCWpc27pV1SJMO910Q7Tw090y9v8fHH0ihhVHH4rGyqaGzdu8i2HWc9jMXGQHn6ir63VWmyHzJP1WmTDegHxGFSWWnUwPRzI5tl688P18BV9DZNHxwobUqKV7Lhpa97cuN3AuAQQ5tm3pYfoUeMU3RJKKmgroLhacZ6qx0QXZbrUK7xgXgTM5qEW3kA9DFyiexVLRytQbVEnrOd8xdC8UV3mjWQCyOapYlk99WjjEN2zqGrbCigu9e42yU91WSncrLW8UledWJvvVszqpQdgF923fLKZPUC2vj9olta8VqNCWXNjrBPPxsT4xVnRLJDSQw8Cu+jWPNGkOsuMrreu2El/6ysLGi9OveinqTp1x1MXxgnWg8YqesDpJysyWy+vUvBMjGpRZv8XGq9Atey3VwxdeashWhb1bgP1YLGJ7l8bPwsolnrr4JmY5lXnWOrICdLkjHqql60lBtAhTA+L2zbRg445WnKKa4reaB5H9EZ1aIoUjTXIh4MOELUNdYvoIRdWeYZdc1V2OkuHKte9RXoObdwr6CzSdP1ohfnPMD0sQ90iuvcZGAVpe/7pdJYGN1MAaL4Ci65RbdFbYT7w/FDLULeIHnhyYXGgGVmYFzwToRp0R6qz1ESvVVe6XQejdTYfepIk/FANL/oOfsYBvYR00hK91vyX6Sx5ckN18hix9HcW5rf23kdoLU5W0QNPq3x7W0JY1xS9vkH/tXSWMIWlWnWV23WmxedD8OnQfHznRWee0/Ms4ScURa9vmaueB6NqoqNtScBohaAevOhBNwhv1klEvmt+YUkUjCrQveLEd9LY/BDUgxcdfsQJ/ISa6MS8CkYVRddcpAGjFYJ6PKfoxqKMs7MEaay17xwkYVsL3T0x0am7JjCqKrre7ToYrYDuVhA9+I7Nco8Ang2D7HAwqoBpj2yEouiSerCij3ekf5HdDUaVRa/XgIUXacBoBXR3fz2eT/TOooyzswRpGfwlEgsB2NZCd09I9L+05nxnaYleqy57uw5GK6C7NUQPXOp9e3uHn6A7axD1Qly3p8GoAh2TbFtURBfUgxc9eDGAXwECz3rzyy6GgVF90RfXPR11VEQX1IMXPXztHX6C66z+muOijLOzBAGjCos0bGujLMMGPk5/e/vkHrJhZ/XFkjuB0RiiM3cS8qKfj/vgBy78VlFe9NDMIX+eTp9sDJ71xHZvDEYVAKMKO2mIVpf7pqDH7fBTuk30wHjyUYU6HO7gmbzmVGeJA0aZ1cEhQJvrLVOO89+78NHdJnpgfK8OIDf3aYuKbl/vhuZHEl1a9bZF8z2CwPjOR3eb6GH54sfdpL1PDjzrA7MoQ3eWDmB0YU8uB4re3QYddNPGP1i1ix401Fe4T1tU9L+O2yNovQJgNEd2J03danyPIEgP2+tsNtFDZvUPYp/2/bATE73ercLVfYHGRxPd3bYQCkvQl3mGHKYHj1V01/vvBoW88DpOkc0P7wrcKdMFGq8AGO2qzkShQNFB8SpqBuvBYBXdf4GmDib4Os5+x3aWNx7zJrQ9ouiSizQ4cIz8KFwPErvovrlc6/YA3rq8U4Vvg/B4jAlNVwCM1tjvLPzBiprt9wj66IE4RPezknZXZPBqHVR6xOeuCFoeVXT7GoInRAkSiNJeAd46oXuIfvaw8v6JVSAwzPcvMuTVodByBcCowWGo6lhbDFc3z3uf+7YPuFQCRT+dnWP9g6n3AvkIWd/coze8uhPMRxZ92E4arKhJvR6W18Jwqu7U3KeOnONGoViVoVQnpvceYd6xKFMBthUAoyb9d9IQYZ1eTis+5hiFjvncU3TrncJ73U74Wg6GecdpRV18n1mD5dii991J419Drro0sCysoYc9by/wKhPKlSJ9e1ueqUPrOt8eEua9d6eAWQXAaM+2Gl8BxddUWM9paptb9ZASnZM9Lzt8dqp+wqvZN8zzO2W6gFEFwGhXwrCdNBjWb1+8H6169hY9PPCu975bdeaSj1Vpwn1uNe2gx9wXME+C0QeIHrRIA9l6MRDAaEm3nj2vh5uQkx12u9XyI2e5MqcO7oz5TmdBRUT32aMBuREYVQCMAr47aTBbr+rfM62m+pjTQ1R0liryfDOqN1d3WJgPufcFowqAUcRnJ431bFm61e5oGoLMwT2O81NMhyGq8dl80CoXGFUAjBI4G+248MlW13mTTD17odOa7Kq3vYb65kw2H7aeDUYVAKMU1jVjzNa7hZLJVgsfMih1RJf1TDTw3XG1tzWHfyEBowpQdgH+iSCRzMJnSD+kzzAQO5etuhipsQ6eeXRA6DNqMKoAGCVhdtLAvEZc6LQf1gHVB7nD+CwhCDzLwWzeCPPBdV3AqAJglIZoO2brXP17bHX7vFcJ5ES3nGgMnlVAffPq6nfvlBmx6N0oRUQ1csKn/eic7CyB5LGb7G0FeNZAZ/P8vMgCrVGAsw2YizSBt6ndVldnuJNP3XoiKfqu3KMNTwzAsxZENt/jgRW0RgEwylJnocFny/r26RBED9itrspuJALPusBoCNZ8ZKIvaIeIbN3hR9WjED2HIHuqMnPgKXiG4LxnWd+ggLYoQJhlQdXtYZ0U3ZInDUH4KG16kQY8I8Fs3rk2z3WWEmCUBbN137NlWy3X0Vz8/HTy/gI844Aw77/TBlqiABilsa6th/ixpzpTAPFD86mVBPCM5xfCIr82z3bWQ0WHC/d+C9ki5uhKEcRFp9YMwTOev90e49fm2c5SA4wCsLZ+D90/1bSdnirHKTqRfIBnLPVOmeAwD+1QAIy2wWS0R02autV6miuITjz7Bc84mkUZ7EDHvAjNUACMmsAq0/Hnt8frTlWrFRbiahREx10e4BlHa1EGs3lbmIdWKABGazBbLxZhwmvScH0oiYbo0GLwjAE2ILBr8wg0Ip7oRFSql5WCa9IUlqplLlzSlkBF9O58BJ7RgObs2jwBtEEBtEpl6+0L07qngqDQHGbIJxC9ozq6RsHtlCHW5okwD01QAIxito5r64Gq563GXFgUJdHb95jgGYVtp4xjND1IdCKsE1Eo8Inhib7rFUVL9Jbq4BmB40UBooPbQwcaoEDLIMw7eCF2VCfiE6K5KFOhJrp5uYJniEfdFszmzbV5sK9AYw2zdcvaelBNGmYlWxQ90Y1FGvAM8NwpYwnzYN/Ker1eLdfrwPhZmgpeWw/Z76d6g16iJ7qxkwY8A8295z02mwfrLOvlh/kyUMC7Ia4LjyVgkYbbkyCJpujV03XH8A19oZ/O5sE6zW6J7/n6vvd38srWSbg7E6B+9qBzs1agKXqzk8ZxgQdX2iRGGxinoF/29JX9jMmk9yNTag2CoH72oKm5ruhNTmJVvU+RHszmmeoNJozkb9aDSWtfwh8CmXi9l/erfbNWoCt6ozr419C3Zjpk80QxJpOdtW7Hh3VuJ+rneO6EqfGZwuJori36yX2B2xZlXMDaPFGRqcJVWfWdHeyguG3fOovHu/ZVyNO7WSvQFr2+XedUH3gOCmbzzHD3qKZLqy5WE89ZkyaW5vqin+wXuMCJR5jNd+vN+1a0RtXNeus9w7q3r1XIU9g10UFf9LPtHty/powVzOY7Yf4AApO053WsjNW/AGKhui2qxdM8gujN0hwKK3eK4S8IZGbzvgeVmgW5sN76gdIqCMsiTR0G2KREjgii192Ha8+CJ1dS+VZVb97/jIqqqir8Vp4hgtHeqkOGU6/URtA8iuinyqN75wIfXE/VIDcESVeRzQccMbeiFK+iBhgNh9lJ0zx7gM5TIIroTFgTPY26NIXz8H63w5VXlvczXDhNfgBGe0A6bTx7gM57WtHJtWfqb4NFpwrSBp1t1T6AqF2YF4wOUd2c0oxpDjpPgUiiExf4kEUZgpbB9mgFYW28m9/s3POj1R4QTxTN6wA6T4FYotfz96F0VPLckwV2lhHmA0+mXdaDHFb3wOgw1au7mdaAgM5TIJronbVnoi7LMMBonY8Fiv6OYb3lx3A6vrczejCqQDzRW/fk4TVlXIDRUzW9g6wOPtkndo4meNOKcp1pDowqEE90M6wR85qK6FmYtzxPpfmkFZc8Hdq4m7l2UhswqkBE0Y21Z8FFmQowWiF47jgY7U09vGE9HowqEFP0Oqzd5TXnOytwSreddQRG+9N5WNCkNmBUgaii12HNzFy0RQdRncBPgB/iqjeXPxhVIK7oLdVFNec7a6SiLw6k5rwfgkQWvZ7MxBZlnJ0Fmo5E9N8jefmDUQVii16HNalFGVdn+T5VNcAtGIQfw6lvWw8+fkgSW/S/VEjTFH20I70Jeq0FKjCqQGTRf438Xe4e3dpZoOlIRG8mutZMB0YViCv6b+tBiNhqnLWzQNNxiN4qo2WoDkYViCt6+6Gn2Lq7tbMCHqYXRLlP7yxZNKkcGFUgquj14yThJ2zWzgpekYshej3NwSNnMKpATNEbB+mdNIMAoxXBouNG6I4fApo3zx66eyrAqAIRRTd3ysjumuE7a3f8/gRVHSypffOGHwKYzx46qoNRBeKJ3n6ECGFNQfRyI0XopH4n9s0bfgjQ0rnzxBGMKhBN9O5Gf2nVwWi9bz1Q9GqTHPFMHYz2o+N6eycNGFUgluhYdyWoEEGo6OYu5oAN0G/FHoqKww78EAD2vrf6BjpPgUiiEztl5N5u6Yre3bce9HC1tTGys08OjPaB2BBq7qSBzlMgjujkThnnW5z9RMd963tQ1sIa982Lik6+z2bczUDnPa3odCiHXSNDKCzBnvd8g2PAUF8Sv3E/3ko/BDSnfW7uZqDzFIgiOrW/fzH83fQW5Nst5b51/1dc3osvQLQoTsgCo8Gwb+nWyR103pOKztclEFykwZowxnzsncvVCzPEBeRTD88BOc0VVMHQWgVFiAii215ThEy2J1jBsb1v3XNZrrUnEt9jpAoRB2HLXeu3QaADxdEX3V5hinmLM2z8eFSa8prW37vfgum9b+mRAuvaRKTKUqcYotdnhTK9xc333kAVCvKESntpqQLqSQuGeabe/FDNjbsZZhFYDm3R67NC2cg4SHWs4Lim1k+9VP/gvglZXb8w73zeUGf21FUribboXLbaYEluHBBh3TZIHNkc/44Dkc33CPPEogyoHqkcha7odbkZ2zCGtzj9gFpi+JZph7Xlzu3dkTVjmA+sJUcuynSp8lpl1XVF99GcWpd3dyFk6y7Fc9gkfmULESXno63evANPH+t7WLAuiaro9fYQcI5R3TYKGsh662CcZE3K/uGZOmHS6BvmiWcPNPUkAMYF0RS9PpaCdM8kZJGG6XiwznBedTK6D59RXkBWqPTJ5gPyFvWzPHRFb05sAs8A3500mK1XFRzBPM9uvfooWC65TTIkhSWsUOnM5ulnDyTqp/aoim6c0kU518FxD5uD2bpRwRHsK1AbY6INR8hdKXVWrTRqoptHkYBnBLbF2qKj7bdN0ABN0R0XYIegTULqJ/Epit46eRM8o7CGQMzWu4V5oQUKtC1CvXkmzIc9XyBOLRVHSfT21QqeUfA7aYhRBZ95zAmMUG+eCPOBpdNOEVTXEb0TocAzEmYnjef8CW1QAIx6ZPOhewbyViuenX5SE72Ti4BnNMSuEj5bJztLGTCaY83mCZ/sFB7oqq4ieveuw+qlQWepkgjrbLL02PPT2WjE7pRh6XShyu26hujQYM5BwFykgbhJh/VRiE5eoD+2PIWF7URBFETHs0I5B5E604WQaTnLdAyik9n8T7jmjR+KizTyohNnhYJnPD93CktYh85SBIx2gTBfXsLwQZ669YqLNOKiU2eFsh4SYLfZw/qoRKey+cCtYIY5tdMYpUWvz1c1WwqesfQ9y3Q0ovdbm6f9IPtSAmnRyasTPKMhkiH/3TTQEgXAKAeEef+dNmarqag5PtGZeQg8o8C4eAvZkgRtUQCM8vzCNOV5AbdaTeRHEsiKTmvu0Vmwtn4Pfd0J2vJY0btlcL3DfNsi3glJICo6d28JnrXBsN6jJg00RgEwyvJLaO4X5jutVlFdUnROc3tnUfNfj9edoDUKgFGOZqcMZvOOnTZcp0reuAmKzq8Xg2c1XLYeXpMGjCoARjlaizJh2Ty0WkF1OdF5zbnOwrDeJDvB1UnAqAJglAF2ylDRjP4q+nGWV11M9PpsTSLRBM8WVLbe7ohQ1cGoAmCUhmo6cYHTa7PYavmdNFKiW5/7g2eYrePaunUnDQJGFQCjJNxOGWJtngjzRKutfdsHIdHtV2PHL3LfOhD4hAqMKgBGKWw7ZTzCPNXqen+OkOoyotfzDqn5KdTx8tqgd9IwgFEFaMttHK8vObN5stWWfKkPMqI7co3GJS5bJwnadQJGFQCjiEeb7dk83WpZ1QNF3613q9V613nbz6F51VlEMkOEdYOQ/WVgVAEwCnjulLFEO6bVzBoIrYeLANFXS+Otz49lY4hpUEPhqD1bJwlYpAGjjxDd//UldgCA0ZIqOjTdzurhxFf03Qre831flqWYXJqfqGzdEtYNuEwYAaMKgNEuQbknZvNZmGdb3X6uYdFDSnSujEN2ebmfBJ3xqvbeP+RdkwasKgBGO1A36FaIffN8Lxqq2/TwwEt0Sw2HVV1ThmvtwCoOsLrFAHbji96nxjFm83w1jXqvgk0P+BaBh+hr5rIqg0pRQJepk7I7fvcK60RXOlQH0wqA0RZ9a9hjNs+8SlvW7/mEwN7SgxaihVt0W9GOnE/YKVNClOzoUZnJMzkC67FFty3KuMBsniyskUfVpUMPVyEVH9EtoaRiSd6s7bqK963B5leTBuwrAEYNHIsyTich78FaePmuOWvYLXCGeJfoXgU28cRxqLaIa+v+eNVrgZbHFb1H3ZwukM1DvfnT6ZBC5xO4xrpD9DX+IkXbCijeK6xTPWoZRdB0BcBojXdNGTsQ5rv1k3rpAdhF9y2f/G5ckpitHyxaeXaGe76EtscUvX8tPPglyIPMdKmPHgR20b0LpVcVNrG64tGzEoUdd2YMbVcAjFYIHk1C9GFdbz5cDxqr6AGnn6zIbP0Y8n66Fec9MDQ+ouiD69t2/YBoWdwRB+rBYhP9DD/F874DxZv5CDzrgasmDbReATDa1pxrWrjo5AA6hOlhcd8metAxR0sirLtGSBCOEAqtjya6VM36rh+QDwcdQGQb6hbR/Q/ByGlflR6dFYhjJw00XwEwuhi4KEPQlsAM8/aVuC7v1OqOW3TP24OKeqjjiiy6NkR1+lk1ND+S6JLn0BB+GGE+8PxQy1C3iB54cmFxoBlZbx0864d1VwoYVQCMOtokILox3ENPkoQf8hB9Bz/jgH9CBJ71Vd0yqsBoFNHDa8q4AKOnMpsPPiiY1uJkFT0wur+9LcmHBHRn9cSySANGFQCjjjxDTPQszAeLzsd3XnSySLYN/mgE8Gyw6pgpg9EYostrzvshqAcvetANwpt1EgHP+sPupAGjCnRtOuvZSoouqAcvOvyIE/gJBdHZlRAwqi8615RBgNEKQT2eTnRuzROMKtC22HenjB221dDdUxKdeaIFRrVFF16UcfoB3a0genDybrlHAM8GQe+kAaMKmPYCz50ZLLqkHqzoox3pRn+bqoNRXdHJNkgARiugu/vr8Yyik6MMjCrQWBPaKUPAthq6e2KiU4s0YFRTdCavkACMVkB3a4geuNRLnEmMnSUGZs5gVIHamMKijNMPQT140YMXA/gVIPBsOHCPDEYVRWfuGnVFF9SDFz1oC4XdCHimoDoYVYAxLQrb6uC1d14PXvTge4RPYp+2nujdnTRgVE10fv1fAjCacz7ug0Xn90HzoodmDvnzdPqcW5Xe6TzhAqMK5IaUFmUqiFaX+6agx+3wU7pN9MDHOh9lZxCvtYFnIrR30oBRJdFtz/RVRK93xnq93NLAR3eb6IHx/fNeU+/TNjpLgdauFWi+Aho7Zeyim5vkAuM7jj0f0cPyxfRu0t4nB54JYe5Pg9ariC6/U8YiencbdNBNG/9g1S560FBf4T5to7OUMBZpoPUKaC7KgOiw8X0fpAefxtlFD5nVP4h92vWWOfBMjCaThsZriK64KNMWHfoyf48gTA8eq+gBO98LeeE1rCKbB8/kqHfSQOMVUL1Bb0QHxauoGawHg1V0/wWaOpgQr+PsNEWvV0pgs708xwiaL7j32HrqQWIX3TegtG4PcLjb6psPJprqdek0RWf+3ro1etrvEfTRA3GI7pfBwwQCV2vf0iMe1MmVsuq15nqeYAkSiNL99A13DPQAABEPSURBVOjgEt1nGvmAplFh3r92XGhf2asRC7FTXpTBiprk6mZfPVq4q0s5ry3OBuQjZH1zAeqTkYhOktZ8QE0ZG1hRk1J8mB4NHnXkHPOIbf6IFOb1VT+r7ZQh699TlaVqBuhR4lMx0lZJzlW1DMO847SiXlS36zpnzBv17DXa7ldDzmCIHjleZUJ37MXlU4N2B2FePpv/0VVdbVEG69+z5VbNHh2kh3cVaNqMb9lh4mqWDvOVAb4U9QCUNMds/fblubI4TA/veu+7VSeD+Fh5z6Ccg5JgPXQx6nr2og2GbL0YCDH0CDnZYbdeLj9yliuq9gBL4SVURBQN89w5r8Np6tmD0d5gtl4VSo6hh/ih+RTN1Y31zaXCPHei82CMevZgtB9E1Gsmjhh6xBWdqm8ulM1zZ7cPxaxnD0b74MhvYugRW3SyvrlEmG/qoROVjvrT+k0wGgxm69369zH0eIDoYWePBohej0r+ZJFgWtEDjIZBhHWMctACBR4jOtkBw26IckM7YdU7eQIYDQHmNfpChzYo8CjRyWx+yNp8YUn4jPlObghGvcFsnat/D21Q4IGiy4b50pSo6vvOj4FRP3BtnQjr0xFdMJuvbDlPBvQHfgqM+hB4YStLkfNo0cWy+dqY2J4KPN8UjDoJOlt2SqLLhPnamNQiDWoeKjqRrLqjGLRDgXGILrA231izn+XuC3WwJBi1AfOW34WsLEXOWEQns/mQMG+Y2wnsqSB/A4yyYLbuewghtESBEYk+bG3etEcqFgQdLcAoDRG1/NcgYugxLtEHZPMtg8R8HAStuZ8fA/OTCHKMTvTe2Xzb4kDVmVwQjALh2brdDx1GKHq/0dIxCffYIXB3fWC0DRHWw9cc1MUYrehkB9rnxa7NAaqzXwWjJjAv9VtdhNYoMFbRyWzetjYPRrnh6oRfyQWjNZitc2vrLsCoAiMWPSzMg9G+izTUDbrdj6C19Vl0N97ZPBjtuZPG9kwejC7k9waAUQXGLrp3Ng9Gm6frIUtz1u+AUYFs3cMPcZ5AdL/RBEYdo5bBGh06Jolkk45CIYBRBZ5DdI8OBqMn+/xM4sgDWgZh3sELcRZ9MJjNm2EejObwmTiJXXPDD8zWh4Z1ux+yPJHo9rV5MFrAqL5er1fL9boTxNkb9LYfRNSxryGEAEYVeC7RLdk8GC3B2/X18sN8GWgNH+U0L/xQfi8PjCrwbKKz2TwYrWirvlvie77LQnf3er1Gtt4FjCrwhKLTo41N1c6G6vTLnsXrnm7NzxjWI57sIMhzik7Nq2z1huY5KSN5xqquL8JdPbt916BKVQ2wq8Czik5l81y9+VKuFQZ2g7QsaExrTtTPEQ7rFWBagScWnQrzdH2ePHK7iii/5yWNqcsGFNcI6xVgXYHnFp3K5qnFtINX4ewlqXnMmniz6L5ANg/15k+ng2U6b/iAtICofqkU1iug8xR4AdHpMN9ugWfV7HbtEqyMpRjWK6DzFHgN0ReLXxDIzOZ3IC+NWZALfnB/0AzrFdB5CryK6Asq36pLq3qfUVFVVYXf2otVonAAnTeLzpMbwik4z+YDjphbUYpXUYM1Lu2HMq8lOjUPH3cBtfHf3jFbb/IDMKrnhyqvJjo13IPOtvroKL7u+y7bYD/0eEHR4d4ahLXxvqWSggf5ocRrit4K84En09bny+HqHhjV90OFVxXdyMcCRX9nFJ9FDwM8i9VZWZjfgqwOPtkndmA0lh/CvLboWZgPPDL27e2TGuSP9kOUVxc9+Jxg27E3YDSmH4K8vuiBU/osuhDgWczOAlGdwE9UgNGYfggyi47AT4zCD0Fm0RH4iVH4IcjLi+77VNWA2D7zeD8kmUc6Aj8xCj8EmUVH4CdG4Ycgs+gI/MQo/BDk9UUPeJheMN+niwCexeyseXEGmJdhgRX8xCj8EOT1Rffc/dzAn+EIRmP6IUiI6OklSTbpd3bzm26S5JLCJxjAs6idFTipv8MP1IDRqH50yUTYXIxFhe+Uf+O2hb/o+0zmNE2Sy+k7SdLv9JJs4EM04FnUzgqM75bzx8FoVD9Aj83lM0mSuoTCOkk81fQW/TPZHIv/n1w2m+x97kuSeI518CxqZwWuyfHRfVSiH5NMhHwMlqTioq+TTbmbJEmS5FDaeArRw/J3Pncfl+ibXI+LIfrG+O92fEXfbKowUmm9SZLEcw4Bz+J2VlAqxy68j0v0tBahEnrnPwg9Rf+uf/u7+u1PbxsP76yAoW6Z0Ucl+ibZlWJ8l3/59B+EvqJXwT3/bdtooADPIneW/ysu71bXwGhkPwyKW6fsdqr6m/nfHQTfpwf8dg14FruzvAM8vzBzGuF9+mHTa0oPFz3gt2vAs+id5XnbZg3u4xP9u8ypT2FTerDo2W9v4a8OwLP4neU1rVvWZXLAaHw/WmyMqJsa07uLUNFbywG+gGfxO2vnofoHt9+9AozG96OFObj979LDRb8Yv/35LNn7yUv1D2sSNxY/DMzcPWja9RU9LVfazd9+kmXYCkdpAsd8Ph4/ar6NARgypfuKfikvqmx9txI99b60wLPHdJYth3+35+2j8qPCFD2L7pYF5Daeom9w7fV7433rBp49qLPObBK/8lp6AKMP8qPk2CyZZP/V/07ad4m+GODHpA7v35uEfrWTADx7WGetSdmXnqtNYPRhfhRsqhG4TTRET5PN9+mYJptdmiTp4fR9yR/yeAKePbKzVp2M7sNvlJ9GKPouU/14+kyTS8iU7p3IZbdqxe9+X/LLyt/E6DrrvF4tP3KWy+7hDlbA6GP9OJ12l00uxmfIXXrILdvhu17O//b//Rzw7NGd1RMwOgI/drkYIXfpU9gj97J+HNK0ClNBN2yz6M/rx85YMsmeggXMU7Poz+pH2iyZhM3os+jP60eWT3/m/y1NsnurAGbRn9WPY6n57hKq+Sz68/qRZi8hpAH7U2tm0Z/Xj+/sLn3j/85JTfIf/EkcuTMpLcx++PsRQ/T/wLPn7KyX8SP5A38ThzjiXp7ZD38/kn/hb+L8C54pMPvh70eiH0+iRMUIcfF1/EgW/4O/CvMH/FJh9sPbj0T90oo0QGY//P1I1C+t/4FXSsx++PqRZOcXwj8IEikoLmY//P3IRNcMKNGC4uyHvx+56P/+wD9K2Yhym1Mx++HnR1JYgX+V4SdqX81+ePqRlGZUIkrUmDj74e1HJfriL3xkMBFzn4bZD5baj1r0xb+yZn6+IofE2Q8rph+JaUYwpvx9UFfNftC0/Ejadv739+vnv6H/+e9PtIUMjtkPmx8J2Jl5eWbRJ8gs+gSZRZ8gs+gTZBZ9gsyiT5BZ9Akyiz5BZET/ud3gbw/m63b7mrq4HDKiX5LkAH98LJskGd2FOBZkRE+TZA9/fCybJJlHOoOM6NskucAfH8pXkgxw7StN7/DHB/OZplINkBE96+Ir/PWRZOXuBphPkg387bH8I9gkoez9MqyPxbkOjO7J6CaHo2AwFRI9G+pRXt/2ZOBAz76/hT8+lKEemUjdp29HpXpWemfQdHMcW3y/SsYescWZrPbJSDL462X4FbgZ2f2I5EAXXJHbZ3XNxjARbjcCDbmNa766iSYZgsuwX9kAuzx4ReSaSb4RmI/TMal+lA08omvvt0z2zWX/KOFvn3lR5FTk7jFTfSTJXNYUyXUD4QcuP2ne7cnlsk23+5uNoGF0tfzQ7XbbbtO0qIKdpHupBYPjCAJXxj6LXaLtkH/K9rMthfdjc7mkW9Klr2OaXi4hP7VJj5JLRHlCuNk/NE/5yqYrodhVo/Ro9ZqPvmz82akvj0tnyvpqrhzrD1zymJJujyrP+b7S4kCD/UNm99s+75+NeHr8+OfpX4dt4ZtxS/KTK35Jjw9/dne9V7HmsrlcPtM05f9ve7sdbjfHoLxmn7mxv1H8X2arHg2OH+zBSDZR5LnAphpPqZKzPfnZppeQGSu5pFsiNPykachkleVFShFmNDtnrvXqzlU8cZEhH8e3Q2pnUwmbtoLy16XKOy7Wr6fpLbejmkiMaLvUT3lnnI7uOW0ot22eCzTpV54bPPBetsOY9sj95A9ox/dsvg/XuzFfZWsr3Uz1kYxqY+RPkqT/vITmC/MZ1D678YN/fiDj2g2bJpv72LZj9OenWDC/JclmXA/nR7YFepNcRrfbrj8/WdS6bgY+5pVnZKLfx7dPaQjHJLltx7ctd2wvO2xeaKDn81W6GddGssUIRU9fZkbP+doM2pWrxPxaky4jfCNgFl2brzHmKLPoyqQj21W7mEWfJrPoE2QWfYLMok+QWfQJMos+QWbRJ8gs+gSZRZ8gs+gTZBZ9gsyiT5BZ9Akyiz5BZtEnyCz6BJlFnyBaol+Pwvt+e78uMGyj5av40UKrKEEiXEJy37P+y3Uz6AXYV/GjjYToP/hmbV6vBT7Yn33fvv8JeL/kVfxwIiE6URj2S7azbsav7e0VpIrKEU2BoJv/OzOR/SgaytXauBfVOa7GV0dVEPhC1N66pVtOlnCM18Gu1Tv/9DuBt7piRL3b/Ei0jiamH7cLNLT1SXTT3w8nSiNdlrR5HeySvfabvR9GD8BDkly2t3urTujFt9hiTD+ygbu95VGLmqsvSXK53/Lal+bfhF5+1RrpXdpHqny1y8j9c8PZ1Jhgr8brYNdLboqtj7vfVk2qv/Llq2VMP4yGUjavRdHLVgVzbz+cKIz0Km6Vf7slya2IVpcqtG3Mfy9mr2Yazj5ffqSc7/bEaEiJv7X/OTX/B3yA4kF+2JRsuenrhxOFkV51Vvm3W5JsW923r8snLcqLuTV9ZZ1VfqT5fbDpEP1izpVHz2OFHuHHhZnUS1rvOfv64QSbEU5nhBQ1mMwRkl3+x0vpffa/t9c8k8nrnabJ5f5zzcqypNXn03xm3lY/Ss3fG1vBiuy3ja9c7aOpJr4fnYYiLTd9/XCiNaen5gjJ1xWu5aFZ1dlZ103h71cxMOqTdm6Q0hK+Hom/LWp77UqEmUnuw4/042bGEZqOm1Kvumtl760Rsqk+dstLcmzqr9Uf/7odq7N/bt2euOHPs3lcLXqrtzf24fQgP4qG2qo7X42CigF+OIky0i/Vx27V9Z0Yk911W92U3lqfr7jBz19t5c+vtzQrP3kxov/FW/SYflxveelkfqHt2p3wPf1wEmWkF00t4uG221nHIhynG66zDkny2frDdUMYtDXp4reYFduPhb1S4nXT/TdPP5w8YqRf6jrt5Ry4b/6Z6KzuXOiheZ56N/9Dak6X9YNoqMkVr8GnndOvnYBW1+eozmvFReb2VIaam/e61U+bpwP2zN51/SAa2rpnx3H+ZNm7OUKMrv0qP3hclPUVmRHSimo/kPFuk2Yt81A+gNyaA6jnfbqqH8emodUfTT/y4uddhcd2n55V4t9cLs3DLXYuzOPgZnvLzlu5lq5u81VmtrP2hrPF3XJuq0ps02a95JrX3S5KbDfzZ+o5FUb041rM//mZM2ljqv4fh4Rsi9BzNinRzbx1YR0hZQX/6tP/lP/rkrKdZa5ZmxXTm/XO+iv1KpkxTELW3qP5sd9AQ00/CDdHtvaebsprctNclfvNpbwD/blcCk/Samzmx2glm7Ig+jUroH85Lm7lNV1/3jDQPDMzbF2qv2UnBFRfKX/bPHDH++lUTD/KLyepEbANP9ILtmVcT9nU6SZNgC2txdv8yfvxHLthMVS2uFvOzbttRlTJbSR+PMkWaOvesqNlgGR7y9iOjM84/HiWfe8p3x8XW9C8bqyPYKMzCj+e5mUHPodJrXPdlf/iQ+CbE8+PF3jD5YcdHs9FPD/m15omyCz6BJlFnyCz6BNkFn1qLBaL/wPRMWYH4EdwAAAAAABJRU5ErkJggg==\"\n",
        "height=\"250\"></center>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aC33OfEqG47W",
        "colab_type": "text"
      },
      "source": [
        "Let's evaluate the Chain on a single example:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LBn-KOP10d_o",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "x = np.array([1, -1])\n",
        "y, dual = chain.forward(x)\n",
        "print(y)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Sfpb9YLXHGFD",
        "colab_type": "text"
      },
      "source": [
        "In addition to the output value `y = f(x)`, we also have now the dual function stored in `dual`. Remember, this dual function (which we wrote before as $\\hat{f}$) maps a tangent\\* vector $\\yhat$ at $\\y$ to a tangent vector $\\xhat$ at $\\x$, in other words, it gives us $\\xhat = \\hat{f}(\\yhat)$. \n",
        "\n",
        "The tangent vector $\\xhat$ tells us that if we were to start at $\\x$ and move a small distance $\\epsilon$ along $\\xhat$, the output $\\y$ would move a small distance $\\epsilon$ along $\\yhat$. In other words, the dual function $\\hat{f}$ tells us which direction to move in the input space if we want to move a certain direction in the output space. This is exactly what we need for optimization!\n",
        "\n",
        "> <small>\\*By tangent vector $\\xhat$ at $\\x$, we mean a vector $\\xhat$ that lives in the \"tangent space\" at $\\x$. You can think of the tangent space of a vector being the space of _possible velocities_ of that vector, if you think of that vector being able to change in time. Seen from this perspective, backprop allows us to easily compute the (linear) relationship between the velocities of inputs and the velocities of outputs of a function.</small>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Cw2iXiSi__h6",
        "colab_type": "text"
      },
      "source": [
        "<center><img src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAA+gAAAGQCAMAAAAKk7pTAAAAYFBMVEUAAAD////LTVjdj5a/Hi7ux8v24+X68fKpqKr6+vvu7u+9vsDd3t/Nzs/m7/U9hrNso8WLttCgw9m61ORVlbvy9/rT5OzR3NiuzL+XwK0HZzk6gl9nj3x+qpXv9fLy9/REBP9NAAAACXBIWXMAAEnRAABJ0QEF/KuVAAAgAElEQVR4nO2d2XrqOhJGgyEJHWYDJmDY5/3fsj8PKg0lybMt4n9ddJ8dCKEoLSSVZPljAQD480B0AGYARAdgBkB0AGYARAdgBkB0AGYARAdgBkB0AGYARAdgBkB0AGYARAdgBkB0AGYARAdgBkB0AGYARAdgBkB0AGYARAdgBkB0AGYARAdgBkB0AGYARAdgBkB0AGYARAdgBkB0AGYARAdgBkB0AGYARAdgBkB0AGYARAdgBkB0AGYARAdgBkB0AGYARAdgBnQX/WcDemTHPuC3ZIc20Sc/ndtAH6LvupK1784vwX7U+CXYT5q/ie5x/BnRWWjNP4rOr/Bn2lUYom/Zzxqy7R7IrrMgfyWOMEA+BGHEAdEJiN4nyIcAokuQEAKiC9CuCIguQcMKC+RDANElSAgB0QVoVwREl6BhhQXyIYDoEiSEgOgCtCsCokvQsMIC+RBAdAkSQkB0AdoVAdElaFhhgXwIILoECSEgugDtioDokl4T8vtKH+zxGkB0AqILILokoIT8u72ej4x/7AnVQHQCogsguiSQhNyynlzwy55QDUQnILoAoksCSMi/36eUPOPOnlINRCcgugCiSyZOSHJ/6ZJnvNjTqoHoBEQXQHTJhAnZmj254MmeWuPFILoAogsgumSihOzvT+a3hD29GohOQHQBRJdMkJDbPbX35ESLsjtEJyC6AKJLRk7IzduTC27s9yqB6AREF0B0ybgJcVqu9/EtqnEQnYDoAoguGTchL2Z4Jvnzvr51rcZBdAKiCyC6ZNyE3Jjkr99/WRz6pD1lv1gJRCcgugCiS0ZOCJc848fs6dnvVQLRCYgugOiSkRMieu70flbfhFmFb152h+gERBdAdMnICXnlPblZVZcdevlfzXe7Q3QCogsgumTkhGzvFoflzP3+W/4/e07lC0N0AUQXQHRJCBe1KEW4W9tqHEQnILoAoksCSAgN3NNEVOuar69BdAKiCyC6ZPqE/Gr7ZNqW3SE6AdEFEF0yeUL+pXLgLvfONY4LohMQXQDRJZMnRFbcb8o/G+92h+gERBdAdMnUCVEq7uq/G+92h+gERBdAdMnUCTG3vUL0zkB0AUSXTJwQY+BO5jdeX4PoBEQXQHTJtAmRA3fqwtN2ZXeITkB0AUSXTJuQ1Bi4y7J7093uEJ2A6AKILpk0IXLgvjN/Ztkp6wWiExBdANElUyZEDtyfMo6Wu90hOgHRBRBdMmVC5MBdieOn3SZYiE5AdAFEl0yYEDlw/1XjaFd2h+gERBdAdMl0CVEG7loc7cruEJ2A6AKILpkuIXLg/k+Lo90mWIhOQHQBRJdMlpC7slXGJnrDsjtEJyC6AKJLpkqIPG3iZcTRbhMsRCcgugCiS6ZKiDJwN+LYtiq7Q3QCogsgumSihMiB+y+Lo1XZHaITEF0A0SXTJGSr73HX42hVdofoBEQXQHTJNAmR92DL97TrcTzblN0hOgHRBRBdMklC9IG7GccrO/r9dWt2WQtEJyC6AKJLpkiIrLiXFTc9jn8tbo8O0SUQXQDRJVMkJNUH7n+nYYUBRBdAdMkECfk1Bu4QvV8gugCiS8ZPiHpjlhIZx+v+H/uFWkB0AqILILpk/IQ8zYG7EsctTbPbtbQAohMQXQDRJaMnhA/c1TjuaZo2v5UqRFeB6AKILhk7IazivtDjyPr05vdShegKEF0A0SVjJ4RV3BdGHL9pmjY+1R2iK0B0AUSXjJwQOXBXu20tjt9Wo3eITkB0AUSXjJsQS8V9weJ4pWmKe6+1B6ILILpk3IRYKu4LFkeSpmnadJUNohMQXQDRJaMmxLyjosCI49limg7RCYgugOiSMRNirbgveBy/Lbp0iE5AdAFEl4yZEHm+s/G2jTiyJbam9TiITkB0wR8SfffTke2m6yv87Gq+CVlxfxmPbPU4ctGf7Pe99BIH+4Dfkrr5cLN9q3blpp84OreBPkR/I9ZyCb3iTb/SR/pI2Y8H56+I/k6tInwC6dG3Hdltur7CtuabkBX3m/nQTn+JbH0tTdkLeOklDvYBvyW7HhrFG7UrD73EgTm6oN6Q11VxX/A4cs9xOGRbMEcXoBgnGSsh/yy3Qif0OO4QvROBCLL5ZD9sBkQn3kh0ZeDOHjPOdS88xw0c2hKIIKvVavn1B+KA6II6CdnJijt7zIijmKE33QQL0YkwBFmuMqLPb/ZQXSA68T6i+wbu/Oq1bHWNPckPRCfCEOS5KonaDuEhOvE2osvzna3nxyhx/Cs8T5ueBAvRiTAEiVZE1G4ID9GJdxH95h24a3GUnvPKfAUQnQhCkN1KI1o2H8JDdOJdRPcP3NU4dq0qcRBdJQhBPlcmy6ZDeIhOvInoco+7o8JGcfxXdujNw4LoRBCCREz0xq5DdOI9RK8auCtxvFoO3CG6QhCCMMlFZa7+dB2iE+8hutzjzh4qEXHc0jZ7ZXIgOhGCIHzkrrjOnm0HohNvIfq9auBOcYiBu/N5HiA6EYIgS+a3qnpUq1uH6MQ7iF49cKc42g/cIbpCCIJoU/SIa19nxQ2iE+8guqy4u5fGizjKgfuz1U2ZIDoRgCBfutXf30ub6xUrbhCdeAPRZcXdc2RMEcez/cAdoisEIIih9TJzf8kL8f4qPEQnwhd9m1YP3Ms4ftsuoRd/CKILwhN9VfTdX5HpOkSvR/iiW2/MwsjiSMpKHO6m2pXpBfkWPbbaped86qp7J+oQnQhe9BoV90UZR6eBO0RXmF6QcnFt+S20juR0/EtZeYvYb6pAdCJ00ZMaFfdFEce908AdoitML8hS9OO8S8/4FD/2jtwhuiR00etU3BdFHEWH/tyzx2oC0YnpBaGBuRjDi1k6kbse+cvuEJ0IXHQ5cPdU3Bd5HL8dltBzIDoxuSCfcmBu79IzvpeRf+QO0SVhiy4r7lWnSPxs7q1Om1D/GEQXhCJ65rZtll4XiE6ELXq9ivtC6dHbVuIgusrkgqgldXeXXglEJ4IW/bfuwF326PaltX91/IfoxNSCfKsldecsvRqIToQsuvOOihZ+Njv3VWu3irW5AohOTC3Ipyp6hy4dohMhi07nO1dU3BdFHC+X6Oe0zpgAokumFiTS1s7az9IhOhGw6A0G7nIdnT1AlfvKF4HoxNSClGKLl2jdpUN0IlzRvTdmYfxstv/Zi3HZNTHPV+WOG4iuMLEg4so1kQ85S/dueOVAdCJc0eWNWeqc25zFcbfti0sLxbPhwcv/QhCdmFgQsS3ux/hB1Y5XBkQnghW90cC9jOPJTnPPhgVFL3+rnOtDdGJiQco5+ZryQbP0hl06RCdCFV1W3GsM3Lf39Jk+0/vd3DHzT7FbOu8AohPTCiJG7ko+Ws7SIToRquhygl75/v4rz49K2UUtN/0Oy3e/6RCdmFaQ0upIyUfLWTpEJwIVXQ7c2aSb/fnC74fF9NRYP795xwcQnQhC9KWaj3azdIhOhCl6g4r7fynjJbbH/Xsak/J/vnocRCemFUQsrqn5aDdLh+hEmKLLY+Ksd1RUeQrN7/npkL8HW+m9FhCdmFQQsS1Oz0erWTpEJ4IUXZ7vXHnR6V1oTnHcXhC9K5MKIkbuej5azdIhOhGi6A0q7uUEPf1V40jsV7ZUvhZEF0wqSDlI/zLy0WaWDtGJEEWngXt1xZ1u2PA293l/C6YU5EvobOSjzSwdohMBCtKg4l6eKnN4n9s/vwchit5mlg7RifAEaVFxT6xx/Pu9P5/pXW6s295f2a4aV+EdohMTb5jJTnT+ZPloMUuH6ER4olfeCl3ylMfEsTi2FzkwyNVOnvq/GRCdCOC4Z0s+ms/SIToRnOgNKu5nZYOMGUf2dZG+7rdXNj7I9sP95hex3e4v5553iE6EKUjzWTpEJ4ITvfnAPe/49Tjyfe3lmD0fIdzuNEL4dU3+IToRqCCNZ+kQnQhN9AYD97t6vrMWR+a5HA/kfTp14y/nlwhEJ0IVpGmXDtGJwESvcyt08VTtTmtaHKl2betL/eL4Kf7BXg2iq4QqyGfDWTpEJwITveXAXY/jrn9N/KpfHC+IXoNgBWnYpUN0IizRGwzcX/qlakoc/4xDaV7qC77cwwWITgQjyNdSPw+y4SwdohNBid6g4l7ufX2K3a5KHHfj119qH57f+8V+ejREJ0IRZBlFxl0Um62lQ3QiKNEbHAdZLqHTVFyJIzVOmUu1Pvz2ejrOpoLoRBiCrKOIid5slg7RiZBElwN3h4qSX/OMCbVHN260WG+IANEVghDkM4qiJeu4G83SIToRkOhy4G4fWivoFfeFN45bvevaIbpCCIIsV1FkmYk3mqVDdCIg0b3nO2vFOVFxVzp+dxzuhXMDiE4EIEjen7OfNuzSIToRjujyVuiWivtdtfofOxzOF0daa4wA0VUC2Ouezc+t+WgyS4foRDCiy9MmbGtf2Vj9dc++Af67iUNlnnvlCe446k36IbrK5IJ8ZZ6v7flo0KVDdKL3QDbPj4wnfcTrx+Pj4yNds99T2O2Uirtl4C5n5Qpafc0Zx69rkMCA6MTkgmSeR458NJilQ3Si50DWH8Qj/5DXK/rBhv1mwe8rWyujDt2Vm9+n7rne8TvjuNedokN0yeS3Tc49d+Wj/lo6RCd6DeRnlffdu8UmzcxeLxbZ/z83P7t1Wvyb8ysVr6q4/zr7c08c+ctf2I8tQHRiakGWmeg7Vz7qz9IhOtFnILvVx8dT/Hfeh6cfH2nxg2c2fme/qq6d+wbuguT2yra+Pl838/xHZxx8FT1J7V8mEJ2YWJDvfOTuzkftWTpEJ3oMZKsNz9NivF4OmosBPP/Yje685h0VLahx3J5PMvvGp+jmzjkzjg5AdEGndvVZIXrtWTpEJ/oLJBu3K5/rpRC9iHHjmKXfTc/tfW0NlDh+1dK9ttE951/q+CsQnQhCdOccvX6XDtGJ/gJ5fnyoc6aiLFcqVVTi2bv9Z3ru6GtrIONIHqrbKavFvVzDBohOTH1/9MhXdW8wS4foRG+BbA2RC7e/1H+wjpR16LX2pFuRcWhXnN/YOMF9o0WITgTRo7vW0Rf1u3SITvQWSGpU1fM5uuhXf7J/rNhvshm6U8JKZBxP1e1U695L8x2jBohOTCxIvl0mWr3c+ag5S4foRJ+iaz/NRZcjq83a8pkzzR1nv9TA6NHL4v0zPy1OWZu/eTbPQHQiiOW1KPK8BK2lf7KHFCA68bPZbTuyy18iSp/qy9zywfra/8rM8sfjwZ5UE4ojmw8883/csiMht1kPnxYP7rIvgV/X6+16+Cj+iugbFlrrfLT6HKNoFa2iiD1AyDPe2UPq60wcR/kmwhB9GIpZ+dr/2szyx+PBntSYzOznc/3M/7/8d/p8vZ75cbD3gQIu+DOiT0zZpS/d74Jm6Z7nhEIgQ/efjuxsLxEVi2t+mOWPx8P7C262ypuQN2/7Lf4t99HfnS/giKMZf6dH7/pJbLt9mNtsjp5dv+ZEdumuZ+QpZT9q+kb68OMvX71mL7MbWIpx3efoGTvjrmu37K5rL+dd1zxxNGNk0ffsJz0x/dy27NI9M/Coxiwdc3RiUNGtm9sVLKLbrlGtwxzvphofTuxn/RCAIIXo0Td7QFBnLR2iEwMFslG2xbm5Mc8fZ+eT/cxQ9NPhcDhe2Y/7IJATZlyHzBTU6NIhOjFQIKl1JxyDdeltl9HnJ/r1eDgMZnooZ8ZFUeTeEVPj5qoQnRgokJVyQYuHrWm6fxbtYW6i557Hh8PhUuPYy8aEIEh1l169lg7RiWEC+eFT9I3t4jVz8O7ay1LNzERPLofDZb+4XoYxPQhBKutx1bN0iE70eFFL+kGn+RVXtGivHFlFX9yUPj1t7/nMRM88Lwbt2US9bV3DTRCC3Mp6HHuAqJylQ3Sizy2wcrD+ZFP0H+eUXax6m2fGNGNWomdD9rj87/NF/ndvhCHIsmrwXjlLh+hEX4FstAtRP9gU3bzmRXuB2+v123p2XjAn0WNtvL4/HnpfZgtEkGXVEltVlw7Rib4CKfrwcrvLWv2H+Am/eE1iJOSaxEnDanKHOJKksOZdRD8fDif107meeh+9hyW6w+Ias3SITvQVyEPZCvezMs+Z2D0+Vr5o9YRkI9PDpVnb7RDH5VBwuRyPx+MpzjgnSdJi39k4PfrZ/GjOfY/dQxGkctdMRZcO0Ylee/RisP7z+PhItap7ZD0vTkFLyKkUr1Hr7RDHwUVz03GZqqCfdvXVcZYO0Ym+Atk+hOebVfZfL3lGXDZ9X/n/ipqQPWnXZOY5hOiXxptRILqgp3bVcZYO0YneAsm79NWTzpvI//1Iy0OkKkJVExJL047siU6GEL256RBd0FO7+uo2S4foRH+BbMsTID+exSv+iLu2PJ6Vf8IheoM+fRDRD7Vu/KAA0QV9tatlxdjd36VDdKLXQDbr6KlGtVtHT9sRUgyX6PX79GFEbzKmWEwmenLue3PcaIJ82vQsydvVV5Xo3i4dohPhrT8numk1R8/t49gzu1WarVCPL/o+LhYNLnGfF6ePJchn5DnGtWhXeZfuEd3bpUN0IsCNJsc28+T2cSRM7vZ9+uii5ysUp9Pp2O9O2JEEWTr8LCjaVVWP7u3SIToRoOjXi2ZaPdMHE72RP2OLnnmeb57J5zv9DeDHEWTp7IlzlB7d3e2rS2z8hSA6EeLW0WuLPn040Zus548set6fk+c9bnkfRZBszL1cus9mV3p09piKp0uH6ESYe8RPumo1pp/9iX48mNTv08cVPX/neRGh+djDzwiCfEerVbZs9uk0XRbjeFet4Z6lQ3Qi0ItBDNOrB6W9iU5XhSVJcj7HcbYllv2OCxbHoBzp/V7qz3FGiqMqH7nn3/p/GeTt6rNihr7wdukQnQj1qq+mfXr7OLTlvI69Io9jQPbyk8muVT32WHYfXJCvSOnHl6uVbRaetavsTulLy2M6zi4dohPBXt5pmF4lYE+id5zmWuIYjuLan+Llr20uwXEztCBfupRLi6NFuyoG7p9f/FEVZ5cO0Ylwr+OOG5nej+gVf6QSWxyDkX8VNtzRM1oc/nx8GtvXv2zr6dvNT7nVPb+wZWkb3pe4unSITgR8YMPZPnm204vo/j9RA2scQ9HPW7Yy/NDd/IHF4u0m0nGr7urSIToR8skshuneTWq9iP5W57rve660qwQhyGdk4jbdsZYO0Ymgj2CK65veh+iXtxL9XK9K2YoQBPlaFSP2z6UcwDvr744uHaITYZ+1luib5Dwb39vHcXSJvj+5/5wDVxxDoNbi+iYI0ZV6+6dwnQ35BfZZOkQnAj9Usbbp/Yt+vWQnpzfDGccAHIerxYUhiHYXUmE6e1aJvUuH6ETop6capjvvVtC/6McWI2N3HP2xjwuKb77yH+90OOTXcrn0L5cV6O2q6qBIa5cO0Yngj0k2N747TO9d9FObQrwnjt7Qv/kEjo+lJQMK8lkayabbS9NivV19VxwUae3SIToR/nnopun2zqtv0ani32ii7oujL/hefO+cphXDCZLtgouWESub5aLqFhvt6rNFlw7RiTe48cG1zia5PkR/yTiUDfDHBp2lN45+yT+UgWpxwwmyXK2yNbIv3qfnjmq1NrNdVdTjbF06RCfe4g4nhum2Zbb2ccjV+iPFoVUGHGMIGxVx9MmQtbjBBFmK/paVzQpNtSeb7arq7GdaS5fPgOjEe9zKqNr0DnHEXHRjeFx7YDyi6PkbG2Zf3GCCfJHdTOslU5+3q4p6nKVLh+jEm9yzzNgkx6/I7BLHiYlu7NSxzxYsjCf6kPviBhNkKWbhX0z0iA/mWbuq6tL5LB2iE+9yc0LTdHPdq1McRzZHP2u17doj5PFEL4oIw+yLG0qQL1KZ998R2+xiaVd1u3T6CoHoxNvchdQ8rNVo493iOBk9+mKRKKN3Pn5wMZ7oQ+6LG0qQJVXb2II37+Jt7ep/ZT3uf+ylC6hLF50+RCfe53bDpun6sLVjHHkHflLjUGr99cvu44k+aC1uIEEiISCvvPEu3tquPv2DdzZLh+jEG91X3DgdVje9Yxz5a6/1OMREPcjDIQetxQ0kyKfYELdkE3L+E3u7Km/c4lpiM2fpEJ14I9GZ6WrxvWsc2WufjDiSZhP0MUUfthY3sCDffM2cjeUd7apul14OFyA68U6is2U2Zdmrcxx71qMXm/JY2c/HaKIPW4sbWBA+TudjeVe7Wvp3zRhr6RCdeC/Rmek0fe4ex5716PnfM3rNs3dL7GiiD7ovbmhB+FIaV9/Vrr78V7EZs3SITryZ6GyZTZjeQxxn1qNn+uv/zEb4ngHzuKIPVosbVpBi5K5ta+fqO9tVxRKbPkuH6MS7iW6scFNBaqQ44oN39/tool8GrcX1KMj5wt6lpfu2rKK78lFexeY6VUqfpUN04u1EN69QL0ty48QhrnVxjd/HEv06bC2uN0Hyb2Xzs+IVdtsqujMfS389TltLh+jE+4luXrdalOTGiUP+ZdZP5Ywl+sC1uJ4EKQdf5pUJvMJu6eM9+YhYn/61lNprs3SITryh6Oy61awwPkocaoHAek3bWKIPuy+uL0HKr0VzYyHvvnkfv/DkQ5wOu/zMd8h95wfKyYfVWTpEJ95RdFZ8PySjxHHl3y8GY4k+cC2uJ0HERMcY/XDReR+/8OVDua1DxGtzX8osHaIT7ym6WXw/nMeIw5gyWHrUsUS3+dMnPQkiPjC9S2fd95Krv/DmQzU911yrzClr6RCdeFPR+c2Oh4/D3GxvEW0k0QfeF9eb6GfrJ2Vq/WVcRC7w5EO9twO7VxN16RFEl7yr6Iu9UXy/3NhTGlIZR6J16ZYOfSzRE0s/2St9CXKxzdKX+jL6d7RifXyOLx/fn8vlMsqOk7VcySZn6RCdeFvR2c531/mwtakRhzpjsP25kUQfeoreOo79SQwzyuU1W5f+qXn9FWmXoSi0bldylu6PYxM9n94nQHSNiUTnJbmOY9k6cciCv9WzkUQ/Mnl6pl0c5+x9lf9dtqv8y9hcoFgqYi9F98terUO7oll6tHHsq8melH5kpBvzzz4fkfzDEF0ymejsxCfLWXINqBfH/mj7VinvoDCS6Pkb4DX//mgzlYovauddtqv4cOBb44rBen5XtUxG+yp6l3Ylu/Ts71gvgPnJJX9m//vUH1llD8jnQXRiOtHZftj6Z8FYqBtHvjVP79Cz2zedr2OJnjhHFH3ROA5Zvijfl2hXR+swi/rxrGe3r6J3alfRSlPd0q1/fHysF4vFOjNdM2CT/eRB/4TokglFN+pj3fq5+nHEF+PvFEOLeD+o6Oc4Vv4a6yf7pFkc+5P6fVvULqra1Vd+j9T8JArHFL1Lu9K6dPPGEBnpx0ce4oZ16Sv9JxBdMqXobD9sh4l6gziueiWOdtJcftlT+yJfZSgq/ZeBa+5N86EPq5pee/DpGLl3aVffkd/0ddGfs/5biC7n7RBdMqnolol6WwXax6G8hUs8kIEX+hrbD96hN8yHsXspH+y429W30Xm7Ru5d2tVyZaB/kfw8PsriX/ShzcgXi21eoVOe6oyjNhBd0mnIa+6SO7YcvrePQ+/SBtnIkkiJ4qFn6I2vPdDjz7+DnO1qaT3/lY2t+yvGFWgVuTX12Xn//VIeehnmO+OoD0SXdJvbsr0z7VRrHYfxTWNbYO/MnobFiX2bfa80zIcxfVp42hXb7mq9FD2nfbuKmOf6kIH67KL/Vv9Mvuam9P+uOBoA0SUdi1jm3pl262yt4zBu4c4e74WLMnB3n3zRE1X5SGLtHej7kU97T7syF82dA/eeRdfG7mKGXozcn/pD2hTdGUcDILqkc7Xa3DvT5jbCbeMwOvSBZs/Zot7leDp2qULUxpuPJCuy61+l8qvuEheDDUe7+jSsc62hLzq1K6a5ef+nZ/nKudZqrOYU3RVHE0IR/W9wNDv1+2hhGV8yQ1l4LTYNXE5Dd+eZ6CxIwfpVhqn9VIzdLyf2C8av59KtxT/zrjdiz+oKs3y1Wtlecl303woX9pM+CEP0XVc2m86v0f0VdmvT9CN7SgXt41D/tl4mu/TrZZIMPDkv2dk/iaMyGY+1R4oxu/pb9lfYRatVRB9bMXtmzxGv0DofzPJ82zsnt/qp/WY+RdfeBfutpoQi+p8Yui9+NjdzRb3p8L1LHHLjjub1vujmxrGzR6z50Gfi+gzleDjVu0XWZ7bnNbt+/LPYHec8oX3Iobt4jx9sX1xei1vXiKMBmKNLrA2rEVkc5op6w+p7tzj2xV/XS3HiHV3sO0GDxZoP/Xp8PdB97bvbfqqVsqW13F4yXDGuZG1OyEv11T/rjKM+EF1ibViNyOMwT6NoVhjrGsc1u3pL/4vqdOKdXLfnQ58c+acknnb1GWUilvtfPXjycT17P0y2X8Ze2F+Z3Xehvtb5e+KoiyeO2kB0QRGHuaLeaPjeQxwb/e/5+sCgEflItHD0EZN/CXPQdpVfKHc6O2dEVRtmSljNffE0N8pBdIWARGcHxDbZvtJ/HPrC27C7VnsliyM5Hy/6e9YHTP7vrQHblbxk8eiqfrAu3dahf7GRO1tFh+gqIYleftu38qv/OEbYGjsI142or+s2H+rHM1i7Ous118vpbBu01ZihFxe0aON0vlEOoiuEJTq/crXu3vfe4zBKBtozL8d4hPXwNiTaRaeaRcpw6eIeOOcM1K72ZnLzBJ/ZR6lfvma7IL0UPWI/0a5lg+gKgYnOh+/8jBMrvcfhmdPuRXfk12UK9HetKSQeOlp7UY1B2tWeTcxkis3xxdI/brdei/7BfgLRFUITnW1KrVmT6z0OfZSptUT5Di9H1kanxdi4rr6Xq3usbDJAu7qyWZmO8a35XSzkOW/HuPg2h+7FIXJaGR6iK4QnOq++1zkidoA49rF0XWuF+giUvc6I7JPTUd/Np39Fag+dau/z679dWUftpuvGS/jb1UMvxuVXuBhTdIiuEKDofPheoyY3TBzX88liTOUJ8SOQnOUkkgQAAA4ISURBVGNRc0uc763t1v0h2tX+XOW6mWRLPhReWo296M/1xTWIrhKi6Jbqe+W1ncPFcT5d+BjY0Q+dLsfTiVeXesctiP4d2XJewfLRHGs+9FKhiVnxcORDoKymbT4+IHoFYYrOq+9VNblB49C3nngW2OXB8ce438L8XnsLmi/aV41vT3tteD4a48qHOh8yMmw+1Z2PnJ9VcejzM826882DraJDdJVARbcM3/0LbSPG4alsa43Yv/usDsk5ieP4mI/Rtc5Z+zvavELbz3cMT/S8Lsdya/24KvOxTkVPHv3wje4QXSNU0dlCdsU+uRHj0Ief7p5Ws+yYd/NZRx8bff0+UTF/x/5quinauxP9Y5cZhDUfzajIB08uz26NfPzbrJ+X9c56jQtE1whXdEul1nM6y1SiawNOffbORO/vEV30vfHQJe64wm/PRyO8+bha5ups5N6sXT3MgyLHiKMmEF3gisO8dNVz7eqYcezPsljomSDX7J1rPqIN0D1/KEmCvwupeYseHl+OL46f9VF7i/n+14f5JyG6JGjRixso6e3B0amPHkcS592SJrpeptP80x5x6+x+RBNB/0PG7DZw0a2a2xYIPHHszMqbef5rAUSXhC364so6dT6Xy5kkjn0Sa2Nk/c1qT20jet2XeyfR2XKKgE823HH8PIx9cZZ97guIrhG46LayjXWmHkQcni1z7u8qd6XeI7roFS+WZbyARffsd2fP9cSxNhbNd/xYqRyILgledH6DNutMPTzR606q3b+ki27swj2e4sQ+uAlWdLfm1hV/dxz5ERMfdLnLjR9CUQDRJeGLbtknZ5mpBxHH4prEtDO1u+j6TNzy7WYnUNEtszDXR1LgjmOtdegbl+cQXeEdRLcstLGGEYbogiQ565e1tRJd+yX/fiGVIEWvunqNvYAvjmyOLq5T2+bd+8qafYgueQvRLQttZsMPS3SGpx7vEf2S7bDJhuiNlsVDFJ3nT4dti/PHkXXpj9dmsykG8asNe8ZAcbQCogtqxMGuXTV2vwcu+qLcAnfOd8bppfqjgmWq2pjwRLeuqF0SZcpum5f44tjR7tePR+rQHKKrvIvotk5BvU79DUQfi9BEt2t+1rYYsppLZRzbKP1YfTwjp+UQXeN9RLctwcbUQCA6EZbods2zxCklCL4tLvyNP/WB6IK6cVg6dTHmg+hESILYt8EV389KNq0zFohOzE10y0xdFOUgOhGOIJYxmLgFu35xkLXYCNGJ2YnO76UuugOIToQiiF1zWi1RLvWzbIuD6CozFN3WS2TnTEF0IgxB1jxP+pFgyoqjbXENoivMUXTbTP1wvEJ0IgRB7HNz7VaVyheBZVscRFeZp+i2jXKHO0QXTC+IXXPjEgX5HGvNHaIrzFR0a0uyXenSDIgu6NauLMlh+5uUxTVn4iA6MVvRrVdI1N8NbgeiC7q0K4fmzOaT6wECohPzFd12+Iy6f6YNEF3Qul1dHZpb1skvlRmD6MScRbddverrH6qB6IKW7cp1hZrN5vxYausRIgREJ+YturUoV+c+bQ4guqBVPmyzKbHblRPLjTMuIDoxc9Ht4/eq9uMEogta5MN1eoxd88XiVHmTLYgumb3o9m7EPyJ0AtEFzVdB7Jpf3JmoMceC6AREt4/fnf2IF4guaJgPy2bF9lmQQHQCoi8cCzq2Om8VEF3QKB+OClzbcZUEohMQPcda7W1elYPogvr5sH70+RftH2hXC4iuEUBCrKWg6nKPDkQX1M3H/ujS/I+0K4iuEkRCbOP3hqpDdEHNE39s365iMwNEl0B0QS9xWAeRTdbaILqgTj6s36zyCjWILoHogn7isC61NVAdogsq87G3fquqYyiILoHogr7isC611VYdogsq8uEasx9OcqoE0SUQXdBfHPYl3XqqQ3SBNx+uMbv+KUN0CUQX9BmHbVdsPdUhusCdj/3J+vHy3TEQXQLRBSPcMEAdVjqA6AJXPs7W8VJeaDd3x0B0CUQX9B2HtSpXudgG0QXWfDgLcPpRcCUQXQLRBb3H4diw5Vcdogss+bAXP9xDJYgugeiCQW7Ty1rkoWIPPEQXmPlwd+YXV/EDoksgumCQOKzbYg+ugxAguoKeD8c1qNapOcHz0RiITkB0whaHS3VHLwTRBUo+3J25dWpO2PLREIhOQHTCHodDdfu8EqILRD6uzjK789tSYM9HIyA6AdEJVxwu1S3HSEJ0QZGPxHFtmnf+I3DlowEQnYDohDsOp+qx0SdBdMHPZusZsh9O/EuS4c5HbSA6AdEJXxwu1Y0WC9EFruvM64zZS3z5qAlEJyA64Y/Dsa6u140heoG7yl5RgFPx56MWEJ2A6ERVHI51dbWHgugVlje5uh+iSyC6YJw47Hvg824q79YhuvP600adeQ5El0B0wVhxuFeLsvW2mYueOC9Mqz8zl0B0SR+i7346st10fYWfXQ9vYqw4zi/WhkVTjjd/RnQWdiWJq4hRdObNG8kfalfsA25KH6KDpnhatGdH5zuxa/qRrJ0DnULzNfuNORFIj77tyG7T9RW2fbyJMeO4eVy3bpl7M3aNPsyTc4xTfCAtM/N32hXm6II3nEu5J+uHi/9a1jegfj6usa/6djgc299UCXN0CUQXTBCH+0ykFpWnsKiZj32F5WznYCMgugSiCyaJ4+pcbstdf99+vU4+kpN7TNNL/BBdAtEFU8XhXTd+W9er8nE9e0YzeeTHE9pVCUSXvHNCrrGvZ7sc37EO782Hfx3tUGyMQbsiILrkzRPim63n7f7dJuzOfFTNymmTINoVAdElb58Qz/kKRcde57rMcLDmY181XpdbgdGuFCC65C8kxHf5dc7pfTp2lo/ruXK8rliOdqUC0SV/JCHuA42pY0/eYsau5WN/PlbFZViOdqUC0SV/JSG7nfcazUL2OPzyHOVjH1cP162rC2hXBESX/B3Rq6rwZfd3Clv2PI5z7Dklxms52pUKRJf8JdGLZWbmA+MS8Jw9PlXshqFvLNfeN7QrAqJL/pjoNfaAl73hMbhx/Dmu6fjhcPLsY0e7IiC65O+JXrdfL2wPokR3Teo7frjEtgG7BO2KgOiSPyn6Ij8/rc40t+zb/eoMSRLXm4+X77VGhQHtioDokj8req0do5LjKU5G1f2a1Cy5ESfXrFwH7YqA6JK/LHqda0AM3bPefeBC3f7cYDJOb6xivK6AdkVAdMkfF32Rr0g31uoYx0nfs/ckyQRv1Ic3lnyBdqUC0SUzEH2RV7RbGHa4HI/xqVMff03O5zg+tvG7lLzxTn20KwKiS2Yieu5c4wGzptzxeIrj+JxkuDr7ff7oOY7j07G93QWXU7OeXIB2RUB0yXxEXxRT9i6yj8axw5YetCsCokvmJXpOUmtHzYSOd6v+o10REF0yQ9EXxRXe4XXtWc1/pvmwANElSAjRJo4kHNupH4foAoguQUKI1nEkSas1rx4VP6lV/dnng4DoEiSE6BhH421q3blkm/FYyQ35EEB0CRJC9HI31aTZzvOWZIY7C27IhwCiS5AQos/bJue+9y98tt+ucvMN8iGA6BIkhBji/ujXzPhs5wtztj6XfJ9NUntZHPkQQHQJEkIMIbpKueEtPuY4PS8ezp6YbZFjrzJGHGhXBESXoGGFBfIhgOgSJISA6AK0KwKiS9CwwgL5EEB0CRJCQHQB2hUB0SVoWGGBfAggugQJISC6AO2KgOgSNKywQD4EEF2ChBAQXYB2RUB0CRpWWCAfAoguQUIIiC5AuyIgugQNKyyQDwFElyAhBEQXoF0REF2ChtWS6zm/tuVovUd5e5APAUSXICHE2KLnN4S45BetH12nvLcB+RBAdAkSQowset6ZZ9eYx9l/sIfbg3wIILoECSHGFT07GP6yKD0/HHq8LSPyIYDoEiSEGFX0c2Z3nB0PXxw30eM0HfkQQHQJEkKMKvpF2J3knl/YE9qDfAggugQJIcYUPe/QD/l/5iW5PuvuyIcAokuQEGJM0fNbt5UVuHPz+xp7QT4EEF2ChBBjip6P3E/sx72AfAggugQJIUYUfd93AU4F+RBAdAkSQowo+rnvJTUV5EMA0SVICDGi6LGsxfUP8iGA6BIkhBhR9OOh591wKsiHAKJLkBBiRNEvYrvMECAfAoguQUKIEUS/ZrdlSpJik8yp+O/a91SrC/IhgOgSJIQYXvSr/faq7HndQD4EEF2ChBDDi75njg8xV0c+BBBdgoQQIwzd90lO3rEfk5I+r0VfIB8KEF2ChBDjFeOG3BeHfEggugQJIUYTfdB9cciHBKJLkBBiNNEH3ReHfEggugQJIUYTfdB9cciH5A+Jvtt2ZNfHS7AfNSWMOMYSfdB9cYvFn8lHGHGEITrokbFEH3RfXCY66JEQRP/fD+gT9gEPQlGL6/ewCZVvtIk++R/7gJvSXXTwjgxciwOhAdHnycC1OBAaEH2eDFyLA6EB0efJsPviQHBA9FlyHXZfHAgOiD5LUIubGxB9lqAWNzcg+ixBLW5uQPRZMvC+OBAcEH2ODL0vDgQHRJ8jqMXNDog+R+K+75IMQgeiz5EjtsvMDYg+RzBFnx0QfYbssYo+OyD6DIkxcp8dEH0+7MXp7SdsdJ8dEH0u7I+Hw6XYI4Oa+/yA6DOhuOVarvcZpbj5AdFnQizvpXjCPvf5AdFnQiF6NnTP7pd86fteayBwIPpMSITn2draBZW4uQHR58LpcLiczuesZz+hP58dEH02xMdimn5BHW6GQPQZsU/iOME1a7MEogMwAyA6ADMAogMwAyA6ADMAogMwAyA6AH+dxWLxf+6WHpUh3SHjAAAAAElFTkSuQmCC\"\n",
        "height=\"200\"></center>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QfCV_z6HBxMs",
        "colab_type": "text"
      },
      "source": [
        "What 'seed' value should we choose for $\\yhat$? That's up to us, and depends on how we *want* our network's output to change when we optimize it. That depends on how our scalar loss value $\\L$ depends on $\\y$. \n",
        "\n",
        "In some applications, like most supervised learning, the loss $\\L$ can be computed directly within the network, and the final output of this training network is hence $y = \\L$. Therefore we should set $\\yhat = \\idld{\\y} = \\idld{\\L} = 1$.\n",
        "\n",
        "In other applications, the final loss is computed outside the network, and we set $\\yhat$ as appropriate. This happens for example in some policy gradient methods that are used in reinforcement learning."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NJgDU00nlZqr",
        "colab_type": "text"
      },
      "source": [
        "## Defining a loss function"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6la3Pwf9lbO4",
        "colab_type": "text"
      },
      "source": [
        "From the previous discussion, you should understand that before we can train our network, we need to define how the loss $\\L$ depends on the output of our net. That will allow us to fix a value for $\\yhat$, which is needed to 'seed' the backpropagation operation in order to compute $\\idld{\\th}$, the gradients of our parameters. \n",
        "\n",
        "To keep things simple, we will be implementing the so-called L2 loss, which measures how far the output $y$ is from the true output $y'$, and defines the loss to be:\n",
        "\n",
        "$$\\L = (y - y')^2$$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T8LihTWWTB6J",
        "colab_type": "text"
      },
      "source": [
        "Here is this function. It takes the model, the input to pass to the model, and the target output. It then computes the actual output, and calculates the gradients of the parameters such as to minimize this loss."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s-cxJ_jiGQmx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def L2_loss(model, x, y_target):\n",
        "    # compute the output of the model:\n",
        "    y, dual = model.forward(x)\n",
        "    \n",
        "    # compute the loss\n",
        "    loss = np.mean((y - y_target) ** 2)\n",
        "    \n",
        "    # compute y_hat\n",
        "    y_hat = 2.0 * (y - y_target) # = d(loss)/dy\n",
        "    \n",
        "    # trigger backpropagation\n",
        "    dual(y_hat)\n",
        "    \n",
        "    return loss"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7rFjxb6DUEfO",
        "colab_type": "text"
      },
      "source": [
        "Let's try it out! The following code will feed random vectors to our previous multi-layer-perceptron, and compute a loss that measures how similar the output of the network is to the input of the network. In other words, the task here is for the MLP to behave like the identity function:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p8Xnr7sbf_Wi",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "x = y = np.random.randn(2)\n",
        "L2_loss(chain, x, y)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IJqCvUbjioTx",
        "colab_type": "text"
      },
      "source": [
        "We can now use the gradients computed by `l2_loss` to do SGD. Re-evaluating the loss on the original example should show that it has decreased:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F-0A6Td9iwd-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "chain.descend(0.1)\n",
        "L2_loss(chain, x, y)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hp4MjCCz7r1_",
        "colab_type": "text"
      },
      "source": [
        "**Exercise**: how would you modify the above code to make the loss *increase*?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "coug01whnvJB",
        "colab_type": "text"
      },
      "source": [
        "## The training loop "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D_bjG1m6g7Hq",
        "colab_type": "text"
      },
      "source": [
        "Let's define a `train` function that applies this process repeatedly to train the network on a task. The loss as a function of time will be plotted in a log plot:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tZkOLMSFhCvS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "def to_array(x):\n",
        "    return np.array(x, copy=False, ndmin=1)\n",
        "\n",
        "def train(model, data_generator, total_batches, batch_size=64, learning_rate=0.01, loss_function=L2_loss):\n",
        "    losses = []\n",
        "    \n",
        "    period = np.floor(total_batches / 10) # how often to print loss\n",
        "    avg_loss = 0 \n",
        "    \n",
        "    print(\"batch\\tloss\")\n",
        "    \n",
        "    for batch_number in range(1, total_batches + 1):\n",
        "      \n",
        "        # collect gradients from `batch_size` examples\n",
        "        loss = 0 \n",
        "        for i in range(batch_size):\n",
        "            x, y = data_generator()\n",
        "            loss += loss_function(model, to_array(x), to_array(y))\n",
        "        loss /= batch_size\n",
        "        losses.append(loss)\n",
        "    \n",
        "        # periodically print the loss\n",
        "        avg_loss += loss    \n",
        "        if batch_number % period == 0: \n",
        "            print(f\"{batch_number}\\t{avg_loss/period:0.3g}\")\n",
        "            avg_loss = 0\n",
        "        \n",
        "        # do a step of SGD, using a gradually decreasing learning rate\n",
        "        schedule = np.sqrt(1.0 - (batch_number - 1) / total_batches)\n",
        "        step_size = learning_rate / batch_size * schedule\n",
        "        model.descend(step_size)\n",
        "\n",
        "    plt.semilogy(losses)\n",
        "    plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6RbixQoNjEYJ",
        "colab_type": "text"
      },
      "source": [
        "This function takes the following arguments:\n",
        "\n",
        "| argument | meaning |\n",
        "| ---- | ---- |\n",
        "| `model` | the model to train |\n",
        "| `data_generator` | a function that will generate `(x, y)` example pairs | \n",
        "| `total_batches` | the number of batches to train on before stopping |\n",
        "| `batch_size` | the number of examples to combine in each batch |\n",
        "| `learning_rate` | the size of the gradient descent steps to take |\n",
        "| `loss_function` | the loss function to use |\n",
        "\n",
        "Note that we defined *default* values of `batch_size=64`, `learning_rate=0.01`, and `loss_function=L2_loss`, so these arguments are optional."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ndG3SP4R__Xf",
        "colab_type": "text"
      },
      "source": [
        "Let's put the `train` function to work! To do this we first define a generator function that will make a random 2-vector and returns it as both the `x` and `y` values, since we want out network to reproduce the input as its output. Essentially, we will be teaching the network to act like the *identity function*:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wrjl9tzYAach",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def make_random_data():\n",
        "    x = np.random.randn(2)\n",
        "    return x, x"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cEyXn4YkAdOi",
        "colab_type": "text"
      },
      "source": [
        "Let's define a simple multi-layer perceptron for this task, and begin training it:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nSXbrgF3jGkc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "mlp_identity = Chain(Linear(2, 3), Relu(), Linear(3, 2))\n",
        "train(mlp_identity, make_random_data, 2000)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zZi6wS73f0pS",
        "colab_type": "text"
      },
      "source": [
        "You should see a curve and table that shows the loss going down slowly. It should be in the range 0.05 - 0.3 by the end of training."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y6-8wzTqll2D",
        "colab_type": "text"
      },
      "source": [
        "Now that we have trained the net, let's try the trained MLP on a particular input and see how close it is to reproducing the input vector of `[1, -1]`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-BO3C7zllsq0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "x = np.array([1, -1])\n",
        "y, _ = mlp_identity.forward(x)\n",
        "print(y)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bh-5lmdPl3jc",
        "colab_type": "text"
      },
      "source": [
        "It's not 100% perfect, but it's pretty good! \n",
        "\n",
        "**Exercise**: try write a simpler network that just consists of a single linear layer, and train it on this task. What shape must this linear layer have?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Wq48vFW2jbm-",
        "colab_type": "text"
      },
      "source": [
        "## Training on MNIST"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w4mLbZx5p_jm",
        "colab_type": "text"
      },
      "source": [
        "We are now going to train on *half* of the famous MNIST (pronounced \"em-nist\") dataset of handwritten digits (taking only the digits 0 through 4). "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7mbc4spFoX_K",
        "colab_type": "text"
      },
      "source": [
        "### Getting the training data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ojL1Grdroahh",
        "colab_type": "text"
      },
      "source": [
        "To access the data, we will use tensorflow - but we will not use any other functionality from tensorflow."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BwPs_xjqpQwV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "mnist = tf.keras.datasets.mnist\n",
        "\n",
        "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
        "x_train, x_test = x_train / 255.0, x_test / 255.0"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q9g9us2ysaXX",
        "colab_type": "text"
      },
      "source": [
        "The variables `x_train` and `x_test` each store a numpy array that contains the 60,000 images and the labels respectively. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fH60SAkgslCD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(\"x shape is\", x_train.shape)\n",
        "print(\"y shape is\", y_train.shape)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oiHgQCgQocZI",
        "colab_type": "text"
      },
      "source": [
        "The dimensions 28, 28 are the width and height of the images in the training set."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Z9b1rI-ioK4h",
        "colab_type": "text"
      },
      "source": [
        "### Visualizing the training data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HXGTufqyh3Pw",
        "colab_type": "text"
      },
      "source": [
        "Let's visualize some of the training data. Here's a function that plots an array as an grayscale image:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OSwM-ZeTh67N",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "def plot_image(array):\n",
        "    plt.imshow(array, cmap='gray_r');"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vWrrOx5wifBs",
        "colab_type": "text"
      },
      "source": [
        "The following code will take the first 10 image arrays from the training set, stack them horizontally into one 2-dimensional array, and plot it using our new function. We are also printing the first 10 labels so we can compare them."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DvA58UJliBIX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "plot_image(np.hstack(x_train[1:10]))\n",
        "print(y_train[1:10])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e6zfyFjgjh7h",
        "colab_type": "text"
      },
      "source": [
        "As you can see, the *i*'th label corresponds to the *i*'th handwritten digit!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QRpoxPLzoSup",
        "colab_type": "text"
      },
      "source": [
        "### Sampling from the training data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VxyMqwaNsxbW",
        "colab_type": "text"
      },
      "source": [
        "We now write a function that will sample at random an image and its matching label, returning them as a pair. The `while` loop will make sure we get a digit that is either zero, one, two, three, or four - it'll keep trying again until it gets one:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KhUF1mZiswwj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def sample_random_mnist_example():\n",
        "    # keep sampling until we get a digit in range 0..4\n",
        "    while True:\n",
        "        i = np.random.randint(60000)\n",
        "        x = x_train[i].flatten()\n",
        "        y = y_train[i].flatten()\n",
        "        if y[0] < 5: break\n",
        "    return x, y"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E1ybXvm5tBXo",
        "colab_type": "text"
      },
      "source": [
        "Let's try this function out. Notice that the single 1 x 28 x 28 array representing the image into a 784-dimensional vector. Similarly, the label is stored in a 1-dimensional vector."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RBsvKD-htASM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "x, y = sample_random_mnist_example()\n",
        "print(\"x shape is\", x.shape)\n",
        "print(\"y shape is\", y.shape)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b1mTO75sxbsG",
        "colab_type": "text"
      },
      "source": [
        "### Optional reading: can we expect problems from using L2 loss for classification?\n",
        " \n",
        "The training data labels are the integers 0, 1, 2, .., 9, corresponding to a training image of the handwritten digit zero, one, two, etc. \n",
        "\n",
        "To re-use our previous code, the loss function we will use for this problem is the L2 loss. However, this choice of loss function has several problems. In fact, it's really bad!\n",
        "\n",
        "To see why, let's look at two kinds of misclassification the net might do:\n",
        "\n",
        "1. the net predicts **y = 0** but the actual label is **y' = 4**. This gives a **high** L2 loss value of $\\L = (y - y')^2 = (0-4)^2 = 16$\n",
        "2. the net predicts **y = 0** but the actual label is **y' = 1**. This gives a **low** L2 loss value of $\\L = (y - y')^2 = (0-1)^2 = 1$\n",
        "\n",
        "But this difference not ideal, because we as human beings care equally about both these misclassifications! Using the L2 loss function will train the net to minimize one kind of error much more than the other kind of error.\n",
        "\n",
        "Similarly, our net will usually predict non-integer values like `3.8` or `5.2`, which we will choose to interpret as predictions for the nearest integer, like four or five. Again, this is unnatural, because it only allows the network to express uncertainty between neighboring numbers three versus four -- it cannot express uncertainty between zero and four!\n",
        "\n",
        "A much better choice of loss function is called the *cross-entropy loss*, which has none of these problems. We will leave this as an exercise in a later section.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yTk-bP5o04Lp",
        "colab_type": "text"
      },
      "source": [
        "### Training the net "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iZ99iBHcp0yi",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "mnist_mlp = Chain(Linear(28*28, 100), Relu(), Linear(100, 1))\n",
        "train(mnist_mlp, sample_random_mnist_example, 1000, learning_rate=0.1)  "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i8CBMdQFq_KG",
        "colab_type": "text"
      },
      "source": [
        "You should see a steadily decreasing loss curve above, with a final loss of around 0.05 to 0.2 (try again if the loss remains high by the end of training).\n",
        "\n",
        "Let's try the trained net out on an example from the test set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Hrqg0sQ0q-YO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "x = x_test[1]\n",
        "plot_image(x)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z2Agi9our-u7",
        "colab_type": "text"
      },
      "source": [
        "You should see an image of a handwritten digit two above. So if we are lucky, the net should predict a label that is close to the value `2.0`:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "S3dUAAFMrpS3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "y, _ = mnist_mlp.forward(x.flatten())\n",
        "y"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0hHzf5ZE4OJB",
        "colab_type": "text"
      },
      "source": [
        "We can try to evaluate the performance by hand, but this isn't very scientific. So let's measure the accuracy on the entire test set, which contains 10,000 examples. We'll do this by looping over all examples in the test set, testing whether the output of the net, when rounded to the closest integer, gives the true label. We'll then return the fraction of examples that were correctly classified:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-rVlnHTI4jQh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def measure_mnist_accuracy(model, to_prediction=np.round):\n",
        "  \n",
        "    number_correct = number_seen = 0\n",
        "    for i in range(10000):\n",
        "        x = x_test[i]\n",
        "        y_target = y_test[i]\n",
        "        \n",
        "        # skip labels not in 0..4\n",
        "        if y_target >= 5: \n",
        "            continue      \n",
        "            \n",
        "        # get net's prediction\n",
        "        y, _ = model.forward(x.flatten())\n",
        "        \n",
        "        # update count of number of correctly predicted examples\n",
        "        if to_prediction(y) == y_target: \n",
        "            number_correct += 1\n",
        "        number_seen += 1\n",
        "        \n",
        "    return number_correct / number_seen"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bRZXvZVA6D96",
        "colab_type": "text"
      },
      "source": [
        "We've written this as a function so that it is easy to use again later. The first argument is the model whose predictions should be measured. We also have an optional `to_prediction` argument to allow us to test nets that have output of different forms than a single real number (we'll need this later when we investigate cross-entropy).\n",
        "\n",
        "Let's try it out on the model we just trained:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3OgxrAtAkcsi",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "measure_mnist_accuracy(mnist_mlp)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J0dVMWH0krnQ",
        "colab_type": "text"
      },
      "source": [
        "You should see an accuracy in the range 70-90%. If you don't, try running the `train` step again and testing the accuracy one more time -- sometimes the random initialization of the net just doesn't allow it to learn as well."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PEUh9MYSDf-z",
        "colab_type": "text"
      },
      "source": [
        "### Exercise: switching to cross-entropy loss"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IFWrwKscDkZZ",
        "colab_type": "text"
      },
      "source": [
        "We should be able to improve our training process by using the so-called **cross-entropy loss** (CE loss) instead of L2 loss. This will avoid the problem of some misclassifications (e.g. 0 vs 4) contributing much more loss than others (0 vs 1).\n",
        "\n",
        "For simple situations like MNIST, the CE loss can be defined as follows, where $y$ is a vector of probabilities, one for each class, and $i$ is the index of the _true_ class:\n",
        "\n",
        "$$\\mathrm{CE}(y, i) = - log(y_i)$$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PZQTATj7FW4W",
        "colab_type": "text"
      },
      "source": [
        "**Simple exercise**: convince yourself that minimizing this loss will encourage the likelihood of the _true_ class to be high. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M1D3iPN8ivWs",
        "colab_type": "text"
      },
      "source": [
        "How do we get our network to produce a vector of probabilities? The most common way is to use the so-called **softmax function**, which is defined as:\n",
        "\n",
        "$$\\textrm{softmax}(\\x) = \\frac{e^{\\x}}{\\sum e^\\x}$$ \n",
        "\n",
        "In English terms: we exponentiate the input vector, and then normalize. This last step ensures the components of the vector will add up to 1, so that the vector can be interpreted as a vector of probabilities (because probabilities over all the classes must sum to 1)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a8qnyyMKjjhS",
        "colab_type": "text"
      },
      "source": [
        "We won't spend much time explaining cross entropy loss and softmax, though there is a lot more to say. Instead, we will cut to the code. This code will combine both the cross entropy and softmax operations into a single expression that is more numerically stable:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ajs1T79iD2jS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def cross_entropy_loss(model, x, y_target):\n",
        "  \n",
        "    # compute the output of the model\n",
        "    y, dual = model.forward(x)\n",
        "    \n",
        "    # compute the loss\n",
        "    index = round(y_target[0]) \n",
        "    log_softmax = y - np.log(np.sum(np.exp(y)))\n",
        "    loss = -log_softmax[index]\n",
        "    \n",
        "    # compute y_hat\n",
        "    y_hat = np.exp(log_softmax) \n",
        "    y_hat[index] -= 1\n",
        "    \n",
        "    # trigger backpropagation\n",
        "    dual(y_hat)\n",
        "    \n",
        "    return loss"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TfoTv9Gij50g",
        "colab_type": "text"
      },
      "source": [
        "**Exercise**: derive from scratch the code for `loss`, which combines the computation of softmax and the cross entropy into one step. \n",
        "\n",
        "**Harder exercise**: derive the code for `y_hat` from first principles. Hint: remember you are trying to compute $\\idld{y}$, and you will need to use the chain rule through the combined log-softmax operation. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TwmkxTNfOIof",
        "colab_type": "text"
      },
      "source": [
        "Let's try it out:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F_cfCbifOJoP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "mnist_mlp_ce = Chain(Linear(28*28, 100), Relu(), Linear(100, 10))\n",
        "train(mnist_mlp_ce, sample_random_mnist_example, 1000, learning_rate=0.1, loss_function=cross_entropy_loss)  "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ROIgnzJzOd0f",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "measure_mnist_accuracy(mnist_mlp_ce, to_prediction=np.argmax)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BPWhfN9ckOro",
        "colab_type": "text"
      },
      "source": [
        "We should see the accuracy here is much higher than when we used L2 loss! In fact, it should be in the range 95-99%. Here, the theoretical problems with L2 loss corresponded to a large deficit in actual accuracy!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A85P5E6mqCA3",
        "colab_type": "text"
      },
      "source": [
        "## Conclusion"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "glGUQ3fyrUbU",
        "colab_type": "text"
      },
      "source": [
        "In this practical we discussed what *gradients* of a differentiable function represent, how reverse-mode automatic differentiation allows us to calculate gradients, and how these correspond to vector-Jacobian products. We used this theoretical foundation to implement a simple deep learning framework that we used to train a multi-layer perceptron to solve the MNIST classification task, with good accuracy.\n",
        "\n",
        "By following this practical and understanding the underlying ideas you'll be in a much better position to understand how modern deep learning frameworks work, why it is efficient, and what you might need to do to implement your own advanced layers if you need to."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H4BtEZY560av",
        "colab_type": "text"
      },
      "source": [
        "## Extra reading\n",
        "\n",
        "* [Blogpost: Backpropagation by Chris Olah](http://colah.github.io/posts/2015-08-Backprop)\n",
        "\n",
        "* [Video: 3b1b episode about backpropogation](https://www.youtube.com/watch?v=tIeHLnjs5U8)\n",
        "\n",
        "* [Wikipedia page on automatic differentiation](https://en.wikipedia.org/wiki/Automatic_differentiation)\n",
        "\n",
        "### Advanced:\n",
        "\n",
        "* [Video: Lecture on Automatic Differentiation from the 2018 MLSS in Stellenbosch](https://www.youtube.com/watch?v=23ksBFYWIcM)\n",
        "\n",
        "* [Video: Some Principles of Differentiable Programming Languages from POPL2018](https://www.youtube.com/watch?v=qhPBfysSYI8)\n",
        "\n",
        "* [Wikipedia page on dual numbers](https://en.wikipedia.org/wiki/Dual_number)\n",
        "\n",
        "* [Paper: The simple essence of automatic differentiation](https://arxiv.org/abs/1804.00746) (ironically, this is not a particularly simple paper)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aEOanf1Uvaom",
        "colab_type": "text"
      },
      "source": [
        "# Appendix"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oQztwaYixYWu",
        "colab_type": "text"
      },
      "source": [
        "This appendix will explain how to derive the code for the backward pass of the Linear and ReLU layers, using the vector-Jacobian product. It is optional reading, but strongly encouraged to help build your intuition and confidence with the mathematics that underlies automatic differentiation. It will also give you the confidence to try to write your own layers!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rNWYBtLHwn4j",
        "colab_type": "text"
      },
      "source": [
        "## Recap of differentiation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EVF-jlkL7Gp-",
        "colab_type": "text"
      },
      "source": [
        "Here is a quick list of some basic rules for rewriting sums, products, and derivatives that you can quickly review to make sure they are familiar. Here, $\\newcommand{\\defineblock}[2]{\\newcommand{#1}{#2{\\hspace{0.1ex}\\blacksquare \\hspace{0.1ex}}}#1} \\defineblock{\\xa}{\\red}$ and $\\defineblock{\\xb}{\\blue}$ stand for *any* expression you like, and remember, multiplication $\\times$ is not _normally_ written explicitly, it is implied when two expressions are next to each other, but we will use $\\times$ here to be clearer.\n",
        "\n",
        "1. Differentiating a sum:\n",
        "\n",
        "$$\n",
        "\\partialfrac{\\left(\\xa + \\ngapp \\xb\\right)}{x} \\gapp = \\gapp \\partialfrac{\\xa}{x} + \\partialfrac{\\xb}{x}\n",
        "$$\n",
        "\n",
        "2. Differentiating a product:\n",
        "\n",
        "$$\\partialfrac{\\left(\\xa \\times \\ngapp \\xb\\right)}{x} \\gapp = \\gapp \\xa \\times \\partialfrac{\\xb}{x} + \\xb \\times \\partialfrac{\\xa}{x}$$\n",
        "\n",
        "However, it's more common that *some* parts of an expression **do not** depend on $x$. Then you can apply simpler rules (which are special cases of the rules above). Again, $\\defineblock{\\xc}{\\color{gray} }$ and $\\defineblock{\\xd}{\\color{orange}}$ stand for *any* expression you like.\n",
        "\n",
        "3. Differentiating a sum when $\\xc$ **does not** depend on the variable $x$:\n",
        "\n",
        "$$\\partialfrac{\\left(\\xc + \\ngapp \\xd\\right)}{x} \\gapp = \\gapp \\partialfrac{\\xd}{x}$$\n",
        "\n",
        "4. Differentiating a product when $\\xc$ **does not** depend on the variable $x$:\n",
        "\n",
        "$$\\partialfrac{\\left(\\xc \\times \\ngapp \\xd\\right)}{x} \\gapp = \\gapp \\xc \\times \\partialfrac{\\xd}{x}$$\n",
        "\n",
        "5. Differentiating an expression $\\xc$ that **does not** depend on $x$ at all:\n",
        "\n",
        "$$\\partialfrac{\\xc}{x} = 0$$\n",
        "\n",
        "Lastly, there are two rules for sums $\\Sigma$ that are very useful:\n",
        "\n",
        "6. Factoring sum of products when $\\xc$ **does not** involve the summation index $i$:\n",
        "\n",
        "$$\\sum_i \\xc \\times \\ngapp \\xd \\gapp =\\gapp \\xc \\times \\sum_i \\xd$$\n",
        "\n",
        "7. Sum of products with a Kronecker delta:\n",
        "\n",
        "$$\\sum_i \\delta_{ij} \\times \\xd \\gapp = \\underbrace{\\hspace{1ex} \\xd \\hspace{1ex}  }_{\\textrm{$i$ replaced with $j$}}$$\n",
        "\n",
        "This last rule, and the meaning of the symbol $\\delta$, will be explained in the next section."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IsjG4FqK2Ge4",
        "colab_type": "text"
      },
      "source": [
        "#### The Kronecker delta symbol $\\delta_{ij}$\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S_3vllyB8cLq",
        "colab_type": "text"
      },
      "source": [
        "In the derivations that follow, we will be deriving expressions for the various vector-Jacobian products. A very common situation is that we will need to take the partial derivative of an indexed expression, like $b_i$ (which represents the $i$'th component of a vector $\\vec b$), with respect to something else. \n",
        "\n",
        "As you know, the symbol $i$ stands for an integer (this is always between 1 and some fixed integer that is the size of the vector or array, which is also left symbolic, like $n$ or $m$). \n",
        "\n",
        "Now, let's calculate the partial derivative of the component $b_i$ with respect to the component $b_j$. Here, $j$ is another symbolic integer. \n",
        "\n",
        "$$\\partialfrac{b_i}{b_j} = \\hspace{1ex}?$$\n",
        "\n",
        "Because $i$ and $j$ are *symbolic*, the value of this partial derivative depends on what values $i$ and $j$ take. Let's imagine the case where $i = 1$ and $j = 2$. Then the partial derivative $\\ipartialfrac{b_1}{b_2} = 0$, because $b_1$ and $b_2$ are independent parameters that have no effect on each other. But if $i = 3$ and $j = 3$, then the partial derivative $\\ipartialfrac{b_3}{b_3} = 1$, because the derivative of something with respect to itself is always 1!\n",
        "\n",
        "Therefore, what we need to fill in the \"?\" in the formula is a special symbol that depends on $i$ and $j$ that is equal to $1$ if $i$ is the same as $j$, and is equal to $0$ if $i$ is different from $j$. With this symbol we can keep computing symbolically without having to write out the different cases for $i$ and $j$.\n",
        "\n",
        "This symbol is very useful and is called the Kronecker delta. It is written $\\delta_{ij}$ (it can use any symbols, not just $i$ and $j$). \n",
        "\n",
        "Now we can write a symbolic result for the partial derivative from above:\n",
        "\n",
        "$$\\partialfrac{b_i}{b_j} = \\delta_{ij}$$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VjqgsWGw6sGV",
        "colab_type": "text"
      },
      "source": [
        "How does this really help us with our calculations? It turns out that there is at least one 'trick' that makes the Kronecker symbol become very useful, and that is in simplifying sums. Consider the sum: \n",
        "\n",
        "$$\\sum_{i \\le n} x_i \\delta_{ij} = x_1 \\delta_{1j} + x_2 \\delta_{2j} + \\dots + x_n \\delta_{nj}$$\n",
        "\n",
        "Because the Kronecker is zero except for the single term $x_i \\delta_{ij}$ in the sum for which $i = j$, we can replace the entire sum with the single case $x_j$. \n",
        "\n",
        "This gives us the rule for how sums $\\sum_i$ and $\\delta_{ij}$ interact: you can replace the sum with a single expression in which all $i$ have been replaced with $j$ (this rule obviously works for any two symbolic indices, not just $i$ and $j$).\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mWqZzugswL_Q",
        "colab_type": "text"
      },
      "source": [
        "## Deriving the gradients for Linear layer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uv_p7h7Evgtz",
        "colab_type": "text"
      },
      "source": [
        "This section explains how to derive the code that implements the gradients for the Linear layer."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ot0EfCDz6xXB",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "The important thing is to understand that simplifying the vector-Jacobian product mathematically tells us the *numpy* code that we can use to efficiently compute the values of $\\What$, $\\bhat$, and $\\xhat$, using the values of $\\W$, $\\b$, $\\x$, and $\\yhat$!\n",
        "\n",
        "We must now do a bit of calculus to compute the actual values to use above. But before we begin, remember that the quantities we are working with are arrays, which means we will often need to work in terms of the *components* of a particular gradient, e.g. $\\hat{b}_i$ or $\\hat{\\W}_{ij}$. After we understood what the gradient is, we will switch back to array operations that we can express using the _numpy_ library. \n",
        "\n",
        "Before we start on the calculus, we're going to recap some simple rules for differentiation that we will need along the way."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zRHUX7Kd6pqk",
        "colab_type": "text"
      },
      "source": [
        "### Deriving the bias gradient"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "qM0zWIQ68jrI"
      },
      "source": [
        "We're now ready to begin deriving gradients. To help us, let's remind ourselves of the fundamental equation for our layer, because we'll need to substitute this into our calculations:\n",
        "\n",
        "$$ y_i = b_i + \\sum_j W_{ij} \\hspace{1mm} x_j$$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XrQLKK_r9lKy",
        "colab_type": "text"
      },
      "source": [
        "Let's start with $\\bhat$, as this is simplest. Note that we switch from the hat notation to $\\partial$ notation at the beginning of the derivation, and switch back to hat notation at the end (the \":=\" means \"notation for\")."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8epzfRR9arKY",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "\\begin{align}\n",
        "\\hat{b}_i :=& \\gapp \\dld{b_i} \\\\\n",
        "=& \\sum_j {\\dld{y_j} \\partialfrac{y_j}{b_i} } \\because{vector-Jacobian product} \\\\\n",
        "=& \\sum_j {\\dld{y_j} \\partialfrac{\\left(  b_j  + \\sum_k W_{jk} \\cdot x_k \\right)}{b_i}} \\because{definition of $y_j$} \\\\\n",
        "=& \\sum_j {\\dld{y_j} \\partialfrac{b_j }{b_i}} \\because{$\\W$ and $\\x$ are independent of $\\b$} \\\\\n",
        "=& \\sum_j {\\dld{y_j} \\delta_{ij}}\\\\\n",
        "=& \\gapp \\dld{y_i} \\because{$\\delta_{ij}$ selects term with $j = i$} \\\\\n",
        ":=& \\gapp \\hat{y}_i \n",
        "\\end{align}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MCGk8vX0Ndit",
        "colab_type": "text"
      },
      "source": [
        "In other words, the gradient of the bias vector $\\mathbf b$ is simply the output of the linear layer, $\\mathbf y$. This is intuitive if you think about it: because the biases are added last, increasing each component of the bias directly increases the corresponding component of the output by the same amount."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RBep3kkinCLc",
        "colab_type": "text"
      },
      "source": [
        "**Exercise**: after reading the derivation above, try to recreate it from scratch without looking.\n",
        "\n",
        "Hint: It is very easy to make mistakes when doing calculus on arrays. The safest technique, which we are using here, is to analyze the result on a specific index, which lets us show that $\\bhat = \\yhat$ by showing that the components are equal ($\\hat{b}_i = \\hat{y}_i$). "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B5QZ6l6bquT7",
        "colab_type": "text"
      },
      "source": [
        "### Deriving the weights gradient"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "oG6CZJNp8cAj"
      },
      "source": [
        "The next calculation is for $\\hat W$. The first few steps are basically the same, so we'll focus on the part that is different:\n",
        "\n",
        "\n",
        "\\begin{align}\n",
        "\\hat{W}_{ij}\n",
        "=& \\sum_p {\\dld{y_p} \\partialfrac{\\left(  b_p  + \\sum_q W_{pq}x_q \\right)}{W_{ij}}} \\because{definition of $y_j$} \\\\\n",
        "=& \\sum_p {\\dld{y_p} \\sum_q \\partialfrac{W_{pq} }{W_{ij}} x_q  } \\because {$\\b$ is independent of $\\W$}\\\\\n",
        "=& \\sum_p {\\dld{y_p} \\sum_q \\delta_{(i,j)(p,q)} x_q } \\because{different components of $W$ are independent}\\\\\n",
        "=& \\sum_p {\\dld{y_p} \\sum_q \\delta_{ip} \\delta_{jq} x_q } \\because {$(i,j)=(p,q)$ implies $i=p$ and $j=q$} \\\\\n",
        "=& \\sum_p {\\dld{y_p} \\delta_{ip} x_j } \\because {$\\delta_{jq}$ selects term with $q=j$} \\\\\n",
        "=&  \\gapp {\\dld{y_i}  x_j } \\because{$\\delta_{ip}$ selects term with $p=i$}  \\\\\n",
        ":=& \\gapp \\hat{y}_i x_j \n",
        "\\end{align}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QnYYQshwfAVw",
        "colab_type": "text"
      },
      "source": [
        "The expression we've found for the components of the gradient matrix $\\hat{\\mathbf W}$ is in fact the components of an operation called the \"outer product\" of two vectors, which is written as follows:\n",
        "\n",
        "$$\\hat{\\mathbf W} = \\hat{\\mathbf y} \\otimes \\mathbf x$$\n",
        "\n",
        "In numpy this is implemented using the function `np.outer`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "74fXSCe07Zuk",
        "colab_type": "text"
      },
      "source": [
        "**Optional Exercise**: look up the definition of `np.outer` using Google and try to write a function that takes `y_hat` and `x` and returns `W_hat`. Test it on some random arrays using `np.random.rand` of the made-up shapes for `y_hat` and `x` and make sure it runs without an error and returns a `W_hat` array of the shape you expect. We'll see the correct definition later, so you can compare your answer with that definition."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JY_kH2tRAcJ6",
        "colab_type": "text"
      },
      "source": [
        "### Deriving the input gradient"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2ALkYeYiVgNZ",
        "colab_type": "text"
      },
      "source": [
        "Lastly we need to derive the expression for $\\hat{\\mathbf x}$. Again the start is pretty similar, so we'll start at the step of the derivation where things become different:\n",
        "\n",
        "\\begin{align}\n",
        "\\hat{x}_i =& \\sum_j {\\dld{y_j} \\partialfrac{\\left(  b_j  + \\sum_k W_{jk}x_k \\right)}{x_i}} \\because{definition of $y_j$} \\\\\n",
        "=& \\sum_j {\\dld{y_j} \\partialfrac{\\left(\\sum_k W_{jk}x_k \\right)}{x_i}} \\because {$\\b$ is independent of $\\x$} \\\\\n",
        "=& \\sum_j {\\dld{y_j} \\sum_k W_{jk}\\partialfrac{x_k}{x_i}} \\\\ \n",
        "=& \\sum_j \\dld{y_j} \\sum_k W_{jk} \\delta_{ki} \\\\\n",
        "=& \\sum_j \\dld{y_j} W_{ji} \\because {$\\delta_{ki}$ selects term with $k = i$}\\\\\n",
        ":=& \\sum_j \\hat{y}_j W_{ji} \n",
        "\\end{align}\n",
        "\n",
        "This expression gives the components of a matrix product, so $\\mathbf{\\hat{x}}$ can be writen as:\n",
        "\n",
        "$$\n",
        "\\mathbf{\\hat{x}} = \\mathbf{W}^T \\mathbf{\\hat{y}} \n",
        "$$\n",
        "\n",
        "Note that $\\mathbf{W}$ is transposed, which is a result of the indices in our component expression being swapped ($W_{ji}$ vs $W_{ij}$). We can implement this using the function `np.dot`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r3u0JNQ_7vM4",
        "colab_type": "text"
      },
      "source": [
        "**Optional Exercise**: look up the definition of `np.dot` and try to write a function that takes `y_hat` and `W` and returns `x_hat`. As before, test it on some random arrays to make sure it produces the right shape of array."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VGT-OYfqweuE",
        "colab_type": "text"
      },
      "source": [
        "## Deriving the gradient for the ReLU layer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SeWP50aqwiDs",
        "colab_type": "text"
      },
      "source": [
        "Recall that $y_i = \\mathrm{relu}(x_i)$, so \n",
        "\n",
        "\\begin{align}\n",
        "\\hat{x}_i =& \\sum_j {\\dld{y_j} \\partialfrac{\\left(\\relu{x_j}\\right)}{x_i}  } \\because{definition of $y_j$} \\\\\n",
        "=& \\sum_j {\\dld{y_j} \\partialfrac{\\left(\\relu{x_i}\\right)}{x_i} } \\gap \\delta_{ji} \\because {relu is elementwise}\\\\\n",
        "=& \\sum_j \\dld{y_j} \\step{x_i} \\gap \\delta_{ji} \\because {derivative of relu is step}\\\\\n",
        "=& \\gap \\dld{y_i} \\step{x_i} \\because {$\\delta_{ji}$ selects term with $j = i$}\\\\\n",
        ":=& \\gapp \\hat{y}_i \\gap \\step{x_i}\n",
        "\\end{align}\n",
        "\n",
        "We can write this as $\\xhat = \\yhat \\odot \\step \\x$, where $\\odot$ means the element-wise multiplication of the two vectors.\n",
        "\n",
        "Let's check if this makes sense in the two cases: \n",
        "\n",
        "* if $x_i<0$, changing $x_i$ has no effect on $y_i$, because $y_i=\\relu{x_i}=0$ is constant around the value of $x_i$. Therefore the gradient $\\hat{x}_i=0$. \n",
        "\n",
        "* if $x_i>0$, changing $x_i$ changes $y_i$ by the same amount. Therefore the gradient $\\hat{x}_i = \\hat{y}_i$. \n",
        "\n",
        "This is exactly what the elementwise multiplication with $\\step{\\x}$ achieves! Good!\n",
        "\n",
        "In `numpy` we can write the step function using the overloaded\\* array syntax `x > 0`, and we can write the element-wise multiplication $\\odot$ using the overloaded array syntax `x * y`. This overloaded syntax makes our definition of `relu` very easy, as you can see below.\n",
        "\n",
        "<small>* Note that the term \"overloaded\" means that `>` and `*` syntax have been defined to have a special meaning for numpy arrays. Python normally only defines these operators for single numbers.</small>"
      ]
    }
  ]
}