07. MLP Neural Networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MLP Neural Networks\n",
    "In this notebook we will build several different multi-layer perceptron models. Each time, we will add some common functionality to the model. We will start with building an easy to analyze system and build the overall complexity each time.  \n",
    "\n",
    "## Getting some data to test with:\n",
    "Let's start by grabbing some data for a (small) hand written digits dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1797, 64)\n",
      "(1797,)\n",
      "0.0 1.0\n",
      "[0 1 2 3 4 5 6 7 8 9]\n"
     ]
    }
   ],
   "source": [
    "# lets load up the handwritten digit dataset\n",
    "from sklearn.datasets import load_digits\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "import numpy as np\n",
    "\n",
    "ds = load_digits()\n",
    "X = ds.data/16.0\n",
    "y = ds.target\n",
    "\n",
    "print(X.shape)\n",
    "print(y.shape)\n",
    "print(np.min(X),np.max(X))\n",
    "print(np.unique(y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAk4AAAGGCAYAAACNCg6xAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAExBJREFUeJzt3V2IleXaB/B7XrRm3sAcDaKMiInQiYTSFLSiwAOxDwz6\n0NSoESoqiJoO6iAPKjsIUiKwgwrtQ4vqICVKpC0h4RiaEXgwhhVBSSX50Yc2fa73YHzZu32wva6n\n1vLx2b8fPHRy/71XXmtWf5bG1dVqtQoAAMf2P8f7BQAAnCgUJwCAIMUJACBIcQIACFKcAACCFCcA\ngCDFCQAgSHECAAgaEznU1dU1sZQyt5TyeSllpJ0viJTuUso5pZRNrVZrfyZoprVlps1jps1jps0T\nn2mr1TrmU0pZVEppeWr7LIrM0UxPqMdMm/eYafMeM23ec8yZhr5xKqPNuKxdu7b09/cHI7Tb8PBw\nWbJkSSlH55P0eSlmWjdm2jxm2jxm2jyZmUaL00gppfT395dp06ZVf2W0S5Wve8203sy0ecy0ecy0\neY45U385HAAgSHECAAhSnAAAghQnAIAgxQkAIEhxAgAIUpwAAIIUJwCAIMUJACBIcQIACFKcAACC\nFCcAgCDFCQAgSHECAAhSnAAAghQnAIAgxQkAIEhxAgAIUpwAAIIUJwCAIMUJACBIcQIACFKcAACC\nFCcAgCDFCQAgSHECAAhSnAAAghQnAIAgxQkAIEhxAgAIUpwAAIIUJwCAIMUJACBIcQIACBpzvF/A\nv9uzZ0868+yzz6bOz5gxI33H/Pnz05mTTjopnWHUk08+mTr//fffp+9YsmRJOtPX15fOMOqrr75K\nnX/44YfTd0yZMiWduffee9OZJnrvvffSmWeeeSZ1/owzzkjfsXz58nTGZ291v/zyS+r8Cy+8kL7j\ntttuS2fqpHbF6dtvv01n3nzzzdT5sWPHpu+45ppr0hmq27JlS+r8vn370nfMmzcvnVGcqsuW29de\ney19x+WXX57OKE6jPvvss3Rm7dq1qfNViu0jjzySzlDdH3/8kTq/bdu29B0nenHyR3UAAEGKEwBA\nkOIEABCkOAEABClOAABBihMAQJDiBAAQpDgBAAQpTgAAQYoTAECQ4gQAEFS7XXVLly5NZ3bv3p06\nf+DAgfQdPT096czQ0FA6M2vWrHSmiXp7e1Pn169fn75j06ZN6UyVBdFNtHfv3nQmu6cs+x4opZRd\nu3alM020YsWKdOa5555LZ956663U+auuuip9R5Udeueff346w6gNGzakzl988cVteiX15RsnAIAg\nxQkAIEhxAgAIUpwAAIIUJwCAIMUJACBIcQIACFKcAACCFCcAgCDFCQAgSHECAAhSnAAAgtq65PeL\nL75IZ7ILe0vJL+2tsjy0ymJgS35HVVkIW2Vpb1YTf687JbsItJRSZs+enTq/ePHi9B133313OtNE\nVZalV/m9u+iii1Lns4ueS7Gw968YGRlJZ5566qnU+UceeSR9x6FDh9KZKsaPH9+WX9c3TgAAQYoT\nAECQ4gQAEKQ4AQAEKU4AAEGKEwBAkOIEABCkOAEABClOAABBihMAQJDiBAAQ1NZddT/88EM6c8UV\nV6QzVXbPZc2cObPtd5wIXn311XTmzjvvTGcOHjyYzmRNnz697Xc0VZVdaJMnT06dv+GGG9J3DAwM\npDNNVOUzscrPXHa36I033pi+o8q+te7u7nSmiarslBweHk6dnzNnTvqO5cuXpzMTJkxIZ+666650\nJsI3TgAAQYoTAECQ4gQAEKQ4AQAEKU4AAEGKEwBAkOIEABCkOAEABClOAABBihMAQJDiBAAQpDgB\nAAS1dcnvd999l85cffXVbXglf92BAwfSmSpLCetuwYIF6cz8+fPTmZ6ennQm6/Dhw+nM+PHj2/BK\njq8qS1RXr16dzqxbty6dyXr66afbfkdTVVkM/NNPP6XOz5s3L31HlczGjRvTmbovBt6xY0c6s3Dh\nwnRmcHAwnclatmxZOvOPf/yjDa+kGt84AQAEKU4AAEGKEwBAkOIEABCkOAEABClOAABBihMAQJDi\nBAAQpDgBAAQpTgAAQYoTAECQ4gQAENTWJb+nnnpqOrN9+/Y2vJI/q7LUdGhoKJ259dZb0xk6Z/fu\n3enMpEmT2vBKjq8nnnginamypDOrymdB3Re1Nk3297vK8t377rsvnVm1alU6c//996cznTRu3Lh0\npsri5pUrV6bOv//+++k7qrjkkks6ck+Eb5wAAIIUJwCAIMUJACBIcQIACFKcAACCFCcAgCDFCQAg\nSHECAAhSnAAAghQnAIAgxQkAIKitu+rOOOOMdGbz5s3pzLZt21LnX3zxxfQdVdxyyy0duQf+ioGB\ngXSmys6x7L7HmTNnpu+o8u9y5513pjMzZsxIZ+puxYoV6cy8efNS57/77rv0Ha+//no6c8cdd6Qz\ndTd58uR05sCBA+nM3r17U+enTp2avmNwcDCdqdMeSt84AQAEKU4AAEGKEwBAkOIEABCkOAEABClO\nAABBihMAQJDiBAAQpDgBAAQpTgAAQYoTAECQ4gQAENTWJb+9vb3pTJUFvEuXLk2dv+KKK9J3vPvu\nu+kMo6osZ8wua12zZk36jrfffjudmTNnTjpTd5MmTUpntm7dms5kl4cuW7YsfUeV90FfX18608Ql\nv6eddlo6c91117XhlfxZlYW9jz32WBteyX+HU045JXX+4MGD6Ttuv/32dKZOfOMEABCkOAEABClO\nAABBihMAQJDiBAAQpDgBAAQpTgAAQYoTAECQ4gQAEKQ4AQAEKU4AAEFt3VVXxZlnnpnOXHPNNanz\n/f396TvorFmzZqXO//rrr+k7pk+fns5QXU9PT+p89j1QSrX3wQUXXJDONFGVnX3Zz94qLrzwwrbf\nwT+NHTs2dX7JkiXpO8aNG5fO1EntilOVHxI/WM1z2223tfU8nTdhwoTU+Soz9T6o7rLLLutIhnrL\nLvl96aWX2vRK6ssf1QEABClOAABBihMAQJDiBAAQpDgBAAQpTgAAQYoTAECQ4gQAEKQ4AQAEKU4A\nAEGKEwBAkOIEABCkOAEABClOAABBihMAQJDiBAAQpDgBAAQpTgAAQYoTAECQ4gQAEKQ4AQAEKU4A\nAEGKEwBAkOIEABCkOAEABClOAABBihMAQJDiBAAQpDgBAAQpTgAAQYoTAECQ4gQAEKQ4AQAEKU4A\nAEGKEwBAkOIEABA0Jniuu5RShoeH2/hSyPqXeXRXiJtpDZlp85hp85hp86Rm2mq1jvmUUhaVUlqe\n2j6LInM00xPqMdPmPWbavMdMm/ccc6bRb5w+P/rPxaWU3cEM7TellLKu/HM+Gf+fMdN6MdPmMdPm\nMdPmCc+062gD/s+Hurr+9+gvurvVah35q6+Ov8dfmYuZ1pOZNo+ZNo+ZNk9mLqHiBACA/6sOACBM\ncQIACFKcAACCFCcAgCDFCQAgSHECAAhSnAAAghQnAIAgxQkAIEhxAgAICi357erqmlhKmVtGl9+N\ntPMFkdJdSjmnlLKp1WrtzwTNtLbMtHnMtHnMtHniM221Wsd8SimLSiktT22fRZE5mukJ9Zhp8x4z\nbd5jps17jjnT0DdOZbQZl7Vr15b+/v5ghHYbHh4uS5YsKeXofJI+L8VM68ZMm8dMm8dMmycz02hx\nGimllP7+/jJt2rTqr4x2qfJ1r5nWm5k2j5k2j5k2zzFn6i+HAwAEKU4AAEGKEwBAkOIEABCkOAEA\nBClOAABBihMAQJDiBAAQpDgBAAQpTgAAQYoTAECQ4gQAEKQ4AQAEKU4AAEGKEwBAkOIEABCkOAEA\nBClOAABBihMAQJDiBAAQpDgBAAQpTgAAQYoTAECQ4gQAEKQ4AQAEKU4AAEGKEwBAkOIEABCkOAEA\nBClOAABBihMAQJDiBAAQpDgBAAQpTgAAQWOO9wv4d2+//XY6M27cuNT5Sy+9NH0HnZV9Hxw5ciR9\nx/XXX5/OMGrPnj3pzMjISOr8xx9/nL7j66+/TmdOPvnkdGZgYCB89rfffkv/+sfDjh070pnh4eHU\n+Ysuuih9R39/fzozZkzt/tN2XGzevDmd+fnnn1Pnr7zyyvQdJ7ravbueffbZdKavry91XnGqv+z7\nYN++fek7FKfqPvzww3Rm//79qfPr1q1L3zE0NJTO9Pb2pjM333xz+Ozvv/+e/vWPh02bNqUzzz//\nfOr84OBg+o7zzjsvnVGcRlX5GTp48GDq/H9jcfJHdQAAQYoTAECQ4gQAEKQ4AQAEKU4AAEGKEwBA\nkOIEABCkOAEABClOAABBihMAQJDiBAAQVLuFPrt27Upn1q9fnzq/cuXK9B3nnntuOvPJJ5+kM01U\nZXlodqarVq1K30FnTZw4MXV+9erV6Tsef/zxdCa7m6uUUrq7u8NnqywRPh527tzZ9juqfPa+8847\n6cwbb7yRztTdoUOH0pk1a9a04ZX8WVdXVzoze/bsdGbr1q3pTLv4xgkAIEhxAgAIUpwAAIIUJwCA\nIMUJACBIcQIACFKcAACCFCcAgCDFCQAgSHECAAhSnAAAghQnAICg2i35Pf3009OZTz/9NHW+t7c3\nfcf8+fPTmZGRkXQmszz0RHHvvfe2/Y4q86G6BQsWtP2Op59+Op35+OOP05nNmzenM000ffr0dKav\nry91fsWKFek7JkyYkM5UeR9Mnjw5nemkw4cPd+Sea6+9NnU++x4opZQNGzakM3XiGycAgCDFCQAg\nSHECAAhSnAAAghQnAIAgxQkAIEhxAgAIUpwAAIIUJwCAIMUJACBIcQIACFKcAACCarfkt8qixaGh\nodT5gwcPpu+YOXNmOtPEhb1VfPPNN+nM7NmzU+cnTZqUvoNRdV2M+9BDD7X9jlJK2bp1azozZ86c\nNryS42tgYCCdOeuss1LnP/vss/QdVZb8VlkWX3cTJ07syD2vvPJK6vxNN92UvuPAgQPpTJ34xgkA\nIEhxAgAIUpwAAIIUJwCAIMUJACBIcQIACFKcAACCFCcAgCDFCQAgSHECAAhSnAAAghQnAICg2i35\nXb16dTrzwAMPpM5/9NFH6TsWLlyYzlSxYMGCjtzTSVUWOk6dOjV1/tVXX03fMXfu3HRm/Pjx6Uzd\nVVmI+sEHH6Qz69evT2eytm3bls5UWSzeRD/++GPb76jyHqiylL2JP6dVlsZnl6WXUkpPT0/q/KOP\nPpq+Y8uWLenMoUOH0pl2vQ984wQAEKQ4AQAEKU4AAEGKEwBAkOIEABCkOAEABClOAABBihMAQJDi\nBAAQpDgBAAQpTgAAQbXbVVdFXXdN7dmz53i/hFro7+9PZ7I7rfbt25e+o8r+wS+//DKdmTRpUjrT\nSVX2OVXZKblmzZrU+e3bt6fvqOtnQaft3bs3nZkyZUo6s2rVqtT5Tz/9NH3HVVddlc689dZb6UwT\n99tt3bo1ncm+dzr1+TY4OJjOVPmcivCNEwBAkOIEABCkOAEABClOAABBihMAQJDiBAAQpDgBAAQp\nTgAAQYoTAECQ4gQAEKQ4AQAEKU4AAEG1W/K7Y8eOdGbcuHGp8w8++GD6jipuuOGGjtxTd/fcc086\nMzQ0lDpfZbnr8PBwOrNhw4Z05q677kpn6m758uXpTG9vb+r81KlT03cwauLEielMdj6llLJ06dLU\n+f3796fvOOuss9KZl19+OZ1p4s9pFdmlvVU+C1auXJnObNu2LZ1pF984AQAEKU4AAEGKEwBAkOIE\nABCkOAEABClOAABBihMAQJDiBAAQpDgBAAQpTgAAQYoTAECQ4gQAEFS7Jb+bNm1KZ5YtW9aGV/Jn\ng4OD6UyVxbNNNH/+/HTm0UcfTZ2vsjTy2muvTWeq/Ls00caNG9OZ7M92d3d3+g5GVfm9q/Lz0NPT\nkzpfZZHwwMBAOpNdPtxUVRbw7ty5M3V+37596Tt27dqVzmSXD7eTb5wAAIIUJwCAIMUJACBIcQIA\nCFKcAACCFCcAgCDFCQAgSHECAAhSnAAAghQnAIAgxQkAIKh2u+rmzp2bzowfP74Nr+TPLrzwwrbf\n0VRjxuTfZtn3wemnn56+o6+vL5059dRT05kmuu+++9KZs88+uw2vhL/L4sWL05lp06alzp988snp\nO6r8nFb5zGmiKv89zc70yJEj6TtO9M/R2r27ZsyY0ZEMnVPlQyw7U++Bzrr++uuP90vgbzZnzpyO\nZOgcn4vt4Y/qAACCFCcAgCDFCQAgSHECAAhSnAAAghQnAIAgxQkAIEhxAgAIUpwAAIIUJwCAIMUJ\nACBIcQIACFKcAACCFCcAgCDFCQAgSHECAAhSnAAAghQnAIAgxQkAIEhxAgAIUpwAAIIUJwCAIMUJ\nACBIcQIACFKcAACCFCcAgCDFCQAgSHECAAhSnAAAghQnAIAgxQkAIEhxAgAIUpwAAIIUJwCAIMUJ\nACBIcQIACBoTPNddSinDw8NtfClk/cs8uivEzbSGzLR5zLR5zLR5UjNttVrHfEopi0opLU9tn0WR\nOZrpCfWYafMeM23eY6bNe445066jg/yPurq6JpZS5pZSPi+ljBwzQKd0l1LOKaVsarVa+zNBM60t\nM20eM20eM22e8ExDxQkAAH85HAAgTHECAAhSnAAAghQnAIAgxQkAIEhxAgAIUpwAAIL+D6u49RLy\n4kAWAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10ba5a240>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "fig, ax = plt.subplots(nrows=2, ncols=5, sharex=True, sharey=True,)\n",
    "ax = ax.flatten()\n",
    "for i in range(10):\n",
    "    img = X[y == i][0].reshape(8, 8)\n",
    "    ax[i].imshow(img, cmap='Greys', interpolation='nearest')\n",
    "\n",
    "ax[0].set_xticks([])\n",
    "ax[0].set_yticks([])\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAk4AAAGGCAYAAACNCg6xAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3VFsVvd5P/Af/7DhKgQZiIxUokHDImyl1hoKVkqiMolN\nKIo0fJOGOFyEaGtVkk2REy1SFrJqZFEzDV9EkHTVhKuNBrpogmiqKhq4iLRyAU2yzksNAgOJSjWz\ngL2GaGZK9f4vmv9fueA8v5+PsXnfN5+PdK6+PMcnfnxePzl638fzGo1GAgAg7//c6AsAAGgVBicA\ngEIGJwCAQgYnAIBCBicAgEIGJwCAQgYnAIBCBicAgELzS/7RvHnzlqaUNqWUzqeUpmbzgpiWjpTS\nypTS4UajcWk6hXratPS0/ehp+9HT9lPe00ajkT1SSgMppYajaY+Bkj7qaUsdetp+h56236Gn7Xdk\ne1r0xCn9ZjJO+/btSz09PYUls+vv//7vw/xf/uVfKrOHH344rN28eXOYL1iwIMznyujoaNq6dWtK\nn/Rnms6n1Fw9HRoaCvMvfvGLldn+/fvD2nvuuSfM//iP/zjM58pnrae/+tWvKrNHHnkkrF25cmWN\nK5p77dbTH//4x7Vrc/fpv//7v4f5P//zP4f5XP1MtFtPv/Wtb4X5okWLKrMLFy6EtX/6p38a5s1y\nH0+np6WD01RKKfX09KQ1a9bUv7Lr6POf/3yYR8PN7/zO74S1d911V5h3dHSE+Q1Q53Fv0/V02bJl\nYf6FL3yhMlu4cGFYm/t5aZbvwad8Jnr627/925XZnXfeGdauXr261jXdQG3R09OnT9euzd2nOU34\nM9EWPb311lvDfPHixZXZRx99FNY2Yc9ysj315nAAgEIGJwCAQgYnAIBCBicAgEKlbw5vOm+99Vbt\n2twnfd54440wP3jwYO2vTbXoDYgppXT8+PHKrKurK6zN9fzxxx8P887OzjCnnkOHDlVmp06dmtG5\njx49GuZN+CGPlvDSSy/Vrv2rv/qrMN+9e3eY/9u//VuYN+EbjVvCxMREmEf34n333RfWdnd3h/n/\n/M//hHkz3qeeOAEAFDI4AQAUMjgBABQyOAEAFDI4AQAUMjgBABQyOAEAFGrZPU5f/vKXw/z222+v\nzHbt2hXWLlmyJMxz+2XsEqnngQceCPMXX3yxMov6nVJ+R5Q9TbNj1apVYd7f31+Z7d27N6z93d/9\n3TB/7733wtx9Ws/dd98d5l//+tcrs+9+97thbe4+3bx5c5hTz5YtW8I82t119uzZGX3tZtzTlOOJ\nEwBAIYMTAEAhgxMAQCGDEwBAIYMTAEAhgxMAQCGDEwBAoZbd47Rt27Ywv+222yqz3N6J3B6nZcuW\nhTn1rFixIsyHh4crs8HBwbD25ZdfrnVNzMz27dvD/J577qnMcvdhT09PmNvTNDtyu7m6u7srs9x9\nmtvdxezI7ceK9jyNj4+HtSdPnqx1Tc3MEycAgEIGJwCAQgYnAIBCBicAgEIGJwCAQgYnAIBCBicA\ngEItu8fpypUrtWsPHToU5hMTE2He2dlZ+2tT7aGHHgrz9evXV2Zf//rXw9qOjo5a10RsamoqzHN7\nnI4dO1b7a//kJz+pXUu1CxcuhPnQ0FDtc+/atat2LbPnL/7iL2rX5vYatuM+NU+cAAAKGZwAAAoZ\nnAAAChmcAAAKGZwAAAoZnAAACjXtOoLcR2K7u7vDfM+ePZXZ2NhYWHv//feH+Q9/+MMwt65gdhw9\nerQyy60yOHjw4PW+HArkVn/84he/qMyGh4fD2ieffDLMffS9ntdffz3MBwcHw/ynP/1pZXbixImw\ndt26dWHO7Mj1/MCBA7WylFL6wQ9+EOYPPvhgmDcjT5wAAAoZnAAAChmcAAAKGZwAAAoZnAAAChmc\nAAAKGZwAAAo17R6npUuXhvnixYvD/NFHH63MLl26FNbedtttYf7qq6+G+fbt28P8s2pqairMb7/9\n9tr1IyMjta6Jmeno6AjzJUuWhPnJkycrs6GhobB227ZtYU49GzduDPPvfve7YR7dx/Y43RinTp2a\nUf2XvvSlyiy3x+mll14Kc3ucAADamMEJAKCQwQkAoJDBCQCgkMEJAKCQwQkAoJDBCQCgUNPuccrt\nh+nv7w/zz33uc5VZbgdUbj9MtCOKarmeTkxMhPnatWsrs/3799e6JmbX4OBgmH/jG9+ozF555ZWw\ndvPmzbWuidiKFSvCPHefRvu3jhw5UuuamJlcT7/61a+G+f3331+Z9fb2hrV79+4N81bkiRMAQCGD\nEwBAIYMTAEAhgxMAQCGDEwBAIYMTAECh0nUEHSmlNDo6OouXMj0ffPBB7dqPP/54Rud+5513wnzB\nggXTvqY6PtWP+HP+19ZyPb169WpldvLkybD2pptuqnVNc63devr++++HedTTc+fOhbXNch/mtFpP\no56kNLPX3tOnT4d5blVMs/is9TSqn5ycDGvffffdMP/oo4/CfK5Mq6eNRiN7pJQGUkoNR9MeAyV9\n1NOWOvS0/Q49bb9DT9vvyPZ03ieNDM2bN29pSmlTSul8SmkqW8Bc6UgprUwpHW40GpemU6inTUtP\n24+eth89bT/FPS0anAAA8OZwAIBiBicAgEIGJwCAQgYnAIBCBicAgEIGJwCAQgYnAIBCBicAgEIG\nJwCAQgYnAIBCBicAgELzS/6RP0rYtPyhyfajp+1HT9uPnraf8p42Go3skVIaSCk1HE17DJT0UU9b\n6tDT9jv0tP0OPW2/I9vToidO6TeTcdq3b1/q6ekpLJld58+fD/Pjx49XZi+++GJY+w//8A9hfued\nd4b5XBkdHU1bt25N6ZP+TNP5lOa2pxcvXgzz//qv/wrzZ555pjJbsmRJWPvNb34zzPv6+sJ8rrRa\nT69evRrm//iP/xjmr7zySmV2yy23hLV/9Ed/FOaDg4NhPldarac5udfeI0eOVGY/+clPwtqnnnoq\nzL32zo7ca3PU87/+678Oa1944YUwb8Welg5OUyml1NPTk9asWVP/yq6jm2++Ocz/8z//s/a5u7u7\nw7xZvgefUudx75z39MKFC2H+y1/+MswXLFhQmS1cuDCsveOOO8JcT+uZmoov8+jRo7XPPX9+/PK0\nbNmyMNfT2ZF77f35z39emeXuU6+9N0butfm3fuu3KrPodTml9uypN4cDABQyOAEAFDI4AQAUMjgB\nABQqfXN403nttdfCfMeOHZXZnj17wtrokz4ppbRu3bow59qGh4fDPOpZSin19/dXZrfffntYu3v3\n7jDfuHFjmHNtuU9J5Xoa3YtLly4Na7ds2RLmuU/VLV++PMw/q3Jv+M+92TcS3cMpee29Uf71X/81\nzHOfSo488cQTYZ57DWlGnjgBABQyOAEAFDI4AQAUMjgBABQyOAEAFDI4AQAUMjgBABRq2T1OmzZt\nCvOJiYnKbPv27WHtPffcE+aTk5Nh3tnZGeafVV/5yldmVB/tYnr88cfD2sWLF8/oa3Ntuf1X27Zt\nC/OhoaHKbGxsbEbnzu2B4to6OjrCPPd9X7t2bWWW28PU19cX5nv37g1z6rn33nvDfMmSJZVZ7o9t\nd3V11bqmZuaJEwBAIYMTAEAhgxMAQCGDEwBAIYMTAEAhgxMAQCGDEwBAoZbd45TbB/LEE09UZqdO\nnQprR0dHa10Tsdx+rPXr14d5tPPn0KFDYe2RI0fCnHqmpqbCPHevRbuacj3L7ZBiduR26B0/frwy\n+/73vx/WDg4O1romZmb58uVhHt2nuT1O+/fvr3VNzcwTJwCAQgYnAIBCBicAgEIGJwCAQgYnAIBC\nBicAgEIGJwCAQi27xynnvvvuq8y6u7vD2j179oR5Z2dnrWv6rOvo6Ajzu+++O8yjPU65/S92/syO\nvXv3hvlMdqLl7lNujC996UthvmXLlsost6tt165dta6JmZmcnKxd+/DDD4d57nW/FXniBABQyOAE\nAFDI4AQAUMjgBABQyOAEAFDI4AQAUKhl1xG8/PLLYR59dH3VqlVh7bp162pdE7GpqakwHx4enqMr\n4XrJ3StLliwJ823btlVmO3bsCGtzqxCYHY8++miYL168uDLr6uq63pfDdbBz587aefS7NqWUtm/f\nXuuampknTgAAhQxOAACFDE4AAIUMTgAAhQxOAACFDE4AAIUMTgAAhVp2j9Njjz0W5sePH6/MHnro\nobB20aJFta6JWEdHR5hPTEyEeX9/f2WW2yWya9euMKee3B6ny5cvh3lfX19ltmXLlrA2t3tm+fLl\nYU49q1evDvP77ruvMvve974X1p44cSLM7dibHWfPng3zb3/725VZ7rV3cnIyzDs7O8O8GXniBABQ\nyOAEAFDI4AQAUMjgBABQyOAEAFDI4AQAUMjgBABQqGX3OB05ciTMo/0w0T6glFJasWJFrWtiZn7x\ni1+EeW9vb+1z2w9zY+TutWhX0549e8Jae5pujL1794b5k08+WZmNjY2Fte7TGyPa05RSSl/5ylcq\nsyVLloS1rbinKccTJwCAQgYnAIBCBicAgEIGJwCAQgYnAIBCBicAgEKl6wg6UkppdHR0Fi9lek6f\nPl27dnJyMszfeeedMF+wYEHtr309faofHTXKm66nFy9eDPOPP/649rlPnjwZ5jfddFPtc19P7dbT\nDz74oHbt+++/H+Zvv/127XPPpXbrac74+Hjt2lbpebv19Pz582EevfZevXo1rG2WnuVMq6eNRiN7\npJQGUkoNR9MeAyV91NOWOvS0/Q49bb9DT9vvyPZ03ieNDM2bN29pSmlTSul8SmkqW8Bc6UgprUwp\nHW40GpemU6inTUtP24+eth89bT/FPS0anAAA8OZwAIBiBicAgEIGJwCAQgYnAIBCBicAgEIGJwCA\nQgYnAIBCBicAgEIGJwCAQgYnAIBCBicAgELzS/6RP0rYtPyhyfajp+1HT9uPnraf8p42Go3skVIa\nSCk1HE17DJT0UU9b6tDT9jv0tP0OPW2/I9vToidO6TeTcdq3b1/q6ekpLJld7777bpg/88wzldl/\n//d/h7Xf+973wnzlypVhPldGR0fT1q1bU/qkP9N0PqXW6unNN99cmR05ciSs/dWvfhXmg4ODYT5X\n2q2nP/7xj8N8//79ldm5c+fC2j179oT5nXfeGeZzpd16mrtP9+7dW5ldvnw5rH3ppZfC/JZbbgnz\nudJuPb148WKYP/3007XP/eKLL4Z5V1dX7XNfT9PpaengNJVSSj09PWnNmjX1r+w6+vWvfx3mCxYs\nqMzmz4//s3MvuKtXrw7zG6DO496W6+miRYsqs5///Odh7ec+97kwb5bvwae0RU9Pnz4d5gsXLqzM\ncvdpd3d3mDfL9+BT2qKnufu0s7OzMvvf//3fsPb3fu/3ap/7BmmLnl64cCHMo/s0p7e3N8yXL19e\n+9yzJNtTbw4HAChkcAIAKGRwAgAoZHACAChU+ubwpnPixIkwHxsbq33u+++/P8zPnDlT+9xUO3Dg\nQJivWrWqMjt79mxYOzw8HOY7duwI8yZ8U2pLuHQpXody7Nix2ud+4YUXwvzgwYO1z/1ZlnujcF9f\nX5hH92nudTn32nv06NEw7+joCPPPqsnJyTDfsGFDmOfe4B3JvbZGn8JsVp44AQAUMjgBABQyOAEA\nFDI4AQAUMjgBABQyOAEAFDI4AQAUatk9TuvWrZu1cz/yyCOzdm6qRftfUkrpjTfeqMxGRkZmdG77\nX2bH0qVLa9cuXrw4zHfv3l373FS7cuXKjOqj18/cTp+urq4ZfW2ubXx8PMxz+7UGBwcrs9zv4k2b\nNoX5zp07w7wJ/wiwJ04AAKUMTgAAhQxOAACFDE4AAIUMTgAAhQxOAACFDE4AAIXado9TtANmYmIi\nrH3qqadqXRMzk+vps88+W5m98sorYe2DDz5Y65qYmVtvvbV27eHDh8O8Gfe7tIMVK1aEeW7vTrSr\nKbeba//+/WFu31o9uZ6uX78+zIeGhiqz3A6oAwcOhHkr3seeOAEAFDI4AQAUMjgBABQyOAEAFDI4\nAQAUMjgBABQyOAEAFGrZPU5TU1NhHu1qyu2ssCvkxnjooYfCfMOGDZWZPU3NadGiRbVr77jjjut4\nJVwvP/rRj2rX5nbojYyMhHlu1xvXlvudtnr16jA/duxYZZb7fbp58+Ywb0WeOAEAFDI4AQAUMjgB\nABQyOAEAFDI4AQAUMjgBABRq2XUE27dvD/PFixdXZuPj42HtiRMnwtxHYuvJrZDIuXjxYmV24cKF\nsHb58uUz+trUk7uXIm+99VaYb9y4sfa5qXbp0qUwjz6anlJKx48fr8xeeOGFsLavry/MG41GmHNt\nudfe4eHhMO/v76/MotfllFL64he/GOb/8R//EebNuB7IEycAgEIGJwCAQgYnAIBCBicAgEIGJwCA\nQgYnAIBCBicAgEItu8dpYmIizDds2FCZ/eEf/mFYu2nTpjD/5S9/GebNuHeiGeS+L729vWEe7QvJ\n9cQepxvjjTfeCPNoP8xzzz0X1trjdGNEO/JSSuns2bOV2cjISFgb/TxQX+61d/369WEevfbed999\nYe2OHTvC/PXXXw/zBx98MMxvBE+cAAAKGZwAAAoZnAAAChmcAAAKGZwAAAoZnAAAChmcAAAKtewe\np2eeeSbM+/r6KrNDhw7N6GtPTU2FuT1O9WzZsmVGOc0nd58+8cQTldndd999vS+HArmdZ6+99lqY\n/8Ef/EFlltsXtHv37jBndvzTP/1TmH/ta1+rzHJ7mgYHB8N88+bNYd6MPHECAChkcAIAKGRwAgAo\nZHACAChkcAIAKGRwAgAoVLqOoCOllEZHR2fxUqbn5MmTN+xr/+xnPwvzW265ZU6u41P9qLP/oOl6\neu7cudq1uZ+Hm266qfa551K79TTXlytXrlRm4+PjYe3bb79d65rmWrv19PTp07Vro36nlNLIyEiY\n534m5kq79fTixYthnutbJNezd955J8wXLFhQ+2tPx7R62mg0skdKaSCl1HA07TFQ0kc9balDT9vv\n0NP2O/S0/Y5sT+d90sjQvHnzlqaUNqWUzqeU4u2PzKWOlNLKlNLhRqNxaTqFetq09LT96Gn70dP2\nU9zTosEJAABvDgcAKGZwAgAoZHACAChkcAIAKGRwAgAoZHACAChkcAIAKGRwAgAoZHACAChkcAIA\nKGRwAgAoNL/kH/mjhE3LH5psP3rafvS0/ehp+ynvaaPRyB4ppYGUUsPRtMdASR/1tKUOPW2/Q0/b\n79DT9juyPS164pR+Mxmnffv2pZ6ensKS2fXhhx+G+Z/92Z/VPvd3vvOdMF+wYEHtc19Po6OjaevW\nrSl90p9pOp9Sc/X0+PHjYf7nf/7nldkXvvCFsPbFF18M866urjCfK+3W05ncp3/5l38Z1q5cubLO\nJc25duvpu+++G+aPPfZYZeY+Tf+/ppl6OjQ0FOYjIyOV2fDw8PW+nBtiOj0tHZymUkqpp6cnrVmz\npv6VXUeTk5NhvnDhwtrnvuuuu8K8o6Oj9rlnSZ3HvU3X04mJiTCfP7/6xzXX797e3jBfvnx5mN8A\nbdHTmdynd955Z1i7evXqWtd0A7VFT3/961+Hufu0rKaZerps2bIwP3fuXGXWLP8N11G2p94cDgBQ\nyOAEAFDI4AQAUMjgBABQqPTN4U3n1VdfDfNjx45VZqtWrQpr33vvvTBvwTeltoTnnnsuzJcsWVKZ\njY6OhrWPP/54mB88eDDMqSd3n46Pj1dmuU9Y7d27t9Y1MTMvvPBC7drcp+I2bNgQ5mfOnKn9tan2\n+uuvh/nly5crs0cffTSsbcf71BMnAIBCBicAgEIGJwCAQgYnAIBCBicAgEIGJwCAQgYnAIBCLbvH\nad26dbVrN2/eHOZHjx6tfe6U7HmqK9rpk1JKX/3qVyuzU6dOhbVvvvlmmJ84cSLMZ/Lz9lk2NjYW\n5n/3d39XmX3jG98Ia3M9u+OOO8K8s7MzzKkn2sU0MjIyh1dCqdwfV/7yl79cme3YsSOsXbt2bZgP\nDAyEeTPep544AQAUMjgBABQyOAEAFDI4AQAUMjgBABQyOAEAFDI4AQAUatk9Trm9E5GhoaEwHxwc\nDPPc3gnqeeSRR8I82heyePHisPb5558Pc3uaZsdMdrzk7vGHHnoozM+cORPm1LN79+4wv+222yqz\nVatWhbW5fWvMjmeeeSbMc/da5Kc//WmYP/roo7XPfaN44gQAUMjgBABQyOAEAFDI4AQAUMjgBABQ\nyOAEAFDI4AQAUKhp9zhNTk6G+e233x7m69evr8y6urrC2l27doU59eR6mtv5E5mYmAhzu7dujI6O\njjC/fPlyZXbx4sWwNrcfhtkxPDwc5tGupqjf3Dif//znw3xsbKwyy+09bMffp544AQAUMjgBABQy\nOAEAFDI4AQAUMjgBABQyOAEAFGradQSnT58O8w0bNoT57t27K7Pe3t6wdmpqKsxzH7Hm2l599dUZ\n1UcrB9auXRvW7ty5M8zb8SOzzWD79u1h3tPTU5mNj4+Hte7D2ZF7/RsaGgrzw4cPV2YnTpwIax9/\n/PEw379/f5j7magn933ftm1bZZb7eWjH11ZPnAAAChmcAAAKGZwAAAoZnAAAChmcAAAKGZwAAAoZ\nnAAACjXtHqd169aF+cjISO1zR7tjUkrpvffeC/PVq1fX/tqfZbme5kS7uXKiHVDMnlOnToX5ww8/\nXJmNjY2Fte7T2ZHbhZS7lxYtWlSZbdy4Max97LHHwlzP68nt5jp06FCYHzhwoDIbHh4Oa3O7u2b6\ne+FG8MQJAKCQwQkAoJDBCQCgkMEJAKCQwQkAoJDBCQCgkMEJAKBQ0+5xynnzzTfDvLe3tzLL7XFa\nsWJFrWsiltvXsWfPnjAfGhqqzJYtW1a7ltnzwx/+MMwHBwcrs9x+mL6+vjC302d2HD9+PMy7u7tr\nn7u/vz/Mc/c515bbzbVt27Yw37JlS2W2fv36sDb6XdyqPHECAChkcAIAKGRwAgAoZHACAChkcAIA\nKGRwAgAoVLqOoCOllEZHR2fxUqbn4sWLYf7xxx9XZleuXAlr33nnnTBfsGBBmM+VT/Uj/qzptTVd\nT99///0wv3r1amWW6+nPfvazML/lllvCfK60W08//PDDMP/ggw9qn/vcuXNh/vbbb9c+9/XUbj09\nefLkrJ17cnIyzJvlPm63ns7kPvxM/j5tNBrZI6U0kFJqOJr2GCjpo5621KGn7Xfoafsdetp+R7an\n8z5pZGjevHlLU0qbUkrnU0pT2QLmSkdKaWVK6XCj0bg0nUI9bVp62n70tP3oafsp7mnR4AQAgDeH\nAwAUMzgBABQyOAEAFDI4AQAUMjgBABQyOAEAFDI4AQAUMjgBABQyOAEAFDI4AQAUMjgBABSaX/KP\n/FHCpuUPTbYfPW0/etp+9LT9lPe00Whkj5TSQEqp4WjaY6Ckj3raUoeett+hp+136Gn7HdmeFj1x\nSr+ZjNO+fftST09PYcmNNTQ0VJlduHAhrH366afDvKurq9Y1XW+jo6Np69atKX3Sn2k6n1Jz9fTD\nDz8M829961u1z71r167atXOp3Xp68eLFMP+TP/mTymzJkiVh7Xe+850wX7BgQZjPlXbrae4+/f3f\n//3K7OGHHw5rH3vssTDX0xvjySefrMzeeuutsPZv/uZvwryvr6/WNV1v0+lp6eA0lVJKPT09ac2a\nNfWvbA4tW7asMvvoo4/C2t7e3jBfvnx5rWuaRXUe9zZdTycnJ8O8s7Oz9rmb5b9xGtqip7n/SYl+\nES5cuDCsveuuu8K8o6MjzG+Atuhp7j6NRK/LKelps4pee+fPj8eIO+64I8yb8HuQ7ak3hwMAFDI4\nAQAUMjgBABQyOAEAFCp9c3jTefnll8M8+lTd4sWLw9oNGzaE+ZkzZ8Kceg4fPhzmhw4dqsxyPX3+\n+efD/KmnngrzJnxTaksYHh4O87GxsVpZSin97d/+bZg/++yzYU49ufs0Er0up5TS2bNnw/zgwYO1\nvzbVTpw4EebRa+/69evD2gceeCDML1++HObNyBMnAIBCBicAgEIGJwCAQgYnAIBCBicAgEIGJwCA\nQgYnAIBCLbvHad26dbVrt23bFua5XSNTU/HfALTzp55bb701zFetWlWZ5f54aK6nuZ+JJvzDzi1h\nyZIltWtnep/azTU77r333lk795tvvjlr56Za7g/xRo4dOxbm/f39Yd6Kv089cQIAKGRwAgAoZHAC\nAChkcAIAKGRwAgAoZHACAChkcAIAKNSye5xmIrf/ZXBwcI6uhE+75557wnxsbKxWllJKJ0+eDHN7\nmmbHwMBAmEf34vDwcFi7ePHiMG/F/TCt4Oabb65dG+1iSymlM2fO1D439XV2doZ5tFMtd5+OjIzU\nuqZm5okTAEAhgxMAQCGDEwBAIYMTAEAhgxMAQCGDEwBAIYMTAEChpt3jlNvB0tfXF+b9/f2V2aFD\nh8La3K4R+19mx+uvv167Ntez1atX1z439b311lthntu/FXnttdfCPLebhnpeffXVMI/2a+X6ffTo\n0TDfuHFjmFPPD37wgzCPdjXl9qlt3rw5zFvx96knTgAAhQxOAACFDE4AAIUMTgAAhQxOAACFDE4A\nAIWadh3BTO3evbsy+/a3vx3Wdnd3h3nuI7E++n5tuRUT3/zmN8N8586dldmOHTvC2hMnToT5unXr\nwpx6HnjggTDftm1bZZb7mHN0j6fko+uz5fvf/36YP//885XZ5cuXw9rnnnsuzPW0ntxr75YtW8J8\nz549ldnAwEBYu3bt2hl97WZ8bfbECQCgkMEJAKCQwQkAoJDBCQCgkMEJAKCQwQkAoJDBCQCgUNPu\ncero6Ajz/v7+MI/2+jz99NO1run/ee2118L82WefndH521Vul0hPT0+Y/+hHP6r9tQ8cOBDmzbgr\npBXkejoxMVH73LnaN998s/a5qa+rqyvMx8bGKrO+vr6wNrePjXpm+vs06un4+HhY29vbG+Znz54N\n82Z8bfbECQCgkMEJAKCQwQkAoJDBCQCgkMEJAKCQwQkAoJDBCQCgUNPuccoZHh4O8507d1Zm999/\nf1i7fv0+gSmnAAABjUlEQVT6MH/ggQfCnGvr7OwM86NHj4b59u3bK7PcbpktW7aEOfXk9sMcP348\nzJ944onKbHR0NKx95ZVXwpzZsXv37jD/2te+VpnlXrcHBwdrXRMzM5O+dHd3h7W536f33ntvmDcj\nT5wAAAoZnAAAChmcAAAKGZwAAAoZnAAAChmcAAAKla4j6Egp//HgufThhx+G+fj4eGV29erVsPbK\nlSth/u6774b5Rx99FObXy6f6EX8m/Nqarqe5vnzwwQeVWe7n4eTJk2F+0003hflcabee5r7v0b32\n8ccfh7Xnzp0L87fffjvM50q79fTixYthPpOeRq/bKenpbMm9fkavvTm536cjIyNhnvuZuF6m1dNG\no5E9UkoDKaWGo2mPgZI+6mlLHXrafoeett+hp+13ZHs675NGhubNm7c0pbQppXQ+pTSVLWCudKSU\nVqaUDjcajUvTKdTTpqWn7UdP24+etp/inhYNTgAAeHM4AEAxgxMAQCGDEwBAIYMTAEAhgxMAQCGD\nEwBAIYMTAECh/wuKMUFkIAph0wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10ba5a198>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(nrows=5, ncols=5, sharex=True, sharey=True,)\n",
    "ax = ax.flatten()\n",
    "digit = 8\n",
    "x_digits = X[y == digit]\n",
    "for i in range(25):\n",
    "    img = x_digits[i].reshape(8, 8)\n",
    "    ax[i].imshow(img, cmap='Greys', interpolation='nearest')\n",
    "\n",
    "ax[0].set_xticks([])\n",
    "ax[0].set_yticks([])\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1437, 64)\n",
      "(1437,)\n",
      "(360, 64)\n",
      "(360,)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X,y,test_size = 0.2)\n",
    "print(X_train.shape)\n",
    "print(y_train.shape)\n",
    "print(X_test.shape)\n",
    "print(y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_____\n",
    "# Single Layer Neural Networks are just Linear and Logistic Regression\n",
    "The single layer network can be summarized with the notation given by the following representation:\n",
    "![SingleLayerNetwork](PDF_Slides/SingleLayerNetwork.png)\n",
    "\n",
    "When we calculated objective functions for linear regression and logistic regression we used the maximum likelihood priciple, showing that (under Gaussian noise) this led to minimizing the squared error of the output.\n",
    "\n",
    "This is identical to the objective function for generic, single layer neural networks:\n",
    "$$ \\sum_i^M (\\mathbf{y}^{(i)}-\\mathbf{\\hat{y}}^{(i)})^2 $$\n",
    "\n",
    "Here, the $\\mathbf{y}^{(i)}$ is a one-hot-encoded representation of the class for the $i^{th}$ instance.\n",
    "Adding in terms of single layer neural network explicitly:\n",
    "$$ J(\\mathbf{W}) = \\sum_i^M (\\mathbf{y}^{(i)}-\\phi(\\mathbf{W}\\cdot \\mathbf{x}^{(i)}))^2 $$\n",
    "\n",
    "Adaline network objective function:\n",
    "$$ J(\\mathbf{W}) = \\sum_i^M (\\mathbf{y}^{(i)}-\\mathbf{W}\\cdot \\mathbf{x}^{(i)})^2 $$\n",
    "\n",
    "To solve, we need the gradient:\n",
    "$$  \\nabla J(\\mathbf{W}) $$\n",
    "\n",
    "And then we can use the Widrow-Hoff learning Rule:\n",
    "$$ \\mathbf{W}\\leftarrow\\mathbf{W} +\\eta\\nabla J(\\mathbf{W}) $$\n",
    "\n",
    "Which can be separated for each row (they are independent):\n",
    "$$ \\mathbf{w}\\leftarrow\\mathbf{w} +\\eta[ \\mathbf{X}*(\\mathbf{y}-\\mathbf{\\hat{y}})] $$\n",
    "\n",
    "$$ \\mathbf{w}_{row}\\leftarrow\\mathbf{w}_{row} +\\eta[ \\mathbf{X}*(\\mathbf{y}_{row}-\\mathbf{\\hat{y}}_{row})] $$\n",
    "\n",
    "The same is true for the update equation of the perceptron with a sigmoid activation function:\n",
    "$$ \\mathbf{w}\\leftarrow\\mathbf{w} +\\eta[ \\mathbf{X}*(\\mathbf{y}-\\mathbf{g(x)})] $$\n",
    "\n",
    "$$ \\mathbf{w}_{row}\\leftarrow\\mathbf{w}_{row} +\\eta[ \\mathbf{X}*(\\mathbf{y}_{row}-\\mathbf{g(x)}_{row})] $$\n",
    "\n",
    "This means that the update equations are identical to the case of linear and logistic regression, using one-versus-all! So all of this we have already looked at and solved! Yay!! But, what about when we add more layers?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multi-layer Case\n",
    "In the multilayer perceptron, we need to adjust some notation:\n",
    "![MultiLayerNetwork](PDF_Slides/MultiLayerNetwork.png)\n",
    "\n",
    "### Formulation of multi-layer perceptron\n",
    "Our objective function becomes:\n",
    "$$ J(\\mathbf{W}) = \\sum_k^M (\\mathbf{y}^{(k)}-\\mathbf{a}^{(L)})^2 $$\n",
    "\n",
    "And the update equation becomes:\n",
    "$$  w_{i,j}^{(l)} \\leftarrow w_{i,j}^{(l)} - \\eta \\frac{\\partial J(\\mathbf{W})}{\\partial w_{i,j}^{(l)}}$$\n",
    "\n",
    "However, this is not tractable to compute by hand for each network architecture, so we expand the gradient calculation using the chain rule to get partial derivatives:\n",
    "$$  \\frac{\\partial J(\\mathbf{W})}{\\partial w_{i,j}^{(l)}} = \\frac{\\partial J(\\mathbf{W})}{\\partial\\mathbf{z}^{(l)}} \\frac{\\partial\\mathbf{z}^{(l)}}{\\partial w_{i,j}^{(l)}}   $$\n",
    "\n",
    "The second term in the partial derivatives is easy to calculate as $a_j^{(l)}$ (see lecture notes), such that:\n",
    "$$  \\frac{\\partial J(\\mathbf{W})}{\\partial w_{i,j}^{(l)}} = \\frac{\\partial J(\\mathbf{W})}{\\partial\\mathbf{z}^{(l)}} a_j^{(l)}   $$\n",
    "\n",
    "And our update equation would be:\n",
    "$$  w_{i,j}^{(l)} \\leftarrow w_{i,j}^{(l)} - \\eta \\frac{\\partial J(\\mathbf{W})}{\\partial\\mathbf{z}^{(l)}} a_j^{(l)}$$\n",
    "\n",
    "To calculate the other term, we need to continue to use the chain rule:\n",
    "$$ \\frac{\\partial J(\\mathbf{W})}{\\partial\\mathbf{z}^{(l)}} =  \\frac{\\partial J(\\mathbf{W})}{\\partial\\mathbf{z}^{(l+1)}} \\frac{\\partial \\mathbf{z}^{(l+1)} }{\\partial\\mathbf{z}^{(l)}} $$\n",
    "\n",
    "After some derivation, we can see that: \n",
    "$$ \\frac{\\partial \\mathbf{z}^{(l+1)} }{\\partial\\mathbf{z}^{(l)}} = \\text{diag}[\\mathbf{a}^{(l+1)}*(1-\\mathbf{a}^{(l+1)})] \\cdot\\mathbf{W}^{(l+1)}  $$\n",
    "\n",
    "and\n",
    "$$ \\frac{\\partial J(\\mathbf{W})}{\\partial\\mathbf{z}^{(l)}} = \\text{diag}[\\mathbf{a}^{(l+1)}*(1-\\mathbf{a}^{(l+1)})] \\cdot\\mathbf{W}^{(l+1)} \\frac{\\partial J(\\mathbf{W})}{\\partial\\mathbf{z}^{(l+1)}} $$\n",
    "\n",
    "This set up a recurrence relation such that if we know $\\frac{\\partial J(\\mathbf{W})}{\\partial\\mathbf{z}^{(l)}}$ for the final layer, we can use it to get the derivatives for the previous layers. For a two layer network, the final layer gradient can be formulated as:\n",
    "$$  \\frac{\\partial J(\\mathbf{W})}{\\partial\\mathbf{z}^{(2)}} = \\frac{\\partial}{\\partial\\mathbf{z}^{(2)}} (\\mathbf{y}^{(k)}-\\phi(\\mathbf{z}^{(2)}))^2 $$\n",
    "\n",
    "$$  \\frac{\\partial J(\\mathbf{W})}{\\partial\\mathbf{z}^{(2)}}  = -2(\\mathbf{y}^{(k)}-\\mathbf{a}^{(3)})*\\mathbf{a}^{(3)}*(1-\\mathbf{a}^{(3)}) $$\n",
    "\n",
    "where $\\mathbf{y}^{(k)}$ is the $k^{th}$ instance and $\\mathbf{a}^{(3)}$ is the feedforward result for the $k^{th}$ instance. We can make this more explicit by writing the $k^{th}$ instance of $\\mathbf{a}^{(3)}$ as $_k\\mathbf{a}^{(3)})$. Now the update becomes:\n",
    "\n",
    "$$  \\frac{\\partial J(\\mathbf{W})}{\\partial _k\\mathbf{z}^{(2)}}  = -2(\\mathbf{y}^{(k)}-{_k\\mathbf{a}^{(3)}})*{_k\\mathbf{a}^{(3)}}*(1-{_k\\mathbf{a}^{(3)}}) $$\n",
    "\n",
    "Also recall that $\\cdot$ represent matrix multiplication and $*$ represents elementwise or row-wise multiplication. The update equation can be given by:\n",
    "\n",
    "$$ \\mathbf{W}^{(l)} \\leftarrow \\mathbf{W}^{(l)} -\\eta \\frac{\\partial J(\\mathbf{W}^{(l)})}{\\partial _k\\mathbf{z}^{(l)}} \\cdot {_k\\mathbf{a}^{(l)}} $$\n",
    "\n",
    "For single elements stochastic gradient descent. Or can be given by the following for batch gradient descent:\n",
    "\n",
    "$$ \\mathbf{W}^{(l)} \\leftarrow \\mathbf{W}^{(l)} -\\eta \\sum_k\\frac{\\partial J(\\mathbf{W}^{(l)})}{\\partial {_k\\mathbf{z}^{(l)}}} \\cdot {_k\\mathbf{a}^{(l)}} $$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Programming a Two Layer Perceptron\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Example adapted from https://github.com/rasbt/python-machine-learning-book/blob/master/code/ch12/ch12.ipynb\n",
    "# Original Author: Sebastian Raschka\n",
    "# This is the optional book we use in the course, excellent intuitions and straightforward programming examples\n",
    "# please note, however, that this code has been manipulated to reflect our assumptions and notation.\n",
    "import numpy as np\n",
    "from scipy.special import expit\n",
    "import sys\n",
    "import pandas as pd\n",
    "\n",
    "# start with a simple base classifier, which can't be fit or predicted\n",
    "# it only has internal classes to be used by classes that will subclass it\n",
    "class TwoLayerPerceptronBase(object):\n",
    "    def __init__(self, n_hidden=30,\n",
    "                 C=0.0, epochs=500, eta=0.001, random_state=None,\n",
    "                 activ_func=\"sigmoid\", cost_func=\"quadratic\"):\n",
    "        np.random.seed(random_state)\n",
    "        self.n_hidden = n_hidden\n",
    "        self.l2_C = C\n",
    "        self.epochs = epochs\n",
    "        self.eta = eta\n",
    "        self.activ_func = activ_func\n",
    "        self.cost_func = cost_func\n",
    "        \n",
    "    @staticmethod\n",
    "    def _encode_labels(y):\n",
    "        \"\"\"Encode labels into one-hot representation\"\"\"\n",
    "        onehot = pd.get_dummies(y).values.T\n",
    "            \n",
    "        return onehot\n",
    "\n",
    "    def _initialize_weights(self):\n",
    "        \"\"\"Initialize weights with small random numbers.\"\"\"\n",
    "        W1_num_elems = (self.n_features_ + 1)*self.n_hidden\n",
    "        W1 = np.random.uniform(-1.0, 1.0, size=W1_num_elems)\n",
    "        W1 = W1.reshape(self.n_hidden, self.n_features_ + 1) # reshape to be W\n",
    "        \n",
    "        W2_num_elems = (self.n_hidden + 1)*self.n_output_\n",
    "        W2 = np.random.uniform(-1.0, 1.0, size=W2_num_elems)\n",
    "        W2 = W2.reshape(self.n_output_, self.n_hidden + 1)\n",
    "        return W1, W2\n",
    "    \n",
    "    @staticmethod\n",
    "    def _sigmoid(z):\n",
    "        \"\"\"Use scipy.special.expit to avoid overflow\"\"\"\n",
    "        # 1.0 / (1.0 + np.exp(-z))\n",
    "        return expit(z)\n",
    "    \n",
    "    @staticmethod\n",
    "    def _softmax(z):\n",
    "        e_x = np.exp(z - z.max(axis=1, keepdims=True))\n",
    "        out = e_x / e_x.sum(axis=1, keepdims=True)\n",
    "        return out\n",
    "\n",
    "    \n",
    "    @staticmethod\n",
    "    def _add_bias_unit(X, how='column'):\n",
    "        \"\"\"Add bias unit (column or row of 1s) to array at index 0\"\"\"\n",
    "        if how == 'column':\n",
    "            ones = np.ones((X.shape[0], 1))\n",
    "            X_new = np.hstack((ones, X))\n",
    "        elif how == 'row':\n",
    "            ones = np.ones((1, X.shape[1]))\n",
    "            X_new = np.vstack((ones, X))\n",
    "        return X_new\n",
    "    \n",
    "    @staticmethod\n",
    "    def _L2_reg(lambda_, W1, W2):\n",
    "        \"\"\"Compute L2-regularization cost\"\"\"\n",
    "        # only compute for non-bias terms\n",
    "        return (lambda_/2.0) * np.sqrt(np.mean(W1[:, 1:] ** 2) + np.mean(W2[:, 1:] ** 2))\n",
    "    \n",
    "    def _cost(self,A3,Y_enc,W1,W2):\n",
    "        '''Get the objective function value'''\n",
    "        if self.cost_func == \"cross entropy\":\n",
    "            cost = -np.mean(np.nan_to_num((Y_enc*np.log(A3)+(1-Y_enc)*np.log(1-A3))))\n",
    "        else: # quadratic\n",
    "            cost = np.mean((Y_enc-A3)**2)\n",
    "            \n",
    "        L2_term = self._L2_reg(self.l2_C, W1, W2)\n",
    "        return cost + L2_term\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class TwoLayerPerceptron(TwoLayerPerceptronBase):\n",
    "    def _feedforward(self, X, W1, W2):\n",
    "        \"\"\"Compute feedforward step\n",
    "        -----------\n",
    "        X : Input layer with original features.\n",
    "        W1: Weight matrix for input layer -> hidden layer.\n",
    "        W2: Weight matrix for hidden layer -> output layer.\n",
    "        ----------\n",
    "        a1-a3 : activations into layer (or output layer)\n",
    "        z1-z2 : layer inputs \n",
    "\n",
    "        \"\"\"\n",
    "        A1 = self._add_bias_unit(X, how='column')\n",
    "        Z1 = W1 @ A1.T\n",
    "        if self.first_func == \"sigmoid\":\n",
    "            A2 = self._sigmoid(Z1)\n",
    "        elif self.first_func == \"softmax\":\n",
    "            A2 = self._softmax(Z1)\n",
    "        else: \n",
    "            A2 = Z1\n",
    "        A2 = self._add_bias_unit(A2, how='row')\n",
    "        Z2 = W2 @ A2\n",
    "        A3 = self._sigmoid(Z2)\n",
    "        return A1, Z1, A2, Z2, A3\n",
    "    \n",
    "    def _get_gradient(self, A1, A2, A3, Z1, Z2, Y_enc, W1, W2):\n",
    "        \"\"\" Compute gradient step using backpropagation.\n",
    "        \"\"\"\n",
    "        # vectorized backpropagation\n",
    "        if self.cost_func == \"cross entropy\":\n",
    "            sigma3 = (A3-Y_enc)\n",
    "        else: # quadratic\n",
    "            sigma3 = -2*(Y_enc-A3)*A3*(1-A3)\n",
    "        \n",
    "        if self.first_func == \"sigmoid\":\n",
    "            sigma2 = (W2.T @ sigma3)*A2*(1-A2)\n",
    "        elif self.first_func == \"softmax\":\n",
    "            sigma2 = (W2.T @ sigma3)*(A3-Y_enc)\n",
    "        else: # linear\n",
    "            sigma2 = (W2.T @ sigma3)     \n",
    "\n",
    "        grad2 = sigma3 @ A2.T\n",
    "        grad1 = sigma2[1:,:] @ A1\n",
    "        \n",
    "        # regularize weights that are not bias terms\n",
    "        grad1[:, 1:] += W1[:, 1:] * self.l2_C\n",
    "        grad2[:, 1:] += W2[:, 1:] * self.l2_C\n",
    "\n",
    "        return grad1, grad2\n",
    "    \n",
    "    def predict(self, X):\n",
    "        \"\"\"Predict class labels\"\"\"\n",
    "        _, _, _, _, A3 = self._feedforward(X, self.W1, self.W2)\n",
    "        y_pred = np.argmax(A3, axis=0)\n",
    "        return y_pred\n",
    "    \n",
    "    \n",
    "    def fit(self, X, y, print_progress=False):\n",
    "        \"\"\" Learn weights from training data.\"\"\"\n",
    "        \n",
    "        X_data, y_data = X.copy(), y.copy()\n",
    "        Y_enc = self._encode_labels(y)\n",
    "        \n",
    "        # init weights and setup matrices\n",
    "        self.n_features_ = X_data.shape[1]\n",
    "        self.n_output_ = Y_enc.shape[0]\n",
    "        self.W1, self.W2 = self._initialize_weights()\n",
    "\n",
    "        self.cost_ = []\n",
    "        for i in range(self.epochs):\n",
    "\n",
    "            if print_progress>0 and (i+1)%print_progress==0:\n",
    "                sys.stderr.write('\\rEpoch: %d/%d' % (i+1, self.epochs))\n",
    "                sys.stderr.flush()\n",
    "\n",
    "            # feedforward all instances\n",
    "            A1, Z1, A2, Z2, A3 = self._feedforward(X_data,self.W1,self.W2)\n",
    "            \n",
    "            cost = self._cost(A3,Y_enc,self.W1,self.W2)\n",
    "            self.cost_.append(cost)\n",
    "\n",
    "            # compute gradient via backpropagation\n",
    "            grad1, grad2 = self._get_gradient(A1=A1, A2=A2, A3=A3, Z1=Z1, Z2=Z2, Y_enc=Y_enc,\n",
    "                                              W1=self.W1, W2=self.W2)\n",
    "\n",
    "            self.W1 -= self.eta * grad1\n",
    "            self.W2 -= self.eta * grad2\n",
    "            \n",
    "\n",
    "        return self"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Epoch: 1/200"
     ]
    },
    {
     "ename": "ValueError",
     "evalue": "operands could not be broadcast together with shapes (51,1437) (10,1437) ",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-38-1d401c15d5c9>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mget_ipython\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_cell_magic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'time'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"nn = TwoLayerPerceptron(**params)\\nnn.fit(X_train, y_train, print_progress=1)\\nyhat = nn.predict(X_test)\\nprint('Test acc:',accuracy_score(y_test,yhat))\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/Users/danh/anaconda/envs/machinelearning/lib/python3.5/site-packages/IPython/core/interactiveshell.py\u001b[0m in \u001b[0;36mrun_cell_magic\u001b[0;34m(self, magic_name, line, cell)\u001b[0m\n\u001b[1;32m   2113\u001b[0m             \u001b[0mmagic_arg_s\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvar_expand\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mline\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstack_depth\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2114\u001b[0m             \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuiltin_trap\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2115\u001b[0;31m                 \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmagic_arg_s\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2116\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2117\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<decorator-gen-59>\u001b[0m in \u001b[0;36mtime\u001b[0;34m(self, line, cell, local_ns)\u001b[0m\n",
      "\u001b[0;32m/Users/danh/anaconda/envs/machinelearning/lib/python3.5/site-packages/IPython/core/magic.py\u001b[0m in \u001b[0;36m<lambda>\u001b[0;34m(f, *a, **k)\u001b[0m\n\u001b[1;32m    186\u001b[0m     \u001b[0;31m# but it's overkill for just that one bit of state.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    187\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mmagic_deco\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 188\u001b[0;31m         \u001b[0mcall\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mlambda\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    189\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    190\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mcallable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/danh/anaconda/envs/machinelearning/lib/python3.5/site-packages/IPython/core/magics/execution.py\u001b[0m in \u001b[0;36mtime\u001b[0;34m(self, line, cell, local_ns)\u001b[0m\n\u001b[1;32m   1178\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1179\u001b[0m             \u001b[0mst\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mclock2\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1180\u001b[0;31m             \u001b[0mexec\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mglob\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlocal_ns\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1181\u001b[0m             \u001b[0mend\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mclock2\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1182\u001b[0m             \u001b[0mout\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<timed exec>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32m<ipython-input-37-ceb07639dc5d>\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, X, y, print_progress)\u001b[0m\n\u001b[1;32m     82\u001b[0m             \u001b[0;31m# compute gradient via backpropagation\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     83\u001b[0m             grad1, grad2 = self._get_gradient(A1=A1, A2=A2, A3=A3, Z1=Z1, Z2=Z2, Y_enc=Y_enc,\n\u001b[0;32m---> 84\u001b[0;31m                                               W1=self.W1, W2=self.W2)\n\u001b[0m\u001b[1;32m     85\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     86\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mW1\u001b[0m \u001b[0;34m-=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0meta\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mgrad1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-37-ceb07639dc5d>\u001b[0m in \u001b[0;36m_get_gradient\u001b[0;34m(self, A1, A2, A3, Z1, Z2, Y_enc, W1, W2)\u001b[0m\n\u001b[1;32m     36\u001b[0m             \u001b[0msigma2\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mW2\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mT\u001b[0m \u001b[0;34m@\u001b[0m \u001b[0msigma3\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0mA2\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0mA2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     37\u001b[0m         \u001b[0;32melif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfirst_func\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"softmax\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 38\u001b[0;31m             \u001b[0msigma2\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mW2\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mT\u001b[0m \u001b[0;34m@\u001b[0m \u001b[0msigma3\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mA3\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0mY_enc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     39\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;31m# linear\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     40\u001b[0m             \u001b[0msigma2\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mW2\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mT\u001b[0m \u001b[0;34m@\u001b[0m \u001b[0msigma3\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: operands could not be broadcast together with shapes (51,1437) (10,1437) "
     ]
    }
   ],
   "source": [
    "%%time \n",
    "nn = TwoLayerPerceptron(**params)\n",
    "nn.fit(X_train, y_train, print_progress=1)\n",
    "yhat = nn.predict(X_test)\n",
    "print('Test acc:',accuracy_score(y_test,yhat))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Example adapted from https://github.com/rasbt/python-machine-learning-book/blob/master/code/ch12/ch12.ipynb\n",
    "# Original Author: Sebastian Raschka\n",
    "# This is the optional book we use in the course, excellent intuitions and straightforward programming examples\n",
    "# please note, however, that this code has been manipulated to reflect our assumptions and notation.\n",
    "import numpy as np\n",
    "from scipy.special import expit\n",
    "import pandas as pd\n",
    "import sys\n",
    "\n",
    "# start with a simple base classifier, which can't be fit or predicted\n",
    "# it only has internal classes to be used by classes that will subclass it\n",
    "class TwoLayerPerceptronBase(object):\n",
    "    def __init__(self, n_hidden=30,\n",
    "                 C=0.0, epochs=500, eta=0.001, random_state=None,\n",
    "                 first_func=\"sigmoid\", cost_func=\"cross entropy\"):\n",
    "        np.random.seed(random_state)\n",
    "        self.n_hidden = n_hidden\n",
    "        self.l2_C = C\n",
    "        self.epochs = epochs\n",
    "        self.eta = eta\n",
    "        self.first_func = first_func\n",
    "        self.cost_func = cost_func\n",
    "        \n",
    "    @staticmethod\n",
    "    def _encode_labels(y):\n",
    "        \"\"\"Encode labels into one-hot representation\"\"\"\n",
    "        onehot = pd.get_dummies(y).values.T\n",
    "            \n",
    "        return onehot\n",
    "\n",
    "    def _initialize_weights(self):\n",
    "        \"\"\"Initialize weights with small random numbers.\"\"\"\n",
    "        if self.first_func == \"relu\":\n",
    "            init_bound = np.sqrt(6. / (self.n_hidden + self.n_features_ + 1))\n",
    "            W1 = np.random.uniform(-init_bound, init_bound,(self.n_hidden, self.n_features_ + 1))\n",
    "\n",
    "            init_bound = np.sqrt(2. / (self.n_output_ + self.n_hidden + 1))\n",
    "            W2 = np.random.uniform(-init_bound, init_bound,(self.n_output_, self.n_hidden + 1))\n",
    "        else: \n",
    "            W1_num_elems = (self.n_features_ + 1)*self.n_hidden\n",
    "            W1 = np.random.uniform(-1.0, 1.0,size=W1_num_elems)\n",
    "            W1 = W1.reshape(self.n_hidden, self.n_features_ + 1) # reshape to be W\n",
    "\n",
    "            W2_num_elems = (self.n_hidden + 1)*self.n_output_\n",
    "            W2 = np.random.uniform(-1.0, 1.0, size=W2_num_elems)\n",
    "            W2 = W2.reshape(self.n_output_, self.n_hidden + 1)\n",
    "        return W1, W2\n",
    "    \n",
    "    @staticmethod\n",
    "    def _sigmoid(z):\n",
    "        \"\"\"Use scipy.special.expit to avoid overflow\"\"\"\n",
    "        # 1.0 / (1.0 + np.exp(-z))\n",
    "        return expit(z)\n",
    "    \n",
    "    @staticmethod\n",
    "    def _softmax(z):\n",
    "        e_x = np.exp(z - z.max(axis=1, keepdims=True))\n",
    "        out = e_x / e_x.sum(axis=1, keepdims=True)\n",
    "        return out\n",
    "    \n",
    "    @staticmethod\n",
    "    def _relu(Z):\n",
    "        return np.maximum(0,Z.copy())\n",
    "    \n",
    "    @staticmethod\n",
    "    def _add_bias_unit(X, how='column'):\n",
    "        \"\"\"Add bias unit (column or row of 1s) to array at index 0\"\"\"\n",
    "        if how == 'column':\n",
    "            ones = np.ones((X.shape[0], 1))\n",
    "            X_new = np.hstack((ones, X))\n",
    "        elif how == 'row':\n",
    "            ones = np.ones((1, X.shape[1]))\n",
    "            X_new = np.vstack((ones, X))\n",
    "        return X_new\n",
    "    \n",
    "    @staticmethod\n",
    "    def _L2_reg(lambda_, W1, W2):\n",
    "        \"\"\"Compute L2-regularization cost\"\"\"\n",
    "        # only compute for non-bias terms\n",
    "        return (lambda_/2.0) * np.sqrt(np.mean(W1[:, 1:] ** 2) + np.mean(W2[:, 1:] ** 2))\n",
    "    \n",
    "    def _cost(self,A3,Y_enc,W1,W2):\n",
    "        '''Get the objective function value'''\n",
    "        if self.cost_func == \"cross entropy\":\n",
    "            cost = -np.mean(np.nan_to_num((Y_enc*np.log(A3)+(1-Y_enc)*np.log(1-A3))))\n",
    "        else: # quadratic\n",
    "            cost = np.mean((Y_enc-A3)**2)\n",
    "        L2_term = self._L2_reg(self.l2_C, W1, W2)\n",
    "        return cost + L2_term\n",
    "    \n",
    "    def _feedforward(self, X, W1, W2):\n",
    "        \"\"\"Compute feedforward step\n",
    "        \"\"\"\n",
    "        A1 = self._add_bias_unit(X, how='column')\n",
    "        Z1 = W1 @ A1.T\n",
    "        if self.first_func == \"sigmoid\":\n",
    "            A2 = self._sigmoid(Z1)\n",
    "        elif self.first_func == \"relu\":\n",
    "            A2 = self._relu(Z1)\n",
    "        elif self.first_func == \"softmax\":\n",
    "            A2 = self._softmax(Z1)\n",
    "        else: \n",
    "            A2 = Z1\n",
    "        A2 = self._add_bias_unit(A2, how='row')\n",
    "        Z2 = W2 @ A2\n",
    "        A3 = self._sigmoid(Z2)\n",
    "        return A1, Z1, A2, Z2, A3\n",
    "    \n",
    "    def _get_gradient(self, A1, A2, A3, Z1, Z2, Y_enc, W1, W2):\n",
    "        \"\"\" Compute gradient step using backpropagation.\n",
    "        \"\"\"\n",
    "        \n",
    "        # vectorized backpropagation\n",
    "        if self.cost_func == \"cross entropy\":\n",
    "            sigma3 = (A3-Y_enc)\n",
    "        else: # quadratic\n",
    "            sigma3 = -2*(Y_enc-A3)*A3*(1-A3)\n",
    "        \n",
    "        if self.first_func == \"sigmoid\":\n",
    "            sigma2 = (W2.T @ sigma3)*A2*(1-A2)\n",
    "        elif self.first_func == \"softmax\":\n",
    "            sigma2 = (W2.T @ sigma3)\n",
    "        elif self.first_func == \"relu\":\n",
    "            sigma2 = (W2.T @ sigma3) \n",
    "            Z1_with_bias = self._add_bias_unit(Z1,how='row')\n",
    "            sigma2[Z1_with_bias<=0] = 0\n",
    "        # relu derivative only zeros out certain values! easy!\n",
    "        else: # linear\n",
    "            sigma2 = (W2.T @ sigma3)\n",
    "        \n",
    "        grad1 = sigma2[1:,:] @ A1\n",
    "        grad2 = sigma3 @ A2.T\n",
    "        \n",
    "        # regularize weights that are not bias terms\n",
    "        grad1[:, 1:] += W1[:, 1:] * self.l2_C\n",
    "        grad2[:, 1:] += W2[:, 1:] * self.l2_C\n",
    "\n",
    "        return grad1, grad2\n",
    "    \n",
    "    def predict(self, X):\n",
    "        \"\"\"Predict class labels\"\"\"\n",
    "        _, _, _, _, A3 = self._feedforward(X, self.W1, self.W2)\n",
    "        y_pred = np.argmax(A3, axis=0)\n",
    "        return y_pred\n",
    "    def score(self, x, y):\n",
    "        res = self.predict(x)\n",
    "        return accuracy_score(y, res)\n",
    "    \n",
    "    def set_params(self, **parameters):\n",
    "        for parameter, value in parameters.items():\n",
    "            setattr(self, parameter, value)\n",
    "        return self\n",
    "\n",
    "            \n",
    "    def get_params(self, deep=True):\n",
    "        return {\n",
    "            'eta': self.eta,\n",
    "            'n_hidden': self.n_hidden,\n",
    "            'C': self.l2_C,\n",
    "            'epochs': self.epochs,\n",
    "            'eta': self.eta,\n",
    "            'first_func': self.first_func,\n",
    "            'cost_func': self.cost_func\n",
    "        }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "# just start with the vectorized version and minibatch\n",
    "class TLPMiniBatch(TwoLayerPerceptronBase):\n",
    "    def __init__(self, alpha=0.0, decrease_const=0.0, shuffle=True, \n",
    "                 minibatches=1, **kwds):        \n",
    "        # need to add to the original initializer \n",
    "        self.alpha = alpha\n",
    "        self.decrease_const = decrease_const\n",
    "        self.shuffle = shuffle\n",
    "        self.minibatches = minibatches\n",
    "        self._estimator_type = 'tlpnn'\n",
    "        # but keep other keywords\n",
    "        super().__init__(**kwds)\n",
    "        \n",
    "    \n",
    "    def fit(self, X, y, print_progress=False):\n",
    "        \"\"\" Learn weights from training data. With mini-batch\"\"\"\n",
    "        X_data, y_data = X.copy(), y.copy()\n",
    "        Y_enc = self._encode_labels(y)\n",
    "        \n",
    "        # init weights and setup matrices\n",
    "        self.n_features_ = X_data.shape[1]\n",
    "        self.n_output_ = Y_enc.shape[0]\n",
    "        self.W1, self.W2 = self._initialize_weights()\n",
    "\n",
    "        delta_W1_prev = np.zeros(self.W1.shape)\n",
    "        delta_W2_prev = np.zeros(self.W2.shape)\n",
    "\n",
    "        self.cost_ = []\n",
    "        self.score_ = []\n",
    "        for i in range(self.epochs):\n",
    "\n",
    "            # adaptive learning rate\n",
    "            self.eta /= (1 + self.decrease_const*i)\n",
    "\n",
    "            if print_progress>0 and (i+1)%print_progress==0:\n",
    "                sys.stderr.write('\\rEpoch: %d/%d' % (i+1, self.epochs))\n",
    "                sys.stderr.flush()\n",
    "\n",
    "            if self.shuffle:\n",
    "                idx_shuffle = np.random.permutation(y_data.shape[0])\n",
    "                X_data, Y_enc, y_data = X_data[idx_shuffle], Y_enc[:, idx_shuffle], y_data[idx_shuffle]\n",
    "\n",
    "            mini = np.array_split(range(y_data.shape[0]), self.minibatches)\n",
    "            mini_cost = []\n",
    "            for idx in mini:\n",
    "\n",
    "                # feedforward\n",
    "                A1, Z1, A2, Z2, A3 = self._feedforward(X_data[idx],\n",
    "                                                       self.W1,\n",
    "                                                       self.W2)\n",
    "                \n",
    "                cost = self._cost(A3,Y_enc[:, idx],self.W1,self.W2)\n",
    "                mini_cost.append(cost) # this appends cost of mini-batch only\n",
    "\n",
    "                # compute gradient via backpropagation\n",
    "                grad1, grad2 = self._get_gradient(A1=A1, A2=A2, A3=A3, Z1=Z1, Z2=Z2, \n",
    "                                                  Y_enc=Y_enc[:, idx],\n",
    "                                                  W1=self.W1,W2=self.W2)\n",
    "\n",
    "                # momentum calculations\n",
    "                delta_W1, delta_W2 = self.eta * grad1, self.eta * grad2\n",
    "                self.W1 -= (delta_W1 + (self.alpha * delta_W1_prev))\n",
    "                self.W2 -= (delta_W2 + (self.alpha * delta_W2_prev))\n",
    "                delta_W1_prev, delta_W2_prev = delta_W1, delta_W2\n",
    "\n",
    "            self.cost_.append(mini_cost)\n",
    "            self.score_.append(accuracy_score(y_data,self.predict(X_data)))\n",
    "            \n",
    "        return self"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "params = dict(n_hidden=50, \n",
    "              C=0.19, # tradeoff L2 regularizer\n",
    "              epochs=200, # iterations\n",
    "              eta=0.001,  # learning rate\n",
    "              random_state=1, \n",
    "              first_func=\"relu\",\n",
    "              cost_func=\"quadratic\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epoch: 200/200"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test acc: 0.944444444444\n",
      "CPU times: user 2.3 s, sys: 154 ms, total: 2.45 s\n",
      "Wall time: 2.46 s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "nn = TLPMiniBatch(**params)\n",
    "nn.fit(X_train, y_train, print_progress=1)\n",
    "yhat = nn.predict(X_test)\n",
    "print('Test acc:',accuracy_score(y_test,yhat))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.26 s, sys: 1.38 s, total: 3.64 s\n",
      "Wall time: 1min 56s\n",
      "Best score: 0.965205288796\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV, RandomizedSearchCV\n",
    "params = {\n",
    "    \"n_hidden\": [50],\n",
    "    \"eta\": [.0001, .001, .1],\n",
    "    \"epochs\": [200],\n",
    "    \"C\": [0.0, .5, .2],\n",
    "    \"first_func\": ('sigmoid', 'linear', 'relu'),\n",
    "    \"cost_func\": ('quadratic', 'cross entropy'),\n",
    "}\n",
    "\n",
    "\n",
    "cv = GridSearchCV(TLPMiniBatch(), params, n_jobs=-1) # runs in parallel to get the job done as quick as possible\n",
    "%time cv.fit(X_train, y_train)\n",
    "print('Best score: '+str(cv.best_score_))\n",
    "y_hat = cv.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion matrix\n",
      " [[40  0  0  0  1  2  0  0  0  0]\n",
      " [ 0 25  0  0  0  0  0  0 10  0]\n",
      " [ 0  1  1 29  0  2  0  0  1  2]\n",
      " [ 0  0  2 35  0  4  0  0  0  0]\n",
      " [ 1  0  0  0 37  0  0  0  0  0]\n",
      " [ 0  0  0  1  0 29  0  0  0  0]\n",
      " [ 0  0  0  0  0  0 37  0  0  0]\n",
      " [ 0  0  0  0  0  2  0 35  0  0]\n",
      " [ 1  0  3  1  0  2  0  0 19  3]\n",
      " [ 6  1  0  0  0 16  0  0  5  6]]\n",
      "{'n_hidden': 50, 'epochs': 200, 'cost_func': 'cross entropy', 'first_func': 'relu', 'C': 0.2, 'eta': 0.001}\n"
     ]
    }
   ],
   "source": [
    "from mlxtend.plotting import plot_confusion_matrix\n",
    "from sklearn import metrics as mt\n",
    "\n",
    "conf = mt.confusion_matrix(y_test, y_hat)\n",
    "print(\"Confusion matrix\\n\",conf)\n",
    "print(cv.best_params_)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "\n",
    "WOW. That was **sooooo sloooowwwww!** Why?\n",
    "### Self-Test: Where is our biggest bottle neck?\n",
    "- A. Feedforward calculation\n",
    "- B. Number of gradient calculations\n",
    "- C. `for` loop in gradient\n",
    "- D. One-hot encoded vector format of Y\n",
    "___\n",
    "\n",
    "Also, how did our objective function perform?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuQHWd55/Hv0+fM/SaNNJZkjWTJtnyRrxhZZI0xsFmI\nZJKI22IbQpIF1uUUJqF2k8UUVVSqslVZh2yKZXFQKawTNgtxlotBFYQNNgYvscEagSxLtmXLkkAj\n6zK6z0hzOzPP/tF9Rq3DjGZGmp5+R/p9qqZOn7e7jx61jufn9+23u83dERERCU2UdwEiIiKjUUCJ\niEiQFFAiIhIkBZSIiARJASUiIkFSQImISJAUUCIiEiQFlIiIBEkBJSIiQSrmXcBUmjt3ri9ZsiTv\nMkRE5Cw2bdp0yN3bxtvuggqoJUuW0NHRkXcZIiJyFmb2y4lspyE+EREJkgJKRESCpIASEZEgKaBE\nRCRICigREQmSAkpERIKkgBIRkSApoBJbOo/xk1cP5V2GiIgkFFCJdU/v5LPf2Zp3GSIiklBAJQqR\nMeSedxkiIpJQQCUKZgwNK6BEREKhgEpEkTGsgBIRCYYCKlEwDfGJiIQk04Ays1Vmtt3MdpjZA2fZ\n7lYzK5nZ+ye771QpFDTEJyISkswCyswKwEPAamA5cI+ZLR9juweB709236mkc1AiImHJsge1Etjh\n7jvdfQB4BFgzynafAL4JHDyHfadMIVJAiYiEJMuAWgjsSb3vTNpGmNlC4D3Alya7b+oz7jWzDjPr\n6OrqOudiIzOUTyIi4ch7ksTngU+5+/C5foC7r3P3Fe6+oq1t3CcIj6kQoR6UiEhAsnzk+15gUep9\ne9KWtgJ4xMwA5gJ3mllpgvtOqUgX6oqIBCXLgNoILDOzpcThcjfwwfQG7r60vGxm/wD8i7t/28yK\n4+071Qqm66BEREKSWUC5e8nM7gceBwrAw+6+zczuS9avney+WdUKutWRiEhosuxB4e4bgA0VbaMG\nk7v/4Xj7Zikywx3cnWTIUUREcpT3JIlgFKI4lDRRQkQkDAqoxEhAaZhPRCQICqhElAzrDZ/zhHcR\nEZlKCqhEITkS6kGJiIRBAZUo96B0DkpEJAwKqET5HJSuhRIRCYMCKqFJEiIiYVFAJU5PklBAiYiE\nQAGVUA9KRCQsCqhEQZMkRESCooBKRJGugxIRCYkCKqHroEREwqKASug6KBGRsCigEiPXQakHJSIS\nBAVUQpMkRETCooBKRHrchohIUBRQiXIPSkN8IiJhUEAl9MBCEZGwKKASkSZJiIgERQGVOD1JIudC\nREQEUECNiMoX6mqIT0QkCAqohCZJiIiEJdOAMrNVZrbdzHaY2QOjrF9jZlvMbLOZdZjZ7al1u83s\nhfK6LOsETZIQEQlNMasPNrMC8BDwDqAT2Ghm6939xdRmTwLr3d3N7Ebg/wLXpNa/3d0PZVVjWqTH\nbYiIBCXLHtRKYIe773T3AeARYE16A3fvcR9JhAYgt3Qo6IGFIiJByTKgFgJ7Uu87k7YzmNl7zOxl\n4LvAR1KrHHjCzDaZ2b1j/SFmdm8yPNjR1dV1zsVqiE9EJCy5T5Jw90fd/Rrg3cBfpFbd7u43A6uB\nj5vZHWPsv87dV7j7ira2tnOuI9IkCRGRoGQZUHuBRan37UnbqNz9aeByM5ubvN+bvB4EHiUeMszM\n6R5Uln+KiIhMVJYBtRFYZmZLzawauBtYn97AzK40i7suZnYLUAMcNrMGM2tK2huAdwJbM6xVDywU\nEQlMZrP43L1kZvcDjwMF4GF332Zm9yXr1wLvA37fzAaBXuCuZEbfPODRJLuKwNfc/bGsaoXUEJ/O\nQYmIBCGzgAJw9w3Ahoq2tanlB4EHR9lvJ3BTlrVV0iQJEZGw5D5JIhQjj3zXEJ+ISBAUUImRR76r\nByUiEgQFVKKgO0mIiARFAZXQJAkRkbAooBKaJCEiEhYFVGLkgYXKJxGRICigEuUHFmqIT0QkDAqo\nhCZJiIiERQGVGLkOSj0oEZEgKKASug5KRCQsCqhEQXeSEBEJigIqEakHJSISFAVUSiEy9aBERAKh\ngEopmOmBhSIigVBApUSRHvkuIhIKBVRK3INSQImIhEABlVKIFFAiIqFQQKUUItMQn4hIIBRQKepB\niYiEQwGVEpl6UCIioVBApagHJSISDgVUSqTroEREgpFpQJnZKjPbbmY7zOyBUdavMbMtZrbZzDrM\n7PaJ7psFTZIQEQlHZgFlZgXgIWA1sBy4x8yWV2z2JHCTu98MfAT48iT2nXIa4hMRCUeWPaiVwA53\n3+nuA8AjwJr0Bu7e4z7SZWkAfKL7ZiEy3c1cRCQUWQbUQmBP6n1n0nYGM3uPmb0MfJe4FzXhfZP9\n702GBzu6urrOq+BCZLqbuYhIIHKfJOHuj7r7NcC7gb84h/3XufsKd1/R1tZ2XrVEutWRiEgwsgyo\nvcCi1Pv2pG1U7v40cLmZzZ3svlNFkyRERMKRZUBtBJaZ2VIzqwbuBtanNzCzK83iR9ma2S1ADXB4\nIvtmQZMkRETCUczqg929ZGb3A48DBeBhd99mZvcl69cC7wN+38wGgV7grmTSxKj7ZlVrWWTGkPJJ\nRCQImQUUgLtvADZUtK1NLT8IPDjRfbOmSRIiIuHIfZJESPQ8KBGRcCigUqJI10GJiIRCAZWiIT4R\nkXAooFLiSRIKKBGRECigUtSDEhEJhwIqpaAelIhIMBRQKVGk50GJiIRCAZVSMA3xiYiEQgGVUog0\nxCciEgoFVEqkSRIiIsFQQKUU9MBCEZFgKKBSIt3NXEQkGAqoFE2SEBEJhwIqRZMkRETCoYBK0XVQ\nIiLhUEClFEyPfBcRCYUCKkWPfBcRCYcCKiXSAwtFRIKhgEopRCigREQCoYBKiTSLT0QkGAqoFF0H\nJSISDgVUiq6DEhEJx4QCysz+cSJto2yzysy2m9kOM3tglPUfMrMtZvaCmT1jZjel1u1O2jebWcdE\n6jxfkRnu4AopEZHcFSe43XXpN2ZWAN54th2SbR4C3gF0AhvNbL27v5jabBfwVnc/amargXXAm1Lr\n3+7uhyZY43krRAbEEyWKBZuuP1ZEREZx1h6UmX3azLqBG83sRPLTDRwEvjPOZ68Edrj7TncfAB4B\n1qQ3cPdn3P1o8vanQPs5/S2myEhAqQclIpK7swaUu/+luzcBn3P35uSnyd3nuPunx/nshcCe1PvO\npG0sHwW+l/7jgSfMbJOZ3TvWTmZ2r5l1mFlHV1fXOCWdXWRxQA3rdkciIrmb6CSJfzGzBgAz+z0z\n+xszu2yqijCztxMH1KdSzbe7+83AauDjZnbHaPu6+zp3X+HuK9ra2s6rjkJyNNSDEhHJ30QD6kvA\nqWQSw38GXgP+9zj77AUWpd63J21nMLMbgS8Da9z9cLnd3fcmrweBR4mHDDNV7kHpYl0RkfxNNKBK\nHk9tWwN80d0fAprG2WcjsMzMlppZNXA3sD69gZktBr4FfNjdX0m1N5hZU3kZeCewdYK1nrPyOShd\nCyUikr+JzuLrNrNPAx8G3mJmEVB1th3cvWRm9wOPAwXgYXffZmb3JevXAp8F5gB/a3HvpeTuK4B5\nwKNJWxH4mrs/Num/3SRpkoSISDgmGlB3AR8EPuLu+5Oez+fG28ndNwAbKtrWppY/BnxslP12AjdV\ntmft9CQJBZSISN4mNMTn7vuBrwItZvbbQJ+7j3cOasZRD0pEJBwTvZPEB4DngH8PfAD4mZm9P8vC\n8lDQJAkRkWBMdIjvM8CtyYw6zKwNeAL4RlaF5SGKdB2UiEgoJjqLLyqHU+LwJPadMXQdlIhIOCba\ng3rMzB4H/il5fxcVkx8uBLoOSkQkHGcNKDO7Epjn7n9mZu8Fbk9WPUs8aeKCMnIdlHpQIiK5G68H\n9Xng0wDu/i3ii2oxsxuSdb+TaXXTTJMkRETCMd55pHnu/kJlY9K2JJOKchRFCigRkVCMF1CzzrKu\nbioLCUG5B6UhPhGR/I0XUB1m9h8rG83sY8CmbErKT0E9KBGRYIx3DuqTxPfE+xCnA2kFUA28J8vC\n8hBpkoSISDDOGlDufgC4LXle0/VJ83fd/YeZV5aD4kgPKudCRERkYtdBuftTwFMZ15I7XQclIhKO\nC+5uEOdD10GJiIRDAZUycqsj9aBERHKngEoZGeJTD0pEJHcKqBQ98l1EJBwKqBRNkhARCYcCKkWT\nJEREwqGASinoOigRkWAooFI0SUJEJBwKqBRNkhARCUemAWVmq8xsu5ntMLMHRln/ITPbYmYvmNkz\nZnbTRPfNgp4HJSISjswCyswKwEPAamA5cI+ZLa/YbBfwVne/AfgLYN0k9p1yUflCXQ3xiYjkLsse\n1Epgh7vvdPcB4BFgTXoDd3/G3Y8mb38KtE903yxoiE9EJBxZBtRCYE/qfWfSNpaPAt+b7L5mdq+Z\ndZhZR1dX13mUmxriUw9KRCR3QUySSB7n8VHgU5Pd193XufsKd1/R1tZ2XnVE6kGJiARjQo/bOEd7\ngUWp9+1J2xnM7Ebgy8Bqdz88mX2nmiZJiIiEI8se1EZgmZktNbNq4G5gfXoDM1sMfAv4sLu/Mpl9\ns1DuQQ0pn0REcpdZD8rdS2Z2P/A4UAAedvdtZnZfsn4t8FlgDvC3FvdeSslw3aj7ZlVrmSZJiIiE\nI8shPtx9A7Chom1tavljwMcmum/WNElCRCQcQUySCEWkBxaKiARDAZVS7kFpiE9EJH8KqJSRu5lr\niE9EJHcKqBQzIzIoaRqfiEjuFFAVGmqKnBwo5V2GiMhFTwFVoammSE+fAkpEJG8KqAqNtepBiYiE\nQAFVoaGmSLd6UCIiuVNAVWisKdLTr4ASEcmbAqpCU63OQYmIhEABVUE9KBGRMCigKjTWVKkHJSIS\nAAVUhcbaIj0DJd3uSEQkZwqoCk01Rdzh1OBQ3qWIiFzUFFAVGmriJ5Cc1HkoEZFcKaAqNNbGAaVr\noURE8qWAqtCU9KA0k09EJF8KqArlHpRm8omI5EsBVaFxpAc1mHMlIiIXNwVUhXJA6RyUiEi+FFAV\nmmp1DkpEJAQKqArlaeY6ByUikq9MA8rMVpnZdjPbYWYPjLL+GjN71sz6zexPK9btNrMXzGyzmXVk\nWWdaVSGiphjRo2dCiYjkqpjVB5tZAXgIeAfQCWw0s/Xu/mJqsyPAHwPvHuNj3u7uh7KqcSy6o7mI\nSP6y7EGtBHa4+053HwAeAdakN3D3g+6+EQhqypzuaC4ikr8sA2ohsCf1vjNpmygHnjCzTWZ271gb\nmdm9ZtZhZh1dXV3nWOqZGtWDEhHJXciTJG5395uB1cDHzeyO0TZy93XuvsLdV7S1tU3JH9xYU6Rb\nPSgRkVxlGVB7gUWp9+1J24S4+97k9SDwKPGQ4bTQM6FERPKXZUBtBJaZ2VIzqwbuBtZPZEczazCz\npvIy8E5ga2aVVmisKegclIhIzjKbxefuJTO7H3gcKAAPu/s2M7svWb/WzOYDHUAzMGxmnwSWA3OB\nR82sXOPX3P2xrGqt1FirSRIiInnLLKAA3H0DsKGibW1qeT/x0F+lE8BNWdZ2No01VQooEZGchTxJ\nIjdNtUUGSsP0l/RUXRGRvCigRtE48lRdBZSISF4UUKMoB9TXO/awpfMYfYMKKhGR6ZbpOaiZ6toF\nzcyqr+Ivv/cyAMXIuPKSRq67tIXrFzZzY3sL1y9soaZYyLlSEZELl7l73jVMmRUrVnhHx9TcV9bd\n2XOkl62vH2fb68fZuvcE214/zqGeAQCqixE3t8/ijUtmc+uS2bzxslZa6qqm5M8WEbmQmdkmd18x\n7nYKqIlzdw529/OLXx2jY/cROn55lK17j1MadiKDmxfN4o6r2rjjqjZubp9FFFlmtYiIzFQKqGnS\nOzDE5j3HePa1Q/z41UNs6TyGO8xvrmX1DfN51w0LuGXxbIWViEhCAZWToycH+NErB/nulv08/WoX\nA6Vh2mfX8b5b2nnfLe0snlOfa30iInlTQAWgu2+QJ146wDc37eVfXzuEO7xpaSvvf2M7d96wYOTp\nvSIiFxMFVGBeP9bLo7/Yyzc2dbLr0Enqqgqsun4+v3ntJbzlyjZa6jXBQkQuDgqoQLk7P//VUb6x\nqZMNL+zneO8ghci4ZfEs3nb1Jbzt6jaWL2gmuQ+hiMgFRwE1A5SGhnm+8xhPvdzFj145yNa9JwCY\n11zDW69q4+1XX8Kbl82luVa9KxG5cCigZqCDJ/r40Std/Hh7F0+/2kV3X4liZLzxstm87epLuOOq\nuVw7v1kzAkVkRlNAzXCDQ8P84lfHeGr7QX60vYuX9sW9q9aGav7N5XO47co5vPmKuVw2p17DgSIy\noyigLjD7j/fxkx2HeOa1Qzyz4zD7T/QBsHBWHbddMYc3XzmX266YwyXNtTlXKiJydgqoC5i7s/PQ\nSZ7ZcYh/3XGYZ3ce5njvIABL5tSzcmkrK5fOYeWSVha11qmHJSJBUUBdRIaGnZf2neDZ1w7z3O4j\nbNx9hGOn4sCa31zLrUtb49Ba0sqySxp1DktEcqWAuogNDzs7unr42a4jPLfrCM/tOsyBE/0AzKqv\nYsVlrbxpaSu3Lm3lukubqSroqSsiMn0mGlC6lcEFKIqMq+Y1cdW8Jj78G5eN3Jn9ud1xWD236whP\nvHQAgPrqArcsns3Kpa3cuqSVNyyeRW2VHiMiIvlTD+oidfBEXzwcuOsIP9t1hO0HunGHqoJxw8KW\n+BzWUj1GRESmnob4ZFKOnxqk45dHRkJrS2f8GBEzuHpe08iQ4MolrZopKCLnRQEl56V3YIhf7DnK\nxl1H2bj7CJt+eZTewSEgnil465LTgaVrsURkMoI4B2Vmq4D/ARSAL7v7f6tYfw3w98AtwGfc/a8n\nuq9kq666wG1XzOW2K+YC8YXD214/wcZdcS/rBy8d4OubOgG4pKlmJKxuXdLKNfObNFNQRM5bZj0o\nMysArwDvADqBjcA97v5iaptLgMuAdwNHywE1kX1Hox7U9CnPFHxuVzyt/bldR9h3PL54uLm2yIok\nrFYunc0NC2dRXdRMQRGJhdCDWgnscPedSUGPAGuAkZBx94PAQTN712T3lXylZwr+XjJTsPNoLxuT\n67B+tusIP3z5IAC1VRE3L5oV97CWtnLzolk06Qa4IjKOLANqIbAn9b4TeNNU72tm9wL3AixevHjy\nVcqUMDMWtdazqLWe997SDsChnn46dh/huV1HeW73Yb741A6GfwhmcPncBm5sn8WN7S3c2D6L6y5t\n1vR2ETnDjL8Oyt3XAesgHuLLuRxJmdtYw6rrF7Dq+gVA/IThn//qGM/vOcaWzmP8ZMchHv3FXgAK\nSY/spvYWrl/YwrULmrl6fhONeuqwyEUry//69wKLUu/bk7as95VANdVW8dar2njrVW1AfE/BAyf6\neb4zDqwtncf53tb9PLLxdOd5cWs91y5o4pr5zVy7oIlrFzSzaHa9JmGIXASyDKiNwDIzW0ocLncD\nH5yGfWWGMDPmt9Qyv2U+v3XdfICRc1kv7+/m5X0neHl/Ny/tO8H3XzxAeT5PfXWBy9sauKKtkSva\nGkeWl85t0DChyAUks4By95KZ3Q88TjxV/GF332Zm9yXr15rZfKADaAaGzeyTwHJ3PzHavlnVKuFI\nn8t6x/J5I+29A0O8ciAOq5f3d7Pz0Ek6dh/lO5tfT+0L7bPr4tCaGwfXZXPqWdxaz6Wz6nTPQZEZ\nRhfqyozWOzDEzkM9vNZ1kp1d8etrB3vYeaiHvsHhke0ig0tn1bFodhxYi+fEIbi4tZ5Fs+tobajW\nxcYi0ySEaeYimaurLnDdpS1cd2nLGe3Dw86+E33sOXKKXx05NfL6qyOnePLlAxzqGThj+5pixIKW\nWua31HJpSx0LZtWyoKWOBS3x66Wzammpq1KIiUwjBZRckKLIWDirjoWz6viNy+f82vqT/SU6j/aO\nhNe+473sO97HvuN9/HTnYQ509zM0fOboQl1VgQUttcxrrqWtqYZLmmpoS34uaaodWZ5VV6VJHCJT\nQAElF6WGmiJXz2/i6vlNo64fGna6uvt5/Xgv+4/38fqxcoD1cvBEP5v3HONgd98Zw4hlxchGwqqt\n8XSIza6vZk5jNbPrq2ltOP2jiR0io1NAiYyiEJVnGI5953Z35+TAEF3d/Rw80UdXT3+83B2/dnX3\ns+94H893HufIyX6Gxzjd21BdYHZDNXMaqpldDq76alob49fZDdW01FUxq76Klrr4p66qoOFGueAp\noETOkZnRWFOksabI0rkNZ912aNg50TvI4ZMDHD01wOGeAY6klo+eGuDwyXj51QM9HDk5MHL3+NFU\nFyKaK0JrVl0VzeXldHuy3FxXRXNtFTXFSOEmM4ICSmQaFCJjdtJDmqjegSGOnBrg6MkBjvcOcrx3\nkGOnBkeWj/eebj9woo9XDnRzvHeQ7r7SWT+3qmA01VbRWFOkqbb8U0VTTWq5tkhjajleVzWyfUN1\nUefZJHMKKJFA1VUXWFgdT/SYjNLQMN19pTjQelOBdmqA7v4S3X0luvsG6ekrL5fYc+QUPal1Yw1H\nlplBY3VxJMjqq+OeZH11gYaaIg01BRqqizSc0VakobxcXaS+pnB6HwWejEIBJXKBKRaiSffW0tyd\nUwNDSWANcqKvlAqzuIfW3X96+WR/iZ7+EqcGhjjU08/JgRKn+uP9+0u/PolkLHVVhTjYasqBV6C+\nOg67+uoidVUF6qoLZ75WtlW2J+91kfbMpIASkTOY2UiPZ17z2JNEJqI0NMypwSFO9pc42Z+8JgF2\ncqCiLQnFU/0levqHODVQ4ljvIHuP9dI7METv4NDI62QVI/v1EEuFWW11gfpUW21VgZqqiNpi/FpT\nLFCbvNYUo3h9MRpzm4J6g1NCASUimSkWIpoLEc1T+Pyv4WGnvzRM72AcYn2DQ/QODHNqoETv4FD8\nfnCIUwNxoPWVlwdTy6nAO947GK8bGOJU0jaZnt9oipGdDrEk0KqLETVVBWqT1/S6eDkOuupCRHUx\nfq0qGNXFeN+qglFTjJLlZH2yXU25LbW+3DaTw1IBJSIzShTZSE+o9RyHMcczPOwMDA3TPzhMX2mI\n/sFh+ktD9CWv/aVh+gbj15H2wXL7aNuklgfjUOwfHGKgYpv+0hCDQ1N7+7lCZGeEXU2xvJwKtYrX\nYnn7QrxNMbVcVYi484b5LJs3+jWEU0kBJSJSIYqM2ige6mthep/+PDzsDA4PM1AaZnDIGSjFywND\np18HhyraSqfbBofisBsYGmaw5AwMDY18Vn9qu5F9ku17+ksj7aVhT17j/QaTzysNO0PDztXzGxVQ\nIiIXmygyaqICNcUw7zAyNOxM16ChAkpERCZsOs9pae6liIgESQElIiJBUkCJiEiQFFAiIhIkBZSI\niARJASUiIkFSQImISJAUUCIiEiRzn9r7PuXJzLqAX57HR8wFDk1ROVmaCXXOhBphZtQ5E2qEmVHn\nTKgRZkad51PjZe7eNt5GF1RAnS8z63D3FXnXMZ6ZUOdMqBFmRp0zoUaYGXXOhBphZtQ5HTVqiE9E\nRIKkgBIRkSApoM60Lu8CJmgm1DkTaoSZUedMqBFmRp0zoUaYGXVmXqPOQYmISJDUgxIRkSApoERE\nJEgKqISZrTKz7Wa2w8weyLseADNbZGZPmdmLZrbNzP4kaf9zM9trZpuTnzsDqHW3mb2Q1NORtLWa\n2Q/M7NXkdXaO9V2dOl6bzeyEmX0yhGNpZg+b2UEz25pqG/PYmdmnk+/pdjP7rRxr/JyZvWxmW8zs\nUTOblbQvMbPe1DFdOx01nqXOMf+NAzqW/5yqb7eZbU7aczmWZ/ndM73fS3e/6H+AAvAacDlQDTwP\nLA+grgXALclyE/AKsBz4c+BP866votbdwNyKtr8CHkiWHwAezLvO1L/3fuCyEI4lcAdwC7B1vGOX\n/Ps/D9QAS5PvbSGnGt8JFJPlB1M1LklvF8CxHPXfOKRjWbH+vwOfzfNYnuV3z7R+L9WDiq0Edrj7\nTncfAB4B1uRcE+6+z91/nix3Ay8BC/OtalLWAF9Jlr8CvDvHWtJ+E3jN3c/nriNTxt2fBo5UNI91\n7NYAj7h7v7vvAnYQf3+nvUZ3/767l5K3PwXas65jPGMcy7EEcyzLzMyADwD/lHUdZ3OW3z3T+r1U\nQMUWAntS7zsJLAjMbAnwBuBnSdMnkqGVh/McOktx4Akz22Rm9yZt89x9X7K8H5iXT2m/5m7O/AUQ\n2rGEsY9dqN/VjwDfS71fmgxJ/djM3pJXUSmj/RuHeCzfAhxw91dTbbkey4rfPdP6vVRAzQBm1gh8\nE/iku58AvkQ8HHkzsI94SCBvt7v7zcBq4ONmdkd6pcfjALlf02Bm1cDvAl9PmkI8lmcI5diNxcw+\nA5SAryZN+4DFyffhPwFfM7PmvOpjBvwbp9zDmf/zlOuxHOV3z4jp+F4qoGJ7gUWp9+1JW+7MrIr4\nC/JVd/8WgLsfcPchdx8G/o5pGJYYj7vvTV4PAo8S13TAzBYAJK8H86twxGrg5+5+AMI8lomxjl1Q\n31Uz+0Pgt4EPJb+wSIZ5DifLm4jPR1yVV41n+TcO7VgWgfcC/1xuy/NYjva7h2n+XiqgYhuBZWa2\nNPk/7LuB9TnXVB6P/l/AS+7+N6n2BanN3gNsrdx3OplZg5k1lZeJT55vJT6Gf5Bs9gfAd/Kp8Axn\n/B9qaMcyZaxjtx6428xqzGwpsAx4Lof6MLNVwH8BftfdT6Xa28yskCxfntS4M48akxrG+jcO5lgm\n/h3wsrt3lhvyOpZj/e5hur+X0z07JNQf4E7imSqvAZ/Ju56kptuJu9BbgM3Jz53APwIvJO3rgQU5\n13k58Qye54Ft5eMHzAGeBF4FngBac66zATgMtKTacj+WxIG5DxgkHrv/6NmOHfCZ5Hu6HVidY407\niM87lL+ba5Nt35d8DzYDPwd+J+djOea/cSjHMmn/B+C+im1zOZZn+d0zrd9L3epIRESCpCE+EREJ\nkgJKRESCpIASEZEgKaBERCRICigREQmSAkokQ2Y2ZGfeRX3K7pSf3Ok6lOu2RKZcMe8CRC5wvR7f\npkZEJkk8XzK+AAABs0lEQVQ9KJEcJM/8+SuLn6H1nJldmbQvMbMfJjc2fdLMFift8yx+5tLzyc9t\nyUcVzOzvkmf2fN/M6pLt/zh5ls8WM3skp7+myHlRQIlkq65iiO+u1Lrj7n4D8EXg80nb/wS+4u43\nEt989QtJ+xeAH7v7TcTPEtqWtC8DHnL364BjxHcegPhZPW9IPue+rP5yIlnSnSREMmRmPe7eOEr7\nbuDfuvvO5Kac+919jpkdIr4Vz2DSvs/d55pZF9Du7v2pz1gC/MDdlyXvPwVUuft/NbPHgB7g28C3\n3b0n47+qyJRTD0okPz7G8mT0p5aHOH1e+V3AQ8S9rY3JnbJFZhQFlEh+7kq9PpssP0N8N32ADwH/\nL1l+EvgjADMrmFnLWB9qZhGwyN2fAj4FtAC/1osTCZ3+r0okW3Vmtjn1/jF3L081n21mW4h7Qfck\nbZ8A/t7M/gzoAv5D0v4nwDoz+yhxT+mPiO+IPZoC8H+SEDPgC+5+bMr+RiLTROegRHKQnINa4e6H\n8q5FJFQa4hMRkSCpByUiIkFSD0pERIKkgBIRkSApoEREJEgKKBERCZICSkREgvT/AabX+h18psHy\nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1038b0f60>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(range(len(nn.cost_)), nn.cost_)\n",
    "plt.ylabel('Cost')\n",
    "plt.xlabel('Epochs')\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vectorized coding operations in gradient\n",
    "So we need to derive equations for vectorized operations. These can be derived by specifically writing out our current gradient calculations. If we plug in the equations for the final layer gradient, we have that:\n",
    "\n",
    "$$  \\Delta_k^{(2)} = -2(\\mathbf{y}^{(k)}-{_k\\mathbf{a}^{(3)}}){_k\\mathbf{a}^{(3)}}(1-{_k\\mathbf{a}^{(3)}})\\cdot {_k\\mathbf{a}^{(2)} } $$\n",
    "\n",
    "where we use $\\Delta_k^{(2)}$ in place of $\\frac{\\partial J(\\mathbf{W}^{(l)})}{\\partial _k\\mathbf{z}^{(2)}} \\cdot {_k\\mathbf{a}^{(2)}}$ to simplify the notation. Therefore, the total batch gradient calculation is given by:\n",
    "\n",
    "$$ \\mathbf{\\Delta}^{(2)} = \\sum_k \\Delta_k^{(2)} $$\n",
    "\n",
    "This first operation is relatively easy to vectorize because each column in $\\mathbf{A}^{(2)}$ corresponds to the $k^{th}$ instance of $_k\\mathbf{a}^{(2)}$. Therefore we can simply transpose $\\mathbf{A}^{(2)}$ and multiply it by the elementwise multiplications of $\\mathbf{y}^{(k)}$ and $_k\\mathbf{a}^{(3)}$:\n",
    "\n",
    "$$ \\mathbf{\\Delta}^{(2)} = -2(\\mathbf{Y}-\\mathbf{A}^{(3)})*\\mathbf{A}^{(3)}*(1-\\mathbf{A}^{(3)})\\cdot [\\mathbf{A}^{(2)}]^T $$\n",
    "\n",
    "For convenience, we define another variable, $\\mathbf{\\sigma}^{(2)}$:\n",
    "\n",
    "$$ \\mathbf{\\sigma}^{(2)} = -2(\\mathbf{Y}-\\mathbf{A}^{(3)})*\\mathbf{A}^{(3)}*(1-\\mathbf{A}^{(3)}) $$\n",
    "\n",
    "such that:\n",
    "\n",
    "$$ \\mathbf{\\Delta}^{(2)} = \\mathbf{\\sigma}^{(2)}\\cdot [\\mathbf{A}^{(2)}]^T $$\n",
    "\n",
    "Further details about this calculation can be found in the lecture notes or `PDF_slides/VectorizedBackPropagation.pdf` in this same repository.\n",
    "\n",
    "![VectorizedDerivation](PDF_slides/VectorizedBackPropagation.pdf)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "\n",
    "Similarly, the calculation of the gradient for the hidden layer, from our derivations was:\n",
    "\n",
    "$$  \\Delta_k^{(1)} = \\frac{\\partial J(\\mathbf{W})}{\\partial \\mathbf{z}^{(2)}}\\cdot \\mathbf{W}^{(2)} \\cdot \\text{diag}[{_k\\mathbf{a}^{(2)}}*(1-{_k\\mathbf{a}^{(2)}})] \\cdot \\mathbf{a}^{(1)}$$\n",
    "\n",
    "$$ \\mathbf{\\Delta}^{(1)} = \\sum_k \\Delta_k^{(1)} $$\n",
    "\n",
    "This can also be vectorized by recognizing that $\\frac{\\partial J(\\mathbf{W})}{\\partial \\mathbf{z}^{(2)}}$ is a matrix with columns equal to $\\frac{\\partial J(\\mathbf{W})}{\\partial {_k\\mathbf{z}^{(2)}}}$ for each instance $k$. We have already solved for this: $ \\mathbf{\\sigma}^{(2)} $. With some algebra, we can find that the computation is represented vectorized as:\n",
    "\n",
    "$$  \\mathbf{\\Delta}^{(1)} = [\\mathbf{W}^{(2)}]^T \\cdot \\mathbf{\\sigma}^{(2)} * \\mathbf{A}^{(2)} * (1-\\mathbf{A}^{(2)}) \\cdot \\mathbf{A}^{(1)} $$\n",
    "\n",
    "And we define an intermediate variable $\\mathbf{\\sigma}^{(1)}$ as:\n",
    "\n",
    "$$ \\mathbf{\\sigma}^{(1)} =  [\\mathbf{W}^{(2)}]^T \\cdot \\mathbf{\\sigma}^{(2)} * \\mathbf{A}^{(2)} * (1-\\mathbf{A}^{(2)}) $$\n",
    "\n",
    "such that: \n",
    "\n",
    "$$  \\mathbf{\\Delta}^{(1)} = \\mathbf{\\sigma}^{(1)}\\cdot \\mathbf{A}^{(1)} $$\n",
    "\n",
    "Let's get programming! All we really need to do is update the `_get_gradient` class method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class TwoLayerPerceptronVectorized(TwoLayerPerceptron):\n",
    "    # just need a different gradient calculation\n",
    "    def _get_gradient(self, A1, A2, A3, Z1, Z2, Y_enc, W1, W2):\n",
    "        \"\"\" Compute gradient step using backpropagation.\n",
    "        \"\"\"\n",
    "        # vectorized backpropagation\n",
    "        sigma2 = -2*(Y_enc-A3)*A3*(1-A3)\n",
    "        grad2 = sigma2 @ A2.T\n",
    "        \n",
    "        sigma1 = (W2.T @ sigma2)*A2*(1-A2)\n",
    "        grad1 = sigma1[1:,:] @ A1\n",
    "        \n",
    "        \n",
    "        # regularize weights that are not bias terms\n",
    "        grad1[:, 1:] += W1[:, 1:] * self.l2_C\n",
    "        grad2[:, 1:] += W2[:, 1:] * self.l2_C\n",
    "\n",
    "        return grad1, grad2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epoch: 500/500"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.938888888889\n",
      "CPU times: user 4.67 s, sys: 228 ms, total: 4.89 s\n",
      "Wall time: 2.61 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "params['epochs'] = 500\n",
    "nn = TwoLayerPerceptronVectorized(**params)\n",
    "\n",
    "nn.fit(X_train, y_train, print_progress=1)\n",
    "yhat = nn.predict(X_test)\n",
    "print('Accuracy:',accuracy_score(y_test,yhat))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X+UXWV97/H355wzPzKTSUKSSZD8ILFEWUFAcYRasYBd\nKFht1GqFWm0Vm4tLtL2tvcLtXbZd9i6X1VW5Wmpurhd/lCr33lZsSlMQFEorIpm0kEAkGEIgiUB+\nkmSSzI8z871/7GdmTiaTZJLMztnJfF5rzTp7P3vvM888i5UPz7Of/WxFBGZmZkVTqncFzMzMRuOA\nMjOzQnJAmZlZITmgzMyskBxQZmZWSA4oMzMrJAeUmZkVkgPKzMwKyQFlZmaFVKl3BcbTzJkzY8GC\nBfWuhpmZHcXq1at3RET7sc47owJqwYIFdHZ21rsaZmZ2FJKeG8t5HuIzM7NCckCZmVkhOaDMzKyQ\nHFBmZlZIDigzMyskB5SZmRWSA8rMzArJAZWs2fIy//azHfWuhpmZJQ6oZPlDG/nMPzxR72qYmVni\ngEokEfWuhJmZDXFAJSVBhCPKzKwoHFCJgAHnk5lZYTigkmyIzwllZlYUuQaUpGskrZe0QdLNRznv\nDZKqkt57vNeOW10Bj/CZmRVHbgElqQzcBlwLLAaul7T4COd9Hvj+8V47zvV1QJmZFUiePahLgQ0R\nsTEieoE7gSWjnPcJ4O+BbSdw7biRJ0mYmRVKngE1B9hcs78llQ2RNAd4N/DV47225juWSuqU1Ll9\n+/YTrqzAd6DMzAqk3pMkbgU+HREDJ/oFEbE8IjoioqO9/ZhvED6irAd1wpebmdk4y/OV71uBeTX7\nc1NZrQ7gTkkAM4G3S6qO8dpxVfIsPjOzQskzoFYBiyQtJAuX64DfrD0hIhYObkv6BnB3RHxPUuVY\n1443yc9BmZkVSW4BFRFVSTcB9wJl4PaIeFLSjen4suO9Nq+6ZjyLz8ysSPLsQRERK4GVI8pGDaaI\n+J1jXZunbJTRCWVmVhT1niRRGCVPkjAzKxQHVCLEgBPKzKwwHFCJ5AE+M7MicUAlXovPzKxYHFBJ\nthafE8rMrCgcUIlXkjAzKxYHVCL8ynczsyJxQCVezdzMrFgcUEnJs/jMzArFAZVIfg7KzKxIHFCJ\np5mbmRWLA2qQh/jMzArFAZWUvJSEmVmhOKASge9BmZkViAMqcQfKzKxYHFCJ8FJHZmZF4oBK/ByU\nmVmxOKAGya98NzMrEgdUovTpYT4zs2JwQCVKCeV8MjMrBgdUUkoJ5XwyMyuGXANK0jWS1kvaIOnm\nUY4vkbRG0mOSOiVdXnNsk6S1g8fyrCcMD/H5WSgzs2Ko5PXFksrAbcDVwBZglaQVEbGu5rQfACsi\nIiRdBPxf4Pya41dFxI686nhofbNP55OZWTHk2YO6FNgQERsjohe4E1hSe0JEdMXwrIRW6jjCpqEh\nPieUmVkR5BlQc4DNNftbUtkhJL1b0lPAPwEfqTkUwP2SVktaeqRfImlpGh7s3L59+wlX1j0oM7Ni\nqfskiYi4KyLOB94FfLbm0OUR8VrgWuDjkn75CNcvj4iOiOhob28/4Xoo3YVyQJmZFUOeAbUVmFez\nPzeVjSoiHgJeKWlm2t+aPrcBd5ENGeZmqAflIT4zs0LIM6BWAYskLZTUCFwHrKg9QdJ5Sjd/JF0C\nNAE7JbVKakvlrcBbgSdyrGvNg7p5/hYzMxur3GbxRURV0k3AvUAZuD0inpR0Yzq+DPh14EOS+oCD\nwPvTjL7ZwF0puyrAtyPinrzqCn4OysysaHILKICIWAmsHFG2rGb788DnR7luI3BxnnUbaXCIz89B\nmZkVQ90nSRSN88nMrBgcUImGZ0mYmVkBOKCSkmfxmZkVigMqGV6Lr67VMDOzxAGVDC115JtQZmaF\n4IBKfAvKzKxYHFDJcA+qzhUxMzPAATXEr3w3MysWB1TiIT4zs2JxQCVezdzMrFgcUImfgzIzKxYH\nVDK8Fl9962FmZhkHVDI8xOeEMjMrAgfUIL/y3cysUBxQyeD7oMzMrBgcUMnwWnzuQpmZFYEDKpGH\n+MzMCsUBlfhBXTOzYnFAJSWvZm5mVigOqBH8HJSZWTE4oJKhV757kM/MrBByDShJ10haL2mDpJtH\nOb5E0hpJj0nqlHT5WK8d97qmT4/wmZkVQ24BJakM3AZcCywGrpe0eMRpPwAujojXAh8BvnYc146r\noXtQef4SMzMbszx7UJcCGyJiY0T0AncCS2pPiIiuGJ6V0MpwPhzz2vE2vBafI8rMrAjyDKg5wOaa\n/S2p7BCS3i3pKeCfyHpRY742Xb80DQ92bt++/YQr6yE+M7NiqfskiYi4KyLOB94FfPYErl8eER0R\n0dHe3n7C9fCDumZmxZJnQG0F5tXsz01lo4qIh4BXSpp5vNeOBw3dg3JCmZkVQZ4BtQpYJGmhpEbg\nOmBF7QmSzlNKBkmXAE3AzrFcO948xGdmViyVvL44IqqSbgLuBcrA7RHxpKQb0/FlwK8DH5LUBxwE\n3p8mTYx6bV51hZoelAPKzKwQcgsogIhYCawcUbasZvvzwOfHem2ehh/TdUKZmRVB3SdJFEUptYR7\nUGZmxeCASgZf+e7noMzMisEBNciv2zAzKxQHVOJZfGZmxeKASkpezdzMrFAcUMnwWnz1rYeZmWUc\nUMngJAkP8ZmZFYMDKhlei88JZWZWBA6oxHegzMyKxQGVDC515OegzMyKwQGVeBKfmVmxOKAS55OZ\nWbE4oBKvZm5mViwOqKQ09ByUE8rMrAgcUIm8Fp+ZWaE4oIYMDvE5oszMisABlbgHZWZWLA6opCT3\noMzMisQBlfh1G2ZmxeKASobX4qtvPczMLOOASoZWM69zPczMLJNrQEm6RtJ6SRsk3TzK8Q9IWiNp\nraSHJV1cc2xTKn9MUmee9cx+X/bp56DMzIqhktcXSyoDtwFXA1uAVZJWRMS6mtOeBa6IiN2SrgWW\nA5fVHL8qInbkVcdD65t9Op/MzIohzx7UpcCGiNgYEb3AncCS2hMi4uGI2J12HwHm5lifo5JX4zMz\nK5QxBZSkvxlL2QhzgM01+1tS2ZHcAPxzzX4A90taLWnpUeq2VFKnpM7t27cfo0pH5h6UmVmxjHWI\n74LanTR89/rxqoSkq8gC6vKa4ssjYqukWcB9kp6KiIdGXhsRy8mGBuno6DjheBl6DupEv8DMzMbV\nUXtQkm6RtA+4SNLe9LMP2Ab8wzG+eyswr2Z/biob+TsuAr4GLImInYPlEbE1fW4D7iIbMsyNJ0mY\nmRXLUQMqIj4XEW3AFyJiSvppi4gZEXHLMb57FbBI0kJJjcB1wIraEyTNB74LfDAinq4pb5XUNrgN\nvBV44rj/uuPgB3XNzIplrEN8d0tqjYj9kn4LuAT4HxHx3JEuiIiqpJuAe4EycHtEPCnpxnR8GfAZ\nYAbw1+l9TNWI6ABmA3elsgrw7Yi458T+xLHxWnxmZsUy1oD6KnBxek7pD8mG5L4FXHG0iyJiJbBy\nRNmymu2PAh8d5bqNwMUjy/Mkr8VnZlYoY51mXo3sX+4lwF9FxG1AW37VOvU8xGdmVixj7UHtk3QL\n8EHgzZJKQEN+1Tr1hnpQHuQzMyuEsfag3g/0AB+JiBfJZuR9Ibda1YF7UGZmxTKmgEqh9LfAVEnv\nALoj4lu51uwUG34fVJ0rYmZmwNhXkvgN4FHgfcBvAD+R9N48K3aq+TkoM7NiGes9qD8G3pAemkVS\nO3A/8Hd5VaxeHE9mZsUw1ntQpcFwSnYex7WnBXmtWDOzQhlrD+oeSfcC30n772fE802nu5Jn8ZmZ\nFcpRA0rSecDsiPgjSe9heDHXH5NNmjhjDN+Dqm89zMwsc6we1K3ALQAR8V2ydfOQdGE69s5ca3cK\nDb3y3QFlZlYIx7qPNDsi1o4sTGULcqlRnQyvxeeEMjMrgmMF1LSjHJs0nhWpN7+w0MysWI4VUJ2S\nfndkoaSPAqvzqVJ9DA/xOaHMzIrgWPegfp/stRcfYDiQOoBG4N15VuxU8+s2zMyK5agBFREvAb+U\nXsn+mlT8TxHxw9xrdop5LT4zs2IZ03NQEfEA8EDOdamrkt8HZWZWKGfUahAnw89BmZkViwMqGZok\nUed6mJlZxgE1aGiauSPKzKwIHFBJScc+x8zMTh0HVDL4yne/D8rMrBhyDShJ10haL2mDpJtHOf4B\nSWskrZX0sKSLx3rtuNc1fTqfzMyKIbeAklQGbgOuBRYD10taPOK0Z4ErIuJC4LPA8uO4dpzrm306\nn8zMiiHPHtSlwIaI2BgRvcCdwJLaEyLi4YjYnXYfAeaO9drxNvwcVJ6/xczMxirPgJoDbK7Z35LK\njuQG4J+P91pJSyV1Surcvn37SVQ343tQZmbFUIhJEmkppRuATx/vtRGxPCI6IqKjvb39JOpwwpea\nmVkOxvrK9xOxFZhXsz83lR1C0kXA14BrI2Ln8Vw7nryauZlZseTZg1oFLJK0UFIjcB2wovYESfPJ\n3tL7wYh4+niuHW8lvw/KzKxQcutBRURV0k3AvUAZuD0inpR0Yzq+DPgMMAP46/QcUjUN1416bV51\nhdrnoPL8LWZmNlZ5DvERESuBlSPKltVsfxT46FivzdPQc1CeaG5mVgiFmCRRBH7lu5lZsTigksEh\nPueTmVkxOKBqSDDgm1BmZoXggKrRUCpRdUCZmRWCA6pGQ1lU+wfqXQ0zM8MBdYiGSok+B5SZWSE4\noGpUSiV6+z3EZ2ZWBA6oGo0e4jMzKwwHVA0P8ZmZFYcDqkalJPo8xGdmVggOqBoN5RK97kGZmRWC\nA6pGY6Xke1BmZgXhgKrhIT4zs+JwQNXwEJ+ZWXE4oGo0ehafmVlhOKBqNJRLVD3EZ2ZWCA6oGtk9\nKPegzMyKwAFVo6Hie1BmZkXhgKrRWPY9KDOzonBA1aiU5HtQZmYFkWtASbpG0npJGyTdPMrx8yX9\nWFKPpE+NOLZJ0lpJj0nqzLOeg7wWn5lZcVTy+mJJZeA24GpgC7BK0oqIWFdz2i7gk8C7jvA1V0XE\njrzqOFJjuURv1QFlZlYEefagLgU2RMTGiOgF7gSW1J4QEdsiYhXQl2M9xqyhLL/y3cysIPIMqDnA\n5pr9LalsrAK4X9JqSUvHtWZHUPEkCTOzwshtiG8cXB4RWyXNAu6T9FREPDTypBReSwHmz59/Ur+w\noVyirz+ICCSd1HeZmdnJybMHtRWYV7M/N5WNSURsTZ/bgLvIhgxHO295RHREREd7e/tJVDd7oy7g\nBWPNzAogz4BaBSyStFBSI3AdsGIsF0pqldQ2uA28FXgit5omlXLWHNUBD/OZmdVbbkN8EVGVdBNw\nL1AGbo+IJyXdmI4vk3Q20AlMAQYk/T6wGJgJ3JWG2SrAtyPinrzqOqghBVRfNaAx799mZmZHk+s9\nqIhYCawcUbasZvtFsqG/kfYCF+dZt9EMDvF5uSMzs/rzShI1Bof4PJPPzKz+HFA1Bof4vNyRmVn9\nOaBqNKQhvl0HeutcEzMzc0DVmDe9hZLgN/7nj/mvd63l+Z0H6l0lM7MJywFV45L5Z/HDP7yS975+\nLn+3egu/8pcP8tm717HnQCFWYjIzm1AUcebcb+no6IjOzvFZ+Hzb3m6+dP/T/J9Vm5ne2sQX3nsR\nV50/a1y+28xsIpO0OiI6jnWee1BHMGtKM597z0X84ycuZ+bkRj78jVX86Yon6fdismZmp4QD6hgu\nOGcq3/v4m/idX1rANx7exMfuWE13X3+9q2VmdsZzQI1Bc0OZP/21C/iTdy7m++te4qPf7ORAb7Xe\n1TIzO6M5oI7Dh9+0kC++72IefmYHv/ZXP2Ldz/fWu0pmZmcsB9Rxeu/r5/I3N1zG3oN9vOuvf8TX\nf/QsA74vZWY27hxQJ+BN583kn3/vzbz5vJn82T+u44O3/4Qtu/3MlJnZeHJAnaAZk5v42m938Ln3\nXMhjz7/M2770EN959HnOpGn7Zmb15IA6CZK4/tL53Puff5mL503jlu+u5UO3P8pzO/fXu2pmZqc9\nB9Q4mHtWC3fccBmfXXIB//7cbq7+0kPcev/Tno5uZnYSHFDjpFQSH3zjAn74qSt52wVnc+v9P+Nt\ntz7EA+u31btqZmanJQfUOJs9pZmvXP867rjhMsol8eGvr+Ij31jFM9u76l01M7PTigMqJ5cvymb6\n3XLt+Tz67C7e9qWHsoVnD3rhWTOzsXBA5aipUuY/XfELPPCpK3lfx1xu/9GzXPXFB7njkeeo+q29\nZmZH5YA6Bdrbmvjcey7i7k9czqJZk/lv33uCX/3yv/GjDTvqXTUzs8JyQJ1CF5wzlTuX/iJf/cAl\n7O+t8oGv/YTf/VYnm3Z4WrqZ2Ui5BpSkayStl7RB0s2jHD9f0o8l9Uj61PFce7qSxLUXvoL7/+AK\n/uhtr+bhDTu4+kv/wudW/pS93b4/ZWY2KLeAklQGbgOuBRYD10taPOK0XcAngS+ewLWnteaGMh+/\n6jwe+NSVvOu1c1j+rxt5yxcf5DuPPu93TpmZkW8P6lJgQ0RsjIhe4E5gSe0JEbEtIlYBI7sOx7z2\nTDFrSjNfeN/FrPj45SyY0cot313LO7/ybzyycWe9q2ZmVld5BtQcYHPN/pZUlve1p6UL507l/934\nRr5y/evYc7CP65Y/wsfuWM3mXV6E1swmpkq9K3CyJC0FlgLMnz+/zrU5OZJ458XncPXi2Sx/aCNf\nffAZ7lv3EkteO4cbr3gli2a31buKZmanTJ49qK3AvJr9ualsXK+NiOUR0RERHe3t7SdU0aJpbijz\nyV9ZxAOfupIPvvFcVq59gau/9BC/+61OOjft8orpZjYh5NmDWgUskrSQLFyuA37zFFx7xjh7ajN/\n8s4L+MRbFvHNhzfxzR9v4r51L/Hq2W1cf+k83v26uUxtaah3Nc3McqE8/29c0tuBW4EycHtE/HdJ\nNwJExDJJZwOdwBRgAOgCFkfE3tGuPdbv6+joiM7Ozpz+mvrb31PlHx//Od9+9HnWbNlDU6XENa85\nm3dedA5vftVMmirlelfRzOyYJK2OiI5jnncmDRed6QFV64mte/jOo89z95oX2HOwj7bmCm9dfDZv\nv/Bs3vgLM2hpPO1vL5rZGcoBNUH0Vgf40TM7uPvxF/j+uhfZ112lsVLisoXTueJV7Vz56ln8Qnsr\nkupdVTMzwAE1IfVU+1n17G4eXL+NB9Zv45nt2RJKs6c08YYF07ls4XTesHA6r5rVRqnkwDKz+nBA\nGZt3HeBfnt7OT57dxapnd/Hi3m4Apk5q4JL507hwzlQunJt9zp7S5F6WmZ0SDig7RESwZffBobB6\nbPPL/GzbPgZXVWpva+LCOVO54JwpLJrdxqtnt7FwZiuNFa8nbGbja6wB5TvpE4Qk5k1vYd70Ft77\n+rkAHOit8tMX9rJmyx7Wbt3D2i17eHD9tqHQqpTEwpmtvOrsNl41q41FsyezYEYrC2a2eBKGmeXO\n/8pMYC2NFV5/7nRef+70obLuvn42bt/Pz7btY/2L+3j6pS6e2LqHlWtfoLazPautiQUzW1k4o5Vz\nZ7ZknzNamT+jhclN/s/KzE6e/yWxQzQ3lFl8zhQWnzPlkPKDvf08s72L53YeYNPO/Ty7Yz/P7dzP\nD57axo6unkPOndJcYc5ZLcyZNom5Z03inGnNzJnWwpyzJjFn2iRmTm70/S4zOyYHlI3JpMYyr5kz\nldfMmXrYsX3dfUPBtWX3QbbuPsjWlw+yedcBHtm4k66e6iHnN1VKzJ7SzOwpTcxqa2ZW+pxd+zml\nmSnNFQeZ2QTmgLKT1tbccMTwAthzsG8otH7+cvb54p5utu3r5qcv7OXB9d3s7+0/7LrBIJs5uZHp\nrU3MaG3krNZGZrQ2Mn2Un5bGsgPN7AzigLLcTZ3UwNRJDYcNG9bq6qmybW832/b18NLebranz237\neti+r4ctuw+wZsvL7D7QS1//6DNPmyolprc2Mq2lkamTKkxpzn7vlPT7pzRXmNrSMEp5A80NJYeb\nWcE4oKwQJjdVmNw+mVe2Tz7qeRHBvp4qu7p62XWgN/vcn7b397Kzq5c9B3vZe7DKczsPsLe7j70H\n+0btodVqKIuWxgqTmyq0NpVpbUrbjRVaDysb3m5pqjA5HWsdur7i6flm48ABZacVSUxpzno9C2gd\n83V9/QPs666y52AWWHsO9rG3O30erLK3u48DPVW6evrZ31Nlf2+Vrp4qL+3tZn9PP/t7q+zvqR6x\n9zZSY6U0HHaNFdqaB4OuwuTGCpPT/uSa4JvR2sTMtkbaJzdxVkujV/uwCc8BZRNCQ7k0dK/qZPRU\n+7PASiG2vybUunqy/f09VfYNbffT1VOlq7vKrv29PL/rQHZud/WovbpySUxvbeScqc3Mn9HKudNb\nOHdGC+fOaGXBjBba27zyh535HFBmx6GpUqapUj7poAMYGAgO9PXT1V2lq6ePnV297OjqZUdXdt9t\nR1cPW18+yOObX2bl2hfoHxjuvbU0lofCasHM9DmjlYUzWx1edsZwQJnVSamk7N5bUwVo5rxZRz63\nr3+ArbsP8tyuAzw39BzaAda/uI/71r1EtSa8mhvSNP40hf/sKc3MntJMe1sT01oamNbSyFktDUyb\n1Ehbc8VDiVZYDiiz00BDuZT1lGa2Au2HHKv2D/Dzl7t5dud+Nu3Yz+ZdB3gpzYJ8Yuse7v/pS3T3\nDYz6vSVlsyyntTQyZVIDk5vKh00WGZwoUjsZpLmhTHNDieaGMk2VUtof3i479GwcOKDMTnOVcon5\nM1qYP6OFK17VftjxiGBvd5Xt+3rYc7CXlw/0sftAHy8f6GXPwT52H8jK9hzsY39PlZ1dvcP303r7\n6a2OHm5H01AWTZUsxJoqZZoaSjSn/cZKiYZyicZyiUpZh21nP8PbjZUSlVLar5RoGGW7XBZliUpJ\nlErZZ3nET6UkShKVUolSCSql0lHOkXuWBeCAMjvDSRp6Fu1E9FYHOJBmNR7ozSZ9dPf201MdoLuv\nn+5qPz192XZWNjBcVu0fKu/p66e7b4De6gBd1Sp9/QP0VYO+gYHh7f603Z9t1w5d1sNoQVdWFl4l\nQUlZoGlomxH7NdulwWO11zJif7TvqjleGjx/tN8FIttX+l6RtgfLSeWHlWXfwSjlIvtuar7vHRe9\ngvNmteXf/rn/BjM7rTVWSjRWsgegT7WIGA6r/qB3KMCGQ6yvf4D+gTj0J4LqQNDfn20fdnwgHY+g\nv3+A/oD+gQH6Bw79HD5n+HuqA0FEVreBCAYCBiIrO3Q/GBgYLjvy+VlZ/8DA0LFDzh84/PzDf9eh\nxyK13SHbMFTvUcvJvo+0Pdo5g149u80BZWYTmyQaK/KDzwUxGG6napKoA8rMzMZkcHjwVMn1f0sk\nXSNpvaQNkm4e5bgkfTkdXyPpkppjmyStlfSYJL8m18xsgsmtByWpDNwGXA1sAVZJWhER62pOuxZY\nlH4uA76aPgddFRE78qqjmZkVV549qEuBDRGxMSJ6gTuBJSPOWQJ8KzKPANMkvSLHOpmZ2Wkiz4Ca\nA2yu2d+SysZ6TgD3S1otaWlutTQzs0Iq8iSJyyNiq6RZwH2SnoqIh0aelMJrKcD8+fNPdR3NzCwn\nefagtgLzavbnprIxnRMRg5/bgLvIhgwPExHLI6IjIjra2w9/it7MzE5PeQbUKmCRpIWSGoHrgBUj\nzlkBfCjN5vtFYE9EvCCpVVIbgKRW4K3AEznW1czMCia3Ib6IqEq6CbgXKAO3R8STkm5Mx5cBK4G3\nAxuAA8CH0+WzgbvSKwMqwLcj4p686mpmZsWjqF2/4jQnaTvw3El8xUzA09rdDrXcFhm3Q8btkDnZ\ndjg3Io55T+aMCqiTJakzIjrqXY96czsMc1tk3A4Zt0PmVLWDF7gyM7NCckCZmVkhOaAOtbzeFSgI\nt8Mwt0XG7ZBxO2ROSTv4HpSZmRWSe1BmZlZIDigzMyskB1RyrHdXnUkk3S5pm6QnasqmS7pP0s/S\n51k1x25J7bJe0tvqU+vxJ2mepAckrZP0pKTfS+UTqi0kNUt6VNLjqR3+LJVPqHYYJKks6T8k3Z32\nJ1w7jPY+vrq0Q/YK34n9Q7bSxTPAK4FG4HFgcb3rlePf+8vAJcATNWV/Adyctm8GPp+2F6f2aAIW\npnYq1/tvGKd2eAVwSdpuA55Of++EagtAwOS03QD8BPjFidYONe3xB8C3gbvT/oRrB2ATMHNE2Slv\nB/egMmN5d9UZI7JV4XeNKF4CfDNtfxN4V035nRHRExHPki1LNerCvaebiHghIv49be8Dfkr2upcJ\n1RaR6Uq7DeknmGDtACBpLvCrwNdqiidcOxzBKW8HB1RmLO+uOtPNjogX0vaLZOshwgRpG0kLgNeR\n9R4mXFukYa3HgG3AfRExIdsBuBX4L8BATdlEbIfR3sd3ytuhyO+DsjqJiJA0YZ4/kDQZ+Hvg9yNi\nb1qkGJg4bRER/cBrJU0jW6j5NSOOn/HtIOkdwLaIWC3pytHOmQjtkBz2Pr7ag6eqHdyDyozl3VVn\nupckvQIgfW5L5Wd020hqIAunv42I76biCdkWABHxMvAAcA0Trx3eBPyapE1kw/xvkXQHE68diNHf\nx3fK28EBlRnLu6vOdCuA307bvw38Q035dZKaJC0EFgGP1qF+405ZV+l/Az+NiL+sOTSh2kJSe+o5\nIWkScDXwFBOsHSLiloiYGxELyP4N+GFE/BYTrB105Pfxnfp2qPdskaL8kL2X6mmyGSh/XO/65Py3\nfgd4AegjGy++AZgB/AD4GXA/ML3m/D9O7bIeuLbe9R/HdricbKx9DfBY+nn7RGsL4CLgP1I7PAF8\nJpVPqHYY0SZXMjyLb0K1A9ls5sfTz5OD/x7Wox281JGZmRWSh/jMzKyQHFBmZlZIDigzMyskB5SZ\nmRWSA8rMzArJAWWWI0n9aUXowZ9xWylf0oLaFenNzjRe6sgsXwcj4rX1roTZ6cg9KLM6SO/b+Yv0\nzp1HJZ2XyhdI+qGkNZJ+IGl+Kp8t6a70zqbHJf1S+qqypP+V3uP0/bQSBJI+qew9V2sk3VmnP9Ps\npDigzPI50eqrAAABYUlEQVQ1acQQ3/trju2JiAuBvyJbRRvgK8A3I+Ii4G+BL6fyLwP/EhEXk73L\n68lUvgi4LSIuAF4Gfj2V3wy8Ln3PjXn9cWZ58koSZjmS1BURk0cp3wS8JSI2pgVrX4yIGZJ2AK+I\niL5U/kJEzJS0HZgbET0137GA7NUYi9L+p4GGiPhzSfcAXcD3gO/F8PuezE4b7kGZ1U8cYft49NRs\n9zN8X/lXgdvIelurJPl+s512HFBm9fP+ms8fp+2HyVbSBvgA8K9p+wfAx2Do5YJTj/SlkkrAvIh4\nAPg0MBU4rBdnVnT+vyqzfE1Kb6oddE9EDE41P0vSGrJe0PWp7BPA1yX9EbAd+HAq/z1guaQbyHpK\nHyNbkX40ZeCOFGICvhzZe57MTiu+B2VWB+keVEdE7Kh3XcyKykN8ZmZWSO5BmZlZIbkHZWZmheSA\nMjOzQnJAmZlZITmgzMyskBxQZmZWSP8fnYEijlW16U0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10f63deb8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(range(len(nn.cost_)), nn.cost_)\n",
    "plt.ylabel('Cost')\n",
    "plt.xlabel('Epochs')\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Back to slides]\n",
    "## Adding complexity and functionality\n",
    "So now we can calculate the gradient quickly, but we need to account for more optimization methods in this new space because the optmiization surface can be highly non-convex. We need to steer clear of local optima as best as possible.\n",
    "\n",
    "To start, let's add some momentum to the calculation and perform mini-batching of the gradient updates to help improve performance. Our new class can inherit from the old one. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Example adapted from https://github.com/rasbt/python-machine-learning-book/blob/master/code/ch12/ch12.ipynb\n",
    "# Original Author: Sebastian Raschka\n",
    "# This is the optional book we use in the course, excellent intuitions and straightforward programming examples\n",
    "# please note, however, that this code has been manipulated to reflect our assumptions and notation.\n",
    "class TLPVectorizedMiniBatch(TwoLayerPerceptronBase):\n",
    "    def __init__(self, alpha=0.0, decrease_const=0.0, shuffle=True, minibatches=1, **kwds):        \n",
    "        # need to add to the original initializer \n",
    "        self.alpha = alpha\n",
    "        self.decrease_const = decrease_const\n",
    "        self.shuffle = shuffle\n",
    "        self.minibatches = minibatches\n",
    "        # but keep other keywords\n",
    "        super().__init__(**kwds)\n",
    "    \n",
    "    def _feedforward(self, X, W1, W2):\n",
    "        \"\"\"Compute feedforward step\n",
    "        -----------\n",
    "        X : Input layer with original features.\n",
    "        W1: Weight matrix for input layer -> hidden layer.\n",
    "        W2: Weight matrix for hidden layer -> output layer.\n",
    "        ----------\n",
    "        a1-a3 : activations into layer (or output layer)\n",
    "        z1-z2 : layer inputs \n",
    "\n",
    "        \"\"\"\n",
    "        A1 = self._add_bias_unit(X, how='column')\n",
    "        Z1 = W1 @ A1.T\n",
    "        if self.activ_func == \"sigmoid\":\n",
    "            A2 = self._sigmoid(Z1)\n",
    "        else: \n",
    "            A2 = Z1\n",
    "        A2 = self._add_bias_unit(A2, how='row')\n",
    "        Z2 = W2 @ A2\n",
    "        A3 = self._sigmoid(Z2)\n",
    "        return A1, Z1, A2, Z2, A3\n",
    "    \n",
    "    def _get_gradient(self, A1, A2, A3, Z1, Z2, Y_enc, W1, W2):\n",
    "        \"\"\" Compute gradient step using backpropagation.\n",
    "        \"\"\"\n",
    "        # vectorized backpropagation\n",
    "        if self.cost_func == \"cross entropy\":\n",
    "            sigma3 = (A3-Y_enc)\n",
    "        else: # quadratic\n",
    "            sigma3 = -2*(Y_enc-A3)*A3*(1-A3)\n",
    "        \n",
    "        if self.activ_func == \"sigmoid\":\n",
    "            sigma2 = (W2.T @ sigma3)*A2*(1-A2)\n",
    "        else: # linear\n",
    "            sigma2 = (W2.T @ sigma3)     \n",
    "\n",
    "        grad2 = sigma3 @ A2.T\n",
    "        grad1 = sigma2[1:,:] @ A1\n",
    "        \n",
    "        # regularize weights that are not bias terms\n",
    "        grad1[:, 1:] += W1[:, 1:] * self.l2_C\n",
    "        grad2[:, 1:] += W2[:, 1:] * self.l2_C\n",
    "\n",
    "        return grad1, grad2\n",
    "    \n",
    "    def predict(self, X):\n",
    "        \"\"\"Predict class labels\"\"\"\n",
    "        _, _, _, _, A3 = self._feedforward(X, self.W1, self.W2)\n",
    "        y_pred = np.argmax(A3, axis=0)\n",
    "        return y_pred\n",
    "    \n",
    "    def fit(self, X, y, print_progress=0):\n",
    "        \"\"\" Learn weights from training data.\"\"\"\n",
    "        X_data, y_data = X.copy(), y.copy()\n",
    "        Y_enc = self._encode_labels(y)\n",
    "        \n",
    "        # init weights and setup matrices\n",
    "        self.n_features_ = X_data.shape[1]\n",
    "        self.n_output_ = Y_enc.shape[0]\n",
    "        self.W1, self.W2 = self._initialize_weights()\n",
    "        \n",
    "        delta_W1_prev = np.zeros(self.W1.shape) # for momentum\n",
    "        delta_W2_prev = np.zeros(self.W2.shape) # for momentum\n",
    "\n",
    "        self.cost_ = []\n",
    "        for i in range(self.epochs):\n",
    "\n",
    "            # adaptive learning rate\n",
    "            self.eta /= (1 + self.decrease_const*i) # decreasing learning rate\n",
    "            # there are many forms of adaptive learning rates out there!\n",
    "\n",
    "            if print_progress>0 and (i+1)%print_progress==0:\n",
    "                sys.stderr.write('\\rEpoch: %d/%d' % (i+1, self.epochs))\n",
    "                sys.stderr.flush()\n",
    "\n",
    "            if self.shuffle:\n",
    "                ridx = np.random.permutation(y_data.shape[0])\n",
    "                X_data, Y_enc = X_data[ridx], Y_enc[:, ridx]\n",
    "\n",
    "            # use numpy split to split into equal sized batches \n",
    "            # num batches== self.minibatches\n",
    "            mini = np.array_split(range(y_data.shape[0]), self.minibatches)\n",
    "            mini_cost = []\n",
    "            for idx in mini:\n",
    "\n",
    "                # feedforward\n",
    "                A1, Z1, A2, Z2, A3 = self._feedforward(X_data[idx],\n",
    "                                                       self.W1,\n",
    "                                                       self.W2)\n",
    "                \n",
    "                cost = self._cost(A3,Y_enc[:, idx],self.W1,self.W2)\n",
    "                mini_cost.append(cost) # this appends cost of mini-batch only\n",
    "\n",
    "                # compute gradient via backpropagation\n",
    "                grad1, grad2 = self._get_gradient(A1=A1, A2=A2, A3=A3, Z1=Z1, Z2=Z2,\n",
    "                                                  Y_enc=Y_enc[:, idx],\n",
    "                                                  W1=self.W1, W2=self.W2)\n",
    "\n",
    "                delta_W1, delta_W2 = self.eta * grad1, self.eta * grad2\n",
    "                self.W1 -= (delta_W1 + (self.alpha * delta_W1_prev)) # update with momentum\n",
    "                self.W2 -= (delta_W2 + (self.alpha * delta_W2_prev)) # update with momentum\n",
    "                delta_W1_prev, delta_W2_prev = delta_W1, delta_W2\n",
    "\n",
    "            self.cost_.append(mini_cost)\n",
    "            \n",
    "        return self"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epoch: 200/200"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.122222222222\n",
      "CPU times: user 7 s, sys: 125 ms, total: 7.12 s\n",
      "Wall time: 3.93 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "nn_mini = TLPVectorizedMiniBatch(**params,\n",
    "                          alpha=0.001,# momentum calculation\n",
    "                          decrease_const=0.0001, # decreasing eta\n",
    "                          minibatches=50, # minibatch size\n",
    "                          shuffle=True)\n",
    "\n",
    "    \n",
    "nn_mini.fit(X_train, y_train, print_progress=5)\n",
    "yhat = nn.predict(X_test)\n",
    "print('Accuracy:',accuracy_score(y_test,yhat))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAH1BJREFUeJzt3Xu8F3W97/HXmwUIigroOoiAgsVOyQvWytrt9t6paWgX\nopuw3cb2WIimhVkJ1elh7S5WmuaJZGNxNmaKXTR5JGlK7tM5qcmCjRdUYnlL2FyWV/RoXD/nj5kl\nP38u1vyANb+ZtX7v5+Mxj5n5zmV9Z0zffb+/78woIjAzMyubPkVXwMzMrDMOKDMzKyUHlJmZlZID\nyszMSskBZWZmpeSAMjOzUnJAmZlZKTmgzMyslBxQZmZWSn2LrkA9HHjggTF69Oiiq2FmZsDSpUuf\njojmrP0aIqBGjx5Na2tr0dUwMzNA0pO17OcuPjMzKyUHlJmZlZIDyszMSskBZWZmpeSAMjOzUnJA\nmZlZKTmgzMyslBxQWR54ABYtKroWZmYNxwGV5eqr4fTTi66FmVnDcUBl6dMHIoquhZlZw3FAZZFg\n+/aia2Fm1nAcUFkkt6DMzArggMriLj4zs0I4oLK4i8/MrBAOqCxuQZmZFcIBlcUtKDOzQuQaUJIm\nSFopqU3SzE62ny7pfkkPSLpL0jFp+ShJd0p6SNIKSZ+tOOZiSWskLU+nU/O8Bg+SMDMrRm5f1JXU\nBMwGTgJWA0skLYyIhyp2exz4x4h4TtIpwFzg7cBW4MKIWCZpX2CppNsrjr08Ii7Nq+6v4S4+M7NC\n5NmCOg5oi4jHImIzsACYWLlDRNwVEc+lq/cAI9PytRGxLF1+EXgYGJFjXXfOXXxmZoXIM6BGAE9V\nrK+m65A5C/htdaGk0cCxwJ8qis9PuwbnSRqy51Xtgrv4zMwKUYpBEpKOJwmoi6rKBwG/AmZExMa0\n+CrgMGA8sBa4bCfnnCapVVJre3v77lfOXXxmZoXIM6DWAKMq1kemZa8h6Wjgx8DEiHimorwfSTj9\nLCJu7CiPiPURsS0itgNXk3Qlvk5EzI2IlohoaW5u3v2rkDpOuPvnMDOzXZZnQC0BxkoaI6k/MBlY\nWLmDpEOAG4EzIuLPFeUCfgI8HBHfrzpmeMXqJODBnOrf8QeTuQPKzKyuchvFFxFbJZ0H3AY0AfMi\nYoWk6en2OcBXgQOAHyWZxNaIaAH+DjgDeEDS8vSUX4qIRcB3JY0HAngCODuvawCSLj5IBkr0KUWP\nqJlZQ8gtoADSQFlUVTanYvmTwCc7Oe7/AtrJOc/o5mp2zS0oM7NCuEmQpaPV5IAyM6srB1SWjhaU\nn4UyM6srB1QWd/GZmRXCAZXFXXxmZoVwQGVxF5+ZWSEcUFncxWdmVggHVBZ38ZmZFcIBlcVdfGZm\nhXBAZXELysysEA6oLG5BmZkVwgGVxYMkzMwK4YDK4i4+M7NCOKCyuIvPzKwQDqgs7uIzMyuEAyqL\nu/jMzArhgMriLj4zs0I4oLK4BWVmVggHVBa3oMzMCuGAyuJBEmZmhXBAZXEXn5lZIRxQWdzFZ2ZW\nCAdUFnfxmZkVwgGVxV18ZmaFyDWgJE2QtFJSm6SZnWw/XdL9kh6QdJekY7KOlTRU0u2SVqXzIXle\ng7v4zMyKkVtASWoCZgOnAOOAKZLGVe32OPCPEXEU8K/A3BqOnQksjoixwOJ0PT9uQZmZFSLPFtRx\nQFtEPBYRm4EFwMTKHSLiroh4Ll29BxhZw7ETgfnp8nzgQzleg1tQZmYFyTOgRgBPVayvTst25izg\ntzUcOywi1qbL64BhnZ1M0jRJrZJa29vbd7XulSdK5m5BmZnVVSkGSUg6niSgLtqV4yIigE6TIyLm\nRkRLRLQ0NzfvfuXcxWdmVog8A2oNMKpifWRa9hqSjgZ+DEyMiGdqOHa9pOHpscOBDd1c7+oKJnN3\n8ZmZ1VWeAbUEGCtpjKT+wGRgYeUOkg4BbgTOiIg/13jsQmBqujwVuDnHa3AXn5lZQfrmdeKI2Crp\nPOA2oAmYFxErJE1Pt88BvgocAPxISRBsTbvlOj02PfUlwM8lnQU8CXw8r2sA3MVnZlaQ3AIKICIW\nAYuqyuZULH8S+GStx6blzwAndm9Nu+AuPjOzQpRikESpuYvPzKwQDqgsHV18bkGZmdWVAyqLW1Bm\nZoVwQGXxIAkzs0I4oLJ4kISZWSEcUFncxWdmVggHVBZ38ZmZFcIBlcVdfGZmhXBAZXEXn5lZIRxQ\nWdzFZ2ZWCAdUFnfxmZkVwgGVxS0oM7NCOKCyuAVlZlYIB1QWD5IwMyuEAyqLu/jMzArhgMriLj4z\ns0I4oLK4i8/MrBAOqCzu4jMzK4QDKou7+MzMCuGAyuIWlJlZIRxQWdyCMjMrhAMqiwdJmJkVIteA\nkjRB0kpJbZJmdrL9cEl3S9ok6fMV5W+StLxi2ihpRrrtYklrKradmuc1uIvPzKwYffM6saQmYDZw\nErAaWCJpYUQ8VLHbs8BngA9VHhsRK4HxFedZA9xUscvlEXFpXnV/DXfxmZkVIs8W1HFAW0Q8FhGb\ngQXAxModImJDRCwBtnRxnhOBRyPiyfyq2gV38ZmZFSLPgBoBPFWxvjot21WTgeurys6XdL+keZKG\ndHaQpGmSWiW1tre378afTbmLz8ysEKUeJCGpP/BB4BcVxVcBh5F0Aa4FLuvs2IiYGxEtEdHS3Ny8\nJ5VI5u7iMzOrqzwDag0wqmJ9ZFq2K04BlkXE+o6CiFgfEdsiYjtwNUlXYn7cgjIzK0SeAbUEGCtp\nTNoSmgws3MVzTKGqe0/S8IrVScCDe1TLLG5BmZkVIrdRfBGxVdJ5wG1AEzAvIlZImp5unyPpIKAV\n2A/Yng4lHxcRGyXtQzIC8OyqU39X0ngggCc62d69PEjCzKwQuQUUQEQsAhZVlc2pWF5H0vXX2bH/\nDzigk/IzurmaXXMXn5lZIUo9SKIU3MVnZlYIB1QWd/GZmRXCAZXFXXxmZoVwQGVxF5+ZWSEcUFnc\nxWdmVggHVJaOLj63oMzM6soBlcUtKDOzQjigsniQhJlZIRxQWTxIwsysEA6oLO7iMzMrhAMqi7v4\nzMwK4YDK4i4+M7NCOKCyuIvPzKwQDqgsfg7KzKwQDqgsbkGZmRXCAZXFgyTMzArhgMriQRJmZoVw\nQGVxF5+ZWSFqCihJP62lrFdyF5+ZWSFqbUG9uXJFUhPw1u6vTgm5i8/MrBBdBpSkWZJeBI6WtDGd\nXgQ2ADfXpYZFcxefmVkhugyoiPh2ROwLfC8i9kunfSPigIiYVac6FstdfGZmhai1i+83kvYBkPTP\nkr4v6dAc61Ue7uIzMytErQF1FfCypGOAC4FHgWuyDpI0QdJKSW2SZnay/XBJd0vaJOnzVduekPSA\npOWSWivKh0q6XdKqdD6kxmvYPW5BmZkVotaA2hoRAUwEfhgRs4F9uzogHUgxGzgFGAdMkTSuardn\ngc8Al+7kNMdHxPiIaKkomwksjoixwOJ0PT9uQZmZFaLWgHpR0izgDOAWSX2AfhnHHAe0RcRjEbEZ\nWEAScK+KiA0RsQTYsgt1ngjMT5fnAx/ahWN3nQdJmJkVotaAOg3YBPz3iFgHjAS+l3HMCOCpivXV\naVmtArhD0lJJ0yrKh0XE2nR5HTCss4MlTZPUKqm1vb19F/7s606U1sYBZWZWTzUFVBpKPwP2l/R+\n4K8Rkfkb1B56V0SMJ+ki/LSkf+ikXkESZK8TEXMjoiUiWpqbm3e/Fu7iMzMrRK1vkvg4cC/wMeDj\nwJ8kfTTjsDXAqIr1kWlZTSJiTTrfANxE0mUIsF7S8LRew0meycqX5BaUmVmd1drF92XgbRExNSI+\nQRIW/yPjmCXAWEljJPUHJgMLa/ljkvaRtG/HMnAy8GC6eSEwNV2eSj0eGO7TxwFlZlZnfWvcr0/a\nkunwDNkP+W6VdB5wG9AEzIuIFZKmp9vnSDoIaAX2A7ZLmkEy4u9A4CYl3Wt9gesi4tb01JcAP5d0\nFvAkSYsuX5K7+MzM6qzWgLpV0m3A9en6acCirIMiYlH1fhExp2K5Y8BFtY3AMTs55zPAibVVu5u4\nBWVmVnddBpSkN5KMmvuCpA8D70o33U0yaKIxuAVlZlZ3WS2oK4BZABFxI3AjgKSj0m0fyLV2ZeFB\nEmZmdZc1SGJYRDxQXZiWjc6lRmXkLj4zs7rLCqjBXWwb2J0VKTV38ZmZ1V1WQLVK+lR1oaRPAkvz\nqVIJuYvPzKzusn6DmkEy3Pt0dgRSC9AfmJRnxUrFXXxmZnXXZUBFxHrgnZKOB45Mi2+JiN/nXrMy\ncRefmVnd1fQcVETcCdyZc13Ky118ZmZ1V+urjhpbnz5uQZmZ1ZkDqhZuQZmZ1Z0DqhYDB8LLLxdd\nCzOzhuKAqsVBB8Hatdn7mZlZt3FA1WL4cAeUmVmdOaBqcfDBDigzszpzQNVi+HBob4ctW4quiZlZ\nw3BA1WL48GS+bl2x9TAzayAOqFocfHAyf+qpYuthZtZAHFC1ePvbk4d1b7ml6JqYmTUMB1Qtmpvh\nhBPguutg27aia2Nm1hAcULWaPh2eeAJ++cuia2Jm1hAcULWaNAkOPxy+/W2/9sjMrA4cULXq0wdm\nzoT77oNFi4qujZlZr5drQEmaIGmlpDZJMzvZfrikuyVtkvT5ivJRku6U9JCkFZI+W7HtYklrJC1P\np1PzvIbX+Kd/gkMPhW98w60oM7Oc5RZQkpqA2cApwDhgiqRxVbs9C3wGuLSqfCtwYUSMA94BfLrq\n2MsjYnw61a85068fXHQR3HMP3Nm4n8cyM6uHPFtQxwFtEfFYRGwGFgATK3eIiA0RsQTYUlW+NiKW\npcsvAg8DI3Ksa+3OPDN5cPeb3yy6JmZmvVqeATUCqHyydTW7ETKSRgPHAn+qKD5f0v2S5kkaspPj\npklqldTa3t6+q3925wYMgAsvhN//Hu6+u/vOa2Zmr1HqQRKSBgG/AmZExMa0+CrgMGA8sBa4rLNj\nI2JuRLREREtzc3P3Vuzss2HoULeizMxylGdArQFGVayPTMtqIqkfSTj9LCJu7CiPiPURsS0itgNX\nk3Ql1tegQXDBBcmbJZYvr/ufNzNrBHkG1BJgrKQxkvoDk4GFtRwoScBPgIcj4vtV24ZXrE4CHuym\n+u6a886D/faDb32rkD9vZtbb9c3rxBGxVdJ5wG1AEzAvIlZImp5unyPpIKAV2A/YLmkGyYi/o4Ez\ngAckdTRRvpSO2PuupPFAAE8AZ+d1DV0aPBg+/Wm45BJ45JHkIV4zM+s2igZ4nqelpSVaW1u7/8Tt\n7clzUR/7GMyf3/3nNzPrhSQtjYiWrP1KPUii9Jqb4VOfSl4i609xmJl1KwfUnrrgguStEldcUXRN\nzMx6FQfUnho9Gk47DebOheefL7o2Zma9hgOqO3zhC/DSSzBnTtE1MTPrNRxQ3WH8eDj5ZPjBD+Cv\nfy26NmZmvYIDqrt88Yuwbh1ce23RNTEz6xUcUN3lhBPgLW+B730Ptm8vujZmZj2eA6q7SEkr6s9/\nhoU1vTDDzMy64IDqTh/5CIwZk7z+qAEegDYzy5MDqjv17QuzZsGSJXDbbUXXxsysR3NAdbepU+GQ\nQ+BrX3MrysxsDzigulv//kkr6p574I47iq6NmVmP5YDKw5lnwsiRbkWZme0BB1Qe9toraUX98Y9w\n++1F18bMrEdyQOXlrLOST3HMmuXnoszMdoMDKi977QVf/zosWwa/+EXRtTEz63EcUHk6/XQ48kj4\nyldgy5aia2Nm1qM4oPLU1JR8Er6tDa68sujamJn1KA6ovJ16KrzvfXDxxbBmTdG1MTPrMRxQeZOS\n1tOWLXDhhUXXxsysx3BA1cNhhyWj+W64ARYvLro2ZmY9ggOqXi66CN7wBjj3XHjllaJrY2ZWerkG\nlKQJklZKapM0s5Pth0u6W9ImSZ+v5VhJQyXdLmlVOh+S5zV0mwED4N/+Lfkcx9e+VnRtzMxKL7eA\nktQEzAZOAcYBUySNq9rtWeAzwKW7cOxMYHFEjAUWp+s9w4knJg/wXnopLF1adG3MzEotzxbUcUBb\nRDwWEZuBBcDEyh0iYkNELAGqHxLq6tiJwPx0eT7wobwuIBeXXgrDhsEnPgEvv1x0bczMSivPgBoB\nPFWxvjot29Njh0XE2nR5HTBsTypZd4MHw7//Ozz0EMyYUXRtzMxKq0cPkoiIADp9XbikaZJaJbW2\nt7fXuWYZTjoJZs6Eq6+GBQuKro2ZWSnlGVBrgFEV6yPTsj09dr2k4QDpfENnJ4iIuRHREhEtzc3N\nu1Txuvj61+Fv/xamTYNVq4qujZlZ6eQZUEuAsZLGSOoPTAYWdsOxC4Gp6fJU4OZurHP99OsH11+f\nzD/4QXjhhaJrZGZWKrkFVERsBc4DbgMeBn4eESskTZc0HUDSQZJWA58DviJptaT9dnZseupLgJMk\nrQLek673TIceCr/6VfKuvsmTYevWomtkZlYaigb44mtLS0u0trYWXY2dmzsXzj47GTRx+eVF18bM\nLFeSlkZES9Z+fetRGcswbVoyqu+KK+CQQ+CCC4qukZlZ4RxQZXHZZbB6NXzuc3DQQTBlStE1MjMr\nlAOqLJqa4Npr4emnYepUGDoU3vveomtlZlaYHv0cVK8zYAD8+tcwbhxMnAi33lp0jczMCuOAKpvB\ng5NPchxxRBJSt9xSdI3MzArhgCqjAw5IQuqoo2DSJLi5Zz7qZWa2JxxQZTV0KNxxBxx7LHz0o34l\nkpk1HAdUmQ0eDL/7XfJKpClT4DvfgQZ4bs3MDBxQ5bf//klInXZa8oLZc8/1GyfMrCF4mHlPMGAA\nXHcdjB6dtKKeeip5j9+++xZdMzOz3LgF1VP06QOXXAJXXQW//S28/e3w8MNF18rMLDcOqJ5m+vRk\n8MQzz8Db3gY33FB0jczMcuGA6omOPx6WLYNjjknegj5jBmzeXHStzMy6lQOqpxoxAv7jP5Jw+sEP\n4N3vTt7lZ2bWSzigerJ+/ZLPc9xwA9x/f/Jg77XXeii6mfUKDqje4OMfh+XLk3f4nXFG8mBve3vR\ntTIz2yMOqN7ijW+EP/whGYb+m9/Am9+cvHjWzKyHckD1Jk1N8MUvwtKlMHJk8h6/SZPgySeLrpmZ\n2S5zQPVGRx4J99yTPDf1u98lb0b/9rc90s/MehQHVG/Vvz9cdFHyMO+pp8KXvgRHH5085OtBFGbW\nAzigertDDoFf/jIJpm3bkrD6+79PhqibmZWYA6pRTJgAK1Ykr0p6/PHkYd+TTkq6As3MSsgB1Uj6\n909eldTWljw/dd99yac8PvABuPtud/2ZWankGlCSJkhaKalN0sxOtkvSlen2+yW9JS1/k6TlFdNG\nSTPSbRdLWlOx7dQ8r6FXGjgweQPFY48lgyf++Ed45zuTd/tdcw1s2lR0Dc3M8gsoSU3AbOAUYBww\nRdK4qt1OAcam0zTgKoCIWBkR4yNiPPBW4GXgporjLu/YHhGL8rqGXm/QoOQbU3/5C/zoR/DyyzB1\nKowaBV/5SlJuZlaQPFtQxwFtEfFYRGwGFgATq/aZCFwTiXuAwZKGV+1zIvBoRPhhnrwMGgTnnJP8\nRnX77Um337e+BYcemgyomD0b1q8vupZm1mDyDKgRwFMV66vTsl3dZzJwfVXZ+WmX4DxJQzr745Km\nSWqV1Nru1/7URoL3vAduvhkefRS+8Q147jk47zw4+GA4+WSYNw+ef77omppZAyj1IAlJ/YEPAr+o\nKL4KOAwYD6wFLuvs2IiYGxEtEdHS3Nyce117nTFj4MtfhgcfhAcegFmzkt+szjoLhg2D978ffvhD\nWLXKgyvMLBd5BtQaYFTF+si0bFf2OQVYFhGv9i9FxPqI2BYR24GrSboSLU9HHpm0platgiVLkhbV\nI4/A+efD3/wNvOENcO65ybv/nnmm6NqaWS+RZ0AtAcZKGpO2hCYDC6v2WQh8Ih3N9w7ghYhYW7F9\nClXde1W/UU0CHuz+qlunJGhpgcsuS4aqt7Ulv08ddRT89KfJe/8OPDAJtHPOST798fjjbmGZ2W5R\n5Pgfj3QI+BVAEzAvIr4paTpARMyRJOCHwASSkXpnRkRreuw+wF+AwyLihYpz/pSkey+AJ4Czq0Lt\ndVpaWqK1tbW7L88qbd4M996bvFH9D3+Au+6CF19Mtg0ZAuPHJ18AHj8+mY44Inkuy8wajqSlEdGS\nuV+eAVUWDqgCbNuWjAq8667kW1XLlycfVXzllWR7v37J96uOOSb5NMgRRyTro0cnb2U3s16r1oDq\nW4/KWANqakpeTnv00TvKtm1LugU7Amv58mRY+zXX7NhnwIDkd60xY5Jh7tXTgQcmXY1m1us5oKx+\nmprgTW9KptNO21H+/PPJoIuHHkrevv7ww0mQLV4ML7302nPsvXfyAtyOwBo1KhkCP3z4jvnQodDX\n/9M26+n8b7EVb/BgeMc7kqlSRPIc1pNPvn76y19g2bKdf9p+//3hgAOSsOqYV04dZUOGJH9///2T\naZ993EIzKwkHlJWXtCNQjj228302bYJ16+C//iuZ1q5NhrpXTs8+m7TInn02aa119btrnz6w775J\nWO23X/KWjb33Tt5fuPfeO6b99kumgQOT39MGDEimgQNfOx8wIGnN9euXzDumyvXK5T59HJBmKQeU\n9Wx77bWju68W27bBCy/sCK7nnkvWX3gBNm7cMe+YXnwxGdjx/PPJuwpfeSXpdty4EbZsyeeadhZe\nXQVbx9TUlIRcx7yzqXpbZ4EovX7aWfnO9qvcv7PzZ6131K1jOSL557d9e7JcvU9ZlKkuu6Oy/tX/\nPDu89a3w4Q/nXhUHlDWWpqYdrbI9tWkT/PWvybxj+ZVXknnl8tatO6YtW3a+3tW2WvbtmDr+I145\nbdv22v/AV5ZXi3j9tLPyne1XuX9n589a75g6wmj79teHauU+ZVG2UdEdQb4r+1cvV5dJ8C//4oAy\nK7W99komM8tFqd/FZ2ZmjcsBZWZmpeSAMjOzUnJAmZlZKTmgzMyslBxQZmZWSg4oMzMrJQeUmZmV\nkgPKzMxKqSE+WCipHXhyD05xIPB0N1Wnp/I98D0A3wPwPeiwJ/fh0IhoztqpIQJqT0lqreXrj72Z\n74HvAfgegO9Bh3rcB3fxmZlZKTmgzMyslBxQtZlbdAVKwPfA9wB8D8D3oEPu98G/QZmZWSm5BWVm\nZqXkgDIzs1JyQGWQNEHSSkltkmYWXZ+8SJonaYOkByvKhkq6XdKqdD6kYtus9J6slPTeYmrdvSSN\nknSnpIckrZD02bS8Ye6DpAGS7pV0X3oPvpaWN8w9AJDUJOk/Jf0mXW+o6weQ9ISkByQtl9SaltX3\nPkSEp51MQBPwKHAY0B+4DxhXdL1yutZ/AN4CPFhR9l1gZro8E/hOujwuvRd7AWPSe9RU9DV0wz0Y\nDrwlXd4X+HN6rQ1zHwABg9LlfsCfgHc00j1Ir+tzwHXAb9L1hrr+9NqeAA6sKqvrfXALqmvHAW0R\n8VhEbAYWABMLrlMuIuIPwLNVxROB+enyfOBDFeULImJTRDwOtJHcqx4tItZGxLJ0+UXgYWAEDXQf\nIvFSutovnYIGugeSRgLvA35cUdww15+hrvfBAdW1EcBTFeur07JGMSwi1qbL64Bh6XKvvy+SRgPH\nkrQgGuo+pN1by4ENwO0R0Wj34Argi8D2irJGuv4OAdwhaamkaWlZXe9D3z09gTWGiAhJDfFMgqRB\nwK+AGRGxUdKr2xrhPkTENmC8pMHATZKOrNrea++BpPcDGyJiqaR3d7ZPb77+Ku+KiDWS/htwu6RH\nKjfW4z64BdW1NcCoivWRaVmjWC9pOEA635CW99r7IqkfSTj9LCJuTIsb7j4ARMTzwJ3ABBrnHvwd\n8EFJT5B06Z8g6Voa5/pfFRFr0vkG4CaSLru63gcHVNeWAGMljZHUH5gMLCy4TvW0EJiaLk8Fbq4o\nnyxpL0ljgLHAvQXUr1spaSr9BHg4Ir5fsalh7oOk5rTlhKSBwEnAIzTIPYiIWRExMiJGk/z7/vuI\n+Gca5Po7SNpH0r4dy8DJwIPU+z4UPVKk7BNwKslorkeBLxddnxyv83pgLbCFpP/4LOAAYDGwCrgD\nGFqx/5fTe7ISOKXo+nfTPXgXSb/7/cDydDq1ke4DcDTwn+k9eBD4alreMPeg4rrezY5RfA11/SQj\nl+9LpxUd/+2r933wq47MzKyU3MVnZmal5IAyM7NSckCZmVkpOaDMzKyUHFBmZlZKDiizOpG0LX0z\ndMfUbW/HlzS68k30Zr2BX3VkVj+vRMT4oith1lO4BWVWsPS7O99Nv71zr6Q3puWjJf1e0v2SFks6\nJC0fJumm9JtN90l6Z3qqJklXp99x+l36JggkfUbJN67ul7SgoMs022UOKLP6GVjVxXdaxbYXIuIo\n4Ickb9MG+J/A/Ig4GvgZcGVafiXwvyPiGJJveK1Iy8cCsyPizcDzwEfS8pnAsel5pud1cWbdzW+S\nMKsTSS9FxKBOyp8AToiIx9KX1a6LiAMkPQ0Mj4gtafnaiDhQUjswMiI2VZxjNMmnMcam6xcB/SLi\nG5JuBV4Cfg38OnZ878ms1NyCMiuH2MnyrthUsbyNHb8xvw+YTdLaWiLJvz1bj+CAMiuH0yrmd6fL\nd5G8URvgdOD/pMuLgXPg1Y8L7r+zk0rqA4yKiDuBi4D9gde14szKyP9Pyqx+BqZfqu1wa0R0DDUf\nIul+klbQlLTsfOB/SfoC0A6cmZZ/Fpgr6SySltI5JG+i70wTcG0aYgKujOQ7T2al59+gzAqW/gbV\nEhFPF10XszJxF5+ZmZWSW1BmZlZKbkGZmVkpOaDMzKyUHFBmZlZKDigzMyslB5SZmZXS/wdOoRK9\n3LIKNQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10f9d6ef0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# take the mean of each minibatch epoch\n",
    "cost_avgs = [np.mean(x) for x in nn_mini.cost_]\n",
    "\n",
    "plt.plot(range(len(cost_avgs)), cost_avgs, color='red')\n",
    "plt.ylabel('Cost')\n",
    "plt.xlabel('Epochs')\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using different optimization techniques\n",
    "While the above implementation is good for a number of applications, we have yet to use more advanced optimization algorithms. Why is that? Shouldn't we remedy this situation?\n",
    "\n",
    "### Self-test: Should we try quasi-Newton methods on the MLP?\n",
    "- A. Yes. Quasi-Newton methods converge much faster\n",
    "- B. Yes. Quasi-Newton methods are guaranteed to find the global optimum\n",
    "- C. No. There is no gurantee that Quasi-Newton methods will work better in a non-convex space.\n",
    "- D. No. With so many weights, the Hessian calculation is too complex. \n",
    "\n",
    "___\n",
    "\n",
    "## Quasi-Newton updates for the MLP\n",
    "Let's try to perform L-BFGS with on the previous models. The easy way of performing this is to add wrapper methods to the `_get_gradient` function in order to pack and unpack the data into the Weights matrices.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from scipy.optimize import fmin_bfgs\n",
    "from scipy.optimize import fmin_l_bfgs_b\n",
    "class TLPVectorizedBFGS(TwoLayerPerceptronVectorized):\n",
    "    \n",
    "    def __init__(self, gtol=1e-5, **kwds):        \n",
    "        # need to add to the original initializer \n",
    "        self.gtol = gtol\n",
    "        \n",
    "        # but keep other keywords\n",
    "        super().__init__(**kwds)\n",
    "        \n",
    "    @staticmethod\n",
    "    def _pack(in1, in2):\n",
    "        '''Pack and flatten input vectors '''\n",
    "        return np.hstack((in1.flatten(),in2.flatten()))\n",
    "    \n",
    "    def _unpack(self, in_tot):\n",
    "        '''Undo packing according to layer weight sizes'''\n",
    "        out1 = in_tot[:self.W1.size].reshape(self.W1.shape)\n",
    "        out2 = in_tot[self.W1.size:].reshape(self.W2.shape)\n",
    "        return out1, out2\n",
    "    \n",
    "    def _calc_cost_gradient_packed(self,W,X_data,Y_enc):\n",
    "        '''Unpack and get cost, gradient for bfgs'''\n",
    "        W1, W2 = self._unpack(W) \n",
    "        # feedforward all instances\n",
    "        A1, Z1, A2, Z2, A3 = self._feedforward(X_data,W1,W2)\n",
    "        \n",
    "        cost = np.sum((Y_enc-A3)**2)\n",
    "        L2_term = self._L2_reg(self.l2_C, W1, W2)\n",
    "        cost = cost + L2_term\n",
    "        #perform back prop to get gradients\n",
    "        grad1,grad2 = self._get_gradient(A1=A1, A2=A2, A3=A3,Z1=Z1, Z2=Z2, Y_enc=Y_enc,\n",
    "                                         W1=W1, W2=W2)\n",
    "        return cost, self._pack(grad1,grad2)\n",
    "    \n",
    "    def _cost_packed(self,W,X_data,Y_enc):\n",
    "        '''Unpack and calculate MSE for bfgs'''\n",
    "        W1, W2 = self._unpack(W)\n",
    "        _, _, _, _, A3 = self._feedforward(X_data,W1,W2)\n",
    "        return np.sum((Y_enc-A3)**2)\n",
    "    \n",
    "    def fit(self,X,y,print_progress=0):\n",
    "        '''Learn weights from training data'''\n",
    "        X_data, y_data = X.copy(), y.copy()\n",
    "        Y_enc = self._encode_labels(y)\n",
    "        \n",
    "        # init weights and setup matrices\n",
    "        self.n_features_ = X_data.shape[1]\n",
    "        self.n_output_ = Y_enc.shape[0]\n",
    "        self.W1, self.W2 = self._initialize_weights()\n",
    "        \n",
    "        # make initial matrices into single row vector\n",
    "        W = self._pack(self.W1,self.W2)\n",
    "        \n",
    "        if print_progress>0:\n",
    "            def callback(xd):\n",
    "                callback.counter += 1\n",
    "                if callback.counter%print_progress==0:\n",
    "                    sys.stderr.write('\\rEpoch: %d/%d (max)' % (callback.counter,callback.epochs))\n",
    "                    sys.stderr.flush()\n",
    "\n",
    "            callback.counter = 0\n",
    "            callback.epochs = self.epochs\n",
    "            \n",
    "        else:\n",
    "            callback = None\n",
    "            \n",
    "        # compute gradient optimum with bfgs\n",
    "        W_best,_,props = fmin_l_bfgs_b(\n",
    "                        x0=W,\n",
    "                        func=self._calc_cost_gradient_packed,\n",
    "                        maxfun=self.epochs,\n",
    "                        callback=callback,\n",
    "                        pgtol=self.gtol,\n",
    "                        args=(X_data, Y_enc))\n",
    "        \n",
    "        self.W1, self.W2 = self._unpack(W_best)\n",
    "        if print_progress:\n",
    "            print(props)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epoch: 204/500 (max)"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'warnflag': 0, 'grad': array([-0.0193883 , -0.00119921,  0.00149961, ...,  0.03169827,\n",
      "       -0.01317503,  0.02611571]), 'funcalls': 363, 'task': b'CONVERGENCE: REL_REDUCTION_OF_F_<=_FACTR*EPSMCH', 'nit': 204}\n",
      "Accuracy: 0.986111111111\n",
      "CPU times: user 2.17 s, sys: 222 ms, total: 2.39 s\n",
      "Wall time: 2.12 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "nn_bfgs = TLPVectorizedBFGS(**params, gtol=1e-3)\n",
    "\n",
    "nn_bfgs.fit(X_train, y_train, print_progress=1)\n",
    "yhat = nn_bfgs.predict(X_test)\n",
    "print('Accuracy:',accuracy_score(y_test,yhat))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow! That was pretty fast compared to the other implementations we have seen. But don't let this fool you. The L-BFGS implementation is great for smaller datasets, but minibatch/stochastic methods are more optimal as the number of instances in our data gets larger. Eventually they over power the L-BFGS algorithm's ability to move along the curvature effectively and efficiently.  \n",
    "___\n",
    "# Using a bigger handwriting dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# more data for handwriting recognition?\n",
    "# Let's use Raschka's implementation for using the mnist dataset:\n",
    "# https://github.com/rasbt/python-machine-learning-book/blob/master/code/ch12/ch12.ipynb\n",
    "import os\n",
    "import struct\n",
    "import numpy as np\n",
    " \n",
    "def load_mnist(path, kind='train'):\n",
    "    \"\"\"Load MNIST data from `path`\"\"\"\n",
    "    labels_path = os.path.join(path, '%s-labels.idx1-ubyte' % kind)\n",
    "    images_path = os.path.join(path, '%s-images.idx3-ubyte' % kind)\n",
    "        \n",
    "    with open(labels_path, 'rb') as lbpath:\n",
    "        magic, n = struct.unpack('>II', lbpath.read(8))\n",
    "        labels = np.fromfile(lbpath, dtype=np.uint8)\n",
    "\n",
    "    with open(images_path, 'rb') as imgpath:\n",
    "        magic, num, rows, cols = struct.unpack(\">IIII\", imgpath.read(16))\n",
    "        images = np.fromfile(imgpath, dtype=np.uint8).reshape(len(labels), 784)\n",
    " \n",
    "    return images, labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rows: 60000, columns: 784\n",
      "Rows: 10000, columns: 784\n"
     ]
    }
   ],
   "source": [
    "X_train, y_train = load_mnist('data/', kind='train')\n",
    "print('Rows: %d, columns: %d' % (X_train.shape[0], X_train.shape[1]))\n",
    "\n",
    "X_test, y_test = load_mnist('data/', kind='t10k')\n",
    "print('Rows: %d, columns: %d' % (X_test.shape[0], X_test.shape[1]))\n",
    "\n",
    "# don't forget to normalize\n",
    "X_train = X_train/255.0\n",
    "X_test = X_test/255.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHiFJREFUeJzt3XmcndP9wPFPaol9D9WiscTWWqO28qMIIUrtUcReYt+p\ntrZYqwhiX0PUWpRWa19KEQa1i6X2fYt9CfP7w+v7POfO3MRMcufeM3c+73/y9DzP3DlOnzvf55zn\nnO/p1draiiRJuflBoysgSVI1BihJUpYMUJKkLBmgJElZMkBJkrJkgJIkZckAJUnKkgFKkpQlA5Qk\nKUuTd+bi2WabrbVv375dVJXuo6Wl5d3W1tY+k/o5tud3bM/aq0Wb2p4l79Ha6mh7dipA9e3blwcf\nfHDia9UkevXq9VItPsf2/I7tWXu1aFPbs+Q9WlsdbU+H+CRJWTJASZKyZICSJGXJACVJypIBSpKU\nJQOUJClLBihJUpYMUJKkLBmgJElZMkBJkrJkgJIkZckAJUnKkgFKkpQlA5QkKUsGKElSlgxQkqQs\nGaAkSVkyQEmSsmSAkiRlyQAlScqSAUqSlCUDlCQpSwYoSVKWDFCSpCwZoCRJWTJATYT+/fv3b3Qd\nmontKfUsHf3OG6AkSVkyQEmSsmSAkiRlyQAlScqSAUqSlCUDlCQpSwYoSVKWDFCSpCwZoCRJWZq8\n0RWYFK+88goAJ598clF20kknAbD33nsDsOeeexbn5p577jrWTpI0KexBSZKy1O16UK+99lpxvNRS\nSwHw4YcfFmW9evUCYPjw4QCMHDmyOPfOO+/Uo4pN6ZxzzgFg5513Lsq+/fZbAJ555pmibMEFF6xv\nxTL25ZdfAvD1118XZXfffTdQ3sdbb711cW7yybvd17Em3n333eJ43LhxAIwePRqA9ddfvzj3gx90\n7nl62223BeCss84qyiabbLKJrqfgqaeeAmCNNdYoyh555BEA+vTpU/PfZw9KkpSlbvPI9tJLLwGw\n6qqrFmUffPABUPaaAGaccUYAevfuDcDbb79dnHvhhRcA+MlPflKU+UQ1YbfeeisA++yzD1D9KTZt\n/54qevEnnHBCUXbbbbcBcP/994/359IRgUMOOaSLapeXN998E4CLLroIgLPPPrs4F73yl19+Gai8\n3zp7n1144YUAzDzzzEXZkUceCZR/H3L17LPPAuXfuGWXXbaR1SnEvbz66qvX5ffZg5IkZckAJUnK\nUpZDfOlL5RjaGzhwIFBOLR+fJZdcEoCjjjoKgJVWWqk4169fP6BySGH77bevQY2b15gxYwD44osv\nGlyTfKSTbWKJQ/z7+eefF+daW1sBmHfeeYuyWWedFYCWlhag8gX+0KFDga552ZyTgw46CIBRo0bV\n5ffF0hMoJ/nMP//8dfndEyuG1p9++mmgsUN8cR9DOfQYfxe6mj0oSVKWsuxB7b///sXxiBEjOvWz\nd955JwCffvopABtssEFx7uqrrwbg4YcfntQqNrUnn3yyOD7ssMMqzi299NLF8U033QTAtNNOW5d6\nNUr0HuMF+xlnnFGcGzt27Hh/brHFFgPKexLKadRzzDEHAG+99Va7z2r2HtSvfvUroHoP6kc/+hEA\n++23H1BOmoDqE3T+/e9/A3DNNdfUvJ6NdMoppwCw5pprNrgm8MknnxTHxxxzDFCZAKEr71d7UJKk\nLBmgJElZymqILyZApF3/9AUdVA7ZbbTRRgBsueWWRVnk21tkkUUAOPDAA4tzV111VdXP1Heee+45\nANZZZ52i7P3336+45thjjy2OY81Zs7vnnnuAyv/28Vl00UWL47vuuguAGWaYoSh77733aly77ie+\nw23vLSiH8aabbroOfdZOO+0ElN/3WD+V2m677YrjdA1kzr755ptGV6GQZo8J0d5dzR6UJClLWfSg\nYjX9hHLrbbHFFkCZEw7Kl/lp2eDBgwGYZpppgPKlK5RPZxdffHFRFlNezXQO5557LlB9Kv+GG24I\nwC9/+cu61ikHkZGgmsg9uNpqqwHl8gao7DmFWDbRk8X3sFr7dNZDDz0EVObza2ueeeYpjnPOd/j6\n668Xx2mGkUar1tMdMGBAXX63PShJUpYa9jiRPvEcd9xxQJl3KqbgQrnIMRYxTjnllMW5WJQb/3bU\nZ599Vhwff/zxQDmts6ep1hbpdN5YWDps2LD6Viwjp59+OgArrLACUC4ah/Je7ehU+zQ3pCZOZISH\ncoF0eh+3lS5byVks24AJ//fUSyzVeeyxx9qdi78LXc0elCQpSwYoSVKW6j7EFyvpY6U4lNPKY9ry\njTfeWJxbYIEFgMr8fLX0v//9r0s+N3cxESXdEK6ayCSx8MILd3WVsjX99NMDsMsuu0zyZ8UWHOqY\nmKoPsO+++wLwxBNPFGVfffXVeH925ZVXBjq/0WGjPP744+3KOvv6opZ+//vfA5WTNxZffHGg8lVL\nV+oe/89JknqcuvegYiFdtTxc9913H1B92/Cpp566ayvWw0QOs//85z/tzm2yySbF8TbbbFOvKnVr\nsQj8o48+KspiQXi60V5kMQ+DBg0qjuebb76urGI2ovd+xRVXAHDDDTeM99rrr7++OJ7QhoUzzTQT\nUG6CCOVOBlNMMcXEV7bBlltuuS79/C+//BKovC9jt4fLL7+83fUxmWyqqabq0noFe1CSpCzVvQe1\n6667ApXphiL1SbWeUy1FZuR0TLqnpT164IEHANh6663bnYss0+nC53o9KXUH8R40HZOPbdqrjQhU\nu99CLAy/4IILirLu8q5kYrzxxhvF8aqrrgrA888/X7PPj3s3TdPVDNKkBRMS92Tcc2kG/XjPHu/r\nTj311OJcpFRKl0lEBvX47qfv/+uV4ig07zdCktStGaAkSVmqyxBfukFgTBtNX3imL+W7UgyhpL97\nmWWWqcvvbqR0mGD55Zcf73Uxpb/ZNyDsiDSb9KuvvgqUQ1NprsLI+RhDdmuvvXZx7tJLLwUqN3wL\nsdziH//4R1H2m9/8BoDJJptskuufsxhW78jw+vdtWBhickS6kV4jp2hPjLiXoPwbtd566wGw0EIL\nTfBn7733XqBs0zTnYGSGjwkX6RKfmIqftlV8/+OejowSUP/NNO1BSZKyVJceVGyZDeW0xjTLeDrV\ntlbiCbVajr2NN964OD744INr/rtzc8IJJxTHE3oKTffO6qmi5/TII48UZW2n+kZuPoDVV18dgPnn\nnx+Azz//vDj36KOPAnD//fe3+z1vvvkmANtuu21RFtPM09+Xc/btzphzzjmL45ioc+WVVwKV25p3\nZAHoeeedVxwfeuihtapiwx1xxBHFcdxPd9xxR4d+tl+/fkDZC4/RECjzmXZUTPuPe7SRi/TtQUmS\nsmSAkiRlqWHjB+n6mo5u79wRMbR3xhlnAHDAAQcU5/r27QuUOaagfjmlGiE2PYssB9WkQ0z1fgGa\ni3RCRGzfkN43IYZPhgwZUpTFfRzbI6y77rrFuciM0rt376IstjSJIcR0HdQqq6wCwKabblqUxTqr\nat+Rueaa63v+y/IUOTd32GGHifr5yMkHzTXEl4p1itXWK3a1v//97xX/e7vttqt7HYI9KElSlhrW\ng9pqq61q9lnp9six+WG8yE57CGmGhJ4gptBX2w57rbXWAmDEiBF1rVNOYgrz8OHDi7KYKBIZzKHc\n8j3aLO39xxbuO+64I1CZfXuxxRYD4LLLLivK4oVzTBbafffdi3Pnn38+ACNHjizKIl9dSPP1jRkz\n5vv+E5tSbPOu+thwww0b9rvtQUmSslSXHlS6IC+O46kU4I9//ONEfW4shEyfQmPb+D322AOAk046\naaI+uxnE9uLVppZHT6GZ38F9nxhrT6fXx7ueNIt2//79AXjmmWcAOPPMM4tzkYMvppenPdJ4ZzXD\nDDO0+93xXir214GyJ7fRRhsVZW17/d3hfk7f6cV24T/96U+LsonNLn7zzTcD9VvYr8azByVJypIB\nSpKUpboM8aW57+I48ptBuYJ6++23BypfUMf2zmeddRZQbrQH8OKLLwLlqmuAwYMHA+UQX08UubbS\nPGZtpUNLPVW1LdxjmUK6FGHs2LFA9S25QyxriHsYJn77jMiP1vY4d88++ywAhx12WFEWm969//77\nRVlHhvhiyHT06NFFWXy3q+U2jDx2bg9TO/E6JiYCQf031bQHJUnKUsOmmacvUqMHFTm2ZpllluJc\nvGStJjJHDxw4sCjbbbfdalrP7iKdah8Lc+MJPl0oGgsbzVheLtyOnGNQ5o2855572l2/5ZZbAjBg\nwICiLO7B2HK8mTcd/D7bbLMNUD33YDq5o9qkkbZikkq68V61Ld9jCnQs3m1k3rhmE+09oZGYrtZz\nv02SpKwZoCRJWarLEF+6BmKNNdYA4JZbbml3XUycSIerwuyzzw7A0KFDi7KJXT/VjNIXx23bL4ay\nwC01UrfeeitQbvYG5dBeuj3EZpttBpQv4Jt9Q8GuMGzYsEn+jNiiJ81Cc/jhhwPNsy1Jjm677bbi\nOLaXqRd7UJKkLNXlsSN9KRov8GOLZpjwlPAjjzwSKHOdzTrrrF1RRfVAMXkktnJve6zOiSnl6Sah\nJ554Yqc+Y9FFFwXKvxnpZobxNyDt3arrpBmAGsUelCQpS3UfuI1cZ+kiyWoLJtU5P/7xj4vjQYMG\nAZX55KSuFvtTHX300UXZ//3f/wGVez9Fdv3YZ2i99dYrzkUPtpZ7xKlzIhdkmnOyUexBSZKyZICS\nJGXJuZlNIh0SufbaaxtYE/V06ZTvddddF6jM1qG8xVTyRmaQCPagJElZMkBJkrJkgJIkZckAJUnK\nkgFKkpQlA5QkKUsGqInQ0tLS0ug6NBPbU+pZOvqdN0BJkrJkgJIkZckAJUnKkgFKkpQlA5QkKUsG\nKElSlgxQkqQsGaAkSVkyQEmSsmSAkiRlyQAlScqSAUqSlCUDlCQpSwYoSVKWDFCSpCwZoCRJWTJA\nSZKyZICSJGXJACVJypIBSpKUJQOUJClLBihJUpYMUJKkLBmgJElZMkBJkrJkgJIkZalXa2trxy/u\n1esd4KWuq0638ZPW1tY+k/ohtmfB9qy9SW5T27OC92htdag9OxWgJEmqF4f4JElZMkBJkrJkgJIk\nZckAJUnKkgFKkpQlA5QkKUsGKElSlgxQkqQsGaAkSVkyQEmSsmSAkiRlyQAlScqSAUqSlCUDlCQp\nSwYoSVKWDFCSpCxN3pmLZ5tttta+fft2UVW6j5aWlndrsbum7fkd27P2atGmtmfJe7S2OtqenQpQ\nffv25cEHH5z4WjWJXr161WTLZtvzO7Zn7dWiTW3PkvdobXW0PR3ikyRlyQAlScqSAUqSlCUDlCQp\nSwYoSVKWDFCSpCwZoCRJWTJASZKyZICSJGXJACVJypIBSpKUJQOUJClLBihJUpYMUJKkLBmgJElZ\nMkBJkrJkgJIkZckAJUnKkgFKkpQlA5QkKUsGKElSlgxQkqQsGaAkSVkyQEmSsmSAmgj9+/fv3+g6\nNBPbU+pZOvqdN0BJkrJkgJIkZckAJUnKkgFKkpSlyRtdAeXn3XffLY5/8YtfADBu3DgAnn/++YbU\nSVLPYw9KkpQle1AqHH744QCceeaZRdk777wDwJAhQxpSJ0k9lz0oSVKWDFCSpCw5xNdDffrppwBs\nsskmRdmNN94IQK9evYqy5ZZbDoDTTjutjrWTJHtQkqRMZd+D+vbbbwH48ssvx3vNyJEji+PoGTz5\n5JMADB8+vDh38MEHAzBixIiibOqppwbghBNOAGDo0KG1qHa2Ygr5fvvtB8BNN93U7poLLrigOP75\nz38OlO0k5eyrr74qjgcOHAhULo3473//C8BMM81U34ppotiDkiRlqWE9qLFjxxbH33zzDVA+3aRP\n9R9++CEAZ599dqc+v2/fvgDsu+++Rdl5550HwIwzzliUrbzyygCsttpqnfr87uqjjz4CYNSoUeO9\nJtoOYOGFF+7qKkkd8vHHH1f8m5p22mkBaGlpKcruuOMOAJZYYomizJGA7sUelCQpSwYoSVKW6j7E\n9+qrrwKw5JJLFmUffPBBzT7/Bz/4LubGcF7apd9+++0BmH322Yuy6aabDoA+ffrUrA65SXPrrb32\n2gC0tra2u+7+++8HYJlllqlPxZrcX/7yFwC++OKLouyxxx4D4JRTTml3/VJLLQXAgw8+WIfa5eON\nN94ojqNdXnzxxXbXxfBdtXyQMckp2hfKe7xfv35FWUy66kmiLS+88EIA/vWvfxXnHnjggXbXX3LJ\nJQDMPffcANx8883FuW222QaofA3QlexBSZKyVPce1KyzzgrAHHPMUZR1pge15pprtvusq6++uijr\n3bs3AKuuuuqkVLOpXHrppcVxPH1uueWWQOWU++mnn76+FWsCY8aMAcplDbHYGeDcc88FqvdW08XQ\n4dFHHwVg6aWXLsoeeuih2lU2U/fcc09x/Kc//Wm810011VQA7LnnnkVZfPfTyVAh2njXXXctynrK\nJIm0TTfddFMA3nrrLaDyftxwww0BeOWVV4qy+NsQ0usjN2e9Fu7bg5IkZckAJUnKUt2H+KKLHS/s\nAK666ioAVlhhBQA22mijdj+30korAfC3v/2tKJtyyikBePPNN4uyk08+ubYV7sZiQsRdd91VlC24\n4IIAnHjiiYDDeuPzySefFMdbbbUVUK7TS8XwdKzNSYdDYpj5zjvv7NDvjBf46RrBZnb66acDcMAB\nB7Q7t88++wCVrwJ22WUXAKaZZpqiLIb2IuNJDGMB/PCHPwTKTTebWdw7MSFi0KBBxbm4l3/9618D\ncOSRRxbnYgJJrEUF2G677QC47LLL2v2eFVdcsYa1/n72oCRJWWpYJol44gFYfPHFgbJHlD5RxUvT\nYcOGVVyTiiclgGOOOab2le1mYppyZORIX8jvsMMOAEwxxRT1r1g3EJMd4mkT4IUXXujwz6e9+VjC\nkPbG3nvvPQDWXXddoPp06uWXX77jFe7Gol0+++yzomyBBRYA4NBDDwXKNky9//77xXH0BqLdI6ME\nwBlnnAHA5JNnn3J0kt1+++0ArLXWWu3ObbbZZgCcf/75QDmRLHX33XcXx217TumU8g022GCS69oZ\n9qAkSVnK4tGibUSfeeaZ210TC/gidx5Un6rbU6WLQW+99dbxXjfbbLMBMMMMM3Toc6+88kqgei/i\nwAMP7EwVu4UjjjgCmHCvKaY7A1x00UUA9O/fH6i+4Dud2nzqqacC1XtO8X7wnHPO6WStu6eY/hz3\nGJTT6g855BAAjj322OJc7GgQ76cALr74YqBs9/Qd9Prrr98V1c5Guth77733Bsq/idF+UH5Pq/Wc\nwl577TXec5dffnlxnL7/qwd7UJKkLBmgJElZymKIr620uzl69GgArrnmGgCeeOKJ4tzPfvaz+lYs\nY+lwZ7RZTD2N/IRQOUTaVmScSD8rXlY/99xz7a4/6KCDgHILD+ie09Yff/zx4jjNU9bW/PPPD8AN\nN9zQrqyjXn755fGeGzJkCFD/YZRGmWuuuQBYffXVi7IY4osMEZtvvnlxbosttgCq5+KLKevVlqg0\nmzPPPBMoh/WgHL4bPHgwAL/73e+Kc20nRI0bN644jqUTzz77bFEWSyViCLGRuTntQUmSspRlDyqd\nSh4bFcaL//TFZ0wFThfixTTInjaBIqZHQ7mYOXpO6VN+28kRr732WnEcbZwuog7RM5pvvvmKsnj6\n2mSTTYqyeKGabgqZu6OOOqo4TqeEh1j0GC/sO9priokr0aMFuO6666p+NjT/S/22Yvp3te3XIzdc\nOuU+nuzT73YsSRkwYECX1TMH6SSoWHKTtkP0nGIqeTUxPT+mnUM5PT210047AbDjjjtOQo1rwx6U\nJClLWfagUrPMMgtQZokeOHBgcW748OEV/0L5BBFj0dUW+jWTmHpbbVp07Oeyxx57FGWRAT72iDru\nuOOKcxdccAFQmV4mekf7778/ULmocpFFFgHg7bffnsT/isZK33m+/vrrQOV08ehRdvZeiv2gfvvb\n37Y7FwvVY++difn8ZhGLczsqzbYdqY46umyiu0pTEaXpnMJJJ50EwKeffgqU6eOgHNW49957gcp3\nxtELq7aYv1pShHqzByVJypIBSpKUpeyH+MKyyy4LVE4zj2mW6Ur0yMQbU1FjaAq65xTo7/P0008D\nlS8+Q0wD33nnnYuyGALYb7/9ABg1alRxLiY2pENSf/jDH4BySDD9PXH9euut166sO1luueWK445m\nHh+fdIPB3Xbbrd35mPIb/9/01GE9KJdBpFuKV9vcMURW+ZEjR3ZtxTI02WSTFceRezTN+xivQiY0\nOWyeeeYBKielxGSUdFg/3TCz0exBSZKy1G16UGHOOecsjuPlddpDWGONNYBy6vAzzzxTnEtzSjWL\nRx55ZLzn0nYJMekhMp2n7rvvPqDMCQfl5Iu0LEQbN2NOvomVZumv9jT717/+FYB11lmnbnXK1dCh\nQwE499xzi7IJ9QB62tKRVJr/MTKPp1PwYyv2RRddFCh7m1AuAI9M7+m56EHF/xe5sQclScqSAUqS\nlKVuN8SXim5vbK0N5cvEyDd17bXXFudiuG+hhRaqUw27XmyAl75c3nbbbSuuSbNFxCSTuD7W6kA5\njBcTIqDcNr7a9dUmZvRUsQ4lXvxDZQ7EkA4B9iQff/xxcRxD7bGtSDp0t8oqqwBlO/35z38uzsUa\ntZ4uNhBMJ0l0ROTbS/8mxj268MIL16ZyNWYPSpKUpW7Xg0qfoiLjcayQhspMvVD5xFrtRX+zSJ9C\nJ/QyOZ6Y4prYHh7KDMiff/55URYZ4+O6CW161hPFCv9on7TXFG2cruqPDSN7mpaWluI4cr2FdIPG\nyFge3+m0B7XEEkt0ZRWbXuTzq3aPxkhJbuxBSZKylH0PKqZPnnbaaUCZLw7g1VdfHe/PxbuoGK+F\n5pymGhndI6szlG0UPaJ0cfPYsWMrfj7enUD5nildtHf88ccDzbnIeWJ9/fXXxXEsMq22hCEW6qb5\nI5vxHpyQeO9bbZ+m6FUttthiRVlkk991113bXd/ZvbdUKW3n7sIelCQpSwYoSVKWshrii+799ddf\nX5QdccQRAIwZM6ZDn7HaaqsB5eZy/fv3r2UVsxO53dKcbtGO/fr1Azo+rFQtF9+SSy5Zk3o2g9ja\nZJ999inKzjrrrIpr0qG+GNbqacN6qX/+858AfPDBB0VZbCq61FJLAZVbSdx2221AubleunwizSKj\nznvssccaXYVOswclScpSw3pQkVUbynxQsRHZww8/3KHPWHPNNQE4/PDDi7KYVt5TnlpjU8I77rij\nKIsceTENv5roBaQ9zHiibebp+JMiJpi07TVBmQNt4403rmudctd2WUN6HD2n0aNHF+ciV2RMx0/z\nPK6//vpdW9kmV21T09zZg5IkZckAJUnKUl2G+NLMBHvttRdQpoyHctO9CYntCQ455JCiLF7gx0SB\nniydzJBu4KhJF2vxTjzxxHbnFl98cQBuv/32utapu3jrrbfalc0+++xAORx63XXXtbsmJlfktHle\ndxebvn5fvsic5F07SVKP1SU9qBdffBGAo48+GoBbbrmlOPfSSy99789PM800xfGwYcMA2GWXXQCY\ncsopa1VNqUPiHjz99NPbnTv00EOB7rnVfT1EDzMVk0xiCnmfPn2KczFC0h2zHuQupulHfk2Ap556\nCqjs6c4777z1rdgE2IOSJGWpS3pQsa31eeedN95r0rHlzTff/LvKTP5dddKFoulWx1K9pHvttM1f\nePDBBxfHK664Yt3q1B3F1PA0h2bkKBwwYABQTi0HGDx4cB1r1zMNHz68OF5rrbWAylyeI0aMACpz\ncjaKPShJUpYMUJKkLHXJEN++++5b8a/U3YwaNao4vuSSS4Ayt+Huu+9enEtf8Ku9GKIfMmRIUZYe\nq/5WWmml4njTTTcF4IorrijKIovHySefDDR2Ypo9KElSlrLKZi7lYtCgQcXxQQcdBMDFF18M2GtS\n99a7d+/iOCavLLTQQkVZLKs47LDDgMZOlrAHJUnKkgFKkpQlh/ikKhZZZJHieNy4cQ2sidR1Yrgv\nMqK0PW40e1CSpCwZoCZCS0tLS6Pr0ExsT6ln6eh33gAlScqSAUqSlCUDlCQpSwYoSVKWDFCSpCwZ\noCRJWTJASZKyZICSJGXJACVJypIBSpKUJQOUJClLBihJUpYMUJKkLBmgJElZMkBJkrJkgJIkZckA\nJUnKkgFKkpQlA5QkKUsGKElSlgxQkqQsGaAkSVkyQEmSsmSAkiRlyQAlScqSAUqSlKVera2tHb+4\nV693gJe6rjrdxk9aW1v7TOqH2J4F27P2JrlNbc8K3qO11aH27FSAkiSpXhzikyRlyQAlScqSAUqS\nlCUDlCQpSwYoSVKWDFCSpCwZoCRJWTJASZKyZICSJGXp/wExUf3fhO0TIwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1105c7358>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "fig, ax = plt.subplots(nrows=2, ncols=5, sharex=True, sharey=True,)\n",
    "ax = ax.flatten()\n",
    "for i in range(10):\n",
    "    img = X_train[y_train == i][0].reshape(28, 28)\n",
    "    ax[i].imshow(img, cmap='Greys', interpolation='nearest')\n",
    "\n",
    "ax[0].set_xticks([])\n",
    "ax[0].set_yticks([])\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmcjfX///HHGU1j7DIYPsgSKiVZSn1iWrRIkiVC5Ctp\nsSR9SFJRQlla+EhEIilFlg9ZspRKU0xDSQiRncIYs885vz/O73rPHDOYM87MdZ2Z5/1262auc11n\nvq/v+3PN9breu8vj8SAiIuI0IXYHICIikh0lKBERcSQlKBERcSQlKBERcSQlKBERcSQlKBERcSQl\nKBERcSQlKBERcSQlKBERcaRL/Lk4IiLCU7169TwKJXhs2rTpuMfjKX+xv0fl6aXyDLxAlKnKM4Pu\n0cDKaXn6laCqV6/Oxo0bcx9VAeFyufYG4veoPL1UnoEXiDJVeWbQPRpYOS1PNfGJiIgjKUGJiIgj\nKUGJiIgjKUGJiIgjKUGJiIgjKUGJiIgjKUGJiIgj+TUPyik+/vhjAJKSkvjll1945513zLnrr79e\n8wwkoNLS0khISDDHW7ZswePxALB169Zsv3PppZcC0KFDB0qUKEFIiN4Fc8Pj8RAfH8/y5csBmDVr\nFps3b+aXX34BoHTp0naG5yjWPblgwYJznnO5XBw+fJj+/fsD8Omnn+JyuWjRogXgvPLUX42IiDhS\nUNSgduzYAcBvv/3GihUreP/99wHftwLLli1baNiwIQAxMTH5HGlwc7vdnDhxwuez0qVLc8klQXGb\n5Jmnn36ad999N1ff7dWrFz169OC///0vAOHh4YEMrUCKi4tjzZo1AEyfPp2lS5f6nC9evDihoaF2\nhOZICQkJREdH8/TTTwPe5+TZzn5WWv926tQJl8tF1apVAejfvz9dunShYsWK+RH6BTnuyRMfH0+3\nbt0A2Lx5M4B5aJ4+fRqPx8Ott94KwNdff53l+263m1OnTuVPsDbatGkTAI0aNfL7u8nJyQD8/PPP\nTJo0iZSUFABSUlJYvHixz7Xvvfcejz322EVGG9ySkpKIioryK7mkpaUB8NVXXzFz5kzTpNKgQYM8\niTHYHTx4EIBRo0Yxffp0c4/Wrl2b4cOHm/IcOXIknTp1olixYrbF6jR9+/Zl1qxZF/U7/vrrLwAG\nDRrElClTePTRRwEYMGCAaa62g5r4RETEkRxTg7KqpQ888AC7d+8+53WHDx+mRIkSgLe29ffff3Pf\nffcB8OeffwLQtGnTvA3WAerXr5/ja8+cOQN4a11btmzhq6++AjC1paioKAA6d+7MmTNnWL16tfnu\ngAEDCn0NasqUKYSEhFCkSJEcf8ftdgPQrl07Fi9ezMqVKwHVoDL7/fffAbj//vs5cOAAAImJiTz/\n/PP06NED8C6uGhoaas6PHDmS66+/3pZ4nSq7QWHWwLHIyMgLfr9fv34cOXLEHP/xxx8MHToUgJYt\nW3LttdcGKFL/OSZBvfLKKwBZklPRokVN9bVRo0aUL5+xQnt4eDgTJ040iQmgTp06TJs2Le8Dtpk/\nbfDNmjUDIDY2Fo/HY9qfPR4PvXr14q233gK8bfs1atQwCcrj8fDMM88EOPLgk5v+DqvZ1HoJ6NSp\nU0BjKgispvtmzZqZl86HH36Yxo0b+/Qrn6148eL5El+wWLVqFWPHjjV/x4B5Zq5YseKCI/PatWvH\n33//DeCYvieLIxLUr7/+aoaRZlarVi2WLVtGrVq1zvndffv2+Rx3795d7dNnsYY4FytWjGbNmjF6\n9GgAatSoQalSpczDYPbs2Tz55JPme1deeSVDhgzJ/4ClULjpppt8/j2f5557zvz80EMP5VlMwahi\nxYq8/vrrpqbz7rvvmlrVvHnz6Nixo9/Dx62aV5kyZQIbrJ/UByUiIo7kiBrUa6+9Rnx8vDlu1aoV\nAGPGjMm29pSUlATAjz/+6DPqrFWrVrRp0yaPow0+M2bMAKBKlSpcdtllWc5bIwKfeOIJEhMTqV27\nNgAbNmwwTS/in++//97uEAqUzM34klWRIkV45JFHAG8//sMPPwx4m5g7dux4we9bU3csnTt3BjDD\nz+3iiAQ1YMAAM8y0fPnyzJw5E+CcD0drJYnevXsD0KRJEwDmzJmjB2o2zjegYvbs2fTp0wfwdlDX\nqFHDPFztrt4Hs2PHjpmfa9as6dO2Hxsba4ZRn+3GG2/M89iC2W233WbrsOdgULp0aZYsWZLj648c\nOWKeueBNcK+++moeROY/RySoG2+8Mds5TdmJiYmhb9++5jg0NNT0kyg5+ScmJoZevXqRmpoKePuc\n5s6dS7ly5WyOLPgcPXoU8A6OOH36NIMHDzbn/vzzTypVqmSO4+LizMTJUqVKmXl9Xbt2VYLKRlxc\nnKnl9+jRQ8tGBViLFi3MYgjgnQtVtGhRGyPKoP+lRUTEkRxRg/JHkyZNfIagzp8/n3vvvdfGiIJP\nbGwsALfffjupqanUrFkTgPXr16v2dB7W0PHffvuN7du3A7B06VJ2795tltWy+kfPZt2zNWvWpG3b\ntjzwwAMAVKpUKdt+QcmwZs0a0yQ6cOBAm6MpWNauXcuuXbvM/dm4cWNHzSMNqgT15ptv4na7far4\nVv+T5ExsbKwps/T0dK666iq+++47QH1O55OQkMAVV1wBeCeLn0+NGjXYs2ePOd6wYYPu04uwevVq\n8zdfoUIFm6MpGKzV+QcPHkxKSgrVqlUDvPOmnCQoElR6ejrgnTEdEhJisv3nn39ORESEnaEFlZiY\nGG6//XZTngDr1q1TYsqBIkWKmNrl4cOHGTduHOCdL9KmTRufl6aUlBQzMOWvv/7immuuyf+AC5CD\nBw9y8803A87bDiJYTZgwAfA+E1wuFwMGDACcV77qgxIREUdyfA0qNTWVVatWAd7NtQAziu+ee+45\n75Io4hUXFwdAnz59iIuLMyPKFixYoBpoDoWFhTF79mzAW57WG312W5GEh4eb+SPWKtEiTrFixQpe\nfvllwLtmZJ06dRy7OoejE1RycjIDBw7kvffeM599+umntG/fHkDJKQcSEhLMAqV//vknNWvWZNmy\nZYB33ULJOS30mv+Sk5NZvnw5rVu3tjuUAiEhIYFx48aZZ2dISAgrVqxwbN+eoxPUqVOnfJLT1Vdf\nTYcOHWyMKPjs37/fzMK/5JJLWLlypRm1J3nHqpla850kd6Kjo0lMTPSZVya5Fx0dzdq1a81xjx49\n+Ne//mVjROenPigREXEkR9agrGVirJEm1oiozJlfcqZEiRJme4JOnTqp9pRPrLXMzt6hWPzz4Ycf\nAs7bBiJYWdvCW4YMGZKr7WTyiyMTlLUO1OTJkwFMh57ThkAGg8qVK7N//34AxyxfIuKPMmXKUKpU\nKbvDCGqPP/44AFu3bgUy1jHdvHkzcXFxNGrUyLbYzkdNfCIi4kiOq0EdPnyYU6dOmeOhQ4eaIb2S\nO6p55r/7778fgJ07dxIWFmZzNMErJiaG8uXLU7JkSbtDCWrWdhrW6D1r1/GpU6eya9cu2+K6EMcl\nqI8++og5c+YAULt2bfr16+ezzbtIMAgPDwc4727Qcm7WljqxsbG88MILNkdTcA0ePNjRo/gcl6Ba\ntWplts+YPXu2kpNIIXTkyBHzc5cuXWyMpGCy+qRGjx5tcyTnpz4oERFxJMfVoK666irS0tLsDkNE\nbPTMM8/4/CsXJ/MC0cHE5c9Md5fLdQzYm3fhBI3LPR7PRbc9qjwNlWfgXXSZqjx96B4NrByVp18J\nSkREJL+oD0pERBxJCUpERBxJCUpERBxJCUpERBxJCUpERBxJCUpERBxJCUpERBxJCUpERBxJCUpE\nRBxJCUpERBxJCUpERBxJCUpERBxJCUpERBxJCUpERBxJCUpERBzJrx11IyIiPNWrV8+jUILHpk2b\njgdi8zKVp5fKM/ACUaYqzwy6RwMrp+XpV4KqXr06GzduzH1UBYTL5QrIjpgqTy+VZ+AFokxVnhl0\njwZWTstTTXwiIuJISlAiIuJISlAiIuJISlAiIuJISlAiIuJISlAiIuJISlAiIuJIfs2DkoLv5MmT\nzJkzB4D+/ftz//33M3fuXACKFi1qZ2j5KiEhgVtuuQWAzZs38/zzz+f4u6NGjcLlcgHg8Xjo1asX\nFSpUAKBTp05ce+21gQ9YJAfS0tLYuXMnixYtAuDLL7/km2++MeeHDh3Ka6+9Zld4WagGJSIijhRU\nNaj09HRSUlKYMmWK+ezQoUOMHTvWHDdo0MCcv/HGG/M9xmCSkJBAdHQ0Tz/9NOB924+Pj+evv/4C\nwOVysXjxYoYNGwbAuHHjbIs1vx04cICQEO/7m8vlYvTo0T61orN/9ng85tqGDRsSGxtrfteMGTPM\n+TFjxvDdd9/p3vTDjBkzzM//+c9/OHXqFFdffTUAzzzzDAD33nsvAJGRkfkfoIMdOnSIxMRE1q1b\nB8Dy5cuZP3++Oe/xeMx9Dt7700k1KMcnqNTUVCZOnAjAqlWrWL58uXk4WDIfb9682fwPoIeAr/T0\ndE6ePGkS0pEjR1i7dq05n/nBm1nXrl3zLUanqF27tlmS5ujRo+dcnqZcuXI5us/27dsHQI0aNfj7\n778DF2gBk5SUxPLly5k3bx7gfaCeOnXK5750uVxs27YNgN69ewNQt25dALZu3ZrPETvPZ599ZpL6\nDz/8QFxcnM8LFED9+vXNvx999JH5bp8+ffI52vNTE5+IiDiSI2tQcXFxAOzcuZPhw4ezdOlSn/NF\nihQB4MorrwQgPj4egL17A7KeY4FjvT3Nnj2bXr16ZXmbupDLL788z2ILBhUqVDBNSLm1cOFCIOdl\nXpjExsayadMmAEaMGMHBgwf9/h2HDx8GYM+ePdSoUSOg8QWbxx57jNOnT/t81rZtWwDatWtH69at\nufTSSwH49ddffWpQL774Yv4FmgOOSVCpqakAvPPOO6ZP6ejRo1muCw8P57vvvgPg+uuvJzU11bSv\n3n333QBUqlQpHyJ2th07dgDwyy+/0L9/f8DbpHe2Hj16mD/o//u//6NJkyY+140YMYLLLrssHyIu\n2Kxm1czt/YVVTEwMGzZs4IMPPgBg+/btJCYm5vj7TZo04aeffvL5rEyZMgCFPjkBREdHExMTY47b\ntWtHWFhYtteuWrUKj8fDzTffDECpUqXyJcacckSCSk1NNR3xmQc8gPeGq127NgB33XUXd999N/Xq\n1TPnt2zZYhITwP3330/fvn3zIWrn2rx5My1atADgxIkTPucaN25MnTp1ALj22msZMGAAoaGhAFSu\nXNnnpaBx48YMHDgwn6IuuI4ePeoz4OLWW2+1NyCbtWjRwrSSnEuDBg0Ab1/gmDFjfM7t27eP2267\nzeczp73526lu3bqmT+5crBrW3LlzcblctGzZEuCcicwuep0TERFHckQNCqBs2bKAd8TY448/DniH\njEZERFCyZEkgo+/JsmPHDlq1amWO+/Xrx7Bhw0yNoLByuVzmjf3aa6/l7bffBrxlXK1aNUqXLu1z\nfUJCAuBtAnS5XOYt6vXXX6dYsWL5GHnBNGrUKNPv99Zbb6lMgYoVK5qRZM899xxffPEFAFFRUVx9\n9dVUrlwZyL7J6eym6lq1atGhQ4c8jrhgeeONNwBvH1TFihV54oknbI4oe45IUKGhoQwZMiTH11v9\nVUOHDuXo0aP069cP8DYPWp1/hVn9+vX59ttvAUzz6LkkJCRwzz33mOPGjRvz+uuvA96HhVyco0eP\nMmPGDDM44oEHHrA5IvutXr2aiIgIqlataj7zp9lz+vTp5udixYoxevRoSpQoEcgQCzxrtRjw9uWX\nK1fOxmjOzREJyh8pKSlmqZidO3fSqFEjM4G0sNecMrtQYrL07t3bDDpp2rQpq1at0ht+AK1Zs4Yz\nZ86YznunPgjy0/XXX5/r744cOdJnoumUKVNo165dIMIqNA4cOMA///wDeAftjBo1yuaIzk19UCIi\n4khBV4PavXs3O3fuBLx9Kp988olqTrm0ePFi/ve//5nmJ9WeAscaDTlw4EBcLhcff/wxgMo3F9LS\n0vj9998BGD9+PElJSabV5L777rMztKA0YcIEM4rv9ttvNyMmnSioEtSuXbuoX7++aW/euHGj5j3k\ngrVsT/fu3YmPjzfrl+nhGTiLFy8GvBNIIyMjueGGG2yOKHjNmjXLLGkE0LNnTwYMGGBjRMHr0KFD\nPmsb9uzZ08ZoLiwoEtT27dsBTN/Tr7/+CkC1atVsiymYjR49GsAkpx9//NHmiAqWhIQEM0rK5XKZ\nUZTiv+3bt/Poo4+aUak33HADb731ls1RBa/ExMQLzkFzEvVBiYiIIzm+BpWammpmjaelpfH000+r\n5pRLSUlJdO3alQULFgBQp04d1q9fbzbTk8AYNWoUu3btArwroWSeqyc5k5ycDEDDhg0JCQnhpptu\nAmDlypVqis4Ft9sNwPDhw/F4PGZ+WefOne0M64IcnaD27t1L3759zUKQ3bp1480337Q5quC1detW\nFi1aZJpLMu/0KoGTed5T69at9UD1U2pqqpnbmJyczA033GD2KFJZ5o61x9ucOXNwuVz897//tTmi\nnHF0gho7dixLly41nfhO2kgrmDz33HMAZtXiHj16AKijOQ8sW7aMw4cPmwTlz1bx4k1IAwcO9OnI\nHzNmjCaNX6S5c+f6HLdu3dqmSPyjPigREXEkR9agoqOjAW91tHLlynz//fcAVKlSxc6wgtKOHTvM\n26i1srm1rJTmjwWe1YTSq1cvADWh5pDVjN++fXuio6PN+puxsbGFfj+yQOvevXvQLA3luASVmJho\ntssoWbIk69ev16CIXLAWJ73qqqvMZ/Xq1WPLli12hVSgpaSkAN5h0R6Ph2uuucbmiIJHUlKSab6P\njo6mWbNmTJ06FdBmmYHy5ZdfAt7nwksvvRQ0+5IFR5QiIlLoOKYGlZSUBMCHH35otn+eOHGi3qBy\nadq0aYB3oqg1yCTzCsYSWNYWELGxsURGRtK9e3fAu+SRmvmyZw0lHzRoEB9++CEAbdq04YMPPnDc\nzq7B7PTp06xfvx7wPg+OHTsWNCvwOCZBHT9+HICnnnqKESNGAJh9ocQ/p0+f9hmOv2TJEiBjJQ7J\nOx6Ph8OHD3PHHXcA3t1N9WKQVWpqqtmtedGiRbRv3x7AbAMvgWONKLXcf//9Zp5e8eLF7QgpxxyR\noJKSksyGWdWqVTN9UJdc4ojwgk5KSopZULdt27Y+/VCSN6w3/sjISJ9h5lZNVjKkp6fTp08fPvvs\nMwC2bNniszeUBFaJEiXMMP09e/YwadIkwsPDbY4qZ9QHJSIijuSIKsqrr75KbGws4F1p29r+XXKn\nXLlypKWl2R1GoVK6dGnAuxmcnF+vXr3Yt28fBw8eBAiat/lgtnbtWrtDyBWXNRw5Rxe7XMeAvXkX\nTtC43OPxlL/YX6LyNFSegXfRZary9KF7NLByVJ5+JSgREZH8oj4oERFxJCUoERFxJCUoERFxJCUo\nERFxJCUoERFxJCUoERFxJCUoERFxJCUoERFxJCUoERFxJCUoERFxJCUoERFxJCUoERFxJCUoERFx\nJCUoERFxJCUoERFxJCUoERFxJL+2fI+IiPBUr149j0IJHps2bToeiN01VZ5eKs/AC0SZqjwz6B4N\nrJyWp18Jqnr16mzcuDH3URUQLpcrIFs2qzy9VJ6BF4gyVXlm0D0aWDktTzXxiYiIIylBiYiIIylB\niYiIIylBiYiII/k1SEJEJNBmzpzJo48+ao5fe+01n/M9e/akQoUK+R2WOIASlIjYKioqivHjx5vj\nxYsX880335jjYcOGsWTJEgBatmyZ7/GJfQpUE5/b7SYtLc3853a77Q7J8TweD8nJySQnJ/Ptt98y\nePBgXC4XLpeLAQMGsH79ertDlAKuRo0aDBgwwPy3cuVKdu7cyc6dO03Nqn379rRv394kKgmsH3/8\nkR9//JHmzZvjcrl49NFHfWq1dilQCUpERAqOoGzis2pGqampzJ07l+PHjwOwceNG5s2bZ64bO3Ys\nzz77rC0xOtHOnTuZO3euz2cpKSmMGjXK5zOXywXAO++8w+rVq4mOjgagWLFi+RNoEDl+/DhfffWV\nz2cej4d+/foBcOLECZ9zbrebZs2amTK/5ZZb8ifQIHLJJZdQo0YNACZNmsR1111nyvOhhx5i/vz5\n3HPPPXaGGPTi4+MBWLBgAe+//z7ff/894L0/XS4XCxYsAGD69Om2xQhBkqDS09MB2L9/Px999BE7\nduwAYPbs2T7XeTwe83AFWL9+faFNUNYNuH//fnOTzZo1i2PHjvlcl7nMQkNDqVOnDn/++ScAZ86c\nYevWrSQlJQFKULt37wbgyJEjrFy5EoApU6act0wz348AISEhfP/999x9990AfPfddzRo0CCvQw9a\noaGhPPXUU6xatQrw9k8tWbJECcpPJ0+eBGDixImMHz+etLQ0ABISErK93il9fWriExERR3JcDcrt\ndpusfuLECd5//332798PwAcffODX7+revXvA4wsGixYtYsCAAQDs3XvuJa+GDRtGWFgYd9xxB+Ct\nIV111VW0bdsWgGXLlnHPPfdQokSJvA/awbZt28azzz7Lzz//DMCxY8fweDxA1hoSeN8+s/scvGUK\nkJycDGBqp3J+o0ePBmDJkiWsWLGCxMREAMLDw+0My/HWr1/P2rVrefPNNwE4derUBb9Tvnx5Zs2a\nldeh5YhjEtTq1asB+Pzzz3nvvffOeV2xYsVo06YNkJGAfvjhBwBeffVV3G439957LwAPPPBAXobs\nWPv37/dJTDfddBMAERERNGzY0CSga665hpCQkCzftR6i4B0CfOmll+ZD1M518uRJ06RnsVakLlKk\nCCNGjDB9JgBNmzbN8jusRFS8eHEA6tevD8DVV1+dFyEXOJdffjngLfc///zT3N9XXnmlnWE50vHj\nx5k8eTIAI0eONM15mVWsWBGAVq1aMWPGDJ9z48aN45JLnJEabInC7XabyXj//PMPgHkAbNu2zefa\n8PBwWrduDcATTzxBZGRklpvSulnDwsJITEzkwQcfBLJ/uy0Mevfubfo4AKpWrQp4y+dCrNoqQGRk\nJL179w58gEGmdu3a1K5d29yHjRo1olOnTjn+flJSknlpslg13FKlSgUu0ALMqindddddTJ06lcWL\nFwNKUJlZg8X+/e9/s3PnTp9z1apVA7wv7TVr1qRjx44AZvCJ5aGHHjLnnEB9UCIi4ki21KA2bNhg\nhjFu3rw5y/moqCjAWz0tW7bseZtBDh48yMKFCwFITEzk4Ycf5s477wQKbw0qNDSUK664Ilff/fDD\nD83PX3zxBWXLlg1UWEErIiIiS80+p06fPk3Lli3ZsGED4G09eOqppwpt/2huWVNLzpw5g8fjMf2m\nkuHw4cOAtxXk4YcfBqBEiRLUq1ePbt26Ad4ae0JCgpmOM3/+fAA6d+4MeIeV56SlJb/YkqA+/vhj\nk5gGDRpEhw4dfM7XrVsXyFnzR+ZlUTweD8OHD6dy5coBjrhwGD9+PHPmzDHll9skJxnWrVvHDz/8\nYF6WGjRokGWtObkwa9rEnDlzaNmypenDkwzXXHMNAHv27KFcuXKAt4/0bN988w09e/Y0x7Vr1zZT\nUYoWLZoPkeacLQlq0qRJPPbYY4C307NMmTK5+j2HDx9mwYIFZtTf119/bTpTJedSUlIAWLFiBSVK\nlDCT9qybXPxn1bgeeeQRAOrUqQN4E1bp0qVtiytYHTx40Px8+eWXExoaamM0zna+hXW/+eYbU1sC\n6Nq1K2PHjnVcYrKoD0pERBzJlhqUy+UKyOz5N998k++//55mzZoBcOONN2ZbpZVzS0lJMTWlM2fO\n8NJLL5kRP5I7cXFxZimjuLg4rr76atauXQug2lMuWVMlwPt3Lv6x5o0NGjSIU6dOmftwxIgRREZG\n2hnaeTljsLufpk2bBsDkyZNJSEhgzJgxAI4Zux9M0tPTTft+o0aNGD58uL0BFQBly5b1GaAzdepU\nNZdehB07dpilepo2bUq7du1sjii4JCYmmi6Vn376iRtuuIFx48YBULNmTTtDu6Cge6Lv2rWLiRMn\nAt43/vfff58mTZoA2XcIyrmlpKRw7733mnJ7++23bY4oeFmrTIwcORK3221aCF544YVsJ+5Kzg0b\nNszMg/rggw8K/com/lq5ciUff/wx4K3Bf/bZZ2ZupNOpD0pERBwp6GpQr732Gr/++qs5btasmZr2\ncumtt95i3bp1NGrUCPDOQBf/JSYm8uKLLwKwfPlyQkJCzAz9s6dQiH+WLFnC0qVLzbxJrRzhn+jo\naHr06GGO582bFzS1JwiyBDVw4EDmzZvHddddB3gnmalD33+HDh0C4JVXXiE8PDzLOnOSc9u2baNn\nz5789NNP5rM//vhD9+VFSE5ONhNLly5dStOmTbW9hp+sPdysJc+sieHBtv9YUCQoa6299957j8TE\nRPMWVa1aNdWecsHqs0tISGDQoEFcdtllNkcUfOLi4gAYNWoUP/30k1ntZOrUqT4Lx4r/7rnnHjP5\n/r777vPZhFQubPv27WY1nfj4eHr06JFlQdhgoT4oERFxpKCoflhNUNZYfmsotGpP/vv888/NDrAD\nBgzghRdesDmi4GStUWgNJ586dSqQ/VYbcmHW3/bLL7/MN998Y+bpvPfee45aGy4YjBo1ykwdqVSp\nkk8fVLBx/BN+3759DB061BxPnjxZa8Tl0q+//kq3bt1ITU0FoE2bNtruwQ/WJoNdunQxi5c2aNBA\nyxddpN27d/PZZ58BMGHCBCBjrqOTJ5E6jbW+6YIFC8zSRb/88ktQN+GriU9ERBzJ8TWodevW8fff\nf5vj8PBwTcjNpeHDh5OcnMycOXMAaN68uc0RBReraXnRokVmJ+J+/fqp9pRL1uTmqKgos+AzwI8/\n/qjVyv2Unp5OixYtAO8E/DVr1gAEde0JgiBBne3mm2+2O4SgZe0XY+0MW1j3y8qN5ORkdu/ebY7H\njx8PZKxWLv67/vrrgYwRkZI7breb7t27m36ndevW+axdGMwcn6C6detG165dzbH15iqSnw4cOGA2\nd4OMLduzOaTSAAAVo0lEQVRF7PbRRx8xd+5czpw5A2CWhSoI9LQXERFHcnyCcrlcFClSxPynZqnc\n+/bbb3G73ZQoUUILbooUEN27d8ftdhMeHl6gak8ALo/Hk/OLXa5jwN68CydoXO7xeMpf7C9ReRoq\nz8C76DJVefrQPRpYOSpPvxKUiIhIfnF8E5+IiBROSlAiIuJISlAiIuJISlAiIuJISlAiIuJISlAi\nIuJISlAiIuJISlAiIuJISlAiIuJISlAiIuJISlAiIuJISlAiIuJISlAiIuJISlAiIuJISlAiIuJI\nSlAiIuJIl/hzcUREhKd69ep5FErw2LRp0/FA7K6p8vRSeQZeIMpU5ZlB92hg5bQ8/UpQ1atXZ+PG\njbmPqoBwuVwB2bJZ5eml8gy8QJSpyjOD7tHAyml5qolPREQcSQlKREQcSQlKREQcSQlKREQcSQlK\nREQcKagS1IEDB3jssccICQkhJCSETp06cfr0abvDEhGRPODXMHO7LFy4EIB27dpRvnx5HnzwQQBi\nY2OpWbMm27ZtAyAiIsK2GKXgi4uLA6Bs2bK43W4AQkJCeOWVV6hVq5a5zuPxcMsttwDw3XffAfDv\nf/8bgKpVq+ZnyCJBzfEJ6vfff6dz584AlC5dmq+//porr7wSgCNHjlC5cmV+/vlnAO68807b4nSS\nffv2UaNGDQDS09NtjqbgCA0NBeC6665j8+bNALhcLl5++WWf6zweD5dddhkAJ06cALxJDWDRokU0\nadLE/C4RJ0lOTubDDz80x0WKFOHRRx+1LZ6gauITEZHCw9E1KLfbzcKFC7nkEm+YmzZtombNmuZ8\nyZIlKVq0KK1atQLg5MmTFCtWzJZYncTlcuFyuQCIjo7mxhtvtDmigiE8PByAoUOH0qlTp/Nea9Wc\nzj5u1qwZJ0+eVA3qAqy+5R9++AGAtLQ0AHr06JHt9U888YT5t1KlSnkfYJCLj48HYP78+QDMmjUL\ngPXr15OammqeHwDr1q0DYPbs2fkbJA5PUNu2beOFF15gxowZAD7JCaBYsWL07NmTyZMnA96mFfGW\ng1UWTZs2pVGjRgAsW7aMChUq5Pj3REdHEx0dzZNPPgmgh+r/16FDB9O3BLB7926GDx9ujk+cOJEl\nQcn5HTt2DPA2Kc2bN4833ngDgD179uTo+6+88goAU6dO5dChQ3kTZJBat24dR44cAWDs2LHs2LHD\n9KEmJCRkuT5zcgKYM2cOYE+CUhOfiIg4kqNrUFbGbtOmjc2RBJfMTXyNGjUiNjYWgCuuuILGjRub\nwSQdOnSgdu3aPt+Njo5m7ty5AMyYMYMzZ87Qtm1bQCPQMmvatKnPz126dDHHn3zyCQ8//HC232vb\nti1hYWF5Hp8T7du3D/AO3Jk4cSI7d+4057Zu3Qp4a+k7duzI9f+Nxx577OKCLADWrFnDmjVrAJg2\nbRr//PPPRQ2WmjBhQqBC85ujExTAgw8+SMmSJbM95/F4SEtLo3HjxoCaoCyZm/hKlSpl2u+XLVsG\nwMqVKwFv8oqPj6dXr14AvP/++7hcLjMCMDIyErfbTbly5fL7/4Wg1rlzZ0JCsm+cGDJkCJdeemk+\nR2S/H3/8kWbNmgGQkpKS4+9VrVqV6tWrM3bsWJ/Pv/76awCee+45n8/PfuEqDA4ePMjbb79tXiwP\nHjxomvAupGPHjjRp0oTHH38c8N6fVpcJwN13322a+O3g6ARVrVo1jh07Zh6wRYoU8TmfkJDA1KlT\n6dOnD0Ch/MPPTuYaVGb33nuvz78vvPACSUlJ5nzFihXp1KmT6etbt24dAwYMyIeIC44pU6YQEhKS\npfwnTpwIYF6mCpvx48eTmpp6zvNFixYFICwsjIkTJ5ph+c2bN6dUqVI+1+7YsYOYmBhzHBISwqBB\ngwC44447Ah264zVu3JjDhw+f83y3bt3MVAhr+oOlePHihIaGmjl+H330kc/5Ll262FrjVx+UiIg4\nkqNrUE899dR5z1sjU7p27Zof4QSNqlWr0rx5c8A79N56cz27CbR8ed8NLV999dUsv6tKlSoauu+H\nU6dOZfmsaNGi1K9f34ZonOPTTz/lgQceADKGOGd22223Ad5+0gvZvn07n376qTkuX768aUWpXLly\nIMINCi+99BJAltrTI488YmpMxYsXp2zZsmaqztmSkpJYtmwZQ4YMAbyrpURFRTFs2DAAbr755rwK\nP0ccnaAuxLqpK1asaHMkzmMl7d69e5sb2N9BDtbwUrmwxMREwDtA4mzXXXedWeqoMLNWhLkYf/75\np+lLtbRs2bJQrphi9Q0dOnSIW265hbvuuguAChUqZOkOOZd//vnHDIIC71y/t99+2zEvVI5MUFYH\n/8mTJ30+T0hI4IMPPuCXX34BvKOCXC4XR48eBbyJyppMWdhZkxU9Hg9Tp04Fsq8hnY/mk+TMyZMn\nTSe+dW9mNn78+PwOqcCxOv379u3L0qVLzed16tRh/PjxWfpWCgPrb3zatGl+f9d6abVGSFutJAsX\nLnRMcgL1QYmIiEM5pgZljSbbs2cPb731FuAd9pyZx+PxGR0VEhJCmTJlzBvq+PHjqVKlSj5F7GzW\ncNvsRvNdiDW7fP/+/WYZKTm3f/75hzFjxphjt9tthpm3bNnSZ86U+O/gwYOmlcSa02fVHh588EGK\nFy9uW2zBKCkpyfRfWcvHrV27FnDeXEdHJKjPPvvMdOpt377dfB4REUHPnj19rp0/fz67du0CvJ2j\n27ZtM0NSJYOVoJYsWeL33JC///4bgF27dvm1NFJhFBcXxxNPPJHlxck6zs0LgviaPXu26cS3rFix\nAoBrr73WjpCC2v79+9mwYYM5jo+Pz3GfVX5zRIJq164d1apVA7wzyh966CHAO+os88gzt9vN3r17\nTYJas2aNktMFWHOe/GHNO4mMjOS6664LdEgFyqJFi8ys/eycvX6kZLDWKzx58qSZHG6x5j7u37/f\n56UVvPMjreeF+Ofo0aP07t3brNxRr149Ro0a5djRj+qDEhERR3JEDapIkSJmS4jzbQ0xefJkPv30\nU9q3bw9gNi6UwCpdujQAdevWZdy4cbmqhRUWmVcxz6xEiRIAZoUDycpqSs5u7pM14tRapdxqgpo9\nezYdO3Z0bJOUUyUnJwMwc+ZM1q1bZ0bt9e7dm9atW9sZ2nk5IkFdiLUUf//+/XG5XIwYMQLgnOud\nieQ1a0HTc22rsWTJEgAN2jmPc03KPXToENOnT8/2XIMGDZSc/JScnGyaoYcMGUKxYsXMPMl+/frZ\nGdoFOT5Bud1uWrZsaY5Hjx5N3bp1bYyo4LMW84yLi8uyDpp4bdq0Cch+5Qi3202DBg3yO6QC4eDB\ngzRu3NjMwatQoQKDBw+mb9++AIV2JfjcSklJ4ZlnnmHKlCkAtGrVitGjR3PNNdfYHFnOqAoiIiKO\n5Pga1M8//8zPP/8MeFeKePzxx9W0l8esNQ5jY2PNmn7i63zDyHV/5l6TJk18VjDp0qULzz77rI0R\nBafdu3cDMHjwYBYsWGD6kT/55JOgmjfm+AT1xx9/mJ+nTJliOvAl71j9Kh6Px2xuKL6svqWwsDDT\nAS25c+jQIbp3725+rlSpEosWLQI0zyk3EhMTzcTxBQsWcNlll/Hee+8BBFVyAjXxiYiIQzm+BhUZ\nGcktt9wCaOv3/DJv3jzA23ylibrZs+7J+vXr89NPP/mcK126tFaQ8MOkSZP46quvzHGHDh1o0qSJ\njREFt+uuu86n5SkmJoZ//etfNkaUe46vQUVFRfH111+bLZ4l/3g8Hs2BuoDslpL64YcfKFGihJkL\nJedXsmRJKlasaP7r0aOH3SEFtcmTJxMWFkZYWBhRUVHn3W3X6RxfgxL7NGzY0O4QHC8iIoJt27bZ\nHUZQGzJkSJa19iT3WrRoYfYnC3aOr0GJiEjhpBqUZDFy5Eiff0VE7OCydq/N0cUu1zFgb96FEzQu\n93g85S/2l6g8DZVn4F10mao8fegeDawcladfCUpERCS/qA9KREQcSQlKREQcSQlKREQcSQlKREQc\nSQlKREQcSQlKREQcSQlKREQcSQlKREQcSQlKREQcSQlKREQcSQlKREQcSQlKREQcSQlKREQcSQlK\nREQcSQlKREQcSQlKREQcya8t3yMiIjzVq1fPo1CCx6ZNm44HYndNlaeXyjPwAlGmKs8MukcDK6fl\n6VeCql69Ohs3bsx9VAWEy+UKyJbNKk8vlWfgBaJMVZ4ZdI8GVk7LU018IiLiSEpQIiLiSEpQIiLi\nSEpQIiLiSEpQIiLiSH6N4pPg5/F4ePfddwHo06cPAHPnzgXgoYcesi0uEZGzKUEVMu+//z79+vUD\nICTEW4Fes2YNoASVG3v37uW3337jxIkTAHTt2pXGjRtTuXJlAJ544gkALrnE+6d255132hOow+3Z\nsweAmjVrUqZMGVOeF7J7926+/fZbOnbsCEDRokXzLMZg5Ha7SU1NZd68eQD8/fffAKxcuRKA5cuX\nA/Doo48CUK9ePa666ipzn1rPCLuoiU9ERBzJthrUyZMnAVi9enWWc4cPHwagf//+uN3uLFnc7XYD\n3uweGRlJ06ZNAejcuTP33Xef3qLO4fjx40yYMCHL5xs2bAAgPj6eEiVK5HdYjjRp0iTatWsHQK9e\nvUhLS8v2uh07dvDXX3+Z45CQEGJiYoiJiQHgf//7HwBhYWEAHDlyhJIlS+Zl6EEtJCSE06dPM2DA\nAADuu+8+qlWrRqVKlQDvPbp161ZSU1MB6NSpE2fOnGHQoEEAbNy4kapVq9oTvIMkJiYC8MYbb/DK\nK69kOe/xeABwuVwAzJgxw+f8zJkzAejWrVseRnlhtiWovXu9E4k7dep0zmtcLhchISGmEC1WwnK5\nXBw5coRFixYBsHDhQo4dO6YEdZbjx48D0Lx5c3bs2EG9evUA78330ksv8dtvvwFKUJZdu3YxZswY\nnn766YD9zuTkZABatmzJl19+qSR1Hh6Ph4kTJwIwceJEypQpQ9myZQFISkri0KFDWb5j3ePWy2th\nduDAAZo1awZ4n7O1atWiWLFi5vzo0aMJDQ0FMhLUzp07gYx+aevYbmriExERR7KtBhUeHg5AyZIl\nOX36tM85qzmkTp06pKenU6RIEZ/z6enpgPeNf9++ffkQbfA6duwYUVFRAGzfvp1KlSoxfPhwANq1\na8fq1atZtWoVAAkJCXaF6Sg1atSgbt26Pm/qZcqUoUePHrn6fTNnzjRN2hs2bKB58+Z89dVXAJQr\nV+6i4y3oTp48acqvePHiNG/enG+++cbnmrZt2wIQGRmZ7/E5zfTp000LVZcuXZg2bdoFW5XKl89Y\nt7VMmTJmIJXdbEtQderUAWDevHm0bNky23MxMTFZmvcyO3XqFHfddRebNm3Ku0CDlJVsoqKi2L59\nO+Ad4bRmzRpTvmd78803TdNKYRYSEsLYsWPp1asXAB9//DHFihWjWrVqufp9d911F/fee6853rJl\ni1kw9O677774gAuADz/80Oe4b9++gPcBm1np0qWpXLmyafKzPP/880DGy21h9txzz/HAAw8A3mfp\nhZJTamoqr776qjl+6qmnfBKWnWwfZt6wYcMsn/3yyy8AfPHFF6ajOjtJSUnExcX5fLZp0yYN5SVj\nWHOLFi245pprAHjppZfOmZzEV8OGDc1Ah4tVs2bNgPyegmzz5s0+x9deey0AN954Y5ZrP//8c5/j\nSy+9lFq1auVdcEEmLCyM+vXr5+jatLQ0hg0bxoIFCwBvTcrqh3IC9UGJiIgj2V6DKlOmDNOnTzcT\nxQAzwulCG3tVrFiRxo0b+4w4iY2NVQ0K71slwDvvvGNzJHK2W2+9lRYtWtgdhqPcd999ACxevJii\nRYty6623ZntdfHw848aN8/ls0KBBlClTJq9DLJBmzpzpU54ff/yxo/rxbE9QRYoUoXv37qapb+jQ\noSZBZdf8dyGZE52I3ZKSkhg5cqTPZ6GhoVkG/hR27du3B7xzIDt27MgVV1yR7XWPP/44P/30kzm+\n8847GTZsWL7EWJDs3r0bgCFDhgAZw8ut4elOYXuCAu9YfKvNecmSJRf1u5YsWcIjjzwSiLAKnZ49\ne9odQoEzf/58PvroI7vDcDyrBvTCCy9ke95qJVm4cCGQMRdy+PDhprVAcq5BgwYAnDlzhrp165oJ\n/FbftVOoD0pERBzJWenSTydPnuT33383y3a43W5at25tc1Qi3qY9yBgubalbty4ffPCBHSEFrfj4\neP7zn/8A3nItWrSoWYHfWuZMcu6xxx4jPj4e8A7bX7p0qeNqThZnRpVDe/fu5eeffzZzpexeeTfY\nzZgxQ/OgAuzsaRCdO3c268pJzixatMisaQjewVH333+/jREFr88//5zp06ebl/pZs2ZRo0YNm6M6\nt6BOUCJSsA0cOJBp06aZ40svvdQsDCs5Z03cX7x4MS6XyyzG6/QWJ1U5RETEkVSDEuPs/hLJvbOH\nllvNzzmd4V/YWauSv/vuu6SkpJjPJ0yYwJNPPmlXWEEpOTnZjNrbtWsX5cuXz3bbHScK+gRltaWC\nltq/WKVLl7Y7hAIhPT09y5YQ1hpxbdq0sSOkoJKWlmZ2d7aS05VXXgmcf3seySo5OZk+ffqwa9cu\nwLuYbnR0tM1R5VxQJ6jnn3/eZzHZmTNn6iF7ESZNmpTlzV/8t23bNrPhm0UP1pwbM2YMX3zxhTmu\nV68e69atA+Cyyy6zKarg9Pvvv/uMGn3jjTeoUqWKjRH5R31QIiLiSEFZg1qxYgUA33//PZCxB4xW\nkLg4p06dsjuEAmHHjh0+x2XLlmXgwIE2RRNcTp06laV/ZMWKFao55dL06dOBjHVNrSWlgkVQJihr\nbx2reU8LogbGli1bzERIyb3Bgwf7HEdFRVGvXj2bogkOycnJgHfposwvSuPHj6dChQp2hRW0/vrr\nL8A7zwkwm5IG29+2mvhERMSRgrIGZY3W08oRgfXtt98yYsQIRo8ebXcoUsgcPHgQyNoa0q9fP638\nngvWFhqnT5+mbt26VKxY0eaIcicoE5SVmM63HbzkzKxZsxg+fDjg3Y34xRdftDegAui1116zO4Sg\nY225I/5LSEgw/fPg3eGhePHiNkaUe0GZoDILCwujXLlydocRtCpUqMDkyZPtDqNAeeaZZ+jfvz/g\nXXvvQhtviq+iRYsSExMDoNpTLvzzzz+m/B566CFHr7V3IWojExERRwrKGtRnn30GeLcn7tevH1FR\nUTZHJJKhT58+ZodSyRnrLT89Pd3mSIJflSpVCkw5ujIvFXTBi12uY8DevAsnaFzu8XjKX+wvUXka\nKs/Au+gyVXn60D0aWDkqT78SlIiISH5RH5SIiDiSEpSIiDiSEpSIiDiSEpSIiDiSEpSIiDiSEpSI\niDiSEpSIiDiSEpSIiDiSEpSIiDjS/wP9oERMAInE+QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10447a240>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(nrows=5, ncols=5, sharex=True, sharey=True,)\n",
    "ax = ax.flatten()\n",
    "sevens = X_train[y_train == 7]\n",
    "for i in range(25):\n",
    "    img = sevens[i].reshape(28, 28)\n",
    "    ax[i].imshow(img, cmap='Greys', interpolation='nearest')\n",
    "\n",
    "ax[0].set_xticks([])\n",
    "ax[0].set_yticks([])\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### L-BFGS Optimization for MNIST\n",
    "This is a nice test to show that mini-batching has better conversion properties for some datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epoch: 79/100 (max)"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'task': b'STOP: TOTAL NO. of f AND g EVALUATIONS EXCEEDS LIMIT', 'nit': 79, 'warnflag': 1, 'funcalls': 101, 'grad': array([ -0.41771607,   0.04243388,  -0.09627657, ...,   1.52028738,\n",
      "        -0.33353427,  24.48449635])}\n",
      "Validation acc: 0.9439\n",
      "CPU times: user 2min 10s, sys: 32.6 s, total: 2min 43s\n",
      "Wall time: 2min 4s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "nn_bfgs = TLPVectorizedBFGS( \n",
    "                      n_hidden=50, \n",
    "                      C=0.1, \n",
    "                      epochs=100, # max iterations\n",
    "                      gtol=1e-9,\n",
    "                      random_state=1)\n",
    "\n",
    "nn_bfgs.fit(X_train, y_train, print_progress=1)\n",
    "yhat = nn_bfgs.predict(X_test)\n",
    "print('Validation acc:',accuracy_score(y_test,yhat))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MiniBatch Gradient Descent for MNIST"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epoch: 100/100"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation acc: 0.9458\n",
      "CPU times: user 2min 24s, sys: 49.3 s, total: 3min 13s\n",
      "Wall time: 2min 55s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "nn_mini = TLPVectorizedMiniBatch( \n",
    "                          n_hidden=50, \n",
    "                          C=0.1,  \n",
    "                          epochs=100, \n",
    "                          eta=0.001,\n",
    "                          alpha=0.001,\n",
    "                          decrease_const=0.00001,\n",
    "                          minibatches=50, \n",
    "                          shuffle=True,\n",
    "                          random_state=1)\n",
    "\n",
    "nn_mini.fit(X_train, y_train, print_progress=1)\n",
    "yhat = nn_mini.predict(X_test)\n",
    "print('Validation acc:',accuracy_score(y_test,yhat))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHQxJREFUeJzt3X20XXV95/H3NzcJSbghIeQKIQGCkkojEmVSZMBlH6wt\ntE7jdKSGQkelDtIZFVxVCrpc7bgca+sTOmVkobWlg4X6QG2qCFrFmakgkKA8ptgQEBIJSXjIA4Tc\nPHznj985vSeHe29ucu+5e+ee92utvfbZv733yfds9X787f3be0dmIklS3UyqugBJkgZjQEmSasmA\nkiTVkgElSaolA0qSVEsGlCSplgwoSVItGVCSpFoyoCRJtTS56gLG0ty5c3PhwoVVlyFJGsaqVas2\nZ2bf/rabUAG1cOFCVq5cWXUZkqRhRMRPR7Kdp/gkSbVkQEmSasmAkiTVkgElSaolA0qSVEsGlCSp\nlgwoSVItGVBNq1bBd75TdRWSpIYJdaPuqHziEyWkfvKTqiuRJGEPakBvL2zfXnUVkqQGA6rJgJKk\nWjGgmpoBlVl1JZIkOhxQEXF2RDwUEWsi4vJB1p8cEbdHxM6IeN8g63si4kcR8Y1O1gmUgMqEHTs6\n/k9JkvavYwEVET3AVcA5wGLgvIhY3LbZ08B7gE8M8TWXAKs7VeM+envL3NN8klQLnexBnQ6sycy1\nmdkP3AAsa90gMzdm5l3ArvadI2IB8JvAFzpY44CZM8t827Zx+eckScPrZEDNBx5vWV7XaBupK4HL\ngL3DbRQRF0XEyohYuWnTpgOvsskelCTVSi0HSUTEG4GNmblqf9tm5jWZuTQzl/b17fcFjUMzoCSp\nVjoZUOuB41qWFzTaRuIs4Lci4lHKqcFfiYjrxra8NgaUJNVKJwPqLmBRRJwYEVOB5cCKkeyYmVdk\n5oLMXNjY73uZeUHnSsWAkqSa6dijjjJzd0S8C7gF6AG+mJkPRMTFjfVXR8QxwErgCGBvRFwKLM7M\nrZ2qa0gGlCTVSkefxZeZNwE3tbVd3fJ5A+XU33Df8X3g+x0ob18GlCTVSi0HSVTCYeaSVCsGVNO0\naTBpkj0oSaoJA6opwgfGSlKNGFCtDChJqg0DqpUBJUm1YUC1MqAkqTYMqFYGlCTVhgHVqrfXYeaS\nVBMGVKuZM+1BSVJNGFCtPMUnSbVhQLUyoCSpNgyoVs2Ayqy6EknqegZUq95e2LsXXnih6kokqesZ\nUK18orkk1YYB1aoZUA41l6TKGVCtmq/csAclSZUzoFp5ik+SasOAamVASVJtGFCtDChJqg0DqpUB\nJUm1YUC1MqAkqTYMqFYGlCTVhgHVasYMiPA+KEmqAQOqVYQPjJWkmjCg2hlQklQLBlQ7A0qSasGA\namdASVItGFDtDChJqgUDqp0BJUm1YEC16+11mLkk1YAB1W7mTHtQklQDBlQ7T/FJUi0YUO2aAZVZ\ndSWS1NUMqHa9vbBnD+zcWXUlktTVDKh2PjBWkmrBgGpnQElSLRhQ7ZoB5VBzSaqUAdXOHpQk1YIB\n1W7mzDI3oCSpUgZUO3tQklQLHQ2oiDg7Ih6KiDURcfkg60+OiNsjYmdEvK+l/biIuDUiHoyIByLi\nkk7WuQ8DSpJqYXKnvjgieoCrgDcA64C7ImJFZj7YstnTwHuAN7Xtvhv4w8y8OyJmAqsi4jtt+3aG\nASVJtdDJHtTpwJrMXJuZ/cANwLLWDTJzY2beBexqa38iM+9ufN4GrAbmd7DWAQaUJNVCJwNqPvB4\ny/I6DiJkImIh8GrgjiHWXxQRKyNi5aZNmw6izDYzZpS5ASVJlar1IImI6AW+BlyamVsH2yYzr8nM\npZm5tK+vb/T/6KRJcPjh3gclSRXrZECtB45rWV7QaBuRiJhCCacvZeaNY1zb8HyiuSRVrpMBdRew\nKCJOjIipwHJgxUh2jIgA/hJYnZmf6mCNg/OdUJJUuY6N4svM3RHxLuAWoAf4YmY+EBEXN9ZfHRHH\nACuBI4C9EXEpsBg4Ffg94L6I+HHjKz+QmTd1qt592IOSpMp1LKAAGoFyU1vb1S2fN1BO/bX7ZyA6\nWduwDChJqlytB0lUxoCSpMoZUIMxoCSpcgbUYHp7HWYuSRUzoAZjD0qSKmdADaY5zDyz6kokqWsZ\nUIPp7YXdu6G/v+pKJKlrGVCD8YGxklQ5A2owBpQkVc6AGkwzoBzJJ0mVMaAGc+SRZf7009XWIUld\nzIAaTPO1HWPxfilJ0kExoAbTDKjNm6utQ5K6mAE1mLlzy9welCRVxoAazGGHlZt1DShJqowBNZS+\nPgNKkipkQA3FgJKkShlQQzGgJKlSBtRQDChJqpQBNZRmQPlEc0mqhAE1lL4+2LULtm6tuhJJ6koG\n1FC8WVeSKmVADcXHHUlSpQyoofg0CUmqlAE1FHtQklQpA2ooBpQkVcqAGsrhh8O0aQaUJFXEgBpK\nhDfrSlKFDKjhGFCSVBkDajgGlCRVxoAaTl+fN+pKUkUMqOHYg5KkyhhQw+nrg+eegx07qq5EkrqO\nATUc74WSpMoYUMPxcUeSVBkDajj2oCSpMgbUcAwoSarMiAIqIv73SNomHANKkioz0h7UK1oXIqIH\n+HdjX07NzJ4NkycbUJJUgWEDKiKuiIhtwKkRsbUxbQM2Av8wLhVWKaIMlPBmXUkad8MGVGb+aWbO\nBD6emUc0ppmZeVRmXjFONVbLm3UlqRIjPcX3jYg4HCAiLoiIT0XECfvbKSLOjoiHImJNRFw+yPqT\nI+L2iNgZEe87kH3HjQElSZUYaUB9Dng+IpYAfwg8DPzNcDs0rlNdBZwDLAbOi4jFbZs9DbwH+MRB\n7Ds+DChJqsRIA2p3ZiawDPiLzLwKmLmffU4H1mTm2szsB25o7P9vMnNjZt4F7DrQfceNASVJlRhp\nQG2LiCuA3wO+GRGTgCn72Wc+8HjL8rpG20iMeN+IuCgiVkbEyk2dCJK+Pnj2WdjVnqGSpE4aaUC9\nBdgJXJiZG4AFwMc7VtUByMxrMnNpZi7ta963NJaajztyJJ8kjasRBVQjlL4EzIqINwIvZOaw16CA\n9cBxLcsLGm0jMZp9x5Y360pSJUb6JInfAe4EzgV+B7gjIt68n93uAhZFxIkRMRVYDqwYYV2j2Xds\nGVCSVInJI9zug8AvZOZGgIjoA/4J+OpQO2Tm7oh4F3AL0AN8MTMfiIiLG+uvjohjgJXAEcDeiLgU\nWJyZWwfb9+B+4ig1A8pTfJI0rkYaUJOa4dTwFCPofWXmTcBNbW1Xt3xuXs8a0b6VsAclSZUYaUDd\nHBG3ANc3lt9CHcJjPBx1VHnk0ZNPVl2JJHWVYQMqIk4Cjs7M90fEbwOvbay6nTJoYuLr6YGXvQxW\nr666EknqKvvrQV0JXAGQmTcCNwJExCsb6/5DR6uri1NPhXvuqboKSeoq+7uOdHRm3tfe2Ghb2JGK\n6mjJEnj4Ydi+vepKJKlr7C+gZg+zbvpYFlJrS5ZAJtx/f9WVSFLX2F9ArYyI/9LeGBHvAFZ1pqQa\nWrKkzD3NJ0njZn/XoC4F/j4izmcgkJYCU4H/2MnCauWEE2DWLANKksbRsAGVmU8CZ0bELwOnNJq/\nmZnf63hldRLhQAlJGmcjug8qM28Fbu1wLfW2ZAn89V/D3r0waaTP2JUkHSz/0o7UkiVlFN8jj1Rd\niSR1BQNqpBwoIUnjyoAaqVNOKaf2DChJGhcG1EhNnw4/93MGlCSNEwPqQCxZYkBJ0jgxoA7EkiXw\n6KOwZUvVlUjShGdAHYjmQIl77622DknqAgbUgXAknySNGwPqQBx7bHmBoQElSR1nQB2ICAdKSNI4\nMaAO1NKlJaBeeKHqSiRpQjOgDtSZZ0J/P6zqnreNSFIVDKgDdeaZZf6DH1RbhyRNcAbUgerrg0WL\nDChJ6jAD6mCcdRbcdlt5DbwkqSMMqINx5pmweTP8679WXYkkTVgG1ME466wyv+22auuQpAnMgDoY\nJ58Ms2d7HUqSOsiAOhiTJpXTfPagJKljDKiDddZZ8OCD8PTTVVciSROSAXWwmvdD3X57tXVI0gRl\nQB2s00+Hnh5P80lShxhQB2vGDHj1qx0oIUkdYkCNxllnwZ13wq5dVVciSROOATUaZ50FO3bAD39Y\ndSWSNOEYUKNx9tkwfTpcf33VlUjShGNAjcbMmfCmN8Hf/V15BYckacwYUKN1wQXlXqibb666Ekma\nUAyo0XrDG8orOK67rupKJGlCMaBGa8oUWL4cVqyALVuqrkaSJgwDaiycfz7s3Alf+1rVlUjShNHR\ngIqIsyPioYhYExGXD7I+IuKzjfX3RsRpLeveGxEPRMT9EXF9REzrZK2jcvrpcNJJnuaTpDHUsYCK\niB7gKuAcYDFwXkQsbtvsHGBRY7oI+Fxj3/nAe4ClmXkK0AMs71StoxZRBkt8//vw+ONVVyNJE0In\ne1CnA2syc21m9gM3AMvatlkG/E0WPwRmR8S8xrrJwPSImAzMAH7WwVpH7/zzyyvgr7226kokaULo\nZEDNB1q7E+sabfvdJjPXA58AHgOeALZk5rcH+0ci4qKIWBkRKzdt2jRmxR+wk06Cc86BT34Snnqq\nujokaYKo5SCJiDiS0rs6ETgWODwiLhhs28y8JjOXZubSvr6+8Szzxf78z2HrVvjIR6qtQ5ImgE4G\n1HrguJblBY22kWzzq8AjmbkpM3cBNwJndrDWsXHKKXDhhXDVVfDww1VXI0mHtE4G1F3Aoog4MSKm\nUgY5rGjbZgXwnxuj+c6gnMp7gnJq74yImBERAbweWN3BWsfOhz9c7o264oqqK5GkQ1rHAiozdwPv\nAm6hhMuXM/OBiLg4Ii5ubHYTsBZYA3we+K+Nfe8AvgrcDdzXqPOaTtU6pubNg/e/H77yFd+2K0mj\nEJlZdQ1jZunSpbly5cqqy4Dt22HRIjj++PJCw8mTq65IkmojIlZl5tL9bVfLQRKHvN5e+PSny8sM\nP/jBqquRpEOSAdUpy5fDO99ZRvataL/0JknaHwOqk668Ek47Dd76VnjkkaqrkaRDigHVSdOmlcES\nmXDuufDCC1VXJEmHDAOq01760vL4o1Wr4O1vh717q65Ikg4JBtR4WLYMPvYxuOEG+NCHqq5Gkg4J\njn8eL5ddVp4u8dGPwoknwjveUXVFklRrBtR4iSiPQHrsMbj44hJSr3991VVJUm15im88TZkCX/5y\nuYn3wgvhueeqrkiSasuAGm9HHAGf/3zpSf3Jn1RdjSTVlgFVhde+tlyD+vSn4Z57qq5GkmrJgKrK\nn/0ZzJlTnjaxZ0/V1UhS7RhQVZkzBz71KbjjDrjm0HhQuySNJwOqSuefX0byfeAD8MwzVVcjSbVi\nQFUpAj75SXj2Wfj4x6uuRpJqxYCq2pIlcN558JnPwIYNVVcjSbVhQNXBhz8M/f3wkY9UXYkk1YYB\nVQcnnQS///tlsISv5ZAkwICqjw99CHp64I//uOpKJKkWDKi6mD8f3v1uuO46uO22qquRpMoZUHVy\nxRXl/VG//dvw+ONVVyNJlTKg6uTII2HFCnj+eXjTm8pckrqUAVU3ixfD3/4t/OhH5Q28mVVXJEmV\nMKDq6I1vhD/90/Jqjksv9Vl9krqSLyysq8sugyeeKDfw/uQncP31MHt21VVJ0rixB1VXEXDllXD1\n1fBP/wRnnFGCSpK6hAFVd+98J3z3u/DUU3DaaSW0du+uuipJ6jgD6lDwutfBqlXwi78I730vvOY1\nZVmSJjAD6lBx/PHwjW/AV75Srk39wi/A7/4urF5ddWWS1BEG1KEkAt785hJKl11W7pl6xStg+fIy\nLF2SJhAD6lA0axZ87GPw6KNw+eXwzW+W61NnnAHXXgs7dlRdoSSNmgF1KJs7Fz760fJYpCuvhC1b\n4G1vg3nzyk2+3/62AyokHbIMqIlg9my45BJ48EH43vfKY5JuvBF+/dfh2GPhwgvh61+H556rulJJ\nGrHICfQonaVLl+bKlSurLqMeXngBvvWt8jSKb32r9K4OO6yMBPy1XyvTKaeU61qSNI4iYlVmLt3v\ndgZUF9i1C/75n+Ef/xFuuaX0tACOPhpe+9qBackSmDKl2lolTXgjDSgfddQNpkyBX/7lMgGsW1eu\nT916K/zgB/C1r5X26dNh6dIy2OL008vnE06wlyWpEvagBOvXl6D64Q/h9tvh7ruhv7+smzOnjBA8\n9VR45SvLtHhxCTNJOgie4tPB27kT7ruvPK1i5coSWA8+WK5rAUyaBC97WbmG9fM/DyefXKaXvxyO\nOKLa2iXVngGlsbVnD6xZA/feCw88APffX6Y1a/Z9Hcgxx5SgevnLS4i99KUD81mzqqtfUm3U4hpU\nRJwNfAboAb6QmR9rWx+N9b8BPA+8LTPvbqybDXwBOAVI4MLMvL2T9WoYPT0DwXPuuQPt/f2wdi08\n9BD8y7+U+UMPletaTz2173fMmVOCauHCcm3rhBPKI5ya8yOP9HqXpH/TsYCKiB7gKuANwDrgrohY\nkZkPtmx2DrCoMb0G+FxjDiW4bs7MN0fEVGBGp2rVKEydOnCKb9myfddt2QKPPAIPP1zma9eW6d57\ny3MFm6cMm3p74bjjYMGCgWn+/IHp2GOhr6+cYpQ04XWyB3U6sCYz1wJExA3AMqA1oJYBf5PlPOMP\nI2J2RMyj9KZeB7wNIDP7gf4O1qpOmDULXvWqMrXLhE2b4Kc/hcceK9NPf1qeirFuXTmN+MQTL37l\n/eTJ5TTivHllPth09NHwkpeUwLNHJh2yOhlQ84HHW5bXMdA7Gm6b+cBuYBPwVxGxBFgFXJKZPgph\noogoIfKSl5Qnsw9m927YsKGMMly/vgTWz35WPm/YUELtzjth48YXBxnAtGkDYdWc5s7dd+rrK/Oj\njipP5LB3JtVGXe+DmgycBrw7M++IiM8AlwMfat8wIi4CLgI4/vjjx7VIddjkyQOn+oazezds3lxC\na8MGePLJElpPPll6aRs3lnC7556yXfupxaaIch1szpx9Q2zOnIHpyCP3nY46qvQUe3rG/vdLXa6T\nAbUeOK5leUGjbSTbJLAuM+9otH+VElAvkpnXANdAGcU3+rJ1yGme9jvmmJFt//zzJbg2by7zTZvK\ngI6nny7T5s1lef16+PGP4Zlnhn+OYTPYZs8u06xZZWp+bm1vzo84YmCaNav09jwdKe2jkwF1F7Ao\nIk6khM5y4HfbtlkBvKtxfeo1wJbMfAIgIh6PiJdn5kPA69n32pV08GbMGBhFOFI7d5bweuaZMj37\n7ECgNcPt2WfLwJBnninD77dsKW3btu3/+6dM2Te0msHVujxzZrmu1jofbDLsNEF0LKAyc3dEvAu4\nhTLM/IuZ+UBEXNxYfzVwE2WI+RrKwIi3t3zFu4EvNUbwrW1bJ42vww4rAzPmzTvwfffsga1bS1g9\n+2z5vHVrCbDm5/blLVvK9bbVq8vnbdtKSI5ET89AWB1++MDU2zv41FzXuu2MGWVq3Xf6dK/RaVx5\no650qNi1qwTVtm2wffvA5/apdd1zz5Vp+/aBeXOb7dth794Dq2H69H0DrDlNnz50W/t82rTyuTm1\nbt+cpk61FziB1eJGXUljaMqUgcEaYyGz9Mqa4dUMsOefL9OOHS9et2NHWffccwPbPf986fU9+eS+\n+z7//Mh7fe0mTSpB1jo1w2vatNKjbc6bba3btS63btuct05DrZs82ZCsmAEldauIgT/ic+d25t/Y\ns6eMmmwG1o4dL55aQ6112rlzYN/29h07yunS9u2a87E4MxTx4hCbMqX07qZOHVjX/Nw6b51a25r7\nN+et39G+rr1tqPbJkyfsqVcDSlLn9PQMXMcaL5nldGgzuJpTc3nnzjL19w98bg241vbW9f39A1Nz\n//7+gbBsb28uN2vp5OWUSZMGQmyoqTXoJk8u01Cfm8vtQTh5cvnP9Nxzy1sNOsyAkjSxRAz0NGbO\nrLqaAbt3l7BqhlczuFqDrLm+fbvB5v39A9+5a1f53Lq+dV3rd+/eXabm6d3m8q5dpcfb/Nz8vuZ3\n7tlT5lDeZGBASdIE0eyBHOrvUjvQgTWjYEBJkkZuHK93Tcwra5KkQ54BJUmqJQNKklRLBpQkqZYM\nKElSLRlQkqRaMqAkSbVkQEmSamlCvW4jIjYBPx3FV8wFNo9ROYc6j8UAj0XhcRjgsRhwMMfihMzs\n299GEyqgRisiVo7kHSXdwGMxwGNReBwGeCwGdPJYeIpPklRLBpQkqZYMqH1dU3UBNeKxGOCxKDwO\nAzwWAzp2LLwGJUmqJXtQkqRaMqAkSbVkQDVExNkR8VBErImIy6uuZ7xExHERcWtEPBgRD0TEJY32\nORHxnYj418b8yKprHS8R0RMRP4qIbzSWu/JYRMTsiPhqRPxLRKyOiH/fjcciIt7b+N/G/RFxfURM\n65bjEBFfjIiNEXF/S9uQvz0irmj8DX0oIn59tP++AUX5gwRcBZwDLAbOi4jF1VY1bnYDf5iZi4Ez\ngP/W+O2XA9/NzEXAdxvL3eISYHXLcrcei88AN2fmycASyjHpqmMREfOB9wBLM/MUoAdYTvcch78G\nzm5rG/S3N/5uLAde0djnfzX+th40A6o4HViTmWszsx+4AVhWcU3jIjOfyMy7G5+3Uf4Izaf8/msb\nm10LvKmaCsdXRCwAfhP4Qktz1x2LiJgFvA74S4DM7M/MZ+nCYwFMBqZHxGRgBvAzuuQ4ZOb/BZ5u\nax7qty8DbsjMnZn5CLCG8rf1oBlQxXzg8ZbldY22rhIRC4FXA3cAR2fmE41VG4CjKyprvF0JXAbs\nbWnrxmNxIrAJ+KvG6c4vRMThdNmxyMz1wCeAx4AngC2Z+W267Di0Geq3j/nfUQNKAEREL/A14NLM\n3Nq6Lsu9CBP+foSIeCOwMTNXDbVNtxwLSq/hNOBzmflq4DnaTmN1w7FoXF9ZRgnsY4HDI+KC1m26\n4TgMpdO/3YAq1gPHtSwvaLR1hYiYQgmnL2XmjY3mJyNiXmP9PGBjVfWNo7OA34qIRymneX8lIq6j\nO4/FOmBdZt7RWP4qJbC67Vj8KvBIZm7KzF3AjcCZdN9xaDXUbx/zv6MGVHEXsCgiToyIqZQLfSsq\nrmlcRERQrjOszsxPtaxaAby18fmtwD+Md23jLTOvyMwFmbmQ8t+B72XmBXTnsdgAPB4RL280vR54\nkO47Fo8BZ0TEjMb/Vl5PuU7bbceh1VC/fQWwPCIOi4gTgUXAnaP5h3ySRENE/Abl+kMP8MXM/B8V\nlzQuIuK1wP8D7mPgussHKNehvgwcT3mFye9kZvvF0gkrIn4JeF9mvjEijqILj0VEvIoyWGQqsBZ4\nO+X/1HbVsYiI/w68hTLi9UfAO4BeuuA4RMT1wC9RXqnxJPDHwNcZ4rdHxAeBCynH6tLM/Nao/n0D\nSpJUR57ikyTVkgElSaolA0qSVEsGlCSplgwoSVItGVBSB0XEnoj4ccs0Zg8VjYiFrU+ZliaayVUX\nIE1wOzLzVVUXIR2K7EFJFYiIRyPizyPivoi4MyJOarQvjIjvRcS9EfHdiDi+0X50RPx9RNzTmM5s\nfFVPRHy+8b6ib0fE9Mb272m84+veiLihop8pjYoBJXXW9LZTfG9pWbclM18J/AXlKSYA/xO4NjNP\nBb4EfLbR/lng/2TmEsoz8R5otC8CrsrMVwDPAv+p0X458OrG91zcqR8ndZJPkpA6KCK2Z2bvIO2P\nAr+SmWsbD+vdkJlHRcRmYF5m7mq0P5GZcyNiE7AgM3e2fMdC4DuNF8cREX8ETMnMj0TEzcB2ymNp\nvp6Z2zv8U6UxZw9Kqk4O8flA7Gz5vIeB68q/SXlL9GnAXY2X7UmHFANKqs5bWua3Nz7fRnmSOsD5\nlAf5Qnm19h8ARERP4423g4qIScBxmXkr8EfALMrDTaVDiv+vSuqs6RHx45blmzOzOdT8yIi4l9IL\nOq/R9m7KW2zfT3mj7dsb7ZcA10TE71N6Sn9AecPrYHqA6xohFsBnG69rlw4pXoOSKtC4BrU0MzdX\nXYtUV57ikyTVkj0oSVIt2YOSJNWSASVJqiUDSpJUSwaUJKmWDChJUi39f8MI8UgyyXqKAAAAAElF\nTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10d7455f8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# take the mean of each minibatch epoch\n",
    "cost_avgs = [np.mean(x) for x in nn_mini.cost_]\n",
    "\n",
    "plt.plot(range(len(cost_avgs)), cost_avgs, color='red')\n",
    "plt.ylabel('Cost')\n",
    "plt.xlabel('Epochs')\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaYAAAEYCAYAAAAXsVIGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXd8FOX2/99nE0JLCD10IgiCIFIURKQI4rVhA6Vciopi\nQ+Ha0GtDxKsofvWH4BWwICIWrlgQQUGqBtQgTa9iBWmhJyEhBcj5/TGT3CUkm93NTHY3PO/Xa17J\nzlPOmc8+M2eesjOiqhgMBoPBEC54Qu2AwWAwGAzemMBkMBgMhrDCBCaDwWAwhBUmMBkMBoMhrDCB\nyWAwGAxhhQlMBoPBYAgrHA1MItJORJKcrPNUxujpLEZP5zGaOovR0yLgwCQio0UkWURyRGSWd5qq\nbgJSRaSfj/IrROTmAG2qiGSKSIa9vRqo3wHYihKRiSKyS0QOi8h6Eanukq2KIvKaiGyzbW0QkUvz\n093S06vscFvboMr7aaO9iKwTkSP23/Zu2bLtzRGRFBFJF5FfvI/NDT1FpKWIfCwi+0TkoIh8LiJn\nlPIwirNVW0S+FpEDIpImImtEpJsbtoqx30JEskVkTv6+cqBpd6/rSv6mItLfDXu2zRW2jvn2tuSn\nRbqetr1+IvKDfWxJInJmoHUE02PaBUwEXi8m/W3g1iDqLYmzVTXW3ly7kAJPAOcDXYFqwDAg2yVb\n0cB2oCcQDzwCvC8iiV55XNFTRGoA/wR+dLpuLxsxwMfAHKAG8Cbwsb3fLZ4BmqlqNeBKYKKIdPJK\nd1rP6sAnwBlAAvAt1jG7QQZws22nOjAJWCAi0S7ZK8w04Lsi9kespqq62uu6EgtcgaXzYjfseTHa\ny27hIBGxeopICyz/b7PtLgA+CbiNqmpQG1ZwmlXE/oZAFlCxiLSngONYF/oMYKqfthQ43c+8s4BX\ngCXAYWAl0NTPsjVsv5oHq0tpN2AT0N9NPe2yrwB3ACuAm13S82JgJyBe+/4CLikjLc8AdgPXu62n\nVx017fZay2k9C9XjAfrZtuqWgZaDgPeB8cCcQmnlQlO7rjeAN1zWsqRzLmL1BEYDnxVqp1lAn4B8\nLIW4RQYmOy0daOfvlwJ8Cjzow5Zi9dRSgPlAoo+8s2wxewAVgf8HfOWPLbtMKjDOtvULcKebjbSQ\n/QS7wbVyWc/OQLLdaEo6SUqj5z+ARYX2LQDudVnHl4Ejdrv5Hoh1U89Cea8Gdruhp1eeTUCufXwz\ny6BdVrPPhUYUEZjKg6Z2vqp2Pb1c1nMFsA/YD3xdlL1I1ZOTA1MU1jVtTCAauTUEcBirG+cXqnpF\nCVl6AmuBKlgB8VMRaa+qx4rJv1BVVwGIyMNAmog0VtXtJdhqhDWk1hI4DWgBfCkiv6jqEn+PJxhE\npAJWF/hNVf25ULJjeopIFNaFe7Sq5omIP1UGq2cskFZoXzoQ54/RYFHVO0TkLqzh2F5ATqEsTrdP\nAESkEdZw1z0lZA1Wz3x/2olIJeAawM1h0XyeBF5T1R0+2ktEa2pzLVawWOln/mAZB/wX6+ZiENZw\nbHtV/d0rT6TquRSYJCK9gCSsY43Bunb7jVvLxeOweh6OoKqrVDVXVVOBMUAi0NpHke1eZTOAg0AD\nP0xl2X8nqGqWWhOR7wKXBeW4n4iIB3gLq6GOLiKLk3reAWxS1bUBlAlWzwysu21v4rFOOldR1eOq\n+hXWzcbthZIdbZ8AIlIH+AJ4WVXfKSF7sHoWoKrZtp0HReTsQP31F3uxykXACyVkjXhNgRHAbLVv\n9d1CVb9R1cOqmqOqb2L1mgpfYyJST/umegQwFWsYvTZWEN4RiK+O95hEpCFWhNxSTBanvnRft/qN\nvfyJxRpT3eVHnZvsv94+utpIxboFfQ1rGO8yVT1aKN1pPfsAPUUk/0SoCXSw79iKCooQvJ4/AveK\niHid7O2wGm1ZEQ00z//gRvu0F5J8AXyiqk/5USRYPYuiAtAM2Bhk+ZLohXUj+JfdW4oFokTkTFXt\nCOVDUxFpjHWsbizcKgnF63oW6Xqq6n+A/9hlqwMjKXrRTLEEs1w82h5GiMJqoJUKrbjoCSxT1cLD\nJ/nswTqR/LXXxl5yHGUL9H9YE+o/+Sh2mYhcYK/+ehJYq6rbfeQHwO5KrwYeFmspd2usrvan/vob\nBP/G6v31U9WsItId1RO4wbbX3t6SsVYiPuyjTFB6Yo2FHwfutvW8G+ukWhaAv34jInVFZJCIxNrt\n5W/AYOBLr2xOt89qwOfA16r6oJ/FgtJTRM7LLycilUVkHNYNzTf++hsEM7ACe357eQVYCPzNK0/E\naurFMCCp0HCa44hIdRH5W/51U0T+jjWX470KMKL1FJFO9vlXB6v9fFLE9IRvApmQsm96x2NdXLy3\n8V7pC4ErfZTvijWRegiYYu9bBPyzmPy9se4cMoG9wEdACx/1z+J/K0oygFXAaV7pxdqy0xtiNZIM\n4A/g1kA1CkDLprZ++Sts8re/u6VnEeVX4P+qvGD07ACswxom/R7o4KKedbDmB1Kx5rI2A7cUyuN0\n+xxhf4eZhb7DJk7riXXB2og1FHrQPtYebulZjA/jOXlVXsRq6pXnZ2BkGehXB6v3cNhup2uBvuVJ\nT+ArrzY6HagaqE5iV+QIItIOmK6qXR2rNHAfZgE7VPWRUPngFEZPZzF6Oo/R1FmMnhaOzjGptVgg\nZIKWN4yezmL0dB6jqbMYPS3MQ1wNBoPBEFY4OpRnMBgMBkNpMT0mg8FgMIQVQc0x1a5dWxMTEx12\nJXxZt27dflWt41b9Rk9nMXo6y6mmJxhNnSZQPYMKTImJiSQnJwdTNCIRkW1u1m/0dBajp7OcanqC\n0dRpAtXTDOUZDAaDIawwgclgMBgMYUXEBqbs7Gx27drFrl27yM7OZtq0aaxZs4Zdu4J95Fj5Jzs7\nG7MK0z22b9/O7bffjsfj4fffXX2yjcFQrom4wLRhwwYefvhhOnfuTOPGjWncuDFdu3blgQce4IIL\nLqBx48YlV3KKMmLECEaOHMn69ev9yp+ZmUleXp7LXkU+Bw8epH///rRu3Zrp06cD8K9//SvEXhkM\nJ5Obm8uYMWPweDxcdtllHD9+PNQuFUlZvZK5VPzyyy+0bl38Wy7OO+88zjvvvDL0KDIZOHAgAwYM\n4OKLL6ZDhw4+8/7444+cc8455OTkmOBUBJmZmdxyyy28++67xMbGMm/ePGbNmsXnn3/O9ddfT05O\ncc/fNBRHXl4ehw4d4r333uPw4f+9GeW6666jQYMGVKpUKYTeRR6ZmZlcc801LF26tGCkRESoVasW\nderU4dtvvyU+Pp6jR4+GXXsNy8B05MgRXn31VXr16kW7du2Ijo6mRo0axMXFcfjwYQYOHEj79u05\n//zzOe2006hSJaB3UJ2yBHIXv2TJEnJycujZs6eLHkUmR44c4bLLLuO7777j888/57zzziMu7sR3\nHzZv3ryY0obC7Nq1i6VLl/LBBx/w6acnP8j/n//8J+3atWPGjBmce+65IfAw8tiyZQt9+/Zlxw7r\nNUgiwvnnn8+kSZNo164dIoKqkpaWxtlnu/Y6r+AJ5gm5nTp1UrfIycnR7t27q8fj0eTk5IL9hw4d\nUlXV1NRUzcvLc81+UQDJ6uITh93UM5/du3drjRo1VET0nXfeKTF/7969VURO+A6cItL13Lt3r37y\nySeamZl5Utq8efNURAraa1kQ6Xpecskl6vF4tEmTJjpu3Dg9dOiQZmZmamZmpqakpOiMGTM0Ojpa\nK1eurGPHjnXVl3wiWdP9+/drs2bNVERURLRSpUo6bdo0PXr06El5f/zxRxURPXbsmGv+qAauZ9j1\nmG699Va+/vprXnjhBdq1a1ewv3p16y3D8fHxoXItolm/fj2pqdYLMatVK/xS2RM5duwYOTk5VKlS\nhTZt2pSFexFFnTp16NevX7HpCQkJBe3V4JtffvmFr7/+mg8//JDLL7+cqKioE9KrVKnCLbfcQkJC\nAtdccw2vvvoq99xzj5lL9sHIkSP5888/Cz6vWLGCLl26FJk3f44pIyMjrK6tYbX4ITc3l9mzZ5OQ\nkMCoUaOoUKFCqF0qF+Tm5vLQQw8BVqO97DLfb4r/9ttvSUpKYsyYMWZcPwBSU1P517/+xahRo0Lt\nSsTQuXNnnn32Wa688sqTgpI3l19+OX379uXIkSMcOnSoDD2MPD755BMAKlasyDfffEPnzp2LzfvW\nW28B4XfDH1aBac2aNTRv3pyff/7ZXBAd5LHHHmPTJuut8U888USJ+adOLcs3n5cf3njjDWrXrl1w\nE2AombS0NL8uilFRUVSuXLkMPCo/3HrrrZx77rmISJHpmZmZfPHFF2XslX+E1VDel19+SY8ePUoc\najL4z7p165gxYwYALVq0oG7duiWWyZ8wNfjPkSNHeP7557ntttvMTVUALFiwgD59+pSY7+DBgyxd\nupQmTZrQqlWrMvAscvn+++/58MMPeeyxx3zme+2119i0aROnn356GXnmP2EVmGbOnEl6ejqdOnWi\nX79+NGrUKNQuRTz3339/wdzSsmXLiI4u+Sv/6quvAMyQVABMnjyZtLQ0rrzyylC7ElFcccUVfuV7\n7733OHLkCHfeeScxMTEuexXZtG/fnvbt2/vMo6qsWbMGsJbjhxthFZhSUlLweDyMHj2au+++m0ce\neYQLL7yQ3377jdatW9OsWTMA/vjjD9q0aWN6Vn6wYsUKwJpbql+/fkBlCy+BNhRNdnY2q1ev5qmn\nnjphwY7BGWbNmsXo0aOZOHEi99xzT6jdiViys7PJzs5m06ZNbNmyhffeew+ACy64IMSeFUEgS/jU\n5aWOkyZNUo/H49fWoEEDveOOO1zxozBE6NLRTZs2qYho27Zt9fDhw6qqmpeXpzk5OQXb0aNHC/7P\nX4YvInrzzTe7tiw/UvUsjrvvvltPO+00TU1NLTZPTk6Ovvrqq9qjRw/duXOno/bLm57e/PDDD+rx\neLRTp06anp5eZnbLm6ZZWVk6cuRIrVevXsEychHR0047TbOzs123H6ieYbX44b777uPMM8/k9NNP\nx+Px7VpKSgrTp09n5syZZeRd5DFv3jzAmv+YPHky48eP57HHHqNSpUpUqlSJihUr0qdPn4LPq1ev\nLvgFeI0aNYqdNDX8j+TkZKZMmcJ9991X7CR+dnY2nTt35vbbb6dv377Url27jL2MTGbNmsWll17K\noEGDWLZsmenBB8njjz9OtWrVeP3119mzZ88Jaeeeey4VK1YMkWc+CCSKaRlG+//+97+6ceNG7du3\nr8+e00033eS6L0To3VP9+vVPuDsqamvbtq02a9ZMO3furCKiF154oYqIbt261RWfVCNXz8IcPXpU\nu3Xrpk2bNtX9+/eflJ6ZmamTJ0/WWrVqadOmTXXLli2u+FFe9MwnIyND77nnHq1QoYIOGzZMjx8/\nXqb2VcuPpl9//bV6PJ5iz/8GDRro7t27XfcjUD3DqsfkTevWrWnXrh1DhgwBIDo6moceeoitW7cy\nevToEHsXGbz88ssANGvWjD59+tC/f3/69+/PjBkz2LhxIxs3buS7775jw4YNfP3112zevJkmTZpw\n/vnnk5CQEGLvw58pU6aQlJTE4sWLqVWrVsH+bdu2MWDAAGJjY7n//vvp3LkzW7dupWXLliH0NnIY\nMmQIL774Ip07d+buu+8mNze3YAFPYbKysswbBYrh6NGj9OjRA1U9aeVdz549Wbp0KXv27OHGG28M\nkYc+CCSKaRlHe1XV3NxcvfPOO0/qKVWoUEFvuukmzcnJcd0Hysndkz9kZmZqbm6uqzbKg57Tpk1T\nEdHFixfrtGnTtEGDBioiGh0drYMGDdLffvvNdR/yKQ96qv5vPumll17StWvXnrD98MMP+vTTT+vE\niRP1wQcf1A4dOujEiRO1SZMm6vF4HPelPGiam5t7Ug/pzjvvPOFRWlWqVFERcd2XQPUMW1G9ycjI\n0GHDhmnDhg3V4/Foq1atdNq0aWVmvzw00nAi0vU8fvy4tmrV6qST/uqrr9bNmze7arsoIl3PfI4f\nP64HDhzQI0eOlJg3KytLVa1rw4EDBxz3pTxoeuzYMT3//PMLnpf3wgsvnHTTOXLkyLAMTGG1XLw4\nqlatyuzZs1mxYgXLly/ngQceoGrVqqF2y3CK8tdff7FlyxYABgwYwNChQ0lMTKRVq1bmNzalwOPx\nULNmTb/y5v+IuWrVquZaUAxRUVEsWbKEZ555hpEjR9K0adOT8jRq1CgsH/0WEYEpn169etGrV69Q\nu2E4xUlMTDTvqDJEBFWqVGHChAnFpo8fP77EZ2eGgrBd/GAwGAwG9/H1kNdQYQKTwWAwGMIKE5gM\nBoPBEFaItWAiwEIi+4BtzrsTtjRV1TpuVW70dBajp7OcgnqC0dRpAtIzqMBkMBgMBoNbmKE8g8Fg\nMIQVJjAZDAaDIawwgclgMBgMYYUJTAaDwWAIK0xgMhgMBkNYYQKTwWAwGMIKE5gMBoPBEFaYwGQw\nGAyGsMIEJoPBYDCEFSYwGQwGgyGsMIHJYDAYDGGFCUwGg8FgCCtMYDIYDAZDWOFoYBKRdiKS5GSd\npzJGT2cxejqP0dRdROR5Ebk91H6UNUEFJhEZJCI/iUimiPwuIt0BVHUTkCoi/XyUXSEiNwdgq7uI\nZBTaVET6B+N7AHaH23b89jVIO4WP7biIvASu6dlSRD4WkX0iclBEPheRMxw4lOLs9RORH+xjSxKR\nM92yZdtLFJHPROSQiKSIyFQRiQZ39LTLlOkx2jbLpH3atlqLyDIRSROR30Tkmvw0l9pobRH5WkQO\n2DbXiEi3Uh6GL3tqX8vyz8FXHa5/tIgki0iOiMwqIr2PiPwsIkdEZLmINPVKngz8U0Riiqk70fY/\nOgB/KorICyKyyz5PXhaRCgEfmH+2REQmishO+7tcISJtSioXcGASkb7AJOBGIA7oAfzhleVt4NZA\n6y0OVV2tqrH5G3AFkAEsdspGYUSkBvBP4Ee3bORT6NjqAVnAPK8sjuoJVAc+Ac4AEoBvgY8drL8A\nEWmB5f9ttt0FwCeBnERB8DKwD6gPtAd6And4pTuqZyiOsSzbp30cHwOfAjWBUcAcEWnplc3pNpoB\n3IzVPqtjXW8WuNxuzvY6F50O9ruAicDrhRNEpDYwH3gUS99k4L38dFXdDfwMXOmgPw8C5wBtgZZA\nR+ARB+v35jrgJqA71vGtAd4qsZSqBrQBScBIH+kNsS6uFYtIewo4DmRjNb6pQdh/A3jDR/os4BVg\nCXAYWIn19sRAbLyCdTFbAdwcqI/BbsAIrCAvZahnTUCBWk7rCYwGPvP67LGPpY+LGv4EXOb1+Tlg\nult6BnqMkdY+sS5eGYXa5BfAk2XRRm09+9lttK4bmtp1n+6mjradicCsQvtGAUlen6vaWrby2vcw\nxVzzgL9s/zPsrasffiQD13t9HgJsL0Gfu7GuTfvtc8rj5zGPA973+twGyC6pXEA9JhGJwoq0dewu\n/Q57qKRyfh5V3QkcxbojPwFVfRhYDYxW685ktF3vpyLyoB/2qwIDgDdLyPp34EmgNrAB644uvw6f\ntkSks32Mr5TkjwuMAGar/Q2Cu3ra9ABSVPWAjzxB61kIsbe2fuYPhheBgSJSRUQaApfi1bsuAz39\nOcZIbZ8FbuB1fG5pKiKbsALaJ8CrqrrXR/bSttFV9tDvfBFJLCGvk7QBNuZ/UNVM4Dd7fz4/AWcX\nU76H/be6re8aEWkiIqki0sRPHwRoJCLxPvJcg9XuOgJXYfWC8MPWu0BzewqhAtY1ruTRrgAjfgOs\n6JmMNVRSG/gaeKpQvp1Aj2LqWEGQd3nAMOBPvO7eisgzC3jX63Ms1h1bYz/qj7KP7bzS+hrEsTW1\n/TytiDS39Gxk1z3YJT1bAZlALyAGa7giD3jIRR1bA+uAY3ZbnVW4vTipZ6DHGGntE6iAdaf8gP3/\nxUAu8HkZtdFKwGBghBtt1M7fw/7uqgNTgR+AaBe0LKrH9BrwTKF9XwM3eH3uC/xRTJ2Jdjv321/b\nj6+BOljTB9/YddQvJr8Cl3h9vgP40k9bMcD/s+s4hnX9PukaV3gLdI4py/77kqruVtX9wP8BlxXK\nFwekBli3P5zUoyiG7fn/qGoGcBArqJbEHcAmVV0bvItBMwz4SlX/LCLNcT1FpA7WkMzLqvpOCdmD\n0lNVf8b6zqYCu7FuZP4L7AjSbZ+IiAfrbmw+1pBIbaAG1hyFN47pGeQxRkz7VNWjwNXA5UAKcC/w\nPicfnyvnvKpm2+3zQREprtcAwWuKqq5S1VxVTQXGYF3sWwfvdUBkANUK7YvHGpLMx2ltnwLWY/Us\nk4CPsHq8e3yU2e71/zb81BZ4DOgMNMa6yXgCWCYiVXwVCigwqeohrAbpHRhOCBL28EkMsKW4agKx\n6VVvY6y70tl+ZG/sVS4Wax5llx/l+gDX2F36FOB84HkRmRq4xwEznCKGKN3Q0548/wL4RFWf8qNI\nsHqiqv9R1baqWgt4HOuk/y5Qn/2kJtAEax4jR63hyTfwunFyQ88gjjGi2qeqblLVnqpaS1X/BjTD\nWjQDuHfOF6KCbbc4gm6jxSClKBsIP+I1TGdPVzTnxIUtrfEa7itEMO01S1VHq2pDVW0GHADWqWqe\nj2KNvf5vgv/atsfqze5Q1WOqOgvrZtHnytVglou/AdwlInXtC9w/sFbs5NMTWKaqOcWU34PvBlYc\nw7AmCX/3I+9lInKBvcTySWCtqm4vqRBwA1YjaG9vyVgR/uEg/PUbETkfawJ5XhHJjuopItWAz4Gv\nVdXfeZNg9UREOolIlN1Dm4EVDH/2199AsHvwfwK3iUi0iFTH6s1s8srmePsM4hgjrX22E5FK9rzd\nfVjD+LO8sjjdRs/L10dEKovIOKwVet/4KBaUpiLSRkTa299fLNYI0E6seR1HsNtiJayh2Chby/wV\nhh8CbUWkv53ncWBjofbTE1hUTPX7sIaOA9G3oYg0sJdyn4c1/Px4CcXuF5EadgdhDF4rB0vgO+A6\nEUkQEY+IDMO6yfjNZ6kgxkkrYC3JTcXq2k8BKnmlLwSu9FG+K/ALcAiYYu9bBPyzBLs/42M1oFe+\nWfxvhU4GsAqvMU1/bHnlXUEZzDEB04G3iklzVE+sC7VizYtkeG1N3NAT+AprWOKgfZxVXdayvf29\nHcJaQfQ+kOBm+wzkGCO0fT5n65Fh+3d6oXSn22hPrB5CvqYrKWb+qrSaAr2xenqZwF6sYa0WDus3\n3j7nvLfxXukXYV3fsuzvNNErrT7WKFWMj/onYAWoVOA8rB6Nr3O6B7AVOGIf+99L8N97Vd4B4Hkg\nyk4ryVYlYBrWMHc68D1e81XFbWIXdgQRaYe1NLerY5UG7sMsYIequrUuv8wwejqL0dN5jKbuIiLP\nA7+r6ssh9EGxgrXvXo6DOPqDNbV+BR6yBlreMHo6i9HTeYym7qKq94bah1BgHuJqMBgMhrDC0aE8\ng8FgMBhKi+kxGQwGgyGsCGqOqXbt2pqYmOiwK+HLunXr9qtqHbfqN3o6i9HTWU41PcFo6jSB6hlU\nYEpMTCQ5OTmYohGJiGxzs36jp7MYPZ3lVNMTjKZOE6ieZijPYDAYDGGFCUwGg8FgCCtMYDoFycvL\nIy/P12OxDAZDeeDw4cO89NJLnHnmmYgIIsL999/PqlWrOH78eKjdK5awCEyPPvooX375ZajdOCVI\nTU3lueeeo0WLFkRFRXHVVVexf//+ULsVUVxzzTV4PB48Hg8DBgxg4sSJfPLJJwwaNIi4uDi++86t\nZ9SWT+69914aNWrElClTSE9PZ+3atZx55pncd999oXYtYvnss8/weDzEx8fz2GOPMXDgQN5//33e\nf/99qlatSp8+fahQoQJ33nlneN6kBvPsp06dOqmTAFq/fn3duXNnUOUnT56sWVlZjvrkDZCsLj6L\nzGk9/WXz5s1aq1YtjYuL0/T09DKzG+l6ioi2atVKH3jgAT127FjB/ry8PL3jjjs0JiZGv/nmG1d9\n8KY86Ll3794T9r3xxhtav359V+36IpI1Xb16tXbo0EFFRM8++2w9dOjQSXmSkpK0SZMmKiKamprq\nmi/5BKpnWPSYatSowZ49e3j//fcDKpeVlcVHH33EAw88wMMPu/qA5XJJ27ZtWbhwIUeOHOHPP4t6\nDZShOFavXs2kSZOIiooq2CcivPjiiwwfPpyRI0eSmZkZQg8jh8aNGxMbG3vCvmuvvZaUlBS2bCnu\nTRqG4vjiiy/YsGEDDRo0YPny5VSvXv2kPF27dqVu3boh8M4/wiIwLVu2DLC69Nu3b+fw4cPF5k1L\nS2Pt2rU88MADdOrUif79+wOweHHJb+s1nEyXLl1o167dKbV01Ql++OGHIvdXqFCBiRMnkpaWxoAB\nA8rYq8jku+++o0KFCkWm7dy5s4y9KT+MGTOGGjVqhNqNoHD0Ia7B0q5dO4YMGcI777xD69atiYmJ\noWLFijRr1owjR44wcuRIAA4cOMCLL75Ienr6SXWMGDGirN2OKA4fPsynn37KF198Qc2aNRk2bBit\nWrUiNTWV1NRU+vXrF2oXI4YqVaowfPhw1qxZQ8OGDU9KT0hI4IMPPuDqq6/m6NGjxV50DRZF3bkf\nO3YsBJ6ULxo1auQzfcSIEaxbt45FixYxaNCgMvLKP8Kix+TxeHjrrbd44YUXyMrKIi0tjb1797J2\n7Vo2bdrEmDFjGDNmDBMmTCA9PZ2RI0eSkJBQUP6tt95izJgxITyC8CUnJ4clS5bQpEkTbr/9dpKS\nkvjwww/p0KEDLVu2ZNiwYfTt25c6dVz7kXu549///jcHDhygS5curFu37oS0lJQUhg8fToMGDVBV\nFixYECIvI5v58+cDJV9cDcXz9NNP+0zfuNF6Ke6GDRvKwp2ACIvAlM9dd93FE088QY0aNahfvz4d\nO3YEICoqiqioKB588EFWr15NZmZmQa8pOjqaIUOGULFixVC6Hrb88MMPfPjhh0ycOJFt27axZcsW\nfv31V9atW8f27dtZunQpqamppKWlhdrViGHYsGF89dVXpKWl0aVLF8aMGcOhQ4cAyMjIYM6cOcyZ\nM4eHHnopUBaFAAAgAElEQVSIQYMGkZWVFWKPI489e/ZwySWX0LJly1C7ErFs3bqVX3/9tdj0cAxI\nBQSyUkLLYEWJN7///ruuXr1aN2/erJs3b9a8vDx9++231ePxFGyzZs1y3Q8ieIVO9erVdf369Sft\nX7VqlQLaqlUrjYmJ0QYNGuiff/7pmh/eRLKe3uzevVtvv/12FRGNi4vTK6+8Ul944QUVER08eLBu\n375dRUQzMzNd9aO86JnPvn37NCEhQefOnVumdr2JZE23bdumjRs3VhHRLl266MKFC3XLli26ZcsW\nXbp0qV5//fV6/fXXa1xcnIqIDh8+3DVf8glUz7AT1Rd5eXl61VVXFQSlevXqaXZ2tut2I7mRdu/e\n/SSNNmzYoE2aNNHu3btrRkaGJicna8uWLbVy5conLdt1g0jWszB5eXmakpKiF154oTZr1kxFREVE\nPR6P3nLLLSoi+t1337nqQ3nSU1V14MCBKiLao0cPHTBggI4cOVL/+OOPMvUh0jV99dVXC9qiP5vb\nBKpnWA3llcTevXsLxuxr1qzJ6tWrzRBeCYwdO/YEjdLS0rjqqquoUqUKn332GVWrVqVTp06sWbOG\nxo0b07dv3xB6G3mICAkJCSxbtozNmzfz119/MWTIEFSVtm3bAsWv4DMUzYIFCzj99NML5pF37txJ\n8+bNadasGbm5uSH2LjIYMWIE33//PUOHDqVdu3YMHTqUhIQEEhISGDp0KN9++y3Tp08PtZvFEhar\n8vzlqaeeAqBatWpMnz6d008/PcQehT8PPPAAiYmJdOzYkQ0bNnDDDTdw2mmn8cEHH5zw25GaNWuy\nZs0aunbtSm5uLjExMSH0OjKpUqUKVapU4bLLLmPu3LkMHjyY9957j6VLl3LDDTeE2r2IISUlhQoV\nKlCpUiUAjh8/TkZGBv/3f//HhRdeyNy5c2natGmIvQxvoqOjad++PbNnzyYjI4PY2FgOHjwIWOc6\nQHZ2dihd9EnE9Jiys7N55ZVXAHj++ee59tprQ+xRZDB27FjOOeccRo0aRdeuXcnKyuKTTz4paJze\n5Aenm2++maNHj4bA2/JFxYoVad68ecEiHoN/xMXFFQQlsBY/xcfH88QTT3DjjTfSvn37EHoXeeTf\ngNasWbPI8z4ciZjA9PLLLxc8dND71/YG34waNYqrrrqKTZs28emnn7Jp0ybi4uKKzV+zZk0+/PBD\nevToYX5L4hCRcjGIBG6++WbatGkTajfCmszMTM466yyeffZZUlNTQ+1OUEREYMrOzmbWrFkFn7t3\n7x46ZyKMmJgYPvzwQ9auXUufPn38mpM7fPgwkyZNolKlSvznP/8pAy/LJz/88ANz5szhoosuCrUr\n5YqPP/7Y9Oh9ULVqVb799lvmzp1LzZo16dixIx9//PFJDxdOSkoCCMsefUTMMVWoUIFq1aoBUKlS\npSJ/bW9wlh49evDOO+8waNAgkpKS6Ny5c6hdijjuv/9+wPSYnKZKlSqhdiHsqVy5MitXrqRnz55s\n2LCBa665hujoaOLj4wvy5PemJk6cGCo3iyUiekw7duxgzZo1AIwcOdKsxCsj+vfvzyWXXMLgwYND\n7UpEkpSURGJiohl6dpivv/461C5EBPHx8axatYp77rkHsB7zdODAgYLt+PHjnHvuufTq1Su0jhZB\nRAQmb2688cZQu3DK4PF4WLBggc9fjxuKp0aNGiQnJ5sbKQc5fvw4Dz30ENHRETHYE3KqVavGc889\nx9GjR/niiy+YMmUKTZs25corr+Thhx9m+fLlJyw0CRci4tutW7cuF198MV999VVYP6q9PJL/1kuD\n/7Rr144zzjiDAQMGmGE8B0lPT2fChAn8+OOPpk0GgIgQFRXFRRddxEUXXcTo0aND7VKJRERgqly5\nMvPnzycjI8M8bNQQ9rRt25affvop1G6UO6pVq8bkyZOZPHlyqF0xuExEBCawglPlypVD7YbBYDAY\nXEasxxgFWEhkH7DNeXfClqaq6lpXzejpLEZPZzkF9QSjqdMEpGdQgclgMBgMBreIuFV5BoPBYCjf\nmMBkMBgMhrDCBCaDwWAwhBUmMBkMBoMhrDCByWAwGAxhhQlMBoPBYAgrTGAyGAwGQ1hhApPBYDAY\nwgoTmAwGg8EQVpjAZDAYDIawwgQmg8FgMIQVJjAZDAaDIawwgclgMBgMYYWrgUlEnheR2920cSph\n9HQeo6mzGD2d5ZTVU1V9bsBoIBnIAWYVSosB/gNsBRToVSi9PrAdiCmm7kS7XHRJfhRT/svSlPej\n/hHAOiAd2AE8W1pbJeh5HrAEOAjsA+YB9d3WE2gGfAocBvYDz7qhp23rH0CKrenrQEUH6vSl6Zl2\n2iF7Wwqc6aamgAATgZ1AGrACaOOWpl52HTkffOlZKN9jtr2L3G6jTh+jj/rbAp/b54E6VKev9pmv\nR4bX9qjL7dPx65oPW4OALbatvcCbQLWSyvnTY9pln2SvF5P+FTAU62JzAqq6G/gZuNIPOwEhIn8H\nKjhdbyGqAGOB2kAXoA9wXynr9KVnDWAGVmNrihUo3shPdENPEYnBCobLgHpAI2COU/UXsvU34EEs\nHZtiBcQnHKjal6a7gIFY32Ft4BPg3fxEl9rodcBNQHegJrAGeMvB+k/C4fOhpHMeEWmOdZy7vfeX\ng3P+KPA+MNLBOkvUE6iuqrH29mT+Tpf0dOO6VhxJQE9VrYZ1vkdjaeGbACLfRHzfPe2gUI/J3v8w\n8EYxZf7ixLuFrn76Eg/8gtXD8Hm3YKffDfyBdRf0HOAJMvrfAyxw6E7Cp552no7AYTf1BEYBqwPw\nO2g9gbnAv7w+9wZSnNDTzzYaDdwJHHFZ03HA+16f2wDZbrXRQM4Hp/QEFgOXYY2WXFQoLeLPeeB0\nHOox+dITP3o8buhZqA6f1zUn9LTriQVmA5+VlLcsFj/8BJxdTFoP+2/+3cIaEWkiIqki0sRHnf8C\n/k0RvbRiuAY4B+tCfxXW3Sx+2irs749+5nWCouw5red5wFYRWSQi+0VkhYicVYJfwerZBtjo9Xkj\nkCAitUqwV2pEJBXIBl7Caj/eOK3pu0BzEWkpIhWwhk4Wl+BiadpooOdDqRCR64AcVf2smCzl6Zwv\nK7aJyA4ReUNEahdKc0PPwnWUdF0LWk8RuUBE0rBGgPoDL5bkUFkEpsNAdX8zq+pfqlpdVf8qKl1E\nzgG6YV1g/GWSqh6063wRGOyPrUJ2b8L6YiYHYDdoRKQd1hj+/YWSHNUTa+huEDAFaAAsBD62h/iK\nI1g9Y7HmXPJJt//G+Xk4QaOq1bHuukcD6wslO63pbqwh7i1AFtaQ1z9KqDYoTYM8H4JGROKwgsQY\nH9nKxTlfRuwHzsUa2u6EdS68XSiP0+2zgACua0HrqapfqWo81rXmOaxetk/KIjDFAalOVCQiHuBl\nYIyqHgug6Hav/7dhXYADsXs18DRwqaruD6RsMIjI6cAirONcXSjZMT1tsoCvVHWRquZiNdBaQGsf\nZYLVMwOo5vU53v572M/ypUJVM4FXgNkiUtcryWlNHwM6A42BSljzaMtEpIqPMgFrWorzoTSMB95S\n1a0+8kT8OV9WqGqGqiar6jFV3YN143SxfQOQj9PtEwj4ulZqPVV1J9bIwbsl5S2LwNSaE4dvvNEA\n66qGFd3fE5EU4Dt7/w4R6e6jXGOv/5tgTUb6hYhcAswE+qnq5gD9DRgRaYq1cuxJVS1qwtxJPQE2\nBVEuWD1/5MQhibOBPap6IED7pcGDNfnb0Guf05q2B95V1R32BWcW1sKWM32UCUbTYM+H0tAHuFtE\nUmybjYH3RWScV56IPudDTL4+3tdmp9tnMNc1p/SMBpqXmMuPCatorLu+p7FWFlXCa6IOqGjv2wFc\nbP8vXulfANcXU3cV4DjQ0s/JM8FaOZa/nYv1xTSk+OWUirXEtAaWuD8Do/y01xs4APTwJ7+fdRar\np30cvwP3+SjvmJ52mTOAI8BFQBTWkNPvLul5CdYcwZl2+RXAMy5r2hfoYB9bNawhy11AJRc1fRxr\nKC8B6wIzDMjEmgdwTNNgzgcH9KxVyOZ2rKHKWDf0DOYYS9lGxT7eM+16KlHKnzSUoGcX+xz02Nq+\nBywvVN7p9hnQda2Uev4daGL/3xRYCcwvsZwfFY+3HfPexnulby0iPdFOq48VsIo9SYAJWL/ZScWa\niG+CNeTTxA/fEglshc4B4Hkgyk7zaQtYDhzjxN8YLCplIy1WT6wLmhayl+FV1hU9gWuB37DmfFbg\n4zc3pdHTznMPsMe29QbO/I7Jl6bX2SdShq3LQqCdm5piXXimYc01pQPfA5e4pWkg50Np9Swi71ZO\n/h1TJJ/z+fV7b1tdbJ+DgT+xblx2Y61aq+dy+wzoulZKPZ+y/c+0/84AapWkmdiFXUFEngd+V9WX\nXTNSsg8KtFDV30Llg1MYPZ3HaOosRk9nOVX1dDUwhQPlqZGGA0ZP5zGaOovR01lCoad5iKvBYDAY\nwopy32MyGAwGQ2RhekwGg8FgCCuigylUu3ZtTUxMdNiV8GXdunX7VbWOW/UbPZ3F6Oksp5qeYDR1\nmkD1DCowJSYmkpycHEzRUnP06FFuueUWTj/9dB555JEysSki29ysP5R6hgKjp7MYPZ3HaOosgeoZ\ncUN5aWlpzJ49m2eeeYbs7OxQuxNx7N69m3r16iEi/PVXuDwuzGAwGP5HxAWmfBo0aIDHE7Huh4Rx\n48bRtGlT9u3bx1lnnUX16n4/F9JgMEQIM2fO5LHHHmP48OGICB6Pp+Cvx+OhTp06zJ07l7y8vFC7\nWiwRe2UfNGgQMTG+HoBtKMx7773H8ePHOeuss1i1ahXVqlUruZChRLp06YKIMGDAgFC7Ui65+uqr\nueaaa0LtRsRw22238dRTT/H222/j8Xi46qqruOeee2jbti0iwsGDBxk2bBiDBg3i8OEyeX5ywAQ1\nxxQq0tLSaNWqFbVq1WLChAmhdieiqFevHvv27WPHjh3Ur18/1O6UC5588kmeeOIJ8vLyGDhwIHPn\nzi1IW7RoEYMHDy74nJrq+MOhTwnWr1/Pxx9/TIsWLULtSkQwduxY0tPTqVq1aol5e/bsSfXq1enY\nsSPfffddifnLkogKTN9++y0HDx7kjTfeCLUrEcWSJUs4cOAAt99+OwkJCQX7Dx8+zPfffw9Ax44d\niYtz/bVI5Ya9e/cydepU8vLyaNGiBYMGDSoYWv7000/p378/R48eBaB///6hdDViyc3NpX///ogI\nf//730PtTkTwwAMPUKlSJb/yLly4kPPPP5/vv/+evLy8sJoaiZjAlJmZyf3330+DBg3MkEmAHDx4\nkLy8PLp3747H42Hjxo2MHz+ejRs3sm2btVimadOmxMfH06hRI9566y0z/1QCF110Efv27eOMM85g\n5cqV1K1rvd5pwYIFDBw4kKNHj7Jw4UK6d+9OxYoVQ+xtZDJ37ly2bt0KcELv01A8DRr4/5qk2NhY\nXn75Zf72t7/xyy+/0KpVKxc9C4yICUzffPMNmzZt4tZbb6Vq1aocPXqUvLw8c9L7Qf6w5+WXX876\n9evp1q0bOTk5J+TJD1CbNm2ib9++LFmyxASnYti/fz+7dlmvo7nvvvsKghLAm2++SXZ2NjVr1uTM\nM88kNjY2VG5GLKrK4sWLuemmmwr2mflQd7jgggsYMGAAF110ETt27Ai1OwWET9/NB7m5ubzyyisA\nPPTQQ+Tl5TFq1Cj69etHVlZWiL0Lb5YtW8Yvv/wCwLRp0xgyZAg5OTlcccUVfPvtt2zduvWE7Zln\nnuH7779n+vTpIfY8fHnppZc4ePAgN9xwAzfccEPB/sWLF7N48WIA3n33XZo2bRoiDyOb/fv3c/nl\nlwPQrFmzEHtT/unWrRt79+7l119/DbUr/yOY94t06tRJy5IZM2aoiOiAAQNUVXXz5s0qIioiunPn\nTtftA8laivexlLS5qeebb76pHo/nhK1hw4Z68ODBYst4PB6NiYnRpKQkV3yKZD1VVePj41VEdMyY\nMaqq+tNPP+mcOXM0NjZWRUQTExN13759rvrgTaTr6c2OHTs0KipKRUT37t2rdevWVRHR3bt3l5kP\nquVLU1XVY8eOnbDl5eUVpO3bt089Ho8uX77cNfuB6hn2PaaUlBSefvppAO666y6OHDnCkCFDAGjc\nuDHx8fGhdC/iqFevHsuWLaNGjRrF5qlfvz7Hjh1j1apVZehZ5JCeng7Azp07GTJkCOeccw7Dhg0j\nMzMTsIZFe/bsyU8//RRKNyOOPXv20L17d6KiotixYwe1a9cmLy+Pjh07njBcavCP7OxsPvvsM8aN\nG0dCQgIxMTEF2w033MC4ceP45ptvqFy5cqhdPZlAopiWYbQ/duyYrlu3Tps2bVrQO8rNzdVvvvmm\n4PNpp52m6enprvtCBN89efeY6tevr9OmTSuxzJYtW9Tj8WiNGjVc8SmS9VTVgvZX0nb22WfrJ598\n4qovqpGvp6pqenq6NmvWTOPi4gp66t9++61GRUXpnDlziizzww8/6Pr1613xJ9I1/eWXX/Sss846\nabSkqK1bt25atWpVPXLkiGv+BKpnWC5+yM7OZujQocyfP/+E/YUXOmzdupX4+Hg2b95MmzZtytLF\niGTZsmV+rbwxv3Pyn549exIbG8vq1atJT0+nc+fOREdHk5SUxKZNmxg9ejS9e/f263clpzLvvvsu\nf/75J9OmTaNr166AtfQ5JiaGK6+88qT8r732GosXL+a2224ra1fDnpycHLp37052djZ33XUX3bt3\nB+D8889HVYmPj2fx4sWkpqZy7733smbNGqpWrRpWPaewC0zZ2dmMHz++ICjFxsYyadIkatSowcyZ\nM1m+fPlJZZKTk01g8oNGjRr5lS9cfw0eLmzcuJFFixZx6aWXcsYZZxATE0Pv3r1ZsWIFs2bNIjEx\nkaSkJAYOHMj27duZO3cut9xyS6jdDmvyFzE988wzLFiwAICVK1cSHx/P/PnziYuLY8GCBezZsweA\n7du3M2nSpIKLruF/PPjgg+zbt48hQ4bw4osvFpkn/7d1HTp0oEePHmXpnl+EXWAaP348zz77bMHn\n6dOnM3jwYI4dO8Zdd90FgIjQtWtX+vXrx9VXX80ZZ5wRKnfLJRMnTgQwF9NiOOusszjrrLNO2Ocd\n9CtVqkTv3r259tprmTlzJk899ZTRsgRuvfVWOnfuzLx585g/f37BzxcyMzMLtDv77LNJS0ujX79+\nfPbZZ6F0N6zJ/ymDP9SsWZPo6Ojwa5+BjPtpGYyPeo/Rjx07Vo8fP66qql9++aUCKiI6dOhQ1+wX\nBRE83rxr1y6tUqWKejwenTNnjmZlZZVYxuPxaPXq1TUlJcUVnyJZz+LYuHGjioj+9NNPBfvWrl2r\nIqJVqlRx1XZ50zM3N1dFREeMGKHHjh0rU9v5RLKm119/vXo8nhKvk7m5udqrVy/1eDzat29f1/xR\nDVzPsF2Vd8YZZzB+/PiCx2Ts3r0bEWHs2LHMnj07xN5FDvXr1+fmm28GYPjw4Zx33nl89NFHRT5Z\nOC8vj3379gEwatSoEx5fZPCP/BV7AC1atCA2Ntb81i5A3n77bcD68XJUVFSIvYlccnNzi32CeHZ2\nNmeffTarVq2iTp06fPrpp2XsXQkEEsXU5Wh/+PBhFRH973//W2Ta9u3bXbFbEkTw3ZM3u3bt0jFj\nxmhMTEyxK3SaNGniuh/lRc/CzJ07V2NiYrR+/fr666+/am5uro4aNUpFxFW75UnPnJwcbdmypQ4d\nOvSE39qUNZGuaX6vKTo6Wh999FHdvHmzrl+/XseNG6e1atVSj8ejUVFR+uWXXxaMSrlJoHqGpajh\nRqQ30sKkpKTomDFjTghIp512ms6cOVP37Nnjuv3ypqc3Q4YMKRiKvuSSSwr+d5PypGdGRoaKiC5Z\nsqTMbBZFpGu6Y8cOHTp0qM9l4itXrnTVB28C1TNsh/IM7pGQkMCLL77I8ePHC7Y//viDm2++2fyQ\nsZS88sorPPHEEwB8/vnnIfYmMomKiqJ169ahdiOiadiwIa+++ipLliw54bFZAJMnT2b79u1ccMEF\noXHOD8JuVZ7BEMnExcUxduxY8vLyCgLUyJEjQ+xV5JCWlsaVV15Jw4YNQ+1KxFOxYkV69+5N7969\nee2110LtTkCYHpPB4DBxcXE8/vjj5OXlkZeXx8yZM0PtUsTQoEEDPvjgg1C7YQgxJjAZDAaDIaww\ngclgMBgMYYVYCyYCLCSyD9jmvDthS1NVreNW5UZPZzF6OsspqCcYTZ0mID2DCkwGg8FgMLiFGcoz\nGAwGQ1hhApPBYDAYwgoTmAwGg8EQVpjAZDAYDIawwgQmg8FgMIQVJjAZDAaDIawwgclgMBgMYYUJ\nTAaDwWAIK0xgMhgMBkNYYQKTwWAwGMIKE5gMBoPBEFaYwGQwGAyGsMIEJoPBYDCEFSYwGQwGgyGs\ncDUwicjzInK7mzZOJYyezmM0dRajp7Ocsnqqqs8NGA0kAznArCLSqwAvA/uBNGCVV1p9YDsQU0zd\niYAC0SX54VWmIvACsAs4ZNuu4G/5QDbgFSDDa8sBDpeyzmL1BP5eyN4RW59OLuo5AlgHpAM7gGcD\nKR9qPf1so9cDPwGHgf8CV7vcRm8Ajhc61l4uaToI2GJ/f3uBN4FqLut5M/CbfVyLgQblRU/b3j+A\nFFvT14GKpairIvAa1ksBDwMbgEsL5ekD/Gyf78uxXqrnpp5hf87702PaBUy0v6CimAHUBFrbf/+R\nn6Cqu23Br/TDjr88CJwDtAVaAh2BRxysvwBVvU1VY/M34B1gXimrLVZPVX27kL07gD+A7+10N/Ss\nAowFagNdsE6S+xysvwCX9AQfmopIQ2AOcA9QDbgfmCsidW2f3NAUYI33sarqCofrzycJ6Kmq1YBm\nQDSWFqXBl569gH8BV2Gd739ifY9A5OspIn/Dusb0AZpiafpEKaqMxgosPYF4rGvV+yKSaNurDcwH\nHsXSMxl4L7/wKXvOBxD5JnLyHX4rrKhb7B0a8DDwRjFpf2FF+/xo2tUPP5KB670+DwG2+8ivwN1Y\nF/j9wHOAJ4jIXxXrjqenQ3cSJ+lZRJ7lwONu6llEHfcACyJNTx9ttAuwt9C+fd7auNBGbwC+CsBv\npzSNBWYDn7mo52TgZa/PDWz/m5cHPYG5wL+8PvcGUpxqo3adm4D+9v+jgCSvtKpAFtDKLT2LqCPs\nzvnSzjF1xuqiPiEi+0Vks4j0L5TnJ+DsYsr3sP9WVyuirhGRJiKSKiJN/PRBgEYiEu8jzzVYvayO\nWHd6NwEEaKs/1gVtlZ9+lQoRaYqlz+xCSW7r2QP4sYQ8kaRnMvCTiPQTkSgRuRprOGGTVx43NO1g\nnxO/iMijIhJdgp9BayoiF4hIGtZJ3x94sQRbTiL237Ze+yJZzzbARq/PG4EEEalVgj2/EJEErJGe\n/HPsBHuqmok1TNrGq9gpd86XNjA1wmqQaVh3TqOBN0WktVeew0B1fytU1b9Utbqq/lVMlsXAGBGp\nIyL1sCI5WN3T4pikqgftOl8EBvtpy5sRwGy1Q38ZMBxYrap/FtrvtJ4FiMhNWI1vcglZI0ZPVT2O\nFdzfwQpIc4Fb7QtAPk5rugrrvKiLdTIOxhpC9EXQmqrqV6oaj3U+Pgds9fdYgmAxcJ2ItBORysBj\nWHfU3udfJOsZi3U9yyfd/hvnx6H4REQqAG8Db6rqz8XYy7fpbe+UO+dLG5iygKPARFXNVdWVWMNP\nF3vliQNSS2nHm6eA9ViTiEnAR7YPe3yU2e71/zasIOo39t1AL07uvbjJcKyJ7MI4rScAdk/iaayJ\n2f0lZI8YPUXkIqzJ3V5ADNZY/6si0t4rm6Oaquofqvqnquap6mZgAjCghGKl0tS2uxMrcLwbaNkA\nbCwFxgMfYAXArVgXzh1e2SJZzwysuch88kdiDvtZvkhExAO8BeRi3cAXZy/fpre9U+6cL21g2lTE\nvsLRsDUndo195S0RVc1S1dGq2lBVmwEHgHWqmuejWGOv/5tgTe4GwjDga1X9I8ByQSEi3bC++P8U\nkeyonra9S4CZQD/7xC+JSNKzPdZK0WT7wvYd8A1wkVcexzUtog4pIU9pNc0nGmgeZFm/UNVpqtpC\nVROwAlQ08INXlkjW80dOHDY7G9ijqgf8d+9ERESwVuYlYM0tHS3OnohUxfr+vIfWTr1z3o8Jq2ig\nElZkfcv+P9pOq4A1Hvqona8bVqT3nrj7Aq/FCoXqroK1DLRlSX54lWmIddEW4DysSH6xj/wKfAnU\nsMX9GRjlrz27ji3ATYGUCUZPrzwzsLq8RZV3Ws/eWMG9h5/5w0pPP9poT6xx7fb25w728V7sVd5p\nTS8FEuz/W2FdtB93Q1Osnxg0sf9vCqwE5ruoZyWsYTXBukCtwGuxQDnQ8xKspeJn2uVXAM+UUs9X\ngLVAbBFpdbCG8vrb2j4LrHVZz7A/5/2pcLztmPc23iu9DbAGyMT6jcg1Xmn1sbr4Ra7Bt/NMwLpw\npGIFmiZY3dsmxeTvgTV8cMQ+2L/7IWr+ipIDwPNAlJ3m05adp6t9bHGlaZwB6FnJ1qJPEWXd0HM5\ncIwTf2uwKFL09FPT0Vg3UIdtv+91WdPJWEPLmba9Cfj4rV1pNMUa2t5h29qBdVNTyy09seY6Ntn2\nUrCCV1R50dPOc49tLx14g9L9jqmp7U92oXPs7155LsK62GdhBcJEl/UM+3Ne7IKuICLPA7+r6suu\nGSnZBwVaqOpvofLBKYyezmM0dRajp7Ocqnq6GpjCgfLUSMMBo6fzGE2dxejpLKHQ0zzE1WAwGAxh\nRbnvMRkMBoMhsjA9JoPBYDCEFSU91qNIateurYmJiQ67Er6sW7duv6rWcat+o6ezGD2d5VTTE4ym\nThOonkEFpsTERJKTk4MpGpGIyDY36zd6OovR01lONT3BaOo0gepphvIMBofZv38//fv3Z9GiRaF2\nxe6i/jsAABywSURBVGCISILqMRkMhqLZv38/PXv2ZMuWLWzevJlLL7001C5FFH/88Qfz588/Yd/U\nqVNJSEjgxRdf5LzzzsN6wo+hPGMCk8HgEPPmzeORRx7h999/B6BNmzYllDAAZGRkkJuby+zZs/ni\niy9YvHjxSXn++usvunXrxtGjR4mKigqBl5HL3r17SUlJ4bHHHuOvv/7inHPOoW7dugDcfffdBf+H\nE2E9lJeRkUHz5s3xeDxUqFABj8eDx+Ohd+/e/Prrr6F2L6JYtWoV55xzDtHR0QVbVFQUPXr04Nxz\nzy3Yt3Xr1lC7GpF89NFHDB48mN9+s36DeOmll/Lhhx+G2KvIoFq1atSuXZt77rmHxYsXk5WVRWpq\nKnl5eSdtJij55siRI1x99dV4PB6ioqLweDy0b9+ehQsXMmrUKCZOnMjVV19NRkYG77//PvXq1ePR\nRx8NtdsnE8zznzp16qRlQUZGhtauXVtr1qypGzdu1NWrV2uLFi1URHTSpEll4oOqKpCspXj2WEmb\n23quXLlS4+LiNCoqShMSEvSf//ynfvTRR7p7927Nzs7WrKwsjY2N1aioKJ0wYYKrvqhGvp6FSU9P\n127duqnH41FAPR6P/vHHH2VmP9L13Llzp/bu3VtFREVEc3JyXLXnD5Gq6ZAhQzQqKko9Ho+OGjVK\nV65cqampqUXmXbhwoXo8Hm3QoIErvngTqJ5hPZRXtWpVrrjiCt58802io6O54IILSE5O5vXXX+fJ\nJ5+kV69edO7cOdRuhjVHjx7lpZdeok+fPjz11FO0aNGCChUqnJQnf9x+yJAhoXAzotm3bx9r165F\nRPB4PIwYMYImTfx9eaihQYMG1KhRA4AuXbrg8Xj49ddf2bx5M1u3bmXq1KkFea+44gp69erFtdde\nGyp3w5YpU6bw7rvvMmPGDPr16+fXEJ2qkpKSUgbeBUZYByaAW265hTfffJN+/foxffp03nnnHVas\nWMHhw4fJzs4OtXthT4UKFZg3b57PPLNmzeLIkSO0bduWBg0Cfj/dKU+VKlWoW7cue/fuBWDx4sXs\n27ePevXqhdizyCMuLo4RI0Ywb948jh07dlL61KlTmTp1KlWrVuX1119nwIABZjEE8Oqrr/KPf/yD\njh07MnLkSL/LiUhY6hf2galy5cqICH/++ScXX2y9GLdSpUp8/vnn9OjRo4TShpLYtm0b9957LwAP\nP/wwlStXDrFHkUe9evW48cYbmTRpEmBNNv/73//miSeeCLFnkUFaWhrff/89AEuXLi3YHxUVRUJC\nQpFldu3axcCBA/n000+57LLLysTPcGb58uWoKsOHDw+onIbpI+nCevEDQIcOHUhKSmLKlCkFkb1v\n375ceOGFIfYs8lFVlixZwpEjR6hRowbdunULtUsRy7hx4074PHv2bHbtCvYltKcWqampJyy6SUhI\nYPz48axZs4YdO3YUucXGxgKwYMEC8vJ8vbz61GDmzJk8/PDDAQemcO0xhdXEXUlUqFBBRUQXLVpU\npnaJ0InQkvjss880KipKo6KidOHChWVmt7zq+fTTTxcsfvB4PNqsWbMysRvpeqakpGjNmjVVRLRJ\nkya6e/fuEsssXLiwYLHEZ5995rhPka6pP1x44YUqIlq/fn3XbQWqZ9j3mIoiJiYm1C6UC95//30A\nmjdvXjBMagiesWPHsnLlSqpWrYqIsGPHDiZMmMDOnTtD7VpYk5CQwI8//sicOXPYsGGDX3NzzZo1\nK/j/p59+ctO9csnevXtJTk5GRPh//+//hdqdk4iYwJSVlVUwHpq/gscQPDt27GD27NkAPPvss0RH\nh/10Y9hTqVIlevTowXXXXUd8fDzHjx9nwoQJnH/++aF2LeypV68eQ4YMMed2GTFlyhQyMzPp2bMn\n1113XajdOYmICUxJSUkcP36cihUr0qFDh1C7E9Hk5uby6P9v79zDqirzPf55N1gICJmFVxTxBIYa\neGlkZg4NT6lpDXnhmCmnHO+mjUftiFo2mdqUJ82yvDDeKku7WWe8YIeyiDTylqVZ1vECgoooKKKg\ngLznj7XZZ4PAvri3ay18P8+zH/de7/tb6+uX992/td73XWs/9xyVlZUkJibSv39/vSU1KFatWsXb\nb79t+5ybm6ujmobJggULbO/V3Khr7Ny5k9WrVyOEICkpSW85tWKaxJSTkwNAcnKyzkrMT2pqKmvX\nriUgIICJEyfqLadBEhUVpbcEU1BRUeHW4oVvv/0W0L4P1L2MzpOTk8PAgQPJy8tj0aJFLi0tv5GY\nIjGVlpby0ksvATB06FCd1ZibgoICRowYAcD8+fPVkns3+e233/jtt99qLdu6dSv9+vWzTeSqVWN1\n89prr/Hll1+6HHfo0CEAbr/9dmOuKjMoPXv2JD8/n27dujF+/Hi95dSJKSYWiouLbc/GCwgI0FmN\neZFS8vLLL1NcXAxod9ErXKe4uJjY2FgeeeQRxo0bZ9v+wQcfsHHjRk6cOMHVq1cRQhAXF8fGjRt1\nVGtsWrVqRVJSEqmpqdxzzz3XPJWkNk6fPg1oc81G/nI1EmVlZUycOJG8vDxatGhBamqqU17rhSkS\nk8IzrF69mkWLFgHazzPcdttttrIdO3aosXoXEEKwdu1a3n333Wrb27VrR1JSEsnJyXTs2FEndeZh\n2LBhfPrpp9x77722bUII4uPjSUxMpFu3bgB8//33bNiwga+++gqA7du306NHD7VC1wlWrlzJ2LFj\nEULw4osvMnPmTL0lOcQ0iUlKqS7Zr5PMzExAW5ETFBRERUUFmZmZTJkyhbVr1+qszjw0adKEzMxM\n9u3bB8Ds2bNJSEige/fuDBkyRGd15mP9+vWsWbOGtLQ0NmzYgJSSr776ypaEapKUlKRWOjpJfn4+\nU6dOtd1I+/nnnxMdHU1aWhr79++31XNnONWbmCYxqaR0/VT9zs3Zs2c5ffo0AwcO5Ndff+X1118n\nMjJSZ3XmIiIigoiICACVjK4TX19fxowZw8iRIykvL6eyspLly5fzwQcfsGvXLoQQSCm5//77Wbhw\nIZ07d9ZbsmkICQnh1VdfZezYsQCkp6fz9ddf2070KysrsViMt9TAeIpqoUmTJsTHxwMQGRmpvgjc\npGoFzpw5cwgNDWX37t1MnDiRJ554wpCNU3Fz4ePjg5+fH/7+/kydOpWdO3faFo9IKdm2bRsxMTHq\nnjsXGT16NNnZ2WRlZbF+/XqeeeYZ2xXUgAEDeOaZZ/SWeA2m+As3btyYzZs3ExwczJUrV5g2bZre\nkkzJ9OnTSUtLY/fu3cTFxTF//nx1T5hCcRMQGhoKaFf3Q4YMYe7cuTorqh9TJCbQflqgvLxcbxmm\nxt/f3zbPpFAoFEZFjd8oFAqFwlCIqufPuRQkxBkg2/NyDEs7KeWd3tq58tOzKD89y03oJyhPPY1L\nfrqVmBQKhUKh8BZqKE+hUCgUhkIlJoVCoVAYCpWYFAqFQmEoVGJSKBQKhaFQiUmhUCgUhkIlJoVC\noVAYCpWYFAqFQmEoVGJSKBQKhaFQiUmhUCgUhkIlJoVCoVAYCpWYFAqFQmEoVGJSKBQKhaFQiUmh\nUCgUhsKriUkIsVAI8aQ3j3Ezofz0PMpTz6L89Cw3rZ9SyjpfwK3AKrTfDSkGfgD62ZXfAnwMZAES\niK8R3xLIAW6pY/9h1jjf+nTUiHkM+BW4AOQDbwNBzsa78gKWAxftXleA4uvYnyM/Y4HPgULgDPAR\n0NLLfgpgHnACKALSgU5m8NNJT6OAPcA56+sLIMqbnlrjwoHNVk1ngf/yhqfWY00B8qx9YjVwq7f8\nrFH3b1Zvenm5jd4KLAJOWv+GS4FGXvLSo/3BifZZ5Yd9v3iuAfn5F+Bqjf9fvKM4R1dMvlZT/gQE\nA7OAD4UQYXZ1tgP/bu0Y1ZBSngIOAY84OI4rfAv8SUoZhNb5fdEakseRUo6XUgZWvYD1aMnCXRz5\n2RT4B1pja4fWkNfY6fGGn4OBkUAccDuQCaz14P5teMFPcOzpSWAIcIf1tRF4306Txz0VQtyCdoLx\nJdACaAO866n91zjWg8AM4AG0NhMOvHAdu3SmzyOE6IDWdk7Zb/dSG50B9AA6AxFAN6sub+Dp/uCU\nn8Btdn1jbtXGBuAnQKZ9v5dSpjuMcCMD7gcSa9meSy2ZEHgWWFPHvo5T/Wzh9y5qCQTeAVLrqSOB\nScBRtDPXVwCLG//vALRE8ScPn1HU6qe1rBs1rig87ScwHfjQ7nMn4LJZ/XTQRn2BiUCJlz0dC3zj\ngl63PQXWAX+3+3w/kOdtP4HPgIfQRkt61SjztJ97gEftPg8Dcrzkp0v94Xr9xIkrHpP7+Rdgu6se\nuTTHJIRojpZhD7oQ9gsQXUfZfdZ/q84WMoUQbYUQ54UQbevR8a9CiCK0L7ZE4DUHGgainSF0A/qj\nnRHhzLHsSEQbXstwoq5TOOHnfbWUedrP94EOQogIIUQjYDjal059GNJPq4ZaPRVCnAcuA28Af68R\n5mlPY4EsIcRWIcRZIUS6EKKLA+nuetoJ+NHu849AcyFEMwfHc4ra/BRCDAauSClT6wjzeJ+vKQto\nI4QIrqeOu3660x+cpp4+ny2EyBVCrBFC3FGjzMx+AnS19oPfhBDPCSF8HSpyIcs3QhufT6mjvK4r\npt7A0TpiwnBj/N4uvjUwG4hwkO372n2eAGxz41jbgNnu6HTTz3vQ5privOkn2jzh69a4CuAY0N5s\nfjrpaYBV78Ne9jQNKAf6Wf2dhna2Wdc8gdueAkdqxDay7i/MG34CTYD/rdo/tV8xedrPecAO4E60\nodGd1n20rKP+9fjpUn/wgJ+BaF/4vkBztDn7/2lAfoYD7dEW2nUBfgZmOopz6opJCGFBG2ctA55y\nJsaOJsB5F2OcQkp5Au1s5n0HVXPs3mcDrVw5jvVsIB5t2PC6ceSnEOJfgK3Af0gpv6lR7Gk//wb8\nDggF/NDmJ74UQvjXE2MoP637dNhGpZSX0BZgvCOECLEr8rSnpWjDF1ullGXAAqAZcHc9Me56ehEI\nsvtcddZb7GR8rdTj52xgrZQyq55wT/v5IrAPbeHAt8B/oyX+0/XEuOunO/3BIXX5KaW8KKXcI6Ws\nkFKetpb1EUI0sQs3rZ9SyqNSymNSykop5QFgDvBvjuIcJiYhhEBbVdIcbVy03BlBdtxN9aEGe6SL\n+6oNX6CDgzqhdu/bok2Iu8LjwA4p5VEX467BkZ9CiHZoZ1VzpZS1Tbp62s8Y4H0pZa61c7yFtggj\nqp4Yw/gJLrdRC+CPdrVdhac93e9GnLueHqT6ME80cFpKWeDi8W048PMBYJIQIk8IkWfV/aEQYrpd\nHY/6KaUslVI+JaVsLaUMBwqAvVLKynrC3PXTnf5QLy62zyp/7L+bzeznNYdHGzp0KNLRpdhy4Dsg\nsI7yW9HOLHKBPtb3wq48DbuJthqx/mhLCesciqslJgloa33fDvga+KSe+hJt2Kip1dxDwFhnj2fd\nx6/ASFdi3PET7cvyCPCf9cR72s/n0VZWNkfrDI8Dl9DGrA3vpxOe9ga6Aj5oVxaL0TqVnxc9jQRK\ngF7W406x/l3rG8pzy1OgL9qK2ChrfDrwshf9bIY2/FP1ykFbyRZoV8fTfrZGO0MXaPN3OUCfeupf\nj58u9QcP+NnT2l4sVm8/AL6qUcfMfvYDmlvfdwR+Ap53GOdgp+2soi5TfR16kl2dLGsd+1eYtawl\nWsKqtUNa68xBmwQ/bzWprfUYbeuo/6J1n5es//4DaObA1KoVJQXAQsDHWlbvsax1fm89VhN3G6az\nflo7haxRdtEu3ht++gFL0Jb9XgC+x2482ch+OunpYGtHumj1ZQtwjzc9tcYMAg5bPU2nnnthPODp\nVLRhmAtotxdcz31MDvt8jfpZXHsfk6fb6H3W45SgndTUqsUTfrraHzzQPoeizWNdsh7zHaBFA/Jz\ngbVtXrLGz8GJe6aENdgrCCEWAkeklEu9dhDHGiRwl5TysF4aPIXy0/MoTz2L8tOz3Kx+ejUxGYGG\n1EiNgPLT8yhPPYvy07Po4ad6iKtCoVAoDEWDv2JSKBQKhblQV0wKhUKhMBSOHw1RC3fccYcMCwvz\nsBTjsnfv3rNSyju9tX/lp2dRfnqWm81PUJ56Glf9dCsxhYWFsWfPHndCTYkQItub+1d+ehblp2e5\n2fwE5amncdVPNZSnUCgUCkOhEpNCoVAoDIVKTDcpn332GcnJyVgsFoQQtGnThvz8fL1lmZKPP/5Y\nbwkNmhUrVtjaacuWLTl16pTjIAWXLl3iqaeewmKx2Pzr2rUrGRkZXLlyRW959WKoxFRSUkJsbCwT\nJkzgyJEjHD58mOLi63pIssKOnj170qhRI3x8fHj44YfZsGED69at48qVK9x///307dtXb4mmIi8v\nDx8fH373u9/pLaVBcfbsWWbMmIGPjw8+Pj6MHz+eNm3aEBUVRX5+PqGhoY53chNTUlJCly5dCAoK\nYv/+/Vy8eJHKykqklOzcuZOysjL8/f2ZP38+V69e1Vtu7bjz/Kfu3btLT7Fv3z7Zo0cPGRwcLP39\n/WXr1q2lEEI2atRI+vj4SH9/fxkcHCynTp0qDxw4IA8cOCBLS0s9dnxnAPZIN5+V5czLk37Wxdy5\nc6Wvr6+0WCyyY8eO8uTJk/Ly5cu28p07d8pz587JlStXel1LQ/BTSikHDx4sY2Nj5alTp27I8eqi\nofhZRVxcnLRYLNJiscgxY8bIt956S5aWlsoRI0bYtnsbM3t66tQpabFY5PDhw2stv3jxos3HoqIi\nr+mwx1U/3VqV50lmzpzJ3r17uffeewkPD2fWrFmUl5cTFBRERUUF3333HZs2beL48ePExcVRVFRE\nSEgIr7zyCn379uXOO722orNBsXLlSiortafah4SEcPbsWVq2bGkrb9euHZ06dSIvL49Ro0bpJdM0\nHDx4kPLycjIzM2stLyws5NKlS4SGhnLo0CG++Ub7Wa3u3bvTrVu3GynVVBQWFpKVlQVAYmIiy5Yt\nw8fHR19RJmPjxo0AREfX9aO3/8+nn37K8OHDvS3JZXRPTG+99RbDhg0jJiaGhQsXXlMeERHBE088\nAcDly5fJzc3l/fffZ/LkyYSHh/Pee+8RERFxo2Wbjo8//phevXpRXFzM9u3b+cMf/kCrVq14+umn\niYyMZMmSJeTl5REUFOR4Zwo2b958zRdmcXExAwYMAKCgoICSkhIiIiLIzc3lwIEDADRv3py7776b\nbdu23XDNZmDVqlWcOHGCsLAwXn311VqTkmqj9ZOTk1NveePGjRk+fDhvv/32DVLkOronpubNm7Nl\nyxanzor8/Pxo374948eP59ixY6xZs4YXXniBlStX0rhx4xug1rz06NGDY8eO8cUXX/DYY49RUlLC\n4cOHefLJJ6vVW7FihU4KzUNxcTHff/89K1as4Pjx4zRr1oyAgADGjRtHQUEBlZWVHDx4EIDJkydz\n9epVJk+eDMDp06fp3bu3nvINy3fffcesWbMAiI+Pp02bNoA2Z7Jp0yY++eQTAFvyV7iHxWIx/Pel\n7okJtITjDD/88AMDBgygoKCAS5cuAZCWlkZWVhZ3313fr1YrAJo2bUpCQgK5ubmkpKSwd+9eUlNT\nbeVJSUn8+c9/1lGhORg5cqTtS/KXX35h8+bNBAQEEBgYyNatW5FScu7cOQAiIyO5cOECycnJlJWV\nMXToUJYtW6anfMNSVFRERUUFACdOnLDdgPrTTz/ZhpeDgoKYOXOmbhrNQNV34erVq5kyZUqtdaqG\nk999911DDuUZbuKuPlJTU6UQQgohZGxsrAwJCZH//Oc/vX5cTDwRWh+//PKLbRI0NDRUnj59+oYc\n1+x+BgcHy/j4eHn8+HH517/+VRYWFjqMmTx5srRYLDI5OdnjeszuZxX79++XgYGBtjZZ26tr1643\nRIuZPb169aocPHiwtFgscs6cObK0tFRWVlbK8vJyefnyZblkyRLp7+8vLRaLbNeunayoqPCalipc\n9dNQy8Ud0adPHyZNmoSfnx/Lly/nyJEjPPLII3rLMi1Vk8ygnaEePXpUPzEmori4mHHjxhEaGsri\nxYtp2rSpw5jFixffAGXmpkuXLvTr16/ato4dO1b7PG3atBspyZRYLBZWrFhBdHQ0s2fPJiAggAkT\nJtCvXz/CwsJIS0sjNTWVDh06kJOTU23UxCgYYijPWXx8fHjppZcIDg4mMTGR+Ph4li5dyi233KK3\nNNORmZlJUlKS3jJMx/Lly0lISODRRx91Ombp0qXMmzePadOmIYTwojrzs3r1apo0aUJ6ejqdO3dm\nwYIFjB49mu3btwPQvn17nRWag+DgYDIyMjh8+DDvvPMOAM8++yzt27cnNDQUi8XCjBkzGDNmDGvX\nriUhIUFnxdUxVWICbUXJ888/z6BBg/jjH/9IUVERH330kd6yTMesWbM4f/683jJMx4QJExg4cCAW\ni3ODDYsWLWLTpk2sXLkSX1/TdbcbTmBgIKtWraKgoIBmzZpx5coVm2+RkZF07txZZ4XmITAwkJiY\nGGJiYmotf/DBBw17Um+qobwqLBYL0dHRHDlyhJ9++omSkhK9JZmKPXv28OOPP1bb1rhxY5o0aaKT\nIvMgpaSwsJCLFy/WW+/48eNMnz6d6dOnk52dTXh4+A1S2DBo1qwZALt27SI9PR2AmJgYAgMDdVTV\nsGjdujWPP/44Z86c0VvKNZgyMQGUlZXx888/k5+fT6dOnfSWYxqOHj3Kgw8+SEVFBU8//bRte0xM\njPLRCYQQZGRkMGbMGEpLS68pP3nyJG+88QZ33XUXZ8+eZdmyZaSlpemg1FxcvnyZ5ORkNmzYUG37\n/Pnzbe/t26vCM3Ts2JGMjAy9ZVyD6RJTcXEx7733Hl27duWBBx7g3LlzDBs2TG9ZpiEvL4/z58+T\nkJBAYmKibbshl4wakKqluB9++CEjRoygsLCQUaNG0bNnT3r27ElcXBzNmjUjOzubN998k1GjRtGh\nQwedVRuftLQ0Fi5caJtLAigvL1ejIR6gvLzc9tQXs2D4Qe+MjAy2bNlCSkoKFy5cALTHuiQkJDB1\n6lSGDh2Kv7+/zirNw8SJEwFYt24d69atA7T7w7p06aKnLNNw8OBBKisrKSoqok+fPkRFRdnKUlJS\n6N+/v47qzIu/vz8Wi4XFixdz9epVCgsLWb9+PQC+vr4MGjSI7t2766zSXJSXl9OqVSuGDBnCm2++\neU35mTNnePbZZw15w7fhE1PPnj35+uuv6d27N1FRUQwaNIgWLVrQvHlzvaU1CLp27Urbtm31lmEq\nLBYLTZs2Zffu3XpLaTD06tWL8PBwDh8+zJIlS6qV9e/f35akFM7j6+vLpEmTeOGFF8jKyuLRRx+t\ndnvNvHnzKCsrY8eOHTqqrB3DJ6Zbb72V5557Tm8ZDZLo6Gi2bdtGcHCw3lIUChYvXsxDDz1UbVt0\ndLR6TJabCCGYNm0aFy5c4LPPPmPEiBHX1PHz82PLli06qKsf080xKTxDly5dSE9PV0lJYRg6d+5M\nSkoKISEhzJ49m5SUFNVGrxM/Pz9eeeUVdu3adc2V0cSJE8nOzua+++7TSV3dGP6KSeFZ9u3bp7cE\nhaJWWrduzejRoxk9erTeUhocjRs3JjY21rg/DFgDdcWkUCgUCkOhEpNCoVAoDIXQHvzqYpAQZ4Bs\nz8sxLO2klF77qVzlp2dRfnqWm9BPUJ56Gpf8dCsxKRQKhULhLdRQnkKhUCgMhUpMCoVCoTAUKjEp\nFAqFwlCoxKRQKBQKQ6ESk0KhUCgMhUpMCoVCoTAUKjEpFAqFwlCoxKRQKBQKQ6ESk0KhUCgMxf8B\nJgorJ7TP+1AAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1325f2518>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "miscl_img = X_test[y_test != yhat][:25]\n",
    "correct_lab = y_test[y_test != yhat][:25]\n",
    "miscl_lab = yhat[y_test != yhat][:25]\n",
    "\n",
    "fig, ax = plt.subplots(nrows=5, ncols=5, sharex=True, sharey=True,)\n",
    "ax = ax.flatten()\n",
    "for i in range(25):\n",
    "    img = miscl_img[i].reshape(28, 28)\n",
    "    ax[i].imshow(img, cmap='Greys', interpolation='nearest')\n",
    "    ax[i].set_title('%d) t: %d p: %d' % (i+1, correct_lab[i], miscl_lab[i]))\n",
    "\n",
    "ax[0].set_xticks([])\n",
    "ax[0].set_yticks([])\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Is 94% good for the MNIST dataset? Well, no not really. Its pretty embarassing. \n",
    "\n",
    "But we have some additional implementation details that are needed for practical implementation of neural networks, in order to get them training more efficiently and more meaningfully. These are visited in the next notebook, `08. Practical Implementation of Neural Networks`\n",
    "### [Back to Slides]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda env:machinelearning]",
   "language": "python",
   "name": "conda-env-machinelearning-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}