lec5-neural-network.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CS231n Winter 2016: Lecture 5\n",
    "## Topics\n",
    "- Neural Network\n",
    "- Activation functions\n",
    "- Data Preprocessing\n",
    "- Weight Initialization\n",
    "- Batch normalization\n",
    "- Sanity check\n",
    "- Process Visualization\n",
    "\n",
    "## Sources\n",
    "- video: https://www.youtube.com/watch?v=gYpoJMlgyXA\n",
    "- original notes by Andrej Karpathy:\n",
    "  - http://cs231n.github.io/neural-networks-2/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/gYpoJMlgyXA?rel=0&amp;controls=1&amp;showinfo=0\" frameborder=\"0\" allowfullscreen></iframe>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "id = 'gYpoJMlgyXA'\n",
    "from IPython.display import HTML\n",
    "HTML(f'<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/{id}?rel=0&amp;controls=1&amp;showinfo=0\" frameborder=\"0\" allowfullscreen></iframe>')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import math\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sigmoid activation function\n",
    "### Problems\n",
    "- it kills gradient for big values (saturated neurouns \"kills\" the gradients)\n",
    "- sigmoid outputs are not zero-centered\n",
    "  - example: when the input for a neuron (x) is always positive all coeficiens of W will be all positive or negative\n",
    "  $$\n",
    "  f(\\sum_i w_i x_i + b)\n",
    "  $$\n",
    "  and optimization will look like zig-zag path, slow convergence\n",
    "  TODO: I don't really get why is it\n",
    "- exp is bit compute exptensive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fe220d5ce80>]"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3Xl8VOXd/vHPdyb7Rla2sCMgixAg\nIu7UFdSKu7jW4lOXFh+r1q320Vbb+rjU7Vc3XGq11h0tWhRX6o6EXfawhzVAgEAg6/37Y0afGBMy\ngUnOTHK9X695MXPOSebiZLgy3Oece8w5h4iItC4+rwOIiEj4qdxFRFohlbuISCukchcRaYVU7iIi\nrZDKXUSkFVK5i4i0Qip3EZFWSOUuItIKxXj1xNnZ2a5Hjx5ePb2ISFSaOXPmFudcTmPbeVbuPXr0\noKCgwKunFxGJSma2OpTtNCwjItIKqdxFRFohlbuISCukchcRaYUaLXcze9bMNpvZtw2sNzN7xMwK\nzWyemQ0Lf0wREWmKUN65PweM3sf6MUCf4O0K4PEDjyUiIgei0XJ3zn0KbNvHJmOB513A10C6mXUK\nV0AREWm6cJznngusrfW4KLhsQ90NzewKAu/u6dat23492YxV2/hs2Rbi/EZcjI84v4+4GH/gfoyP\n+BgfCbF+4oP342P8xMf6SIz1kxjr/36dz2f79fwiItGgRS9ics5NBCYC5Ofn79eHt85aXcIjHy07\n4CwJwcJPioshMc5PclzgfnK8n8S4GFLi/aTEx5ASH0tKQgypCTGkxseQkhBDu8RY2iXGkpYQS1pi\nLH79ohCRCBOOcl8HdK31uEtwWbO48tjeXHFMLyqqa6isdlRU1Xx/K6+qpryqJngL3q8M3N9bWc3e\nyhr2VFazpyLwuKyimj2V1ZRVVFFWUU1ZeTUbduylrKKaXeVV7C4PLG9ManwMaYmxZCbHfX/LSIoj\nMzmWjOQ4slPiaZ8aT05qPNkp8STE+ptr94iIAOEp98nABDN7GTgM2OGc+9GQTDiZWWC4JQaIb85n\ngqrqGnaXV1NaXsmu8ip27qli555KdtRzKymroGR3BcuLd1Gyu4LdDfxiSEuIoX1aAh3S4unULpHO\n7RLo2C6RTukJdG6XSG5GIinxns0MISKtQKMNYmYvAaOAbDMrAu4AYgGcc08AU4BTgEKgDPh5c4X1\nQozfR7skH+2SYpv8tXsrqykpq2BLaQXFu/ZSXFpOcWk5m0vL2byznE2le/l82RY2l+6lps4gVWZy\nHN0yk+iWmUT3rCS6ZibRPTOJg9qnkJXSzL/RRCTqmXP7NfR9wPLz850mDguoqq5hU2k5G7bvYcOO\nvRSV7GHNtjLWbNvNmm1lrN++l+pa7Z+RFMtB7VOCt1QOap9C/06ptE9N8PBvISItwcxmOufyG9tO\n//ePADF+H7npieSmJ9a7vrK6hvXb97Byy26WF++mcHMphZt38d63Gykp+78TlXJS4xnYOY2BndMY\n0KkdAzun0T0rCTMd8BVpa1TuUSDW76N7VjLds5IZ1e+H67buKmfppl0s3LCTBet3sHD9Tj5ftoWq\n4Dv99KRYhnfLYFj3DIZ3z2BIl3QS43RAV6S1U7lHuayUeA5Piefw3lnfL9tbWc2yTbv4dv0OZq8p\nYebqEj5avBmAGJ8xoHMaI3pkcmSfbA7rmUlSnF4GIq2NxtzbiJLdFcxeGyj6glUlzF67nYqqGuL8\nPoZ1T+foPjkcdVA2g3Lb6bx9kQgW6pi7yr2N2ltZzTcrt/FF4RY+W7aFhRt2AoGDtcf378DJAzty\ndJ9snZMvEmFU7tIkW3aV80XhFqYtKebDRZso3VtFUpyfUf1yOHlgR35ycHvSEpp+OqiIhJfKXfZb\nRVUNX6/YytQFG3l/4SaKS8uJ8/s4vn97zh7WhWP75RDr10cBiHhB5S5hUVPjmL12O/+et4F/zVnH\n1t0VZKfEcUZeLmcP70L/TmleRxRpU1TuEnaV1TVMW1LMGzOL+GjxJiqrHQM7p3HRYd05c2iuTrEU\naQEqd2lW23ZX8Pbc9bwyYy0LN+wkPSmWC0d045LDu9OpXf0XY4nIgVO5S4twzvHNym387YtVvL9w\nIz4zxhzSifFH9mBotwyv44m0Opp+QFqEmXFYrywO65XF2m1l/P3LVbwyYy1vz13PiJ6ZXH9iX0b2\nymr8G4lIWOmdu4TdrvIqXp2xlif+s5zNpeUc0TuL60/sS36PTK+jiUQ9DcuI5/ZWVvPi9DU8Pq2Q\nLbsqOKZvDtef2Je8ruleRxOJWip3iRhlFVW88NVqnvjPckrKKjlxQAd+d2p/umclex1NJOqo3CXi\n7Cqv4rkvVvL4tOVUVjt+cUxPfvWTgzRxmUgThFruusxQWkxKfAwTjuvDx78ZxamDO/HoJ8s5/i//\n4e256/HqTYZIa6VylxbXIS2BB8/P4/WrDicjKY5rXprNBU99zZKNpV5HE2k1VO7imfwembx9zVH8\n8YxBLN5YyqmPfMZDHy6lsrrG62giUU/lLp7y+4yLR3bnkxtG8dMhnXnow2Wc+dgXLN2kd/EiB0Ll\nLhEhIzmOB8/P44mLh7Fh+15Oe+RznvzP8h98MLiIhE7lLhFl9KBOTL3uGI47uD13v7uY8578ilVb\ndnsdSyTqqNwl4mSnxPP4xcN46Pw8lm0qZczDn/HGzCKvY4lEFZW7RCQz44yhubx/3bHkdU3nhtfm\nctub8ymvqvY6mkhUULlLROvYLoEXLh/BVcf25sXpazjvia9Yt32P17FEIp7KXSJejN/HLWMO5slL\nhrOieDenPfIZny4t9jqWSERTuUvUOHlgRyZfcxTtUxP42d++4ZGPllGjs2lE6qVyl6jSMzuZN391\nBGfk5fLAB0v55Yuz2FupcXiRulTuEnWS4mJ44Lwh/O7U/kxduJGLnp7Ott0VXscSiSgqd4lKZsZ/\nHd2Lxy4cxvx1Ozj78S9ZvVXnw4t8R+UuUW3MIZ34538dRklZBWc99iVz1m73OpJIRFC5S9TL75HJ\nG1cfQVK8n3ETv+KDhZu8jiTiuZDK3cxGm9kSMys0s1vqWd/NzD4xs9lmNs/MTgl/VJGG9c5JYdLV\nR9K3QypXvlDAi9NXex1JxFONlruZ+YFHgTHAAOACMxtQZ7PfAa8654YC44DHwh1UpDE5qfG8fMVI\nju2bw21vfstzX6z0OpKIZ0J55z4CKHTOrXDOVQAvA2PrbOOAtOD9dsD68EUUCV1SXAxPXpLPiQM6\n8Pu3F/L0Zyu8jiTiiVDKPRdYW+txUXBZbb8HLjazImAKcE1938jMrjCzAjMrKC7WFYbSPOJifDx2\n0TDGDOrIH/+9iImfLvc6kkiLC9cB1QuA55xzXYBTgBfM7Eff2zk30TmX75zLz8nJCdNTi/xYrN/H\nIxcM5dTBnfjzlMU8Nq3Q60giLSqUj51fB3St9bhLcFltlwOjAZxzX5lZApANbA5HSJH9Eev38fD5\nefjNuPe9JVRXO645vo/XsURaRCjlPgPoY2Y9CZT6OODCOtusAY4HnjOz/kACoHEX8VyM38eD5+cR\n4zP+8sFSqp3j1yf09TqWSLNrtNydc1VmNgGYCviBZ51zC8zsTqDAOTcZuAF4ysyuI3Bw9TLnnGZ0\nkojg9xn3nTsEM+OhD5eRlhDL+KN6eh1LpFmF8s4d59wUAgdKay+7vdb9hcCR4Y0mEj5+n3HP2Yew\nq7ySO99ZSGZyHGcMrXtegEjroStUpc2I8ft4eNxQRvbK5DevzeWTJTokJK2Xyl3alIRYP09dmk+/\njqlc/Y+ZzFxd4nUkkWahcpc2JzUhlud+PoKOaQmMf24GSzeVeh1JJOxU7tIm5aTG88LlhxEX4+PS\nZ76hqKTM60giYaVylzara2YSz48fwe6KKi599hu2l+kDP6T1ULlLm9a/UxrP/OxQirbt4ep/zKKy\nusbrSCJhoXKXNm9Ez0zuPusQvlqxlTsmL0CXaEhrENJ57iKt3dnDu1BYvIvHpy2nb/sULjtSFzlJ\ndFO5iwTdeFI/lm/exZ3vLKRHdjKj+rX3OpLIftOwjEiQz2c8eH4e/Tqmcc0/Z1O4WadISvRSuYvU\nkhwfw9M/yyc+1s/45woo2a0zaCQ6qdxF6shNT2TipcPZuHMvV/1jJhVVOoNGoo/KXaQew7plcN85\ng5m+cht/nrLI6zgiTaYDqiINGJuXy/yiHTz9+UqGdktnbJ5mkZTooXfuIvtw85iDObRHBre8MV9z\n0EhUUbmL7EOs38dfLxxGcnwMV/1jJrvKq7yOJBISlbtIIzqkJfDXC4eyemsZN78+T1ewSlRQuYuE\nYGSvLG46uR//nr+BZ79Y5XUckUap3EVCdMUxvThpQAfunrKIGau2eR1HZJ9U7iIhMjPuP28IXTIS\n+dWLsyguLfc6kkiDVO4iTZCWEMvjFw9n595KrntlDjU1Gn+XyKRyF2mi/p3SuOOnA/m8cAsTP1vh\ndRyReqncRfbDuEO7csohHbl/6hLmrN3udRyRH1G5i+wHM+PuMwfTIS2B/35pNqV7K72OJPIDKneR\n/dQuKZaHx+VRVFLG/7z1rddxRH5A5S5yAPJ7ZHLt8X15a856Js0q8jqOyPdU7iIHaMJxBzGiZyb/\n89a3rNyy2+s4IoDKXeSA+X3GQ+fnEeP38d8vzdb87xIRVO4iYdA5PZF7zh7M/HU7uP/9JV7HEVG5\ni4TL6EEdufCwbjz12Qq+XrHV6zjSxqncRcLod6f2p3tmEje8OlenR4qnQip3MxttZkvMrNDMbmlg\nm/PMbKGZLTCzf4Y3pkh0SIqL4YHz89iwYw9/eHuh13GkDWu03M3MDzwKjAEGABeY2YA62/QBbgWO\ndM4NBH7dDFlFosKwbhn8ctRBvD6ziKkLNnodR9qoUN65jwAKnXMrnHMVwMvA2Drb/AJ41DlXAuCc\n2xzemCLR5b+P78PAzmncOmm+Zo8UT4RS7rnA2lqPi4LLausL9DWzL8zsazMbHa6AItEoLsbHQ+fn\nsau8ilsn6dObpOWF64BqDNAHGAVcADxlZul1NzKzK8yswMwKiouLw/TUIpGpT4dUbjq5Hx8u2syr\nBWsb/wKRMAql3NcBXWs97hJcVlsRMNk5V+mcWwksJVD2P+Ccm+icy3fO5efk5OxvZpGoMf7Inhze\nK4s7317Imq1lXseRNiSUcp8B9DGznmYWB4wDJtfZ5i0C79oxs2wCwzSa6FraPJ8v8OlNPjNueG0O\n1fpwD2khjZa7c64KmABMBRYBrzrnFpjZnWZ2enCzqcBWM1sIfALc6JzTVRwiQG56InecPpAZq0r4\n2xcrvY4jbYR5daAnPz/fFRQUePLcIi3NOccvni/gs2VbmHLt0fTOSfE6kkQpM5vpnMtvbDtdoSrS\nAsyMP595CAmxfm58ba6GZ6TZqdxFWkj7tAT+cPpAZq3ZzjOf65CUNC+Vu0gLGpvXmRMHdOD+95dS\nuHmX13GkFVO5i7QgM+NPZw4iKc7PDa/Npapac79L81C5i7Sw9qmB4Zm5a7fz1Gc6e0aah8pdxAOn\nD+nM6IEdefCDpSzbVOp1HGmFVO4iHjAz/njmIFISYjQ8I81C5S7ikeyUeO4cO5B5RTuY+JnOnpHw\nUrmLeOjUQzoxemBHHvpgGYWbNTwj4aNyF/GQmXHXGYNIivdz4+vzdHGThI3KXcRjOanx/P6nA5m9\nZjvPfq6zZyQ8VO4iEWBsXmdO6N+e+99fwopiXdwkB07lLhIBAhc3HUJ8jI+b35hHjYZn5ACp3EUi\nRIe0BG7/aWBq4L9/tcrrOBLlVO4iEeTsYbmM6pfDve8tYfXW3V7HkSimcheJIGbG3WcdQozPuOl1\nDc/I/lO5i0SYTu0Sue3U/kxfuY0Xp6/2Oo5EKZW7SAQ6/9CuHN0nm7vfXczabfpgbWk6lbtIBPpu\neMaAWyfNx6uPw5TopXIXiVBdMpK49ZT+fF64hZdnrPU6jkQZlbtIBLtwRDcO75XFn/69iHXb93gd\nR6KIyl0kgvl8xr3nDKbGOQ3PSJOo3EUiXNfMJG4efTCfLi3mtZlFXseRKKFyF4kCl4zszoiemdz1\nzkI27tjrdRyJAip3kSjg8xn3nj2YyuoafvumhmekcSp3kSjRIzuZG08+mI8Xb2bSrHVex5EIp3IX\niSKXHdGD/O4Z/OHtBRqekX1SuYtEEb/PuO/cIVRU13DLpHkanpEGqdxFokzP7GRuGX0w05YU81qB\nzp6R+qncRaLQpYf3YGSvTO58Z6EubpJ6qdxFopDPZ9x3zhCcc9z8uoZn5MdU7iJRqmtmEr89NTD3\nzIvT13gdRyJMSOVuZqPNbImZFZrZLfvY7mwzc2aWH76IItKQC0d04+g+2fx5yiLWbNXUwPJ/Gi13\nM/MDjwJjgAHABWY2oJ7tUoFrgenhDiki9TMz7jl7MH4zbnx9rj65Sb4Xyjv3EUChc26Fc64CeBkY\nW892dwH3ADr5VqQFdU5P5H9+OoDpK7fx3JervI4jESKUcs8Fak8mXRRc9j0zGwZ0dc79O4zZRCRE\n5w7vwnEHt+ee9xZTuLnU6zgSAQ74gKqZ+YAHgBtC2PYKMysws4Li4uIDfWoRCTIz/vfsQ0iK8/Pr\nV+ZQUVXjdSTxWCjlvg7oWutxl+Cy76QCg4BpZrYKGAlMru+gqnNuonMu3zmXn5OTs/+pReRH2qcm\ncPdZg/l23U4e/mip13HEY6GU+wygj5n1NLM4YBww+buVzrkdzrls51wP51wP4GvgdOdcQbMkFpEG\njR7UkXOHd+HxacspWLXN6zjioUbL3TlXBUwApgKLgFedcwvM7E4zO725A4pI09xx+kByMxK57tU5\nlO6t9DqOeCSkMXfn3BTnXF/nXG/n3J+Cy253zk2uZ9tRetcu4p2U+BgePC+PdSV7uOudhV7HEY/o\nClWRVii/RyZXj+rNqwVFTF2w0es44gGVu0grde3xfRmUm8atk+azuVSXn7Q1KneRViouxsdD5+ex\nu7yKmzS5WJujchdpxQ5qn8pvT+nPtCXFPPvFKq/jSAtSuYu0cpce3p0T+nfgf99dxPyiHV7HkRai\nchdp5cyM+84ZTHZKPBNemqXTI9sIlbtIG5CRHMfD44aydlsZv3vrW42/twEqd5E2YkTPTK47oS//\nmrOe12bqs1dbO5W7SBvyy58cxOG9srjjXws0e2Qrp3IXaUP8PuOhcXkkxfmZ8M/Z7K2s9jqSNBOV\nu0gb0yEtgfvPG8LijaX86d+LvI4jzUTlLtIG/aRfe644phcvfL2at+eu9zqONAOVu0gb9ZuT+jG8\newY3vzGPJRs1/t7aqNxF2qi4GB+PXTSM5PgYrvrHTHbq/PdWReUu0oZ1SEvgsYuGsXZbGde/Mpea\nGp3/3lqo3EXauEN7ZHLbqf35cNEmHptW6HUcCROVu4hw2RE9OCOvM3/5YCnTlmz2Oo6EgcpdRDAz\n7j5rMP06pHLty3NYu63M60hygFTuIgJAYpyfJy8ZjnOOK1+YqQucopzKXUS+1z0rmYfHDWXRxp38\n5jUdYI1mKncR+YGfHNyem0cfzDvzNvDQh0u9jiP7KcbrACISea48phcri3fzyMeFdM9K5uzhXbyO\nJE2kcheRHzEz7jpjEGtLyrhl0jy6ZCRyWK8sr2NJE2hYRkTqFRfj4/GLhtMtM4kr/zGTlVt2ex1J\nmkDlLiINapcUy7OXHYrPjPHPzWB7WYXXkSREKncR2afuWclMvGQ460r2cOULM6moqvE6koRA5S4i\njcrvkcl95w5m+spt3PS6TpGMBjqgKiIhGZuXS1HJHu6buoR2ibH8/vSBmJnXsaQBKncRCdkvR/Vm\nx55KJn66grTEWG44qZ/XkaQBKncRCZmZceuYg9m5p5L/93EhqQkxXHFMb69jST1U7iLSJGbGn848\nhNLyKv48ZTGpCbFcMKKb17GkDpW7iDSZ32c8eF4eu8ur+O2b80lNiOG0wZ29jiW1hHS2jJmNNrMl\nZlZoZrfUs/56M1toZvPM7CMz6x7+qCISSb67yCm/ewbXvTKHTzQPfERptNzNzA88CowBBgAXmNmA\nOpvNBvKdc4OB14F7wx1URCJPYpyfZy47lH4dU7nyhZl8vHiT15EkKJR37iOAQufcCudcBfAyMLb2\nBs65T5xz383u/zWgWYZE2oi0hFheGH8Y/ToECv7d+Ru8jiSEVu65wNpaj4uCyxpyOfDugYQSkeiS\nkRzHi784jMFd0vnVP2fx5uwiryO1eWG9QtXMLgbygfsaWH+FmRWYWUFxcXE4n1pEPJaWEMvz40cw\nslcW1786l5e+WeN1pDYtlHJfB3St9bhLcNkPmNkJwG3A6c658vq+kXNuonMu3zmXn5OTsz95RSSC\nJcfH8OxlhzKqbw63TprP375Y6XWkNiuUcp8B9DGznmYWB4wDJtfewMyGAk8SKHYdMhdpwxJi/Tx5\nST6jB3bkD28v5LFphTinuWhaWqPl7pyrAiYAU4FFwKvOuQVmdqeZnR7c7D4gBXjNzOaY2eQGvp2I\ntAFxMT7+euFQxuZ15t73lnDH5AVUVWs2yZYU0kVMzrkpwJQ6y26vdf+EMOcSkSgX4/fx4Hl5dExL\n4MlPV7BmWxn/74KhpCbEeh2tTdCUvyLSbHw+49ZT+nP3WYfw2bItnPvEV6zbvsfrWG2Cyl1Emt0F\nI7rx95+PYN32PZzx6BfMK9rudaRWT+UuIi3iqD7ZTLr6COJjfJz35Fe89+1GryO1aip3EWkxfTqk\n8tavjqR/pzSufnEmD7y/hGp9qlOzULmLSIvKTonnpV+M5JxhXXjk40Iufno6m0v3eh2r1VG5i0iL\nS4j1c9+5Q7j/3CHMXlvCKQ9/zpeFW7yO1aqo3EXEM+cM78LkCUeRnhTLRc9M56EPl2qYJkxU7iLi\nqb4dUpk84UjOHJrLQx8u49JnNUwTDip3EfFcUlwMfzl3CPeePZiCVSWc+MCnTJpVpGkLDoDKXUQi\ngplx3qFdmXLt0RzUPoXrX53L5X8vYMMOXfS0P1TuIhJReuek8OqVh3P7aQP4cvkWTnrgU17+Zo3e\nxTeRyl1EIo7fZ4w/qidTf30MA3PTuGXSfC555hvWbitr/IsFULmLSATrnpXMP/9rJH88YxCz15Rw\n/AP/4f6pS9hdXuV1tIincheRiObzGReP7M4H1x/LKYM68tdPChl1/zReK1hLjU6bbJDKXUSiQuf0\nRB4aN5RJvzyC3PREbnx9Hqc/+jnfrNzmdbSIpHIXkagyrFsGk64+gofH5bF1VwXnPfkVV70wk0Ub\ndnodLaKE9GEdIiKRxOczxublctKAjjz56XKe/mwl7y3YyMkDO3DNcX0YlNvO64ieM69OL8rPz3cF\nBQWePLeItC7byyp49otV/O2LlZTureKE/u255rg+DOma7nW0sDOzmc65/Ea3U7mLSGuxY08lz3+5\niqc/X8mOPZUc2zeH8Uf15OiDsvH5zOt4YaFyF5E2q3RvJS98vZpnP1/Jll0V9MxO5pKR3Tknvwtp\nUf4Zrip3EWnzyquqee/bjfz9y1XMWrOdpDg/Zw7N5dLDe9CvY6rX8faLyl1EpJb5RTt4/qtV/Gvu\neiqqahjSpR1nDs3ltCGdyU6J9zpeyFTuIiL1KNldwRuzinhz9joWrN+J32cc3SebM4fmcuKADiTF\nRfZJhCp3EZFGLN1Uyluz1/GvOetZt30PSXF+jju4PScO6MCovu1plxR54/MqdxGRENXUOGas2sZb\nc9bzwcJNbNlVTozPGNEzkxP6d+CE/h3olpXkdUxA5S4isl9qahxzi7bz4aJNfLhwM0s2lQLQOyeZ\nIw/K5ojeWYzslUV6Upwn+VTuIiJhsGZrGR8s2sRny4r5ZuU2yiqqMYMBndI4oncWh/fOYmjXDDKS\nW6bsVe4iImFWWV3DvKLtfFm4lS+Xb2Xm6hIqqmsA6JGVxNBuGQztlk5e13T6d0oj1h/+6btU7iIi\nzWxvZTVz1m5nztrtzF5Twqw12ykuLQcgLsZHvw6p9O+USv9OaYFbx7QDPkgbarlH9jk/IiIRLCHW\nz8hegTF4AOcc63fsZfaaEuas2c6ijTv5cNFmXi0o+v5rctMTuWl0P8bm5TZrNpW7iEiYmBm56Ynk\npidy2uDOQKDwi0vLWbhhJ4s2lLJow05yUpv/oimVu4hIMzIz2qcl0D4tgVH92rfY84Y02m9mo81s\niZkVmtkt9ayPN7NXguunm1mPcAcVEZHQNVruZuYHHgXGAAOAC8xsQJ3NLgdKnHMHAQ8C94Q7qIiI\nhC6Ud+4jgELn3ArnXAXwMjC2zjZjgb8H778OHG9mrWPyZBGRKBRKuecCa2s9Lgouq3cb51wVsAPI\nCkdAERFpuhb9gGwzu8LMCsysoLi4uCWfWkSkTQml3NcBXWs97hJcVu82ZhYDtAO21v1GzrmJzrl8\n51x+Tk7O/iUWEZFGhVLuM4A+ZtbTzOKAccDkOttMBn4WvH8O8LHz6tJXERFp/Dx351yVmU0ApgJ+\n4Fnn3AIzuxMocM5NBp4BXjCzQmAbgV8AIiLiEc/mljGzYmD1fn55NrAljHHCRbmaRrmaLlKzKVfT\nHEiu7s65Rse1PSv3A2FmBaFMnNPSlKtplKvpIjWbcjVNS+Rq0bNlRESkZajcRURaoWgt94leB2iA\ncjWNcjVdpGZTrqZp9lxROeYuIiL7Fq3v3EVEZB8ittzN7FwzW2BmNWaWX2fdrcHphZeY2ckNfH3P\n4PTDhcHpiMP+6bXB7zsneFtlZnMa2G6Vmc0Pbtfsny1oZr83s3W1sp3SwHb7nMq5GXLdZ2aLzWye\nmb1pZukNbNci+ysSp7I2s65m9omZLQy+/q+tZ5tRZraj1s/39ubOFXzeff5cLOCR4P6aZ2bDWiBT\nv1r7YY6Z7TSzX9fZpsX2l5k9a2abzezbWssyzewDM1sW/DOjga/9WXCbZWb2s/q2aRLnXETegP5A\nP2AakF9r+QBgLhAP9ASWA/56vv5VYFzw/hPA1c2c9y/A7Q2sWwVkt+C++z3wm0a28Qf3XS8gLrhP\nBzRzrpOAmOD9e4B7vNpfofz9gV8CTwTvjwNeaYGfXSdgWPB+KrC0nlyjgHda6vUU6s8FOAV4FzBg\nJDC9hfP5gY0EzgP3ZH8BxwBRVXwhAAAD8klEQVTDgG9rLbsXuCV4/5b6XvdAJrAi+GdG8H7GgWSJ\n2HfuzrlFzrkl9awaC7zsnCt3zq0ECglMS/y94HTDxxGYfhgC0xGf0VxZg893HvBScz1HMwhlKuew\ncs697wKzhgJ8TWCeIq9E5FTWzrkNzrlZwfulwCJ+PAtrpBoLPO8CvgbSzaxTCz7/8cBy59z+Xhx5\nwJxznxK4Sr+22q+jhrroZOAD59w251wJ8AEw+kCyRGy570MoUxBnAdtrFUl924TT0cAm59yyBtY7\n4H0zm2lmVzRjjtomBP9r/GwD/w0MZT82p/EE3uXVpyX2V8RPZR0cBhoKTK9n9eFmNtfM3jWzgS0U\nqbGfi9evqXE0/AbLi/31nQ7OuQ3B+xuBDvVsE/Z95+lnqJrZh0DHelbd5pz7V0vnqU+IGS9g3+/a\nj3LOrTOz9sAHZrY4+Bu+WXIBjwN3EfjHeBeBIaPxB/J84cj13f4ys9uAKuDFBr5N2PdXtDGzFOAN\n4NfOuZ11Vs8iMPSwK3g85S2gTwvEitifS/CY2unArfWs9mp//YhzzplZi5yi6Gm5O+dO2I8vC2UK\n4q0E/ksYE3zHVd82YclogSmOzwKG7+N7rAv+udnM3iQwJHBA/yhC3Xdm9hTwTj2rQtmPYc9lZpcB\npwHHu+BgYz3fI+z7qx5Nmcq6yPYxlXW4mVksgWJ/0Tk3qe762mXvnJtiZo+ZWbZzrlnnUAnh59Is\nr6kQjQFmOec21V3h1f6qZZOZdXLObQgOU22uZ5t1BI4NfKcLgeON+y0ah2UmA+OCZzL0JPAb+Jva\nGwRL4xMC0w9DYDri5vqfwAnAYudcUX0rzSzZzFK/u0/goOK39W0bLnXGOc9s4PlCmco53LlGAzcB\npzvnyhrYpqX2V0ROZR0c038GWOSce6CBbTp+N/ZvZiMI/Dtu1l86If5cJgOXBs+aGQnsqDUc0dwa\n/N+zF/urjtqvo4a6aCpwkpllBIdRTwou238tcQR5f24ESqkIKAc2AVNrrbuNwJkOS4AxtZZPAToH\n7/ciUPqFwGtAfDPlfA64qs6yzsCUWjnmBm8LCAxPNPe+ewGYD8wLvrA61c0VfHwKgbMxlrdQrkIC\n44pzgrcn6uZqyf1V398fuJPALx+AhOBrpzD4WurVAvvoKALDafNq7adTgKu+e50BE4L7Zi6BA9NH\ntECuen8udXIZ8Ghwf86n1lluzZwtmUBZt6u1zJP9ReAXzAagMthflxM4TvMRsAz4EMgMbpsPPF3r\na8cHX2uFwM8PNIuuUBURaYWicVhGREQaoXIXEWmFVO4iIq2Qyl1EpBVSuYuItEIqdxGRVkjlLiLS\nCqncRURaof8PoVY29fvyqmEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe220dc2470>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "class Neuron:\n",
    "    def __init__(self, inputs = 3):\n",
    "        self.weights = np.random.randn(inputs,)\n",
    "        self.bias = np.random.rand()\n",
    "\n",
    "    def forward(self, inputs):\n",
    "        \"\"\" assume inputs and weights are 1-D numpy arrays and bias is a number \"\"\"\n",
    "        value = inputs * self.weights\n",
    "        if len(inputs.shape) > 1:\n",
    "            value = np.sum(value, axis=-1)\n",
    "        cell_body_sum = value + self.bias\n",
    "        firing_rate = 1.0 / (1.0 + np.exp(-cell_body_sum)) # sigmoid activation function\n",
    "        return firing_rate\n",
    "\n",
    "data = np.linspace(-10, 10)\n",
    "plt.plot(data, Neuron(1).forward(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Relu\n",
    "- Solves almost all problems of Sigmoid. Except\n",
    "### Problems\n",
    "- Not zero-centered output\n",
    "- An annoyance - gradient of value less then 0 is 0.\n",
    "  - dead Relu - it could comes on initial state - when W define that way that it only produce 0 gradient on input data\n",
    "  - or it could become dead Relu after some iteration with high learning rate (step) optimization could set such high \n",
    "  values for W so it gets dead gradient and will never update. Andrej told that it possible to get 10% dead neurons \n",
    "  that way.\n",
    "  - decrease changes - init bies = 0.01 (slightly positive numbers) - controversial point\n",
    "  \n",
    "## LeakyRelu\n",
    "- Solve probem of `die relu`.\n",
    "\n",
    "## PPeLU (Parameteric Rectifier)\n",
    "$$\n",
    "f(x) = max(\\alpha x, x)\n",
    "$$\n",
    "and is learned $\\alpha$ by backprop\n",
    "\n",
    "## Exponential Linear Units (ELU)\n",
    "\n",
    "## Maxout Neuron\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing\n",
    "- zero-centering\n",
    "```\n",
    "X -= np.mean(X, axis=0)\n",
    "```\n",
    "- normalize\n",
    "```\n",
    "X /= np.std(X, axis=0)\n",
    "```\n",
    "- decorrelat (PCA)\n",
    "    ```python\n",
    "    # get the data covariance matrix\n",
    "    cov = np.dot(X.T, X) / X.shape[0]\n",
    "    # the columns of `U` are the eigenvectors and `S` is a 1-D array of the singular values.\n",
    "    U,S,V = np.linalg.svd(cov)\n",
    "    # To decorrelate the data, we project the original (but zero-centered) data into the eigenbasis\n",
    "    # Notice that the columns of `U` are a set of orthonormal vectors, so they can be regarded as basis vectors\n",
    "    Xrot = np.dot(X, U) # decorrelate the data\n",
    "    # covariance matrix of Xrot would be diagonal\n",
    "    ```\n",
    "  - [SVD](https://en.wikipedia.org/wiki/Positive-definite_matrix#Negative-definite.2C_semidefinite_and_indefinite_matrices)\n",
    "  - reduce dimensionality `np.linalg.svd` returns U -- the eigenvector columns are sorted by their eigenvalues,\n",
    "  - so we could get top few eigenvectors and discarding the rest:\n",
    "  ```python\n",
    "  Xrot_reduced = np.dot(X, U[:,:100])\n",
    "  ```\n",
    "  - this way we would keep data which have the most variance\n",
    "\n",
    "- whitening - covariance matrix is identity matrix, normalize scale \n",
    "```python\n",
    "# divide by the eigenvalues (which are square roots of the singular values)\n",
    "Xwhite = Xrot / np.sqrt(S + 1e-5)\n",
    "```\n",
    "  - if the input data is a multivariable gaussian this tranformation makes them mean with zero and identity covariance matrix.\n",
    "  - Warning: Exaggerating noise.\n",
    "\n",
    "### For images commonly use zero-centering\n",
    "- subtract the mean image [width, height, rgb]\n",
    "- subtract per-channel mean [r,g,b]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Weight initialization\n",
    "\n",
    "### Shouldn't do\n",
    "- $W = 0$\n",
    "- small numbers\n",
    "```\n",
    "W = 0.001 * np.random.randn(D, H)\n",
    "```\n",
    "it works ok for small network but for big one you will get a problem: \n",
    "  - if we start with `mean = 0.0` and `std = 0.01` after each layer `mean` becomes more centered and `std -> 0`. and finally collapses to the `0`. Almost the same will be for gradient\n",
    "- std = 1.0\n",
    "```\n",
    "W = 1.0 * np.random.randn(D, H)\n",
    "```\n",
    "  - if we start with `mean = 0.0` and `std = 1.0` after few layers `W` will be saturated to `-1` and `1`. Gradients also be all `0`\n",
    "  \n",
    "### Should do\n",
    "- xavier initialization for `tanh`\n",
    "```\n",
    "W = np.random.randn(fan_in, fan_out) /np.sqrt(fan_in)\n",
    "```\n",
    "where `fan_in` - number of inputs\n",
    "  - it still decrease standard deviation but not so dramatic as previous\n",
    "  - it doesn't work for `ReLU` - decrese of standard deviation is much more rappid. Because `Relu` you 'kill' of distribution\n",
    "  \n",
    "- He et al. 2015\n",
    "```\n",
    "W = np.random.randn(fan_in, fan_out) /np.sqrt(fan_in / 2)\n",
    "```\n",
    "- biases set to zero "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Batch normalization (Ioffe and Szegedy)\n",
    "just make unit gaussian activation\n",
    "$$\n",
    "\\hat{x}^{(k)} = \\frac{x^{(k)} - E[x^{(k)}]}{\\sqrt{Var[x^{(k)}]}}\n",
    "$$\n",
    "it works because it's completely (vanilla) differentiable function\n",
    "- for each feature independently\n",
    "- usage: `FC -> BN -> tanh -> FC -> BN -> tanh`. \n",
    "Because we are not sure that `tanh` would like gaussian on input. we shift and scale input:\n",
    "\n",
    "$$\n",
    "y^{(k)} = \\gamma^{(k)} \\hat{x}^{(k)} + \\beta^{(k)}\n",
    "$$.\n",
    "\n",
    "and learn thoese params. But it could learn to disable batch norm by getting:\n",
    "\n",
    "$$\n",
    "\\gamma^{(k)} = \\sqrt{Var[x^{(k)}]}\n",
    "$$\n",
    "$$\n",
    "\\beta^{(k)} = E[x^{(k)}]\n",
    "$$\n",
    "\n",
    "### Features\n",
    "- improves gradient flow through the network\n",
    "- allow higher learning rates\n",
    "- reduces the strong dependencies on initialization\n",
    "- act as a form of regularization in a funny way, and slightly reduces the need for dropout, maybe\n",
    "\n",
    "### Details\n",
    "- in test time mean/std are fixed and could be estimated during training with running averages - because we want more deterministic function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sanity check of nn\n",
    "## check the loss function\n",
    "\n",
    "\n",
    "- for 10 classes, with W init by `mean = 0` and `std = 0.0001` and `reg = 0.0` it should ~ log(10).\n",
    "- with `reg = 1e3` it should going up\n",
    "\n",
    "## try to overfit small pease data\n",
    "- 20 examples, 10 labels\n",
    "- turn off regularization `reg = 0`\n",
    "- use vanilla `sgd`\n",
    "- loss should be near `0`. and `100%` accuracy\n",
    "\n",
    "## Hyper parameter optimization\n",
    "- tune  learning rate\n",
    "  - sometime cost decrease slowly but in contrast accuraccy step much quicker. The reason that accuracy just takes into account bigger score, but accuracy (in softmax) works with all scores, so they could be near each other but we already have right leaders\n",
    "- coarse -> fine\n",
    "- grid search -> random layout ([Random Search for Hyper-Parameter Optimization by Bergstra and Bengio](http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf))\n",
    "  - because usually one parameter is better than others so when we use grid search we just get much less data samples on the most significant parameter then if we randomize input a little bit\n",
    "- implementation in cluster: `worker` run the model for random hyperparameters and store the result to files, name of file reflect performance, `master` launch/kill the `worker`-s\n",
    "- usually developers prefer single validation set instead of cross-validation\n",
    "- use log scale range for learning rate, but linear scale for drop-out\n",
    "- start from wide range of parameters but less epoch (maybe 1) and then narrow range but increase epoches\n",
    "- _Bayesian Hyperparameter Optimization_ - more sophisticated way to search optimal hyperparameters doesn't show significant results for ConvNets. [Discussion](http://nlpers.blogspot.com/2014/10/hyperparameter-search-bayesian.html)\n",
    "\n",
    "### Evalution and Model Ensembles\n",
    "- the best hyper parameters for the single model but with different initialization (problem - less variation)\n",
    "- top few of hyperparameters (problem - could increase sub-optimal)\n",
    "- different checkpoint of a single model (proc - very cheap, cos - less variation)\n",
    "- _running average of parameters during training_\n",
    "- [Geoff Hinton on “Dark Knowledge”](https://www.youtube.com/watch?v=EK61htlw8hY) - _\"distill\" a good ensemble back to a single model by incorporating the ensemble log likelihoods into a modified objective._ \n",
    "  \n",
    "## Visualize process\n",
    "- for example visulize all loss functions on cluster\n",
    "  - https://lossfunctions.tumblr.com/ loss function could tell a lot\n",
    "- accuracy\n",
    "- different between parameters and scale of your update of those parameters \n",
    "  - it should be about 1e-3. (`dW * learning rate / W`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}