lec4-computational-graph-backpropagation-neural-networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CS231n Winter 2016: Lecture 4\n",
    "## Topics: \n",
    "- Computational Graph\n",
    "- Backpropagation\n",
    "- Neural Networks\n",
    "\n",
    "## Sources\n",
    "- video: https://www.youtube.com/watch?v=i94OvYb6noo\n",
    "- original notes by Andrej Karpathy: ...\n",
    "  - http://cs231n.github.io/optimization-2/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/i94OvYb6noo?rel=0&amp;controls=1&amp;showinfo=0\" frameborder=\"0\" allowfullscreen></iframe>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "video_id = 'i94OvYb6noo'\n",
    "HTML(f'<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/{video_id}?rel=0&amp;controls=1&amp;showinfo=0\" frameborder=\"0\" allowfullscreen></iframe>')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient\n",
    "- **gradient check** - validate of algorithm implementation correctness by comparising analytically computing gradient with numerical gradient.\n",
    "- **gradient update** - Gradient Descent\n",
    "- alternatives to gradient descent - [LBFGS](https://en.wikipedia.org/wiki/Limited-memory_BFGS)\n",
    "\n",
    "### Mini-batch\n",
    "it works because images (traingin data) are correlated with each other.\n",
    "- single sample -- Stochastic Gradient Descent (SGD) (on-line gradient descent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computational Graph\n",
    "**TODO:** maybe there is the way to show this graph, like gradient flow and try it interactive?\n",
    "- **add gate** - gradient distributor. _Spread gradient to all gates_\n",
    "- **max gate** - gradient router. _Because only max $x_i$ could influent on gradient of $max(x_1, ..., x_n)$_\n",
    "- **mul gate** - gradient ... \"switcher\". _One gate takes value of oposite gate and multiple on result gradient_\n",
    "\n",
    "**TODO:** what does _gradients add at branches_ mean? Do we really could have computation graph with multiple result?\n",
    "\n",
    "### property\n",
    "- no any loops\n",
    "- scale/vectorization - derivative/jacobian_matrices\n",
    "\n",
    "## Implementations of gates\n",
    "- https://github.com/torch/nn Torch names it layers. Lue\n",
    "- exp [Caffee](http://caffe.berkeleyvision.org/tutorial/layers/exp.html)\n",
    "    - C++ for exp https://github.com/BVLC/caffe/blob/master/src/caffe/layers/exp_layer.cpp\n",
    "    - CUDA GPU for exp https://github.com/BVLC/caffe/blob/master/src/caffe/layers/exp_layer.cu\n",
    "- [Caffee2 operations sources](https://github.com/caffe2/caffe2/tree/master/caffe2/operators) and [docs](https://caffe2.ai/docs/operators-catalogue.html) C++ and GPU\n",
    "- Tensorflow, Keras and something on js?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Backpropagation\n",
    "- from math - chain rule\n",
    "- the derivative on each variable tells you the sensitivity of the whole expression on its value.\n",
    "- the multiply gate gives big gradient for smaller input and vice versa. This way If you scale data set (X) gradient of weights (W) will increase up to this scale. Thus we should compensate it with learning rate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "\n",
    "## Neural Network\n",
    "Before: Linear function (1 layer)\n",
    "$$\n",
    "f = Wx\n",
    "$$\n",
    "Now: Non-linear function (2 layers)\n",
    "$$\n",
    "f = W_2 max(0, W_1x)\n",
    "$$\n",
    "2nd layer is  activation function Relu\n",
    "\n",
    "- **TODO:** what if we would have hidden layer dimantion less then dimantion of output?\n",
    "- **TODO:** could we just predefine layers how we could think they should look like. For example for fully connected layers we just could place templates of real car\n",
    "- **TODO:** could compare different activation functions (Sigmoid, tanh, ReLu, Leaky ReLu, Maxout, ELU)\n",
    "- another words - it is cernal trick where we have finaly layer which works as linear classifir (for example for binary classification it just split n-dim surface to by n-1 dim plane to 2 sets positive and negative"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}