lec10-rnn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CS231n Winter 2016: Lecture 9\n",
    "## Topics\n",
    "Recurrent Neural Networks, Image Captioning, LSTM\n",
    "**TODO:** add topics. hm maybe there is some plugin to do this quick?\n",
    "\n",
    "## Sources\n",
    "- video: [2016](https://www.youtube.com/watch?v=yCC09vCHzF8), [2017](https://www.youtube.com/watch?v=6niqTuYFZLQ&t=0s&list=PL3FW7Lu3i5JvHM8ljYj-zLfQRF3EO8sYv&index=10)\n",
    "- slides: [2017](http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture10.pdf)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/yCC09vCHzF8?rel=0&amp;controls=1&amp;showinfo=0\" frameborder=\"0\" allowfullscreen></iframe>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "\n",
    "def play_youtube(video_id):\n",
    "    return HTML(f'<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/{video_id}?rel=0&amp;controls=1&amp;showinfo=0\" frameborder=\"0\" allowfullscreen></iframe>')\n",
    "play_youtube('yCC09vCHzF8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Types of RNN\n",
    "- one to many (image capturing)\n",
    "- many to one (sentiment classification)\n",
    "- many to many - output starts after input is ending (machine translation)\n",
    "- many to many - output starts in the same time with input (video classification of frame level)\n",
    "- ? bidirectional rnn\n",
    "\n",
    "## Application\n",
    "- [Multiple Object Recognition with Visual Attention Jimmy Ba, Volodymyr Mnih, Koray Kavukcuoglu](https://arxiv.org/abs/1412.7755) - use rnn for classify image numbers. rnn control attention and learn to read numbers from left to right\n",
    "- [DRAW: A Recurrent Neural Network For Image Generation Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, Daan Wierstra](https://arxiv.org/abs/1502.04623) - generate numbers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RNN\n",
    "$$\n",
    "h_t = f_W (h_{t-1}, x_t)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (Vanila) Recurrent Neural Network\n",
    "the state consist of a single \"hidden\" vectore h\n",
    "$$\n",
    "h_t = \\tanh(W_{hh} h_{t-1} + W_{xh} x_t)\n",
    "$$\n",
    "$$\n",
    "y_t = W_{hy} h_t\n",
    "$$\n",
    "\n",
    "- implementation on numpy https://gist.github.com/karpathy/d4dee566867f8291f086 by Karpathy\n",
    "\n",
    "- [Visualizing and Understanding Recurrent Networks Andrej Karpathy, Justin Johnson, Li Fei-Fei](https://arxiv.org/abs/1506.02078) - run rnn on text and monitor state of the single cell. In 10% cases it shows/markup something interestin like - inner of quotes, nexted expression (nested code), expression of condition, end of line and etc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### RNN for image capturing\n",
    "$$\n",
    "h = \\tanh(W_{xh} * x + W_{hh} * h + W_{ih} * v)\n",
    "$$\n",
    "\n",
    "Dataset: [Microsoft COCO](http://cocodataset.org/#home)\n",
    "\n",
    "research:\n",
    "[Show, Attend and Tell: Neural Image Caption Generation with Visual Attention Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, Yoshua Bengio](https://arxiv.org/abs/1502.03044) - soft attention, we allow model to return to the parts of image on each word and references them in descriptin. And train that lookup.\n",
    "\n",
    "https://youtu.be/yCC09vCHzF8?t=2617"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stack RNNs\n",
    "$$\n",
    "h_t^l = \\tanh W^l \\begin{pmatrix} h_t^{l-1} \\\\ h_{t-1}^{l} \\end{pmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LSTM\n",
    "research\n",
    "- [LSTM: A Search Space Odyssey Klaus Greff, Rupesh Kumar Srivastava, Jan Koutník, Bas R. Steunebrink, Jürgen Schmidhuber](https://arxiv.org/abs/1503.04069) - tried different LSTM architectures but none of them were better than LSTM\n",
    "- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, Yoshua Bengio](https://arxiv.org/abs/1406.1078)\n",
    "\n",
    "\n",
    "$$\n",
    "\\begin{pmatrix} i \\\\ f \\\\ o \\\\ g \\end{pmatrix} = \n",
    "\\begin{pmatrix} sigm \\\\ sigm \\\\ sigm \\\\ \\tanh \\end{pmatrix} W^l \\begin{pmatrix} h_t^{l-1} \\\\ h_{t-1}^{l} \\end{pmatrix}\n",
    "$$\n",
    "update cell. we could forget but also we could contribute new values from input\n",
    "$$\n",
    "c_t^l = f \\odot c_{t-1}^l + i \\odot g\n",
    "$$\n",
    "some output leaking to the hiddent state\n",
    "$$\n",
    "h_t^l = o \\odot \\tanh(c_t^l)\n",
    "$$\n",
    "```\n",
    "i - input\n",
    "f - forget\n",
    "o - output\n",
    "g - ?\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient vanishing of RNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<video autoplay=\"\" loop=\"\" class=\"\" style=\"max-width: 100%; min-height: 409.18px;\"><source type=\"video/mp4\" src=\"//i.imgur.com/vaNahKE.mp4\"></video>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "HTML(f'<video autoplay=\"\" loop=\"\" class=\"\" style=\"max-width: 100%; min-height: 409.18px;\"><source type=\"video/mp4\" src=\"//i.imgur.com/vaNahKE.mp4\"></video>')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "in RNN we have vanishing gradient problem, because we multiple on W on each step (and btw threshold if we use ReLU). In parctice developers clip very big numbers to protect from value exploding\n",
    "\n",
    "#### gradient highway\n",
    "In LSTM we don't have vanishing becase we only have addition for cell (c) (actually we have - forget gate).\n",
    "In this way LSTM reminds ResNets. \n",
    "To fight against forget gate developers sometimes add bias on init and LSTM learns to remove it. But on the first steps we have very good gradient flow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### GRU\n",
    "-  [An empirical exploration of recurrent network architectures](https://dl.acm.org/citation.cfm?id=3045367)\n",
    "\n",
    "simplified version with $c$ and $h$ just a $c$ (single hidden state)\n",
    "\n",
    "$$\n",
    "r_t = sigm(W_{xr} x_t + W_{hr} h_{t-1} + b_r)\n",
    "$$\n",
    "$$\n",
    "z_t = sigm(W_{xz} x_t + W_{hz} h_{t-1} + b_z)\n",
    "$$\n",
    "$$\n",
    "\\tilde{h_t} = \\tanh(W_{xh} x_t + W_{hh} (r_t \\odot h_{t-1}) + b_h)\n",
    "$$\n",
    "$$\n",
    "h_t = z_t \\odot h_{t-1} + (1 - z_t) \\odot \\tilde{h}_t \n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}