lec3-linear-classification-optimization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CS231n Winter 2016: Lecture 3\n",
    "## Topics\n",
    "- Linear Classification, \n",
    "- Optimization\n",
    "- SVN\n",
    "\n",
    "## Sources\n",
    "- video: https://www.youtube.com/watch?v=qlLChbHhbg4\n",
    "- original notes by Andrej Karpathy: http://cs231n.github.io/optimization-1/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/qlLChbHhbg4?rel=0&amp;controls=1&amp;showinfo=0\" frameborder=\"0\" allowfullscreen></iframe>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "video_id = 'qlLChbHhbg4'\n",
    "HTML(f'<iframe width=\"560\" height=\"315\" src=\"https://www.youtube.com/embed/{video_id}?rel=0&amp;controls=1&amp;showinfo=0\" frameborder=\"0\" allowfullscreen></iframe>')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lost function _(Cost function, Objective)_\n",
    "\n",
    "### Multiclass Support Vector Machine (SVN) loss\n",
    "- based on Weston and Watkins 1999 version\n",
    "- **TODO:** take a look on SVN [CS229 lecture notes](http://cs229.stanford.edu/notes/cs229-notes3.pdf)\n",
    "- other alternatives: _kernels, duals, the SMO optimization algorithm, One-Vs-All (OVA), All-vs-All (AVA) strategy, Structured SVM._\n",
    "\n",
    "SVN loss \n",
    "- [hinge loss](https://en.wikipedia.org/wiki/Hinge_loss) (max-margin loss). **TODO: what is it?**:\n",
    "$$\n",
    "L_i = \\sum_{j \\neq y_i} max(0, s_j - s_{y_{i}} + 1)\n",
    "$$\n",
    "where scores:\n",
    "$$\n",
    "s = f(x_i, W)\n",
    "$$\n",
    "`1` is margin\n",
    "- squared hinge loss *L2-SVM* (sometimes could be used). **TODO: why?**\n",
    "$$\n",
    "L_i = \\sum_{j \\neq y_i} max(0, s_j - s_{y_{i}} + 1)^{2}\n",
    "$$\n",
    "- usually at initialization $W$ closes to $0$ (zero). In this case $s$ also would be close to $0$ and $L_i \\approx number of classes - 1$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def l_i_vectorized(x, y, W):\n",
    "    scores = W.dot(x)\n",
    "    margins = np.maximum(0, scores - scores[y] + 1)\n",
    "    margins[y] = 0 # or we could subtract 1 instead\n",
    "    return np.sum(margins)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Regularization\n",
    "problem with this form of $W$ - that we could scale $W$ by factor (>1) and get the same loss result.\n",
    "So we should use regularization:\n",
    "$$\n",
    "L_i = \\frac{1}{N} \\sum_{i=1}^{N} \\sum_{j \\neq y_i} max(0, f(x_i, W)_j - f(x_i, W)_{y_{i}} + 1) + \\lambda R(W)\n",
    "$$\n",
    "\n",
    "in common use:\n",
    "- **L2 regularization** $R(W) = \\sum_k \\sum_l W^{2}_{k,l}$ - tries to spread $W$ and take into account the most features (prefers smaller and more diffuse weight vectors)\n",
    "- L1 regularization $R(W) = \\sum_k \\sum_l |W_{k,l}|$\n",
    "- Elastic net (L1 + L2)  $R(W) = \\sum_k \\sum_l \\beta W^{2}_{k,l} + |W_{k,l}|$\n",
    "- Max norm regularization\n",
    "- Dropout\n",
    "\n",
    "$$\n",
    "L = \"data loss\" + \"regularization loss\" = \\frac{1}{N} \\sum_{i}L_i + \\lambda R(W)\n",
    "$$\n",
    "where N is number of training examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Softmax Classifier (Multinominal Logistic Regression)\n",
    "alternative to SVN loss function.\n",
    "\n",
    "we consider that scores is unnormalized log probability of the classes\n",
    "\n",
    "$$\n",
    "P(Y=k|X=x_i) = \\frac{e^{s_k}}{\\sum_{j} e^{s_j}}\n",
    "$$\n",
    "we use the log likelihood to maximize probability (cross-entropy loss):\n",
    "$$\n",
    "L_i = - log P(Y=k|X=x_i) = - log \\frac{e^{s_{y_i}}}{\\sum_j e^{s_{y_j}}} = - s_{y_i} + log \\sum_j e^{s_{y_j}}\n",
    "$$\n",
    "\n",
    "*Property:*\n",
    "- min = 0, max = infinity\n",
    "- for small $W$, $L_i \\approx - log (1/N)$\n",
    "- for classifier with score maximum for the right classes if we will jiggle scores a little bit we likely to get the same SVN than Softmax. Robustness of SVN and Micromanagement of softmax.\n",
    "\n",
    "*Related*\n",
    "- [Kullback-Leibler divergence](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence)\n",
    "\n",
    "### numerical stability\n",
    "Dividing large numbers can be numerically unstable thus:\n",
    "$$\n",
    "\\frac{e^{s_k}}{\\sum_{j} e^{s_j}} = \\frac{C e^{s_k}}{C \\sum_{j} e^{s_j}} = \\frac{e^{s_k + log C}}{\\sum_{j} e^{s_j + log C}}\n",
    "$$\n",
    "A common chooice\n",
    "$$\n",
    "log C = - max_j f_j\n",
    "$$\n",
    "\n",
    "### interesting to know \n",
    "[Why is softmax activate function called softmax](https://www.quora.com/Why-is-softmax-activate-function-called-softmax)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "without shifting: [  0.   0.  nan]\n",
      "with shifting: [  5.75274406e-290   2.39848787e-145   1.00000000e+000]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:2: RuntimeWarning: overflow encountered in exp\n",
      "  \n",
      "/opt/conda/lib/python3.6/site-packages/ipykernel_launcher.py:2: RuntimeWarning: invalid value encountered in true_divide\n",
      "  \n"
     ]
    }
   ],
   "source": [
    "f = np.array([123, 456, 789]) # example with 3 classes and each having large scores\n",
    "p = np.exp(f) / np.sum(np.exp(f)) # Bad: Numeric problem, potential blowup\n",
    "\n",
    "print(f'without shifting: {p}')\n",
    "# instead: first shift the values of f so that the highest number is 0:\n",
    "f -= np.max(f) # f becomes [-666, -333, 0]\n",
    "p = np.exp(f) / np.sum(np.exp(f)) # safe to do, gives the correct answer\n",
    "\n",
    "print(f'with shifting: {p}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient of Softmax\n",
    "TODO: add description. For example https://stackoverflow.com/questions/41663874/cs231n-how-to-calculate-gradient-for-softmax-loss-function\n",
    "$$\n",
    "\\nabla_{w_{y_i}}L_i = \n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimization\n",
    "- *random search* - render random $W$ and check its loss function\n",
    "- *gradient*\n",
    "  - iterative refinement - update position by some delta on each iteration\n",
    "  - numerical - approximate, very slow, but easy to write (we use it for debug/check)\n",
    "  - analytical - exact, fast, but error-prone (use in practice)\n",
    "- **step-size** (learning rate) - choose right value is very complex trick\n",
    "- mini-batch gradient descent (only few samples - 32/64/128)\n",
    "\n",
    "- **ME:** *we use physic analog for optimization step value (learning rate) problem here. Optization like throwing a ball down to the funnel. But if we wouldn't have friction or any other lost energy ball would bumping around and never would set to the lowest point, so what do we need here is a decrease of energy after each iteration.* And btw it is a common practice of decaying learning rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Practice before CNN popularity\n",
    "- usually don't use raw pixels but extract features: histogram of colors (hue bins), HOG/SIFT/Texton/SSIM features\n",
    "- bag of words of features\n",
    "\n",
    "### ... to be continue [./lec4-computational-graph-backpropagation-neural-networks.ipynb](./lec4-computational-graph-backpropagation-neural-networks.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}