diff --git a/lectures/lec19/Clustering.ipynb b/lectures/lec19/Clustering.ipynb new file mode 100644 index 0000000..9e64dcf --- /dev/null +++ b/lectures/lec19/Clustering.ipynb @@ -0,0 +1,434 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# CS429: Information Retrieval\n", + "\n", + "
\n", + "\n", + "## Lecture 19: Clustering\n", + "\n", + "
\n", + "\n", + "### Dr. Aron Culotta\n", + "### Illinois Institute of Technology" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Recall classification problem notation:\n", + "\n", + "\n", + "- $\\vec{x} \\in \\mathcal{X}$      *instance*, *example*, *input*\n", + " - e.g., an email\n", + "- $y \\in \\mathcal{Y}$      *target*, *class*, *label*, *output*\n", + " - e.g., $y=1$: spam ; $y=-1$: not spam\n", + "- $f: \\mathcal{X} \\mapsto \\mathcal{Y}$      *hypothesis*, *learner*, *model*, *classifier*\n", + " - e.g., if $x$ contain the word *free*, $y$ is $1$." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "** Training data:**\n", + "\n", + "We are given training data $D = \\{(\\vec{x}_1, y_1), \\ldots, (\\vec{x}_n, y_n)\\}$\n", + "\n", + "||free|money| |*label*|\n", + "|:--:|:--------:|:--------:|:--:|:--:|\n", + "||$x_{i1}$|$x_{i2}$| | $y_i$ |\n", + "|$x_1$|0|0||-1| \n", + "|$x_2$|1|0|| 1|\n", + "|$x_3$|1|1||-1|\n", + "|$x_4$|1|0||-1|\n", + "|$x_5$|1|1||1|\n", + "|$x_6$|0|0||1|\n", + "|$x_7$|0|1||-1|\n", + "\n", + "How to classify a new instance? \n", + " \"free money\" -> $\\{1,1\\}$\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**In clustering, we don't get any labels!**\n", + "\n", + "\n", + "||free|money| |*label*|\n", + "|:--:|:--------:|:--------:|:--:|:--:|\n", + "||$x_{i1}$|$x_{i2}$| | $y_i$ |\n", + "|$x_1$|0|0||?| \n", + "|$x_2$|1|0||?|\n", + "|$x_3$|1|1||?|\n", + "|$x_4$|1|0||?|\n", + "|$x_5$|1|1||?|\n", + "|$x_6$|0|0||?|\n", + "|$x_7$|0|1||?|" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**What can we possibly hope to learn using only X??**\n", + "\n", + "- **Patterns of features**: E.g., the terms \"free\", \"money\", and \"credit\" often appear together\n", + "- **Patterns of documents**: E.g., these 1,000 emails all look pretty similar\n", + "\n", + "




\n", + "**Why do this?**\n", + "- Helps the user explore/visualize data\n", + " - E.g., lawyers looking for criminal activity in [Enron emails](https://en.wikipedia.org/wiki/Enron_scandal)\n", + " - Bioligists looking for patterns in DNA\n", + "- May help as a preprocessing step for supervised learning\n", + " - Reduce dimensionality of feature vectors\n", + " - Provide more predictive features than raw features" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Today**: Document clustering\n", + "\n", + "**Types of document clustering**:\n", + "\n", + "- Flat clustering\n", + "\n", + "![flat](images/clustering.gif)\n", + "[source](http://home.deib.polimi.it/matteucc/Clustering/tutorial_html/images/clustering.gif)\n", + "\n", + "


\n", + "\n", + "- Hierarchical clustering\n", + "\n", + "![hclust](images/hclust.png)\n", + "[source](http://cs.jhu.edu/~razvanm/fs-expedition/tux3.html)\n", + "\n", + "


\n", + "\n", + "- Hard vs. soft clustering\n", + "![soft](images/soft.png)\n", + "[source](http://home.deib.polimi.it/matteucc/Clustering/tutorial_html/images/clustering.gif)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Flat Clustering: Problem statement**\n", + "\n", + "Input:\n", + "- documents $D = \\{d_1 \\ldots d_N\\}$\n", + "- desired number of clusters $K$\n", + "- **objective function** that evaluates quality of clustering\n", + "\n", + "Output:\n", + "- An assignment $\\gamma : D \\rightarrow \\{1, \\ldots, K\\}$ that optimizes the objective function" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Similar to our gradient descent recipe:**\n", + "\n", + "1. pick a model\n", + "2. pick an error function\n", + "3. minimize that error function" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**But**, what is a good objective function?\n", + "\n", + "





" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**K-Means**\n", + "- Identify $K$ **cluster centers**\n", + "- Minimize distance from each document to its assigned cluster center.\n", + "\n", + "![kmeans](images/kmeans.png)\n", + "[source](http://blog.mpacula.com/2011/04/27/k-means-clustering-example-python/)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "- Each cluster center represented by one **mean vector** $ \\in \\{\\mu_1 \\ldots \\mu_K\\}$\n", + " - $\\mu_i \\in \\mathbb{R}^V$ if $V$ terms in vocabulary\n", + "- Each document $x$ associated with exactly one mean vector (hard clustering)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**K-Means** Algorithm Overview\n", + "\n", + "1. Pick $K$ mean vectors at random\n", + "2. Iterate until convergence:\n", + " 1. Assign each document $x_i$ to its closest mean vector $\\mu_j$\n", + " 2. Update each mean vector $\\mu_j$ to be the mean of the $x_i$'s assigned to it" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**K-means objective**\n", + "\n", + "- Let $M = \\{\\mu_1 \\ldots \\mu_K\\}$ be the set of mean vectors.\n", + "- Let $r_{ij} = 1$ if $x_i$ belongs to cluster $j$, otherwise 0.\n", + "\n", + "$$E(D, M) = \\sum_{i=1}^N \\sum_{j=1}^K r_{ij} d(x_i, \\mu_j)$$\n", + "\n", + "= \"sum of distances from each document to its assigned cluster center.\"\n", + "\n", + "So, optimal clustering is:\n", + "\n", + "$$M^* \\leftarrow \\mathop{\\rm argmin}_M E(D, M)$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Distance $d$?**\n", + "\n", + "E.g., Euclidean:\n", + "\n", + "$$d(x_i, \\mu_j) = \\sqrt{\\sum_{p=1}^V (x_{ip} - \\mu_{jp})^2}$$\n", + "\n", + "E.g.:\n", + "\n", + "- $x_i = \\{1,2,3\\}$\n", + "- $\\mu_j = \\{1, 4, 1\\}$\n", + "- $d(x_i, \\mu_j) = \\sqrt{(1-1)^2 + (2-4)^2 + (3-1)^2} = 8$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**K-Means** is a **greedy** algorithm to minimize $E$\n", + "\n", + "![kmeansalg](images/kmeansalg.png)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Not guaranteed to find global optimum!**\n", + "\n", + "![nonconvex](images/nonconvex.png)\n", + "[source](http://sebastianraschka.com/faq/docs/visual-backpropagation.html)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**K-Means is sensitive to initialization**" + ] + }, + { + "cell_type": "code", + "execution_count": 42, + "metadata": { + "collapsed": false + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAewAAAEKCAYAAADQN2b/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEQBJREFUeJzt3X+s3XV9x/Hni1XSAsqMkZECVhmKBkFlKm5s2VFDiz8C\nYjICbvPXQoybkWhnCvzTy/xDJemURP8okzEwsmV0YzonXiR4ZjSLvwBbEXDqwi+xiwGh0lKqvPfH\nOZTbu9t7T3+c+z2f3ucj+Sbf7/d8z/2++6X0dT8/zvmkqpAkSZPtsK4LkCRJCzOwJUlqgIEtSVID\nDGxJkhpgYEuS1AADW5KkBizruoD5JPEzZ5KkJaWqMtf5iW9hV9VEbevXr++8hlY2n5XPyefks5rk\nbRKf03wmPrAlSZKBLUlSEwzsfdTr9bouoRk+q9H4nEbjcxqdz2o0rT2nLNRn3qUkNcn1SZJ0MCWh\nWp10JkmSDGxJkppgYEuS1AADW5KkBhjYkiQ1wMCWJKkBBrYkSQ0wsCVJasBYAzvJ1Um2Jtk849xz\nk9yc5J4k00mOHmcNkiQdCsbdwr4GWDPr3CXALVV1MnArcOmYa5AkqXljDeyq+gbwyKzT5wLXDvev\nBd42zhokSToUdDGGfUxVbQWoqp8Dx3RQgyRJTVnWdQHAvKt7TE1N7d7v9XrNra4iSdLe9Pt9+v3+\nSNeOfbWuJKuAf6+q04bHdwG9qtqa5Fjga1X1sr2819W6JElLRterdWW4Pe2LwLuH++8CvrAINUiS\n1LSxtrCTXA/0gOcBW4H1wL8BNwAnAPcC51fVL/fyflvYkqQlY74W9ti7xA+EgS1JWkq67hKXJEkH\nyMCWJKkBBrYkSQ0wsCVJaoCBLUlSAwxsSZIaYGBLktQAA1uSpAYY2JIkNcDAliSpAQa2JEkNMLAl\nSWqAgS1JUgMMbEmSGmBgS5LUAANbkqQGGNiSJDXAwJYkqQEGtiRJDTCwJUlqgIEtSVIDDGxJkhpg\nYEuS1AADW5KkBhjYkiQ1wMCWJKkBBrYkSQ0wsCVJakBngZ3k0iR3Jtmc5PNJDu+qFkmSJl0ngZ1k\nFXAR8KqqOg1YBlzQRS2SJLVgWUf3fQx4EjgyyVPAEcDPOqpFkqSJ10kLu6oeATYA9wEPAr+sqlu6\nqEWSpBZ00sJOciLwIWAV8CiwKck7qur62ddOTU3t3u/1evR6vUWqUpKk8er3+/T7/ZGuTVWNt5q5\nbpqcD5xVVRcNj/8cOKOqPjDruuqiPkmSupCEqspcr3U1S/we4HVJlicJ8Ebgro5qkSRp4nU1hv19\n4Drge8D3gQBXdVGLJEkt6KRLfFR2iUuSlpJJ7BKXJEn7wMCWJKkBBrYkSQ0wsCVJaoCBLUlSAwxs\nSZIaYGBLktQAA1uSpAYY2JIkNcDAliSpAQa2JEkNMLAlSWqAgS1JUgMMbEmSGmBgS5LUAANbkqQG\nGNiSJDXAwJYkqQEGtiRJDTCwJUlqgIEtSVIDDGxJkhpgYEuS1AADW5KkBhjYkiQ1wMCWJKkBBrYk\nSQ0wsCVJakBngZ3k6CQ3JLkryZ1JzuiqFkmSJt2yDu99JfDlqvqTJMuAIzqsRZKkiZaqWvybJs8B\nbq+q313guuqiPkmSupCEqspcr3XVJf4i4BdJrklyW5KrkqzoqBZJkiZeV4G9DDgd+ExVnQ5sBy7p\nqBZJkiZeV2PYDwD3V9V3h8ebgHVzXTg1NbV7v9fr0ev1xl2bJEmLot/v0+/3R7q2kzFsgCT/CVxU\nVT9Ksh44oqrWzbrGMWxJ0pIx3xh2l4H9CuCzwLOAnwLvqapHZ11jYEuSloz9DuzhbO7nV9VPZp0/\nrao2H9wy57y/gS1JWjL2a5Z4kvOBu4F/GX6xyWtmvPwPB7dESZI0n/lmiV8G/F5VvRJ4D/C5JOcN\nX5sz/SVJ0njMN0v8t6rqIYCq+naS1wNfSnICYD+1JEmLaL4W9rYku7+JbBjePeBc4JQx1yVJkmaY\nr4X9fmZ1fVfVtiRnA+ePtSpJkrSHzj7WNQpniUuSlpJJ/C5xSZK0DwxsSZIaYGBLktSABQM7yVuT\n3J7k4SSPJdmW5LHFKE6SJA0sOOksyY+BtwNbFnsGmJPOJElLyYFOOnsA+IHJKUlSd0ZZD3sdcFOS\nPrDz6ZNV9bfjKkqSJO1plMD+KPArYDlw+HjLkSRJcxklsFdW1cvHXokkSdqrUcawv5xk9dgrkSRJ\nezXKLPFtwJEMxq93Mfh+8aqq54y9OGeJS5KWkPlmic/bJZ4kwClVdd9YKpMkSSOZt0t82Lz9j0Wq\nRZIk7cUoY9i3JXnN2CuRJEl7NcoY9t3AScC9wOM8M4Z92tiLcwxbkrSE7PcY9tCag1yPJEnaRwt2\niVfVvcAJwBuG+9tHeZ8kSTp4RukSXw+8Gji5ql6SZCVwQ1WdOfbi7BKXJC0hB7r4x3nAOQzGr6mq\nnwHPPnjlSZKkhYwS2E8Om7kFkOTI8ZYkSZJmGyWw/znJRuC3k1wE3AJ8drxlSZKkmRYcwwZIchaw\nmsFHuqar6qvjLmx4X8ewJUlLxnxj2KNMOvtEVa1b6Nx+FnYY8F3ggao6Z47XDWxJ0pJxoJPOzprj\n3JsOrKTdLgZ+eJB+liRJh6y9BnaS9yfZApycZPOM7X+AzQd64yTHA2/G8XBJkhY03zedXQ/cBHwM\nuGTG+W1V9fBBuPcngY8ARx+EnyVJ0iFtr4FdVY8CjwIXHuybJnkLsLWq7kjSYzCZTZIk7cUo3yU+\nDmcC5yR5M7ACeHaS66rqnbMvnJqa2r3f6/Xo9XqLVaMkSWPV7/fp9/sjXTvSx7rGKckfA2udJS5J\nWuoOdJa4JEnqWOct7PnYwpYkLSW2sCVJapyBLUlSAwxsSZIaYGBLktQAA1uSpAYY2JIkNcDAliSp\nAQa2JEkNMLAlSWqAgS1JUgMMbEmSGmBgS5LUAANbkqQGGNiSJDXAwJYkqQEGtiRJDTCwJUlqgIEt\nSVIDDGxJkhpgYEuS1AADW5KkBhjYkiQ1wMCWJKkBBrYkSQ0wsCVJaoCBLUlSAwxsSZIaYGBLktQA\nA1uSpAZ0EthJjk9ya5I7k2xJ8sEu6pAkqRWpqsW/aXIscGxV3ZHkKOB7wLlVdfes66qL+iRJ6kIS\nqipzvdZJC7uqfl5Vdwz3fwXcBRzXRS2SJLWg8zHsJC8EXgl8q9tKJEmaXMu6vPmwO3wTcPGwpf3/\nTE1N7d7v9Xr0er1FqU2SpHHr9/v0+/2Rru1kDBsgyTLgS8BNVXXlXq5xDFuStGTMN4bdZWBfB/yi\nqj48zzUGtiRpyZi4wE5yJvB1YAtQw+2yqvrKrOsMbEnSkjFxgT0qA1uStJRM3Me6JEnSvjGwJUlq\ngIEtSVIDDGxJkhpgYEuS1AADW5KkBhjYkiQ1wMCWJKkBBrYkSQ0wsCVJaoCBLUlSAwzsEU1Pw+rV\ng216uutqJEkHZNOmwdYQF/8YwfQ0nHce7NgxOF6xAm68Edas6bYuSdJ+2LEDVq4c7D/0ECxf3m09\nM7j4xwHasOGZsIbB/oYN3dUjSToAGzfCrl2DbePGrqsZmYEtSVo6duyAyy+Hxx8fbFNT8MQTXVc1\nEgN7BGvXDrrBn7ZixeCcJKkxGzfC9u3PHG/f3kwre1nXBbRgzZrBmPXT3eBr1zp+LUlN2rkTTj11\nz3ONtLCddCZJ0oRw0pkkSY0zsCVJaoCBLUlSAwxsSZIaYGBLktQAA1uSpAYY2JIkNcDAliSpAQa2\nJEkNMLAlSWpAZ4Gd5Owkdyf5UZJ1XdUhSVILOgnsJIcBnwbWAKcAFyZ5aRe1jGp6GlavHmzT011X\nI0k6IJs2DbaGdLL4R5LXAeur6k3D40uAqqpPzLpuIhb/mJ6G884bLKMKg+U1b7zRFbskqUk7dsDK\nlYP9hx6C5cu7rWeGSVz84zjg/hnHDwzPTaQNG54JaxjsP73UpiSpMRs3wq5dg62RtbChgfWwp6am\ndu/3ej16vV5ntUiSGrdjB1x+OTz++OB4agre977OWtn9fp9+vz/StV12iU9V1dnDY7vEJUnj96lP\nwbp18OSTg+PDD4crroCLL+62rqH5usS7amF/BzgpySrgIeAC4MKOalnQmjWDgH66G3ztWsNakpq0\ncyeceuqe5554opta9lEnLWwYfKwLuJLBOPrVVfXxOa6ZiBa2JEmLYb4WdmeBPQoDW5K0lEziLHFJ\nkrQPDGxJkhpgYEuS1AADW5KkBhjYkiQ1wMCWJKkBBrYkSQ0wsCVJaoCBLUlSAwxsSZIaYGDvo1GX\nQZPPalQ+p9H4nEbnsxpNa8/JwN5Hrf0H7pLPajQ+p9H4nEbnsxpNa8/JwJYkqQEGtiRJDZj45TW7\nrkGSpMXU5HrYkiRpwC5xSZIaYGBLktQAA3sfJDk7yd1JfpRkXdf1TKIkxye5NcmdSbYk+WDXNU2y\nJIcluS3JF7uuZZIlOTrJDUnuGv7dOqPrmiZRkkuHz2dzks8nObzrmiZFkquTbE2yeca55ya5Ock9\nSaaTHN1ljQsxsEeU5DDg08Aa4BTgwiQv7baqifRr4MNVdQrw+8Bf+ZzmdTHww66LaMCVwJer6mXA\nK4C7Oq5n4iRZBVwEvKqqTgOWARd0W9VEuYbBv98zXQLcUlUnA7cCly56VfvAwB7da4H/rqp7q2oX\n8E/AuR3XNHGq6udVdcdw/1cM/mE9rtuqJlOS44E3A5/tupZJluQ5wB9V1TUAVfXrqnqs47Im0WPA\nk8CRSZYBRwA/67akyVFV3wAemXX6XODa4f61wNsWtah9ZGCP7jjg/hnHD2AQzSvJC4FXAt/qtpKJ\n9UngI4Af1Zjfi4BfJLlmOHxwVZIVXRc1aarqEWADcB/wIPDLqrql26om3jFVtRUGjQ3gmI7rmZeB\nrbFIchSwCbh42NLWDEneAmwd9kZkuGluy4DTgc9U1enAdgZdmZohyYnAh4BVwErgqCTv6Laq5kz0\nL88G9ugeBF4w4/j44TnNMuyO2wR8rqq+0HU9E+pM4JwkPwX+EXh9kus6rmlSPQDcX1XfHR5vYhDg\n2tOrgW9W1cNV9RvgX4E/6LimSbc1ye8AJDkW+N+O65mXgT267wAnJVk1nHl5AeDM3rn9PfDDqrqy\n60ImVVVdVlUvqKoTGfxdurWq3tl1XZNo2GV5f5KXDE+9ESfqzeUe4HVJlicJg+fk5Lw9ze7N+iLw\n7uH+u4CJbmAs67qAVlTVb5J8ALiZwS86V1eV/zPMkuRM4E+BLUluZ9DFdFlVfaXbytS4DwKfT/Is\n4KfAezquZ+JU1feHvTTfA34D3A5c1W1VkyPJ9UAPeF6S+4D1wMeBG5K8F7gXOL+7ChfmV5NKktQA\nu8QlSWqAgS1JUgMMbEmSGmBgS5LUAANbkqQGGNiSJDXAwJYOQcPlKN+/iPe7KckjLhMqjY+BLR2a\nngv85b6+afgNWfvjCuDP9vO9kkZgYEuHpo8BJw5Xt/oEQJK/TvLtJHckWT88tyrJ3UmuTbIFOCHJ\ntiRXJPlBkpuTnJGkn+THSd46182q6muAi7xIY2RgS4emS4CfVNXpVbUuyVnAi6vqtcCrgFcn+cPh\ntScBn66qU6vqPuBI4JaqejmDEP4b4A3A24GPLvqfRBLgd4lLS8Vq4KwktzFY/OBI4MUM1ni/t6q+\nM+PanVV183B/C/BEVT01bIGvWsyiJT3DwJaWhgAfq6q/2+Nksgp4fNa1u2bsPwXsBKiqGi6dKqkD\ndolLh6ZtwLNnHE8D701yJECSlUmeP3xt9kSz+SaeLfTa/k5ak7QAf1uWDkFV9XCSbybZDNw0HMd+\nGfBfw4ng2xjM6n6KwRKoe7x9vh8918kkXwdOBo4aLl34F1X11QP9c0h6hstrSpLUALvEJUlqgIEt\nSVIDDGxJkhpgYEuS1AADW5KkBhjYkiQ1wMCWJKkBBrYkSQ34P8U1+0+/dFEnAAAAAElFTkSuQmCC\n", + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAewAAAEKCAYAAADQN2b/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFYRJREFUeJzt3X2QnXV99/H3N10iMdHUyVQlYlOCKVUCKPIcHg50slFg\npOmMqHj3IXWiY1pl2siAjjNZ7YyCM3tXRvtHrEiwY27mJjUWG3FpgCNZvGl8AEM1K01hEpKQOA5I\neFhCGr73H+cYNptk9yzJ2ev8dt+vmTNzXb9zbc53Lw755Pdwzi8yE0mS1NmmVF2AJEkanYEtSVIB\nDGxJkgpgYEuSVAADW5KkAhjYkiQVoKvqAkYSEX7mTJI0qWRmHK6943vYmdlRjxUrVlReQykP75X3\nyfvkverkRyfep5F0fGBLkiQDW5KkIhjYY1Sr1aouoRjeq9Z4n1rjfWqd96o1pd2nGG3MvEoRkZ1c\nnyRJx1JEkKUuOpMkSQa2JElFMLAlSSqAgS1JUgEMbEmSCmBgS5JUAANbkqQCGNiSJBWgrYEdEbdE\nxO6I2DSk7Q0RcXdE/DIi+iJiZjtrkCRpImh3D/tWYNGwthuA9Zl5CnAv8Ok21yBJUvHaGtiZ2Q88\nPaz5KuC25vFtwJ+0swZJkiaCKuaw35iZuwEycxfwxgpqkCSpKF1VFwCMuLtHT0/PgeNarVbc7iqS\nJB1JvV6nXq+3dG3bd+uKiDnAdzPz9Ob5ZqCWmbsj4s3AfZn59iP8rLt1SZImjap364rm47fuBP6y\nefwXwL+OQw2SJBWtrT3siFgN1IBZwG5gBfAd4A7grcBW4OrM/M0Rft4etiRp0hiph932IfGjYWBL\nkiaTqofEJUnSUTKwJUkqgIEtSVIBDGxJkgpgYEuSVAADW5KkAhjYkiQVwMCWJKkABrYkSQUwsCVJ\nKoCBLUlSAQxsSZIKYGBLklQAA1uSpAIY2JIkFcDAliSpAAa2JEkFMLAlSSqAgS1JUgEMbEmSCmBg\nS5JUAANbkqQCGNiSJBXAwJYkqQAGtiRJBTCwJUkqgIEtSVIBDGxJkgpQWWBHxKcj4ucRsSkivhUR\nU6uqRZKkTldJYEfEHGAp8K7MPB3oAj5YRS2SJJWgq6LX3QO8BEyPiJeB1wI7K6pFkqSOV0kPOzOf\nBnqBbcAO4DeZub6KWiRJKkElPeyImAv8LTAHeAZYExHXZObq4df29PQcOK7VatRqtXGqUpKk9qrX\n69Tr9ZaujcxsbzWHe9GIq4GFmbm0ef5nwLmZ+TfDrssq6pMkqQoRQWbG4Z6rapX4L4HzIuL4iAjg\nj4HNFdUiSVLHq2oO+2fAN4GfAD8DAvhaFbVIklSCSobEW+WQuCRpMunEIXFJkjQGBrYkSQUwsCVJ\nKoCBLUlSAQxsSZIKYGBLklQAA1uSpAIY2JIkFcDAliSpAAa2JEkFMLAlSSqAgS1JUgEMbEmSCmBg\nS5JUAANbkqQCGNiSJBXAwJYkqQAGtiRJBTCwJUkqgIEtSVIBDGxJkgpgYEuSVAADW5KkAhjYkiQV\nwMCWJKkABrYkSQUwsCVJKoCBLUlSASoL7IiYGRF3RMTmiPh5RJxbVS2SJHW6rgpf+2bge5n5/ojo\nAl5bYS2SJHW0yMzxf9GI1wMPZebJo1yXVdQnSVIVIoLMjMM9V9WQ+EnAryPi1oj4aUR8LSKmVVSL\nJEkdr6rA7gLOBP4xM88EXgBuqKgWSZI6XlVz2NuBJzLzx83zNcD1h7uwp6fnwHGtVqNWq7W7NkmS\nxkW9Xqder7d0bSVz2AAR8QNgaWY+GhErgNdm5vXDrnEOW5I0aYw0h11lYJ8BfB04DngMWJKZzwy7\nxsCWJE0arzqwm6u5fy8z/3tY++mZuenYlnnY1zewJUmTxqtaJR4RVwMDwL80v9jk7CFPrzq2JUqS\npJGMtEr8M8C7M/OdwBLgnyNicfO5w6a/JElqj5FWif9OZj4JkJkbI+JS4N8i4q2A49SSJI2jkXrY\nz0bEgW8ia4Z3DbgKOLXNdUmSpCFG6mF/nGFD35n5bES8B7i6rVVJkqSDVPaxrla4SlySNJl04neJ\nS5KkMTCwJUkqgIEtSVIBRg3siLgyIh6KiKciYk9EPBsRe8ajOEmS1DDqorOI2AL8KfDIeK8Ac9GZ\nJGkyOdpFZ9uB/zQ5JUmqTiv7YV8P3BURdWDvbxsz83+3qyhJknSwVgL774HngOOBqe0tR5IkHU4r\ngT07M+e3vRJJknRErcxhfy8iutteiSRJOqJWVok/C0ynMX+9j8b3i2dmvr7txblKXJI0iYy0SnzE\nIfGICODUzNzWlsokSVJLRhwSb3Zv141TLZIk6QhamcP+aUSc3fZKJEnSEbUyhz0AvA3YCjzPK3PY\np7e9OOewJUmTyKuew25adIzrkSRJYzTqkHhmbgXeClzWPH6hlZ+TJEnHTitD4iuAs4BTMvMPI2I2\ncEdmLmh7cQ6JS5ImkaPd/GMx8D4a89dk5k7gdceuPEmSNJpWAvulZjc3ASJientLkiRJw7US2P83\nIlYCvxsRS4H1wNfbW5YkSRpq1DlsgIhYCHTT+EhXX2b+e7sLa76uc9iSpEljpDnsVhad3ZSZ14/W\n9ioLmwL8GNieme87zPMGtiRp0jjaRWcLD9P23qMr6YBrgV8coz9LkqQJ64iBHREfj4hHgFMiYtOQ\nx+PApqN94Yg4Ebgc58MlSRrVSN90thq4C/gicMOQ9mcz86lj8Nr/AFwHzDwGf5YkSRPaEQM7M58B\nngE+dKxfNCKuAHZn5sMRUaOxmE2SJB1BK98l3g4LgPdFxOXANOB1EfHNzPzz4Rf29PQcOK7VatRq\ntfGqUZKktqrX69Tr9ZaubeljXe0UEZcAy10lLkma7I52lbgkSapY5T3skdjDliRNJvawJUkqnIEt\nSVIBDGxJkgpgYEuSVAADW5KkAhjYkiQVwMCWJKkABrYkSQUwsCVJKoCBLUlSAQxsSZIKYGBLklQA\nA1uSpAIY2JIkFcDAliSpAAa2JEkFMLAlSSqAgS1JUgEMbEmSCmBgS5JUAANbkqQCGNiSJBXAwJYk\nqQAGtiRJBTCwJUkqgIEtSVIBDGxJkgpgYEuSVAADW5KkAlQS2BFxYkTcGxE/j4hHIuKTVdQhSVIp\nIjPH/0Uj3gy8OTMfjogZwE+AqzJzYNh1WUV9kiRVISLIzDjcc5X0sDNzV2Y+3Dx+DtgMvKWKWiRJ\nKkHlc9gR8QfAO4H/qLYSSZI6V1eVL94cDl8DXNvsaR+ip6fnwHGtVqNWq41LbZIktVu9Xqder7d0\nbSVz2AAR0QX8G3BXZt58hGucw5YkTRojzWFXGdjfBH6dmX83wjUGtiRp0ui4wI6IBcD9wCNANh+f\nyczvD7vOwJYkTRodF9itMrAlSZNJx32sS5IkjY2BLUlSAQxsSZIKYGBLklQAA1uSpAIY2JJGtW7d\nOubOnXtI+wUXXMDq1asrqEiafAxsSaO6/fbbmTVr1kFtAwMDPPjgg5xwwgkVVSVNLga2pFHV63Uu\nvfTSg9ruv/9+pk6dyvnnn19RVdLkYmBLGtGWLVvYsWPHIYG9YcMGzj77bI4//viD2p977jne//73\ns3379vEsU5rwDGxJI7rvvvvo6urioosuOqh9w4YNXHzxxQe13XLLLfT29vLtb3+bl19+eTzLlCY8\nA1vSiOr1OvPnz2fGjBkH2rZu3cq2bdu45JJL2L9/P5s3bwbgIx/5CCtWrMCvFJaOPQO7RX190N3d\nePT1VV2N1B59fX10d3fT3d1NX/ONXq/XmTdv3iHXTZkyhQULFrBx40YeeuihKsqVXr01axqPgnRV\nXUAJ+vpg8WIYHGyc9/fD2rWwaFG1dUnHUl9fH4sXL2aw+Ubv7+/nK1/5Ck8++SQ7d+48cN3AwACr\nVq1i1qxZTJ8+nXXr1nHddddVVbY0doODsHRp4/jKK2HYOoxOZQ+7Bb29r4Q1NI57e6urR2qH3t7e\nA2ENMDg4SG9vL8cddxwnn3wyS5Ys4VOf+hTf/e53ueeeezj33HNZtmwZs2fPZubMmRVWLo3RypWw\nb1/jsXJl1dW0zB62pCN66qmnOOuss1i1atUhz915553jX5B0tAYH4XOfg+efb5z39MDHPlZEL9se\ndguWL4dp0145nzat0SZNJMuXL2fakDf6tGnT2LdvH5dddlmFVUnH2MqV8MILr5y/8EIxvWx72C1Y\ntKgxZ/3bYfDly52/1sSzaNEi1q5dS2/zjX711Vfz0Y9+9JDPX49k9erV9Pf3ExHccMMNXHjhhSxb\ntqxdJUtjt3cvnHbawW0vvlhNLWMUnfzxi4jITq5PmsjuvvtuPvGJT7Bp0yZe85rXVF2ONClEBJkZ\nh32ukwPRwJYkTSYjBbZz2JIkFcDAliSpAAa2JEkFMLAlSSqAgS1JUgEMbEmjWrduHXPnzj2k/YIL\nLmD16tUVVCRNPga2pFHdfvvtzJo166C2gYEBHnzwQU444YSKqpImFwNb0qjq9foh33h2//33M3Xq\nVM4///yKqpImF7+aVNKItmzZwo4dOw4J7A0bNnD22Wdz/JBNEzZu3Eh/fz979uzhhz/8IZ/97Ge5\n+OKLx7tkaUKqLLAj4j3Al2n08m/JzJuqqkXSkd133310dXVx0UUXHdS+YcMGPvzhDx84Hxwc5Dvf\n+Q5f+MIXAFizZg3vfe972bJli8Pm0jFQyZB4REwBvgosAk4FPhQRf1RFLa3q64Pu7sajr6/qaqT2\n6Ovro7u7m+7ubvqab/R6vc78+fOZMWPGgeu2bt3Ktm3buOSSS9i/fz+bN29my5Yt3HTTTTz22GNA\nYzORwcFBHnjggUp+F2lEa9Y0HgWpqod9DvBfmbkVICJuB64CBiqqZ0R9fbB4cWMbVYD+/sbuXe7Y\npYmkr6+PxYsXM9h8o/f397N27Vrq9ToXXnjhIddOmTKFBQsWsHHjRh5//HGuueYaHnjggQOryZ94\n4gkignnz5o377yKNaHAQli5tHF95ZRF7YUN1i87eAjwx5Hx7s60j9fa+EtbQOP7tVpvSRNHb23sg\nrKExxP35z3+eJ598kp07dx5oHxgYYNWqVcyaNYvp06ezbt06rrjiCgDOO++8A9fdeOONLF++nDPO\nOGP8fgmpFStXwr59jUche2FDAYvOenp6DhzXajVqtVpltUiTzdNPP81xxx3HySefzJIlS5g1axZv\netObuOeee/jABz7AsmXLmD9/PjNnzjzo577xjW8we/Zsbrzxxooql45gcBA+9zl4/vnGeU8PfOxj\nlfWy6/U69Xq9pWsr2V4zIs4DejLzPc3zG4AcvvCsU7bXHD4kPm2aQ+KaeIYPiU+bNo1zzjmHffv2\njWkeet26dfzqV79iyZIl7N27l127djFnzpx2lS2NzZe/DNdfDy+91DifOhW+9CW49tpq62rqxO01\nfwS8LSLmRMRU4IPAnRXVMqpFixoBvXBh42FYayJatGgRa9euZeHChSxcuJC1a9fy6KOPctlll7X8\nZ/zgBz9g9+7dXH755ezatYu77rqLXbt2tbFqaYz27oXTToN3v7vxOO00ePHFqqtqSSU9bDjwsa6b\neeVjXYeMnXVKD1uajAYGBnjHO97B+vXrWwrtxx9/nDPOOIPnm0ONmUlE8Mwzzxy0wlzSkY3Uw65s\nDjszvw+cUtXrSxrZtm3bmDdvHgsWLGjp+pNOOok9e/a0uSpp8qqsh90Ke9iSpMmkE+ewJUnSGBjY\nkiQVwMCWJKkABrYkSQUwsCVJKoCBLUlSAQxsSZIKYGBLklQAA1uSpAIY2GPU6jZo8l61yvvUGu9T\n67xXrSntPhnYY1Taf+Aqea9a431qjfepdd6r1pR2nwxsSZIKYGBLklSAjt+tq+oaJEkaT0farauj\nA1uSJDU4JC5JUgEMbEmSCmBgj0FEvCciBiLi0Yi4vup6OlFEnBgR90bEzyPikYj4ZNU1dbKImBIR\nP42IO6uupZNFxMyIuCMiNjffW+dWXVMniohPN+/Ppoj4VkRMrbqmThERt0TE7ojYNKTtDRFxd0T8\nMiL6ImJmlTWOxsBuUURMAb4KLAJOBT4UEX9UbVUd6X+Av8vMU4Hzgb/2Po3oWuAXVRdRgJuB72Xm\n24EzgM0V19NxImIOsBR4V2aeDnQBH6y2qo5yK42/v4e6AVifmacA9wKfHveqxsDAbt05wH9l5tbM\n3AfcDlxVcU0dJzN3ZebDzePnaPzF+pZqq+pMEXEicDnw9apr6WQR8Xrgosy8FSAz/ycz91RcVifa\nA7wETI+ILuC1wM5qS+ocmdkPPD2s+SrgtubxbcCfjGtRY2Rgt+4twBNDzrdjEI0oIv4AeCfwH9VW\n0rH+AbgO8KMaIzsJ+HVE3NqcPvhaREyruqhOk5lPA73ANmAH8JvMXF9tVR3vjZm5GxqdDeCNFdcz\nIgNbbRERM4A1wLXNnraGiIgrgN3N0YhoPnR4XcCZwD9m5pnACzSGMjVERMwF/haYA8wGZkTENdVW\nVZyO/sezgd26HcDvDzk/sdmmYZrDcWuAf87Mf626ng61AHhfRDwG/B/g0oj4ZsU1dartwBOZ+ePm\n+RoaAa6DnQU8kJlPZeZ+4NvABRXX1Ol2R8SbACLizcCvKq5nRAZ2634EvC0i5jRXXn4QcGXv4X0D\n+EVm3lx1IZ0qMz+Tmb+fmXNpvJfuzcw/r7quTtQcsnwiIv6w2fTHuFDvcH4JnBcRx0dE0LhPLs47\n2PDRrDuBv2we/wXQ0R2MrqoLKEVm7o+IvwHupvEPnVsy0/8ZhomIBcCHgUci4iEaQ0yfyczvV1uZ\nCvdJ4FsRcRzwGLCk4no6Tmb+rDlK8xNgP/AQ8LVqq+ocEbEaqAGzImIbsAK4EbgjIv4K2ApcXV2F\no/OrSSVJKoBD4pIkFcDAliSpAAa2JEkFMLAlSSqAgS1JUgEMbEmSCmBgSxNQczvKj4/j690VEU+7\nTajUPga2NDG9AVg21h9qfkPWq/El4H+9yp+V1AIDW5qYvgjMbe5udRNARHwqIjZGxMMRsaLZNici\nBiLitoh4BHhrRDwbEV+KiP+MiLsj4tyIqEfEloi48nAvlpn3AW7yIrWRgS1NTDcA/52ZZ2bm9RGx\nEJiXmecA7wLOiogLm9e+DfhqZp6WmduA6cD6zJxPI4Q/D1wG/Cnw9+P+m0gC/C5xabLoBhZGxE9p\nbH4wHZhHY4/3rZn5oyHX7s3Mu5vHjwAvZubLzR74nPEsWtIrDGxpcgjgi5n5Twc1RswBnh927b4h\nxy8DewEyM5tbp0qqgEPi0sT0LPC6Ied9wF9FxHSAiJgdEb/XfG74QrORFp6N9tyrXbQmaRT+a1ma\ngDLzqYh4ICI2AXc157HfDvy/5kLwZ2ms6n6ZxhaoB/34SH/04Roj4n7gFGBGc+vCj2Tmvx/t7yHp\nFW6vKUlSARwSlySpAAa2JEkFMLAlSSqAgS1JUgEMbEmSCmBgS5JUAANbkqQCGNiSJBXg/wPSwn5Y\nPcvHwwAAAABJRU5ErkJggg==\n", + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAewAAAEKCAYAAADQN2b/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFRxJREFUeJzt3X+QXGWd7/H3N3eIiYlyrakVQSRLNMsiEZQFBCLQsJWJ\niiWbrRIE91eWorbMRandLEW0rMqsW6VgVe6Kv6pwRYJb5lJL1rjcjTghQkuCl40/wLCacTeACUlI\nLItIACNkw/f+Me0wM5n09CTpOf1k3q+qrjp9+nT6M4ehP3Oe83SfyEwkSVJnm1J1AEmSNDYLW5Kk\nAljYkiQVwMKWJKkAFrYkSQWwsCVJKkBX1QGaiQg/cyZJmlQyM0Zb3/FH2JnZUbdly5ZVnqGUm/vK\n/eR+cl918q0T91MzHV/YkiTJwpYkqQgW9jjVarWqIxTDfdUa91Nr3E+tc1+1prT9FGONmVcpIrKT\n80mSdDRFBFnqpDNJkmRhS5JUBAtbkqQCWNiSJBXAwpYkqQAWtiRJBbCwJUkqgIUtSVIB2lrYEXF7\nROyOiE1D1r0uItZGxM8ioi8ijm9nBkmSjgXtPsK+A1gwYt1SYF1mngbcD3yszRkkSSpeWws7MzcA\ne0asvgK4s7F8J/BH7cwgSdKxoIpz2K/PzN0AmbkLeH0FGSRJKkpX1QGAplf36O3tHVyu1WrFXV1F\nkqRDqdfr1Ov1lrZt+9W6ImIW8H8z88zG/c1ALTN3R8QbgAcy8/RDPNerdUmSJo2qr9YVjdtv3QP8\nRWP5z4F/nYAMkiQVra1H2BGxEqgB3cBuYBnwTeBu4E3AVuDKzPzVIZ7vEbYkadJodoTd9iHxI2Fh\nS5Imk6qHxCVJ0hGysCVJKoCFLUlSASxsSZIKYGFLklQAC1uSpAJY2JIkFcDCliSpABa2JEkFsLAl\nSSqAhS1JUgEsbEmSCmBhS5JUAAtbkqQCWNiSJBXAwpYkqQAWtiRJBbCwJUkqgIUtSVIBLGxJkgpg\nYUuSVAALW5KkAljYkiQVwMKWJKkAFrYkSQWwsCVJKoCFLUlSASxsSZIKUFlhR8THIuInEbEpIr4e\nEVOryiJJUqerpLAjYhZwHfCOzDwT6AI+WEUWSZJK0FXR6+4FXgJmRMTLwKuBnRVlkSSp41VyhJ2Z\ne4DlwDZgB/CrzFxXRRZJkkpQyRF2RMwG/hqYBTwLrIqIazJz5chte3t7B5drtRq1Wm2CUkqS1F71\nep16vd7StpGZ7U0z2otGXAnMz8zrGvf/FHhnZl4/YrusIp8kSVWICDIzRnusqlniPwPOj4hpERHA\nHwKbK8oiSVLHq+oc9o+BrwE/BH4MBPDlKrJIklSCSobEW+WQuCRpMunEIXFJkjQOFrYkSQWwsCVJ\nKoCFLUlSASxsSZIKYGFLklQAC1uSpAJY2JIkFcDCliSpABa2JEkFsLAlSSqAhS1JUgEsbEmSCmBh\nS5JUAAtbkqQCWNiSJBXAwpYkqQAWtiRJBbCwJUkqgIUtSVIBLGxJkgpgYUuSVAALW5KkAljYkiQV\nwMKWJKkAFrYkSQWwsCVJKoCFLUlSASor7Ig4PiLujojNEfGTiHhnVVkkSep0XRW+9q3AtzLzAxHR\nBby6wiySJHW0yMyJf9GI1wKPZOabx9guq8gnSVIVIoLMjNEeq2pI/FTglxFxR0T8KCK+HBHTK8oi\nSVLHq6qwu4CzgS9m5tnAr4GlFWWRJKnjVXUOezvwVGb+oHF/FXDTaBv29vYOLtdqNWq1WruzSZI0\nIer1OvV6vaVtKzmHDRAR3wWuy8z/jIhlwKsz86YR23gOW5I0aTQ7h11lYZ8FfAU4DngCWJSZz47Y\nxsKWJE0ah13Yjdncv5OZj49Yf2Zmbjq6MUd9fQtbkjRpHNYs8Yi4EugH/qXxxSbnDnl4xdGNKEmS\nmmk2S/zjwB9k5tuBRcA/RcTCxmOjtr8kSWqPZrPE/0dmPg2QmRsj4lLg3yLiTYDj1JIkTaBmR9jP\nRcTgN5E1yrsGXAGc0eZckiRpiGZH2B9mxNB3Zj4XEe8GrmxrKkmSNExlH+tqhbPEJUmTSSd+l7gk\nSRoHC1uSpAJY2JIkFWDMwo6I90XEIxHxTETsjYjnImLvRISTJEkDxpx0FhFbgD8GHpvoGWBOOpMk\nTSZHOulsO/AfNqckSdVp5XrYNwH3RkQdePG3KzPzf7crlCRJGq6Vwv574HlgGjC1vXEkSdJoWins\nkzJzbtuTSJKkQ2rlHPa3IqKn7UkkSdIhtTJL/DlgBgPnr/cz8P3imZmvbXs4Z4lLkiaRZrPEmw6J\nR0QAZ2TmtrYkkyRJLWk6JN44vF0zQVkkSdIhtHIO+0cRcW7bk0iSpENq5Rx2P/AWYCvwAq+cwz6z\n7eE8hy1JmkQO+xx2w4KjnEeSJI3TmEPimbkVeBNwWWP51608T5IkHT2tDIkvA84BTsvM34uIk4C7\nM3Ne28M5JC5JmkSO9OIfC4H3M3D+mszcCbzm6MWTJEljaaWwX2oc5iZARMxobyRJkjRSK4X9zxFx\nG/A/I+I6YB3wlfbGkiRJQ415DhsgIuYDPQx8pKsvM+9rd7DG63oOW5I0aTQ7h93KpLNbMvOmsdYd\nZrApwA+A7Zn5/lEet7AlSZPGkU46mz/KuvccWaRBNwA/PUr/liRJx6xDFnZEfDgiHgNOi4hNQ25P\nApuO9IUj4mTgvXg+XJKkMTX7prOVwL3Ap4GlQ9Y/l5nPHIXX/gfgRuD4o/BvSZJ0TDtkYWfms8Cz\nwNVH+0Uj4nJgd2Y+GhE1BiazSZKkQ2jlu8TbYR7w/oh4LzAdeE1EfC0z/2zkhr29vYPLtVqNWq02\nURklSWqrer1OvV5vaduWPtbVThFxCbDEWeKSpMnuSGeJS5KkilV+hN2MR9iSpMnEI2xJkgpnYUuS\nVAALW5KkAljYkiQVwMKWJKkAFrYkSQWwsCVJKoCFLUlSASxsSZIKYGFLklQAC1uSpAJY2JIkFcDC\nliSpABa2JEkFsLAlSSqAhS1JUgEsbEmSCmBhS5JUAAtbkqQCWNiSJBXAwpYkqQAWtiRJBbCwJUkq\ngIUtSVIBLGxJkgpgYUuSVAALW5KkAljYkiQVwMKWJKkAlRR2RJwcEfdHxE8i4rGI+GgVOSRJKkVk\n5sS/aMQbgDdk5qMRMRP4IXBFZvaP2C6ryCdJUhUigsyM0R6r5Ag7M3dl5qON5eeBzcAbq8giSVIJ\nKj+HHRG/C7wd+Pdqk0iS1Lm6qnzxxnD4KuCGxpH2QXp7eweXa7UatVptQrJJktRu9Xqder3e0raV\nnMMGiIgu4N+AezPz1kNs4zlsSdKk0ewcdpWF/TXgl5n5N022sbAlSZNGxxV2RMwDHgQeA7Jx+3hm\nfnvEdha2JGnS6LjCbpWFLUmaTDruY12SJGl8LGxJkgpgYUuSVAALW5KkAljYkiQVwMKWNKY1a9Yw\ne/bsg9ZfeOGFrFy5soJE0uRjYUsa01133UV3d/ewdf39/Tz88MOceOKJFaWSJhcLW9KY6vU6l156\n6bB1Dz74IFOnTuWCCy6oKJU0uVjYkprasmULO3bsOKiw169fz7nnnsu0adOGrX/++ef5wAc+wPbt\n2ycypnTMs7AlNfXAAw/Q1dXFRRddNGz9+vXrufjii4etu/3221m+fDnf+MY3ePnllycypnTMs7Al\nNVWv15k7dy4zZ84cXLd161a2bdvGJZdcwoEDB9i8eTMA1157LcuWLcOvFJaOPgu7RX190NMzcOvr\nqzqN1B59fX309PTQ09NDX+MXvV6vM2fOnIO2mzJlCvPmzWPjxo088sgjVcSVDkup7+ddVQcoQV8f\nLFwI+/YN3N+wAVavhgULqs0lHU19fX0sXLiQfY1f9A0bNvD5z3+ep59+mp07dw5u19/fz4oVK+ju\n7mbGjBmsWbOGG2+8sarY0riU/H7u1bpa0NMD9903fN38+bB2bTV5pHbo6enhvhG/6KeffjqPP/44\nV199NRFBd3c3J5xwAtdffz1XXXUVJ598MnPnzmXx4sXDnjdlyhR+/vOfc8opp0zkjyCNqdPfz5td\nrcsjbEmH9Mwzz3DOOeewYsWKgx675557Jj6QNIl5DrsFS5bA9Omv3J8+fWCddCxZsmQJ04f8ok+f\nPp39+/dz2WWXVZhKOrpKfj+3sFuwYMHAOY758wdupZzvkMZjwYIFrF69mvnz5zN//nw+97nPsWfP\nnoM+f93MypUrWbx4MRHB0qVL+dKXvtTGxNL4lfx+7jlsSaNau3YtH/nIR9i0aROvetWrqo4jTQrN\nzmFb2JIkdYhmhe2QuCRJBbCwJUkqgIUtSVIBLGxJkgpgYUuSVAALW9KY1qxZw+zZsw9af+GFF7Jy\n5coKEkmTj4UtaUx33XUX3d3dw9b19/fz8MMPc+KJJ1aUSppcLGxJY6rX6wd949mDDz7I1KlTueCC\nCypKJU0uXvxDUlNbtmxhx44dBxX2+vXrOffcc5k2bdrguo0bN7Jhwwb27t3L9773PT7xiU9w8cUX\nT3Rk6ZhUWWFHxLuBzzJwlH97Zt5SVRZJh/bAAw/Q1dXFRRddNGz9+vXr+dCHPjR4f9++fXzzm9/k\nU5/6FACrVq3iPe95D1u2bHHYXDoKKhkSj4gpwBeABcAZwNUR8ftVZBmXVasGbtIxqq+vj56eHnp6\neujr6wMGhsPnzp3LzJkzB7fbunUr27Zt45JLLuHAgQNs3ryZLVu2cMstt/DEE08AAxcT2bdvHw89\n9FAlP4vUVIHv51UdYZ8H/FdmbgWIiLuAK4D+ivKMbd8+uO66geX3vQ+GDANKx4K+vj4WLlzIvn37\nANiwYQOrV6+mXq/zrne966Btp0yZwrx589i4cSNPPvkk11xzDQ899NDgbPKnnnqKiGDOnDkT/rNI\nTRX6fl7VpLM3Ak8Nub+9sa5z3XYb7N8/cLvttqrTSEfd8uXLB8saBoa4P/nJT/L000+zc+fOwfX9\n/f2sWLGC7u5uZsyYwZo1a7j88ssBOP/88we3u/nmm1myZAlnnXXWxP0QUisKfT/v+Elnvb29g8u1\nWo1arTbxIfbtg7/7O3jhhd+Ggr/6q2L+KpMO1549ezjuuON485vfzKJFi+ju7uaEE07gO9/5Dldd\ndRWLFy9m7ty5HH/88cOe99WvfpWTTjqJm2++uaLk0iF02Pt5vV6nXq+3tG0ll9eMiPOB3sx8d+P+\nUiBHTjzrmMtrfvazcNNN8NJLA/enToXPfAZuuKHaXNJRNHJIfPr06Zx33nns379/XOeh16xZwy9+\n8QsWLVrEiy++yK5du5g1a1a7Ykvj0+Hv580ur1nVEfb3gbdExCzgaeCDwNUVZRnbiy/C2942fN1v\nflNNFqlNFixYwOrVq1m+fDkAS5YsYdGiRVx77bUt/xvf/e532b17N5dffjm7du0a/GIVC1sdo+D3\n80qOsGHwY1238srHug4aO+uYI2xpEurv7+etb30r69at47LLLhtz+yeffJKzzjqLFxpDjZlJRPDs\ns88Om2Eu6dA68QibzPw2cFpVry+puW3btjFnzhzmzZvX0vannnoqe/fubXMqafKq7Ai7FR5hS5Im\nk2ZH2H6XuCRJBbCwJUkqgIUtSVIBLGxJkgpgYUuSVAALW5KkAljYkiQVwMKWJKkAFrYkSQWwsMep\n1cugyX3VKvdTa9xPrXNftaa0/WRhj1Np/4Gr5L5qjfupNe6n1rmvWlPafrKwJUkqgIUtSVIBOv5q\nXVVnkCRpIh3qal0dXdiSJGmAQ+KSJBXAwpYkqQAW9jhExLsjoj8i/jMibqo6TyeKiJMj4v6I+ElE\nPBYRH606UyeLiCkR8aOIuKfqLJ0sIo6PiLsjYnPjd+udVWfqRBHxscb+2RQRX4+IqVVn6hQRcXtE\n7I6ITUPWvS4i1kbEzyKiLyKOrzLjWCzsFkXEFOALwALgDODqiPj9alN1pP8G/iYzzwAuAP6X+6mp\nG4CfVh2iALcC38rM04GzgM0V5+k4ETELuA54R2aeCXQBH6w2VUe5g4H376GWAusy8zTgfuBjE55q\nHCzs1p0H/Fdmbs3M/cBdwBUVZ+o4mbkrMx9tLD/PwBvrG6tN1Zki4mTgvcBXqs7SySLitcBFmXkH\nQGb+d2burThWJ9oLvATMiIgu4NXAzmojdY7M3ADsGbH6CuDOxvKdwB9NaKhxsrBb90bgqSH3t2MR\nNRURvwu8Hfj3apN0rH8AbgT8qEZzpwK/jIg7GqcPvhwR06sO1Wkycw+wHNgG7AB+lZnrqk3V8V6f\nmbth4GADeH3FeZqysNUWETETWAXc0DjS1hARcTmwuzEaEY2bRtcFnA18MTPPBn7NwFCmhoiI2cBf\nA7OAk4CZEXFNtamK09F/PFvYrdsBnDLk/smNdRqhMRy3CvinzPzXqvN0qHnA+yPiCeD/AJdGxNcq\nztSptgNPZeYPGvdXMVDgGu4c4KHMfCYzDwDfAC6sOFOn2x0RJwBExBuAX1ScpykLu3XfB94SEbMa\nMy8/CDizd3RfBX6ambdWHaRTZebHM/OUzJzNwO/S/Zn5Z1Xn6kSNIcunIuL3Gqv+ECfqjeZnwPkR\nMS0igoH95OS84UaOZt0D/EVj+c+Bjj7A6Ko6QCky80BEXA+sZeAPndsz0/8ZRoiIecCHgMci4hEG\nhpg+npnfrjaZCvdR4OsRcRzwBLCo4jwdJzN/3Bil+SFwAHgE+HK1qTpHRKwEakB3RGwDlgE3A3dH\nxF8CW4Erq0s4Nr+aVJKkAjgkLklSASxsSZIKYGFLklQAC1uSpAJY2JIkFcDCliSpABa2dAxqXI7y\nwxP4evdGxB4vEyq1j4UtHZteBywe75Ma35B1OD4D/MlhPldSCyxs6dj0aWB24+pWtwBExN9GxMaI\neDQiljXWzYqI/oi4MyIeA94UEc9FxGci4j8iYm1EvDMi6hGxJSLeN9qLZeYDgBd5kdrIwpaOTUuB\nxzPz7My8KSLmA3My8zzgHcA5EfGuxrZvAb6QmW/LzG3ADGBdZs5loIQ/CVwG/DHw9xP+k0gC/C5x\nabLoAeZHxI8YuPjBDGAOA9d435qZ3x+y7YuZubax/Bjwm8x8uXEEPmsiQ0t6hYUtTQ4BfDoz/3HY\nyohZwAsjtt0/ZPll4EWAzMzGpVMlVcAhcenY9BzwmiH3+4C/jIgZABFxUkT8TuOxkRPNmk08G+ux\nw520JmkM/rUsHYMy85mIeCgiNgH3Ns5jnw78v8ZE8OcYmNX9MgOXQB329Gb/9GgrI+JB4DRgZuPS\nhddm5n1H+nNIeoWX15QkqQAOiUuSVAALW5KkAljYkiQVwMKWJKkAFrYkSQWwsCVJKoCFLUlSASxs\nSZIK8P8B+U55dyp0QssAAAAASUVORK5CYII=\n", + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "import matplotlib.pyplot as plt\n", + "%matplotlib inline\n", + "\n", + "plt.figure(figsize=(8,4))\n", + "plt.scatter([0,0], [0, 1], color='b')\n", + "plt.scatter([10,10], [0, 1], color='r', marker='^')\n", + "plt.xlim(-1, 11)\n", + "plt.ylim(-1, 11)\n", + "plt.xlabel('term 1')\n", + "plt.ylabel('term 2')\n", + "plt.show()\n", + "\n", + "plt.figure(figsize=(8,4))\n", + "plt.scatter([0,0], [0, 1], color='b')\n", + "plt.scatter([10,10], [0, 1], color='r', marker='^')\n", + "plt.annotate('$\\mu_1$', xy=(5,1), size=20)\n", + "plt.scatter([5], [1], color='k')\n", + "plt.annotate('$\\mu_2$', xy=(5,0), size=20)\n", + "plt.scatter([5], [0], color='k')\n", + "plt.xlim(-1, 11)\n", + "plt.ylim(-1, 11)\n", + "plt.xlabel('term 1')\n", + "plt.ylabel('term 2')\n", + "plt.show()\n", + "\n", + "plt.figure(figsize=(8,4))\n", + "plt.scatter([0,10], [1, 1], color='b')\n", + "plt.scatter([0,10], [0, 0], color='r', marker='^')\n", + "plt.annotate('$\\mu_1$', xy=(5,1), size=20)\n", + "plt.scatter([5], [1], color='k')\n", + "plt.annotate('$\\mu_2$', xy=(5,0), size=20)\n", + "plt.scatter([5], [0], color='k')\n", + "plt.xlim(-1, 11)\n", + "plt.ylim(-1, 11)\n", + "plt.xlabel('term 1')\n", + "plt.ylabel('term 2')\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**Simple solution**\n", + "- Run K-Means many times with different initialization\n", + " - pick one with lowest cost\n", + " \n", + "OR\n", + "\n", + "- Pick initial means to be well-dispersed in data\n", + " - E.g., Each mean is set to an some existing point\n", + " - To pick new mean, pick a point $x$ that is far away from all other previously selected points\n", + " " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "**How to pick K?**\n", + "\n", + "



\n", + "Look for the knee!\n", + "![knee](images/knee.png)\n", + "\n", + "E.g., at K=4 and K=9" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "How do we know k-means actually minimizes the error function?\n", + "\n", + "We can compute gradient w.r.t. $M$, just like we did for logistic regression.\n", + "\n", + "$$E(D, M) = \\sum_{i=1}^N \\sum_{j=1}^K r_{ij} d(x_i, \\mu_j)$$\n", + "\n", + "$$\\frac{\\partial E(D, M)}{\\partial \\mu_{jp}} = \\sum_{x_i \\in C_j} 2(\\mu_{jp} - x_{ip}) $$\n", + "where $C_j$ is the set of documents assigned to cluster $j$\n", + "\n", + "Setting this derivative to zero, we get the following update:\n", + "\n", + "$$\\mu_{jp} = \\frac{1}{|C_j|} \\sum_{x_i \\in C_j} x_{ip}$$\n", + "\n", + "which is just the average frequency of term $p$ for documents in cluster $j$." + ] + }, + { + "cell_type": "code", + "execution_count": 43, + "metadata": { + "collapsed": false + }, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEACAYAAACwB81wAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAE6NJREFUeJzt3HuQXOWZ3/HvA7KQuAQHsSggMCkjHMdcgrcwERijBu9K\ns+tssENswFuEVQErF4ZQFVUijLOlwZWL9YfKl6I2HmkVIC6rZJcMtQKCRhjUC1qWXcU2YK9nLIzL\nAglZsbN4Y8EYC/Hkj25PBjGXHvrM9Ey/309Vl/qc8/R5n1N99Jszb/eZyEwkSd3vqE43IEmaHga+\nJBXCwJekQhj4klQIA1+SCmHgS1IhKgn8iNgYEQci4tkxtn8yIp5pPnZGxHlVjCtJal1VV/h3A8vH\n2f5j4LLM/GfAfwI2VDSuJKlFc6rYSWbujIgzx9n+1IjFp4BFVYwrSWpdJ+bwbwQe7sC4klS0Sq7w\nWxURlwMrgEunc1xJ0jQGfkScD6wHejLz5XHq/OM+kjRJmRkT1VQ5pRPNx1s3RLwL+CZwXWY+P9GO\nMrMrH2vWrOl4Dx6fx+fxdd+jVZVc4UfEJqAGLIiIF4A1wNxGdud64E+Ak4A/jYgADmXmRVWMLUlq\nTVXf0vnkBNtvAm6qYixJ0tvjnbbTqFardbqFKeXxzW4eX/eLycz/TIeIyJnWkyTNZBFBTvOHtpKk\nGczAl6RCGPiSVAgDX5IKYeBLUiEMfEkqhIEvSYUw8CWpEAa+JBXCwJekQhj4klQIA1+SCmHgS1Ih\nDHxJKoSBL0mFMPAlqRAGviQVwsCXpEIY+JJUCANfkgph4EtSISoJ/IjYGBEHIuLZcWq+HBHPRcTT\nEXFBFeNKklpX1RX+3cDysTZGxO8BZ2Xm2cBK4CsVjTtrbNlyH1u23NfpNqS36O/vZ9myq1i27Cr6\n+/s73Y6mUGRmNTuKOBN4IDPPH2XbV4Admfn15vIAUMvMA6PUZlU9zRRDQ0OcdtpiAPbvf5558+Z1\nuCOpob+/n4997HqGhtYCMH/+au6//16WLx/z+k0zUESQmTFR3XTN4S8CXhyxvK+5rgh9fRs4dOhC\nDh26kL6+DZ1uRxq2bt36ZthfDzSCf9269Z1uS1NkTqcbGE1vb+/w81qtRq1W61gv7RoaGuLOO9fy\nyisPANDb+wesXHmTV/mS3rZ6vU69Xp/066Yr8PcBZ4xYPr25blQjA3+26+vbwKuvngT8HQCvvnoS\nfX0buO22WzvbmASsWvXH7Nx5PUNDjeX581ezatW9nW1KEzryQvjOO+9s6XVVBn40H6PZCnwa+HpE\nLAF+Mdr8fTd67bXDnHfeQuDzzTUL+dWvXu9kS9Kw5cuXc//99w5P46xa5fx9N6vkQ9uI2ATUgAXA\nAWANMBfIzFzfrLkL6AFeAVZk5nfG2FfXfWgrSVOp1Q9tK/uWTlUMfEmanJn2LR1JUocZ+JJUCANf\nkgph4EtSIQx8SSqEgS9JhTDwJakQBr4kFcLAl6RCGPiSVAgDX5IKYeBLUiEMfEkqhIEvSYUw8CWp\nEAa+JBXCwJekQhj4klQIA1+SCmHgS1IhDHxJKoSBL0mFMPAlqRCVBH5E9ETEYETsjojVo2xfEBEP\nR8TTEfG9iPijKsaVJLUuMrO9HUQcBewGPgy8BOwCrsnMwRE1a4B5mfmZiDgZ+CGwMDNfH2V/2W5P\nklSSiCAzY6K6Kq7wLwKey8w9mXkI2AxceUTNT4ETms9PAP7PaGEvSZo6cyrYxyLgxRHLe2n8EBhp\nA/BoRLwEHA9cXcG4kqRJqCLwW/EZ4JnMvDwizgIeiYjzM/PgaMW9vb3Dz2u1GrVabVqalKTZoF6v\nU6/XJ/26KubwlwC9mdnTXL4dyMxcO6LmfwL/OTP/srn8KLA6M//XKPtzDl+SJmE65/B3AYsj4syI\nmAtcA2w9omYA+J1mYwuB9wA/rmBsSVKL2p7SyczDEXELsJ3GD5CNmTkQESsbm3M98F+BuyPiGSCA\n/5CZf9fu2JKk1rU9pVM1p3QkaXKmc0pHkjQLGPiSVAgDX5IKYeBLUiEMfEkqhIEvSYUw8CWpEAa+\nJBXCwJekQhj4klQIA1+SCmHgS1IhDHxJKoSBL0mFMPAlqRAGviQVwsCXpEIY+JJUCANfkgph4EtS\nIQx8SSqEgS9JhTDwJakQlQR+RPRExGBE7I6I1WPU1CLiuxHx/YjYUcW4kqTWRWa2t4OIo4DdwIeB\nl4BdwDWZOTii5kTgSWBZZu6LiJMz8+dj7C/b7UmSShIRZGZMVFfFFf5FwHOZuSczDwGbgSuPqPkk\n8M3M3AcwVthLkqZOFYG/CHhxxPLe5rqR3gOcFBE7ImJXRFxXwbiSpEmYM43j/DZwBXAc8FcR8VeZ\n+aPRint7e4ef12o1arXaNLQoSbNDvV6nXq9P+nVVzOEvAXozs6e5fDuQmbl2RM1qYF5m3tlc/jPg\n4cz85ij7cw5fkiZhOufwdwGLI+LMiJgLXANsPaLmz4FLI+LoiDgW+OfAQAVjS5Ja1PaUTmYejohb\ngO00foBszMyBiFjZ2JzrM3MwIvqBZ4HDwPrM/EG7Y0uSWtf2lE7VnNKRpMmZzikdSdIsYOBLUiEM\nfEkqhIEvSYUw8CWpEAa+JBXCwJekQhj4klQIA1+SCmHgS1IhDHxJKoSBL0mFMPAlqRAGviQVwsCX\npEIY+JJUCANfkgph4EtSIQx8SSqEgS9JhTDwJakQBr4kFcLAl6RCVBL4EdETEYMRsTsiVo9T94GI\nOBQR/6qKcSVJrWs78CPiKOAuYDlwDnBtRLx3jLrPA/3tjilJmrwqrvAvAp7LzD2ZeQjYDFw5St2t\nwBbgf1cwpiRpkqoI/EXAiyOW9zbXDYuI04CPZuZ/A6KCMSVJkzRnmsb5IjBybn/c0O/t7R1+XqvV\nqNVqU9KUJM1G9Xqder0+6ddFZrY1cEQsAXozs6e5fDuQmbl2RM2Pf/MUOBl4BfjjzNw6yv6y3Z4k\nqSQRQWZOOHtSReAfDfwQ+DCwH/gb4NrMHBij/m7ggcy8b4ztBr4kTUKrgd/2lE5mHo6IW4DtND4T\n2JiZAxGxsrE51x/5knbHlCRNXttX+FXzCl+SJqfVK3zvtJWkQhj4klQIA1+SCmHgS1IhDHxJKoSB\nL0mFMPAlqRAGviQVwsCXpEIY+JJUCANfkgph4EtSIQx8SSqEgS9JhTDwJakQBr4kFcLAl6RCGPiS\nVAgDX5IKYeBLUiEMfEkqhIEvSYUw8CWpEJUEfkT0RMRgROyOiNWjbP9kRDzTfOyMiPOqGFeS1LrI\nzPZ2EHEUsBv4MPASsAu4JjMHR9QsAQYy8+8jogfozcwlY+wv2+1JkkoSEWRmTFRXxRX+RcBzmbkn\nMw8Bm4ErRxZk5lOZ+ffNxaeARRWMK0mahCoCfxHw4ojlvYwf6DcCD1cwriRpEuZM52ARcTmwArh0\nvLre3t7h57VajVqtNqV9SdJsUq/Xqdfrk35dFXP4S2jMyfc0l28HMjPXHlF3PvBNoCcznx9nf87h\nS9IkTOcc/i5gcUScGRFzgWuArUc08y4aYX/deGEvSZo6bU/pZObhiLgF2E7jB8jGzByIiJWNzbke\n+BPgJOBPIyKAQ5l5UbtjS5Ja1/aUTtWc0pGkyZnOKR1J0ixg4EtSIQx8SSqEgS9JhTDwJakQBr4k\nFcLAl6RCGPiSVAgDX5IKYeBLUiEMfEkqhIEvSYUw8CWpEAa+JBXCwJekQhj4klQIA1+SCmHgS1Ih\nDHxJKoSBrynz0EMP8e53v/st6y+55BI2bdrUgY6k/6/E89PA15TZvHkzCxYseNO6wcFBnnrqKU49\n9dQOdSU1lHh+GviaMvV6ncsvv/xN6x5//HHmzp3LxRdf3KGupIYSz08DX1PiRz/6Efv27XvLf6gn\nnniCD3zgA8ybN2943be//W1uu+02vvrVr/KpT32K559/frrbVWEmc34CHDx4kI9//OPs3bt3Otus\nXCWBHxE9ETEYEbsjYvUYNV+OiOci4umIuKCKcTVz7dixgzlz5vChD33oTeufeOIJLrvssuHlX//6\n11x11VV89rOf5brrruPGG2/k2muvne52VZhWz0+AjRs3sm7dOu677z7eeOON6Wyzcm0HfkQcBdwF\nLAfOAa6NiPceUfN7wFmZeTawEvhKu+PONlu23MeWLfd1uo0p0d/fz7Jly1i2bBn9/f1A49flc889\nl+OPP364bs+ePbzwwgssXbqUw4cPMzAwwOOPP84JJ5zAKaecAsCFF17IwMAAP/nJTzpxKGPq/vfv\nKpYtu2r4/esm7ZyfADfccANr1qwhMzvSf6Uys60HsAR4eMTy7cDqI2q+Alw9YnkAWDjG/rLbvPrq\nq/nOd56W73znaTk0NNTpdiq1bdu2nD9/fgIJ5Pz583Pbtm152mmn5Sc+8Yk31fb19eXRRx+dBw8e\nzCeffDK/9rWv5YYNG3Lp0qVvqjvjjDPywQcfnMajGF/3v38LE+5JuCfnz1+Y27Zt63RblWn3/Bwp\nInLPnj3T2X7Lmrk5YV5XMaWzCHhxxPLe5rrxavaNUtO1+vo2cOjQhRw6dCF9fRs63U6l1q1bx9DQ\n0PDy0NAQn/vc59i/fz8vvfTS8PrBwUHuueceFixYwHHHHcdDDz3ERz7yEX7+859z7LHHvmmf8+bN\n45e//OW0HcNEuvv9W8/Q0FrgeuB6hobWsm7d+k63VZl2z89uM6fTDYymt7d3+HmtVqNWq3Wsl3YN\nDQ1x551reeWVBwDo7f0DVq686S0fCnWTl19+mXe84x2cddZZrFixggULFrBw4UIeffRRrr76am6+\n+WbOPfdcTjzxRE488cS3/Kp88OBBTj755A51/2Ylvn/dbjLn50xVr9ep1+uTf2ErvwaM96AxpbNt\nxHIrUzqDFDKl84UvfCnnzj034ZGER3Lu3HPzi1/8cqfbqsxovzIvXbo0L7nkkpZe/9hjj+UFF1ww\nvPz666/nMccck7t3756qlieljPevrCmdyZyfI3XDlE4VV/i7gMURcSawH7gGOPJrFluBTwNfj4gl\nwC8y80AFY894r712mPPOWwh8vrlmIb/61eudbKlSy5cv5/7772fdunUArFq1ihUrVnDDDTe09PrL\nLruMn/3sZ+zdu5fTTz+der3OOeecw9lnnz2VbbesjPfv3uFpnFWr7mX58uUd7qo67Z6f3Saygk+e\nI6IH+BKNb/1szMzPR8RKGj911jdr7gJ6gFeAFZn5nTH2lVX0pM4YHBzkfe97H9/61re44oorWnrN\njh07+MY3vsHFF19MvV7njjvuYPHixVPcqUr0ds7PTZs2sXPnTvr6+rj66qu59NJLufnmm6e408mJ\nCDIzJqybaeFq4M9u27dv59Zbb+XZZ5/lmGOO6XQ70pt06/lp4EtSIVoNfP+0giQVwsCXpEIY+JJU\nCANfkgph4EtSIQx8SSqEgS9JhTDwJakQBr4kFcLAl6RCGPiSVAgDX5IKYeBLUiEMfEkqhIEvSYUw\n8CWpEAa+JBXCwJekQhj4klQIA1+SCmHgS1Ih2gr8iPiHEbE9In4YEf0RceIoNadHxGMR8bcR8b2I\n+LftjClJenvavcK/HfhWZv4T4DHgM6PUvA78u8w8B7gY+HREvLfNcWeler3e6RamlMc3u3l83a/d\nwL8SuLf5/F7go0cWZOZPM/Pp5vODwACwqM1xZ6VuP+E8vtnN4+t+7Qb+KZl5ABrBDpwyXnFE/GPg\nAuCv2xxXkjRJcyYqiIhHgIUjVwEJ/MdRynOc/RwPbAFua17pS5KmUWSOmdETvzhiAKhl5oGI+EfA\njsz8p6PUzQEeBB7OzC9NsM+335AkFSozY6KaCa/wJ7AV+CNgLXA98Odj1P134AcThT201rQkafLa\nvcI/CfgGcAawB/hEZv4iIk4FNmTmv4iIDwKPA9+jMeWTwB2Zua3t7iVJLWsr8CVJs8eMu9M2Iv51\nRHw/Ig5HxG93up+qRERPRAxGxO6IWN3pfqoUERsj4kBEPNvpXqZCN988GBHHRMRfR8R3m8f3Xzrd\n01SIiKMi4jsRsbXTvVQtIn4SEc8038O/Ga92xgU+jamfjwF/0elGqhIRRwF3AcuBc4Bru+zms7tp\nHFu36tqbBzPzNeDyzHw/cD5wRXMattvcBvyg001MkTdofHnm/Zl50XiFMy7wM/OHmfkcja9/douL\ngOcyc09mHgI207hprStk5k7g5U73MVW6/ebBzHy1+fQYGpnQVe9lRJwO/D7wZ53uZYoELWb5jAv8\nLrUIeHHE8l66KDBK0o03DzanO74L/BSoZ2a3XQl/Afj3jHOf0CyXwCMRsSsibhqvsN2vZb4t49zM\n9dnMfKATPUkT6dabBzPzDeD9EfEPgO0RsTQzu2JKNSI+AhzIzKcjokZ3zRz8xgczc39E/BaN4B9o\n/tb9Fh0J/Mz83U6M20H7gHeNWD69uU6zRPPmwS3AVzNzrPtNZrXM/L8R8RBwId3zGdoHgX8ZEb8P\nzAdOiIj/kZn/psN9VSYz9zf//VlE3E9jCnnUwJ/pUzrd8tN4F7A4Is6MiLnANTRuWusmQfe8X6Np\n+ebB2SQiTv7NnzWPiPnA7wJPd7ar6mTmHZn5rsx8N43/d491U9hHxLHN3zyJiOOAZcD3x6qfcYEf\nER+NiBeBJcCDEfFwp3tqV2YeBm4BtgN/C2zOzIHOdlWdiNgEPAm8JyJeiIgVne6pSs1vrfwhjW+w\nfLf59b6eTvdVkVOBHc05/KeArZn5aId7UusWAjtHvH8PZOb2sYq98UqSCjHjrvAlSVPDwJekQhj4\nklQIA1+SCmHgS1IhDHxJKoSBL0mFMPAlqRD/D+Qfjx3sp5PIAAAAAElFTkSuQmCC\n", + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# Here's an example. \n", + "# Exercise: work out the means/cluster assignments\n", + "# until convergence.\n", + "def plot(points, cluster_assignments, means):\n", + " plt.figure()\n", + " for point, asg in zip(points, cluster_assignments):\n", + " plt.scatter([point[0]], [point[1]], marker='o' if asg==1 else '^')\n", + " for i, m in enumerate(means):\n", + " plt.annotate('$\\mu_%d$' % i, xy=m, size=20)\n", + " plt.scatter([m[0]], [m[1]], color='k')\n", + " plt.show()\n", + " \n", + "plot([(0, 0), (0, 1), (2, 0), (3,1), (3,0)], [0,0,0,1,1], [(1,0), (4,0)])" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.5.0" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/lectures/lec19/images/clustering.gif b/lectures/lec19/images/clustering.gif new file mode 100644 index 0000000..e1e9703 Binary files /dev/null and b/lectures/lec19/images/clustering.gif differ diff --git a/lectures/lec19/images/hclust.png b/lectures/lec19/images/hclust.png new file mode 100644 index 0000000..e39e20c Binary files /dev/null and b/lectures/lec19/images/hclust.png differ diff --git a/lectures/lec19/images/kmeans.png b/lectures/lec19/images/kmeans.png new file mode 100644 index 0000000..a54699a Binary files /dev/null and b/lectures/lec19/images/kmeans.png differ diff --git a/lectures/lec19/images/kmeansalg.png b/lectures/lec19/images/kmeansalg.png new file mode 100644 index 0000000..6469adb Binary files /dev/null and b/lectures/lec19/images/kmeansalg.png differ diff --git a/lectures/lec19/images/knee.png b/lectures/lec19/images/knee.png new file mode 100644 index 0000000..c26b9a3 Binary files /dev/null and b/lectures/lec19/images/knee.png differ diff --git a/lectures/lec19/images/nonconvex.png b/lectures/lec19/images/nonconvex.png new file mode 100644 index 0000000..896db0e Binary files /dev/null and b/lectures/lec19/images/nonconvex.png differ diff --git a/lectures/lec19/images/soft.png b/lectures/lec19/images/soft.png new file mode 100644 index 0000000..4631c90 Binary files /dev/null and b/lectures/lec19/images/soft.png differ