Universal Approximation Theorem.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While studying machine learning theory over the past year or so, I came across an interesting theorem that really sparked my interest. The [Universal Approximation Theorem](https://en.wikipedia.org/wiki/Universal_approximation_theorem) states that any feed forward neural network with a *single* hidden layer containing a finite number of units/neurons can fit any function. More specifically, we can attain any desired non-zero amount of error. In order words, we will always be able to output a prediction $ g(x) $ where $ |g(x) - f(x)| < \\epsilon $ for all values of x. $ f(x) $ is the label and $ \\epsilon $ is a very small positive non-zero number. When I read that for the first time, I almost did a double take because I was so surprised that something like that could be true. \n",
    "\n",
    "Let's think about what it really means. It's basically saying that if, for example, you're given a set of $ n $ training examples $ X $ where each $ x_i \\in X $ is some $ k $-dimensional vector and you have a set of $ n $ training labels $ Y $ where each $ y_i \\in Y $ is a label from the set {0,1} (binary classification), then you will be able to generate a neural network (with one hidden layer) that is able be trained to classify every single one of those $ n $ examples correctly. \n",
    "\n",
    "That was ridiculously interesting to me when I first saw it. \n",
    "\n",
    "And it made me think \"Well, if this theorem is true, shouldn't neural networks be perfect at pretty much any task?\". The key here, though, is that we encounter a **generalization** and **overfitting** problem when we try to fit a neural net too tightly to a particular training set, and expect the same results on the testing set. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, let's say that I want to create a neural network, with one hidden layer, that correctly classifies every example in the MNIST dataset. As you may recall, MNIST has about 55,000 images and is a 10 class classification problem. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# The classic imports\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import random\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
      "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
      "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n",
      "Shape of the input: (1, 784)\n",
      "Shape of the label: (1, 10)\n",
      "Number of total images: 55000\n"
     ]
    }
   ],
   "source": [
    "# Load in the MNIST data and visualize the dimensions\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets(\"MNIST_data/\", one_hot=True)\n",
    "# oneImage is a tuple where oneImage[0] is the image and oneImage[1] is the one-hot label\n",
    "oneImage = mnist.train.next_batch(1)\n",
    "print ('Shape of the input: ' + str(oneImage[0].shape))\n",
    "print ('Shape of the label: ' + str(oneImage[1].shape))\n",
    "print ('Number of total images: ' + str(mnist.train.images.shape[0]))\n",
    "numInputDimensions = oneImage[0].shape[1]\n",
    "totalNumTrainImages = mnist.train.images.shape[0]\n",
    "totalNumTestImages = mnist.test.images.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So now, let's just create a simple neural network with one hidden layer. This is what the architecture will look like. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![alt text](Data/FunctionApproximation.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "One of the big caveats with this theorem is that although it states that a one hidden layer neural network can approximate any function, it doesn't specify how many hidden units will be necessary to attain that 100% classification accuracy. When we think about the task of MNIST digit classification, the inputs will have 784 input features. Let's see what happens if we have a neural net with just 25 hidden units."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "numHiddenUnits = 25"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also have to define some other useful parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "numClasses = 10\n",
    "trainingIterations = 500000\n",
    "batchSize = 32"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's just create our simple neural network. If you'd like to see a more in depth tutorial on that, go ahead and take a look at my other tutorial [here](https://github.com/adeshpande3/Tensorflow-Programs-and-Tutorials/blob/master/Simple%20Neural%20Networks.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph() # Just to make sure that we start with a new graph\n",
    "X = tf.placeholder(tf.float32, shape = [None, numInputDimensions])\n",
    "y = tf.placeholder(tf.float32, shape = [None, numClasses])\n",
    "\n",
    "W1 = tf.Variable(tf.truncated_normal([numInputDimensions, numHiddenUnits], stddev=0.1))\n",
    "B1 = tf.Variable(tf.constant(0.1), [numHiddenUnits])\n",
    "W2 = tf.Variable(tf.truncated_normal([numHiddenUnits, numClasses], stddev=0.1))\n",
    "B2 = tf.Variable(tf.constant(0.1), [numClasses])\n",
    "\n",
    "hiddenLayerOutput = tf.matmul(X, W1) + B1\n",
    "hiddenLayerOutput = tf.nn.relu(hiddenLayerOutput)\n",
    "finalOutput = tf.matmul(hiddenLayerOutput, W2) + B2\n",
    "finalOutput = tf.nn.relu(finalOutput)\n",
    "\n",
    "loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels = y, logits = finalOutput))\n",
    "opt = tf.train.GradientDescentOptimizer(learning_rate = .01).minimize(loss)\n",
    "\n",
    "correctPrediction = tf.equal(tf.argmax(finalOutput,1), tf.argmax(y,1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correctPrediction, \"float\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is just a quick helper function to test how our network is doing throughout the training. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def testOnDataset(imageDataset, labelDataset, numImages, session):\n",
    "    numCorrectClassifications = 0\n",
    "    for i in range(numImages):\n",
    "        image = imageDataset[i]\n",
    "        image = np.reshape(image, (1, numInputDimensions))\n",
    "        label = labelDataset[i]\n",
    "        label = np.reshape(label, (1, numClasses))\n",
    "        pred = session.run(correctPrediction, feed_dict={X: image, y: label})\n",
    "        if pred[0] == True:\n",
    "            numCorrectClassifications += 1\n",
    "    return numCorrectClassifications"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've defined the graph, let's start to train it the model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 0, training accuracy 0.15625\n",
      "step 20000, training accuracy 0.9375\n",
      "step 40000, training accuracy 0.9375\n",
      "step 60000, training accuracy 0.96875\n",
      "step 80000, training accuracy 1\n",
      "step 100000, training accuracy 0.9375\n",
      "step 120000, training accuracy 0.9375\n",
      "step 140000, training accuracy 0.96875\n",
      "step 160000, training accuracy 0.96875\n",
      "step 180000, training accuracy 0.96875\n",
      "step 200000, training accuracy 0.96875\n",
      "step 220000, training accuracy 1\n",
      "step 240000, training accuracy 1\n",
      "step 260000, training accuracy 1\n",
      "step 280000, training accuracy 1\n",
      "step 300000, training accuracy 1\n",
      "step 320000, training accuracy 1\n",
      "step 340000, training accuracy 0.96875\n",
      "step 360000, training accuracy 1\n",
      "step 380000, training accuracy 1\n",
      "Number of digits (in training set) classified correctly: 54571/55000\n",
      "Number of digits (in test set) classified correctly: 9652/10000\n"
     ]
    }
   ],
   "source": [
    "sess = tf.Session()\n",
    "init = tf.global_variables_initializer()\n",
    "sess.run(init)\n",
    "\n",
    "# Training the network\n",
    "halfParamCorrect = []\n",
    "for i in range(trainingIterations):\n",
    "    batch = mnist.train.next_batch(batchSize)\n",
    "    batchInput = batch[0]\n",
    "    batchLabels = batch[1]\n",
    "    _, trainingLoss = sess.run([opt, loss], feed_dict={X: batchInput, y: batchLabels})\n",
    "    if i%20000 == 0:\n",
    "        trainAccuracy = accuracy.eval(session=sess, feed_dict={X: batchInput, y: batchLabels})\n",
    "        print (\"step %d, training accuracy %g\"%(i, trainAccuracy))\n",
    "        numCorrect = testOnDataset(mnist.train.images, mnist.train.labels, totalNumTrainImages, sess)\n",
    "        halfParamCorrect.append(numCorrect)\n",
    "\n",
    "# Testing network on training set\n",
    "numCorrectClassifications = testOnDataset(mnist.train.images, mnist.train.labels, totalNumTrainImages, sess)\n",
    "print (\"Number of digits (in training set) classified correctly: %d/%d\"%(numCorrectClassifications, totalNumTrainImages))\n",
    "\n",
    "# Testing network on test set\n",
    "numCorrectClassifications = testOnDataset(mnist.test.images, mnist.test.labels, totalNumTestImages, sess)\n",
    "print (\"Number of digits (in test set) classified correctly: %d/%d\"%(numCorrectClassifications, totalNumTestImages))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, as you can see, the network was able to reach a decent accuracy, but it still isn't classifying every example in our training set correctly. Now, let's see what happens when we bump up the number of hidden units by a **lot** and create the graph again. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "numHiddenUnits = 2000 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 0, training accuracy 0.25\n",
      "step 10000, training accuracy 1\n",
      "step 20000, training accuracy 1\n",
      "step 30000, training accuracy 1\n",
      "step 40000, training accuracy 1\n",
      "step 50000, training accuracy 1\n",
      "step 60000, training accuracy 1\n",
      "step 70000, training accuracy 1\n",
      "step 80000, training accuracy 1\n",
      "step 90000, training accuracy 1\n",
      "step 100000, training accuracy 1\n",
      "step 110000, training accuracy 1\n",
      "step 120000, training accuracy 1\n",
      "step 130000, training accuracy 1\n",
      "step 140000, training accuracy 1\n",
      "step 150000, training accuracy 1\n",
      "step 160000, training accuracy 1\n",
      "step 170000, training accuracy 1\n",
      "step 180000, training accuracy 1\n",
      "step 190000, training accuracy 1\n",
      "step 200000, training accuracy 1\n",
      "step 210000, training accuracy 1\n",
      "step 220000, training accuracy 1\n",
      "step 230000, training accuracy 1\n",
      "step 240000, training accuracy 1\n",
      "step 250000, training accuracy 1\n",
      "step 260000, training accuracy 1\n",
      "step 270000, training accuracy 1\n",
      "step 280000, training accuracy 1\n",
      "step 290000, training accuracy 1\n",
      "step 300000, training accuracy 1\n",
      "step 310000, training accuracy 1\n",
      "step 320000, training accuracy 1\n",
      "step 330000, training accuracy 1\n",
      "step 340000, training accuracy 1\n",
      "step 350000, training accuracy 1\n",
      "step 360000, training accuracy 1\n",
      "step 370000, training accuracy 1\n",
      "step 380000, training accuracy 1\n",
      "step 390000, training accuracy 1\n",
      "step 400000, training accuracy 1\n",
      "step 410000, training accuracy 1\n",
      "step 420000, training accuracy 1\n",
      "step 430000, training accuracy 1\n",
      "step 440000, training accuracy 1\n",
      "step 450000, training accuracy 1\n",
      "step 460000, training accuracy 1\n",
      "step 470000, training accuracy 1\n",
      "step 480000, training accuracy 1\n",
      "step 490000, training accuracy 1\n",
      "Number of digits (in training set) classified correctly: 54993/55000\n",
      "Number of digits (in test set) classified correctly: 9826/10000\n"
     ]
    }
   ],
   "source": [
    "tf.reset_default_graph() # Just to make sure that we start with a new graph\n",
    "X = tf.placeholder(tf.float32, shape = [None, numInputDimensions])\n",
    "y = tf.placeholder(tf.float32, shape = [None, numClasses])\n",
    "\n",
    "W1 = tf.Variable(tf.truncated_normal([numInputDimensions, numHiddenUnits], stddev=0.1))\n",
    "B1 = tf.Variable(tf.constant(0.1), [numHiddenUnits])\n",
    "W2 = tf.Variable(tf.truncated_normal([numHiddenUnits, numClasses], stddev=0.1))\n",
    "B2 = tf.Variable(tf.constant(0.1), [numClasses])\n",
    "\n",
    "hiddenLayerOutput = tf.matmul(X, W1) + B1\n",
    "hiddenLayerOutput = tf.nn.relu(hiddenLayerOutput)\n",
    "finalOutput = tf.matmul(hiddenLayerOutput, W2) + B2\n",
    "finalOutput = tf.nn.relu(finalOutput)\n",
    "\n",
    "loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels = y, logits = finalOutput))\n",
    "opt = tf.train.GradientDescentOptimizer(learning_rate = .1).minimize(loss)\n",
    "\n",
    "correctPrediction = tf.equal(tf.argmax(finalOutput,1), tf.argmax(y,1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correctPrediction, \"float\"))\n",
    "\n",
    "sess = tf.Session()\n",
    "init = tf.global_variables_initializer()\n",
    "sess.run(init)\n",
    "\n",
    "doubleParamCorrect = []\n",
    "for i in range(trainingIterations):\n",
    "    batch = mnist.train.next_batch(batchSize)\n",
    "    batchInput = batch[0]\n",
    "    batchLabels = batch[1]\n",
    "    _, trainingLoss = sess.run([opt, loss], feed_dict={X: batchInput, y: batchLabels})\n",
    "    if i%10000 == 0:\n",
    "        trainAccuracy = accuracy.eval(session=sess, feed_dict={X: batchInput, y: batchLabels})\n",
    "        print (\"step %d, training accuracy %g\"%(i, trainAccuracy))\n",
    "        numCorrect = testOnDataset(mnist.train.images, mnist.train.labels, totalNumTrainImages, sess)\n",
    "        doubleParamCorrect.append(numCorrect)\n",
    "        \n",
    "# Testing network on training set\n",
    "numCorrectClassifications = testOnDataset(mnist.train.images, mnist.train.labels, totalNumTrainImages, sess)\n",
    "print (\"Number of digits (in training set) classified correctly: %d/%d\"%(numCorrectClassifications, totalNumTrainImages))\n",
    "\n",
    "# Testing network on test set\n",
    "numCorrectClassifications = testOnDataset(mnist.test.images, mnist.test.labels, totalNumTestImages, sess)\n",
    "print (\"Number of digits (in test set) classified correctly: %d/%d\"%(numCorrectClassifications, totalNumTestImages))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ah, we're reached 100% classification accuracy! This does back up the theorem that any dataset (like MNIST) can be fully fit with a single hidden layer neural network, **provided that we have a sufficiently large number of hidden units**. As you saw with the two runs of the neural networks, the first run showed that the number of hidden units wasn't large enough as we weren't able to fully fit the function. When we increased the number of hidden units, the training set accuracy got to 100%. However, the crux of this topic lies in the problem that the test accuracy of the smaller network actually turned out to be greater than the accuracy of the larger network that was able to fully fit to the dataset. This is the classic machine learning problem of **overfitting**. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's a final look at the classification accuracies as the number of training iterations increases. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAFkCAYAAACAUFlOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3X+QXWd93/H3d1fa3Xv12zZIdrGDEwWjEkqR/EOeBKdE\nqV0Hl0BJU6/xgA2UQoxH1ZTWSWpqxZ6m1JnYDtgw1JBxsGEzjCl1QASBoVCwjTVYLphadmvwT4zk\nX/JK3nt39+7u0z/OufLRRb+utHsu0nm/Zs7s3nO+e85zHu3ofvY5zzk3UkpIkiSVpa/XDZAkSdVi\n+JAkSaUyfEiSpFIZPiRJUqkMH5IkqVSGD0mSVCrDhyRJKpXhQ5IklcrwIUmSSmX4kCRJpeo6fETE\nSRFxa0Q8FxGNiPhhRKzuqLk6Ip7Ot38jIlZ2bB+MiJvyfeyOiNsj4pUdNcsi4nMRMRoROyPi0xGx\noKPm5IjYFBFjEbE9Iq6NCAOVJEm/xLp6o46IpcBdwARwHrAK+HfAzkLNFcCHgPcDZwJjwOaIGCjs\n6gbgLcA7gHOAk4Avdhzu8/n+1+W15wCfKhynD/gqMA9YC7wbuAS4uptzkiRJ5YpuPlguIj4KnJ1S\n+u0D1DwN/EVK6fr89WJgB/DulNIX8tfPAhemlL6U15wGbAPWppS2RMQq4P8Aa1JK9+c15wGbgFel\nlLZHxPnA3wEnppSey2v+DfBR4BUppamuekKSJJWi20sU/xz4QUR8ISJ2RMTWiHhfe2NEnAqsAL7Z\nXpdS2gXcC5ydrzqdbLSiWPMw8EShZi2wsx08cncCCTirUPNAO3jkNgNLgNd1eV6SJKkk87qs/1Xg\ng8BfAv+Z7LLKxyJiIqV0K1nwSGQjHUU78m0Ay4HJPJTsr2YF8ExxY0ppOiJe6KjZ13Ha237Y2fiI\nOJ7sctFjwPiBTlSSJO1lCHg1sDml9PyR7Kjb8NEHbEkpfSR//cOI+A3gA8CtR9KQkpwHfK7XjZAk\n6Sj2TrJ5mYet2/Dxc7K5GUXbgH+Rf78dCLLRjeKoxHLg/kLNQEQs7hj9WJ5va9d03v3SDxzXUXNG\nR1uWF7bty2MAt912G6tWrdpPiWbbhg0buP7663vdjEqxz8tnn5fPPi/Xtm3buPjiiyF/Lz0S3YaP\nu4DTOtadBjwOkFJ6NCK2k92h8iPYM+H0LOCmvP4+YCqvKU44PQW4J6+5B1gaEW8szPtYRxZs7i3U\n/GlEnFCY93EuMAo8uJ/2jwOsWrWK1atX76dEs23JkiX2d8ns8/LZ5+Wzz3vmiKctdBs+rgfuiog/\nAb5AFireB/zrQs0NwJUR8QhZOroGeAq4A7IJqBHxGeC6iNgJ7AY+BtyVUtqS1zwUEZuBmyPig8AA\n8HFgJKXUHtX4OlnIuDW/vffE/Fg3ppRaXZ6XJEkqSVfhI6X0g4h4O9ntrB8BHgXWp5T+tlBzbUTU\nyZ7JsRT4LnB+SmmysKsNwDRwOzAIfA24rONwFwE3kt3lMpPXri8cZyYiLgA+CdxN9jyRW4Crujkn\nSZJUrm5HPkgpfZXs4V4HqtkIbDzA9gng8nzZX82LwMUHOc6TwAUHqpEkSb9cfBS55tzw8HCvm1A5\n9nn57PPy2edHr66ecHq0yz+D5r777rvPSUqSJHVh69atrFmzBrKnj289kn058iFJkkpl+JAkSaUy\nfEiSpFIZPiRJUqkMH5IkqVSGD0mSVCrDhyRJKpXhQ5IklcrwIUmSSmX4kCRJpTJ8SJKkUhk+JElS\nqQwfkiSpVIYPSZJUKsOHJEkqleFDkiSVyvAhSZJKNa/XDTjajE+N8+jOR3l89HEmpyeZSTPMpBmm\nZ6b3fD+TZphO2evZklKatX1JktTpzae+mVcvfXUpxzJ87MPuid38ZOdPeOSFR/jJC/nX/PVTu54i\nYRCQJB1bbv+Xtxs+emXkgREu+u8X7Xm9eHAxK49bya8t+zXWvmotv7bs11h53EpevfTVDM0bor+v\nn77ooz+yr33Rt2ddX/QRxKy1LWL29qWDSyn1ts9TgqkpmJjYe3nFK2Dhwt61S9IxaTbfrw7G8NHh\n4ecf5vja8Xzloq+w8riVHF873jf9Y8X0NLz0EuzeDc1m9sbeamVfi0t7XauVvdmPj7/8tfh9+2vn\nPg60zwOtax+vuP99XW67/XZ4xzvK7z9JmiWGjw7NVpNltWWsfdXaXjfl2Dc9DaOjsHNnFgj294Zc\n/L7ZPPjSDhidS6NxZO2dPx8GB2FoaO+vAwMwb97Ly/z5e39fq2UjFe31xe2dy+Dgy0t7/53LP/pH\ns9P/ktQjho8OjVaD2rxar5vxyyml7K/x9pv5Sy/B2Fj2pt5e9vV61y544YUsZBSX0dF9/2V/MP39\n2Rt6canXX/5+wQI4+WRYtGj/S6227zDQ+X0xBPR5c5gkzQbDR4fmVJP6/HqvmzH3UsoCwfbt2fLz\nn+/9/bPPZqGhHTLagWNq6uD7HhzMwkB7WbwYli2D44+HlSuz75ctg+OOe/n7RYt+cQRhX6ME7dAg\nSTpqGT46NFqNoyt8tFqwY0cWGJ5+Gl58MQsNB1qefz4LGa3W3vtavBhWrIATT8wmNXaOHixc+Ivf\nL1iQBYz211otCwmSJO2H7xIdGq0Gtfm/JJddxsbg8cez5YknsnDRXtph45lnfvHSxdBQFiQ6l5NP\nzr4ed1wWMNpBY8WKbKkfRaFLknTUMnx0aE41WTy4uJyDTU7C//t/8Mgj8NhjLweNxx/PXj///Mu1\n/f1ZQDjppCwwnHVW9n17OfHEbFm2LLt8IUnSLynDR4dGq8HyBctnd6e7dsFDD8G2bXsvP/1pdscH\nZKMVp5wCv/IrsHo1vP3t2fe/8ivw6ldnAaO/f3bbJUlSDxg+OjRbszDh9LHHYNMm+Pu/h/vvzy6P\ntJ1yCqxaBW95C7z2tdn3r3kNLF8OPk9EklQBho8OhzXhtNWCu+/OAsemTfDgg9mkyze9CS65JAsY\nq1bBaaf5ZEpJUuUZPjoc8nM+nn02G9nYtAk2b86eWbF8OZx/PvzZn8E//aewZMncN1iSpKOM4aPD\nQZ/zsXNnFi5uuil75sUZZ8CGDdlllNWrfRCVJEkHYfjosN9bbaen4eab4cors6d8XnMNXHppNtoh\nSZIOmeGjIKW07wmn3/42rF8PP/pRFjj+/M+z214lSVLXvEZQ0JppMZ2mXw4fjz4Kf/AH8OY3Zw/g\n2rIF/vqvDR6SJB0Bw0dBo5V96mltOrLLK6tWwT33wG23ZXeznHFGj1soSdLRz8suBc1WE4D6ZRvg\n/pfgwx+GP/5jb4+VJGkWGT4K9ox87G5mTyA99dQet0iSpGOPl10KmlP5yEd9icFDkqQ50lX4iIir\nImKmY3mwo+bqiHg6IhoR8Y2IWNmxfTAiboqI5yJid0TcHhGv7KhZFhGfi4jRiNgZEZ+OiAUdNSdH\nxKaIGIuI7RFxbUQcUZhqj3zUBxYcpFKSJB2uw3mz/jGwHFiRL7/V3hARVwAfAt4PnAmMAZsjovgx\nqzcAbwHeAZwDnAR8seMYnwdWAevy2nOATxWO0wd8leyy0Vrg3cAlwNWHcT577LnsYviQJGnOHM6c\nj6mU0rP72bYeuCal9BWAiHgXsAN4G/CFiFgMvAe4MKX0nbzmUmBbRJyZUtoSEauA84A1KaX785rL\ngU0R8eGU0vZ8+2uBN6eUngMeiIiPAB+NiI0ppanDOK+XJ5waPiRJmjOHM/Lx6xHxs4j4SUTcFhEn\nA0TEqWQjId9sF6aUdgH3Amfnq04nCzzFmoeBJwo1a4Gd7eCRuxNIwFmFmgfy4NG2GVgCvO4wzgko\njHwMeXeLJElzpdvw8X2yyxvnAR8ATgX+Vz4fYwVZQNjR8TM78m2QXa6ZzEPJ/mpWAM8UN6aUpoEX\nOmr2dRwKNV3bM+F0aNHh7kKSJB1EV5ddUkqbCy9/HBFbgMeBPwQems2GzaUNGzawpOMTZ4eHh2m8\npkEkGKwbPiRJ1TUyMsLIyMhe60ZHR2dt/0f0nI+U0mhE/F9gJfBtIMhGN4qjEsuB9iWU7cBARCzu\nGP1Ynm9r13Te/dIPHNdR0/m40eWFbQd0/fXXs3r16l9Y/1ff/ytq00HUnfMhSaqu4eFhhoeH91q3\ndetW1qxZMyv7P6JbUyNiIVnweDql9CjZG/+6wvbFZPM07s5X3QdMddScBpwC3JOvugdYGhFvLBxq\nHVmwubdQ8/qIOKFQcy4wCux16283Gq0G9amABYYPSZLmSlcjHxHxF8CXyS61/APgz4AW8Ld5yQ3A\nlRHxCPAYcA3wFHAHZBNQI+IzwHURsRPYDXwMuCultCWveSgiNgM3R8QHgQHg48BIfqcLwNfJQsat\n+e29J+bHujGl1Oq6F3KNVoNaC1hUP2itJEk6PN1ednkV2TM4jgeeBb4HrE0pPQ+QUro2Iupkz+RY\nCnwXOD+lNFnYxwZgGrgdGAS+BlzWcZyLgBvJ7nKZyWvXtzemlGYi4gLgk2SjKmPALcBVXZ7PXppT\nTeotHPmQJGkOdTvhdPgQajYCGw+wfQK4PF/2V/MicPFBjvMkcMHB2tONRmuM2uSM4UOSpDnkZ7sU\nNCca1CeBupddJEmaK4aPgsb4Li+7SJI0xwwfBY3xl6hN4ciHJElzyPBR0Jwcc+RDkqQ5ZvgoaEyO\nZbfaOvIhSdKcMXwUNFveaitJ0lwzfBQ02s/5cORDkqQ5Y/goaEw3swmnjnxIkjRnDB8FzekJRz4k\nSZpjho+CRprIJpzWar1uiiRJxyzDRy6lRDO1qMcA9NktkiTNFd9lc+NT4wDU+wZ63BJJko5tho9c\no9UAoDbPSy6SJM0lw0euOdUEoN4/1OOWSJJ0bDN85PaMfMz3ThdJkuaS4SPXbOUjH4YPSZLmlOEj\n1x75qA/6gDFJkuaS4SO357LLgOFDkqS5ZPjI7ZlwWlvc45ZIknRsM3zk9ox81Bb1uCWSJB3bDB+5\nPRNOHfmQJGlOGT5yjVaDeTMwf4HhQ5KkuWT4yDWnmtmHyvmJtpIkzSnDR67RalBvAQu820WSpLlk\n+Mg1Jl5y5EOSpBIYPnLN5m5HPiRJKoHhI9do7qI2hSMfkiTNMcNHrjnxkiMfkiSVwPCRaxg+JEkq\nheEj15gcc8KpJEklMHzkmt5qK0lSKQwfuUar6YRTSZJKYPjINaeajnxIklQCw0euMTORhQ9HPiRJ\nmlOGj1xjepxa6of+/l43RZKkY5rhI9dMk9QZ6HUzJEk65hk+co3UotY/2OtmSJJ0zDN8ANMz00zG\nNPX+oV43RZKkY57hg+xOF8DwIUlSCQwfQKPVAKA2v9bjlkiSdOw7ovAREX8cETMRcV3H+qsj4umI\naETENyJiZcf2wYi4KSKei4jdEXF7RLyyo2ZZRHwuIkYjYmdEfDoiFnTUnBwRmyJiLCK2R8S1EdH1\nOTVb+cjHPG+zlSRprh12+IiIM4D3Az/sWH8F8KF825nAGLA5Ioq3ktwAvAV4B3AOcBLwxY5DfB5Y\nBazLa88BPlU4Th/wVWAesBZ4N3AJcHW357Jn5GPQ8CFJ0lw7rPAREQuB24D3AS92bF4PXJNS+kpK\n6cfAu8jCxdvyn10MvAfYkFL6TkrpfuBS4Dcj4sy8ZhVwHvDelNIPUkp3A5cDF0bEivw45wGvBd6Z\nUnogpbQZ+AhwWUTM6+Z89sz5GFjYzY9JkqTDcLgjHzcBX04pfau4MiJOBVYA32yvSyntAu4Fzs5X\nnU42WlGseRh4olCzFtiZB5O2O4EEnFWoeSCl9FyhZjOwBHhdNyfTHvmoDy3q5sckSdJh6GqEACAi\nLgT+MVmI6LSCLCDs6Fi/I98GsByYzEPJ/mpWAM8UN6aUpiPihY6afR2nve2HHKKXL7s48iFJ0lzr\nKnxExKvI5mv8bkqpNTdNKt+eCaf1xT1uiSRJx75uRz7WAK8AtkZE5Ov6gXMi4kNkczCCbHSjOCqx\nHGhfQtkODETE4o7Rj+X5tnZN590v/cBxHTVndLRveWHbfm3YsIElS5bsef2zXT+DE6C2eskBfkqS\npGoYGRlhZGRkr3Wjo6Oztv9uw8edwOs71t0CbAM+mlL6aURsJ7tD5UewZ4LpWWTzRADuA6bymi/l\nNacBpwD35DX3AEsj4o2FeR/ryILNvYWaP42IEwrzPs4FRoEHD3QS119/PatXr97z+q/v/2ve+3fv\npVY3fEiSNDw8zPDw8F7rtm7dypo1a2Zl/12Fj5TSGB1v7BExBjyfUtqWr7oBuDIiHgEeA64BngLu\nyPexKyI+A1wXETuB3cDHgLtSSlvymociYjNwc0R8EBgAPg6MpJTaoxpfz9tya35774n5sW7s9pJQ\nY3KMwSnoX+iEU0mS5lrXE073Ie31IqVrI6JO9kyOpcB3gfNTSpOFsg3ANHA7MAh8DbisY78XATeS\njbbM5LXrC8eZiYgLgE8Cd5M9T+QW4KpuT6DZ2EWtBdR9zockSXPtiMNHSul39rFuI7DxAD8zQfbc\njssPUPMicPFBjv0kcMEhNnW/Go1R6i1gwYKD1kqSpCPjZ7sAjfFd1KZw5EOSpBIYPoDm+G5HPiRJ\nKonhA2iMv5SFD0c+JEmac4YPoDk5lk04deRDkqQ5Z/ggu9XWkQ9Jksph+AAarWY24dSRD0mS5pzh\nA2hONZ1wKklSSQwfQGN6nPpUwPz5vW6KJEnHPMMH0JyeoBYGD0mSymD4ABppgjqGD0mSymD4ABpp\nklrfYK+bIUlSJRg+gGZqUTd8SJJUCsMH0IgpavOGet0MSZIqofLhozXdYjoS9f5ar5siSVIlVD58\nNFoNAOrzfbqpJEllMHzk4aM2YPiQJKkMlQ8fzakmAPUBn24qSVIZKh8+9ox8DC3scUskSaqGyoeP\nZisf+Rg0fEiSVIbKh489E05rS3rcEkmSqsHw0b7sUlvU45ZIklQNlQ8fzfbIx4KlPW6JJEnVUPnw\n0WjuAqBW97KLJEllqHz4aI69CMDQQkc+JEkqQ+XDR2NslPokxELvdpEkqQyVDx/N8d3UpoC6TziV\nJKkMlQ8fjeYu6i1ggU84lSSpDIaP8d3UWjjyIUlSSSofPpoTY458SJJUosqHj0YrDx+OfEiSVIrK\nh4/mZCObcOrIhyRJpah8+GhMNbORj8HBXjdFkqRKMHxMNamleRDR66ZIklQJlQ8fzelx6szrdTMk\nSaqMyoePxswEdeb3uhmSJFVG5cNHM01S6xvodTMkSaqMyoePRmpRDyebSpJUFsMHLWr9Q71uhiRJ\nlVH58NGMaerzDB+SJJWl0uEjpUSjf5r6vFqvmyJJUmVUOnxMTE8AUJvvo9UlSSpLV+EjIj4QET+M\niNF8uTsi/llHzdUR8XRENCLiGxGxsmP7YETcFBHPRcTuiLg9Il7ZUbMsIj6XH2NnRHw6IhZ01Jwc\nEZsiYiwitkfEtRHR1fk0Wg0A6gM+Wl2SpLJ0O/LxJHAFsBpYA3wLuCMiVgFExBXAh4D3A2cCY8Dm\niCjey3oD8BbgHcA5wEnAFzuO83lgFbAurz0H+FR7Yx4yvgrMA9YC7wYuAa7u5mTa4aM2aPiQJKks\nXYWPlNKmlNLXUko/SSk9klK6EniJLAAArAeuSSl9JaX0Y+BdZOHibQARsRh4D7AhpfSdlNL9wKXA\nb0bEmXnNKuA84L0ppR+klO4GLgcujIgV+XHOA14LvDOl9EBKaTPwEeCyiDjkx5U2W00A6kOLuukG\nSZJ0BA57zkdE9EXEhUAduDsiTgVWAN9s16SUdgH3Amfnq04nG60o1jwMPFGoWQvszINJ251AAs4q\n1DyQUnquULMZWAK87lDPYc9ll6HFh/ojkiTpCHUdPiLiNyJiNzABfAJ4ex4gVpAFhB0dP7Ij3waw\nHJjMQ8n+alYAzxQ3ppSmgRc6avZ1HAo1B9WcykY+ajVHPiRJKsvhfKLaQ8AbyEYZ/gD4bEScM6ut\nmmMbNmxgyZIlPPfSM/Az+Le1/857X/E6hoeHe900SZJ6bmRkhJGRkb3WjY6Oztr+uw4fKaUp4Kf5\ny/vzuRrrgWuBIBvdKI5KLAfal1C2AwMRsbhj9GN5vq1d03n3Sz9wXEfNGR1NW17YdkDXX389q1ev\n5itb/5Z//uVhbv31f8uJBg9JkgAYHh7+hT/It27dypo1a2Zl/7PxnI8+YDCl9CjZG/+69oZ8gulZ\nwN35qvuAqY6a04BTgHvyVfcASyPijYVjrCMLNvcWal4fEScUas4FRoEHD7XhzbEsxdUXLjvUH5Ek\nSUeoq5GPiPhz4O/JJoguAt4J/DbZGz9kt9FeGRGPAI8B1wBPAXdANgE1Ij4DXBcRO4HdwMeAu1JK\nW/KahyJiM3BzRHwQGAA+DoyklNqjGl8nCxm35rf3npgf68aUUutQz6fReBGA2sKl3XSDJEk6At1e\ndnkl8Ddkb/ajwI+Ac1NK3wJIKV0bEXWyZ3IsBb4LnJ9SmizsYwMwDdwODAJfAy7rOM5FwI1kd7nM\n5LXr2xtTSjMRcQHwSbJRlTHgFuCqbk6m2dhF/wzMX7ikmx+TJElHoKvwkVJ63yHUbAQ2HmD7BNlz\nOy4/QM2LwMUHOc6TwAUHa8+BNMZ3U29BLPAhY5IklaXSn+3SHN9NrQUYPiRJKk2lw0d75MPwIUlS\neaodPibHqE0BdT/VVpKkslQ6fDQnG9nIR63W66ZIklQZlQ4fjVaD+nQf9FW6GyRJKlWl33WbU+PU\nUn+vmyFJUqVUOnw0ppvU0+F8vI0kSTpcFQ8fE9SY3+tmSJJUKZUOH800Sd3wIUlSqSodPhppknrf\nYK+bIUlSpVQ6fDRpUes3fEiSVKZKh48GU9T7h3rdDEmSKqXa4aNvito8HzAmSVKZKh0+mn0z1Of7\naHVJkspU2fAxPTPNRH8yfEiSVLLKho/xqXEAaoN+oq0kSWWqbPhotBoA1AcX9rglkiRVS+XDR23I\n8CFJUpkqGz6aEy8BUK8t6XFLJEmqlsqGj8buFwCo1w0fkiSVqbLho/nSTgBqtcU9bokkSdVS2fDR\nyMNHfeGyHrdEkqRqqW74GBsFoLbAyy6SJJWpsuGj2cjCR33x8T1uiSRJ1VLZ8NFo7gKg5mUXSZJK\nVdnw0RzfzcAU9C9ywqkkSWWqbPhojL9EvQXU/WwXSZLKVNnw0Zx4idoUhg9JkkpW2fDRmBzLRj7m\nzet1UyRJqpTqho9Wg/pMf6+bIUlS5VQ2fDRbTWozlT19SZJ6prLvvo3pJvXkJRdJkspW2fDRnJ6g\nxvxeN0OSpMqpbPhozExQN3xIklS66oaPNEktBnrdDEmSKqey4aOZWtT7BnvdDEmSKqey4aNBi3r/\nUK+bIUlS5VQ2fDRjito8w4ckSWWr7L2mjb5p6n21XjdDkqTKqW746J+h1u/nukiSVLbqXnbpT9Tn\nL+h1MyRJqpyuwkdE/ElEbImIXRGxIyK+FBGv2Ufd1RHxdEQ0IuIbEbGyY/tgRNwUEc9FxO6IuD0i\nXtlRsywiPhcRoxGxMyI+HRELOmpOjohNETEWEdsj4tqIOOg5TU1PMdUP9aGF3Zy+JEmaBd2OfLwJ\n+DhwFvC7wHzg6xGxZ/JERFwBfAh4P3AmMAZsjtjroRo3AG8B3gGcA5wEfLHjWJ8HVgHr8tpzgE8V\njtMHfJXs0tFa4N3AJcDVBzuJ8VYTgNqg4UOSpLJ1NecjpfR7xdcRcQnwDLAG+F6+ej1wTUrpK3nN\nu4AdwNuAL0TEYuA9wIUppe/kNZcC2yLizJTSlohYBZwHrEkp3Z/XXA5siogPp5S259tfC7w5pfQc\n8EBEfAT4aERsTClN7e88xhujANSHFnVz+pIkaRYc6ZyPpUACXgCIiFOBFcA32wUppV3AvcDZ+arT\nyUJPseZh4IlCzVpgZzt45O7Mj3VWoeaBPHi0bQaWAK87UKPHx7LwUastPrSzlCRJs+aww0dEBNnl\nk++llB7MV68gCwg7Osp35NsAlgOTeSjZX80KshGVPVJK02Qhp1izr+NQqNmnibHs0PUFSw5UJkmS\n5sCR3Gr7CeAfAr85S20pzX+59iZowp8su4XFt2QDMMPDwwwPD/e4ZZIk9d7IyAgjIyN7rRsdHZ21\n/R9W+IiIG4HfA96UUvp5YdN2IMhGN4qjEsuB+ws1AxGxuGP0Y3m+rV3TefdLP3BcR80ZHU1bXti2\nX3/0vrfzr5/fxn9bt5HX/NbvH6hUkqTK2dcf5Fu3bmXNmjWzsv+uL7vkweP3ySZ6PlHcllJ6lOyN\nf12hfjHZPI2781X3AVMdNacBpwD35KvuAZZGxBsLu19HFmzuLdS8PiJOKNScC4wCD3IA4+MvAVBf\ndNyBT1aSJM26rkY+IuITwDDwVmAsItojDaMppfH8+xuAKyPiEeAx4BrgKeAOyCagRsRngOsiYiew\nG/gYcFdKaUte81BEbAZujogPAgNkt/iO5He6AHydLGTcmt/ee2J+rBtTSq0DnUc7fNQWLuvm9CVJ\n0izo9rLLB8gmlH67Y/2lwGcBUkrXRkSd7JkcS4HvAuenlCYL9RuAaeB2YBD4GnBZxz4vAm4ku8tl\nJq9d396YUpqJiAuAT5KNqowBtwBXHewkxifGYB7Ulxx/0BOWJEmzq9vnfBzSZZqU0kZg4wG2TwCX\n58v+al4ELj7IcZ4ELjiUNhWNTzZgHgx52UWSpNJV8rNdJiYa1FoQg4O9bookSZVTyfAx3mpSn4pe\nN0OSpEqqZPiYmBqnNm34kCSpFyoZPsanxqlP9/e6GZIkVVI1w8f0BPVk+JAkqReqGT5mJqmlI3my\nvCRJOlzVDB/Tk9SZ3+tmSJJUSZUMHxOpRa1voNfNkCSpkioZPsZpUQ/DhyRJvVDN8JGmqPcP9boZ\nkiRVUiXDx0RMUzN8SJLUE5UMH+MxQ31erdfNkCSpkioZPib6pqkZPiRJ6olKho/xvkR9YEGvmyFJ\nUiVVM3zMg9qg4UOSpF6obPioDy7sdTMkSaqkSoYPgPrQol43QZKkSqps+KjVDB+SJPVCZcNHvb6k\n102QJKmSKhs+arXFvW6CJEmVVNnwUV+4rNdNkCSpkgwfkiSpVJUNH7VFhg9JknqhsuGjvui4XjdB\nkqRKqmymXemDAAAKWUlEQVT4qHnZRZKknqhs+PCzXSRJ6o1Kho++BPP75ve6GZIkVVIlw8fgNERE\nr5shSVIlVTJ8DM1U8rQlSfqlUMl34UHDhyRJPVPJd+Eh5vW6CZIkVVYlw0fN8CFJUs9UMnwMGj4k\nSeqZSoaPoTB8SJLUK5UMH4N9A71ugiRJlVXJ8DFk+JAkqWeqGT76B3vdBEmSKquS4WNwnuFDkqRe\nqWT4GJpX63UTJEmqrIqGj6FeN0GSpMrqOnxExJsi4u8i4mcRMRMRb91HzdUR8XRENCLiGxGxsmP7\nYETcFBHPRcTuiLg9Il7ZUbMsIj4XEaMRsTMiPh0RCzpqTo6ITRExFhHbI+LaiDjoOQ0N1Ls9bUmS\nNEsOZ+RjAfC/gT8CUufGiLgC+BDwfuBMYAzYHBHFW0xuAN4CvAM4BzgJ+GLHrj4PrALW5bXnAJ8q\nHKcP+CowD1gLvBu4BLj6YCcwNOhlF0mSeqXrp22llL4GfA0g9v259OuBa1JKX8lr3gXsAN4GfCEi\nFgPvAS5MKX0nr7kU2BYRZ6aUtkTEKuA8YE1K6f685nJgU0R8OKW0Pd/+WuDNKaXngAci4iPARyNi\nY0ppan/nMDiwYH+bJEnSHJvVOR8RcSqwAvhme11KaRdwL3B2vup0stBTrHkYeKJQsxbY2Q4euTvJ\nRlrOKtQ8kAePts3AEuB1B2rn0NDCrs5LkiTNntmecLqCLCDs6Fi/I98GsByYzEPJ/mpWAM8UN6aU\npoEXOmr2dRwKNfs0OOjIhyRJvVLNu11qi3rdBEmSKmu2P2FtOxBkoxvFUYnlwP2FmoGIWNwx+rE8\n39au6bz7pR84rqPmjI7jLy9s26+bb/oCm//HD/ZaNzw8zPDw8IF+TJKkShgZGWFkZGSvdaOjo7O2\n/1kNHymlRyNiO9kdKj8CyCeYngXclJfdB0zlNV/Ka04DTgHuyWvuAZZGxBsL8z7WkQWbews1fxoR\nJxTmfZwLjAIPHqid//4/beCdv//eIzlVSZKOWfv6g3zr1q2sWbNmVvbfdfjIn7WxkiwIAPxqRLwB\neCGl9CTZbbRXRsQjwGPANcBTwB2QTUCNiM8A10XETmA38DHgrpTSlrzmoYjYDNwcER8EBoCPAyP5\nnS4AXycLGbfmt/eemB/rxpRS60DnMFRf3O1pS5KkWXI4Ix+nA/+TbGJpAv4yX/83wHtSStdGRJ3s\nmRxLge8C56eUJgv72ABMA7cDg2S37l7WcZyLgBvJ7nKZyWvXtzemlGYi4gLgk8DdZM8TuQW46mAn\nMDjgcz4kSeqVw3nOx3c4yETVlNJGYOMBtk8Al+fL/mpeBC4+yHGeBC44UM2++Hh1SZJ6p5J3uwz2\n+6m2kiT1SjXDxzzDhyRJvVLJ8DGvb7bvMJYkSYeqkuFDkiT1juFDkiSVyvAhSZJKZfiQJEmlMnxI\nkqRSGT4kSVKpDB+SJKlUhg9JklQqw4ckSSqV4UOSJJXK8CFJkkpl+JAkSaUyfEiSpFIZPiRJUqkM\nH5IkqVSGD0mSVCrDhyRJKpXhQ5IklcrwIUmSSmX4kCRJpTJ8SJKkUhk+JElSqQwfkiSpVIYPSZJU\nKsOHJEkqleFDkiSVyvAhSZJKZfiQJEmlMnxIkqRSGT4kSVKpDB+SJKlUhg9JklQqw4ckSSqV4UOS\nJJXK8CFJkkpl+JAkSaUyfEiSpFIZPjTnRkZGet2EyrHPy2efl88+P3od9eEjIi6LiEcjohkR34+I\nM3rdJu3N/yDKZ5+Xzz4vn31+9Dqqw0dE/CvgL4GrgDcCPwQ2R8QJPW2YJEnar6M6fAAbgE+llD6b\nUnoI+ADQAN7T22ZJkqT9OWrDR0TMB9YA32yvSykl4E7g7F61S5IkHdi8XjfgCJwA9AM7OtbvAE7b\nz88MAWzbtm0Om6VOo6OjbN26tdfNqBT7vHz2efns83IV3juHjnRfkQ0WHH0i4kTgZ8DZKaV7C+v/\nK3BOSukXRj8i4iLgc+W1UpKkY847U0qfP5IdHM0jH88B08DyjvXLge37+ZnNwDuBx4DxOWuZJEnH\nniHg1WTvpUfkqB35AIiI7wP3ppTW568DeAL4WErpL3raOEmStE9H88gHwHXALRFxH7CF7O6XOnBL\nLxslSZL276gOHymlL+TP9Lia7HLL/wbOSyk929uWSZKk/TmqL7tIkqSjz1H7nA9JknR0MnxIkqRS\nVSZ8+AF0cysi3hQRfxcRP4uImYh46z5qro6IpyOiERHfiIiVvWjrsSAi/iQitkTErojYERFfiojX\n7KPOPp8lEfGBiPhhRIzmy90R8c86auzvORQRf5z//3Jdx3r7fZZExFV5HxeXBztqjri/KxE+/AC6\nUiwgm/D7R8AvTCSKiCuADwHvB84Exsj+DQbKbOQx5E3Ax4GzgN8F5gNfj4hau8A+n3VPAlcAq8k+\n2uFbwB0RsQrs77mW/8H4frL/v4vr7ffZ92OymzhW5MtvtTfMWn+nlI75Bfg+8FeF1wE8BfyHXrft\nWFyAGeCtHeueBjYUXi8GmsAf9rq9x8JC9nEDM8Bv2eel9vvzwKX295z380LgYeB3gP8JXFfYZr/P\nbl9fBWw9wPZZ6e9jfuTDD6DrvYg4lSw9F/8NdgH34r/BbFlKNuL0Atjncy0i+iLiQrLnCt1tf8+5\nm4Avp5S+VVxpv8+ZX88vof8kIm6LiJNhdvv7qH7OxyE6nA+g0+xaQfbGuK9/gxXlN+fYkj/Z9wbg\neyml9rVZ+3wORMRvAPeQPWZ6N/D2lNLDEXE29vecyEPePwZO38dmf89n3/eBS8hGmk4ENgL/K//d\nn7X+rkL4kI51nwD+IfCbvW5IBTwEvAFYAvwB8NmIOKe3TTp2RcSryIL176aUWr1uTxWklIqf2/Lj\niNgCPA78Idnv/6w45i+7cHgfQKfZtZ1sno3/BrMsIm4Efg/4Jymlnxc22edzIKU0lVL6aUrp/pTS\nfySb/Lge+3uurAFeAWyNiFZEtIDfBtZHxCTZX9z2+xxKKY0C/xdYySz+nh/z4SNPy/cB69rr8mHq\ndcDdvWpXlaSUHiX7xSz+Gywmu1PDf4PDlAeP3wfenFJ6orjNPi9NHzBof8+ZO4HXk112eUO+/AC4\nDXhDSumn2O9zKiIWkgWPp2fz97wql138ALo5FhELyH5BI1/1qxHxBuCFlNKTZEOnV0bEI8BjwDVk\ndxzd0YPmHvUi4hPAMPBWYCwi2n+JjKaUxvPv7fNZFBF/Dvw92SdnLwLeSfZX+Ll5if09y1JKY0Dn\nMybGgOdTStvyVfb7LIqIvwC+THap5R8Afwa0gL/NS2alvysRPpIfQFeG08lugUv58pf5+r8B3pNS\nujYi6sCnyO7M+C5wfkppsheNPQZ8gKyfv92x/lLgswD2+ax7Jdnv84nAKPAj4Nz2HRj2d2n2eo6Q\n/T7rXgV8HjgeeBb4HrA2pfQ8zF5/+8FykiSpVMf8nA9JkvTLxfAhSZJKZfiQJEmlMnxIkqRSGT4k\nSVKpDB+SJKlUhg9JklQqw4ckSSqV4UOSJJXK8CFJkkpl+JAkSaX6/ySR/t4bOCXgAAAAAElFTkSu\nQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1282d0a90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(halfParamCorrect, 'r')\n",
    "plt.plot(doubleParamCorrect, 'g')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, what's the moral of the story? Yes, a neural network can fit any dataset and can approximate any function, but the question is **\"Should you allow it to?\"**. When a neural network gets so fit to a particular training dataset, it isn't able to properly generalize, and that's when the test accuracy (which really is the most important metric in an ML system) drops. Our particular network got so fit to the training set because our model complexity was too high in that we had an extremely large number of hidden units. This was necessary to get the 100% classification accuracy on the train set, but hurt our ability to do well on the test set. The key in a lot of machine learning models is to find that crucial balance between having a network that is complex enough to fit the training data, and simple enough to be able to generalize to examples it hasn't seen before. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "More Reading:\n",
    "- [Visual proof](http://neuralnetworksanddeeplearning.com/chap4.html) of theorem\n",
    "- [Mathematical perspective](http://mcneela.github.io/machine_learning/2017/03/21/Universal-Approximation-Theorem.html)\n",
    "- [Cybenko's paper](https://link.springer.com/article/10.1007/BF02551274) and [Hornik's paper](http://www.sciencedirect.com/science/article/pii/0893608089900208) that proved the theorem"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}