03C_Keras_API.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TensorFlow Tutorial #03-C\n",
    "# Keras API\n",
    "\n",
    "by [Magnus Erik Hvass Pedersen](http://www.hvass-labs.org/)\n",
    "/ [GitHub](https://github.com/Hvass-Labs/TensorFlow-Tutorials) / [Videos on YouTube](https://www.youtube.com/playlist?list=PL9Hr9sNUjfsmEu1ZniY0XpHSzl5uihcXZ)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "Tutorial #02 showed how to implement a Convolutional Neural Network in TensorFlow. We made a few helper-functions for creating the layers in the network. It is essential to have a good high-level API because it makes it much easier to implement complex models, and it lowers the risk of errors.\n",
    "\n",
    "There are several of these builder API's available for TensorFlow: PrettyTensor (Tutorial #03), Layers API (Tutorial #03-B), and several others. But they were never really finished and now they seem to be more or less abandoned by their developers.\n",
    "\n",
    "This tutorial is about the Keras API which is already highly developed with very good documentation - and the development continues. It seems likely that Keras will be the standard API for TensorFlow in the future so it is recommended that you use it instead of the other APIs.\n",
    "\n",
    "The author of Keras has written a [blog-post](https://blog.keras.io/user-experience-design-for-apis.html) on his API design philosophy which you should read."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Flowchart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following chart shows roughly how the data flows in the Convolutional Neural Network that is implemented below. See Tutorial #02 for a more detailed description of convolution.\n",
    "\n",
    "There are two convolutional layers, each followed by a down-sampling using max-pooling (not shown in this flowchart). Then there are two fully-connected layers ending in a softmax-classifier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Flowchart](images/02_network_flowchart.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/magnus/anaconda3/envs/tf-gpu/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import math"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to import several things from Keras. Note the long import-statements. This might be a bug. Hopefully it will be possible to write shorter and more elegant lines in the future."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from tf.keras.models import Sequential  # This does not work!\n",
    "from tensorflow.python.keras.models import Sequential\n",
    "from tensorflow.python.keras.layers import InputLayer, Input\n",
    "from tensorflow.python.keras.layers import Reshape, MaxPooling2D\n",
    "from tensorflow.python.keras.layers import Conv2D, Dense, Flatten"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This was developed using Python 3.6 (Anaconda) and TensorFlow version:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.9.0'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The MNIST data-set is about 12 MB and will be downloaded automatically if it is not located in the given path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mnist import MNIST\n",
    "data = MNIST(data_dir=\"data/MNIST/\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The MNIST data-set has now been loaded and consists of 70.000 images and class-numbers for the images. The data-set is split into 3 mutually exclusive sub-sets. We will only use the training and test-sets in this tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of:\n",
      "- Training-set:\t\t55000\n",
      "- Validation-set:\t5000\n",
      "- Test-set:\t\t10000\n"
     ]
    }
   ],
   "source": [
    "print(\"Size of:\")\n",
    "print(\"- Training-set:\\t\\t{}\".format(data.num_train))\n",
    "print(\"- Validation-set:\\t{}\".format(data.num_val))\n",
    "print(\"- Test-set:\\t\\t{}\".format(data.num_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copy some of the data-dimensions for convenience."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The number of pixels in each dimension of an image.\n",
    "img_size = data.img_size\n",
    "\n",
    "# The images are stored in one-dimensional arrays of this length.\n",
    "img_size_flat = data.img_size_flat\n",
    "\n",
    "# Tuple with height and width of images used to reshape arrays.\n",
    "img_shape = data.img_shape\n",
    "\n",
    "# Tuple with height, width and depth used to reshape arrays.\n",
    "# This is used for reshaping in Keras.\n",
    "img_shape_full = data.img_shape_full\n",
    "\n",
    "# Number of classes, one class for each of 10 digits.\n",
    "num_classes = data.num_classes\n",
    "\n",
    "# Number of colour channels for the images: 1 channel for gray-scale.\n",
    "num_channels = data.num_channels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Helper-function for plotting images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Function used to plot 9 images in a 3x3 grid, and writing the true and predicted classes below each image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_images(images, cls_true, cls_pred=None):\n",
    "    assert len(images) == len(cls_true) == 9\n",
    "    \n",
    "    # Create figure with 3x3 sub-plots.\n",
    "    fig, axes = plt.subplots(3, 3)\n",
    "    fig.subplots_adjust(hspace=0.3, wspace=0.3)\n",
    "\n",
    "    for i, ax in enumerate(axes.flat):\n",
    "        # Plot image.\n",
    "        ax.imshow(images[i].reshape(img_shape), cmap='binary')\n",
    "\n",
    "        # Show true and predicted classes.\n",
    "        if cls_pred is None:\n",
    "            xlabel = \"True: {0}\".format(cls_true[i])\n",
    "        else:\n",
    "            xlabel = \"True: {0}, Pred: {1}\".format(cls_true[i], cls_pred[i])\n",
    "\n",
    "        # Show the classes as the label on the x-axis.\n",
    "        ax.set_xlabel(xlabel)\n",
    "        \n",
    "        # Remove ticks from the plot.\n",
    "        ax.set_xticks([])\n",
    "        ax.set_yticks([])\n",
    "    \n",
    "    # Ensure the plot is shown correctly with multiple plots\n",
    "    # in a single Notebook cell.\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot a few images to see if data is correct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUMAAAD5CAYAAAC9FVegAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAHihJREFUeJzt3XmUFNXZx/HvA0LYVQQFFWdOwAVCFBWDu0aBKCogccG4EGM0osEtAaNx1xglKBzRE7YD4QQNigKCUVFAEV8EJIIi4wYiCsRlhLggIsJ9/5i5XdUzPXtXVU/7+5zjmequ6qpnvPSdp27dxZxziIj80DVIOgARkVygylBEBFWGIiKAKkMREUCVoYgIoMpQRARQZSgiAqgyFBEBVBmKiACwS00ObtOmjSssLIwolNzzwQcfUFxcbEnHESeVcf5TGWdWo8qwsLCQZcuW1T6qeqZ79+5JhxA7lXH+UxlnpttkERFUGYqIAKoMRUQAVYYiIoAqQxERoIZPk0Vqa8SIEQBs3boVgDfeeAOAxx9/vNyxgwcPBuCoo44C4MILL4wjRPmBU2YoIoIyQ4nYueeeC8C0adMy7jcr3xd2zJgxAMydOxeAE044AYD99tsvihAlQe+++y4ABx54IAAPPPAAAEOGDIk9FmWGIiIoM5QI+GwQKs4IDzroIABOOeUUAN5///3UvlmzZgGwevVqAKZMmQLAjTfemP1gJVHLly8HoEGDkrxsn332SSwWZYYiIigzlCzy411nzJhRbl/Xrl2BIOtr06YNAC1atADgu+++Sx3bo0cPAF5//XUAPv/884gilqStWLECCP4dDBgwILFYlBmKiBBDZuj7kY0fPx6AvffeO7WvSZMmAJx//vkAtGvXDoBOnTpFHZZE4L///S8AzrnUez4jnDNnDgDt27fP+FnfDxHgrbfeStt3+umnZzVOSd7KlSsBGD16NAAXXXRRkuEAygxFRIAYMsOhQ4cCJRMsVsT3K2vVqhUAXbp0ycq1O3ToAMCwYcOAH+bcdXE644wzgOApMEDLli0BaN26daWfffTRR1Pb4fZDyU/vvPMOAFu2bAHSeyAkRZmhiAiqDEVEgBhukydMmAAE3STCt8BFRUVA0PHyxRdfBGDx4sVAMPzqww8/rPD8jRo1AoKuGr4RP3wef7us2+R4FBQUVPvYv/3tb0AwLCvMd7HxPyV/DB8+HChZggBy47upzFBEhBgyw5NPPjntZ5gfiuVt3rwZCDJF/9fi1VdfrfD8P/rRj4BgoLcf5gWwadMmADp27Fir2CU6Tz31FAC33HILANu2bUvt22uvvQC45557AGjWrFnM0UkUwg9R/Xfaf2+bN2+eREhplBmKiJBjw/F23313AE466aS09zNllWU98cQTQJBdAhx88MEADBw4MFshSpb4oXvhjNDz3Sz81F2SHxYsWFDuvbZt2yYQSWbKDEVEyLHMsDY+/fRTAK644gogfSiYb4+qqsOvxKd///5AMDzPGzRoUGr7rrvuijUmiYdf6iHMD4jIBcoMRUTIg8zwoYceAoIMcbfddkvt80+qJHm+/+eiRYuAoK3QtxnddNNNqWP9dE6SH1555RUAJk2alHrv0EMPBaBXr16JxJSJMkMREepxZvjyyy8DQV8078knn0xt++mjJHl+0s7i4uK09/30beoLmr/mzZsHpPf08H2M/TR+uUCZoYgIqgxFRIB6fJv89NNPA8Hcdz179gTgqKOOSiwmKc+veeKHWHonnngiAHfccUfcIUnM/CQtYWeffXYCkVROmaGICPUwM9y6dSsAzz77LBBM1HD77bcDwZRekpzwanZ33303UH726m7dugHqRpPPPv74YwAWLlwIpE+icuaZZyYSU2WUGYqIUA8zQz8ZqG+DOvXUUwE4+uijE4tJ0t13332p7aVLl6bt88Px1FaY//7xj38A8MknnwDBdzVXKTMUEaGeZIZ+IlCAO++8E4Bdd90VgJtvvjmRmKRi999/f4X7/PBJtRXmv3Xr1qW99lP05SplhiIi5Hhm6J9KXnXVVan3vv/+ewD69OkDqF9hfePLtDpP/X3274/dvn07AF988UW5Y/1Qr5EjR2Y8V8OGDVPb9957L6DlBKI2e/bstNenn356QpFUjzJDERFUGYqIADl6m7xjxw4gmNli7dq1qX2dOnUCggcpUr/4dWmq45xzzgGgffv2QNBFY+rUqXWKwa++F55DUbLHd7L25VVfKDMUESFHM8M1a9YAwQpqYb7bhua/y13+4RbAzJkza32exx57rMpj/MOVBg3S/6737dsXCNbeDjv22GNrHZNUbcaMGUDwsNPPap3rqx0qMxQRIccyQ99Js3fv3mnvjxgxIrWd64/nBaZPn57aHj58OFB+ogavqKgIqLwd8JJLLgGgoKCg3L5f/vKXAHTu3Ll2wUrWfPPNNwA888wzae/76brC3ZtykTJDERFyLDMcO3YsUH4YT7itwcxijUnqprrr4j7yyCMRRyJR8+23foXKfv36AXD11VcnFlNNKDMUESFHMkPfL+nBBx9MOBIRqS2fGfp1kusbZYYiIuRIZujXQP7qq6/S3vejTTTdk4hETZmhiAiqDEVEgBy5TS7Lr5w2b948AFq3bp1kOCLyA6DMUESEHMkMb7jhhrSfIiJxU2YoIgKYc676B5t9Bqyr8sD8UeCca5t0EHFSGec/lXFmNaoMRUTylW6TRURQZSgiAkT8NNnM9gDmlb5sB+wAPit9/TPnXOYZP+t2zS5AeD6ojsANzjnNAhGBhMq4AJgM7Ak44O8q3+gkUcal150M9AE2OOe6RXGNtOvF1WZoZrcBXzvnRpR530rj2BnBNRsBG4DDnHPrs31+SRdXGZvZ3sCezrkVZtYKWA6c6px7Nxvnl4rF+T02sxOArcC4OCrDRG6TzayTmRWZ2cPAKqCDmf0vtH+gmU0o3d7LzKab2TIzW2pmR9bgUr2At1QRxi/KMnbObXTOrSjd/hJ4G9gnut9GMon6e+ycWwBsiuwXKCPJNsODgJHOuS6UZG8VeQAY7pzrDpwD+P+5PcxsTBXXGAj8KxvBSq1EXsZm9mOgK/BqdkKWGorjexyLJEegrHHOlV8LtLyewIGh6f53N7OmzrklwJKKPmRmTYDTgOvqHKnUVtRl3Ap4AhjinPu6ztFKbURaxnFKsjLcEtreCYQXN2kS2jZq10h7GrDEOVdcy/ik7iIrYzNrDEwHJjnnZtUpSqmLqL/HscmJrjWlja6bzWx/M2sAnBnaPRe40r8ws+o2pJ6HbpFzRjbLuLSx/h/ACufcAxGEK7UQ0fc4NjlRGZa6HpgDLALCDzyuBI4xszfMrAi4FCpvazCzlsDPgZnRhiw1lK0yPoGSP3a9zGxF6X+/iDh2qZ5sfo+nAQuBLma23sx+HWXgGo4nIkJuZYYiIolRZSgigipDERFAlaGICKDKUEQEqGGn6zZt2rjCwsKIQsk9H3zwAcXFxVb1kflDZZz/VMaZ1agyLCwsZNmy6oy8yQ/du3dPOoTYqYzzn8o4M90mi4igylBEBFBlKCICqDIUEQFUGYqIAKoMRUSAZCd3rdCWLSXzRQ4dOhSAMWOCGX78Y/Jp06YBUFBQEHN0IpKPlBmKiJCjmeHGjRsBGD9+PAANGzZM7fOdRWfPng3A73//+5ijk9p47bXXABgwYABQMiqgtp577rnUdufOnQHo0KFD7YOTxPjvcd++fQEYPXo0AIMHD04dE/7+R0mZoYgIOZYZfvbZZwAMGjQo4Ugk2+bMmQPAtm3b6nyuWbOC9Z8mTpwIwNSpU+t8XonP559/DqRngABDhgwB4JJLLkm917Rp01hiUmYoIkKOZIYPPFCywNnMmSXrN736atXrgS9cuBAAv4bLIYccAsDxxx8fRYhSS99//z0ATz/9dNbOGR54f//99wNBD4TmzZtn7ToSnZdeegmADRvS150/77zzAGjSpEm5z0RNmaGICDmSGV5zzTVAzZ4aTZ8+Pe3nfvvtB8Bjjz2WOubwww/PVohSSy+88AIAixYtAuD666+v8zk3bdqU2l61ahUA33zzDaDMMJeF24vvuuuujMdceOGFAJQsjR0vZYYiIqgyFBEBEr5N7tOnDxA8BNmxY0eVn2nTpg0Q3A6tW7cOgLVr1wJwxBFHpI7duXNn9oKValu5cmVqe+DAgQB06tQJgBtvvLHO5w93rZH644033kht+0743i67lFRFp556aqwxhSkzFBEhgcxwwYIFqe23334bCBpLK3qAcvnll6e2e/fuDcCuu+4KwPz58wH4y1/+Uu5zf//734HyHTslWuGy8A82pkyZAkCLFi1qfV7/4CT8byiJhnapHf+wM5NevXrFGElmygxFRIgxM/QD830bEkBxcXHGY303mbPOOguAW2+9NbWvWbNmacf6KbzGjh1b7pzDhg0D4NtvvwWCSR0aNWpUu19CKvX4448D6R2sfVthuC23tnx3jHA2eOKJJwKw22671fn8Eq1wRu81btwYgLvvvjvucMpRZigiQoyZ4fbt24GKs0EIhtI9+uijQPDkuDI+M/RPKa+77rrUPj9Ey2eIfpqgjh071ih2qR4/4a7//w7Zaa/1dxWPPPIIEDx5BLjpppsAZfu5zHe4f+WVV8rt83d63bp1izWmTJQZioiQI8PxfHvSpEmTgOplhGX5rO/hhx9Ovbd06dIsRCdV+eKLLwBYvHhxuX1XXHFFnc8/btw4IJjirUuXLql9J510Up3PL9GqbOKVXOrpocxQRIQEMsNMo0yWLFlS5/P6USzhUSdlR7b4p9K+z5tkhx+Av379eiCYhilb1qxZk/a6a9euWT2/RCtTZuif/mfjziFblBmKiKDKUEQEiPE22a99HNVKV36VreXLl6feKzvM7/bbb4/k2j90LVu2BILuEeGJGvwQutatW9f4vJ9++ikQdNnxjjnmmFrFKfF6+eWXgaBLVJgfTrvvvvvGGlNllBmKiBBjZvjUU09l9Xy+m0VRURFQ+XAe31VHHXOj4Vcv80Pv/LA8gNNOOw1I7wyfyZtvvpna9g9M/PRsZSdjaNBAf8PrA78Cnn+QGZYLEzOUpX9VIiLkSKfr2vDTRD300EMVHlNYWAjA5MmTgWACCInGbbfdBqRnAv6OIDxBRyZt27ZNbftMsKKhmxdffHFdwpSYlG3rDU+mcdlll8UdTpWUGYqIUA8zQ79UgJ8YtjJ+2NZxxx0XaUxSonPnzkD6CoX+6X7ZjtNl+enawgYNGgSU7yTv2yglN/nO92WfIoefHGdjSrdsU2YoIkKMmWFliz4988wzaa8vvfRSADZu3Fjheaoz3Xu2n2BLzR166KFpP2vixz/+ccb3w/0Yf/rTn9YuMImMn7Kr7FPkfv36JRFOtSkzFBFBlaGICBDjbbKft8zPOh3mO+aWHaqXaeiev82uzkp6Ur/526yyt1u6Nc5tvrO15wc9XHPNNUmEU23KDEVEiDEzHDBgAADDhw9PvVfZeihV8X9tfHeO8ePHA9C+fftan1Nyi39IprWR65c5c+akve7QoQMQTM6Qq5QZiogQY2boV7HzK98BzJw5E4BRo0bV+Hx//vOfgWAtZMk/fr1rT52tc5tfAXP16tVp7zdp0gTI/YlSlBmKiJDAcDy/NnJ4u3fv3kCwCpqfqPWMM84A4He/+13qM/7JYniFNMlPfrVEP8D/lltuSTIcqYKfWs0PtVu1ahUA+++/f2Ix1YQyQxERcmSihlNOOSXtpwgEGca1114LaI3kXOf7/vrp9XwvgMMOOyyxmGpCmaGICDmSGYpk4tuOpX7Ze++9AZg4cWLCkdSMMkMREVQZiogAqgxFRABVhiIigCpDERFAlaGICACWabX7Cg82+wxYF104OafAOde26sPyh8o4/6mMM6tRZSgikq90mywigipDERFAlaGICBDx2GQz2wOYV/qyHbAD+Kz09c+cc99FdN0+wEigITDWOfe3KK4jyZVx6bV3AV4D3nfO9Y/qOj90CX6PJwN9gA3OuW5RXCPtenE9QDGz24CvnXMjyrxvpXHszNJ1GgHvAD8HPgaWAb90zr2bjfNLxeIq49B5hwHdgGaqDOMRZxmb2QnAVmBcHJVhIrfJZtbJzIrM7GFgFdDBzP4X2j/QzCaUbu9lZtPNbJmZLTWzI6s4/ZHAW865dc65bcBjQL+ofhfJLOIyxswKgF7ApKh+B6lc1GXsnFsAbIrsFygjyTbDg4CRzrkuwIZKjnsAGO6c6w6cA/j/uT3MbEyG4/cBPgq9Xl/6nsQvqjIGGAUMBdQ3LFlRlnGskpzPcI1zblk1jusJHBhaO3d3M2vqnFsCLIksOsmGSMrYzPoDHznnVphZz+yFK7WQN9/jJCvDLaHtnUB4pfAmoW2jZo20G4AOodf7UvlfLIlOVGV8NDDAzPqWnqeVmU12zg2qU7RSG1GVcexyomtNaaPrZjPb38waAGeGds8FrvQvzKyqhtTFQBczKzCzH1GSks/KdsxSM9ksY+fcMOfcvs65QuAC4DlVhMnL8vc4djlRGZa6HpgDLKKknc+7EjjGzN4wsyLgUqi4rcE5tx24CngeKAKmOOfeiTp4qZaslLHktKyVsZlNAxZSktysN7NfRxm4xiaLiJBbmaGISGJUGYqIoMpQRARQZSgiAtSwn2GbNm1cYWFhRKHkng8++IDi4mKr+sj8oTLOfyrjzGpUGRYWFrJsWXU6m+eH7t27Jx1C7FTG+U9lnJluk0VEUGUoIgKoMhQRAVQZiogAqgxFRABVhiIigCpDEREg2cldRUQA2Lx5MwAffvhhhccUFBQAMHLkSAC6du0KwAEHHADAIYccUqcYlBmKiJBwZvjpp58CcM455wBw9NFHA3DZZZcBJT3ls+GLL74A4KWXXgLglFNOAaBRo0ZZOb+I1MxTTz0FwOzZswF48cUXAXjvvfcq/MyBBx4IlAyvA9i2bVva/p0767ZKqTJDERESyAx92wDAT37yEyDI3Pbaay8g+xnhYYcdBkBxcTFAalzm/vvvn5XrSPV9+eWXAPzpT38CYNWqVQDMnTs3dYwy9vywZs0aAB566CEAxo0bl9q3detWAGoy0/4770S7eocyQxERYswMfVbm2wcBPv/8cwCuvLJk0azRo0dn9Zp33XUXAGvXrgWCv0zKCOM3ZcoUAG666Sag/FNDnzEC7LHHHvEFJpFZv75kPahRo0bV6TwHHXQQEDw9jooyQxERYswMX3vtNSB4ahR2yy23ZO06b775Zmp7xIgRAJx5Zsnyreeee27WriPV47ODa6+9FgjuEMzS59ocMmRIavvBBx8EoHXr1nGEKLXgyxGCzO/YY48Fgt4ajRs3BmDXXXcFoEWLFqnPfP311wD84he/AIKsr0ePHgAceuihqWObNm0KQPPmzbP8W6RTZigigipDEREghttk37H6iSeeKLdv4sSJALRt27bO1/G3x7169Sq3b8CAAQC0bNmyzteRmvFNFf5hWUWmTp2a2n7mmWeA4GGLv4X2t12SnC1btgDp37PXX38dgJkzZ6Yde9RRRwGwfPlyIL3LnH+Atu+++wLQoEHyeVnyEYiI5IDIM8M//OEPQNC1wneABjj77LOzdp2XX34ZgI8//jj13sUXXwzABRdckLXrSNXWrVuX2p40aVLaPj+Y3newf/7558t93neW91nl+eefD0C7du2yH6xUy3fffQfAr371KyDIBgFuvPFGAHr27Jnxs5kGUey3335ZjrDulBmKiBBDZui7UPif++yzT2pfXdqA/HCeu+++GwiG/IS7bPg2SYnXihUrUtu+M/Xxxx8PwIIFCwD49ttvAXjkkUcA+Otf/5r6zOrVq4Egy+/Xrx8QtCWqy018fBcY/z3zEyuE2/mHDh0KQLNmzWKOLruUGYqIkMBEDX7qHoDevXsDsNtuuwEwePDgKj/vO237n4sXL07bn812SKmd8NRKPlP3na69Jk2aAPCb3/wGgMcffzy1zw/w94P4fcahp8nx80+I77nnHiCYYHXhwoWpY3yn6vpOmaGICDFkhldffTUA8+fPB2Djxo2pfb79yGcATz75ZJXn88eWHc7VsWNHIGjbkOT861//Kvfev//9bwD69++f8TN+WrVMjjzySCB9OJfEY9GiRWmv/TA53z8wnygzFBEhhszw8MMPB2DlypVA+pPGZ599FoDhw4cDsOeeewIwaNCgCs934YUXAnDwwQenve+XDPAZoiTnvPPOS237bP/VV18F4O233waCfw8zZswA0if99W3I/j0/9Zov+y5dukQWu6QLt+VC8ET/9ttvT73Xt29fIH1yhfpImaGICKoMRUQAsJqsQdC9e3dXWUN3HN5//30guB3u1q0bAM899xyQnUkfvO7du7Ns2TKr+sj8kY0y3rRpU2rbl5MfYlfRA7DwwH/fgf70008H4N133wWCVRPHjBlTp/jCVMaVKztoIpOGDRsCcPnllwPBnIQfffQRAJ06dQKCNY/C/Bo4flKHKB7MVLeMlRmKiJDwusm1cccddwDBXyr/8CWbGaHUTXi43LRp0wA466yzgPIZ4lVXXQXAvffem/qM75Dtp17zQ/XmzJkDBJ2yQQ/MovbHP/4RgPvuu6/CY3bs2AEEGb3/WRP+4emJJ54IpE/pFhdlhiIi1JPM0GcXAJMnTwagVatWgFZSy3V+WiffRcNPzOC7z/hM32eDYTfffDMAb731FhB00/GfgeDfg0TDD8Pzq1r66dS2b9+eOsavc+MzxNrwk0D773p4JTw/yW/UlBmKiFBPMkPf0TPstNNOA9Ini5Xc5TPEiiYAzcSviuZXNfSZ4QsvvJA6xj+51rRe0fBPio844gggeLIfNm/ePCDIFm+77TYAli5dWuPr+bbk//znPzX+bF0pMxQRoR5mhn7tVP+US/Kfb6+aNWsWkP6k0a+xnM21t6VmTj755LTXfsitzwwbNWoEBMtwAFx66aUAjBw5EgjakpOkzFBEBFWGIiJAjt8m+2FX4RXv/KpqenDyw+HX1B02bBiQvj6vb6wfOHAgAAcccEC8wUk5fgZ7v2qef7DiZx8CeO+994BgxvqywmslxUWZoYgI9SQzDA8S79OnT9oxX331FRDMfZeL67FKdvhJOe68887Ue/5B2g033AAE63P7bjkSv86dOwNBl6hHH3203DHh7lEAu+xSUhX5LnPh4ZlxUWYoIkKOZ4aZ+L8gPgPwj+b98B0Nz8p/F110UWp77NixAEyfPh0I2qLKzoQu8fFZ+ahRo4Dg7i3ckfqTTz4BoLCwEAjK1LcBJ0GZoYgI9TAzHD9+PAATJkwA4Le//S0QDOqX/Beerm3u3LlAsJ6vn1ggFzrx/tD5nh9+rfR//vOfqX2vvPIKEGSCfgqvJCkzFBEhxzPD0aNHA3Drrbem3jv++OMBGDx4MAC77747AI0bN445OskFvveAXzbAD9krKioCtJJeLvGrG5bdzhXKDEVEyPHM8LjjjgNg/vz5CUciuc5PHnvIIYcAsHr1akCZoVSfMkMREVQZiogAOX6bLFJdfk2ctWvXJhyJ1FfKDEVEUGUoIgKoMhQRAcD8alTVOtjsM2BddOHknALnXNuqD8sfKuP8pzLOrEaVoYhIvtJtsogIqgxFRICI+xma2R7AvNKX7YAdwGelr3/mnPsuwmvvArwGvO+c6x/VdX7okipjM7sOuKT05Rjn3OgoriOJlvF6YHPp9bY553pEcZ3U9eJqMzSz24CvnXMjyrxvpXHszPL1hgHdgGaqDOMRVxmbWTdgMnAk8D3wHPAb55x6XEcszu9xaWXY1Tn3v2ydszKJ3CabWSczKzKzh4FVQAcz+19o/0Azm1C6vZeZTTezZWa21MyOrMb5C4BewKSofgepXMRl3BlY7Jzb6pzbDrwEnBnV7yKZRf09jluSbYYHASOdc12ADZUc9wAw3DnXHTgH8P9ze5jZmAo+MwoYCuhRebKiKuOVwAlm1trMmgOnAh2yG7pUU5TfYwfMN7P/mNklFRyTNUmOTV7jnFtWjeN6AgeGlgvd3cyaOueWAEvKHmxm/YGPnHMrzKxn9sKVWoikjJ1zb5rZ/cBc4GtgOSXtShK/SMq41JHOuQ1m1g543szecs4tykLMGSVZGW4Jbe8ELPS6SWjbqFkj7dHAADPrW3qeVmY22Tk3qE7RSm1EVcY458YB4wDMbDiwug5xSu1FWcYbSn9+bGZPAj8DIqsMc6JrTWmj62Yz29/MGpDe/jMXuNK/KG08r+xcw5xz+zrnCoELgOdUESYvm2VcesyepT8Lgb7A1GzGKzWXzTI2sxZm1qJ0uzklzwDezH7UgZyoDEtdD8yhpOZfH3r/SuAYM3vDzIqAS6HKtgbJTdks45mlx84ELnfOfRlh3FJ92Srj9sD/mdnrwFJghnNubpSBazieiAi5lRmKiCRGlaGICKoMRUQAVYYiIoAqQxERQJWhiAigylBEBFBlKCICwP8D3P5bzM0W5d8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fab44735a20>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Get the first images from the test-set.\n",
    "images = data.x_test[0:9]\n",
    "\n",
    "# Get the true classes for those images.\n",
    "cls_true = data.y_test_cls[0:9]\n",
    "\n",
    "# Plot the images and labels using our helper-function above.\n",
    "plot_images(images=images, cls_true=cls_true)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Helper-function to plot example errors\n",
    "\n",
    "Function for plotting examples of images from the test-set that have been mis-classified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_example_errors(cls_pred):\n",
    "    # cls_pred is an array of the predicted class-number for\n",
    "    # all images in the test-set.\n",
    "\n",
    "    # Boolean array whether the predicted class is incorrect.\n",
    "    incorrect = (cls_pred != data.y_test_cls)\n",
    "\n",
    "    # Get the images from the test-set that have been\n",
    "    # incorrectly classified.\n",
    "    images = data.x_test[incorrect]\n",
    "    \n",
    "    # Get the predicted classes for those images.\n",
    "    cls_pred = cls_pred[incorrect]\n",
    "\n",
    "    # Get the true classes for those images.\n",
    "    cls_true = data.y_test_cls[incorrect]\n",
    "    \n",
    "    # Plot the first 9 images.\n",
    "    plot_images(images=images[0:9],\n",
    "                cls_true=cls_true[0:9],\n",
    "                cls_pred=cls_pred[0:9])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PrettyTensor API\n",
    "\n",
    "This is how the Convolutional Neural Network was implemented in Tutorial #03 using the PrettyTensor API. It is shown here for easy comparison to the Keras implementation below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "if False:\n",
    "    x_pretty = pt.wrap(x_image)\n",
    "\n",
    "    with pt.defaults_scope(activation_fn=tf.nn.relu):\n",
    "        y_pred, loss = x_pretty.\\\n",
    "            conv2d(kernel=5, depth=16, name='layer_conv1').\\\n",
    "            max_pool(kernel=2, stride=2).\\\n",
    "            conv2d(kernel=5, depth=36, name='layer_conv2').\\\n",
    "            max_pool(kernel=2, stride=2).\\\n",
    "            flatten().\\\n",
    "            fully_connected(size=128, name='layer_fc1').\\\n",
    "            softmax_classifier(num_classes=num_classes, labels=y_true)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sequential Model\n",
    "\n",
    "The Keras API has two modes of constructing Neural Networks. The simplest is the Sequential Model which only allows for the layers to be added in sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Start construction of the Keras Sequential model.\n",
    "model = Sequential()\n",
    "\n",
    "# Add an input layer which is similar to a feed_dict in TensorFlow.\n",
    "# Note that the input-shape must be a tuple containing the image-size.\n",
    "model.add(InputLayer(input_shape=(img_size_flat,)))\n",
    "\n",
    "# The input is a flattened array with 784 elements,\n",
    "# but the convolutional layers expect images with shape (28, 28, 1)\n",
    "model.add(Reshape(img_shape_full))\n",
    "\n",
    "# First convolutional layer with ReLU-activation and max-pooling.\n",
    "model.add(Conv2D(kernel_size=5, strides=1, filters=16, padding='same',\n",
    "                 activation='relu', name='layer_conv1'))\n",
    "model.add(MaxPooling2D(pool_size=2, strides=2))\n",
    "\n",
    "# Second convolutional layer with ReLU-activation and max-pooling.\n",
    "model.add(Conv2D(kernel_size=5, strides=1, filters=36, padding='same',\n",
    "                 activation='relu', name='layer_conv2'))\n",
    "model.add(MaxPooling2D(pool_size=2, strides=2))\n",
    "\n",
    "# Flatten the 4-rank output of the convolutional layers\n",
    "# to 2-rank that can be input to a fully-connected / dense layer.\n",
    "model.add(Flatten())\n",
    "\n",
    "# First fully-connected / dense layer with ReLU-activation.\n",
    "model.add(Dense(128, activation='relu'))\n",
    "\n",
    "# Last fully-connected / dense layer with softmax-activation\n",
    "# for use in classification.\n",
    "model.add(Dense(num_classes, activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Compilation\n",
    "\n",
    "The Neural Network has now been defined and must be finalized by adding a loss-function, optimizer and performance metrics. This is called model \"compilation\" in Keras.\n",
    "\n",
    "We can either define the optimizer using a string, or if we want more control of its parameters then we need to instantiate an object. For example, we can set the learning-rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.python.keras.optimizers import Adam\n",
    "\n",
    "optimizer = Adam(lr=1e-3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For a classification-problem such as MNIST which has 10 possible classes, we need to use the loss-function called `categorical_crossentropy`. The performance metric we are interested in is the classification accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer=optimizer,\n",
    "              loss='categorical_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training\n",
    "\n",
    "Now that the model has been fully defined with loss-function and optimizer, we can train it. This function takes numpy-arrays and performs the given number of training epochs using the given batch-size. An epoch is one full use of the entire training-set. So for 10 epochs we would iterate randomly over the entire training-set 10 times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/1\n",
      "55000/55000 [==============================] - 3s 59us/step - loss: 0.2201 - acc: 0.9341\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7fab41f81ef0>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(x=data.x_train,\n",
    "          y=data.y_train,\n",
    "          epochs=1, batch_size=128)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation\n",
    "\n",
    "Now that the model has been trained we can test its performance on the test-set. This also uses numpy-arrays as input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 0s 35us/step\n"
     ]
    }
   ],
   "source": [
    "result = model.evaluate(x=data.x_test,\n",
    "                        y=data.y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can print all the performance metrics for the test-set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss 0.05883921128492802\n",
      "acc 0.9807\n"
     ]
    }
   ],
   "source": [
    "for name, value in zip(model.metrics_names, result):\n",
    "    print(name, value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or we can just print the classification accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "acc: 98.07%\n"
     ]
    }
   ],
   "source": [
    "print(\"{0}: {1:.2%}\".format(model.metrics_names[1], result[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prediction\n",
    "\n",
    "We can also predict the classification for new images. We will just use some images from the test-set but you could load your own images into numpy arrays and use those instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "images = data.x_test[0:9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are the true class-number for those images. This is only used when plotting the images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "cls_true = data.y_test_cls[0:9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the predicted classes as One-Hot encoded arrays."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = model.predict(x=images)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the predicted classes as integers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "cls_pred = np.argmax(y_pred, axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAD5CAYAAACj3GcTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xe8VNW5//HPg2IQVBCxU86NWECjmGCvV5EoFpQYxViIsUQxlngDSYwSTJAYbPwsV1FfoglWEBGJimJXQISASrGAokEvIqJGiWJ7fn/MXjN7Tp19ph++79eL19kzs8uj68w6z1p77bXM3RERkdy0KncAIiLVRJWmiEgCqjRFRBJQpSkikoAqTRGRBFRpiogkoEpTRCQBVZoiIgmo0hQRSWDdfA7u1KmT19TUFCiU6jBnzpyV7r5pueMoFZVxy6cyTiavSrOmpobZs2fnc4qqY2bvlDuGUlIZt3wq42TUPBcRSUCVpohIAqo0RUQSUKUpIpKAKk0RkQTyunsu0lxXXnklAF988QUAr7zyCgATJkyos+/ZZ58NwF577QXAySefXIoQReqlTFNEJAFlmlJSxx9/PADjx4+v93Mzq/PeTTfdBMC0adMAOOCAAwDo2rVrMUKUMnrjjTcA2H777QG49tprATj33HPLFlNtyjRFRBJQpilFF7JLaDjD3GGHHQA49NBDAXjrrbfSn02ePBmAxYsXAzBu3DgALrroosIHK2U1d+5cAFq1SuVzW2+9dTnDqZcyTRGRBJRpStGE55kfeOCBOp/ttNNOQCaL7NSpEwAbbLABAF999VV63z322AOAl19+GYCPPvqoSBFLuc2bNw/I/B4MGDCgnOHUS5mmiEgCJc80wzi8W265BYCtttoq/VmbNm0AOPHEEwHYYostAOjevXspQ5QC+b//+z8A3D39Xsgwp06dCsCWW25Z77FhHCfAokWLsj474ogjChqnlN+rr74KwHXXXQfAKaecUs5wGqVMU0QkgZJnmkOGDAFg6dKlDe4TxuVttNFGAPTs2bMg1+7SpQsAQ4cOBaB3794FOa/U78gjjwQyd70BNtxwQwA6duzY6LH33ntvejvevykt0+uvvw7A6tWrgewRF5VGmaaISAKqNEVEEih58/zWW28FMsNH4k3vhQsXApkBrk8//TQAM2fOBDKPzb377rsNnr9169ZAZghLuBkRP09opqt5XhrdunXLed8rrrgCyDxOFxeGHoWf0nKMGjUKSC29AZX93VSmKSKSQMkzzYMPPjjrZ1x4hC74+OOPgUzmGf76vPTSSw2e/3vf+x6QeeA/PJ4HsGrVKgC22WabZsUuxTNlyhQAhg0bBsCaNWvSn22++eYAXH755QC0bdu2xNFJMcRvBofvdPjetmvXrhwh5USZpohIAhX9GOXGG28MwEEHHZT1fn1Zam33338/kMlWAXbeeWcABg4cWKgQpUDCI5fxDDMIw0/ClHDSMjzzzDN13tt008pfbl6ZpohIAhWdaTbHihUrABg8eDCQ/Qhf6C9ramC1lM7RRx8NZB6rDAYNGpTeHjFiREljktIIS5zEhQdPKpkyTRGRBFpcpnnDDTcAmYyzQ4cO6c/CnTkpvzB+dvr06UCmLzP0aV188cXpfcM0YdIyzJgxA4CxY8em39t1110BOOSQQ8oSUxLKNEVEEmgxmebzzz8PZMbyBQ8++GB6O0xLJuUXJpdduXJl1vthWkCNpW25nnjiCSB7ZEsYox2mh6xkyjRFRBJQpSkikkCLaZ4//PDDQGbuxT59+gCw1157lS0mqSusCRQejQ0OPPBAAP70pz+VOiQpsTBZT9xPf/rTMkTSPMo0RUQSqPpM84svvgDg0UcfBTITdlx66aVAZqo4KZ/46pEjR44E6s7G3qtXL0DDi1qy5cuXA/Dcc88B2ZPpHHPMMWWJqTmUaYqIJFD1mWaYtDb0kR122GEA7L333mWLSbJdddVV6e1Zs2ZlfRYeo1RfZst3++23A/DBBx8Ame9qtVGmKSKSQFVmmmHCWoA///nPALRv3x6ASy65pCwxScOuvvrqBj8Lj72qL7Ple+edd7Jeh6kfq40yTRGRBKoq0wx3Yc8777z0e9988w0A/fr1AzQus9qEMs1llENoTYR9v/76awA+/fTTOvuGR/Suueaaes+1zjrrpLf/+te/AlpGo9geeuihrNdHHHFEmSLJjzJNEZEEVGmKiCRQFc3zb7/9FsjMhPL222+nP+vevTuQuSEk1SWs25SL4447DoAtt9wSyAxdueeee/KKIax2GZ/DUwonDGYP5VXtlGmKiCRQFZnmkiVLgMyKhXFhOIvmX6xc4SYdwKRJk5p9nvvuu6/JfcJNolatsvOBo446CoDevXvXOWbfffdtdkzStAceeADI3LQNs7RX6+qiyjRFRBKo6EwzDIbt27dv1vtXXnllertahy2sTSZOnJjeHjVqFFB3wo5g4cKFQOP9lKeddhoA3bp1q/PZT37yEwB69OjRvGClYP7zn/8A8Mgjj2S9H6aBiw/7qibKNEVEEqjoTHPMmDFA3cev4n0hZlbSmCQ/ua5rfddddxU5Eim20L8cVoTt378/AOeff37ZYioEZZoiIglUZKYZxnVdf/31ZY5ERJorZJphnfOWQpmmiEgCFZlphjXMP/vss6z3w9M/mkZMRMpFmaaISAKqNEVEEqjI5nltYaXCJ554AoCOHTuWMxwRWYsp0xQRSaAiM83f//73WT9FRCqFMk0RkQTM3Zt/sNmHwDtN7tiydHP3TcsdRKmojFs+lXEyeVWaIiJrGzXPRUQSUKUpIpJAo5WmmW1iZvOif8vN7L3Y6/WKEZCZ9YxdY56ZfWZmv2rimNPN7MNo/0Vm9os8YxhnZkc3sc/vYjEuMLNvzKx9PtcthzKVcTcze9rMFkb/7xot3+iYcpTxKWb2qpm9YmYvmNkP8rlmuZSjjKPr3hHKLMf9y1HGO5rZDDNbY2YX5HRid8/pHzAc+E097xvQKtfzJPkHtAZWAJ2b2O90YHS0vQWwEuhUa591E1x3HHB0gv2PAR4rxv+DUv4rVRkDWwG9ou2NgCXAdpVWxsA+QIdo+0jghXKXUbWUcXTOA4DdgXk57l+OMt4c6A1cDlyQy3mb1Tw3s+5RlnAnsADoYmafxD4faGa3Rtubm9lEM5ttZrPMbM8ElzoEWOTuy3I9wN2XA0uBrmY2wsz+ZmYvALeb2bpmdnUUxytmdnoUYysz+18ze83MHgc6JYgR4ATg7oTHVLRilrG7v+/u86LtfwOvAVvnGlupytjdX3D38N88E+ica4zVoNjfY3d/BljVnNhKWMYfuPts4JtcY8tncPsOwCnuPtvMGjvPtcAod59pZjXAFGAnM9sDONXdz2rk2IEkrIzMrDvQDXgrFuf+7v6lmQ0GVrj77mb2PWCmmT0G7An8F9CTVBa0ELgpOt9lpDKMhxu43gZAH+CMJHFWiaKXsZl9H9gJeCnXoEpdxpHTgEca+bxaleJ7nFiZyjgn+VSaS6Iauil9gO0tsyzFxma2vru/CLzY0EFm1gY4HLgwx3hONLMDgTXA6e7+SXTNB939y2ifvkAPMxsYvW4PbAvsD9zt7t8By8zs6XBSd/9DE9ftDzzj7p/mGGc1KXYZbwTcD5zr7p/ncJ2ylLGZ9QFOBlriWr9FLeNmKNf3OGf5VJqrY9vfkeoTCdrEtg3Y3d3rX36wYYcDL7r7yhz3v9Pd6+vIjcdpwGB3fyK+g5kdkzC2uIHA3/M4vpIVrYwtdQNiIjDW3SfneFjJy9jMegFjgB+7+8fNOUeFK/b3OKlyfY9zVpAhR1HN/rGZbWtmrUjdGAmmAeeEF9EvYS7q9BOa2flmlk8zYCowODRDzGx7M1sfeBY4PuoT2ZpUB3aTzGxjYG/goTxiqgqFLGNLpQ63k7pBcG2tzyqmjKNm6ATgZ+6+OI+YqkKRvsd1VFIZN0chx2n+ltR/zHQgfuPmHGCfqMN2IVHfn5ntYWY31XciM9sQ+G9gUq2PegAf5RHjGOBNYJ6ZzQduJJVtTwDeJdUHMhZIL2piZpeZWb8GzvcT4BF3/yKPmKpJocr4AFJ/FA+xzNCXH0efVVIZDwc6AmOiGAvZDK1UhfwejweeA3qa2TIz+3n0UcWUsZl1NrNlwHnA8CjOto1dvKoeozSzfwD93T3nO11SXVTGLV+1l3FVVZoiIuWmxyhFRBJQpSkikoAqTRGRBFRpiogkkNcaQZ06dfKampoChVId5syZs9LXolm9VcYtn8o4mbwqzZqaGmbPzuUJrJbDzNaqZQFUxi2fyjgZNc9FRBJQpSkikoAqTRGRBFRpiogkoEpTRCQBVZoiIgnkNeSoVFavTs0/OmTIEABuuikzE1Xv3r0BGD9+PADdunUrcXQisjZRpikikkBVZJrvv/8+ALfccgsA66yzTvqzMCj3oYdSk6f/6ldNLqEtFeCf//wnAAMGDABg6dKlzT7XY489lt7u0aMHAF26dGl+cFI24Xt81FFHAXDdddcBcPbZZ6f3iX//y0GZpohIAhWdaX744YcADBo0qMyRSKFNnToVgDVr1uR9rsmTM+uy3XbbbQDcc889eZ9XSuejj1KrX8QzSoBzzz0XgNNOOy393vrrr1+6wOqhTFNEJIGKzDSvvTa1QOGkSal11V566aUmj3nuuecACMt37LLLLgDsv//+xQhRmumbb1LLwjz88MMFO2cYQQFw9dVXA5kRF+3atSvYdaR4nn32WQDee++9rPdPOOEEANq0aVPnmHJRpikikkBFZpoXXJBaKz7JXbKJEydm/ezatSsA9913X3qfH/3oR4UKUZrpqaeeAmD69OkA/Pa3v837nKtWrUpvL1iwAID//Oc/gDLNShbvzx4xYkS9+5x88skAmFlJYsqFMk0RkQRUaYqIJFBRzfN+/foBmZs53377bZPHdOrUCcg0w955JzUh89tvvw3Abrvtlt73u+++K1ywkrNXX301vT1w4EAAunfvDsBFF12U9/njQ46kerzyyivp7fCwQ7Duuqmq6bDDDitpTLlQpikikkDZM81nnnkmvf3aa68BmU7fhm4EnXXWWentvn37AtC+fXsAnnzySQAuu+yyOsfdeOONQN0BtFJc8bIIN2jGjRsHwAYbbNDs84YbQPHfoUq6YSCNCzdt63PIIYeUMJJklGmKiCRQtkwzTNAQ+rgAVq5cWe++YfjQscceC8Af//jH9Gdt27bN2jdMDTdmzJg65xw6dCgAX375JZCZ3KN169bN+4+QRk2YMAHIHsge+jLjfc3NFYapxLPLAw88EIAOHTrkfX4prngLIVhvvfUAGDlyZKnDyZkyTRGRBMqWaX799ddAw9klZB6BvPfee4HMnfLGhEwz3JW98MIL05+FR+tCxhmmn9pmm20SxS65CRNDh//vUJj+5NBKueuuu4DMnVaAiy++GFDroZKFBxtmzJhR57PQcuzVq1dJY0pCmaaISAJlv3ten9DfNXbsWCC3DLO2kEXeeeed6fdmzZpVgOikKZ9++ikAM2fOrPPZ4MGD8z7/zTffDGSmDuzZs2f6s4MOOijv80txNTYBTzWMbFGmKSKSQNkzzfqe+nnxxRfzPm94qij+FFDtJ43CXfgwZlAKI0zEsGzZMiAzvVehLFmyJOv1TjvtVNDzS3HVl2mG0Q6FaIkUmzJNEZEEVGmKiCRQtuZ5WLu8WCvLhVXt5s6dm36v9uOZl156aVGuvbbbcMMNgcywkfiEHeHRx44dOyY+74oVK4DMUKZgn332aVacUlrPP/88kBkqFhceg+7cuXNJY2oOZZoiIgmULdOcMmVKQc8Xhp8sXLgQaPwxrDCESQOgiyOsFhgemQyPUwIcfvjhQPZDB/WZP39+ejvc+AnT/tWelKNVK/3trwZhxclwQzaukifoqE2/bSIiCZR9yFGhhOnHbrjhhgb3qampAeCOO+4AMhOBSHEMHz4cyM4sQgsjPlFLfTbddNP0dsgsG3rk9tRTT80nTCmR2n3R8UlVzjzzzFKH02zKNEVEEqj6TDMskREmMG5MeNxuv/32K2pMktKjRw8ge0XQMJqh9gD12sI0gHGDBg0C6j6MEPpQpTKFhxxq3zWP3ykvxFSBpaJMU0QkgbJlmo0tnvbII49kvT7jjDMAeP/99xs8Ty7LHBT6jr0kt+uuu2b9TOL73/9+ve/Hx4H+4Ac/aF5gUjRhKrjad8379+9fjnDypkxTRCQBVZoiIgmUrXke5s0Ls6jHhQHQtR+xrO+Ry9C8z2XlSqluoXlXu5mnJnllC4Pag/BwyQUXXFCOcPKmTFNEJIGyZZoDBgwAYNSoUen3GlsvqCnhr1cY5nLLLbcAsOWWWzb7nFJZws0+rW1eXaZOnZr1ukuXLkBmko5qo0xTRCSBsmWaYdXIsNIkwKRJkwAYPXp04vP94Q9/ADJrmUvLE9arDzSovbKFFWcXL16c9X6bNm2A6p0wR5mmiEgCZX+MMqxtHt/u27cvkFl1MEwofOSRRwLwy1/+Mn1MuJMaX5FQWqawOmmY6GHYsGHlDEeaEKbsC49ILliwAIBtt922bDEVgjJNEZEEyp5p1ufQQw/N+ikCmYzl17/+NaA1zitdGDsdpm0Mox5++MMfli2mQlCmKSKSQEVmmiL1CX3bUl222morAG677bYyR1IYyjRFRBJQpSkikoAqTRGRBFRpiogkoEpTRCQBVZoiIglY7QldEx1s9iHwTuHCqQrd3H3TpndrGVTGLZ/KOJm8Kk0RkbWNmuciIgmo0hQRSUCVpohIAo1Wmma2iZnNi/4tN7P3Yq/XK1ZQZtbPzF43s8VmNiSH/UfEYnvVzA7P8/rPm1mvJvZpY2YTohhnmFnXfK5ZLuUq4+ja65rZK2Y2KYd9S17GsX2PNzPPdf9KU8bv8R1m9qGZzctx/9PD/ma2yMx+kef1x5nZ0U3ss2P0/V1jZjktj9nohB3u/hHQKzr5cOBzd7+y1kWN1A2l73K5YFPMrDVwPfDfwHJgtpk96O5vNHHoFe4+2sx2Ap4ys808dpfLzNZ1928KEWPkTGC5u3c3s5OAvwAnFvD8JVGOMo65EJgPtM1x/1KXMWa2ETAYmF3I85ZSGcv4NuAG4OYEx9zp7heY2RbAfDOb7O7pFReLUMYrgXOBY3M9oFnNczPrbmYLzexOYAHQxcw+iX0+0MxujbY3N7OJZjbbzGaZ2Z5NnH5PYJG7v+Pua4D7gP65xubu8wEDNo7+0txoZrOAkWa2gZndHsUx18yOjGJsa2bjo79u9wNtcrhUf+COaPs+4Me5xlgNilzGmFk34BBgbNLYSljGACOjf2uSxlnpil3G7v4MsKo5sbn7cmAp0DVqZfzNzF4Abo9aKFdHcbxiZqdHMbYys/81s9fM7HGgUw7X+cDdZwM5V8T59GnuAFzj7j2B9xrZ71pglLv3Bo4DQiHsYWY31bP/1sC/Yq+XRe/lxMz2Br5091BYWwJ7uvtQYBjwqLvvDhwEXGVmbYBfAR+7ew9gBLBr7HxjG2iWpeN096+A1WbWIdc4q0SxyhhgNDAESDzmrVRlbGa7AZu5+9Tan7UgxSzjZjOz7kA34K1YnAe7+0mkWnkrojLeDTjHUt1jxwL/BfQETgX2jp3vMjPrV4jY8plPc0lUQzelD7C9Zdaq3tjM1nf3F4EX87h+bUPM7OfAZ8DxsffHx5ocfYHDzOx30es2QFdgf2AUgLvPNbMF4WB3P7WAMVabopRx1M/0L3efZ2Z9EsRTsjI2s1bAVVRhl0tClfY9PtHMDiSV2Z/u7p9E13zQ3cNypH2BHmY2MHrdHtiWVBnfHf0uLDOzp8NJ3f0PhQown0pzdWz7O1LNpSDe9DFg9ygby8V7QJfY6840/hcwuMLd61v7Nx6nAUe7+5L4DrFfhCRCnMst1Znezt0/aeKYalOsMt4bGGBmR0Xn2cjM7nD3QU0cV8oy7kAqY3kuOnYL4GEzO9zd5yY9WQUrVhk3153uXt8NmdplPNjdn4jvYGbHFDWySEGGHEU1+8dmtm30Fzoe/DTgnPCigaZu3Eygp5l1M7PvkWoKTI6OHRX6qJppKqlO3xBLaKI9C/wsem8XYMcczjUZCF/y44DH8oir4hWyjN19qLt3dvca4CTgsVBhVkoZu/sqd+/k7jVRnLOBfi2swsxS4O9xg8zsfDM7q/mRMhUYbGbrRufb3szWJ1XGx0d9m1sDB+RxjQYVcpzmb0n9x0wn1Q8ZnAPsE3XYLgTOgIb7Qtz9a+A84HFgITDO3V+PPt6Z1B315roUaGepISsLgOHR+9cDm5jZIuASIP3FaKRP82ZgSzNbTKq/7KI84qoWBSnjJlRSGa+NClbGZjYeeI5UErQs6loB6AF8lEeMY4A3gXlmNh+4kVSreQLwLql6YywwIxZLvX2aZtbZzJaRqnOGR3E2Opqjap49t1Qb6RF31xKVLZTKeO1gZv8A+hd6eFipVE2lKSJSCfQYpYhIAqo0RUQSUKUpIpJAPuM06dSpk9fU1BQolOowZ86clWvTrN4q45ZPZZxMXpVmTU0Ns2dX7TwGzWJma9WyACrjlk9lnIya5yIiCajSFBFJQJWmiEgCqjRFRBJQpSkikoAqTRGRBFRpiogkkNc4TRGRYvj4448BePfddxvcp1u3bgBcc801AOy0004AbLfddgDssssuRYlNmaaISAIVlWmuWLECgOOOOw6AvfdOrYt05plnAqknFwrh008/BeDZZ58F4NBDU9M3tm7duiDnF5FkpkyZAsBDDz0EwNNPPw3Am2++2eAx22+/PQBLly4FYM2a7AVDv/uu0CtOpyjTFBFJoOyZZui7ANhxx9SyLSET3HzzzYHCZ5g//OEPAVi5MrUGfXjudtttty3IdSR3//73vwH43e9Si0cuWJBaJHLatGnpfdQCaBmWLEmtdXfDDTcAcPPNN6c/++KLLwBIMin666+/3vRORaBMU0QkgbJlmiHLC/2XAB99lFpr6ZxzUoveXXfddQW95ogRIwB4++23gcxfOmWYpTdu3DgALr74YqDuXdKQgQJssskmpQtMimbZstQ6baNH17cKc+522GEHIHO3vNSUaYqIJFC2TPOf//wnkLlLFjds2LCCXWf+/Pnp7SuvvBKAY45JLed8/PHHF+w6kpuQbfz6178GMi2O1EKUGeeem166nOuvvx6Ajh07liJEaYZQjpDJJPfdd18gMzplvfXWA6B9+/YAbLDBBuljPv/8cwB+/OMfA5ksco899gBg1113Te+7/vrrA9CuXbsC/1fkRpmmiEgCqjRFRBIoefM8DGC///7763x22223AbDppvkvzxKa5YccckidzwYMGADAhhtumPd1JJnQRRJu+jXknnvuSW8/8sgjQOamUWi6h+aelM/q1auB7O/Zyy+/DMCkSZOy9t1rr70AmDt3LpA9lDDcCOzcuTMArVpVbj5XuZGJiFSgkmea//M//wNkhpyEgeYAP/3pTwt2neeffx6A5cuXp9879dRTATjppJMKdh1p2jvvZNawGjt2bNZnYVKF8CDD448/Xuf48FBCyFJPPPFEALbYYovCBys5+eqrrwD42c9+BmSyS4CLLroIgD59+tR7bH0Pq3Tt2rXAERaPMk0RkQRKnmmGoSXh59Zbb53+LJ8+qvAY1siRI4HMo1rxoSyhz1RKa968eentMGh9//33B+CZZ54B4MsvvwTgrrvuAuAvf/lL+pjFixcDmVZD//79gUxfp4YilU4YGhS+Z2GCjfh9iCFDhgDQtm3bEkdXGso0RUQSKPuEHWFKKIC+ffsC0KFDBwDOPvvsJo8Pg+PDz5kzZ2Z9Xsh+Umme+JRdIfMPg9uDNm3aAPCLX/wCgAkTJqQ/CxM9hMkcQgaju+elF+6IX3755UBmIuDnnnsuvU8YvN5SKdMUEUmg5Jnm+eefD8CTTz4JwPvvv5/+LPRvhYziwQcfbPJ8Yd/aj+Fts802QKbvRcrn7rvvrvPeP/7xDwCOPvroeo8J0/XVZ8899wSyH8OT0pg+fXrW6/B4YxhfuTZQpikikkDJM80f/ehHALz66qtA9p3VRx99FIBRo0YBsNlmmwEwaNCgBs938sknA7DzzjtnvR+WyggZp5TPCSeckN4OrYeXXnoJgNdeew3I/D488MADQPbk1KGPO7wXpvQLZd+zZ8+ixS7Z4n3NkBnBcOmll6bfO+qoo4DsSTZaEmWaIiIJqNIUEUnAkqzJUVvv3r29sQ77UnjrrbeATDO8V69eADz22GNAYSb/iDOzOe7eu6AnrWCFKONVq1alt0M5hUcjG7qRF58AIjyocMQRRwDwxhtvAJlVSm+66aa84qtNZdyw2g+n1GedddYB4KyzzgIyc2L+61//AqB79+5AZk2wuLBGVJjco1g3mPIpY2WaIiIJlH1we77+9Kc/AZm/fOEmUqEzTGm++GOO48ePB+DYY48F6mac5513HgB//etf08eEge9hSr/wiOXUqVOBzOB30I2/YvvNb34DwFVXXdXgPt9++y2QaSGEn0mEm8AHHnggkD1VYLkp0xQRSaAqM82QrQDccccdAGy00UaAVi6sdGG6sDB0JUzQEYYVhZZDyC7jLrnkEgAWLVoEZIYvhWMg8/sgxREenwyryIZp+r7++uv0PmEdqJBxNkeYrDx81+MrT4bJqMtFmaaISAJVmWmGAbVxhx9+OJA9qbFUrpBxNjRRbX3CKoRhFdGQaT711FPpfcKdek0XVxzhzvhuu+0GZEYyxD3xxBNAJvscPnw4ALNmzUp8vdDXPWfOnMTHFosyTRGRBKo+0wxrH4e7etLyhf60yZMnA9l3VsMa6cOGDSt9YALAwQcfnPU6PCodMs3WrVsDmeVnAM444wwArrnmGiDT112JlGmKiCSgSlNEJIGqap6Hx+XiK0yGVQx1A2jtEdbEHjp0KJC9vna46TBw4EAAtttuu9IGJ3WEFRnCKpXhBlGYrQrgzTffBDIrMNQWX0us3JRpiogkUJWZZnyygH79+mXt89lnnwGZuReraT1lSSZMzvLnP/85/V64Ifj73/8egHHjxgGZ4UpSej169AAyQ8XuvffeOvvEh40BrLtuqmoKQwnjj9XY76NRAAAG3klEQVSWmzJNEZEEqirTrE/4ixQyijBkITx2pcfqWr5TTjklvT1mzBgAJk6cCGT6ymrP7C+lE7L80aNHA5nWYHzA+gcffABATU0NkCnT0EddSZRpiogkUPWZ5i233ALArbfeCsDpp58OZCZ3kJYvPg3gtGnTgMx63GGCiUoeLL22CCNdpkyZAsDf//739GczZswAMpllmBquEinTFBFJoKoyzeuuuw6AP/7xj+n39t9/fwDOPvtsADbeeGMA1ltvvRJHJ5UgjJYIy2WERy0XLlwIaOXKShJWE629XemUaYqIJFBVmeZ+++0HwJNPPlnmSKTShUmOd9llFwAWL14MKNOU/CnTFBFJQJWmiEgCVdU8F8lVWDPq7bffLnMk0tIo0xQRSUCVpohIAqo0RUQSsLDaW7MONvsQeKdw4VSFbu6+adO7tQwq45ZPZZxMXpWmiMjaRs1zEZEEVGmKiCTQaKVpZpuY2bzo33Izey/2uqgzYpjZumb2iplNymHfEbHYXjWzw/O89vNm1ivHfY83M891/0pTrjI2swvNbEH079wc9j/dzD6M4lpkZr/I8/rjzOzoJvbpaGaTo9/DF82sKp/BLGMZL4u+j/PM7MUc9i9HGfcxs09j/z/+0NR5Gx3c7u4fAb2ikw8HPnf3K2td1Ej1jX7X1MUSuhCYD7TNcf8r3H20me0EPGVmm3msw9bM1nX3bwoZoJltBAwGZhfyvKVUjjKO/sAMAnoD3wCPmdkUd29qJPqd7n6BmW0BzDezye6+MnbeQpfxJcCL7n6Ume0I/D/gkAKevyTK/D3ez90/SbB/qcsY4Cl3b7RyjWtW89zMupvZQjO7E1gAdDGzT2KfDzSzW6Ptzc1sopnNNrNZZrZnDufvRuqXc2zS2Nx9PmDAxtFfmhvNbBYw0sw2MLPbozjmmtmR0fXamtn46K/b/UCbHC83Mvq3Jmmcla7IZdwDmOnuX7j718CzwDG5xubuy4GlQNeolfE3M3sBuD1qoVwdxfGKmZ0exdjKzP7XzF4zs8eBTjlcqifwZHTNBcB2ZrZJrnFWumJ/j/NRwjJOLJ8+zR2Aa9y9J/BeI/tdC4xy997AcUAohD3M7KYGjhkNDAES39o3s72BL919VfTWlsCe7j4UGAY86u67AwcBV5lZG+BXwMfu3gMYAewaO99Yq6fpbWa7AZu5+9SkMVaRYpXxq8ABlmr+tgMOA7rkGpSZdQe6AW/F4jzY3U8CzgRWRGW8G3COmXUFjgX+i1RFeCqwd+x8l5lZ9rKmKS8DA6J99gI6R/9akmJ+jx140szmmNlpSYIqYRkD7GtmL5vZw5ZDF0w+z54vcfdcmqV9gO0ts+zuxma2vru/CNTp54j6IP7l7vPMrE+CeIaY2c+Bz4DjY++PjzU5+gKHmdnvotdtgK7A/sAoAHefa2YLwsHufmo9MbYCrgJOTBBfNSpKGbv7fDO7GpgGfA7MBb7N4TonmtmBpDL70939k+iaD7r7l9E+fYEeZjYwet0e2JZUGd8d/S4sM7OnY/E01I91GXCtmc0jVYG+nGOc1aQoZRzZ093fi5raj5vZInef3sR1Sl3GLwE17v551PKcSKqCblA+lebq2PZ3pJrEQbx5a8Du7v5VjufdGxhgZkdF59nIzO5w90FNHHeFu49uIk4Djnb3JfEdYr8IuepA6q/Zc9GxWwAPm9nh7j436ckqWLHKGHe/GbgZwMxGAYtzOOxOd7+giTgNGOzuT8R3MLOcm/+xGD8l1fca/lAuBVraDCDFLOP3op/LzexBYHegqUqzHGUcth+KuvM6NNYPW5AhR1HN/rGZbRv9csWDnwacE17U19Stda6h7t7Z3WuAk4DHQoVpZqNCP2QzTQXSd2rNLDTDnwV+Fr23C7BjEzGucvdO7l4TxTkb6NfCKswshSzjaJ/Nop81wFHAPdHr883srDxCnQoMNrN1o/Ntb2brkyrj46N+r62BA3KIsYOZtY5e/hKY5u6rGzummhWyjC11/2CDaLsdqXsU86PXlVTGW8S29wS+aerGVSHHaf6W1H/MdGBZ7P1zgH2iDtuFwBlRgI31hTRkZ2B5HjFeCrSz1DCIBcDw6P3rgU3MbBGpO6bpyq+hPs21VCHLeFK07yTgLHf/d/R+D+CjPGIcA7wJzDOz+cCNpFpUE4B3gYWkbjDOCAc00t/1A2Chmb0OHExqREdLV6gy3hJ4wcxeBmYBD7j7tOizSirjgZYa9jYPuIbsrr16Vc1jlJZqBz/i7oeWOxYpHjP7B9C/CMNKpEJUexlXTaUpIlIJ9BiliEgCqjRFRBJQpSkikoAqTRGRBFRpiogkoEpTRCQBVZoiIgn8f6S1hmwDdPmXAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fab423855c0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_images(images=images,\n",
    "            cls_true=cls_true,\n",
    "            cls_pred=cls_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examples of Mis-Classified Images\n",
    "\n",
    "We can plot some examples of mis-classified images from the test-set.\n",
    "\n",
    "First we get the predicted classes for all the images in the test-set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = model.predict(x=data.x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we convert the predicted class-numbers from One-Hot encoded arrays to integers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "cls_pred = np.argmax(y_pred, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot some of the mis-classified images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAD5CAYAAACj3GcTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XncVeP+//HXJ6GBKCGablPE4RQJGTIdjuGoTCWdBiehTvQ1VKafzHQMyZyoVDqGQnHSycnw0KGUkiaUIUUTkukgXb8/9rr22ve492rvew/3/X4+HvfjXnvtNXzurva1P+ta17ouc84hIiKpqZHrAERECokqTRGRCFRpiohEoEpTRCQCVZoiIhGo0hQRiUCVpohIBKo0RUQiUKUpIhJBzXR2btiwoSsqKspQKIVh7ty5651zO+c6jmxRGVd9KuNo0qo0i4qKmDNnTjqHKDhm9nmuY8gmlXHVpzKORpfnIiIRqNIUEYlAlaaISASqNEVEIlClKSISQVp3z0XS9cEHHwBw/PHHA7B+/XoA3n333fg2bdq0yX5gIuVQpikiEoEyTcmJv/3tbwCMHTsWgE2bNgHQokULABo1apSbwESSUKYpIhKBMk3JiWnTpgGlM8xXXnkFgCZNmuQmMJEklGmKiERQEJmmv8N6//33AzB79uz4e0uXLgWgfv36AKxevbrYvldddVV8eejQoZUapyTXt29fANasWQPAvvvuC8DUqVOB2HPQUrX4nhAjRowA4KOPPgJgr732im9z5plnAnDYYYcBsPPO+TteijJNEZEI8jrT9Flkjx49AJg3b16525bMML2XXnopvtyvXz8AmjdvnqkQJaJnn30WgN9//x2AZ555BlCGWRV99dVXAJx99tkArFixAoCaNWPVzptvvhnfdtSoUUDYJ/eee+4B4Oijj85OsBEo0xQRiSAvM81vv/0WgHPPPRcI2zQr0qBBAwC++eabYuuXLFkSX37yyScBuP766zMSp6TmiSeeiC9v2LABgM6dOwPQsmXLMvf58ssv48szZswo9p5/emj33XfPaJySWTVqxHKy77//HoAdd9wRgAkTJgDh/wWAq6++GiA+rufkyZMBZZoiIgVPlaaISAR5eXn+/PPPA+Vfll900UXx5QEDBgBQr149AG6++WYAHnnkkVL7LVq0KKNxSmo2btwYX/Y3gHzXEn9TwHc5uvPOOwH45JNP4vusXLmy2PF8x/e6desC0LBhw/h7l156KRDeUNhjjz0y9FdIVLvuuisQXmL7S25f5l26dIlve9RRRwHw0EMPAfDwww8D0K5dOwA6deqUhYhTo0xTRCSCvMw0X3311TLXH3rooQD8/e9/j6/bb7/9APjpp58AeOedd8o9ru9UK9n14IMPllrnbwT5LmHnnHMOAL/88kvS45XMPD/88MP48syZMwHYf//9ix1fXZpy5/zzzwfCTNMP1jJkyJD4Nv699957D4Aff/yx2O98okxTRCSCvMw0/+///g+Ap59+GoDNmzcDYWf3xMxi3LhxACxcuBCA+fPnl3vcfGoXqQ7GjBkDwGeffVbqvbvuugsI2699hunbtq688sr4to0bN67wPP7/CcBTTz0FwOLFi4Hw0du77747cvySGXXq1AHAzICwk/sFF1xQattatWoBMHLkSAC6deuWjRAjUaYpIhJBXmaavu2yffv2ALz22mtA2EnWP5YV1THHHJOB6CRVflAOf8c80b333lvstb/TOnr0aAD23HPPlM9zyCGHxJdPO+00AI477jgAHnjgAQAOP/xwIGw7lew5/fTTAXjuueeA8GrwlltuiW/jnAPCz3737t2zGWIkyjRFRCLIy0zTmz59OhAOLeX7Zy5YsCDlY7Rt2za+rEwz/+yyyy4A/POf/wSiZZhl2WeffYAwc/XZru9VoUwzd/zwb/73HXfcEX/vt99+A/I7w/SUaYqIRJDXmeZWW20FhO1R/qH+xDurq1atKnNfPzhA//794+v83TvJPT/I7MUXXwyE7dfp8oN4+MzVt20+/vjjgO6i57utt9461yEkpUxTRCQCVZoiIhHk9eV5Sd999x0A69evT7ptPo/HJ6UvnzNt7733rpTjSvq+/vprIOxmlGinnXbKdjiRKdMUEYmgIDJN/6jkZZddBpQ9qIO/yeOHmzryyCOzFJ1sCT/QSmVJ5WpEcmPKlClAOOc9wLbbbguEHeHzmTJNEZEI8jrT9A/t+zl9ysowfWflI444AggH+5DqyXeSvu2224qtP+uss3IRjpShrJljyxq8I18p0xQRiSAvM00/pFifPn2A0nfZdtttt/jyiBEjgLAzu+SPU045BYAbbrghvs5fLYwfPx4IB6StX79+Rs7ph4Tz86tvv/32AFxxxRUZOb5sOX/X/NFHHy31XiE93qpMU0QkgrzKNH2Ged111wFl9+MC6NmzZ3xZGWb+OvDAA4Hi7Yl+kOCBAwcCYbu1n8LEP1bpJ9+qiL/7umzZsvg6P42G54cR9NNfSO6sXbsWKHtQ6h122CHL0Ww5ZZoiIhGo0hQRiSDnl+eJM0Ree+21QPkjF/lRnS+//PLKD0wyJvEmjH8IYeLEiUBY/n6+8lmzZgHF5zIv6fjjjwfgmWeeAcKbShA+hjds2DCg8h7TlOpLmaaISAQ5zzQvvPDC+HKyDPPll18GKs5CJP+0bt06vjx27FggvBE0dOhQoOyssTz33XdfsdeNGjWKL/u5tH13NZFMU6YpIhJBzjJNPwjH7Nmzy93Gd0weMGAAEI72LYXPd0fymacflf/2228HKs44mzRpAoTZZGKXppYtW2Y+WJEEyjRFRCLIeqa5fPlyAPr27QvA//73v3K37d27NwBdu3at/MAkp3znc595+t8i+UaZpohIBFnPNPfaay8AGjduDMDSpUtLbXPSSScBMGjQoOwFJiKVys9F36ZNGwDmzJkTf88PGn7UUUcBMH369CxHlzplmiIiEeTs7nlRURFQPNOsVasWEA7c4b+ZRKTwNWjQAIB//etfQDhHPYRDBvqBW/KZMk0RkQhUaYqIRJCzy/OpU6fm6tQikkP+IRU/n1OhUaYpIhKBKk0RkQhUaYqIRGDlzcOT0s5m64DPMxdOQWjunKs2I4eojKs+lXE0aVWaIiLVjS7PRUQiUKUpIhJBhZWmme1kZvODn9Vmtirh9TaVFZSZXW5mi4Kf/ils39vM1gVxLTGzC9I8/zgz65hkm8EJ/xaLzGyTmRXO5M2BHJZxAzObZGZLgzJrm2T7XJSxmdlDZrbMzBaYWat0zpkr+hxXuM0BZva2mf1iZgNSOrBzLqUfYAhwZRnrDaiR6nFSOE8r4H2gNrA18BqwR5J9egPDguVGwHqgYYltakaIYRzQMcL2nYB/Z+rfIFc/2Srj4JjjgZ7B8jbADvlWxsAZwJRg+ShgZq7LqFDKuFA+x8CuQBvgDmBAKsfdostzM9vbzBab2XhgEdDUzDYkvN/FzEYGy7sGGcUcM5ttZocnOXxL4B3n3M/Oud+AN4lVSilxzq0GPgOamdktZvakmc0ERptZTTO7J4hjgZn1DmKsEWQUS81sOhB15rbzgAkR98lrlVnGZtYAOMw5NxrAOferc+67VGPLYhl3AJ4MzvkW0MjMqsxddX2OwTm3xjk3B9iUamzptGnuB9zrnNsfKHsayZjhwFDnXBvgXMAXwmFm9kgZ238AtA8u3+oCpwBNUw3KzPYGmgOfJMR5gnOuG9AHWOucawscCvQzs2bA2cAewP5AL6BdwvFuNbNTKzjfdsCJwKRUYywglVXGewLrgg/CPDMbYWZ1Ug0qi2XcGPgi4fXKYF1Vos9xROk8e748qKGTORHY18z86/pmVts5NwuYVXJj59xCM7sHeBX4AZgH/J7Cec43s2OBX4DezrkNwTlfdM75OTVOAlqaWZfg9Q7APsAxwATn3GZgpZm9nhDPtUnO2wF4I0qmVEAqpYyJ/b9rA/QH5gL3A1cBNyY5T67KuCrT5ziidCrNHxOWNxNrE/FqJSwb0NY592uqB3bOjQBGAJjZUGBZCruNd86V1ZCbGKcBfZ1z/0ncwMxSvmwoQxegqk5oU1llvBJY4T+sZjYRSKURPttlvIpYdvRO8LoJFWdjhUif44gy0uUoqNm/NbN9zKwGxdsuXgX6+ReWwh1IM9sl+F1ErDH+n8Hry8zs4jRCnQb0NbOawfH2NbPaxNpbOgdtIo2B9qkczMzqE7sEmJJGTAUhk2XsnFsJrAkuwQBOABYH++ZTGU8GugfHOQpY45xbl0Zsea26fo6jymQ/zUHE/pj/EsskvH7AkUGD7WLgQqiwLQTghWDbF4CLnXMbg/Utga/TiPFR4GNgvpktBB4mlm0/B6wg9sEdBbztd0jSFnIWMNU593MaMRWSTJZxf+BpM1sAHEDs7iXkVxlPAVaZ2fLgOP3K2KaqqVafYzNrYmYrgUuBIWa2Mln7ekE9RmlmLwMdnHMp3+mSwqIyrvoKvYwLqtIUEck1PUYpIhKBKk0RkQhUaYqIRKBKU0QkgrRmo2zYsKErKirKUCiFYe7cuetdNRrVW2Vc9amMo0mr0iwqKmLOnFSewKo6zKxaTQugMq76VMbR6PJcRCQCVZoiIhGo0hQRiUCVpohIBGndCBLJho8++giAiy66CICuXbsCcOGFF+YsJqm+lGmKiESgTFPyks8uAU477TQAPvkkNvPBZ599BijTlNxQpikiEoEyTckr9913HwDDhg2Lr1uxYkWxbZo3b57VmCQ9w4cPB+DSSy/NcSSZoUxTRCSCvMw0N2yITb388ccfA/DUU08Vez8xC0mYHa+YRo0aAfD22/ER75Wh5LFNm2KDeC9evBiAzz8Pn3LzZdyiRQsAxo0bl+XoJIoff4zNgTZ48GAAPv30U0CZpohItZRXmabPIG677TYAPvzwwzK3S8wu//jHPwLw22+/AbBkyRIA1qxZA8Dq1avj2yrTzF+PPBKbm2vkyJHlbtOwYUMAmjRpkpWYZMv4zPLBBx8EYPbs2bkMJ+OUaYqIRJDzTDOxvfKSSy4B4KeffgKgQYMGAJx55plAmFUec8wx8X189ujbxJo2bQrAzz//XOr4hx12WOb/AEnLl19+CcDjjz8OgJ/or6wJ//7xj39kLzDZYpdddhkABx54IAC1atXKZTgZp0xTRCQCVZoiIhHk7PLcX4InNvwfcsghAFx33XUAHHnkkQDUrl076fH85XjJLkjnnHNO+sFKpfFdixYsWACU3YXsjDPOAODggw/OXmASyfTp0+PLv//+OwDvv/9+5OMsX74cCLsd+jrhtddei28zc+bMMvf1zXd/+ctfIp83CmWaIiIR5CzTrFOnDgAzZszIyPHuvvtuIMxg99lnHwBatmyZkeNL5dh+++2BsDvR+vXrS23jH1Dwg3j84Q9/yFJ0kqpXXnklvlyjRsW5mL/517Fjx1Lvbdy4EYBffvkFgMaNGwPF/18kDuaSaOedY/Ok+ZvDldXVSZmmiEgEOe9ylK53330XgDvvvLPYet99aaeddsp6TJI6nzX6rKOszu0+y3jooYeK/Zbc81mjb5OGsAz9DJfNmjUDYJdddgHgggsuAMKsEsIuZv7Raa9Hjx5A2E4KcOutt5YZy7p16wBo27btlvwpKVOmKSISQUFmmps3b44vT5s2DQjbMnfYYQcAjjvuuOwHJlvM95io6DHKKVOmAOG0F/5uqeROt27dAHj99dfj63z5+CH9xo8fD4SZZt26dQF49tln4/v4TNM//uz5B1m++OKL+LrnnnsOCB/X/PXXXwH485//DMATTzyR1t+UjDJNEZEICjLT9I/cAdxwww3F3rvjjjsAOOigg7Iak6THP/7qH8G79957S22zatUqIOy3mTh8nGTXrFmzgLDdsnXr1vH3/GfQ92jxj0N7EydOjHy+vffeO77s74r369cPCAf62W233YDwLnplUaYpIhJBQWaaL730Uql1/g6dv9smhclfObRp0ya+zreR+XZrP9yfH9TW340FaNWqVVbirO4effRRAH744QcgbNuE8CmekoOHZ4pv98zVYNTKNEVEIlClKSISQUFdns+bNw8Iu55AOMDDVVddBcC2226b/cAkY3yXsa5du8bX+a4pvluL7xTtRwb3XVAgHCSism8GVFc33XQTEF4aH3300QD079+/Us87ZMiQ+LJ/kGXAgAFAONPDVlttVakxeMo0RUQiKIhM089u579tEkf1PuGEEwDo27dv1uOS7Hj++eeB8OZDybJOnAfKd3SWyuFv1PkrPD84R82alVOVXHPNNUDxoecGDRoEwMknnwxkf2R4ZZoiIhEURKY5atQoIOxqlDgoca9evXISk2SfHljIP759OTHbb9So0RYfz3eWf/jhhwEYO3YsEHZcB+jevTsAe+655xafJx3KNEVEIsjrTNMPE3XttdcWW+/vlEPxu6ySn954441S69q3b5/y/o899hgQ3iUtOVNlWTNXSnb4Hi2JD5VMmDABKP34ZEmJw8n5HhJDhw4F4NRTTwXC+xiJM9DmKsP0lGmKiESQl5mmzxxuv/12IHxUyzv99NOzHpNE5weo7dChQ3ydzxjWrl1b5j6TJ08Gimen/rE5P7e9v3PrH5n0+0B67WmSnB84ww/469s0E+9ud+nSBQgHi/ZXhsuWLSt2rMRBiP0jsXPnzgVg9913B5Jnq7mgTFNEJAJVmiIiEeTl5bkfb2/MmDHF1vfs2ROAQw89NNshyRbw87p8//338XW+29jLL79c4b6JN3f85Xi9evWA8DE6P791YncUqVz+5qwfWWq77bYD4J133olv8+qrrwLQokWLMo+x9dZbA+HYqRA+pFIIM40q0xQRiSAvM83y5jX288hU5Omnnwagc+fOGY1JovMDKPgMEeC7775LaV8/kjuEo4L7zETzP+Xe4MGDgbD7j58PCMKR9UvO9+PdeOONQOE++qxMU0QkgrzMNP2jVN71118PhKOz//LLL/H3Jk2aBMDNN98MwP3335+NECUFvtvICy+8EF/nO0N7w4cPB+DYY48Fwkcl/bBfkp9KtlcmdjhfuHBhtsPJKmWaIiIR5GWm+fbbbxd7/c033wCwePFiAM4///z4e35GQv+oZZTH8yQ7EsukZPkoo5RCo0xTRCSCvMw0O3XqBISDzvppDfzvxD58ffr0AWDgwIHZDFFEqillmiIiEeRlpun7cc2cORMI78b5ARoS+2v6Ie9FRLJBmaaISASqNEVEIsjLy3M/Z7Wfw1pEJF8o0xQRiUCVpohIBKo0RUQisHRm8jOzdcDnmQunIDR3zu2c6yCyRWVc9amMo0mr0hQRqW50eS4iEoEqTRGRCFRpiohEUGGlaWY7mdn84Ge1ma1KeL1NZQRkZs3N7HUzW2xmi8zs7yns09vM1gVxLTGzC9KMYZyZdUyyzeCEf4tFZrbJzHZI57y5kIsyDs670sw+CM4zK4Xtc1HGZ5rZguCc75pZu3TOmSs5LOMGZjbJzJYGZdY2yfa5KGMzs4fMbFlQ1q2SHtg5l9IPMAS4soz1BtRI9TgpnGd3oFWwXA9YDrRIsk9vYFiw3AhYDzQssU3NCDGMAzpG2L4T8O9M/Rvk6idbZRwccyWwY4Tts17GwHaEN0sPBhbmuowKrIzHAz2D5W2AHfKwjM8ApgTLRwEzkx13iy7PzWzvIBMcDywCmprZhoT3u5jZyGB51+DbZo6ZzTazwys6tnPuS+fc/GB5I7AUaJxqbM651cBnQDMzu8XMnjSzmcBoM6tpZvcEcSwws95BjDWCb5ulZjYdaBjpHwTOAyZE3CevVWYZpytbZeyc+8EFnyagLlCluppUZhmbWQPgMOfcaADn3K/OudSmIiWrn+MOwJPBOd8CGplZhV2R0mnT3A+41zm3P7Cqgu2GA0Odc22AcwFfCIeZ2SMVncDM9gT+ALybalBmtjfQHPgkIc4TnHPdgD7AWudcW+BQoJ+ZNQPOBvYA9gd6Ae0SjnermZ1awfm2A04EJqUaYwGpzDJ2wAwzm2tmf4sSVDbL2MzONrMPgReIZUJVTWWV8Z7AuqCym2dmI8ysTqpBZbGMGwNfJLxeSZIkLZ0BO5Y75+Yk34wTgX3NzL+ub2a1nXOzgHLbssysHjAR6O+c+yGF85xvZscCvwC9nXMbgnO+6Jz7X7DNSUBLM+sSvN4B2Ac4BpjgnNsMrDSz1/1BnXPXJjlvB+CNKN+iBaQyy/hw59wqM2sETDezJc65/yY5T9bL2Dn3HPCcmR0H3BwcvyqprDKuCbQB+gNzgfuBq4Abk5wnV5/jlKVTaf6YsLyZWJuIVyth2YC2zrlfUz2wxRqnJwGjnHOTU9xtvHOurFm6EuM0oK9z7j8lztcp1djK0AUYm8b++azSytg5tyr4vdrMXgTaAskqzVyVMc6518xsjJnt6JzbkHyPglFZZbwSWOErZDObCKQyi162y3gV0BR4J3jdhIoz7sx0OQpq9m/NbB8zq0Hsxoj3KtDPv7Akd6cs9rUyGpjvnBte4r3LzOziNEKdBvQ1s5rB8fY1s9rAm0DnoE2kMZDSlJZmVp/YJcCUNGIqCBku4+2CZg3MrC7wJ2Bh8Dpvyjho87NguQ2xm0JVqcIsJpNl7JxbCawJLrMBTgAWB/vmTRkDk4HuwXGOAtY459ZVtEMm+2kOIvbH/JfYt4zXDzgyaLBdDFwYBFheW0h7YjdW/mRhtwg/p0VL4Os0YnwU+BiYb2YLgYeJZdvPASuIFeooID6HcJI2zbOAqc65n9OIqZBkqox3A2aa2fvAbOB559yrwXv5VMbnAgvNbD6xNr3OacRVKDJVxhC7NH/azBYABwB3BOvzqYynAKvMbHlwnH5lbFNMQT17bmYvAx2cc5tyHYtUDpVx1VfoZVxQlaaISK7pMUoRkQhUaYqIRKBKU0QkgrRmo2zYsKErKirKUCiFYe7cuetdNRrVW2Vc9amMo0mr0iwqKmLOnFQeJqg6zKxaTQugMq76VMbR6PJcRCQCVZoiIhGo0hQRiUCVpohIBKo0RUQiSOvueT6YO3cuACeeeCIAO+64IwDTpk0DoEWLFrkJTESqJGWaIiIRFFSm+dNPPwFw0UUXxde99NJLAGzcuLHY73POOQeA999/P5shikiKEkaB58wzzwTwk51xwAEHAHDzzTdnP7AklGmKiERQEJnmBx98AMAFF8SmQX7vvffi7/lvpsRvLYBjjz02O8GJyBZJ/My+8MILQPh5fvHFFwFo3bo1EGai+UCZpohIBHmdaX755ZcADBs2DCieYSYzatQoAA499ND4um7dumUwOsmETz/9FIBjjjkGgLfeeguA5s2b5ywmyY5HHik9S8Z1110HwPr16wG4/fbbAWWaIiIFK68zzTvuiM3DNHr06Mj7/vhjbMbPHj16xNf5TLVVq9hEet27d08zQknXjBkzAFi5MjaH1xtvvAFEK5t58+bFlx999FEA2rePTUR43nnnZSROybw+ffqUWuc/o4899li2w0mZMk0RkQjyMtP0T/mMHTsWCO+olSXKxHD33nsvAJ07x2ZiVaZZ2PxTX4nZ5LfffguE/XWVaRYm/7k++uijcxxJaco0RUQiUKUpIhJBXl6eP/DAA0B4iVWy47q/kQMwefJkIHxc0ndP+s9//lPu8V9++WUg7JbUq1evTIQtW8CXRRRTp04FwkdlE/9/DB8+HIBLLrkkA9FJtj3//PNAWKadOnXKZThlUqYpIhJBXmaaY8aMAUpnmG3btgVg0qRJ8XW77bYbAI0bNwagfv36QMWZZp06dQDYddddMxSxbKnvvvsu5W3Xrl0LQN++fYGwW1nizZ7+/ftnMDrJNp9ZjhgxAtCNIBGRgpeXmWZ5evfuDUDt2rXj677//nsANmzYAMDIkSOTHuf4448H4NRTT810iBJRzZrF/wuecMIJpbbZvHkzAIMHDwbgs88+A2DfffcFwnZsKSzr1q2LL/vHJX2b5v7775+TmFKhTFNEJIKCyjSvvPJKoPiD/ttttx0Ab775ZsrHOeOMMzIbmGwx3ybtffjhh0DYRg3w0EMPAWFvh2233RaAu+66C4Bddtml0uOU9H3++ecA7LzzzgCMGzcu/p6/WvD3G/zjtPlImaaISAR5lWlefvnlQPmPRvo7rf4xy8RtS95pL4tvL+nQoUNacUrm+KsH32PilVdeAWDVqlXxba655ppi+/jhw04//fRshCgZ4nu/3H333UA4IA+En19f1vvtt1+Wo0udMk0RkQhynmkmPrkxYcIEIPzWSSV79FLZVhlm/vETaPnpDvxgs/6OeSLfh8/fRZfC4PtV+362t912W7HXAC1btgRKX1XkI2WaIiIRqNIUEYkgZ5fnfobJiRMnxtf5juol1atXD4A777wTCDs3Q/HG5GSuvvpqAG688UYAttlmm9QDlkrhm1V800nHjh2B4o/Ken4c1JId4iW/LFmyBAg/2/5z68v67LPPBmDRokXxfXzzzC233AKEN/vykTJNEZEIcvaV/fDDDwPw9ddfl7tNu3btALjpppsAOO6440pt88MPPwDh3DCbNm0q93j+G2/77bcHCqPRubrwg2+8/vrrpd47//zzAd3Iy2e+4zrAtddeC4Rd/Px8Tf4KsWvXrkBY5hA+Nnn99dcDUFRUBOTnDLLKNEVEIsh6pjl//nwApkyZknRbPwRYWRmm5wedffHFF4FwVsOK+PZUyR9+/qZvvvkGgEaNGsXf82Vcq1at7AcmKUmcb8vPXe8fb73nnnsAaNasGQANGzYE4Kefforv47sc+W5lt956KxA+Vql5z0VEClTWM82PPvoIKP6YXEmtW7cGSg/d5u+uJw7/5h+9TKVzez7PcFdd/fzzz0A4xYl34YUXxpcbNGiQ1ZgkdX54t8QBc3wbZlnt04l8Fpno4IMPBsI77z5b9W2cidvkijJNEZEIsp5ppvKI5LJly4DwTppvr/z9998B+OqrryIdz/viiy+A4u1lklv9+vUDYM2aNUA4BcnAgQNzFpOkruREaJDZydDGjh0LwOLFi+PrlGmKiBQQVZoiIhHk5fNo/obPgw8+uMXH8KN7DxkyJL4ucTRwyS0/p70fP9Pzc8X4Efklv/nuQ/4mkveNAAAFoUlEQVQ3hA+aNG3aFNiy7kL+MdqzzjoLKH75n+sO78o0RUQiyHqmecQRRwDQokULIOyCtKX8t1mNGsXr/z59+gC6oZCvrrjiCiC8qderVy+g+Bzmkv98FrlixYr4Ot8lsEePHgAsXboUSO2xZT9gR8lBPvJpAA9lmiIiEWQ902zSpAkAPXv2BKINmuG/uQ466KD4ugEDBmQuOMmakiOz+xlC9ahkYUr8HJ588skAnHLKKUB41VeRv/71r0CYlfo2Uj93lB6jFBEpUDm7ez5o0KBiv6V68QMJt2rVCoBjjz02h9FIJvnBN3zH9PL4wYohHITYDxTus9PEu/L5QpmmiEgEedlPU6q+efPmAWHvhh133DGX4UglSDYwjs9IofypbvKRMk0RkQiUaUpO+DZMf7dUpFAo0xQRiUCVpohIBLo8l5wYOnRorkMQ2SLKNEVEIlClKSISgSpNEZEIzM/QuEU7m60DPs9cOAWhuXNu51wHkS0q46pPZRxNWpWmiEh1o8tzEZEIVGmKiERQYaVpZjuZ2fzgZ7WZrUp4vU1lBWVmV5rZIjNbaGbjzWzbJNvfkhDbB2Z2Wprnf8vMWiXZppaZPWdmy8zsbTNrls45cyVXZRycu6aZLTCzF1LYNhdlfJyZzTOzTWbWMZ3z5ZI+xxVuM9DMlpjZ+2Y23cyaJjtuhZWmc+5r51wr51wr4BHgXv/aOfdrcFIzs4xlrGbWHLgYOAQ4EKgFnJPCrv8I4jwPGG2J09fFjpvpjvx9gNXOub2BB4HbM3z8rMhFGSe4HFgYYftsl/FnQHfgmQwfN6v0Oa7QHOBg59wfgcnAHcl22KJ/JDPb28wWm9l4YBHQ1Mw2JLzfxcxGBsu7mtkkM5tjZrPN7PAUTrE1sX/kmkAd4MtUY3POLQQMqG9m48zsYTObDdxmZtuZ2eggjnlm9pcgxjpm9mzwjTMxOHcyHYAxwfIzwMmpxlgIKruMgw/Vn4BRUWPLVhk75z51zn0AbE62bSHS5xicczOccz8HL98BmiTbJ51vlv2IfWPtD6yqYLvhwFDnXBvgXMAXwmFm9kjJjZ1znwP3AV8AXwFrnXMzUg3KzNoB/3POfROs2g043Dk3EPh/wCvOubbA8cDdZlYL+DvwrXOuJXAL0DrheKPKSfEbBzESfFv/aGZVbVDISinjwDDgKiBy940slnF1UN0/x4n+BkxNFls6qe5y59ycFLY7Edg3Icuub2a1nXOzgFklNzaznYDTgT2AjcBEM+vinPtnkvNcZWY9ge+Bzgnrn3XO+UzhJOAUMxscvK4FNAOOAYYCOOfmmdkiv7NzrlcKf2NVVVll3BH4wjk338xOjBCPyjjz9DmOxduTWDPCpUniS6vS/DFheTOxVNpLTIsNaOvbTlJwEvCxc249gJk9D7QDkv1j/8M5NyxJnAZ0dM4tT9ygRLNJqlYBTYHVFmtMr+uc25Bkn0JTWWXcDjjTzM4IjlPPzMY453ok2S/bZVwdVPfPMWb2Z2JXPe1T+fsy0vAbfAN8a2b7BI3JnRLefhXolxBgshR5BXCEmdUOGoFPAJYE+w717RdbaBrQPyEWn76/CXQN1v0ROCCFY00G/If8XODfacSV9zJZxs65gc65Js65IqAb8G9fYeZZGVcr1fFzbGZtiN3IPcNX8Mlk8o7oIGJ/zH+BlQnr+wFHWqxryWLgwiDY8tpCZhKrkOYBHwCbgMeDtw8CVqcR441AXYt1Z1gEDAnWPwDsZGZLgOuDcxPEWV5byAhgNzNbRqwtJfUJ3AtXRso4ibwpYzM7wsxWEqs8RprZgjTiKhTV7XN8F1CXWPPB/CAjrlDBPEYZfFtNdc79OdexSOVQGVd9VaGMC6bSFBHJB3qMUkQkAlWaIiIRqNIUEYlAlaaISASqNEVEIlClKSISgSpNEZEI/j9csRQ8hgaB5wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fab42330208>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_example_errors(cls_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Functional Model\n",
    "\n",
    "The Keras API can also be used to construct more complicated networks using the Functional Model. This may look a little confusing at first, because each call to the Keras API will create and return an instance that is itself callable. It is not clear whether it is a function or an object - but we can call it as if it is a function. This allows us to build computational graphs that are more complex than the Sequential Model allows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create an input layer which is similar to a feed_dict in TensorFlow.\n",
    "# Note that the input-shape must be a tuple containing the image-size.\n",
    "inputs = Input(shape=(img_size_flat,))\n",
    "\n",
    "# Variable used for building the Neural Network.\n",
    "net = inputs\n",
    "\n",
    "# The input is an image as a flattened array with 784 elements.\n",
    "# But the convolutional layers expect images with shape (28, 28, 1)\n",
    "net = Reshape(img_shape_full)(net)\n",
    "\n",
    "# First convolutional layer with ReLU-activation and max-pooling.\n",
    "net = Conv2D(kernel_size=5, strides=1, filters=16, padding='same',\n",
    "             activation='relu', name='layer_conv1')(net)\n",
    "net = MaxPooling2D(pool_size=2, strides=2)(net)\n",
    "\n",
    "# Second convolutional layer with ReLU-activation and max-pooling.\n",
    "net = Conv2D(kernel_size=5, strides=1, filters=36, padding='same',\n",
    "             activation='relu', name='layer_conv2')(net)\n",
    "net = MaxPooling2D(pool_size=2, strides=2)(net)\n",
    "\n",
    "# Flatten the output of the conv-layer from 4-dim to 2-dim.\n",
    "net = Flatten()(net)\n",
    "\n",
    "# First fully-connected / dense layer with ReLU-activation.\n",
    "net = Dense(128, activation='relu')(net)\n",
    "\n",
    "# Last fully-connected / dense layer with softmax-activation\n",
    "# so it can be used for classification.\n",
    "net = Dense(num_classes, activation='softmax')(net)\n",
    "\n",
    "# Output of the Neural Network.\n",
    "outputs = net"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Compilation\n",
    "\n",
    "We have now defined the architecture of the model with its input and output. We now have to create a Keras model and compile it with a loss-function and optimizer, so it is ready for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.python.keras.models import Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a new instance of the Keras Functional Model. We give it the inputs and outputs of the Convolutional Neural Network that we constructed above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "model2 = Model(inputs=inputs, outputs=outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compile the Keras model using the RMSprop optimizer and with a loss-function for multiple categories. The only performance metric we are interested in is the classification accuracy, but you could use a list of metrics here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "model2.compile(optimizer='rmsprop',\n",
    "               loss='categorical_crossentropy',\n",
    "               metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training\n",
    "\n",
    "The model has now been defined and compiled so it can be trained using the same `fit()` function as used in the Sequential Model above. This also takes numpy-arrays as input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/1\n",
      "55000/55000 [==============================] - 2s 37us/step - loss: 0.1958 - acc: 0.9394\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7faac4a441d0>"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model2.fit(x=data.x_train,\n",
    "           y=data.y_train,\n",
    "           epochs=1, batch_size=128)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation\n",
    "\n",
    "Once the model has been trained we can evaluate its performance on the test-set. This is the same syntax as for the Sequential Model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10000/10000 [==============================] - 0s 36us/step\n"
     ]
    }
   ],
   "source": [
    "result = model2.evaluate(x=data.x_test,\n",
    "                         y=data.y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is a list of values, containing the loss-value and all the metrics we defined when we compiled the model. Note that 'accuracy' is now called 'acc' which is a small inconsistency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss 0.061494892170466484\n",
      "acc 0.9811\n"
     ]
    }
   ],
   "source": [
    "for name, value in zip(model2.metrics_names, result):\n",
    "    print(name, value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also print the classification accuracy as a percentage:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "acc: 98.11%\n"
     ]
    }
   ],
   "source": [
    "print(\"{0}: {1:.2%}\".format(model2.metrics_names[1], result[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examples of Mis-Classified Images\n",
    "\n",
    "We can plot some examples of mis-classified images from the test-set.\n",
    "\n",
    "First we get the predicted classes for all the images in the test-set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = model2.predict(x=data.x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we convert the predicted class-numbers from One-Hot encoded arrays to integers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "cls_pred = np.argmax(y_pred, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot some of the mis-classified images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAD5CAYAAACj3GcTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xu8lWP+//HXJ41URCqi0+bhlByKxDhUCEN0MtSUoYzMiMiMchrjEA2ZIWYQMmnEzPzSwSFJSfqWHEqhgzM1ZTpJKnJI1++PdV/rXmu3917r3mvtddi9n4/Hfux1uA+fuva61ue+ruu+LnPOISIi6amR7wBERIqJKk0RkQhUaYqIRKBKU0QkAlWaIiIRqNIUEYlAlaaISASqNEVEIlClKSISQc1Mdm7YsKErKSnJUijFYf78+eucc43yHUeuqIyrP5VxNBlVmiUlJcybNy+TQxQdM1uW7xhySWVc/amMo9HluYhIBKo0RUQiyOjyXCQXvv32WwB69eoFwP777w/AiBEj8haT7LiUaYqIRKBKU0QkAl2eS8FbsWIFAM899xwAtWvXBuDmm28GoH79+vkJTHZIyjRFRCLIeaa5efNmAP773/8C8NBDD223zcUXXwxA69atcxeYFI29994bgJ133jnPkey4vv/+ewDuvvtuAL744ov4e8uXLwdg8uTJaR/PXy3ceOONAAwaNAiAnXbaKfNgs0yZpohIBDnLNH2G6b+Zhg4dWu62I0eOBKBnz54A3HfffQDsueeeVRmiFIkzzzwTgLp16+Y5kh3XFVdcAcBjjz1W7jZmBkD79u2B2J1HAHPnzgXgo48+im+7YcMGAAYPHgzACy+8AMA///lPAJo0aZKt0DOmTFNEJIKcZZrDhg0D4M4770y57datWwF48sknAXj55ZcBePzxxwE4/fTTqyBCKVS+3btWrVpA2N4luTdw4EAAnnjiCQCuvvpqALp16xbf5uijj07ax7c916wZq25++OEHIPycA3z11VcA9OnTB4BZs2YB0KlTJwBeeuml+LbNmjXLxj+l0pRpiohEkLNMc7/99kt67ts7fNsIQKtWrYDwm+hPf/oTAKtWrQKga9euAFx77bXxfYYMGQJAnTp1qiJsyRPfAwvhFYYv44MOOigfIQlh26PvX/Cfxb322ivtY/jMM3H0gy/bmTNnAnDUUUcBsHDhQiBsxwaYOnUqkL92TmWaIiIR5CzTnDhxYtLz888/Hwh7xsty5JFHAtCjRw8AvvzySwBuu+22+DaffPIJAP/4xz8A+NnPfpaliCWfpk+fHn/ss5t02sOlavne7b59+wKwxx57VMl5fO95hw4dAFiyZEn8PT+O248D9W2luaJMU0QkAlWaIiIRmHOu0ju3bdvWpTtNvu/48b/fffddAA477LCU+7722msAXH/99QD83//933bb9O7dGwg7DaoqZTez+c65tlVy8AIUpYyzYc2aNUA4IBrg66+/BsJOAX8bZVVRGReOf//73wD069cv/pq/hdM34ZxyyimRj5tJGSvTFBGJIGctqH6Qqh+ovuuuu6a97/HHHw/A8OHDATjrrLPi7/lBsU899RQAXbp0AcKOJikuU6ZMAeCDDz6Iv3beeecBYYa5ZcsWIBwcvdtuu+UyRMkhP1u/v9EFwg4gPxD+f//7X05jUqYpIhJBzjLNli1bAmGmWZZRo0YBYdb429/+tsztfPslwAMPPJD03ocffphRnJIf33zzDRBO0JDI38DgM0uffaxevRoIh6eAJnWprtq1axd/7DNNPwTR93GcdNJJOYlFmaaISAQ5yzTbtk3uqPK959999138NX9Lpb+N0t9SFYWfquqQQw4B4LTTTou/t/vuu0c+nuTGvffeC8CMGTMAOPnkk+Pv+b8dP2nDs88+m7Svn9AalGkWsh9//BGAikbs+FEvNWok53N+mkgIlznxVx6ff/45oExTRKQg5SzT9FNH+TYrP7bKt0sB7LLLLkCYaVbGsmXLgLD3PHEij0cffRQIJ/7QJB/5t2jRIgAeeeSRpNf9rXIA69atA8Jpybx99tkHgMaNG1dliBKRH1f7n//8B4A33ngDgAkTJiS9X5bOnTsD4QQg/vmJJ54Y38bfuulvr801ZZoiIhHkLNOsV68eABdccEHS64njNceOHQvAuHHjAFi/fj0QbYGm0r799tv4Yz+u6/DDDwfCsV/p3JUk2eHbtV588UUALrvsMgBWrlyZtJ2fpAXCqcBKj4zw7V+Jk9n6u0X8hMWSG37SYID+/fsDyctZpKv0Z3306NFAOHlPIn9lmjhuOxeUaYqIRKBKU0Qkgpyve16Rs88+O+n3Tz/9BMCmTZuStkvsPPITgJSeOdoPS/DzbEI4gPq9994D4A9/+AMAd911F6B11qtKYsN/9+7dAXjllVcq3CedlSb9UKOmTZvGX2vevDkQ3iiROORMsm/jxo0AnHvuufHXfBOJbw5LHJgO0LFjRwC2bdsWf8136vnOI/+Z93OovvPOO9ud2/+NNGjQILN/RETKNEVEIiioTNMPLfEN/n6ijtKzQ6czW7SfET5xUKzvdPCZ5rRp04BwIK2fLEKyw2eY11xzTfy10hmm7wj02/gOw3/961/xbd566620z+k7h95++21AmWZV8zek+FsaAX7xi18A4YqVUZQeVuY7bRM/x4k3xOSDMk0RkQjynmk+99xz8cdXXXUVEE715Ccg9YPRK8NnqwCzZ88GwpXu/PpCc+fOBcJhMP6bUirHDyvy2aNvXyzLrbfeCsDvf/97IMwihg4dut22vv3aDz/xN0icc8458W182fqMVapWVU/L5vsqyppU3E8ZmGvKNEVEIsh7ppnYM+6/tXzvmx/g7DPEn//85xmdy09W66ee81mo7wH0vejKNDPjBzVXlGH++te/BsKrC8/3nvrJpRP5ckmcCk6qJ39TyoABAwDYvHlz/L1GjRoByStU5pIyTRGRCPKeaSZOKPzFF18A4aSzfgopP14zW/y0dInjxACOOOKIrJ5nR+WXJSnLfvvtB4RtljvttFPS+2vXrt1unwsvvBAIb6mTwubbtP3ntnQZV8RPwnH00UcD8NlnnwHJYzGff/55AA499NDMg60EZZoiIhGo0hQRiSDvl+eJLr30UiAcZO4HQvvLM3/71XXXXRff56CDDqrwmH6QO4QdEx9//DFQ8QzSEp0f4Fx6AHvijEN+0HqLFi3KPIaf7cjPYAPhwObSs3lL/vlZy/w6PRCuA+abYG655ZaUx3n99deBcDVZf6OLd8cdd8QfH3PMMZUPOAv0VygiEoFlkm21bdvWzZs3L4vhxPjhBb5jpvRQpMSG5VTZh2+UroifUMDP5VfRBABmNt8517bcDaqZKGW8atUqIBx8vmbNGgCmT58e3+bUU0+t8Bi+rP2xANq0aZN+wFmgMo4u8XZVn2n6z6mfX9Ov3uBXZnjmmWfi+zz99NNAeOutv5HBT7jjJ/+Asge6R5VJGSvTFBGJoKDaND0/icOnn34KwJgxY4Dwtko/4QaEw5SiOOGEEwA444wzgPCbMNdTTFU3fq2exKn7ovJThPnfUhzuvvvu+OMbbrgBCG9LHjlyZNLvdPjPfOmVHgqBMk0RkQgKMtMs7aKLLkr6ndje5W/D9CtN+h5230aT2LvuB8z6iWq1joxIdiRO4O2vCCdOnAjApEmTgOQ2zNL8iJhevXoB0KpVqyqJMxuUaYqIRFCQveeFTD2r1Z/KuPpT77mISI6o0hQRiUCVpohIBKo0RUQiUKUpIhKBKk0RkQhUaYqIRKBKU0QkgowGt5vZWmBZ9sIpCi2cc43yHUSuqIyrP5VxNBlVmiIiOxpdnouIRKBKU0QkggorTTNrYGYLg59VZrYy4fnOVRGQmdU1szeDcywxsz+lsc/tCbG9Z2adM4xhtpm1TrFNiZm9amYLzOwdM/tFJufMl3yUccK5a5rZu2Y2KY1t81HGQ8xsaVC+08ysWSbnzJc8fY5bmNnM4DO82MyuSGOfS8xsbRDXUjO7OMMYxppZtxTb9Aj+Bhea2VtmdnzKAzvn0voBbgGuKeN1A2qke5w0zlMDqBs8/hkwD2ibYp/bgUHB48OAtQTttQnb1IwQw2ygdYpt/gH0Dx4fAXycrf+DfP3kqowTjjsEeAqYlMa2+SjjU4DaweOBwJP5LqNiKWNgX///C9QDPgEOSrHPJcCI4HFjYB3QMIMyHgt0S7HNroR9O0cBi1Idt1KX52Z2QPAN8iSwGGhmZhsS3u9lZqOCx3ub2QQzmxdkkMdVdGzn3Dbn3DfB052JVZxp91Y55xYR+wOoH3zTPGRmbwLDzGxXM3s8iGOBmZ0TxFjHzMYF327jgV0qOoc/FbE/BoDdgejrbhSwqizjYJ8WwGnA6Kix5aqMnXMznHNbgqevA02jxlrIqvhz/IVzbmHweCPwPtAk3dicc6uAz4HmwVXGP81sDvB4cIVyTxDHu2Z2SRBjDTN70MzeN7NpQMM0zrPZBTUmUJc06ppMZm4/BLjQOTfPzCo6zv3AcOfc62ZWAjwPHGZmxwL9nHO/K71DcMnwJnAAcJ9zbn66QQXp9XfOufUWW9FuH+A459w2MxsOvOic62tm9YE3gv/cK4CvnHMtzawNsezWH290EMPCUqf6E/CSmV0N1AEqXmaxOFVZGQMjgMGk8YddWg7LONFvgClRYy0CVVnGAJjZ/sSuDt5KNygzOwBoAXyaEGd759x3ZjYAWOOca2dmtYDXzewl4DhgP+BQYpnuEmBkcLw7gDnOuRfKONcvgTuI/S2elSq2TCrNT5xz6cxc2gk4OPjjhlh2UNs59wbwRlk7OOd+AFoHf/QTzaylc25pivMMNrO+wCagZ8Lr45xz24LHpwNnmtl1wfNdgOZAe2B4cO4FZrY4IZZ+5ZyvD/CIc+4+MzsReMLMDk/41qoOqqSMg3am/zrnFppZpwjx5LqMfbx9gcOBKyPEWiyq7HMMYGb1gPHAQOfc5jTO08fMOgLfA5c45zYE53zGOfddsM3pQEsz6xU83x04kFgZ/yv4W1hhZjP9QZ1zN5Z3Qufc08DTZnYyMDQ4frkyqTS/SXi8jdjlkpd46WNAu6AijMQ595WZzQLOAFJVmnc750akiNOItXF8krhBwh9CFL8BOgZxzg7+OOoD6ytzsAJVVWV8PNDDzLoEx6lnZmOccxel2C/XZYzFOvgGAx0q8zdcBKrscxxcMU4ARjvnnk1ztyedc4NSxGnAAOfcy6XO1z3d2MrinHvFzMaY2R7OuQ3lbZeVIUdBzf6VmR1oZjWAxOCnA5f7J5a6x3IvM9s9eFyH2Dfc+8Hz4b6NqpKmEmvQ9+dqEzycBfQOXjsSSGdVp+UEl+Rm1opYI3p1qjCTZLOMnXNDnHNNnXMlwAXAS77CLKQyNrO2wANAF+fcugxiKgpZ/hwb8Diw0Dl3f6n3rjKzci/n0zAVGOCbE8zsYDOrTayMewZtm02ADqkOFLTrWvC4LbFOoXIrTMjuOM1rif1jXgNWJLx+OXBC0GC7BOgfBHismZW1EPK+wKtm9g6xds3JzrkXg/eOAFaVsU+6bgXqWmzIymJiPYkAfwcamNlS4CZggd/BzEaX8wdyNbGCe4dYL13fDOIqFtkq44oUUhn/hVjnwHiLDUmZmEFcxSJbZdwB+BVwmoXDm84I3msJfJlBjA8DHwELzWwR8BCxq+aniSUzS4h1MM71O5jZHWZWVnvl+cAiM1tIrN22ZxnbJCma2yiDb4MpzrmiHA8pqamMdwxmNhno6pzbmu9YKqNoKk0RkUKg2yhFRCJQpSkiEoEqTRGRCFRpiohEkMngdho2bOhKSkqyFEpxmD9//jq3A83qrTKu/lTG0WRUaZaUlDBvXjp3YFUfZrZDLQugMq7+VMbR6PJcRCSCjDJNkWxbtGgRAJdfHr9jj65duwLw+9//Pi8xiSRSpikiEoEqTRGRCHR5LgXlsssuA2D27Nnx12bNmgXEOiwAevTokfO4pGp9+OGHAPz2t78FoHfv3gD0798/bzGVR5mmiEgEyjSloPTsGZuZa86cOfHX/KQyI0bE5h9Wplk9+OwSoHPn2OKin34aW93i888/B5RpiogUPWWaUlCuuCK2PHbi8KIff/wxX+FIFbjvvvuA8MoBYPny5UnbtGjRIqcxRaFMU0QkgoLMNDdsiC3R8dFHHwHw1FNPJb2f+A1V3oJZjRs3BmDu3PiM9wX97SUxM2bMAGDr1qKc1Fsq4Mt0yZIlACxbFt7J6D/HBx10EABjx47NcXTpU6YpIhJBQWWa/ttl2LBhAHzwwQdlbpeYXR555JFA2O61dGlspd/Vq1cDsGpVuEaXMs3C59u2tAxL9TNyZGz9tVGjRpW7TcOGDQFo2rRpTmKqDGWaIiIR5D3TTGyv9HeDfPvttwDsueeeQDguz2eV7du3j+/js0ffXtKsWTMAtmzZst3xjz322Oz/AySr/Pi8RDVrxv5M//rXv+Y4GsmGL774AoDHHnsMCK8iyrqauPvuu3MXWCUp0xQRiUCVpohIBHm7PPeX4ImNwkcffTQAf/zjHwE44YQTAKhdu3bK4/nL8dJDkM4777zMg5Wcefnll7d7rX79+gAcc8wxuQ5HssAPLXr33XeBsocJdunSBYCjjjoqd4FVkjJNEZEI8pZp1qlTBwgHM2fKdxL4DPbAAw8EoGXLllk5vuTP9ddfn+8QJAO77bYbEA4nWrdu3Xbb+JtQ/CQehx12WI6ii06ZpohIBHkfcpSpt956C4C77ror6XU/fKlBgwY5j0mi88NR/OTDie3YZ555Zl5ikuzwWWO3bt2Asge3++zzwQcfTPpdiJRpiohEUJSZ5rZt2+KPp06dCoRtmbvvvjsAJ598cu4Dk0obP3580vNzzz03/viQQw7JdThSBfyomIpuo3zuueeAcNkLf0NLIVGmKSISQVFmmr79C+Dmm29Oeu/OO+8E4IgjjshpTFI5b775JrD9+EwtaVH9+Fucr7rqKgDuvffe7bZZuXIlEI7bTJw+rlAo0xQRiaAoM83nn39+u9eaN28OwEUXXZTrcKQSNm7cCMDAgQMB+OGHHwDo3r07EPa0SvXjrw7btm0bf823Yfq+CT+l45VXXgnAxRdfHN+2devWOYmzPMo0RUQiUKUpIhJBUV2eL1iwAAiHJUB48//gwYMBqFWrVu4Dk8j8ioS+I8jzl+flrf0kxc8PC+zdu3f8tXHjxgEwc+ZMIGy+eeCBBwB4+umn49u+8847ADRq1KjKYy2LMk0RkQiKItP85ptvALjllluA5BmfTz31VAAGDBiQ87gkOn+V4MvS8x15ibPyy45j4sSJADz88MPA9p/nxLW+fKdhvijTFBGJoCgyzdGjRwPhUKPEyRz69euXl5gkfX7yWYBLL70UCG+F9ZNxdOrUCYC333476XdF2rVrB0CTJk2yF6zkVTHclKJMU0QkgoLOND/66CMAbrzxxqTXfU85JPfASWHwWeScOXMAOP/88+PvJbZNAUyZMiXpdxR+ctu+ffvGX/O9775tdKeddop8XInu1Vdf3e61Dh06pL3/o48+CsCwYcOA7VeqLGvlynxRpikiEkFBZpr+W+XPf/4zAJs3b056/+yzz855TJKan0i2f//+AEyaNCnyMQ444AAAvvzyy/hrfumSGjVi3/Hff/89EI7b/dvf/hbf1j/2baU33HADACeeeGLkWCQ1v6Z5165d46/5LH/NmjVl7vPss88Cydnp6tWrAdi6dSsQjtP1t0z6fQAaN26cldgrS5mmiEgEqjRFRCIoyMtzP4v3mDFjkl73Df5a/7ow3XPPPcD2l+U1a4Z/ZoceeigQduC1adMGgGOPPRYIh5Mlzs6fuD+El+e+oylxeJJv0vEdS9OmTQPguuuuA8LZdJo2bRr1nydl+OmnnwDYtGlT/DU/NHDy5MkV7pvYueMvx+vVqweEa36dc845AOyzzz5ZijhzyjRFRCIoyEzTr31cml9jpCL/+c9/AOjZs2dWY5LULrnkEgBGjhwJhI34iWXhM71M+Mzz9NNPT/oN4ZAjn2H6ISx+4gffqXThhRdmHIeEQ7p8hgjw9ddfp7Wvn8kdwisOP6t7Ia/xpUxTRCSCgsw0582bl/T8pptuAsJJHXybFsCECRMAGDp0KJA8/ERya//99wdg/fr1eYvh4IMPTvp9xRVX5C2WHcG+++4LJLdj+6Fg3v333w9Ax44dgfBWyUGDBuUgwuxTpikiEkFBZppz585Neu4zlyVLlgDQp0+f+Ht+tTp/q2WUW7dEJDsSP3elP4PFmlGWR5mmiEgEBZlp+h5QPyGp7/n0vxPHd/mpxoYMGZLLEEVkB6VMU0QkgoLMNG+99VYgvONj0aJFQDjuL3G85hlnnJHj6ERkR6ZMU0QkAlWaIiIRFOTluV/P2K9vLCJSKJRpiohEoEpTRCQCVZoiIhFYJqu8mdlaYFn2wikKLZxzjfIdRK6ojKs/lXE0GVWaIiI7Gl2ei4hEoEpTRCQCVZoiIhFUWGmaWQMzWxj8rDKzlQnPd66KgMyshZnNNLMlZrbYzFJOvW1ml5jZ2iCupWZ2cYYxjDWzbim2uS7h/2KxmW01s90zOW8+qIwr3KaVmc01s+/NrGgnhcxHGQfnXWFm7wXneSON7fNRxj3M7N3gnG+Z2fEpD+ycS+sHuAW4pozXDaiR7nHSOM++QOvgcT3gE+CgFPtcAowIHjcG1gENS21TM0IMY4FuEbbvDryUrf+DfP2ojLfbZm+gLXAnMCjf5VNMZRwccwWwR4Tt81HGuxJ2iB8FLEp13EpdnpvZAUGW8CSwGGhmZhsS3u9lZqOCx3ub2QQzm2dmb5rZcRUd2zn3hXNuYfB4I/A+0CTd2Jxzq4DPgeZmdruZ/dPM5gCPm1lNM7sniONdM7skiLGGmT1oZu+b2TSgYaT/EPgV8K+I+xQ0lTE451Y75+YBW9ONrZhUZRlnKodlvNkFNSZQF0g5nCiTe88PAS50zs0zs4qOcz8w3Dn3upmVAM8Dh5nZsUA/59zvytvRzPYHDgPeSjcoMzsAaAF8mhBne+fcd2Y2AFjjnGtnZrWA183sJeA4YD/gUGJZ0BJgZHC8O4A5zrkXyjnfrkAnoH+6MRYRlXH1V5Vl7IAZZuaAB51zj6UbVC7L2Mx+CdxBrJI9K1VsmVSanwTfwql0Ag42M/+8vpnVds69AZTbzmFm9YDxwEDn3OY0ztPHzDoC3wOXOOc2BOd8xjn3XbDN6UBLM+sVPN8dOBBoD/zLObcNWGFmM/1BnXM3pjhvV+BV51x6iz0XF5Vx9VeVZXycc26lmTUGppnZUufcaynOk/Myds49DTxtZicDQ4PjlyuTSvObhMfbiLWJeLskPDagnXPuh3QPbLHG6QnAaOfcs2nu9qRzrqzG+sQ4DRjgnHu51Pm6pxtbGXoBT2SwfyFTGVd/VVbGzrmVwe9VZvYM0A5IVWnmrYydc6+Y2Rgz28M5t6G87bIy5Cio2b8yswPNrAaxjhFvOnC5f2JmrSs6lsW+Vh4HFjrn7i/13lVmVu6lXhqmAgP8ZYiZHWxmtYFZQM+gTaQJkNaSlmZWHzgeeC6DmIrCjlrGO5Isl/GuQdMVZlYXOA1YFDwvmDIO2nUteNyWWKdQuRUmZHec5rXE/jGvEes18y4HTggabJcQtP2Z2bFmNrKM43Qg1rFymoXDIvyaFi2BLzOI8WHgI2ChmS0CHiKWbT8NLCfWBjIaiK8hbGZ3mFl57RznAlOcc1syiKmY7FBlbGZNzWwFcCVwi8WG0NTJILZikK0y3geYY2bvAG8CE51z04P3CqaMgfOBRWa2kFi7bc9UJy+qe8/NbDLQ1TlXLXszRWW8Iyj2Mi6qSlNEJN90G6WISASqNEVEIlClKSISQUarUTZs2NCVlJRkKZTiMH/+/HVuB5rVW2Vc/amMo8mo0iwpKWHevHRuJqg+zGyHWhZAZVz9qYyj0eW5iEgEqjRFRCJQpSkiEoEqTRGRCFRpiohEoEpTRCQCVZoiIhFkNE5TRKSyEmaBp0ePHgB+sTNatWoFwNChQ3MfWArKNEVEIlCmKQVhy5bYPM533XUXAN98E65uMGXKFACWLFlS5r4333xzmY+lsCVmmpMmTQLCTPOZZ54BoE2bNkCYiRYCZZoiIhEUdKb52WefAdC+fXsAZs+eDUCLFi3yFpNkh88sp02bBsBf/vIXAObMmQOEGQeEGUm9evUAqF+/PgDLly8HYMKECfFtlWkWj5Ejt18l449//CMA69atA+DPf/4zoExTRKRoFXSmOWPGDABWrIit7/Tqq68CcOGFF6Z9jAULFsQfP/zwwwB06BBbpO5Xv/pVVuKU6HwGcccdd6S9z5VXXgnAZZddBsDjjz8OwIEHHpjd4CQnLr300u1ee/vttwF49NFHcx1O2pRpiohEUNCZZiamTp0KJGeTX331FQAbN27c7j3JjVmzZgHwwAMPRN73/vtjS6QfddRRAFx//fXZC0wKim/TPumkk/IcyfaUaYqIRKBKU0QkgoK+PJ88eXLkffxA6PPOOw9IHkDrL+98R4LkXufOnQH49ttvI++7adMmAC644AIAxo0bB8CZZ56Zpegk3yZOnAiEn9vu3bvnM5wyKdMUEYmgoDPNr7/+Ou1t16xZA8CAAQOA8Da8xM6egQMHZjE6SdcLL7wQf7x582Yg+QogUd26dYHkwe2ls1L/3F+JKNOsPnxm+cgjjwDqCBIRKXoFnWnWrJkc3qmnnrrdNtu2bQPguuuuA+Dzzz8H4OCDDwZgxIgRVRihVMQP8erbt2/8NZ9hls4099xzTyBs03rttdfi75U3tOihhx4C4Kyzzoq/lvhYCtvatWvjj/3NDr78Dz300LzElA5lmiIiERR0prnPPvskPf/ggw8AaNKkSfy1Bx98EIDRo0cDUKtWLSCcAGKvvfaq8jilbN999x0AP/30U8ptL7/8cgBOPPFEAI455pj4e9OnTwfg5ZdfLnPfxIlqlWkWrmXLlgHQqFEjAMaOHRt/z18R1qlTBwhvmS5EyjRFRCIo6EzzmmuuAWDMmDEAvPjiiwCsXLkyvs0NN9yQtI+fWurss8/ORYhSAX/h65uyAAAJBUlEQVSlsPPOO6fc9txzz0167q8YAE455RSg/EzTt2NLYWvXrh0Af/3rXwG488474+/5Nm7/eT7kkENyHF36lGmKiERQ0JmmX1zJT4XvJyL1PeaJ/Pgu34suxcFP09eyZctyt/HjMG+88cYy308cx+mXxCjk3tcdjZ8k2o+lHjZsWNJzCMu/9JVjIVKmKSISgSpNEZEICvry3DcOd+3aFYBu3boByWvCeD179gS2HxAv+TN//nwgvHUSkm+PhLCTqKJyO/LIIwHo1KkTEK4r5PlhKqDL8kKwdOlSAMaPHw+EK4z6z/Mvf/lLABYvXhzfxzfB3X777UDYoVuIlGmKiERQFGmZn3xj5syZ273Xp08fIMxGpXD873//A+DHH3+Mv1b6NsryJu5I5Fed9JlJOvtIbvmB6xB22PlbIn1nnx8a1rt3byB5bXt/hXDTTTcBUFJSAoTTABYSZZoiIhEURaZ57733ArB+/XoAGjduHH/PTyy8yy675D4wqZC/wcCvUw6wevXqpG18u+crr7wCwMknn7zdcfzEHz5zlcKTuELs7NmzgfAW5nvuuQeA5s2bA9CwYUMgeaiYH3Lkhw76VUp9e7XWPRcRKVIFnWlu2bIFgL///e9Jr/fv3z/+2E8pJoUrsUe7dKb55ZdfAmGP6ocffghAgwYN4tv4AetSePz0bn6VUQjbMMvqg0iUOOrB8yuN+p53n636Ns7EbfJFmaaISAQFnWn66cJ8drL33nsDMGTIkLzFJNE9++yz8ce77bZbmdts2LABgLlz5wLJE674iVqk8JReCA2yuxjaE088ASRfbSjTFBEpIqo0RUQiKMjL840bNwLbX5b5dUR23XXXnMckledXmISwk6C8mbm7dOkCwO9+97v4a/4SrTx+QLTknh8+5H8DPPzwwwA0a9YMqNxwIX+rtJ9nNfHyP98D3pVpiohEUJCZ5h/+8AcgHMzcr18/IHkNcylOt912GxBOvuI7gEobOXJk/HF5t0127twZCNe6l9zzWaS/1RVg1KhRAFx00UUAvP/++0B6c2X6CTtKT/JRSBN4KNMUEYmgIDPN0jOz+3Yu3SpZ/E466SQArr32WiBcNdQPck/HvvvuC4RZq+TfoEGD4o/POOMMIJxx/9JLL025/69//WsgzEp9G6lfH0y3UYqIFKmCzDT9hLStW7cGoGPHjnmMRqqCv0GhadOmQJhpVMRnmM888wwQ/n1IYfGTb6Qa9eAnK4ZwEuLrr78eCLPTxF75QqFMU0QkgoLMNBcsWACE2cgee+yRz3CkCvlxeH6N9KFDh263zTHHHAOEoyeUYRYH335dnsQVSDdt2lTV4WSNMk0RkQgKMtP0bZi+J02qr1q1agHh5MNlTUIsUkiUaYqIRKBKU0QkgoK8PB8+fHi+QxARKZMyTRGRCFRpiohEoEpTRCQCc85VfmeztcCy7IVTFFo45xrlO4hcURlXfyrjaDKqNEVEdjS6PBcRiUCVpohIBBVWmmbWwMwWBj+rzGxlwvOdqzIwM6tpZu+a2aQ0tr09Ibb3zKxzhueebWYVzgphZieb2QIz22pm3TI5Xz7lq4zN7Cwz+8DMPjazwWlsn48yLjGzV4NyfsfMfpHJOfMlj2V8jZktNrNFZvakmdVKsX0+yniImS0NyneamTVLeWDnXFo/wC3ANWW8bkCNdI8T4XxDgKeASWlsezswKHh8GLCWoL02YZuaEc49G2idYpv9gMODGLtl+9+fj59clTHwM+BToAVQC3gPOKgAy/gfQP/g8RHAx/kuoyIq4xbAx8AuwbHHAxcUYBmfAtQOHg8Enkx13EpdnpvZAWa2xMyeBBYDzcxsQ8L7vcxsVPB4bzObYGbzzOxNMzsujeO3AE4DRkeNzTm3iFgh1TezsWb2kJm9CQwzs13N7PEgjgVmdk5wvjpmNi74xhlPrKBTnecz59x7wLZU2xajKi7j44Clzrllzrnvgf8HdE03tlyVMeCAesHj3YEv0o2xGFT155jYl+MuxO48rEOE/78cfo5nOOe2BE9fB5qm2ieT2ygPAS50zs0zs4qOcz8w3Dn3upmVAM8Dh5nZsUA/59zvythnBDAYiDxts5kdD3znnFtvsZXs9gGOc85tM7PhwIvOub5mVh94w8ymAVcAXznnWppZG2BewvFGA/c55xZGjaUaqKoybgL8N+H5CuDIdIPKYRn/CXjJzK4m9qE/Nd0Yi0iVlLFzbpmZ3UesnL8HJjvnZqQbVJ4+x78BpqSKLZNK8xPn3LzUm9EJONjCZVjrm1lt59wbwBulN7ZY++B/nXMLzaxThHgGm1lfYBPQM+H1cc45nw2eDpxpZtcFz3cBmgPtgeEAzrkFZrbY7+yc6xchhuqmSso4A7ku4z7AI865+8zsROAJMzvcBddy1URVfY4bAGcTa8baCIw3s17OuX+nOE9ePsfBOQ8HrkwRX0aV5jcJj7cRS6W9xLTYgHbOuR/SPO7xQA8z6xIcp56ZjXHOXZRiv7udcyNSxGnE2h8/SdzAyllXW6qsjFcCiQ3uTYPXUsl1Gf8G6AjgnJttZvWA+sD6yhysQFVVGZ8OfOScWwdgZhOJfbZTVZo5/xxbrINvMNAhnX9fVoYcBd8AX5nZgWZWA+ie8PZ04PKEACvszXLODXHONXXOlQAXAC/5CtPMhvv2i0qaSqyx18fSJng4C+gdvHYk0CqDc1RL2SxjYm1Hh5pZC4v1qJ4PPBvsW0hlvJzgktzMWhHrKKlOFWaSLJfxcuDnZlbbYrXZqcDSYN+CKWMzaws8AHTxFXwq2RyneS2xf8xrxNqovMuBEyw2fGgJ0D8I9lgzGxnxHEcAqzKI8VagrsWGMywm1pMI8HeggZktBW4CFvgdzGx0WX8gZvZzM1tB7A9rlJm9m0FcxSIrZeyc+5HYZdA0YAkw1jn3QfB2wZQxcDUwwMzeAcYCfTOIq1hkq4znEPsiXEBsdMRW4LHg7UIq478AdYk1HywMMuIKFc1tlMG31RTnXFGOlZPUVMbVX3Uo46KpNEVECoFuoxQRiUCVpohIBKo0RUQiUKUpIhKBKk0RkQhUaYqIRKBKU0Qkgv8Pw+w8VYAujkkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7faae86bb080>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_example_errors(cls_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save & Load Model\n",
    "\n",
    "NOTE: You need to install `h5py` for this to work!\n",
    "\n",
    "Tutorial #04 was about saving and restoring the weights of a model using native TensorFlow code. It was an absolutely horrible API! Fortunately, Keras makes this very easy.\n",
    "\n",
    "This is the file-path where we want to save the Keras model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "path_model = 'model.keras'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Saving a Keras model with the trained weights is then just a single function call, as it should be."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "model2.save(path_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Delete the model from memory so we are sure it is no longer used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "del model2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to import this Keras function for loading the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.python.keras.models import load_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading the model is then just a single function-call, as it should be."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "model3 = load_model(path_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then use the model again e.g. to make predictions. We get the first 9 images from the test-set and their true class-numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "images = data.x_test[0:9]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "cls_true = data.y_test_cls[0:9]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then use the restored model to predict the class-numbers for those images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = model3.predict(x=images)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the class-numbers as integers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "cls_pred = np.argmax(y_pred, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the images with their true and predicted class-numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAD5CAYAAACj3GcTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xe8VNW5//HPg2IQVBCxU86NWECjmGCvV5EoFpQYxViIsUQxlngDSYwSTJAYbPwsV1FfoglWEBGJimJXQISASrGAokEvIqJGiWJ7fn/MXjN7Tp19ph++79eL19kzs8uj68w6z1p77bXM3RERkdy0KncAIiLVRJWmiEgCqjRFRBJQpSkikoAqTRGRBFRpiogkoEpTRCQBVZoiIgmo0hQRSWDdfA7u1KmT19TUFCiU6jBnzpyV7r5pueMoFZVxy6cyTiavSrOmpobZs2fnc4qqY2bvlDuGUlIZt3wq42TUPBcRSUCVpohIAqo0RUQSUKUpIpKAKk0RkQTyunsu0lxXXnklAF988QUAr7zyCgATJkyos+/ZZ58NwF577QXAySefXIoQReqlTFNEJAFlmlJSxx9/PADjx4+v93Mzq/PeTTfdBMC0adMAOOCAAwDo2rVrMUKUMnrjjTcA2H777QG49tprATj33HPLFlNtyjRFRBJQpilFF7JLaDjD3GGHHQA49NBDAXjrrbfSn02ePBmAxYsXAzBu3DgALrroosIHK2U1d+5cAFq1SuVzW2+9dTnDqZcyTRGRBJRpStGE55kfeOCBOp/ttNNOQCaL7NSpEwAbbLABAF999VV63z322AOAl19+GYCPPvqoSBFLuc2bNw/I/B4MGDCgnOHUS5mmiEgCJc80wzi8W265BYCtttoq/VmbNm0AOPHEEwHYYostAOjevXspQ5QC+b//+z8A3D39Xsgwp06dCsCWW25Z77FhHCfAokWLsj474ogjChqnlN+rr74KwHXXXQfAKaecUs5wGqVMU0QkgZJnmkOGDAFg6dKlDe4TxuVttNFGAPTs2bMg1+7SpQsAQ4cOBaB3794FOa/U78gjjwQyd70BNtxwQwA6duzY6LH33ntvejvevykt0+uvvw7A6tWrgewRF5VGmaaISAKqNEVEEih58/zWW28FMsNH4k3vhQsXApkBrk8//TQAM2fOBDKPzb377rsNnr9169ZAZghLuBkRP09opqt5XhrdunXLed8rrrgCyDxOFxeGHoWf0nKMGjUKSC29AZX93VSmKSKSQMkzzYMPPjjrZ1x4hC74+OOPgUzmGf76vPTSSw2e/3vf+x6QeeA/PJ4HsGrVKgC22WabZsUuxTNlyhQAhg0bBsCaNWvSn22++eYAXH755QC0bdu2xNFJMcRvBofvdPjetmvXrhwh5USZpohIAhX9GOXGG28MwEEHHZT1fn1Zam33338/kMlWAXbeeWcABg4cWKgQpUDCI5fxDDMIw0/ClHDSMjzzzDN13tt008pfbl6ZpohIAhWdaTbHihUrABg8eDCQ/Qhf6C9ramC1lM7RRx8NZB6rDAYNGpTeHjFiREljktIIS5zEhQdPKpkyTRGRBFpcpnnDDTcAmYyzQ4cO6c/CnTkpvzB+dvr06UCmLzP0aV188cXpfcM0YdIyzJgxA4CxY8em39t1110BOOSQQ8oSUxLKNEVEEmgxmebzzz8PZMbyBQ8++GB6O0xLJuUXJpdduXJl1vthWkCNpW25nnjiCSB7ZEsYox2mh6xkyjRFRBJQpSkikkCLaZ4//PDDQGbuxT59+gCw1157lS0mqSusCRQejQ0OPPBAAP70pz+VOiQpsTBZT9xPf/rTMkTSPMo0RUQSqPpM84svvgDg0UcfBTITdlx66aVAZqo4KZ/46pEjR44E6s7G3qtXL0DDi1qy5cuXA/Dcc88B2ZPpHHPMMWWJqTmUaYqIJFD1mWaYtDb0kR122GEA7L333mWLSbJdddVV6e1Zs2ZlfRYeo1RfZst3++23A/DBBx8Ame9qtVGmKSKSQFVmmmHCWoA///nPALRv3x6ASy65pCwxScOuvvrqBj8Lj72qL7Ple+edd7Jeh6kfq40yTRGRBKoq0wx3Yc8777z0e9988w0A/fr1AzQus9qEMs1llENoTYR9v/76awA+/fTTOvuGR/Suueaaes+1zjrrpLf/+te/AlpGo9geeuihrNdHHHFEmSLJjzJNEZEEVGmKiCRQFc3zb7/9FsjMhPL222+nP+vevTuQuSEk1SWs25SL4447DoAtt9wSyAxdueeee/KKIax2GZ/DUwonDGYP5VXtlGmKiCRQFZnmkiVLgMyKhXFhOIvmX6xc4SYdwKRJk5p9nvvuu6/JfcJNolatsvOBo446CoDevXvXOWbfffdtdkzStAceeADI3LQNs7RX6+qiyjRFRBKo6EwzDIbt27dv1vtXXnllertahy2sTSZOnJjeHjVqFFB3wo5g4cKFQOP9lKeddhoA3bp1q/PZT37yEwB69OjRvGClYP7zn/8A8Mgjj2S9H6aBiw/7qibKNEVEEqjoTHPMmDFA3cev4n0hZlbSmCQ/ua5rfddddxU5Eim20L8cVoTt378/AOeff37ZYioEZZoiIglUZKYZxnVdf/31ZY5ERJorZJphnfOWQpmmiEgCFZlphjXMP/vss6z3w9M/mkZMRMpFmaaISAKqNEVEEqjI5nltYaXCJ554AoCOHTuWMxwRWYsp0xQRSaAiM83f//73WT9FRCqFMk0RkQTM3Zt/sNmHwDtN7tiydHP3TcsdRKmojFs+lXEyeVWaIiJrGzXPRUQSUKUpIpJAo5WmmW1iZvOif8vN7L3Y6/WKEZCZ9YxdY56ZfWZmv2rimNPN7MNo/0Vm9os8YxhnZkc3sc/vYjEuMLNvzKx9PtcthzKVcTcze9rMFkb/7xot3+iYcpTxKWb2qpm9YmYvmNkP8rlmuZSjjKPr3hHKLMf9y1HGO5rZDDNbY2YX5HRid8/pHzAc+E097xvQKtfzJPkHtAZWAJ2b2O90YHS0vQWwEuhUa591E1x3HHB0gv2PAR4rxv+DUv4rVRkDWwG9ou2NgCXAdpVWxsA+QIdo+0jghXKXUbWUcXTOA4DdgXk57l+OMt4c6A1cDlyQy3mb1Tw3s+5RlnAnsADoYmafxD4faGa3Rtubm9lEM5ttZrPMbM8ElzoEWOTuy3I9wN2XA0uBrmY2wsz+ZmYvALeb2bpmdnUUxytmdnoUYysz+18ze83MHgc6JYgR4ATg7oTHVLRilrG7v+/u86LtfwOvAVvnGlupytjdX3D38N88E+ica4zVoNjfY3d/BljVnNhKWMYfuPts4JtcY8tncPsOwCnuPtvMGjvPtcAod59pZjXAFGAnM9sDONXdz2rk2IEkrIzMrDvQDXgrFuf+7v6lmQ0GVrj77mb2PWCmmT0G7An8F9CTVBa0ELgpOt9lpDKMhxu43gZAH+CMJHFWiaKXsZl9H9gJeCnXoEpdxpHTgEca+bxaleJ7nFiZyjgn+VSaS6Iauil9gO0tsyzFxma2vru/CLzY0EFm1gY4HLgwx3hONLMDgTXA6e7+SXTNB939y2ifvkAPMxsYvW4PbAvsD9zt7t8By8zs6XBSd/9DE9ftDzzj7p/mGGc1KXYZbwTcD5zr7p/ncJ2ylLGZ9QFOBlriWr9FLeNmKNf3OGf5VJqrY9vfkeoTCdrEtg3Y3d3rX36wYYcDL7r7yhz3v9Pd6+vIjcdpwGB3fyK+g5kdkzC2uIHA3/M4vpIVrYwtdQNiIjDW3SfneFjJy9jMegFjgB+7+8fNOUeFK/b3OKlyfY9zVpAhR1HN/rGZbWtmrUjdGAmmAeeEF9EvYS7q9BOa2flmlk8zYCowODRDzGx7M1sfeBY4PuoT2ZpUB3aTzGxjYG/goTxiqgqFLGNLpQ63k7pBcG2tzyqmjKNm6ATgZ+6+OI+YqkKRvsd1VFIZN0chx2n+ltR/zHQgfuPmHGCfqMN2IVHfn5ntYWY31XciM9sQ+G9gUq2PegAf5RHjGOBNYJ6ZzQduJJVtTwDeJdUHMhZIL2piZpeZWb8GzvcT4BF3/yKPmKpJocr4AFJ/FA+xzNCXH0efVVIZDwc6AmOiGAvZDK1UhfwejweeA3qa2TIz+3n0UcWUsZl1NrNlwHnA8CjOto1dvKoeozSzfwD93T3nO11SXVTGLV+1l3FVVZoiIuWmxyhFRBJQpSkikoAqTRGRBFRpiogkkNcaQZ06dfKampoChVId5syZs9LXolm9VcYtn8o4mbwqzZqaGmbPzuUJrJbDzNaqZQFUxi2fyjgZNc9FRBJQpSkikoAqTRGRBFRpiogkoEpTRCQBVZoiIgnkNeSoVFavTs0/OmTIEABuuikzE1Xv3r0BGD9+PADdunUrcXQisjZRpikikkBVZJrvv/8+ALfccgsA66yzTvqzMCj3oYdSk6f/6ldNLqEtFeCf//wnAAMGDABg6dKlzT7XY489lt7u0aMHAF26dGl+cFI24Xt81FFHAXDdddcBcPbZZ6f3iX//y0GZpohIAhWdaX744YcADBo0qMyRSKFNnToVgDVr1uR9rsmTM+uy3XbbbQDcc889eZ9XSuejj1KrX8QzSoBzzz0XgNNOOy393vrrr1+6wOqhTFNEJIGKzDSvvTa1QOGkSal11V566aUmj3nuuecACMt37LLLLgDsv//+xQhRmumbb1LLwjz88MMFO2cYQQFw9dVXA5kRF+3atSvYdaR4nn32WQDee++9rPdPOOEEANq0aVPnmHJRpikikkBFZpoXXJBaKz7JXbKJEydm/ezatSsA9913X3qfH/3oR4UKUZrpqaeeAmD69OkA/Pa3v837nKtWrUpvL1iwAID//Oc/gDLNShbvzx4xYkS9+5x88skAmFlJYsqFMk0RkQRUaYqIJFBRzfN+/foBmZs53377bZPHdOrUCcg0w955JzUh89tvvw3Abrvtlt73u+++K1ywkrNXX301vT1w4EAAunfvDsBFF12U9/njQ46kerzyyivp7fCwQ7Duuqmq6bDDDitpTLlQpikikkDZM81nnnkmvf3aa68BmU7fhm4EnXXWWentvn37AtC+fXsAnnzySQAuu+yyOsfdeOONQN0BtFJc8bIIN2jGjRsHwAYbbNDs84YbQPHfoUq6YSCNCzdt63PIIYeUMJJklGmKiCRQtkwzTNAQ+rgAVq5cWe++YfjQscceC8Af//jH9Gdt27bN2jdMDTdmzJg65xw6dCgAX375JZCZ3KN169bN+4+QRk2YMAHIHsge+jLjfc3NFYapxLPLAw88EIAOHTrkfX4prngLIVhvvfUAGDlyZKnDyZkyTRGRBMqWaX799ddAw9klZB6BvPfee4HMnfLGhEwz3JW98MIL05+FR+tCxhmmn9pmm20SxS65CRNDh//vUJj+5NBKueuuu4DMnVaAiy++GFDroZKFBxtmzJhR57PQcuzVq1dJY0pCmaaISAJlv3ten9DfNXbsWCC3DLO2kEXeeeed6fdmzZpVgOikKZ9++ikAM2fOrPPZ4MGD8z7/zTffDGSmDuzZs2f6s4MOOijv80txNTYBTzWMbFGmKSKSQNkzzfqe+nnxxRfzPm94qij+FFDtJ43CXfgwZlAKI0zEsGzZMiAzvVehLFmyJOv1TjvtVNDzS3HVl2mG0Q6FaIkUmzJNEZEEVGmKiCRQtuZ5WLu8WCvLhVXt5s6dm36v9uOZl156aVGuvbbbcMMNgcywkfiEHeHRx44dOyY+74oVK4DMUKZgn332aVacUlrPP/88kBkqFhceg+7cuXNJY2oOZZoiIgmULdOcMmVKQc8Xhp8sXLgQaPwxrDCESQOgiyOsFhgemQyPUwIcfvjhQPZDB/WZP39+ejvc+AnT/tWelKNVK/3trwZhxclwQzaukifoqE2/bSIiCZR9yFGhhOnHbrjhhgb3qampAeCOO+4AMhOBSHEMHz4cyM4sQgsjPlFLfTbddNP0dsgsG3rk9tRTT80nTCmR2n3R8UlVzjzzzFKH02zKNEVEEqj6TDMskREmMG5MeNxuv/32K2pMktKjRw8ge0XQMJqh9gD12sI0gHGDBg0C6j6MEPpQpTKFhxxq3zWP3ykvxFSBpaJMU0QkgbJlmo0tnvbII49kvT7jjDMAeP/99xs8Ty7LHBT6jr0kt+uuu2b9TOL73/9+ve/Hx4H+4Ac/aF5gUjRhKrjad8379+9fjnDypkxTRCQBVZoiIgmUrXke5s0Ls6jHhQHQtR+xrO+Ry9C8z2XlSqluoXlXu5mnJnllC4Pag/BwyQUXXFCOcPKmTFNEJIGyZZoDBgwAYNSoUen3GlsvqCnhr1cY5nLLLbcAsOWWWzb7nFJZws0+rW1eXaZOnZr1ukuXLkBmko5qo0xTRCSBsmWaYdXIsNIkwKRJkwAYPXp04vP94Q9/ADJrmUvLE9arDzSovbKFFWcXL16c9X6bNm2A6p0wR5mmiEgCZX+MMqxtHt/u27cvkFl1MEwofOSRRwLwy1/+Mn1MuJMaX5FQWqawOmmY6GHYsGHlDEeaEKbsC49ILliwAIBtt922bDEVgjJNEZEEyp5p1ufQQw/N+ikCmYzl17/+NaA1zitdGDsdpm0Mox5++MMfli2mQlCmKSKSQEVmmiL1CX3bUl222morAG677bYyR1IYyjRFRBJQpSkikoAqTRGRBFRpiogkoEpTRCQBVZoiIglY7QldEx1s9iHwTuHCqQrd3H3TpndrGVTGLZ/KOJm8Kk0RkbWNmuciIgmo0hQRSUCVpohIAo1Wmma2iZnNi/4tN7P3Yq/XK1ZQZtbPzF43s8VmNiSH/UfEYnvVzA7P8/rPm1mvJvZpY2YTohhnmFnXfK5ZLuUq4+ja65rZK2Y2KYd9S17GsX2PNzPPdf9KU8bv8R1m9qGZzctx/9PD/ma2yMx+kef1x5nZ0U3ss2P0/V1jZjktj9nohB3u/hHQKzr5cOBzd7+y1kWN1A2l73K5YFPMrDVwPfDfwHJgtpk96O5vNHHoFe4+2sx2Ap4ys808dpfLzNZ1928KEWPkTGC5u3c3s5OAvwAnFvD8JVGOMo65EJgPtM1x/1KXMWa2ETAYmF3I85ZSGcv4NuAG4OYEx9zp7heY2RbAfDOb7O7pFReLUMYrgXOBY3M9oFnNczPrbmYLzexOYAHQxcw+iX0+0MxujbY3N7OJZjbbzGaZ2Z5NnH5PYJG7v+Pua4D7gP65xubu8wEDNo7+0txoZrOAkWa2gZndHsUx18yOjGJsa2bjo79u9wNtcrhUf+COaPs+4Me5xlgNilzGmFk34BBgbNLYSljGACOjf2uSxlnpil3G7v4MsKo5sbn7cmAp0DVqZfzNzF4Abo9aKFdHcbxiZqdHMbYys/81s9fM7HGgUw7X+cDdZwM5V8T59GnuAFzj7j2B9xrZ71pglLv3Bo4DQiHsYWY31bP/1sC/Yq+XRe/lxMz2Br5091BYWwJ7uvtQYBjwqLvvDhwEXGVmbYBfAR+7ew9gBLBr7HxjG2iWpeN096+A1WbWIdc4q0SxyhhgNDAESDzmrVRlbGa7AZu5+9Tan7UgxSzjZjOz7kA34K1YnAe7+0mkWnkrojLeDTjHUt1jxwL/BfQETgX2jp3vMjPrV4jY8plPc0lUQzelD7C9Zdaq3tjM1nf3F4EX87h+bUPM7OfAZ8DxsffHx5ocfYHDzOx30es2QFdgf2AUgLvPNbMF4WB3P7WAMVabopRx1M/0L3efZ2Z9EsRTsjI2s1bAVVRhl0tClfY9PtHMDiSV2Z/u7p9E13zQ3cNypH2BHmY2MHrdHtiWVBnfHf0uLDOzp8NJ3f0PhQown0pzdWz7O1LNpSDe9DFg9ygby8V7QJfY6840/hcwuMLd61v7Nx6nAUe7+5L4DrFfhCRCnMst1Znezt0/aeKYalOsMt4bGGBmR0Xn2cjM7nD3QU0cV8oy7kAqY3kuOnYL4GEzO9zd5yY9WQUrVhk3153uXt8NmdplPNjdn4jvYGbHFDWySEGGHEU1+8dmtm30Fzoe/DTgnPCigaZu3Eygp5l1M7PvkWoKTI6OHRX6qJppKqlO3xBLaKI9C/wsem8XYMcczjUZCF/y44DH8oir4hWyjN19qLt3dvca4CTgsVBhVkoZu/sqd+/k7jVRnLOBfi2swsxS4O9xg8zsfDM7q/mRMhUYbGbrRufb3szWJ1XGx0d9m1sDB+RxjQYVcpzmb0n9x0wn1Q8ZnAPsE3XYLgTOgIb7Qtz9a+A84HFgITDO3V+PPt6Z1B315roUaGepISsLgOHR+9cDm5jZIuASIP3FaKRP82ZgSzNbTKq/7KI84qoWBSnjJlRSGa+NClbGZjYeeI5UErQs6loB6AF8lEeMY4A3gXlmNh+4kVSreQLwLql6YywwIxZLvX2aZtbZzJaRqnOGR3E2Opqjap49t1Qb6RF31xKVLZTKeO1gZv8A+hd6eFipVE2lKSJSCfQYpYhIAqo0RUQSUKUpIpJAPuM06dSpk9fU1BQolOowZ86clWvTrN4q45ZPZZxMXpVmTU0Ns2dX7TwGzWJma9WyACrjlk9lnIya5yIiCajSFBFJQJWmiEgCqjRFRBJQpSkikoAqTRGRBFRpiogkkNc4TRGRYvj4448BePfddxvcp1u3bgBcc801AOy0004AbLfddgDssssuRYlNmaaISAIVlWmuWLECgOOOOw6AvfdOrYt05plnAqknFwrh008/BeDZZ58F4NBDU9M3tm7duiDnF5FkpkyZAsBDDz0EwNNPPw3Am2++2eAx22+/PQBLly4FYM2a7AVDv/uu0CtOpyjTFBFJoOyZZui7ANhxx9SyLSET3HzzzYHCZ5g//OEPAVi5MrUGfXjudtttty3IdSR3//73vwH43e9Si0cuWJBaJHLatGnpfdQCaBmWLEmtdXfDDTcAcPPNN6c/++KLLwBIMin666+/3vRORaBMU0QkgbJlmiHLC/2XAB99lFpr6ZxzUoveXXfddQW95ogRIwB4++23gcxfOmWYpTdu3DgALr74YqDuXdKQgQJssskmpQtMimbZstQ6baNH17cKc+522GEHIHO3vNSUaYqIJFC2TPOf//wnkLlLFjds2LCCXWf+/Pnp7SuvvBKAY45JLed8/PHHF+w6kpuQbfz6178GMi2O1EKUGeeem166nOuvvx6Ajh07liJEaYZQjpDJJPfdd18gMzplvfXWA6B9+/YAbLDBBuljPv/8cwB+/OMfA5ksco899gBg1113Te+7/vrrA9CuXbsC/1fkRpmmiEgCqjRFRBIoefM8DGC///7763x22223AbDppvkvzxKa5YccckidzwYMGADAhhtumPd1JJnQRRJu+jXknnvuSW8/8sgjQOamUWi6h+aelM/q1auB7O/Zyy+/DMCkSZOy9t1rr70AmDt3LpA9lDDcCOzcuTMArVpVbj5XuZGJiFSgkmea//M//wNkhpyEgeYAP/3pTwt2neeffx6A5cuXp9879dRTATjppJMKdh1p2jvvZNawGjt2bNZnYVKF8CDD448/Xuf48FBCyFJPPPFEALbYYovCBys5+eqrrwD42c9+BmSyS4CLLroIgD59+tR7bH0Pq3Tt2rXAERaPMk0RkQRKnmmGoSXh59Zbb53+LJ8+qvAY1siRI4HMo1rxoSyhz1RKa968eentMGh9//33B+CZZ54B4MsvvwTgrrvuAuAvf/lL+pjFixcDmVZD//79gUxfp4YilU4YGhS+Z2GCjfh9iCFDhgDQtm3bEkdXGso0RUQSKPuEHWFKKIC+ffsC0KFDBwDOPvvsJo8Pg+PDz5kzZ2Z9Xsh+Umme+JRdIfMPg9uDNm3aAPCLX/wCgAkTJqQ/CxM9hMkcQgaju+elF+6IX3755UBmIuDnnnsuvU8YvN5SKdMUEUmg5Jnm+eefD8CTTz4JwPvvv5/+LPRvhYziwQcfbPJ8Yd/aj+Fts802QKbvRcrn7rvvrvPeP/7xDwCOPvroeo8J0/XVZ8899wSyH8OT0pg+fXrW6/B4YxhfuTZQpikikkDJM80f/ehHALz66qtA9p3VRx99FIBRo0YBsNlmmwEwaNCgBs938sknA7DzzjtnvR+WyggZp5TPCSeckN4OrYeXXnoJgNdeew3I/D488MADQPbk1KGPO7wXpvQLZd+zZ8+ixS7Z4n3NkBnBcOmll6bfO+qoo4DsSTZaEmWaIiIJqNIUEUnAkqzJUVvv3r29sQ77UnjrrbeATDO8V69eADz22GNAYSb/iDOzOe7eu6AnrWCFKONVq1alt0M5hUcjG7qRF58AIjyocMQRRwDwxhtvAJlVSm+66aa84qtNZdyw2g+n1GedddYB4KyzzgIyc2L+61//AqB79+5AZk2wuLBGVJjco1g3mPIpY2WaIiIJlH1we77+9Kc/AZm/fOEmUqEzTGm++GOO48ePB+DYY48F6mac5513HgB//etf08eEge9hSr/wiOXUqVOBzOB30I2/YvvNb34DwFVXXdXgPt9++y2QaSGEn0mEm8AHHnggkD1VYLkp0xQRSaAqM82QrQDccccdAGy00UaAVi6sdGG6sDB0JUzQEYYVhZZDyC7jLrnkEgAWLVoEZIYvhWMg8/sgxREenwyryIZp+r7++uv0PmEdqJBxNkeYrDx81+MrT4bJqMtFmaaISAJVmWmGAbVxhx9+OJA9qbFUrpBxNjRRbX3CKoRhFdGQaT711FPpfcKdek0XVxzhzvhuu+0GZEYyxD3xxBNAJvscPnw4ALNmzUp8vdDXPWfOnMTHFosyTRGRBKo+0wxrH4e7etLyhf60yZMnA9l3VsMa6cOGDSt9YALAwQcfnPU6PCodMs3WrVsDmeVnAM444wwArrnmGiDT112JlGmKiCSgSlNEJIGqap6Hx+XiK0yGVQx1A2jtEdbEHjp0KJC9vna46TBw4EAAtttuu9IGJ3WEFRnCKpXhBlGYrQrgzTffBDIrMNQWX0us3JRpiogkUJWZZnyygH79+mXt89lnnwGZuReraT1lSSZMzvLnP/85/V64Ifj73/8egHHjxgGZ4UpSej169AAyQ8XuvffeOvvEh40BrLtuqmoKQwnjj9XY76NRAAAG3klEQVSWmzJNEZEEqirTrE/4ixQyijBkITx2pcfqWr5TTjklvT1mzBgAJk6cCGT6ymrP7C+lE7L80aNHA5nWYHzA+gcffABATU0NkCnT0EddSZRpiogkUPWZ5i233ALArbfeCsDpp58OZCZ3kJYvPg3gtGnTgMx63GGCiUoeLL22CCNdpkyZAsDf//739GczZswAMpllmBquEinTFBFJoKoyzeuuuw6AP/7xj+n39t9/fwDOPvtsADbeeGMA1ltvvRJHJ5UgjJYIy2WERy0XLlwIaOXKShJWE629XemUaYqIJFBVmeZ+++0HwJNPPlnmSKTShUmOd9llFwAWL14MKNOU/CnTFBFJQJWmiEgCVdU8F8lVWDPq7bffLnMk0tIo0xQRSUCVpohIAqo0RUQSsLDaW7MONvsQeKdw4VSFbu6+adO7tQwq45ZPZZxMXpWmiMjaRs1zEZEEVGmKiCTQaKVpZpuY2bzo33Izey/2uqgzYpjZumb2iplNymHfEbHYXjWzw/O89vNm1ivHfY83M891/0pTrjI2swvNbEH079wc9j/dzD6M4lpkZr/I8/rjzOzoJvbpaGaTo9/DF82sKp/BLGMZL4u+j/PM7MUc9i9HGfcxs09j/z/+0NR5Gx3c7u4fAb2ikw8HPnf3K2td1Ej1jX7X1MUSuhCYD7TNcf8r3H20me0EPGVmm3msw9bM1nX3bwoZoJltBAwGZhfyvKVUjjKO/sAMAnoD3wCPmdkUd29qJPqd7n6BmW0BzDezye6+MnbeQpfxJcCL7n6Ume0I/D/gkAKevyTK/D3ez90/SbB/qcsY4Cl3b7RyjWtW89zMupvZQjO7E1gAdDGzT2KfDzSzW6Ptzc1sopnNNrNZZrZnDufvRuqXc2zS2Nx9PmDAxtFfmhvNbBYw0sw2MLPbozjmmtmR0fXamtn46K/b/UCbHC83Mvq3Jmmcla7IZdwDmOnuX7j718CzwDG5xubuy4GlQNeolfE3M3sBuD1qoVwdxfGKmZ0exdjKzP7XzF4zs8eBTjlcqifwZHTNBcB2ZrZJrnFWumJ/j/NRwjJOLJ8+zR2Aa9y9J/BeI/tdC4xy997AcUAohD3M7KYGjhkNDAES39o3s72BL919VfTWlsCe7j4UGAY86u67AwcBV5lZG+BXwMfu3gMYAewaO99Yq6fpbWa7AZu5+9SkMVaRYpXxq8ABlmr+tgMOA7rkGpSZdQe6AW/F4jzY3U8CzgRWRGW8G3COmXUFjgX+i1RFeCqwd+x8l5lZ9rKmKS8DA6J99gI6R/9akmJ+jx140szmmNlpSYIqYRkD7GtmL5vZw5ZDF0w+z54vcfdcmqV9gO0ts+zuxma2vru/CNTp54j6IP7l7vPMrE+CeIaY2c+Bz4DjY++PjzU5+gKHmdnvotdtgK7A/sAoAHefa2YLwsHufmo9MbYCrgJOTBBfNSpKGbv7fDO7GpgGfA7MBb7N4TonmtmBpDL70939k+iaD7r7l9E+fYEeZjYwet0e2JZUGd8d/S4sM7OnY/E01I91GXCtmc0jVYG+nGOc1aQoZRzZ093fi5raj5vZInef3sR1Sl3GLwE17v551PKcSKqCblA+lebq2PZ3pJrEQbx5a8Du7v5VjufdGxhgZkdF59nIzO5w90FNHHeFu49uIk4Djnb3JfEdYr8IuepA6q/Zc9GxWwAPm9nh7j436ckqWLHKGHe/GbgZwMxGAYtzOOxOd7+giTgNGOzuT8R3MLOcm/+xGD8l1fca/lAuBVraDCDFLOP3op/LzexBYHegqUqzHGUcth+KuvM6NNYPW5AhR1HN/rGZbRv9csWDnwacE17U19Stda6h7t7Z3WuAk4DHQoVpZqNCP2QzTQXSd2rNLDTDnwV+Fr23C7BjEzGucvdO7l4TxTkb6NfCKswshSzjaJ/Nop81wFHAPdHr883srDxCnQoMNrN1o/Ntb2brkyrj46N+r62BA3KIsYOZtY5e/hKY5u6rGzummhWyjC11/2CDaLsdqXsU86PXlVTGW8S29wS+aerGVSHHaf6W1H/MdGBZ7P1zgH2iDtuFwBlRgI31hTRkZ2B5HjFeCrSz1DCIBcDw6P3rgU3MbBGpO6bpyq+hPs21VCHLeFK07yTgLHf/d/R+D+CjPGIcA7wJzDOz+cCNpFpUE4B3gYWkbjDOCAc00t/1A2Chmb0OHExqREdLV6gy3hJ4wcxeBmYBD7j7tOizSirjgZYa9jYPuIbsrr16Vc1jlJZqBz/i7oeWOxYpHjP7B9C/CMNKpEJUexlXTaUpIlIJ9BiliEgCqjRFRBJQpSkikoAqTRGRBFRpiogkoEpTRCQBVZoiIgn8f6S1hmwDdPmXAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fab0c040dd8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_images(images=images,\n",
    "            cls_pred=cls_pred,\n",
    "            cls_true=cls_true)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualization of Layer Weights and Outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Helper-function for plotting convolutional weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_conv_weights(weights, input_channel=0):\n",
    "    # Get the lowest and highest values for the weights.\n",
    "    # This is used to correct the colour intensity across\n",
    "    # the images so they can be compared with each other.\n",
    "    w_min = np.min(weights)\n",
    "    w_max = np.max(weights)\n",
    "\n",
    "    # Number of filters used in the conv. layer.\n",
    "    num_filters = weights.shape[3]\n",
    "\n",
    "    # Number of grids to plot.\n",
    "    # Rounded-up, square-root of the number of filters.\n",
    "    num_grids = math.ceil(math.sqrt(num_filters))\n",
    "    \n",
    "    # Create figure with a grid of sub-plots.\n",
    "    fig, axes = plt.subplots(num_grids, num_grids)\n",
    "\n",
    "    # Plot all the filter-weights.\n",
    "    for i, ax in enumerate(axes.flat):\n",
    "        # Only plot the valid filter-weights.\n",
    "        if i<num_filters:\n",
    "            # Get the weights for the i'th filter of the input channel.\n",
    "            # See new_conv_layer() for details on the format\n",
    "            # of this 4-dim tensor.\n",
    "            img = weights[:, :, input_channel, i]\n",
    "\n",
    "            # Plot image.\n",
    "            ax.imshow(img, vmin=w_min, vmax=w_max,\n",
    "                      interpolation='nearest', cmap='seismic')\n",
    "        \n",
    "        # Remove ticks from the plot.\n",
    "        ax.set_xticks([])\n",
    "        ax.set_yticks([])\n",
    "    \n",
    "    # Ensure the plot is shown correctly with multiple plots\n",
    "    # in a single Notebook cell.\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get Layers\n",
    "\n",
    "Keras has a simple way of listing the layers in the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_2 (InputLayer)         (None, 784)               0         \n",
      "_________________________________________________________________\n",
      "reshape_1 (Reshape)          (None, 28, 28, 1)         0         \n",
      "_________________________________________________________________\n",
      "layer_conv1 (Conv2D)         (None, 28, 28, 16)        416       \n",
      "_________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2 (None, 14, 14, 16)        0         \n",
      "_________________________________________________________________\n",
      "layer_conv2 (Conv2D)         (None, 14, 14, 36)        14436     \n",
      "_________________________________________________________________\n",
      "max_pooling2d_3 (MaxPooling2 (None, 7, 7, 36)          0         \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          (None, 1764)              0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 128)               225920    \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 10)                1290      \n",
      "=================================================================\n",
      "Total params: 242,062\n",
      "Trainable params: 242,062\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model3.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We count the indices to get the layers we want.\n",
    "\n",
    "The input-layer has index 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer_input = model3.layers[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first convolutional layer has index 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.layers.convolutional.Conv2D at 0x7fab0c052d30>"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "layer_conv1 = model3.layers[2]\n",
    "layer_conv1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second convolutional layer has index 4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "layer_conv2 = model3.layers[4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convolutional Weights\n",
    "\n",
    "Now that we have the layers we can easily get their weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights_conv1 = layer_conv1.get_weights()[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This gives us a 4-rank tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5, 5, 1, 16)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weights_conv1.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the weights using the helper-function from above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUoAAADuCAYAAABf005JAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAEIBJREFUeJzt3X9wVfWdxvHPkSRcQn5z0RiQ3DGKVHBRSZEplmUoBbQWRkotrhRBZix2QDfF7uIuoyiOynYGdpHuFBYpg12FVqEtsAJa3JVCGUxsqsEFQUwMDb9CfpBAbhLC2T92t/8xz+fObPZc6vv19+P3c/3m5uFk5nvOCcIwNADAlV0T9QcAgHRHUQKAQFECgEBRAoBAUQKAQFECgEBRAoBAUQKAQFECgJCRSrgwCMJBjlwsN1dmzre1uWZ6mjynqEhmatvbrbGzM3ANjUB2EIQFjlxJSYnMnGlocM2M3TxKZvL6dspM7R//aI1NTWm7twVBEOpdM8vu21dmznbq/TAz078BZhdcK5nVmjWGYTjQGf9/Fy8oCBPFxTLXc+SIzPh216zP/1Gm3szOhaH87qZUlIPM7A1HbtiYMTKz++23XTNjjsy4yZNlpnzXLte8qBSY2aOO3NLvf19mVi1Z4po5bHWlzEy66bjMlE+b5poXlRIz2+jIlQ8ZIjM/OXrUNXOCI7PftZLZXLM6ZzQSieJiq1y/XubOjx0rM8ecM/McGc+Fx0TnPP70BgCBogQAgaIEAIGiBACBogQAgaIEAIGiBAAhpXOUoZl1OXJJxxnJcc6ZsVdekZmGefNkpts5Lyol2dm2dPhwmXvHcUbyja/6Xu9x97/rzKxZN8pMc7M+qB2l7JtvtvLVq2Vuo+M87vw1a3xDj+kTgUPnz3ctNbeszDczKidOmC1aJGO1f9Dfy2MjffctTHFk8hw3vmRc8B3754oSAASKEgAEihIABIoSAASKEgAEihIABIoSAASKEgCElA6cXzKzJkfuc0dGPw/5v8WWLZOZP4u2v/FGs02bZOyR8foAeP3e110j29sflJkzMx+XmfLN9a55UUlm5dkniUky1/JPjoP6GfoBtWZmlpOjM84HLKe7qgvDLDjwW5mrcxwmH1Pnu1ni81K91ohBjvcx1Na65v1ZdAwA9CaKEgAEihIABIoSAASKEgAEihIABIoSAASKEgAEihIAhNReBXHzKOtaXSlzh5N6rYTnWe5mtnmrzixfrjNHD5f7BkbkTGtfW7XdcddN/Tdl5o47trlmfrD45zKz4Tsvy8w517ToHD5sNmaMzt11l84cKHzENfO1h3bIzN8ln3atZea70yoqoxLnrPLZf5W5Yw/rtW7a8Jxv6KFDOrNzp86sXOkaxxUlAAgUJQAIFCUACBQlAAgUJQAIFCUACBQlAAgUJQAIKR04P37cbOZMnWtu/rXMhOfuds1cuLBIZn6rn0Jv06e7xkWmvr7Znnhis8yFj92gF1vwsW/ozOdlxHFO2/r7pkXm+uvNKip0rrpaZ1599YRr5muxN2XmhW+5lrIXfbHodHSY1dTI2Atz9WseFjyjX/FgZpb9zDMyM+z0ab3Qxo2ueVxRAoBAUQKAQFECgEBRAoBAUQKAQFECgEBRAoBAUQKAQFECgBCEoT4t/6dwEJw1s7re+zi9qjQMw4FRf4grYW97z1W+t2bsb29y7W1KRQkAX0T86Q0AAkUJAAJFCQACRQkAAkUJAAJFCQACRQkAAkUJAEJK78yJB0FY6si1ODKFRfpdOGZmdvmyjHx2TZnMXLhQa8lko++FHBGIFxSEieJiHWxokJFkW5trZuy223SoRf80a5uarLG9PW33NggGhGZDHEl93TB4sO9/s7/jRUKXLrmWsk8/rWpM5ztzYrF4mJOTkLlEz6cyc6q//l02M8vO1pmjR7scK52wMGySP9SUirLUzH7nyP3SkXlg8mTf0GRSRmbnbJGZHTvKffMikigutsq1a3Vw6VIZ+fjdd10zb92+XYe2bZOR8uXLXfOiM8TM/sORy5KJioqYa2K54+vm+DfIzMymTQvS+vbAnJyETZ1aKXPrW/Qb/pbfpX+XzczuuENnJk/2bNs3XfP40xsABIoSAASKEgAEihIABIoSAASKEgAEihIAhJTOUXaa2XFH7oFdu3Tovvt8Q8v0AdT2L+llHOfWo5WTY3b33TLW5Tgjeeu+fa6RDaX69gHPmWjPsd4olZX1sRUr8mTurbf0Wj849TeumR8W/IPMjPjLtD2jn5LEpWO2vnGqzE3P+LXMbMn5sW/o4R4Zqap6XGZmzdJnZ824ogQAiaIEAIGiBACBogQAgaIEAIGiBACBogQAgaIEACGlA+fJslH2yQr9gM4zk/VB2nEx3wNQ31vznzKz5cnRMlPepdeJUl2d2aPz9b9ba48ckZnRs4a6Zh7cvFlmJq17QGZqD6T3Q5HzPq2yCdP0d/Jez2I33OCaOeRHP5KZggcfdK1lr7/uy0UlN9ds/HgZ21L5V3qtuetcI9dv0o84H96t1wlD1ziuKAFAoSgBQKAoAUCgKAFAoCgBQKAoAUCgKAFAoCgBQKAoAUBI6c6cgq4zNrV2lcy951irOpl0zRwXO6hDM2fqzMqVrnlRyc/3vR0juMXzcoY618zgO/ruqKee0ut89JFrXGRCM/O8CSRjzRodOnDANfNnP/2pzCyYONG1VtrfmXPypNmyZTqXSMjIa/37u0Y+UlCgQ+fOyYhzHFeUAKBQlAAgUJQAIFCUACBQlAAgUJQAIFCUACBQlAAgpHTg3FpbzXbulLGcKv189TONvpFFU3Rmxgz9Koi65Gu+gREp6DxtU4+tkLldtkhmJn35y66Z1e+/LzO3bx0mM7tbal3zotJn5EjL27NH5hoGDJCZU86ZC+r0of+SMUOcq81z5iIyYoSZY39POPY37hz5jy0tMjO4Tx+ZaXbO44oSAASKEgAEihIABIoSAASKEgAEihIABIoSAASKEgAEihIAhCAM9V00fwoHwVnzvmcg/ZSGYTgw6g9xJext77nK99aM/e1Nrr1NqSgB4IuIP70BQKAoAUCgKAFAoCgBQKAoAUCgKAFAoCgBQEjpVRC5ufEwHk/IXKPjNQ/t7b7zm/36BTLjeMK8NTXV2oULjXqxiOQFgetEcWFmpsyE3d2umW2OjGfDTplZaxim7d7G4/EwkUjIXFhVJTPBiBGumcmaGpmJZWe71qq6eLExnQ+cxzMywkRWlsy1d3TITE6Gs5KGD9eZQ4dkpLanxxovX5bf3ZSKMh5P2LPPVsrcunV6rb17fb/MN92ki2HOHL3OypXlrnlRGWhmyx25GcXFMtNVX++aqd9yYhZzZL7nmhadRCJhlQcPytwlxztWMn71K9fMw2VlMjPM88tuZsH776f1XS+JrCyrvOUWmdtfXS0zX4k735qze7fOjBwpI+WeqzrjT28AkChKABAoSgAQKEoAEChKABAoSgAQKEoAEChKABBSOnCel2c2caLOzX6iUIcq5vqGbt0qI+8tqpWZ/r5pkSm4c5RN/Z0+zN/SV98Ac/mc766nKckGmVm/s0RmOp9L78P8Vl3tun0rY80avdawYa6RwxYulJmLL61yrWX90/amJzMz+yTjVptQqL+7v3Tc57Xn1CnXzFPXXScz4x3r+G574YoSACSKEgAEihIABIoSAASKEgAEihIABIoSAASKEgCElA6cZ17TYyU552Xu8rlmmfnnPr5DtPoB82a+h/Ont+Bkg2U9/7TMzZ+rD5OvnznJNfP422/LzIxWPe/ll13jItPR02M1LS0y9/n39LPa7739dt/QDRtkJNtxaPpq0NbWZe++qx/Cnm+tMtPZmeeaedFx44WH90qRK0oAEChKABAoSgAQKEoAEChKABAoSgAQKEoAEChKABAoSgAQUrozx7q7zU6ckLGNw4fLzILVq10jDy5YIDOjv/tdmcnZscM1LypVJ+MWLJsnc+G+/TITjNWvzzAz67EcmenK13dApPu/trE7R9lQx2s2RtR8oBdrb/cNbWuTkdmbv+Fby5Y4c1GpN7MfyFR4T4fMPPfSv7kmjndkChyZLte09P+OA0DkKEoAEChKABAoSgAQKEoAEChKABAoSgAQKEoAEFI7cH7okJnjMPnlV/TrA2z8cdfI0RUVMvNCfIXMnPxNuWteVAYNyrKFC0tl7sMcnXnpJd/Mn+Tqn9OYMXqdjlnpvbddXa77JOxSzp0yM3TJvb6hjoPpGxObXUu96psYmWuvLbOHHnpTB8t+LCNPf2mPb+gPf6gzNTUy0m/fPtc4rigBQKAoAUCgKAFAoCgBQKAoAUCgKAFAoCgBQKAoAUCgKAFACMLQcRfN/4aD4KyZ1fXex+lVpWEYDoz6Q1wJe9t7rvK9NWN/e5Nrb1MqSgD4IuJPbwAQKEoAEChKABAoSgAQKEoAEChKABAoSgAQUnoVREZGPMzMTMjc8OJzerGmJtfM7vPnZaY2b5TMdHTUWldXY+AaGoGiong4eHBC5rKCbpmpa8h0zSy9pl5mzuffIDOnT9daa2v67m28X78wkZ+vg336yMjHzde7ZnZ09MhMZqaeZ2bW3V3VmM4HzguDIBzkyMUcP4MLra2umf1vu01mPqnNkplk0tcLKRVlZmbCEolKmat8aqNebNMm18yGt96SmXlf0Z9p//70fq/L4MEJ275d/38MyWiQmUeXlrhmro09LjO771slMwsWpPfeJvLzrXL2bB10/CLf+ebfu2b+/vfNMlNcXOhaq74+SOu7XgaZ2c8duVvHjZOZg9u2uWaO3r5dZibMGSIzlZW+7y5/egOAQFECgEBRAoBAUQKAQFECgEBRAoBAUQKAkNI5yoICs/vv17mm+/SZtaInn3TNPPwb/WDhr1frdWpqXOMik/lRlZWUOs5sHzkiI8eO+WY+dos+I/ltxzckSNuj5v8jP99syhQZO/61r8lMpS1xjZzydf29nTXLtZQ9/LAvF5nhoyxjiz4DbIuny8jo3FzfzFhMRvYs2SMz5Y+1ucZxRQkAAkUJAAJFCQACRQkAAkUJAAJFCQACRQkAAkUJAEJKB84Hnf7AXljZT+bOvJiUmTecMz94R2fecCx29qxzYESCsjLLWLFC5k7nD5WZdet8M3/xC52Z8Nd/ITO59c4T7hE5m8y1tccmyFzCsVb7H/RBcjOz3QWfy8yjz+sHy14NYs0nbeim52Tuna1bZWaEc+aH1dfKzOTJ7Y6V9JPozbiiBACJogQAgaIEAIGiBACBogQAgaIEAIGiBACBogQAgaIEACGlO3N6wtBakvqum4uOtWZUVLhmzlh8XmZefNFzut53Aj8yn33mejfAdY5XGtiGDa6Rixd3yszfVkzUC50+7ZoXlcJCsxkzdK7oZ1/Vofb9vqEb9O1iay+1uJb6F9/E6AwYYDZnjoxNfMdxm92pU66RKxxLHTgwVWbmzNF3FJlxRQkAEkUJAAJFCQACRQkAAkUJAAJFCQACRQkAAkUJAEJKB84D53+Q8BxAPnzYNbMlP19mwjFjZKb8o6OueZG5fNmsrU3nli6VkaD/h66RO3bofVvxjZUyk97Hzc0yTtRa0ZOPyFzl3r0ys2fsWNfMLEdmumulq0B3t+ug+E7H/k655x7XyEWLdOb++3Wmvt41jitKAFAoSgAQKEoAEChKABAoSgAQKEoAEChKABAoSgAQKEoAEIIwDP3hIDhrZnW993F6VWkYhgOj/hBXwt72nqt8b83Y397k2tuUihIAvoj40xsABIoSAASKEgAEihIABIoSAASKEgAEihIABIoSAASKEgCE/wJH48F53JAKZwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7faa856a42e8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_conv_weights(weights=weights_conv1, input_channel=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also get the weights for the second convolutional layer and plot them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights_conv2 = layer_conv2.get_weights()[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVMAAADuCAYAAACEaORrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt4VeWZNvB75wA5bEJINgIhhiUgoESkmPHCE1rLoB2opeiHqIwylIJaPkSGWkXqxziIjAdqHWwpUmW4UFAQqR+2aNUiooMYEBEUkcMGwnlDQggh5/X9IfFy+jzP2uzhzZ7Lz/v35714XO/K3nnc2etd7xvyfR9ERHR2Uv6nB0BE9P8DNlMiIgfYTImIHGAzJSJygM2UiMgBNlMiIgfYTImIHGAzJSJygM2UiMiBtET+cSQ72/dyc+WBTp3souPH1bgqta1ZsmePzOrqomhoiIXijfFsRdLTfa91a5E3nTxp1qS0Na4lP98+0eHDIorW1CBWX9/i15iZGfFzcjyRn5sZM2sON0XU/JxWFfaJMjLUeP2WLTHf99sHDtKB3NyIX1DgiTw93a5Jb6zRDzQ12UV1dSKKHj6MWGVly79fMzJ8LxyWB7Ky7KJjx/S8utquUX7HoxUViJ082eLXCAB5eRG/c2dP5K1PGtcCAA0Neh7wu1xfeJ7IysqiOHo0fu9JqJl6ubkovftueeCBB+yiP/9ZjVe3GWyWjB8vs+3bS+INzwmvdWuUXnSRyKvXrjVrsgYM0A+MHGmfaPZsEZV8/HHc8bmQk+Ph5ptLRf503+fMmqerRqv5hMJl9omKi9U41LPn7uARulFQ4GHhQnmdHTsG1FRt0w9UVdlF0aiISu67L87o3PDCYZTecIM8cMkldtHChXq+caNdo/zel/z2t3FG507nzh7++Ef5Wnb9zxfsogrjf/QffmiWHHp8gcgGDTqz3sM/84mIHGAzJSJygM2UiMiBhL4zRadO6vejh47YPfmP+/TvRq3vhgGgrExmynf8LaOgAPiXfxFxVmGhWbI6dqGaD5g+yD7PtGkyu+uueKNzIhwGrrhC5sNe0r8XBYBlN+rfTS2ous2sWXRPwkNzKqt8H/otnSIP/O53Zk3doXI1b7Uk4Lu5V16R2dGj8YbnxOYqDz3WyO+6czfbNf37/1zN0y6za6Yqb82GJa/GG54zrbd+gq5XKDe6X3rJLnr9dTV+buCLZsnoQ5tElt5wKu74AH4yJSJygs2UiMgBNlMiIgfYTImIHGAzJSJygM2UiMiBhKZG1dQAn22V/TfgSUtE9Ee6ceONB8ya9evlFIigJzNd+nhHDnJuklOaTpyoNGvOk4/zAgD693/TrFn0g11K2ire8JxoF6rAzenyMdD/tXSYWbNgoT4Fatw4+zwvGLOJVq4MHJ47qamAtpbE975nlrQ6qCwMAQQ+Mn1pR1nzeV1yHn/u2lV/OvTLL+2axkY9v+22erNm4kS5oEHQcgWu7WxzMYZfJR8nXTnEruncWX/Me+si4wcAYPQh5VnjoMUcvoGfTImIHGAzJSJygM2UiMgBNlMiIgfYTImIHEjobn7G/p24cNpw+R+Z/rJZM3++nodC9ur8/cpeE1lWXcCK7g59r1slSmcrd+EHDjRr+hk3boMWIX7jDTkFYPx4ucJ/i6iqUqdgpGgrtp92u6evmj+tk7EwNoARIxIfmks1uR2xbahcpHnIPHvh5uKJer68zLjLD+C63jJLS2wJof+2rKN70W/+BJH32xyw0sm8eWp86x9WmSWXj5CL4OzcGXd4zkQiwJgxMjc2cwAALPhUn7Vx3y8CFmHX3rRneKH8ZEpE5ACbKRGRA2ymREQOsJkSETnAZkpE5ACbKRGRAwlN4IjldMVz18tpUEONxUwA4NFHtQU9gF/9ylgdBMA7YbkP+InUh+MP0IFYXQ6eK5MLnVx/0K6xZqEELc4yqOSYyHKyAjbGcmj9oUKkPPGYyJtGyv1vmt06s4+a7xwV8LpM1OcZhdoGj8+VU6eAj5VZMNuul1OJvlaiz3PbNO12s6TPZPl+KfG3xR2fE61bA927i3hu8dNmydipt+oH+vY1az74uVy1puRX8j3cUnKO7sKg+XLcAxYtsotqa9X4sTXv2DXFo2S2S+9hf4ufTImIHGAzJSJygM2UiMgBNlMiIgfYTImIHAj5vn/m/zgUOgJgd8sNJ1AX3/fbt/RJeI1J8V24Tl6jQ9+G60yomRIRkY5/5hMROcBmSkTkAJspEZEDbKZERA4k9Gx+pF073+vcWeTrt9jbbRQWhtS8Qwf7PPv2yez48Siqq2P6f8yhjIyIn53tifzUKbvm/PP1vKzMrsnMlFl5eRQnT7b8NUYiEd8rKpIHmprMmmiZ/lbRrqNZh4zjar5++/ZYMu4C5+dH/MJCT+TpJwO2wMnNVeODAWszdDy5Q2TR6mrEamtb/LXMz4/4RUWeyFM/+9Qu6tlTz6uqzJK6cJ7IysqiOHas5d+vABAJhXxPO9Crl1lzKiVbzTOjn9snamwUUbS+HrHGxrjXmVAz9Tp3RunLcqGTUO+uZs299+qbtEyaaP/iTpkqPzA//7yx0ZJj2dkeBg8uFfnGjXbNm8qWUQDwi1/YNb2VfYP+/d+Tc41eURFK16yRB2pqzJrRk+UvEwAUF9vnmdTzdTUPDRmSlCkuhYUe3nxTvpYd3l9m1jQNHabmjz9un+eXH8qakr/+Nf4AHSgq8vDuu/Iacy62FxLCihV6/t57Zsmeq24T2ZAhyXm/AoAHYJ2Sp/zHf5g1mzIuVfM+Y/QcAFAh/0dbssfe/+u/jOWM/hUREQViMyUicoDNlIjIgYS+M60NZWBnxoUif+UVu2bYfLnQMwD0mPOaWbN9u8yS9aBWly7AnDkyz7pC34MbAHCN/l3jgg8/tGvmzxfRkpTDcUbnRl1DCvbEskReNGqIWTN9ob6grnG/5itr0hMdmlPpoQZ0CCk/U+s7QwAzt+rfmU55MOD+w9/9nczq6uINz4nUUBNy0qrlgbffNmvqOio3HwHMOy6/F212QlmD+Vjy1oYGCgqQMm6czNeuNUv6/Ppm/cALcqHrrym/l3j11eCxncZPpkREDrCZEhE5wGZKROQAmykRkQNspkREDrCZEhE5kNDUqNb7dqLr/cNF3nXyZLPmwwf1KVAT19vnKSyU2aRJcYfnRMqWT5HVW3kU7/e/N2s+K5T7pgPAhdPvs0+kPQcd8DinS62i21A0Ro75w0ft/cTfnq/nATNT0L27/nP5H6c8Mthsyv+9TD+gPLPdbPUa+ZmkamxyHrU8Wp6CBUvlNLc77vjIrElP1x//njrVPs+YMTJ78cW4w3Omum0nbBjykMhXrrRrUu+coOa/7KtMJWumPeufoT8S/7f4yZSIyAE2UyIiB9hMiYgcYDMlInKAzZSIyIGE7uZ/XtsVl0bl4tBPNdg11qLKQQtk3LB8tMgerojGGZ0jeXnALbeIuLK/fWf6wlLjLvjVV9vnef99maWmxhudG0VFwOzZIk63F1rX/nlgDgDD1uhTMH4dNDaH1n9Sg1CHL0Senm4vDl23doOah1LtxZ5/8pMfiOzQoTMYoANt2gDXXCPzf/xHY5EPACXGRIMJG+Xv3ddylfdLqr3Au2vRKDBqlMwDJmZg+nTjQFBRg9LMznCVJX4yJSJygM2UiMgBNlMiIgfYTImIHGAzJSJygM2UiMiBhKZGXdCtDuuWyj2kQ13sOTU/+5ncMwoA5k7bb5+oTFlswNqc3rGdtZ0xfPsMkS9p+6lZ42/pqObDpurXDgDLap6RYXXAAgwOHTmRgbmreoh86VK75sABZWMuAH37dreLqvomOjSnCgrCGDfuKpEPHWrXnDOwn5r7s5XX67Q/nSenRn1qv12cOnxYn56mraPTbMLmsfqBaNSseWetXEzlxMnkfRa7sFcTStcovx+lpXaRsQDThNJ1ZklVlVycKFojp4Nq+MmUiMgBNlMiIgfYTImIHGAzJSJygM2UiMiBkH+GD/EDQCgUOgJgd8sNJ1AX3/fbt/RJeI1J8V24Tl6jQ9+G60yomRIRkY5/5hMROcBmSkTkAJspEZEDbKZERA4k9Gx+pHVr38uSz+juQDezplUrPc/Ots+za9cpJd0P3y8PxRniWcvMjPht2ngiz8+3a2pq9Dy/6YhZ8+lBeXOwoSGKxsZYi19jdnbEz8vzRN6hvswu6txZjatP2cPNOqb/99YfOhRLxl3gSH6+7517rjygbU1x2oHyDDXv1Bjws1H+e9GqKsRqalr8tQyFIn5Kiifyrl3tmjZt9DylqtIuKi8XUfTEiaRcIwCEwxE/P98TedAOJKe0NgKgXTu75tx2clGD6MGDiFVUxL3OhJqpl5WF0u9/X+TDYO+p43l6bu1DAwC33aatEjEieHCOtGnj4cYb5eIJ2v4zzb6Q2wwBAG6vmWvWdJ0pF5vYty/gh+JQXp6He++V1zjpoFzk4WvGhjobNhv/twTQb7H+3ws9/nhSprh4556LUm2BnFjMrJmxXF+cZkpFwM9G+e+VvPZa3PG5kJLiITtbvpa/+Y1do+0ZBQBZawIWE1JWwSl59dU4o3MnP9/Dgw/K6wwawpYten7TTXbNrJs+EFnJ6IC9sb6Bf+YTETnAZkpE5ACbKRGRAwl9Z9rQpRuOzZPfjy4r22TWjJ3dR82D9lv3V8gFqEsm1sUfoAORCPCzn8k8LeAnVVxsHJhjL1y7c55cVLnkrhNxRudGh5M7MWntcJHPHWgvgnulvjY0nnrKPs+C6y5OdGhupaV99YL+LWPRYACYMmeOmo+d+JhZk6ucoiwtOd9/f69pPdadkPdGJr9lP9lovV+LVq50NSzncnOBH/9Y5kH3Mjrqa7Zj4EC75rcbLxfZkVPh4MGdxk+mREQOsJkSETnAZkpE5ACbKRGRA2ymREQOsJkSETmQ2NSoBv1JvDHT9OlPAPDEE3q+eHHAibTnqa2H/B3LOn4A/f78iDywY4ddVGY8t903YN/4zz+XmfUwsWO7U7tibK6cBtUrYK/1O+/U86A96GMjRyY4Msf27QOmTpX5l1/aNcZcm7nWDwDAXUuuFVlTU7zBOVJQgJRx40Q85Eq7pMp4nQeUzjJrPvpIZrW1G+KNzpmGBuDQIZl3eN9+lD0cHqbmY8bY57nnHpk1NsYb3Vf4yZSIyAE2UyIiB9hMiYgcYDMlInKAzZSIyIGE7uZnHNiFHtNvF/nBgwvMmq7L9TuExcWT7BNpt9ui0XjDc2L9/jBCU69QjvQLqLpUTf2V9pLeoVTtFuFzgWNzJSMD6NVL5takBAB47z19O4GJE/WV6QEgcsst+oFFi4KG507btsD118s8YKX9Zf31BU2GNdgLJ3fqJLP09LijcyM/X52B0DdgbQ5rBsaIgPXXVy8/JrKSa+2fo2uZoRr0ydgm8sqB+h17wH6Z58+3zzPoGrmg0pKX7UVjvomfTImIHGAzJSJygM2UiMgBNlMiIgfYTImIHGAzJSJyIKGpUcjPB5TFKz6YPsCuWaGfIm1ywNSoFStkNmhQvNE5ccnFmSh9Ry7cUhfOM2veess48JS9cIT/iieykvuOxxueE7m5+vQYbbukZrPyn9Tz6INmza140TiSnKlRFQ1hLIvJ9+aIp+z3a339YTV/+237/ffQVLmqyWuvncEAHfh8RytcelORyPv3t2sKC/X87ut32kXjlQVj9u6NMzp3GtIycCzSQ+RBDcza0ipokaWlS+WCSrv3yD22NPxkSkTkAJspEZEDbKZERA6wmRIROcBmSkTkQMj3z+whfgAIhUJHAOxuueEE6uL7fvuWPgmvMSm+C9fJa3To23CdCTVTIiLS8c98IiIH2EyJiBxgMyUicoDNlIjIATZTIiIHElroJByO+Hl5nshzcuyajKqYmn9x1F5Vo3t3me3ZE8XRo7EzW3HgLGRkRPzsbE/kp07ZNdZeM326Vpk1m6Nyk576+igaG1v+GnNyIn779p7I29UcsIvCxqZCmZl2zZ49ary+vDyWjCk11nUG7c8UrtYXOsHJk2ZNVfvzRHbwYBQVFS3/WrZrF/E7d/ZEnppq16RvWq/mVT0vMWvCx/eJLHr8OGLV1S1+jcBXr+U553giz60/Yhe1199itbV2SWWlzI4ejaKqKv5rmVAzzcvzMHlyqcgHDrRrLlyrbxI3YP5os0ZbNOrqq0vijs+F7GwPgwfLa9y40a6J6f+/QOnc1WZNjzFy5aI9e5Jzje3be5gxQ17jzdsfsYsuu0zPi4vtmvHj1Ti0ZElS5gta13nuuXbN5aVP6wdK5X+n2eoxckPJsWOT81p27uxh2TI5trZt7Zr2HfW+sGaufY0DVk4RWcnzz8cfoCPnnOPhscfk+IbF5tpF2sacAHZG7T/ItRXgHnnkzF5L/plPROQAmykRkQNspkREDiT0nenevQ24556jIvd3BKwQv3ChGnfsaH9nOmqUzKLROINz5Lw2MSy4Rvmed2iuXWTdnFlhLcEPbJsvf/Qlo+0bVi75PtDYKPNlF9ir5t/4g3o1v+oq+27Oe+9ZOw0sCRqeMw0NwFH5dg28Z7ZzyAQ1X2W8xAAQqdDPnQwZlYfR4y/PyAMXXGDW7NmtP0LevUvAPRbtB/mnP8UbnjM7dlTjxhs3iNx/P+A7+2nT1LjrkCFmyciRl4pszpy4wwPAT6ZERE6wmRIROcBmSkTkAJspEZEDbKZERA6wmRIROZDQ1KhLup9E6VNrRb4/Y7BZk7viHTV/eeok+0QzZ4qo5LIk7QjQ0KA/H+p5do31PKk2x6tZRobMgh4adyivMopb31Kmpr36qlmzC8r8HwCR9+zztDLy1gFjc+nECWDVKpkbT7kCALq+pT+euLVwrFmjTdur12eSObd+bxZC4/uK/OTJK8yahU/p+VPt7d+xNGUGUiyWUPs4S6kAlEVAwgFjuPNOPbemMgLIzt6upAEP838DP5kSETnAZkpE5ACbKRGRA2ymREQOsJkSETmQ0O24xnBbVF6l3LkPWJ8jq28P/YB1pw0AWifrfq8iHAb695f5HXeYJbP+9y41n3REWYCi2bx5MtuxI97o3EhPBzp2TKjE2hch/MkndtHBg3p+3XUJnfu/KzcXGDpU5gM8fQcAAKjsq9+1HxKwzo22qMlz+proznXrFsasWfLO/cqVdk1hoZ5bLxcApPxOvpdL/s3YlaAFZGe3Rp8+cguOBQGLtt8+sknN8yL2Z8gf/UjOGFi9+sz6ET+ZEhE5wGZKROQAmykRkQNspkREDrCZEhE5wGZKRORAQlOjNm3Sp1VUznvZrHl6/DY1v+Ya+zybX5CLoBybmpx9yFFeDixdKuI97+rTnwAgY4Vx4M677PP80z/J7Mor4wzOkbw8YMQIEX82coZZUmxstdO0NmDf8r5yAY5kqqkBtm6V+fDlRWbNy5PXqfm4cXJvoGYrlNe/Ql8XxrncPZtww3jleh591C46uE/Ph64xSzZMe01k1eHn4w3PGeu1vL1U37MLACZtfFrNu8sZVl97bbmcTlViv/T/BT+ZEhE5wGZKROQAmykRkQNspkREDrCZEhE5EPL9M98OJBQKHQGwu+WGE6iL7/vtW/okvMak+C5cJ6/RoW/DdSbUTImISMc/84mIHGAzJSJygM2UiMgBNlMiIgcSejY/kpvre9p2F/uMZ30BoGdPNY7F7JLdu7WbYrvh+7FQ8AjPXkpKxE9J8USeG7BtRcgYVVaWXdO+Sj7rH62qQqy2tsWvMRKJ+J7nyQNlZXZRk74FxLYa+zn39HQ9P3ZsfSwZd4EjmZm+16aNyPen2WMuCFeqeU0ruZ1FswzUiCy6bx9i5eUt/1oav5Nby8JmTX29ngftZKO9lw8ciKKiouV/JwEgkp3te9ovYUaGXXTsmBrvbdXNLMlRXubDh6M4fjz+dSbUTL2OHVGqbW5z//1mTdOq1Wo+f759np/+VL45AbnPTUtISfGQk1Mq8htusGvSjJ9i0Dofd6+5VWQlb7wRb3hOeJ6HdevkNabcf59dVKVv9HXt1t+aJdYv56JFoaRMcfHatEHpTTeJ/KGIPeaHr3xTzbd5g8yaHg2fiaxk+PAzGOHZs34nL598uVlj7fU0ebJ9nksukdmoUUlafAiAl5uL0rvvlgfOP98uWrxYjSd5y8ySgQNlds89Z3ad/DOfiMgBNlMiIgfYTImIHEjoO9Mtu8O44Kfyu5jP319u1qSU6XuUL1li3wTwv5A1JcNqz2CEZ693b+BN5WuzF16wa/75nzeoebt2/cyauyf2kuG778YbnhNVVcDatTK/3NpQHcCE7foivEHrWc+cmejIHMvOBvr3F/HCaXbJw9OUL80A9Fj7gV2k/TCPHw8emyONmWFUFsvfyQ9W1Zk1x6paqfkVAbclGhtltkf/1W4ZbdsCP/yhiP/tL/bv2C+nKb9jAGaFd9rnWbVKRA/VB9wt/wZ+MiUicoDNlIjIATZTIiIH2EyJiBxgMyUicoDNlIjIgYSmRvVO3YrS3MtEPmPOf5o1TzyRp+azZwecSHtG3Hqg2LH08sPosPQZka9d+3Oz5skn9ekZQc/zI82TWSt9yopr6V+sR8crlEeN337brKnZrOf/+q/KtKDT3nhDTksCgOuuCxyeMwfq8jGj7HaR77xmtF1U8YQaf5hqP56JK+Sxk8++GHd8LmzfDgwZIvPVq+xf7bwR+qOxzzyjP0oLANdunCWykl8fij9AR8prs/DSl/L37IEH7JqLLuqj5v9Q+rBZs23EQyKrCduPH38TP5kSETnAZkpE5ACbKRGRA2ymREQOsJkSETmQ0N38A2174eEfyjv3/+dBbTHnr/gLX9EPbPzEPlFxsczq7IUbXKpvdw72/0TeuX/Ze90u6tRJzydONEt2zpeLZteGn447PhdSAWjrsN+15Fqz5vrr9XzuzB5mzUt/SWxcrqWmAmHlQl8bqixwftoNS+eqeWb/sWZNn5p1IsvGyfgDdKCwEHj8cZmnpNmfkxYu1O/a3zrdfv1RKhcTx8nkXCPw1aL5L70k84BfMXOhnU8Hyzv2zVbdI7O9e+MM7jR+MiUicoDNlIjIATZTIiIH2EyJiBxgMyUicoDNlIjIgYSmRjU26tunL16cYRe9re9rtGeqPgUFAIYOldnWQ8mZNpS+ZSMKereTBy66yC4y5k407dhllnRd+rLIWleXxx2fC5XnXYK3psupLo8Y058AIG+znMoFAJvKBpg12r5ByVReDixdKvOgfatuuF7fN2jaNLumuPhSke0/nh1ndG5UVurr0zQtlu+vZh8UDtcPdOxo1tx3Z6XIyhac2X7yLnTrUIVlk+U+XNr+V81yIMcMAIdO5Zg1558vsy+/jD8+gJ9MiYicYDMlInKAzZSIyAE2UyIiB9hMiYgcCPm+f+b/OBQ6AmB3yw0nUBff99u39El4jUnxXbhOXqND34brTKiZEhGRjn/mExE5wGZKROQAmykRkQNspkREDiT0bH4kLc330tNF3tCzt1mz27j/FonY5zlwQGa1tVHU18dC8cZ4tiIZGb6XrTxXfd55Zs2JE3reBsYBAGhoEFH0yBHEKitb/hrT0nyvVSt5oE0bu+jUKTXe3dretsQSi62PJeMucGZmxM/J8UTe1GTXxGLWQXvbnNxcuTZFdXUUtbUt/35t1y7iFxR4Is88aK8LgRz92fTacL5Z0nr7FpFF6+sRa2ho8WsEgFAo4gNFIr+k8IhdlGJ8VszNtWu038v9+xErL497nQk1Uy89HaWeJ/Jj7yj7w5w2ZkxiOQBMny6zTZuSs6iCl52N0sGD5YH5882ad1bpL9q1eMc+USwmopIHHog3PCe8Vq1Q2rOnPHDVVXbR1q1qPNbT9xMK8uyzoaRMccnJ8XDbbfK9qS3W0+zZZ619jcrMmu9/X/4s//rX5LxfCwo8LFokr7HPzFvtImNDr51X3m6WdB18gchKotG443OnCMB7Ii299/d2ibYBGAAMGWLXaL+Xt9wSZ2xf4Z/5REQOsJkSETnAZkpE5EBC35lWdemND/4gv5/JPWjXLJtzWD8QcAfqH7pvF1nJsJq443OishJYuVLm48ebJddWVOgHrI27ATy9We5Rfrj+sbjDc2FDzYXI3Cpfx5qNn5g1F110sZoPLLbPM2vqMTV/9tng8bnSqRMwdarM6+vtmrk/XqUfePJJu2jqEyIq2VodPDhHTpwAVq2S+ezwi2bNs3dYi5CvDTjTPCX7acC/d6tLlxQ89JByY3iz/V32Q+FZan5nwHlSOxSIrD4tM97wAPCTKRGRE2ymREQOsJkSETnAZkpE5ACbKRGRA2ymREQOJDQ1au9eYOJEma9buM0u2mrMm+re3a5Zvlxm1vQjx/bmXYxJyiOII0faNStW6PnQgCFv3iwz4/F35/phA0qhTPdYoWwy3+z9l/T8/vvtmqXK65hEaXt3IW+i8ohkmT2dRn1hAOCgPf+vMjVVZI3xBufIOa2PY0K310V++WLlkejT/Dc+0g8EPbMels/zlwxPqH2clcjxHRi9Ypg80KuXWfPwSL0vzZhvrycxpeI+kaUfCni/fAM/mRIROcBmSkTkAJspEZEDbKZERA6wmRIROZDQ7bj6ev1G6LLN9t2xSEQ/NmDmBLNmbM3TIttd+3L8ATrQoYM+Y6FozhSzJloyQ81L7TWzMXeyvNO44YPkLOZSfl4/vDRDDu7vL7Nr8t59Vz+wcaNZ81n/0caRJC2QUV0NfKTcub7rLrNkSn99Qe8ZDfZK+/N/I7dLP/pEchaHRiwG/OEPIv5g1d+bJZU1g9Q84KXEgBJlB4IMucNAi8nNBYYOFXHoDnvh5t97clcQAJiSZi8oNGijPLatOmCR92/gJ1MiIgfYTImIHGAzJSJygM2UiMgBNlMiIgfYTImIHEhoalR+PjBqlMyLA/YB6nGn3OsIALBd7vPUbG6p3Lhnw7qGOKNzo1VTDYpq5LSlumn69CcAeGqgni9ebJ9nwkw5ZWzvkeRMNSkrAx54QOYjRtjLc8ycqU8nuf/q982a9+1DydG2LfCjH4n41rX2tLwXFypTgAAUea3Mmj2PviCyBWn6/leuHc3thgVDl4l8oNz+/WsFFZ+p+YATu8w/rkfWAAAApUlEQVSa++6XC6cErRfj2v7afDy0XS5aE7D+jNqrAOBjTy5m0qx/f5lZa9/8LX4yJSJygM2UiMgBNlMiIgfYTImIHGAzJSJyIOT7cpEG8x+HQkcA7G654QTq4vt++5Y+Ca8xKb4L18lrdOjbcJ0JNVMiItLxz3wiIgfYTImIHGAzJSJygM2UiMgBNlMiIgfYTImIHGAzJSJygM2UiMgBNlMiIgf+H0uin4eRebNrAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7faa8529c240>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_conv_weights(weights=weights_conv2, input_channel=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Helper-function for plotting the output of a convolutional layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_conv_output(values):\n",
    "    # Number of filters used in the conv. layer.\n",
    "    num_filters = values.shape[3]\n",
    "\n",
    "    # Number of grids to plot.\n",
    "    # Rounded-up, square-root of the number of filters.\n",
    "    num_grids = math.ceil(math.sqrt(num_filters))\n",
    "    \n",
    "    # Create figure with a grid of sub-plots.\n",
    "    fig, axes = plt.subplots(num_grids, num_grids)\n",
    "\n",
    "    # Plot the output images of all the filters.\n",
    "    for i, ax in enumerate(axes.flat):\n",
    "        # Only plot the images for valid filters.\n",
    "        if i<num_filters:\n",
    "            # Get the output image of using the i'th filter.\n",
    "            img = values[0, :, :, i]\n",
    "\n",
    "            # Plot image.\n",
    "            ax.imshow(img, interpolation='nearest', cmap='binary')\n",
    "        \n",
    "        # Remove ticks from the plot.\n",
    "        ax.set_xticks([])\n",
    "        ax.set_yticks([])\n",
    "    \n",
    "    # Ensure the plot is shown correctly with multiple plots\n",
    "    # in a single Notebook cell.\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Input Image\n",
    "\n",
    "Helper-function for plotting a single image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_image(image):\n",
    "    plt.imshow(image.reshape(img_shape),\n",
    "               interpolation='nearest',\n",
    "               cmap='binary')\n",
    "\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot an image from the test-set which will be used as an example below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAADV5JREFUeJzt3X+oXPWZx/HPUzeNYKrmNtMYbextc0UJwabLEFYra1dtuAmB6D+SICUFaQoqrlB0xaKr+E9YbYqgVG80NC6tbTGVBAmubqhooJaMJv6Ku+uvG5twzZ0YoSkIadJn/5iTcqv3fGecc2bO3DzvF1xm5jznzHlyyOeemfmeO19zdwGI5wtVNwCgGoQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQ/9DPnc2bN8+Hh4f7uUsglPHxcR0+fNg6WbdQ+M1sVNIDkk6T9Ki7b0itPzw8rEajUWSXABLq9XrH63b9st/MTpP0kKQVkhZLWmtmi7t9PgD9VeQ9/zJJ77j7e+5+TNKvJK0upy0AvVYk/OdJ+uOUxweyZX/HzNabWcPMGs1ms8DuAJSp55/2u/uYu9fdvV6r1Xq9OwAdKhL+g5IWTnn81WwZgBmgSPh3S7rAzL5uZl+UtEbS9nLaAtBrXQ/1uftxM7tJ0n+pNdS32d3fLK0zAD1VaJzf3XdI2lFSLwD6iMt7gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIQfCKrQLL1mNi7pqKQTko67e72MpgD0XqHwZ/7F3Q+X8DwA+oiX/UBQRcPvkp41s5fNbH0ZDQHoj6Iv+y9z94Nm9hVJz5nZ/7j7C1NXyH4prJek888/v+DuAJSl0Jnf3Q9mt5OSnpK0bJp1xty97u71Wq1WZHcAStR1+M3sDDP70sn7kpZLeqOsxgD0VpGX/fMlPWVmJ5/nl+7+TCldAei5rsPv7u9J+maJvQDoI4b6gKAIPxAU4QeCIvxAUIQfCIrwA0GV8Vd9ITz55JO5tU2bNiW3Pffcc5P1008/PVm/7rrrkvVzzjkntzYyMpLcFnFx5geCIvxAUIQfCIrwA0ERfiAowg8ERfiBoBjn79Ctt96aWxsfH+/pvh9++OFk/cwzz8ytLV68uOx2ZoyFCxfm1m677bbktvX6qf8t9Jz5gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiAoxvk79Oijj+bWXn311eS27cba9+3bl6zv2bMnWX/++edzay+99FJy23ZTqH3wwQfJehGzZs1K1ufNm5esT0xMJOupf3vqGgCJcX4ApzDCDwRF+IGgCD8QFOEHgiL8QFCEHwiq7Ti/mW2WtErSpLsvyZYNSfq1pGFJ45KudfePe9dm9a688squap0YHR0ttP3HH+cf+nbXCLQbz969e3dXPXVi9uzZyfqFF16YrF900UXJ+pEjR3JrixYtSm4bQSdn/p9L+vT/ztsl7XT3CyTtzB4DmEHaht/dX5D06V+hqyVtye5vkXR1yX0B6LFu3/PPd/eT11Z+KGl+Sf0A6JPCH/i5u0vyvLqZrTezhpk1ms1m0d0BKEm34T9kZgskKbudzFvR3cfcve7u9Vqt1uXuAJSt2/Bvl7Quu79O0rZy2gHQL23Db2ZPSPq9pAvN7ICZXS9pg6Tvmtnbkq7KHgOYQdqO87v72pxSscFtlGbu3Lm5tSuuuKLQcxe9hqGIrVu3Juup6xsk6eKLL86trVmzpqueTiVc4QcERfiBoAg/EBThB4Ii/EBQhB8Iiq/uRmUmJ3MvDJUk3XDDDcl668ryfHfddVdubWhoKLltBJz5gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiAoxvlRmYceeihZb3cdwNlnn52st/vq7+g48wNBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIzzo6d27dqVW9uwodh0D9u2peeKWbJkSaHnP9Vx5geCIvxAUIQfCIrwA0ERfiAowg8ERfiBoNqO85vZZkmrJE26+5Js2d2SfiCpma12h7vv6FWTmLl27Mj/b3Hs2LHktldddVWyfskll3TVE1o6OfP/XNLoNMt/6u5Lsx+CD8wwbcPv7i9IOtKHXgD0UZH3/DeZ2WtmttnM5pbWEYC+6Db8P5O0SNJSSROSfpK3opmtN7OGmTWazWbeagD6rKvwu/shdz/h7n+VtEnSssS6Y+5ed/d6rVbrtk8AJesq/Ga2YMrDayS9UU47APqlk6G+JyR9R9I8Mzsg6d8lfcfMlkpySeOSftjDHgH0QNvwu/vaaRY/1oNeMAN98sknyfozzzyTW5s9e3Zy23vuuSdZnzVrVrKONK7wA4Ii/EBQhB8IivADQRF+ICjCDwTFV3ejkPvuuy9Z37NnT25txYoVyW0vvfTSrnpCZzjzA0ERfiAowg8ERfiBoAg/EBThB4Ii/EBQjPMj6emnn07W77333mT9rLPOyq3deeedXfWEcnDmB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgGOcP7qOPPkrWb7755mT9+PHjyfrKlStza0yxXS3O/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QVNtxfjNbKOlxSfMluaQxd3/AzIYk/VrSsKRxSde6+8e9axXdOHHiRLI+OjqarL///vvJ+sjISLLe7u/9UZ1OzvzHJf3I3RdL+idJN5rZYkm3S9rp7hdI2pk9BjBDtA2/u0+4+yvZ/aOS3pJ0nqTVkrZkq22RdHWvmgRQvs/1nt/MhiV9S9IfJM1394ms9KFabwsAzBAdh9/M5kjaKukWd//T1Jq7u1qfB0y33Xoza5hZo9lsFmoWQHk6Cr+ZzVIr+L9w999miw+Z2YKsvkDS5HTbuvuYu9fdvV6r1croGUAJ2obfzEzSY5LecveNU0rbJa3L7q+TtK389gD0Sid/0vttSd+T9LqZ7c2W3SFpg6TfmNn1kvZLurY3LaKId999N1lvNBqFnn/jxo3J+qJFiwo9P3qnbfjdfZckyylfWW47APqFK/yAoAg/EBThB4Ii/EBQhB8IivADQfHV3aeA/fv359aWL19e6Lnvv//+ZH3VqlWFnh/V4cwPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0Exzn8KeOSRR3JrqWsAOnH55Zcn663vesFMxJkfCIrwA0ERfiAowg8ERfiBoAg/EBThB4JinH8GePHFF5P1Bx98sE+d4FTCmR8IivADQRF+ICjCDwRF+IGgCD8QFOEHgmo7zm9mCyU9Lmm+JJc05u4PmNndkn4gqZmteoe77+hVo5Ht2rUrWT969GjXzz0yMpKsz5kzp+vnxmDr5CKf45J+5O6vmNmXJL1sZs9ltZ+6e3pWBwADqW343X1C0kR2/6iZvSXpvF43BqC3Ptd7fjMblvQtSX/IFt1kZq+Z2WYzm5uzzXoza5hZo9lsTrcKgAp0HH4zmyNpq6Rb3P1Pkn4maZGkpWq9MvjJdNu5+5i71929XqvVSmgZQBk6Cr+ZzVIr+L9w999KkrsfcvcT7v5XSZskLetdmwDK1jb81vp61sckveXuG6csXzBltWskvVF+ewB6pZNP+78t6XuSXjezvdmyOyStNbOlag3/jUv6YU86RCFLly5N1nfu3JmsDw0NldkOBkgnn/bvkjTdl7Mzpg/MYFzhBwRF+IGgCD8QFOEHgiL8QFCEHwjK3L1vO6vX695oNPq2PyCaer2uRqPR0bzpnPmBoAg/EBThB4Ii/EBQhB8IivADQRF+IKi+jvObWVPS/imL5kk63LcGPp9B7W1Q+5LorVtl9vY1d+/o+/L6Gv7P7Nys4e71yhpIGNTeBrUvid66VVVvvOwHgiL8QFBVh3+s4v2nDGpvg9qXRG/dqqS3St/zA6hO1Wd+ABWpJPxmNmpm/2tm75jZ7VX0kMfMxs3sdTPba2aV/v1xNg3apJm9MWXZkJk9Z2ZvZ7fTTpNWUW93m9nB7NjtNbOVFfW20Mx+Z2b7zOxNM/vXbHmlxy7RVyXHre8v+83sNEn/J+m7kg5I2i1prbvv62sjOcxsXFLd3SsfEzazf5b0Z0mPu/uSbNl/SDri7huyX5xz3f3fBqS3uyX9ueqZm7MJZRZMnVla0tWSvq8Kj12ir2tVwXGr4sy/TNI77v6eux+T9CtJqyvoY+C5+wuSjnxq8WpJW7L7W9T6z9N3Ob0NBHefcPdXsvtHJZ2cWbrSY5foqxJVhP88SX+c8viABmvKb5f0rJm9bGbrq25mGvOzadMl6UNJ86tsZhptZ27up0/NLD0wx66bGa/Lxgd+n3WZu/+jpBWSbsxe3g4kb71nG6Thmo5mbu6XaWaW/psqj123M16XrYrwH5S0cMrjr2bLBoK7H8xuJyU9pcGbffjQyUlSs9vJivv5m0GauXm6maU1AMdukGa8riL8uyVdYGZfN7MvSlojaXsFfXyGmZ2RfRAjMztD0nIN3uzD2yWty+6vk7Stwl7+zqDM3Jw3s7QqPnYDN+O1u/f9R9JKtT7xf1fSj6voIaevb0h6Nft5s+reJD2h1svAv6j12cj1kr4saaektyX9t6ShAertPyW9Luk1tYK2oKLeLlPrJf1rkvZmPyurPnaJvio5blzhBwTFB35AUIQfCIrwA0ERfiAowg8ERfiBoAg/EBThB4L6f6yMEem39pFEAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7faa84ab9208>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "image1 = data.x_test[0]\n",
    "plot_image(image1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Output of Convolutional Layer - Method 1\n",
    "\n",
    "There are different ways of getting the output of a layer in a Keras model. This method uses a so-called K-function which turns a part of the Keras model into a function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.python.keras import backend as K"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "output_conv1 = K.function(inputs=[layer_input.input],\n",
    "                          outputs=[layer_conv1.output])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then call this function with the input image. Note that the image is wrapped in two lists because the function expects an array of that dimensionality. Likewise, the function returns an array with one more dimensionality than we want so we just take the first element."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 28, 28, 16)"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "layer_output1 = output_conv1([[image1]])[0]\n",
    "layer_output1.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then plot the output of all 16 channels of the convolutional layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUoAAADuCAYAAABf005JAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJztnWtwlOd1x3+SkRAIJIEEMkJCAiGwERdjMNiOseMY6vjSxM0kvrvudKbth3bcyQfPxJ1kps200za9jNNMM5m0STppLiSNkzh1L2AwBlObi4Ux96uQABmQhO5CV6R+eOc870pa7bsSu9rV6v/7IlitVq+effa8/3Oec0kbHBxECCHE6KQn+gKEECLZkaEUQogAZCiFECIAGUohhAhAhlIIIQKQoRRCiABkKIUQIgAZSiGECECGUgghApg2licXFBQMlpaWxuta4kptbS2NjY1pib6O0dDaxo/JvLYAhw4dahwcHJyX6OsYjYKCgsGysrJEX8a4qKmpiWrvjslQlpaWsn///vFfVQLZuHFjoi8hIlrb+DGZ1xYgIyOjNtHXEImysjI+/PDDRF/GuFi/fn1Uz5PrLYQQAchQCiFEADKUQggRgAylEEIEIEMphBAByFAKIUQAY0oPivhC0279pfr7+2NwJalHS0sLAHV1ddy8eROA9HTvHrd8+XJmzJgR+Bpa28j09/eTlZU17p8VwTQ2NgJgUxWys7PJzMwEYmM/4klSXZ2MbXgOHz4MwL59+1iwYAEAd911F0BURhLgtttuG/V7aWnR5Yqn4tranruVvad9G0x7eztXrlwBYPr06QBcvHjRfd/WcM6cOcDQfW2iIByho2zsZyPt9fEi11sIIQJIKkUpwtPe3g5ASUkJL7/88rheI1rVKEQ86O3tderv8uXLAGRmZpKXlwfAjRs3AN897+vrc8rT9v+0adPcawxX8T09PcyePRuAiooKwHPtY4UUpRBCBBAzRXnp0iXAuyNY7OHMmTPuexkZGQDccccdACxatAiA8vJyli5dGqvLSEk+85nPAJCbm5vgK0k9+vr6AC9GaP/OyclJ5CWlFB0dHQCcPn2atrY2AI4fPw54B5EDAwMAzJ07F/CUIXhq01SjeUOhXpGpzaamJgCuXbvm6rbte7EkZoays7MTgCtXrgw5fADYu3evW7Dbb78dwJ3ehp58hQZwbVHWrFkD+IcX+fn5zthWVlYC0NraOiIYnErIQMYPyyj46KOPOHfuHADbtm0DPPfODhJsry1cuBCA++67j+LiYsA3rAMDA0l/ejvRdHd3A55B+7//+z/Ad6WPHj3KvHleUyRbt6KiImCo22z7v7293dmIrq4uwBdo2dnZ1NfXA16Tjlgj11sIIQKI2e0vPz8fgPr6eqcaTQ2mpaVx/fp1wA/aWjpEY2OjU5v22LRp0ygsLAQ8SQ2wa9cuABYsWODUZlVVFeDJ9A0bNgCpqSjDYS7Km2++6dbb3JdZs2YBkJWV5ZS7KSGAd999F4Af//jHADz//PMAPPzwwyN+T0NDg/td9ntSiZqaGsDLUT1w4ADg77mDBw86b+fXv/41MDSNx8JJpn76+vrc4YS5f9aC7uWXX+bRRx8d8ftD01tSEVPk+fn5LF68GPBVYE5ODg0NDYDvkVq4bvr06U6NWn5rd3e3O7A5ffo0gGuf95nPfMalH9n7YfYnJn9HzF5JCCFSlJgpSovTVFZWutiNKZH29nYXP7A7SHNzM+DdvS1mYcoyNzeXlStXAn4s05RoVlaWuwvbodGCBQvc3T0VsYB3enq6i82YQr/nnntobW0F/FiOqaSmpiZ3l7U7e01NDX/9138NeMF0gH/5l38Z9Xf/+te/dvG5VFSUS5YsATwVbn+n7dVr1665uOXVq1cBP7XlypUrTnmGJkTbPrTYm3laFRUV7vNge7++vt593zyoVMO8nDVr1ri0Hfucd3d3uwOeuro6wI8Zt7a2usRx29/Tp093hzdnz54FfA/yxIkTfOpTnwIiJ6iPFylKIYQIIGaK0qx/bm5u2FNaU4EW47H4w82bN11ahj1msYxQ7E5eV1fH3r17AT+1o7i42N2ZUxFTlAMDA06xhKq7kpISwD85tHhkU1OTi+mcOnUK8Nb46aefBuBnP/vZqL9z69at7nem8traXh0ts8D2qykfi483NTU5b8dUTkFBgYuPFRQUAOFLTG0vZ2RkpLQnFMrMmTOZOXMm4K9NOGyv9/T0jCjr7OzsdBk1n3zyCeCN+QDvfbHyXvMSYsmE5TLYBrONEW6DhEutsCCvSfLq6mrnWtoHuKysTLlvjKwXtoMF8N2Rq1ev8tprrwW+lrma5eXlzhBPRWxPmgtphB4aRpsSZB9uMwZZWVlK/RqG7dPRehhYKMnW0MIfn/3sZ7n77ruB2FbkuOuK+SsKIUSKkRTZsZHqkI8dOwb47s3p06edm79ixQrAcz3jEcBNBezOa8HvwsJCl7wfjtdffx3wwxqVlZUuRUYEE3rwNvzxI0eOAH761rx58+LS6SZVOX/+PAcPHgT8UIgll+fm5kY9UXE8yLoIIUQASaEow91Vh8ciQtMy7LDHShnjEZNIFSzWWFvrjYbesmXLqM9tampyseA777wTGBrnFEMJ58UMf6y3txeAjz/+2MUybb/GoyY5FbE9+cEHH7h/27rawc1dd90VV3WecEM5WiDcThTtj7e8wfz8fFavXg34izRVTg7HSltbG7t37wb8jWX5qeH493//d3dy+NBDDwFqEBGJaMI9dsDW09PjDoRs38pQRseJEycAOHLkiFtP26eWs20VUPFCrrcQQgSQMEUZ1EjWDhDMBbfs/KKiIpYtWwYw7hknqY7ddQ8dOuSC33/4h3846vOtMqq/v9+pneHpMGIo0aYEhTaptZxWyycUkbHP/qFDh9xjVt107733Dvkab6QohRAigIQpykiB18HBQS5cuAB4PevAj0NWVFS4+IQYilU/WRXOzp07WbVqFRA5NvmjH/0I8FIsrOOK0q3CE+2Bgb0XltaWlZXl6rk1liOY5uZm183Jau+vX7/umnwPP9CNN/o0CCFEABOuKCPdkS22Vltby/bt2wE/xmNKp6KiQqeFo2DxGzvpHhwc5A/+4A9Gff77778P+GWia9euVWxyFMKNI4iE9Uq19LbZs2crNjkG6urq+OCDD9y/wSuCMEVpmS8TxYQZSnPlIm00m6Wxfft253Jb0bvl9aViq69bxVJ/rGGAtQYLqun++c9/DvhjNsxNFyOJ1uW298BCR9ZazNquichUV1cD8Pbbb7vHLAf44Ycfdn0HysvLJ/S65HoLIUQAE64ow2FdVayp76lTp1zqj3UEsTuI3O6RmGuyc+dOANatWwf4g5qG8+qrrwJ+sq65M3K7RzKWYWFXr151LrelWVl3ILndkTGvyEY8HDt2zPUnsHruoqIiHnzwQWBkQUq8kaIUQogAJkRRBt2VrTW8tcgvLi52asfiZvPnz4/jFU5eOjo63Fhg66jywgsvjPr8mpoaF+e1FAvFJmPD5cuXXUzdmtOm6oiHWGNK0g5w6uvrXVrQb//2bwOwadMmV4gy0V2XEl7rDX7zTcs9W7FihfsQL1q0CBibCzSVuHz5sju8+fznPw9Ern2fPXu2m5VjmQRqKhKese65devWcf78ecBvKq0WdZEJnfENfjvAY8eOuamgFnYrLy9PWOhNrrcQQgSQMJlm1SPV1dUuJcCUZVFREffddx+gIHgQTU1NbkaOBbojceHCBVcrrwqn6Bk+6TIcaWlpLqyhPgTRYfXcH3/8MYCrxsnLy3MHuXYomciDXClKIYQIYEIUZU9Pj1OLduf47ne/C3jt3W1m99q1awHvbqI0oOhYvHhxVEryrbfeAryY5lNPPRXvy0oZTEla2o95QiUlJe6gxmLrubm5LqYugunr63ONpU2BW4elnJwct65WbJJIpCiFECKACVGUdXV1rq74nXfeAfwyu+rqaqceLWa2fPlyKcooCRol+7WvfQ3AjfiNNFhMDKWhoYF3330X8NNWbB/n5ua62FllZSXg9Ua0eLEIpqury3maVnRipZ6rVq1ymS8WFx4cHExY56UJMZTZ2dnuA20fVMszu3TpklsccyHvuOMOGcoY8PWvf52tW7cC8NJLLwGwbNkyrW2U9PT0uPQeSxWyxiMXL1507qKlr+Tk5Kg93RhIT0934QszijahsqKiwh2M2YFPIg8f9a4KIUQAaRYwjerJaWkNQG38LieulA4ODiZtCxetbfyY5GsLWt94EtXajslQCiHEVESutxBCBCBDKYQQAchQCiFEADKUQggRgAylEEIEIEMphBAByFAKIUQAMpRCCBHAmGq9CwoKBm0i2mSjpqaGxsbGxFTUR0FBQcGgzVuZbNTW1ib92k7WfQtQVVXVmMyVOVNh747JUJaVlfHhhx+O/6oSyPr16xN9CREpLS11HWomG9aNPlmZzPsWIC0tLanLA0tLS9m/f3+iL2NcbNy4MarnyfUWQogAZCiFECIAGUohhAhAhlIIIQKQoRRCiABkKIUQIoAJmZkjbo22tjYAjh07xsmTJwE4e/Ys4A3AWr58OeClwQCsW7cOgPz8fHJycgA0y2UMdHR0AN5QPFu3mTNnAjBnzhzAG6+qNQ3Gxv2ePXuW2lovy6mhoQHwRtPanrWhbDYnJzMz080pGhgYmMhLDktKGMr+/n63qKmIdaFfsmQJn/70pwHcLPT09HSXI2gTAt944w0ALly4QH9/P+DNUB6NjIwMAIqLi12+6apVqwDYsGGDG+qU6obhzJkzAJw4cQLw1mzGjBmAP3faJgIWFRU54xlp7/X29gLelMGuri73s+AZh0RNFZwo6uvrAW9Nt23bBvgTLbu6utzamcEMXRtb+0j7LnTv3nHHHYD3OQEoLCzktttuA27d2Kb2zhdCiBgQFxlmKua9994DPGs+fCxldnb2uF/f1FR7ezsAM2bMcHd8Gy+aStg431BMzQA88MADQ77GktbWVnfXvnnzZsxfP1l477333Azv6upqwFPkeXl5AC6EYSonJyfHqUvby729vU6520hg24/Nzc1u/SxUsnHjRpYuXQqQssrSVN7s2bOZP38+APfccw8AVVVVnD59GoDDhw8DsG/fPvezpjbta39/v9uLhnlbeXl5bNq0CYCnnnoKgPvvv5/i4uKY/B1SlEIIEUDMFKXFANLT07l27RrgNaIA6Ozs5MiRI4B/p+3s7AS8O7Tdte3ukJaW5uI5BQUFAKxYsQLwYkU2EN0CxZWVlW5wuogtppogNRXlJ5984r5aPMv2XHp6OtevXwf8A55Dhw4B3v61x8yD6u7upru7G4C7774b8Nfv4MGD9PT0APB7v/d7gKeC7PumtlINU8rFxcVO3T355JPu+7YmFy9eBPz4cFNTk/MYbX17enqc53j58mUAd7hZV1fnXsM8gpUrVzo7Yp7AeImZoQwNuNoJlkns7du3uz+2ubkZgLlz5wJeQNf+eHNvWltb3b/tFMxc95MnT7qgsJGTkxMziZ2MVFVVAbB//35aWloA2LNnD+AFwe+//34Afud3fgfw3BwRHXZzNncN/Jt5OOzDeObMGfe+2I17+vTpLiSydu1aAP77v/8b8Pbthg0bAFi9ejXgGUo77Jmq2FpXVFQM+RqEvQ92kHngwAHnlltoIz093YVCbtVQyvUWQogA4ppTE3p3NTfFpLYdyLS1tTm1GSq1LUBuatNcpF/96lf83d/9HQAPP/ww4AVtLdfQlGgqYer7nnvuYceOHYC/Lvv27WPnzp0AvPLKK4Afkli+fLn797Jly4ChaSpPPPEE4Kupxx57LO5/S7Jh3k+0LFq0yH3dvHlz4PM/97nPAfDlL3/ZKR1x64S+D/bV3HzrjTl79uyY5WBKUQohRABxVZQlJSWAVyFi6tJiQufPnwc8hWlK0g548vPzXUxh5cqVAO6A6MyZM05hWdC9p6fHKdZUVJSW8L1w4UJ3B/2t3/otwIu92N9uCdNW+XDt2jUXEzaFPnfuXNasWQPgFJG9JyJ+SE3GB4vZ5+TkuPOMcDbAvNbxIkUphBABxFVR2ilUaIqJceedd7p/W9wy0mmjJZ3euHHDnRra16ysrLC/I1UIzSiwWmP7Cr4itPUwrl275k7Am5qaALh+/XrU7e+NVEwLijdf/epXAT9V5U/+5E9cdsJUIkjJjaX0+MKFC85rMs/K9uaMGTPimrSfFAXSkQyk8R//8R/uuZYKZPWhy5Ytu6VKn8mObZDhG8VcdvBvWlYJMhZkKKPnm9/8JgDvvPMOAA899BDgp7eJoUTjElu9eG1trUv3aWxsBLwwHXh5muHsSKzCSnK9hRAigKRQlJGwVJbt27cDnrqxzjZ2l7a7ihiJpUeMNwldBz3Rc+3aNT7++GPAPzyzFKzy8vKEXddkxfaeVTtNmzbNHYpZuM7WebR9GitvSIpSCCECSHpF+eMf/xjwa22XLFni0oMsoBvaSUcMxdIn7PAnUjx4YGDAHRzZHfpW0yqmEm+//TZ1dXUAPPfcc0B8OjpNFULT3MA7sLFCFauND43Dx5OkNZQmrX/xi18A/unYypUrXT2ouTOp3lB2vHR0dLgcVcsxi0ToOpqBlOsdjH2gd+zY4cJB0VTtiNFpb293udbWmKSkpMQ1LrH9GdqMZzixvMnLwgghRABJqyj/6Z/+CfDvFFahU1RU5NKCUnn8w61gAeyOjo6ou7EMR0oyev75n/8Z8DoEvfjii4A6OI0X27uXLl1yrdRMRba0tDjFbm0VwynJeKSzSVEKIUQASSnJPvroI/bu3Qv4sTVTRkVFRUreHQVTgXagMJ4k/EhDyMRQfvjDHwLws5/9DPAKH2z0QWjllIgeq2Q6d+6cO8SxgWNz5sxxXbMiKfZ4eENSlEIIEUBSKUq7E/zgBz9wp1l2N7EBW6WlpS5mIYZiatCyAcZT+6rYZHTU1NSwdevWIY9t2bKFRx55JEFXNLmxTu9WYNLa2uoKSSwFaO7cua7XZLi9Hc9UtqQylFYnW11d7Vqo2YxeW6xUnS1yK5iBtPb4Nl9ovK8jgvmLv/gLN6/l2WefBeDpp59WqtoYsT136tQpwJ+zVVhY6GyATW4NN410otC7KoQQASSForTWSUePHgW8JFKr57aguLmTcrtHYu5KtPXEV69eBUYmocvtDubNN98EvCmA1tbumWeeAeTtjAdr/2fdgKxTUF5envMio1GS8a4gk6IUQogAkkJR/vSnPwX8UqUlS5a4Wd+VlZWAOgSNxo0bN9zBVzR9Pa9evTpCSU71kanRYMPrLG2tp6eHTZs2AeoMNF46OztdKpulAlnfho6ODlcaWltbC/hDwxJBwg3l6dOnXVDcNmNhYaFzvS2QG8/uxZOZjIwMd1OJhlAjqcOb6PnlL38J+Ddz8Dv3qwpnfHR1dbmeDqHVZOC50mYPzC2//fbbR4iBiWraItdbCCECSLii/Pa3v82uXbsAv21aZWWlm6ljIwxEeG5lBIYOb4IxBWlK5/jx44B3wPD4448n7LpSgenTp7sDMFPlpiyt6xX4VXk9PT1OUU70eBIpSiGECCBhitIOcHbv3u1UkTU5nT9/PpmZmYm6NCEAaG5uZufOnYDfANmSy//4j/9Ye/QWmTNnzphr4s0LmmhvSIpSCCECSJiitP6SS5YscYrS4pL9/f2Kn0VJX1+f67hi6RRWg7xixQqXurJ27VrAU+uWTiQiMzg46GZxWyrQRI0eEOFJVPZLwgxlZ2cnAPfdd5+bsmZVN4sXL44qJ1B4NbJvv/02AB9++CHgu4ldXV2u0sFuQr29vUq1ipKcnBw3n0lMbeR6CyFEAGljcXHT0tIagNr4XU5cKR0cHExc+5EAtLbxY5KvLWh940lUazsmQymEEFMRud5CCBGADKUQQgQgQymEEAHIUAohRAAylEIIEYAMpRBCBCBDKYQQAYyphLGgoGCwrKwsTpcSX2pqamhsbEza2j2tbfyYzGsLUFVV1ZjMCeeTeX2j3btjMpRlZWWunniysX79+kRfQkTKysrYv39/oi9jXGzcuDHRlxCRybxvAdLS0pK66mUyr2+0dkGutxBCBCBDKYQQAchQCiFEADKUQggRgAylEEIEIEMphBAByFAKIUQAEz4zxxoF25yc6dOnu8dsZo4YSizWZaIHxk82uru7aW1tBSA93dMP8+YlbY73pKO3t3fEwMDJNBcr5oby5s2btLe3A5CVlQXAuXPn3KbLyMgAiMnQJpsmeOPGDbfo9vpiKDK2kcnKyqKrqwtg3LOmwzHVB7k1NzcDUF1dTX9/P+DbhTVr1iTsusaKXG8hhAggZoqyqakJgP/8z/90pXgfffQRAKWlpZSUlAD+fOmcnBwA8vLy3Gv09PQAkJmZSW9vr3eB07xLtLv2zZs33R3fVGl2drZTl6moKK9cuQJ4dak2djZ03URssH1le7mlpcV5R6YMly5dCsDMmTPdz0111RiJ69evA3DkyBH3mZ81axYgRSmEEClFzBTliRMnANi9ezdvvfUW4McnDh8+zIIFCwD4zW9+A/gxsxkzZnDjxg3AV5TTpk1zhz2LFi0CcAqzoKCAZ599FoDVq1cDXqwyNzc3Vn9K0mFrNzAwQEdHB+ArysbGRrKzswFfCS1cuDABVzn5qaqqAmDHjh0AfPDBB86TMe/F1FB5eTnFxcWAry4HBgZcrNz2sr0nly9fdnv50UcfBabGYZGp7e7ubq5evQpAYWEh4J1dmEJPdmJmKO2D+8ILL/Diiy8O+V5ubi7Hjh0DYO/evQAcP34c8Nwb20zmPnd0dFBQUAD4G+3kyZMALF++3HUqsd/5wAMPkJmZGas/JWnp7+/nwoULgP9hnj59Oi0tLQDuZnH77bcDsGnTJvezqRiSiCU1NTXuZt/X1wd4H25zx9va2gCorfUa+ezfv9+55ba2fX197r2w98Bcz7a2NjZs2AD4N/3y8nLuuusugJS90ZsgmjVrFnV1dYC/d48dO+YEkWUarFq1CvBEkN2AzBY0NzdTVFQEeEYWfIH25JNPuufFA7neQggRQMwU5cqVK4d8Hc66desAePnll8f1+j/5yU8A785cXl4O4O7GsUg1SmYOHz4MwL59+5yyPn/+PADvvvuuu5NasNwUy9y5c51bbodimZmZzj001WPPf+ihh3juuedG/P7QkEiqYerx6tWrziW08MYXv/hFp3gMCyd1dHTQ2NgIQGdnJ+AdOFouZuhhD0BxcTHLly8H4OzZs4CnIi0lKVUVpe3JTZs2UVpaCngHOwANDQ3OixyuEN98803nMYaGMywUYp7p5cuXAXj66af53ve+B/jhkVgiRSmEEAFMGolgaUXTpk2joqIiwVczsVhqFfiK3WJhn/70p53KtMdqamoAL63I/m135bS0NHcXX7x4MeB3KA+XlL5r1y73++35qYTFFysrK11C/d133w148XFT2w0NDYAfq7x586ZT2qZKe3t7nZoxZWmpcq2trc7zMWWek5PjVH2qYn/znDlznKKsrKwEvEIRW1/bu6bYr1696tS2rde1a9c4ffo04Kt+W7/FixfHRUkaUpRCCBFA0ipKU0Lf+MY3AL82/Pd///ennKK0WE24JPPHHnvMlYbZ3dgUY1dXl3vM7s75+fnk5+cDI1XS/fff717X4kiDg4NT4sR89uzZIx4Ld4pqa5uenu5UkJXk9fX1OfVz5swZwI+h5efnc+nSJcBX8MuWLYvln5DUhCbl2/6zr+Ho7e11Cj80jml73c4n7JT8b/7mb2J/0SEkpaG8cuUKf/VXfwV4uWwATzzxBBB+Q0917MMZLi/PpuOFc6vt8CIcdlg0b968iM+balgaC4y8cWVlZTl3/NChQ4C/jnl5eS6Fzb6aKypGEpruZzf5+vp6d7OxcNDnPve5Cbkeud5CCBFAUirKbdu2OdfPAr+mKCdTfWgyEK4O2Q4a7BAi1MXctWsX4KvUioqKKeF6x4oDBw4AfvqKpWc1NzfzqU99CoBHHnkkMRc3SbE0rCNHjrh/mx3YvHnzhFyDFKUQQgSQVIqyvr4e8O7KloBrStIC4GJshMbUDDt8GN5HsaOjw5WZzZ8/f8hzRTBtbW0cPHgQwPUvMDW+aNEiF1ezmm8RGTvEsbLRjz/+mBkzZgCRPUtLOYplWXNSGUoLgHd2drpTLcsbnEzdkJOFcAc4fX19zn0Z3jyjqqrKueGWSxjP3LRU4/jx425trdmD1YMvX76cBx54IGHXNhmx9oLWrrGrq8s1wvn85z8/6s/Fo++DXG8hhAggKRSlHfm//fbbgJfft379eoBJ04YpmQjnbhsZGRkjlKTlqzU1NTn3UI2Bo8c6BJ0/f35E7p8p9JKSkoh5g2IoLS0t7kDXclIXLlzoWixONFKUQggRQMIVZW9vL1u3bgVwfSYHBwddjNK6j4joGetoAktpmTVrlqvn1niDYOzQwA4hz58/P6SXJfgekXlIIjKWsH/mzBl3MGbdmbZs2ZKwqjwpSiGECCDhivJXv/qVGx1h3btfffVVJZaPAzvltjhZ0Iha6x5vMeKlS5cqNjkG7FR29+7d7v9WW79kyZIhX8c6AneqYqlA77//vktVs96e4XqlThQJM5TWLmnr1q3OddmyZQsADz/8sFzuMTDcIEY7w/uNN94A/A+zao+jp7m52aWzWRVOU1OTS6e64447AOX/Rou1TTO7cOTIEZfDm6gDnFDkegshRAAJU5S//OUvAa8Zp3Wneemll4DRx0mI2PCjH/0I8DsxWQqL3O7oOXToEO+99x7g13NfvHjReUU2+sS+J8Jjh1+WAmTdwtrb210LtWRI1JeiFEKIACZcUVpS+alTpwAvMffLX/4y4I01EGMj2nik0dDQ4ILjlgBtMUoRjI1d3rlzpzt8tAT/F154wR1CyiuKDpv1bWWKZhdKSkr4oz/6o1F/LtoDy1gxYYbSTrDM5bZh8/Pnz9chwgSSl5fnZiNbU9/hEwPFSCwv0jrvf/DBBy50UVxcDHjrec899yTk+iYj3d3dbt6T5fLaqfd9990XcV9OlIE05HoLIUQAE6IoBwYGXODbjv8/+eQTwGtiqlZe46evr8+18rIzJ9KeAAAMrElEQVQ1tbki4dp59fb2OldbtcfRY3XHO3bsALxONsNHO/T19U240pnMtLa2ujneNuvc0qusS1CyIEUphBABTIiiPHv2LPv27QP8uljrUrNixQqXnCvGTlpaGnv27AHgnXfeAaC8vBzwFKXFz6yR7LVr11R3PA6sBvn48eOAF1OztbVuTBqZMTZ6enqcPbBuS7aWyTbvXIpSCCECmBBFmZ6e7hJw7S587733AsmRTDqZmTZtGnPnzgX80Q5WmlhcXDxivO+DDz6ozkBjwNJVtm/fDvgdgwYHB0fs5dC56CKYjIwMFy+3TvBGfX29mzdv4x8SqdgnxFCWlJS4gwWNdIgtLS0trk7WPrhW5XD8+HEWLFgAwKOPPgp4Bw8ylNFjhwt2o7eUlS984QtuqqLNU9fM+bGRlZXlwm7mel++fBnw0oSsUsxa/82YMSNhlU5yvYUQIoC04ZP4Ij45La0BqI3f5cSV0sHBwXmJvojR0NrGj0m+tqD1jSdRre2YDKUQQkxF5HoLIUQAMpRCCBGADKUQQgQgQymEEAHIUAohRAAylEIIEYAMpRBCBDCmEsaCgoJB64o92aipqaGxsTFpa/cKCgoGJ2un99ra2qRf28m6bwGqqqoakznhfDKvb7R2YUyGsqysjIMHD47/qhJIsrfoLy0t5f3330/0ZYyLZG8GUVZWxocffpjoyxg3aWlpSV31MpnXN9qWg3K9hRAiABlKIYQIQIZSCCECkKEUQogAZCiFECIAGUohhAggZqMgbEpdY2MjnZ2dAG5ed+joAZt/kZ7u2ejs7Gz3b02xC89//dd/AbB7927+8i//Eog8duDSpUuAt+42eiMnJyfOVzk5sTEaZ86ccbO7z58/775vI0xsZIGNJSgoKNA8+lukvr6emzdvDnnMxj9kZmYm1Yz0mBlK2zTz58+nu7sb8AcGvfXWW24spRlDmzMyZ84cZzwjzcMwYzp79mz3s/Zag4ODzhCk4kyer3zlKwD87u/+blgDaYYxPz8fgObmZsC7edljtgHDEe4GdeXKFQCOHj1KZWUl4BmHVOOTTz4BYM+ePWzbtg2A/fv3A948otzcXMDfm/Pnz3f/t/k5kbC1LSoqYvXq1YCf07ts2TL3vkzFOUb5+flujLINbTPbAb6oitRcvK+vD/AEmtkb+7kFCxbETHzJ9RZCiABiPoUxMzOTzMxMwHf38vLyqK6uBvw7p7nqfX19zuqHfrW7iD3f3Pnr16+7u3xhYSHgKR0be1lUVBTrPynhvPTSSwB89atfDfv9kpKSIf835XIrnD17FoDvfe97PPvsswA8/vjjt/y6yYa51l/60pfYvHkzAIcOHQLg3LlzTp1b5UlLSwsA1dXVbo+a+z4wMOD267Rp3kfL/l9YWEhrayvgT3IsLCx0n5FkcjMnittuu22EhzT8cz/838MxW5Odnc3p06cBf+2zs7OdR3WrSFEKIUQAEzLX+5lnnuGZZ54Z8pgFzHt6emhqanL/Bi/OaArSHrt27RoAFy5ccHdmU6Br1qxxcctUZMuWLXF53ZqaGsCr1R2NU6dOcf369bj8/mTAYusLFy5k4cKFAKxcudJ932JnFrMNfdyUS1dXFwA3btxwr9fQ0ADAgQMHADh27NiImFtOTs6UVJKRGG+s9rbbbmPv3r2AH09etGhRzBTlhBjKcJSXl0f8vm1Qk9bGpUuX+Ld/+zfA23zgHfCk8vD5e++9Ny6vG8lA/vznPwfg4sWL3H777XH5/ZMB239j7exUW+v1sTh58iTgHVxcuHAB8G/+UzXLw246oWG3uXPnAv4B8PTp00d8pvv6+kaENIzr169z4sQJAJYvXw4Q06wEud5CCBFAXBWlHd2P5845XEkaJSUlzJo1C4C77roLgOeee87dfTSnPDZ84QtfALy0rEipRSI8pkDtIKyxsZHvf//7gHfoA1N3r9pnu76+3ilDSx+0w7LQgx5Thjdu3HDragrUeOONN9i5cyfgpx62tLS4A99bRYpSCCECiIuitIMYC17Hkl27dvGb3/wGgNdeew2IXKUixse6desAL562YsWKBF/N5MWUz9y5c50HZAcMWVlZTl1aQcVUwNYk3EFLY2Mj4MUcTXFbTHfevHkjlKSxbds2d4Bm6VcDAwMudcu80PEydd4dIYQYJzFTlKHxluGlXYODgyNOqax0KZoysFBee+01lx70wAMPjOdSJx12Rx0NiwGbKunv7wf8xNtQHnvsMbZu3QrgEvfDce7cOcDLTrC7eNB1THUaGxv56KOPAL82PLQYwEpBrThi2rRpUzJOGSkFyMpkoy2XNVvQ09PjUhA3bNgAjN22RGLC0oOGb4hwdZyRFtAC4QcOHOD1118HYrsQkxk7NBtOaMOBP//zPwe8tJVIBtLeD9uopaWlIxoXiKFYju/Ro0fdzWp4tVRdXZ3L7wsNSSmP8tb4xS9+AXh7fe3atYC/9hkZGbfschtyvYUQIoCEJZyHEq6+czjf/OY3AXjwwQd55ZVXRn0NMRTrxvKTn/wEgL/927+N+HwLiJtK7e3t1doGYGs8a9asUYsDmpubWbBgwZDnq03b+LF0oh/+8IcAVFRUuENHSwmKVVUOSFEKIUQgSaEo7ZAgXDqRxSat5+LXvva1ibuwFODb3/424JeGWQJ0ONra2lyMbenSpe5xHeKEx2rgradluHhYXV0d4MV6LY3NasOnUkpQrPnBD34A+AeWq1evdrX6lnIVy/hvwg1lb29vWBfE3L1vfOMbADzyyCMAfPGLXxz1uWIoO3fu5Fvf+hYwsqlDOE6dOjXEQIKM5GjcuHGDo0ePAn644tFHH3Xft14Fly9fBmDVqlUuG0Eu9/gxl/t///d/Ab/j/MqVK90BpPUmCJf1MV50SxNCiAASpihD5XG4QxyrurGGv3//938/cRc3ybFms1/5ylf47Gc/C0QekRGa02o5k6aIxFBs31ZXV7tWgeZSh84lslCRNZKeOXOmW9NYKp2phoXijPXr1wNeSpAp9dH6RNwKUpRCCBFAwm5tVg8ebjrg0aNH+c53vgP4cZ8nn3xyxPMUmxyKJYY///zzgKdcht+Bw2HTBysqKtxjWtvwXLx4EYCqqioX97VKEPAVp6VX2QHDzZs3p2z/yVhx8uRJ9uzZA+CSy23sycyZM+Pa5UqKUgghAphwRWmqJ9Kc6T/7sz9zz/v6178+4vtSO+Gxrtq7du0CfPUThJ0W5ufn65R7FEwhmoocGBhg06ZNgHeibbS1tQF+srPFhvv6+lSueIt8//vfd/FgG9dhvSezsrLiEps0JsxQmuGzg4NwhtJy/t566y2XFmQSW4yOpaC8+OKLgFe9BCPrjYdjRtFmVYuR2L61CX82Z2jGjBk89NBDI55vBza2vy0lSG73+Pnud78LeAdo5mrbKBmrn4/UvyAWyPUWQogAJrx7UDglaTOk//Vf/xWAzZs38+qrr0Z8HeFjw9bMPdy9e3dUP2cHamqjNjo2IdT2qHlETzzxxIjntra2MmfOHMB/L8zdHu90wamMJfRbcvmiRYuckrRDsnimBIUiRSmEEAFMiKLs6+uLaPH/9E//FPAPI/7hH/5hIi4rJdi/f7+L4Vj9ayROnTrl1E7oIYQYSXd3tyt4sLpuU4w5OTluEJbFL5csWeIUjtVxKzY5fv7xH/8R8OPC999/vxunYets70e8mRBDGclI/vSnP+V//ud/AH/y37Jly0Y8Ty53ePbs2eNO/h5//PHA57/++usjMgnkcoenvb3dtUSz8IR1KT948KA74bbnrF692rnaanhxa5w+fZp9+/YBnssN3km3Hd5YVsFEZRLo3RRCiAASXnR6+PBhd5ewumQL1IKUZBB1dXU89dRTgc/bsWMHAGVlZU7hS0lGJjMz03WnsZQrO5TJy8tzBzzmgsdj6uhU5Tvf+Y47ODMVX1xc7LyniV5rKUohhAgg4Yqyvr7eTU/70pe+lOCrmXy88sor3HnnnYHP27x5M+B1s5FKj47c3NxRE5kLCwtdfMwmVorYkZaW5kY7lJaWAt7BWKJUuxSlEEIEkDBFae3zN27c6MYTxLP7R6oSjZoMZdasWXFPzp0q2EgHO5Vta2uL2MNABFNVVQV4paDWy9OUpZ1lJIKEGUr7sD7//PPaXBOIfahBhzm3SnFxMeAbTFXf3DrWoGXTpk0utGHTK61uPhHI9RZCiADSxhLYT0tLawBq43c5caV0cHBwXqIvYjS0tvFjkq8taH3jSVRrOyZDKYQQUxG53kIIEYAMpRBCBCBDKYQQAchQCiFEADKUQggRgAylEEIEIEMphBAByFAKIUQAMpRCCBHA/wN+WBbwcEwrEgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7faa847ac1d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_conv_output(values=layer_output1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Output of Convolutional Layer - Method 2\n",
    "\n",
    "Keras also has another method for getting the output of a layer inside the model. This creates another Functional Model using the same input as the original model, but the output is now taken from the convolutional layer that we are interested in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "output_conv2 = Model(inputs=layer_input.input,\n",
    "                     outputs=layer_conv2.output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This creates a new model-object where we can call the typical Keras functions. To get the output of the convoloutional layer we call the `predict()` function with the input image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1, 14, 14, 36)"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "layer_output2 = output_conv2.predict(np.array([image1]))\n",
    "layer_output2.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then plot the images for all 36 channels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVMAAADuCAYAAACEaORrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAIABJREFUeJztnXmUFNX1+D/dw8wAA4PoKAgyjBu4AUZFwYALrrgvx7jiGmNMXBOjRr96Eo+J5qgxalzQaBQT97ifGHfFHRk3cEEBh0UWHUBBBmaYmf79Mb/76lXT09PV/aq6Gu7nn36n6nXXe13Vr++97y6JVCqFoiiKUhjJYg9AURRlXUAXU0VRFAfoYqooiuIAXUwVRVEcoIupoiiKA3QxVRRFcYAupoqiKA7QxVRRFMUBupgqiqI4oFuQzjU1Nam6urqQhpKdhoYGGhsbE2FfJ9scJVoskQhnGMWco8xNXpPJ8P5n6+vrG1Op1MahXeD/s74/r2ET1RyhNOYZaDGtq6tj6tSprFmzxhxrbW0FoEePHp2+r6WlBYD29nYAysrKzLny8vKM71m1apVp9+jRg1122SXIUPNG5mizYsUKwBt3z549u/wcmStAW1sb0PlchWLOUVi8eDEA/fr1C+36iURiTmgfblFXV8eUKVNYuXLlWud69+7t9FrLly837R49ejBq1Cinn98Zci8zhYXL77SioiKUa0f1vEL2Z1aYNm0a4F9fNttsMwCqq6vX6v/9998D3vrUq1cvACorK02fsrKynOepar6iKIoDdDFVFEVxQCA1X7BVimzqvZCPmjF9+nTTHjJkiFGVwyaVStHc3GxUAPDshxtvnLuZz7Y5hml/dE2Y6n3UtLa2smzZMj7//HNzTMxHtbW1AAwdOrTLz1m9erVpd+/ePWOf5uZm0y4rK8uodofJkiVLTPvLL78EYMGCBYBn4x88eLDpI/d5ww03BKCqqiqScbpEzDcTJ04E4L777gNgxowZpo+Y5DbZZBPAb45ZuHAh4D0DJ510EgAHHHCA6bPddtv5THbZKJ1fuaIoSozJSzK1DbxhYf/T9+nTJ5JrCslkMnIJbc2aNZFLMy658847AZgwYQKQm8YSNt26dWOjjTZizJgxBX1OZ9KoTbrWErU2UlNTY9ryHNXX1wNw2223AfDFF1+YPn369AFg0003BTo2eIQBAwb4zh100EEAbLvttqZPZWVlzhJbWIg0/Zvf/AaA008/HfDPc/Lkyb5jL774ojknm5DbbLMNALvuuivgv5fdunXL2XtHJVNFURQHxFYytW2WUZJIJLp0YQqD8vLy0PxXo+DTTz8F4KGHHgLgtNNOK+Zw1mtEsjr33HN9r3PmeB5pb731FuDZVT/++GNzrrGxEYBly5YB3p6HuEFCh50xqn2MXNlggw0AfG5pLlzUVDJVFEWJkLwk0zB57rnnAL/TfnNzc+T2RPv6P/zwA+DZaFw7fK8L3HTTTcUeQqfYO7gigYkD+FdffQXAhx9+aPq8++67gOfofeKJJ5pzI0aMADxpZffddwc8uxt0BGnE0f5t7+bb7a5oamoCvN8BdPwWotzHKAVUMlUURXGALqaKoigOiJ2af9dddwFw8sknm2OVlZWRbc60t7ezatUqPvvsM3NMVENxBp47dy7gd8H46KOPAM/gLfMA2GGHHcIdtNIpqVTK58IzZMgQALbcckvAi8u2N2Bmz54NwMiRIwFPlQfP/CPO4PJciioMHRs1UbkNpVIpWlpafE77oo6Lo7o45ueLzDU9J0WU7l9tbW0sX77cbIpBbqaKRYsWAXDZZZcBcM899wS67sKFC325SLKhkqmiKIoDQpNM33jjDdMWI7841Wbi3nvvBTxXjS222CKsoWUlmUxSWVlpJBeAvn37AjBs2DDAk1Dlnx+8kLT/+7//A4JLo8V2M1m6dCngzU0y5dhuYt9++y3gZdGyv6O4kkgkjMtMNnJN79ZZ2KUtvUQZZJJIJKioqPBlRRIpeebMmQD8+OOPgPcc24gjfi6BCcWkrKyM6urqjNmfsiHfxTnnnJPzexoaGkzbziDVFSqZKoqiOMC5ZCr/hrYNaq+99uryfRdeeCEAl156KQDDhw93PbScSSaTPmlG2iItyz+85D8E2GeffYBgblO2EzRQVHcasauNHj260z4iidsSeakikuQ333wD+CXwgQMH5vw5olEUW7KzJWbRGMLUHFpbW2Pp/gXePQW44IILAHj88cdzfv+sWbNMe8yYMXTrltsyqZKpoiiKA3QxVRRFcYAzNV9E/kmTJgF+d4xsmzF/+MMfAE+tP+KII1wNyTmy8SKbZLZrRj5RUfamRY8ePUo6Nr/UEPchMdXYWZeCIKaaIBsVpYqt1ieTydg+r7feeqtpn3LKKQA5qepff/010JHDVAjilqmSqaIoigOcSaay4SRx0FdddVVO7/vjH/8IwC233ALklvk8asQBWxy2JddpvmMVibQY2anWd0S7kCzt+eatFXej9UEiFWz3vTjG5cvG01ZbbWWOHX300YHf/9Of/jSv66tkqiiK4gBnkqk40/7yl7/ssq9kxgav3soxxxzjaijOkX/kQl2D5HPi4k6zPiK5OYNkTcpGqWoXomUFqYgQR2nUZt68eUD24KBMSICR2ErztQWrZKooiuIAZ5JpkPDPG2+8ca12nKtiupI+xE6XnjCi1JB/7rg6bWfDlY3TDtgoRYJIpKJJ2ZJpHO99vln1pR7WRhttVND1VTJVFEVxgC6miqIoDog0n6m4QZ166qnmmN0uNezSJrmoTbLhlGusb1yJo4qXL++//z7gFZgDT+0TFxv7fsmx9NLO6yLiEphp4ylODvurV68Ggm/oSj5i20m/EFQyVRRFcUAiiJSRSCS+A+Z02TEcBqdSqdDFAZ1jJKwP89Q5OqQU5hloMVUURVEyo2q+oiiKA3QxVRRFcYAupoqiKA7QxVRRFMUBgRwea2pqUrlWcXRNQ0MDjY2NoTu35TJH8Wuzfe3EF09e8/HDi9Mcw6S+vr4xil1gfV47RzaeC/EXjWqOkHmekphb/GEzbabL77EQ3+5c5xnoCnV1dUydOjXvQRWClB4OmzDmKKVjlyxZAnh5COxY4IqKCnbbbTen1+2MTHOU6gHygNbW1oZ2/UQiEYmLS6Z5pueSlR/id999Z48P8OLvJdMUdP6jbG5uNu3KysqSfl4lFl9yScicRYiAjsCGqJ5XyD7PN998E4AHHngA8AdgyL2TTG92qWj5/UkGMan2sfXWW5s+tbW1Oc9T1XxFURQH6GKqKIrigIKDxKV8Q6mnJAsTsfUU005pk0qlaG1tZc4cT9sWNX+nnXYq1rBCw86hkJ6CL5nskCeypYBctGiRaUv6RFEXv/rqK8AzF0DHfY46GMYuzjhjxgzAq/++wQYbANC/f3/TRwoIZko7t2zZMsCbk6jIovbLubgE/IwZM8b3ate9//TTTwHvPtnn6uvrAc8sMG3aNADGjx9v+vTu3duYvrpCJVNFURQH5CWZSplcgO+//x7w/vFLtYyD0NbWxg8//MDjjz9uji1cuBCAyy67rFjDcsqaNWtYuHChT7IYO3Zs6NdtaWkJ/RqZCJIIORO2RJeOvVlhE1VWpZaWFubNm8f//vc/c0yKW8pvUTQiuxy5jE+O2XOUDaf0hO/pmbLilDnKZsstt8zY7gqR6AcMGGCOdevWzWgvXaGSqaIoigMCSabt7e2sWrXKlMsFaGxsBOCJJ54A4JVXXgH8uQVHjhwJeLaJxYsXm3OXXnopAIcffnjgwYdBWVkZffr0oW/fvuaYlK2+/PLLAa8U7IEHHmj6iM3tX//6FwDbbLONOTdx4sQur/vJJ5/4bHthUlZWRnV1tcnbGRVxL8hWilRUVDBo0CDOPPPMYg+l5OmsdLtKpoqiKBESSDJNJpP06NGDzTbbzByT9s477wzA0UcfDfh3QF966SUAnnrqKcDbXQT45ptvAM/OI59nS4bFsM0cccQRpt3U1ATAzTffDMB77723Vv+BAwcC8M477wCe43M2ZIcRcv/3c0EikaBbt25rOZp3hdyjiy++GIDnn38+0HWLZTO1n8XPP/8c8OYiHg2iYQG88MILAHz77bcAPnvk7rvvDsDs2bMBGDFiRFjDVkoMlUwVRVEcoIupoiiKA5xXdhs0aJDvFbwNqN///vd5fWaxnYNPOOEE36srxHwAHepioS48uZJMJqmqqvJtJIobTbq5wY5Zv+aaawDPNJMLtsNzlKYMIZVK+eLpN998c9+rOKZL0ALAhhtuCMBxxx0HwOjRo9f63GzqfXNzc+TP7NKlS01bckHMnDkT8PIs5FtXPg6kUimam5uZP3++Ofbhhx8C3qa3mBPtAIZNN93UvB/g3XffNedkA/m5557zXcs2f/3444/qtK8oihIleUmm9sqdS4oryeYi//RBJZS4OgcXSrE3L+wQ4M6+4yeffNK0JfzO3jjrCvu5KEaJ60QiYUInAV/bxt4Uve666wB/tqiusN39+vXrF9kzm0qlWL16tdksA6+E8fTp0wFPghMXP/CksfPOOw+Am266KZLx5ouEQNvugxJwsOOOO/pebUd9kUglVNTeWL7kkksyXks2F6FDss31XqpkqiiK4oC8RIVc3GjuvPNO0xYH/mLYzOLI8uXLAX9uxWKQ7R9X7FC2NBMkR2cmm2EcNQxJ1PP222+bY4ccckjO7xe7d7HuZSKRoHv37r4gEbtt8+WXX5r2lVdeCeRvR43aJix2/h122MEcs9tdse+++wL+nKzpiCRvB7NUV1fnHGyiq5uiKIoDdDFVFEVxgPMdAVEPJdIE4Prrr3d9mZJE3DLi7KIiLkJ33HEHgM8VZd68eTl/jrgcxT0eXyLW7ByWQZB5xjmfr7j22C5te++9d+DPsXO2lprJTqLv7Jwh6edkw9G+l0HmWVrfiKIoSkxxLpk+88wzgD9je9ylk7ARt5UgBvNiIcXJxBj/xhtvBHq/bEzE/Z7LPZFAiaDjleKIdo7QuCGZ8SXwIh9p1KaUJdNsbm6SfV82nuy8IEEorW9EURQlpjiTTMXtQhxnJ0yY4OqjY4O4VcydOxeAIUOG5PQ+CduUWjpxROxp6VUFpK5Oroh9Lu4VF2R8QeeX/v4gjv1RIeGU8pwWGqYs2kYxgi7CROzl4rJnZ8PLB5VMFUVRHFDwX42Elorzs+R7tJMDFNux2RUiYUpChWySqWTcB38IW1wRe9i4ceMAGDZsWF6fE3dbqZCvXUyI87MskqnUcMolyCYbIpnGMeiiELbaaivAe2YLtX+rZKooiuIAXUwVRVEcULCaL+U5bFeodETtEHebqqoqc07yDWZypo0bUup2jz32AOD+++835wYPHuw7d9JJJ0U8usKQTYpc1F/ZiLPjnOX+lcJ9tBH3IcmsBJ4KL5mk7CxZEvcuZWriSM+ePZ18jqj3peYGlQkpT2/nM5XfbJBy0Nko/W9JURQlBiSCZH9JJBLfAXPCG05WBqdSqY3DvojOMRLWh3nqHB1SCvMMtJgqiqIomVE1X1EUxQG6mCqKojhAF1NFURQH6GKqKIrigEB+pjU1Nam6urqQhpKdhoYGGhsbQ49nyzZHCZEV31obCUmT13xC76Keoz0Pu9Y4ePMII2FJfX19YxS7wOv78xo2Uc0RvHnaz6kkdU5PRG6Hz7oIb851noEW07q6OqZOnZr/qAogSDG3QpA5ijM3eBmVJAuPlNKVfATg5SgQh2k7W7csSJIvsX///gAMGjTI9Nl0001NXoOwyTRHKYD4/PPPA55z84Ybbmj6yJzkYbXj03fddVcAjj32WMDL3Zop01AikYjExWV9el4zITkxXDnxpxPVHMGbp11mfsqUKQDcfvvtALz66quA/5mT/pLT1X6eDz74YMAroLjXXnsBa2d3y3WequYriqI4QBdTRVEUBzjP9ir1x+2kudtvvz2QW5JaSU5s164OS03JRCqVYs2aNSxdutQck9RcUntbXmfMmGH6SLmPhoYGwG+DFLVDbDyieqxatcr06dWrV0ZbbJjYORIuvPBC36vw/fffm7ao/qJW2rXm5fuSz8yk3merWa64J8rfTVTY9tCxY8f6XnNh9uzZpi2/R/meCk0xqJKpoiiKA/KSTL/++mvT/vTTTwFPEhMDr13q+U9/+hPgZd+R94CXkeaAAw4AYJ999gFg//33N31qa2sjk9oSiQTl5eW+zaHOGDp0aMZ2vsSxLIRkTgI48MADfa9BKbWMUkp8aGtr44cffvB5l+QjeUvC7EzIJpVs3AW9hkqmiqIoDshLFLKTo4wfPx5Y259rzz33NO0zzjgjn8v4iLocxvvvv2/aYscdOXIk4OVgLVVWrlzJlClTjBsUwBVXXOH0GldeeSUAV111ldPPDUIqlaK5udnnTiO2fLF3P/nkkwAsXrzY9BkwYADg5d99+OGHzbltt90WgMcffxzw8psq4bJmzRoWLFjgyzt7wgknOL3GRRddBMCRRx5pjgUpOaSSqaIoigPykkyz2R1KnZaWFubMmcNnn31mjk2ePBmAe++9F/Cc9aVUrH3s/PPPB+Dyyy835yRDf1xoa2tj6dKlfPDBB+bY/PnzgcLL3Qp2QcVikUgkqKyszFhQbsSIEQD069cP8HtffPTRRwC88MILgF/6fOihh9Y6lk5LSwtxSm357bffAviCNBobGwFvrldffbU5J8EpUi1CnNbPPPNM06fQIn1BKS8vZ8CAAcycOTO0a0yaNAnwvI+gw/Mm13upkqmiKIoDdDFVFEVxQFF8cV5++WXTFleodGyVxHYuD5uKigoGDx7MKaecYo7ZbfBUwjlzvBBzqcEdBNuJvbW11SRsCJsePXowfPhwjj76aHNM1D5Xav6ECROcfE7YSJ4EG3GLExOA5B3IlYqKishrzNvuPGKyWbJkCQCjR49eq7+4GEmeiXPPPdecE3c4cYYXtz97I2/58uWRBpkkEgnKysqMoz3Aa6+9Bngx9YVy+umnA/7fcmVlZc73UiVTRVEUBxRFMu1MGrWx/2krKipiZdCXf/V8pFEbKT8LHVJQVCV1y8rK6N27t88h+amnngI8l5/NN98cgKOOOsr0EWd9CSvNdk/EhaiYtLe309TUxLJly8wxka4ktFnc3ObNm2f6PPLII0Buz2kcaG9v9wVEDBkypMv3iAYSRIOwN50qKysjdVdMJpP06tXLlFIHL5ubbLClZ3sKSk1NDdCRoSqvMRZ0dUVRFAWISDJ95ZVXAHjssccAuO2227p8j/1Pm0wmI7dBZUMksnzHtHz5ciD6QAQhmUzSu3dvn9Rptzujb9++QHaJNE4kk0l69uyZU0jge++9Z9qSp3XHHXcMbWwucaHRiBsYeHlro8qvGwTbzTAXl8NccrrOmjULgP322w+AnXbaKa+xqWSqKIriAF1MFUVRHBCamm/nDXz66acB+O1vf9vl+2SzwC77USx1uDMKNTlIdJAYvOPMX//6V9M+9NBDiziScJAIoGnTppljv/71r4s1nMhZsWIF4FeD46jeB+HLL7807Vw24ySbXVAXuHRUMlUURXFAaJLpxIkTTVs2XLbccssu35ctU3uxKHTDSZB8iWFU/AwLW3o++eSTiziScBCn9VGjRpljhbrYxAUJLsn2vN16662At/myLpCLNApe0I38ru1ie/mgkqmiKIoDnIt///jHPwB4/fXXzTFx8s6G2G5EMo2TK1ShY5Ew0SASelxYF6VR8EJ5xT5YW1tbzOGEQjaJ9O9//zvg/U4vvfTSSMZUbOxAGalbNm7cOCefrZKpoiiKA5xJppIDUTLs//znPw/0fpH+ogqpzAfJWWp7GuSCzKkUJNIHHngA8MIN7fC9dETSBs/RPe5IOKnYwddFiTQXRFs69thjizyS/FmwYAHg5RWWEOhsTveSk9ju52oPI74rl6IoSgmhi6miKIoDnKn5+ebBlJyIUeYsDYpsjslGWu/evQEv3yV4mWYWLVoEeJlswHMvGjhwYOhjzRfJMSsF8CQuPZuaXyqqvY2odHE2J4XFgw8+aNpvvfUW4C9FUmrIvRQ3SlH7s6n5dg6GfEuWd8b690QpiqKEQCJIBqBEIvEdMKfLjuEwOJVKhV6ZTucYCevDPHWODimFeQZaTBVFUZTMqJqvKIriAF1MFUVRHKCLqaIoigN0MVUURXGALqaKoigOCOS0X1NTk8q3DGqhNDQ00NjYGHoqqZqamtTgwYNNLkiAlpYWwItnrqioAPylb11kuYpyjsW6jwD19fWNUbjUrC/P67o+RyiNeQZaTOvq6pg6dWr+oyqAXXbZJZLr1NbW8uabb5q66gCNjY0AfPLJJwBssMEGgD/hiZRb+f777wGvkifA9ttvD3Qd5RXVHMO8jxLRJq92km+JOkokEpH4C9bW1jJ58mSToAbcJ37uLHH4unAvuyKqOUJpzFPVfEVRFAfoYqooiuKA+BRaignJZNKn4oOXqMRVRu51GakkG4eKsqlUijVr1qx1P10yY8YMADbddFNzrBgJYKRyAHgJP+R1yZIlvlfw7P9iqrKrk4ppRhL6SB7ejTf2zNylVMcsKlQyVRRFcUAgyTSVStHa2uqrSy3/cNtttx1Q+qnNmpubmTVrFnfffbc5JmnKJJO3K55++mnT7tOnj0n1FzapVIqWlhafpNLQ0ACsXRXArk5aaiQSCcrLy31Sl2tmzpwJeJuS0CHlRZXzQu6lfb0tttjC95oJSUUnlQfsypzpY+/evTsAra2t5ljUmodoGXGWiEt75VMURYkJgSTTlpYW5s6dy6RJk8yxRx99FIDZs2cDXs3qbbfd1vQRyWCfffYB4Iwzzljrs++66y7AS9g6aNCgIENzRktLC3PmzOGf//ynOXbNNdcA8Le//Q2AYcOGAbB48WLTRyQ68YXbbbfdurzWk08+adqnn356ZBVZ29vbWb16takbDjBlyhTAc/+S5MFiE7Tp378/AIcddpg5Nn78eAB22GEHwJPii2k7TSaToUmls2bNAuD5558HYPjw4eacaGtRkEgkjN9zEHJ5PuOEaBlhkG9tt3RUMlUURXFAIMm0srKSLbbYgmuvvdYck7bUo5bSHiK9AIwYMaLLzx47dixQPIlU6N27N+PGjePzzz83x+6//37Ak0TffvttwF+BVeYrNrRsTJ8+HYAxY8aYY6NHjy74nzFXysrKqK6uZtSoUeaY3bZpamoy7Xnz5gGenc2OSJF/d7H7zp8/Hyj+/QyLwYMHA/Czn/0MWLvKaVRahmBLw6IlyhhKoSpuMREt7IADDijoc1QyVRRFcYAupoqiKA5w5rTfp08fILioLO4WcSufYru6nHvuuTm/b6uttuqyj2zOyGZNnLE3cIYOHdppv84c1W31s9Td5mzuuOMOAA4//PCijqO9vZ2mpiY+++wzc+zVV18FYPLkyQB88cUXgGd6Ac/JX8wTV1xxhTlnm6/WZR5//HHAM1EWyrrzdCuKohSRooeTSkYm25VqXWXu3LnA2psVxUACMNJx7X4S9UZMJuxQS3H1ko3CF198EYCnnnrK9Jk4cSLgd/0SzjrrLAAeeughoPhSXDKZpLKy0gTNAAwcOBCAgw8+GIBvvvkG8AIzwAstld9dsSVslyxcuNC07TDfdI477jjAc28rFJVMFUVRHFB0ydR2oYo7kqMzqCP6okWLALj11lsB+Mtf/uJ2YHkQphN0+nWKjYRDguemJ+GT3377LQAnn3yy6bP77rt3+ll33nkn0BFkkf7ZxaKsrMxn25a2SGUita5atcr0kWfSdYh0MRGbcLbENqNHjzZtCTDae++9nVxfJVNFURQH6GKqKIrigEjVfKmrZEfViEtVKZBvnPkNN9wAwHXXXedyOLFGXKLi5g4l9bymTZsGeLkIzj777E7fc+GFF5q2RHTdcsstYQ0xNGyTy7qk3ktEnkR+7bHHHmv1ETcxidAE/4acC+L1pCuKopQokUqmK1euBPwO8esqL7zwgmln+qdcV8l3ky4qxCVI3GdyCci4+eabTfsPf/gDQKg5Ul0hLmGSN6GUc9Nm46uvvgKy/85OO+003yt4+RVcoZKpoiiKAyKVTEtdIn3jjTcAL8NVNiSbEHjln0sB0R7E5SeohBlXiVSQeYmTejZpTRz07dpfdthlXBF7tczVdolal5A9mGxh2Q8//DDglfi+5557QhuPSqaKoigOKFgylXBQSa7Qr18/wJ/Ju7KyEvD+SeJcxyUTkn9U8h5KYoRDDjlkrb6SsKWUpFHw7GsS8io2RdkptamqqgJg6623Nsfknz/ukmnfvn1z7it5a3/xi1+ENZxQSPegCDLnUiKXPZirr74agIsvvjj08ahkqiiK4gBdTBVFURxQsJovBvxjjjmm0z4S/yyqYKkhBm4p/5CuCoMXBx2HWPR8kM0KySKUKYuXuNhItim7kFvc1fsgvPbaawAceuihQLB8tkp0dKbev/7666Ytv9kJEyaEPh6VTBVFURyQCJLhPpFIfAfM6bJjOAxOpVIbh30RnWMkrA/z1Dk6pBTmGWgxVRRFUTKjar6iKIoDdDFVFEVxgC6miqIoDtDFVFEUxQGB/ExrampSdXV1IQ0lOw0NDTQ2NobuxKlz7ECSZdh+s658aOvr6xuj2AXWexkuUc0RSmOegRbTuro6pk6dmv+oCmCXXXaJ5Drr+xznzZsHePkU7KxKssCmO++Dl5NAjknBOvkcu08ymYzExSWXezlnTsdQ7CJsLoJL4nAvwyaqOULmea5YsQLwcmVsttlmoVw713mqmq8oiuIAXUwVRVEcEGlyaCW+SMrAjTbaCMhclkNSu+Wb5DuOeQtcl65QoqN3796+12KjkqmiKIoDVDJNI5VKsWbNGlMSGOCzzz4DvCxJIr3Jhgx4mxaSfUkKt9n944YkhAZvo8jeiFHiT0tLC/PmzeOpp54yx7744gsANt64w2HivPPOA9bdJNH58tJLLwFeIT47C5rQ0tJCriH3KpkqiqI4oGDJVCS4TKt6Loi7Q21tLVD8nKdNTU1MnTrVlGEBz32mf//+gJfPtL6+3vTZd999ARg6dCgAQ4YMMeckp+J3330HeGWgzz77bNMnymKDbW1tLF++3Heou94WAAALbElEQVSsuro68OfMnj3btLfYYouMfeS7g7XLaRQTKaEzf/58ADbffPO8PmfatGkADBs2zM3AApJMJunZsycjR440x0STkrLUUn5l0qRJpo+UF1qfkd9sNioqKnK29cfn6VYURSlhCpZM85VIBZEQii2RChUVFdTW1vqKqGUrB5zO4sWLAfj444/NMZFI77vvPgBuueUWwMvOD3DqqafmPeZ8aG9vL1gaHjRoUKfnROJ77LHHzLFcJIGwmTVrFuBpQvlKpEKxJFKhW7dubLTRRj67vBSzPP/88wF49tlnAfjggw9Mn/Hjx0c4yniybNkywJ0tWSVTRVEUB+hiqiiK4oCiuEZdcsklpj127NhiDKFTysvLGThwYN7vF8P+/vvvv9a5nXfeGYATTzwRgOHDh+d9nUIoKysrSMW/9tprAbj00ks77fPwww8DsGrVKnNM4vWLiWwG5ovEgYujeJw21dKRzVB5VfyBIzfeeCMAF1xwgZPPju+ToCiKUkJEKpnKP8BNN91kjsWtjG57ezsrV640G0ngZacZMGAA4DlD54tsEJQCdmDCYYcdBsCxxx7baX9xcBb3nMMPP9ycK/R7c8nSpUsBbwO1V69enfYdMWKEaX/yyScAOTtyK/Fg+vTpABx//PHmmCuJVFDJVFEUxQGRSKYi3bz11lsA3HHHHeZcWDkI8yWZTFJVVdWpE3o+NDQ0AB05GUuNyy+/3LTff/99wHO1ycStt94KeK5utlRn5zYtNkHst3Zo8TPPPBPGcJSQETv/Aw88EOh9QTQQlUwVRVEcoIupoiiKAyJR8yW6R2LbzzrrrCguGxskyquUkPwBoh4BPP/88532lygvibI56qijAHxuZnHMZ5oNcf2SaCmAQw45pFjDUfLg+uuvB+C4447L6/1BnlmVTBVFURwQmmT6xhtvmLZsXEj+wFJHcpXaWaPSnfT/+9//mvZBBx0UzcAcsGDBAsDbJBw1apQ5lykQQbj99tsBz31s1113BTpix0sVkbKvu+66Io9ECcLkyZNNe+LEiQB89dVXoV9XJVNFURQHhCY2PPLII6Ytmb4LCdOMA1InSSTSTJmuxMHfzghVSog0Jraid955p9O+M2bMMG2xmZ5wwglAvBz0g/Kf//wHgEcffRSAPn36FHM4eSMaVFwrPYSFnbHtz3/+c2TXVclUURTFAc4l03fffRfwJ1ewM8qXIosWLQK8UELJg5jJhiiZ13ffffeIRucGSeAhDuoPPvhgl+/53e9+Z9qS6X2bbbYBSm/n3g6Rfe655wCvokKpIRqU3NP1TTI95phjTFs8iKJAJVNFURQH6GKqKIriAOdqvhjrJcPQuoCo9bvssguQPa671NR7QVRCmWO2cjSiBtsbigceeCAQbWFAF0ggQlNTkzmWXmyw1JB7kO1eiCtbqZvgMhFUtZfcIYXmplXJVFEUxQGJIFlREonEd8CcLjuGw+BUKhW6v43OMRLWh3nqHB1SCvMMtJgqiqIomVE1X1EUxQG6mCqKojhAF1NFURQH6GKqKIrigEB+pjU1Nan0OkbioyUbWWVlZW5GlkZDQwONjY2hxyhmmmNURD3HtrY2c0za6RuStu+d3NtC/fHq6+sbo9gFznYvZZ5hhb3q8+qWUphnoMW0rq6OqVOnZjwnTt/i8Gw7zpaXlwPewmtnnpcfpvTpDHEmDxuZ4xNPPGGOidP+6aefHuq1o56jzAu8bFeSh+Cjjz4CYP78+abPjz/+CHg5Squrq825QYMGAZhChOLQbz8HPXv2BCCRSETi4iLzlHGD96cRdiaoKO/le++95/uDiyovQlRzhMxrz5QpUwDYeeedAXeCnF3mvVevXowdOzan96maryiK4gBdTBVFURzgLDZf1KZs6pOoInGqn94ZRx55pGlLguQXX3wRgP32268oY3JN375912pL8TgpO7Iu0KtXr2IPITTa2tpYuXIlq1atMsdEzZ81axbgmWdsU1q/fv0AWLlyJQDDhw8359LzMkhKv7jlXQjrGbVTFjY1Na21j9AZKpkqiqI4IDbVzmQzxJaW4sLo0aMBb4zyzy+lgAGuueYa33suu+wy0z7ggAMA2HPPPUMdp7L+0d7ezo8//ujbnPn6668Br7jh4MGDAU/CBE9qlWfaLjMj/QXZaBQpFjo2e9bVUHS7CGR1dXXOG1sqmSqKojggNpLp0qVLgXhKpoKMLds/srgSTZo0yRx7+eWXAXjvvfdCHF1wbJchKYB48803AzBnTof3ku3GJpKJ2Nv23Xfftc6J7e34448HvDImxaC9vZ0VK1Ywe/Zsc0xK/sp4xdZouxOJTb+1tRWA8ePHm3OdFYW0n4kVK1b4fHjDpKWlhblz5/LCCy+YYyJ1iuuS7FU0NjaaPiK9ih11n332MedmzpwJwFZbbeV7/+rVq02f1tZW8/0oHahkqiiK4oCiS6ZiWxSJZssttyzmcApms802A/zO7nGjra2N5cuXs3DhQnNsjz32AGDChAlA5iAKkUy6d+8ewSgLp62tjaamJqP1gGc3lGJ5Ymv89NNPTR85J1K1HXggngE9evTw9bXtif379zcBKmFTVVXFqFGjGDVqVOjXsne529raugy0iZpnn30WgMcee8wcEwlcypI/8MAD5ty4ceO6/MwVK1bkfC9VMlUURXGALqaKoigOKLqaLy5FH3zwQZFHsv5QVlZGdXW1L7Y+F0pFvRfKy8vp16+f2TAD2HvvvZ1eQ3JR2O4zVVVVoSX8iQtxnN8hhxziew2KbLraa9HIkSNz3mhTyVRRFMUBRZFM7ZVf3IZ+8pOfFGMonTJt2jTTloxKYZZxXr16dSycoGWuCxYsADqy9QjpYcB33323aR9++OEA1NTU+PrYGXji7PaWL9tuuy2Ab5Oiqqqq4DSFuZJKpVizZg1Lliwxxz7//HMA3nrrLcDbmLFd88QV7OSTTwb8ASjFdGcrJhKsYIfEb7DBBlnLntuoZKooiuKASCXTpqYmAP7973+bYzfccEOUQ8iZYcOGRXq97t27R5aHMhu9e/cGYOjQoV32PeOMM7rsY4fmxUHyFsSpXoIS7LGJ21M2WlpaAM92mMt7wiCRSFBeXu5z35L2mDFjALjooosAv2aRz7OWnkwlTvezENITheeb0EUlU0VRFAfoYqooiuKASNV8UQsffPDBKC+rOEZU40wRMKL+SnSVnUs0Dnls01VTmYuYN3KlubkZiMec0pE5yhjFtadQ1zbbHSqqDbYocGVeW3e+EUVRlCISiWR65ZVXAvicp5XSJVtMtkhBIunFNcu9SFlBxycSt2xS5Oo2EyWyUSTubZKzdLfddsvr8yQwwZZGe/XqFYsN00J45ZVXAC8fSHoe16CoZKooiuKASCRTCdO67777oricUkTE/U2k17hJbunSVFDpShy64zYvG5EkpRTySSedVNDnSYnudQ0pEe1Ke1LJVFEUxQGRSKZ2pvJ1HclrWVVVVeSR5IeERT700EOA3zFfKrb+6le/AvzOzZLrVOoOZatSGwfEjijO++lhsODNya6dJDk945ToI5VK+bwUvvnmG8A/7iDIdyOeDuIFEFf7dxAk1Ba8UGBXqGSqKIriAF1MFUVRHBCJmu9anI4zparey6aFlPEQ95qPPvrI9MkWry+qocSox62khSDuQpIVTFTYkSNHmj5S5kRMNnYZZHH9itP80jfRJDb/nHPOyevzJLuXPBPyTJeio/706dMBL9fGYYcdZs7dddddAGyyySZOrlV6346iKEoMSQTJ/JJIJL4D5oQ3nKwMTqVSG3fdrTB0jpGwPsxT5+iQUphnoMVUURRFyYyq+YqiKA7QxVRRFMUBupgqiqI4QBdTRVEUB+hiqiiK4gBdTBVFURygi6miKIoDdDFVFEVxgC6miqIoDvh/KEq7GTrHoFYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7faa849ed048>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_conv_output(values=layer_output2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "This tutorial showed how to use the so-called *Keras API* for easily building Convolutional Neural Networks in TensorFlow. Keras is by far the most complete and best designed API for TensorFlow.\n",
    "\n",
    "This tutorial also showed how to use Keras to save and load a model, as well as getting the weights and outputs of convolutional layers.\n",
    "\n",
    "It seems likely that Keras will be the standard API for TensorFlow in the future, for the simple reason that is already very good and it is constantly being improved. So it is recommended that you use Keras."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises\n",
    "\n",
    "These are a few suggestions for exercises that may help improve your skills with TensorFlow. It is important to get hands-on experience with TensorFlow in order to learn how to use it properly.\n",
    "\n",
    "You may want to backup this Notebook before making any changes.\n",
    "\n",
    "* Train for more epochs. Does it improve the classification accuracy?\n",
    "* Change the activation function to sigmoid for some of the layers.\n",
    "* Can you find a simple way of changing the activation function for all the layers?\n",
    "* Plot the output of the max-pooling layers instead of the conv-layers.\n",
    "* Replace the 2x2 max-pooling layers with stride=2 in the convolutional layers. Is there a difference in classification accuracy? What if you optimize it again and again? The difference is random, so how would you measure if there really is a difference? What are the pros and cons of using max-pooling vs. stride in the conv-layer?\n",
    "* Change the parameters for the layers, e.g. the kernel, depth, size, etc. What is the difference in time usage and classification accuracy?\n",
    "* Add and remove some convolutional and fully-connected layers.\n",
    "* What is the simplest network you can design that still performs well?\n",
    "* Change the Functional Model so it has another convolutional layer that connects in parallel to the existing conv-layers before going into the dense layers.\n",
    "* Change the Functional Model so it outputs the predicted class both as a One-Hot encoded array and as an integer, so we don't have to use `numpy.argmax()` afterwards.\n",
    "* Remake the program yourself without looking too much at this source-code.\n",
    "* Explain to a friend how the program works."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## License (MIT)\n",
    "\n",
    "Copyright (c) 2016-2017 by [Magnus Erik Hvass Pedersen](http://www.hvass-labs.org/)\n",
    "\n",
    "Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n",
    "\n",
    "The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n",
    "\n",
    "THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}