mnist_training.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MNIST training example\n",
    "\n",
    "This notebook presents a full commented implementation of the jacobian optimization method introduced in [1][Relative gradient optimization of the Jacobian term in unsupervised deep learning](https://arxiv.org/pdf/2006.15090.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports\n",
    "\n",
    "The method has been implemented using Python and [JAX](https://github.com/google/jax)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import jax.numpy as jnp\n",
    "import time\n",
    "\n",
    "from jax                 import random, vmap, jit, grad\n",
    "from jax.nn.initializers import orthogonal\n",
    "from jax.scipy.stats     import norm\n",
    "from jax.experimental    import optimizers as opt\n",
    "\n",
    "from experiments.utils   import shuffle_perm, batchify\n",
    "from itertools           import chain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_layers   = 2\n",
    "seed       = 42\n",
    "n_epochs   = 10\n",
    "lr         = 1e-4\n",
    "batch_size = 10\n",
    "\n",
    "# JAX random generator key\n",
    "key = random.PRNGKey(seed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the training dataset\n",
    "\n",
    "We load the MNIST training and validation datasets as simple matrices of size NxD where N will usually be the `batch_size` and D = 784 (MNIST flattened dimensionality). As the first dimension is the batch dimension, we assume the samples to be arranged by rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append('experiments/')\n",
    "from datasets.dataloaders import load_dataset\n",
    "x, val_data, _, _ = load_dataset(\"MNIST\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model definition\n",
    "\n",
    "We implement the `smooth_leaky_relu` activation function\n",
    "$s_L(x) = \\alpha x + (1 - \\alpha) \\log (1 + e^x)$ and the `Dense` fully-connected layer we use. The `dummy` argument to the `Dense` layer represents the `Accumulator` layer described in Appendix D of [[1]](https://arxiv.org/pdf/2006.15090.pdf)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def smooth_leaky_relu(x, alpha=0.01):\n",
    "    \"\"\"Smoothed version of the leaky relu activation function\n",
    "    Inspiration:\n",
    "    https://stats.stackexchange.com/questions/329776/approximating-leaky-relu-with-a-differentiable-function\n",
    "    \"\"\"\n",
    "    return alpha*x + (1 - alpha)*(jnp.logaddexp(x, 0))\n",
    "\n",
    "nonlinearity = smooth_leaky_relu\n",
    "\n",
    "def Dense(params, x, dummy=0):\n",
    "    W, b = params\n",
    "    return jnp.dot(x, W) + dummy + b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we define the functions to initialize the parameters and build the model. In the end, the model will be specified by its `params` and 3 `g` functions; all the 3 functions perform a forward pass through the model, but with some differences:\n",
    "\n",
    "* `g_dummy` takes 3 arguments: `params`, `x`, `dummy`. The `dummy` argument is needed to be able to accumulate the $\\delta$ gradients when needed (Appendix D of [[1]](https://arxiv.org/pdf/2006.15090.pdf)).\n",
    "* `g_layerwise` returns the output `z` (the latent variables) and the activations `ys` for each layer (no non-linearities applied).\n",
    "* `g`: returns `z` only"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_dummy_params(params, x):\n",
    "    return [jnp.zeros(x.shape) for p in params]\n",
    "\n",
    "def get_model(n_features, n_layers,\n",
    "              nonlinearity=smooth_leaky_relu,\n",
    "              key=random.PRNGKey(0),\n",
    "              W_init=orthogonal()):\n",
    "\n",
    "    # Forward pass through the network.\n",
    "    # Returns the latent configurations `z`\n",
    "    # and the activations at intermediate layers `ys`.\n",
    "    def forward(params, dummies, x):\n",
    "\n",
    "        z = x\n",
    "\n",
    "        def step(Wb, dummy):\n",
    "            nonlocal z\n",
    "            y = Dense(Wb, z, dummy=dummy)\n",
    "            z = nonlinearity(y)\n",
    "            return y\n",
    "\n",
    "        ys = [step(Wb, dummy) for (Wb, dummy) in zip(params[:-1], dummies[:-1])]\n",
    "\n",
    "        # last layer (no nonlinearity)\n",
    "        z = Dense(params[-1], z, dummies[-1])\n",
    "\n",
    "        return z, ys\n",
    "\n",
    "    g_dummy = forward\n",
    "\n",
    "    def g_layerwise(params, x):\n",
    "        dummies = get_dummy_params(params, x)\n",
    "        return g_dummy(params, dummies, x)\n",
    "\n",
    "    g = lambda params, x: g_layerwise(params, x)[0]\n",
    "\n",
    "    # parameters init\n",
    "    def init_Wb(key, n_features):\n",
    "        return W_init(key, (n_features, n_features)), jnp.zeros((1, n_features))\n",
    "\n",
    "    params = [init_Wb(k, n_features) for k in random.split(key, n_layers)]\n",
    "\n",
    "    return params, g_dummy, g_layerwise, g\n",
    "\n",
    "params, g_dummy, g_layerwise, g = get_model(x.shape[-1], n_layers,\n",
    "                                    nonlinearity=smooth_leaky_relu,\n",
    "                                    key=key)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loss function\n",
    "\n",
    "The objective function to optimize is the loglikelihood of our model $\\boldsymbol{L}(\\boldsymbol{x}) = \\log p_{\\boldsymbol{\\theta}}(\\boldsymbol{x})$ that we rewrite as $\\boldsymbol{L}(\\boldsymbol{x}) = \\boldsymbol{L}_p(\\boldsymbol{x}) + \\boldsymbol{L}_J(\\boldsymbol{x})$ with\n",
    "\n",
    "$$\n",
    "\\boldsymbol{L}_p (\\boldsymbol{x}) = \\sum_i \\log p_i(\\boldsymbol{g}^i_{\\boldsymbol{\\theta}}(\\boldsymbol{x})) ; \\enspace \\ \\boldsymbol{L}_J (\\boldsymbol{x}) = \\log \\left|\\det \\boldsymbol{J}\\boldsymbol{g}_{\\boldsymbol{\\theta}}(\\boldsymbol{x})\\right|\\,, \n",
    "$$\n",
    "\n",
    "We further separate $\\boldsymbol{L}_J$ in a \"layerwise\" component $\\boldsymbol{L}_J^1$ depending on the intermediate layers activations $\\boldsymbol{y}_k$ and a component $\\boldsymbol{L}_J^2$ depending on the parameters only\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    " \\boldsymbol{L}_J (\\boldsymbol{z}_{k-1}) &= \\sum_{i=1}^D\\log \\left| \\sigma'(y_{k}^{i}) \\right| + \\log \\left| \\det \\boldsymbol{W}_k \\right| \\\\\n",
    "    &=: \\boldsymbol{L}_J^{1}(\\boldsymbol{y}_{k}) + \\boldsymbol{L}_J^{2}(\\boldsymbol{z}_{k-1})\\,.\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "The gradient of the $\\boldsymbol{L}_J^2$ term is calculated explicitly and can be computed efficiently by making use of the relative gradient trick (see section 4 of [[1]](https://arxiv.org/pdf/2006.15090.pdf)). Through automatic differentiation we optimize the latent variables distribution of choice $\\boldsymbol{L}_p$ (a standard Gaussian in this case) and the `loss_layerwise` term $\\boldsymbol{L}_J^1$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "@jit\n",
    "@vmap\n",
    "def log_pdf_normal(s):\n",
    "    \"\"\" Log-pdf for a Gaussian distribution w. mean 0 std 1\"\"\"\n",
    "    return jnp.sum(norm.logpdf(s))\n",
    "\n",
    "def inner_layerwise(sigma_prime, y):\n",
    "    return jnp.log(vmap(vmap(sigma_prime))(y))\n",
    "\n",
    "def loss_layerwise(nonlinearity, ys):\n",
    "    sigma_prime = grad(nonlinearity)\n",
    "    batched = vmap(inner_layerwise, in_axes=(None, 0))\n",
    "\n",
    "    # summing individual layers contributions\n",
    "    # Note: this works fine even with `len(ys)` == 2\n",
    "    full_pass = jnp.sum(batched(sigma_prime, jnp.stack(ys)), axis=0)\n",
    "\n",
    "    # summing over dimension\n",
    "    return jnp.sum(full_pass, axis=1)\n",
    "\n",
    "# Here the `dummies` argument is needed to be able to compute the `delta` terms\n",
    "# (Appendix D in [1]) through the JAX `grad` function.\n",
    "def loss(params, dummies, x):\n",
    "\n",
    "    z, ys = g_dummy(params, dummies, x)\n",
    "\n",
    "    lpdf = log_pdf_normal(z)\n",
    "    lwise = loss_layerwise(nonlinearity, ys)\n",
    "\n",
    "    l = - sum(jnp.mean(li) for li in [lpdf, lwise])\n",
    "\n",
    "    return l, (z, ys)\n",
    "\n",
    "# Function to compute the term L^2_J of the loglikelihood\n",
    "def log_abs_det(params):\n",
    "    Ws = [W for (W, b) in params]\n",
    "    return jnp.sum(jnp.linalg.slogdet(Ws)[1])\n",
    "\n",
    "# Note that here we want to compute the full loglikelihood;\n",
    "# during training we directly optimize only the term `l1 + l2`\n",
    "# and we include the gradient of the term `l3` explicitly\n",
    "# (i.e. the `loss` function we derive includes only `l1 + l2`\n",
    "# and the `l3` term is introduced with `add_det_grad`)\n",
    "@jit\n",
    "def full_loss(params, x):\n",
    "    z, ys = g_layerwise(params, x)\n",
    "\n",
    "    l1 = jnp.mean(log_pdf_normal(z))\n",
    "    l2 = jnp.mean(loss_layerwise(nonlinearity, ys))\n",
    "    l3 = log_abs_det(params)\n",
    "\n",
    "    return l1 + l2 + l3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Relative gradient\n",
    "\n",
    "Here we compute the gradient and apply the relative trick taking care of avoiding matrix-matrix multiplications as described in Section 4 of [[1]](https://arxiv.org/pdf/2006.15090.pdf).\n",
    "\n",
    "The core of the computation happens in the `apply_trick` function, where the updates are computed following Appendix F of [[1]](https://arxiv.org/pdf/2006.15090.pdf) to include biases (note that we transposed the update expressions as we are working with row vectors).\n",
    "\n",
    "The `add_det_grad` function completes the gradient updates by including the gradients of the $\\boldsymbol{L}_J^2$ term of the loglikelihood, that we compute explicitly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_relative_gradient(gradient):\n",
    "\n",
    "    def apply_trick(Wb, z, bp_terms):\n",
    "        W, b = Wb\n",
    "        db, delta = bp_terms\n",
    "\n",
    "        dW = W @ (W.T @ z.T) @ delta + W @ b.T @ db\n",
    "        db = db * (1 + b @ b.T) + b @ W.T @ z.T @ delta\n",
    "\n",
    "        return (dW, db)\n",
    "\n",
    "    def relative_gradient(params, x):\n",
    "\n",
    "        dummies = get_dummy_params(params, x)\n",
    "        ((grads, deltas), (z, ys)) = gradient(params, dummies, x)\n",
    "\n",
    "        dbs = (db for (dW, db) in grads)\n",
    "        bp_terms = zip(dbs, deltas) # backprop terms\n",
    "        zs = chain([x], (nonlinearity(y) for y in ys))\n",
    "\n",
    "        return [apply_trick(*args) for args in zip(params, zs, bp_terms)]\n",
    "\n",
    "    return relative_gradient\n",
    "\n",
    "def add_det_grad(gradient):\n",
    "\n",
    "    def det_grad(params, x):\n",
    "        grad_params = gradient(params, x)\n",
    "        return [(dW - W, db) for ((dW, db), (W, b)) in zip(grad_params, params)]\n",
    "\n",
    "    return det_grad\n",
    "\n",
    "gradient = grad(loss, argnums = (0, 1), has_aux = True)\n",
    "gradient = add_det_grad(get_relative_gradient(gradient))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logging\n",
    "\n",
    "The `train` function accepts a list of `loggers` to log scalar values. Each logger must be a function taking arguments `params` and `epoch` and must return a tuple: the first element is a string indicating the name of the logged quantity, the second element is the value to log. We define in this way a function `log_loss` to log the loglikelihood on the validation set during training, and a `log_time` function to log the execution time of each epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_loss_logger():\n",
    "    \n",
    "    loss_vs_time = []\n",
    "    \n",
    "    def log_loss(params, epoch):\n",
    "        val_loss = full_loss(params, val_data)\n",
    "        loss_vs_time.append(val_loss)\n",
    "        return \"Loglikelihood\", val_loss\n",
    "    \n",
    "    return log_loss, lambda: loss_vs_time\n",
    "\n",
    "log_loss, get_loss_vs_time = get_loss_logger()\n",
    "\n",
    "def timer():\n",
    "    start = time.perf_counter()\n",
    "\n",
    "    def log_time(params, epoch):\n",
    "        nonlocal start\n",
    "        now = time.perf_counter()\n",
    "        delta = now - start\n",
    "        start = now\n",
    "        return \"Time\", delta\n",
    "\n",
    "    return log_time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training loop with Adam optimizer\n",
    "\n",
    "Here we define a standard [JAX-experimental](https://github.com/google/jax/tree/master/jax/experimental) training loop using Adam with default parameters. We only add the `loggers` interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_opt_update(get_params, opt_update, gradients):\n",
    "\n",
    "    @jit\n",
    "    def update(i, opt_state, batch):\n",
    "        params = get_params(opt_state)\n",
    "        grads = gradients(params, batch)\n",
    "        return opt_update(i, grads, opt_state)\n",
    "\n",
    "    return update\n",
    "\n",
    "def train(params, x, gradients,\n",
    "          epochs = 100, batch_size = 10, lr = 1e-3, shuffle = True,\n",
    "          loggers = [], log_every = 10):\n",
    "\n",
    "    opt_init, opt_update, get_params = opt.adam(lr)\n",
    "    update = get_opt_update(get_params, opt_update, gradients)\n",
    "    opt_state = opt_init(params)\n",
    "\n",
    "    # training loop\n",
    "    for i in range(1, epochs + 1):\n",
    "        # TODO: shuffle in-place to reduce memory allocations (first, copy data)\n",
    "        x, _ = shuffle_perm(x) if shuffle else (x, None)\n",
    "        batches = batchify(x, batch_size)\n",
    "        for batch in batches:\n",
    "            opt_state = update(i, opt_state, batch)\n",
    "\n",
    "        # console logging\n",
    "        if i % log_every == 0:\n",
    "            params = get_params(opt_state)\n",
    "            logs = [log(params, i) for log in loggers]\n",
    "            print(f\"Epoch {i}\", end=\" \")\n",
    "            for log in logs:\n",
    "                print('[%s %.2f]' % (log), end=\" \")\n",
    "            print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "\n",
    "See it in action!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1 [Loglikelihood -1432.02] [Time 42.70] \n",
      "Epoch 2 [Loglikelihood -1393.83] [Time 40.14] \n",
      "Epoch 3 [Loglikelihood -1383.73] [Time 40.05] \n",
      "Epoch 4 [Loglikelihood -1379.98] [Time 40.33] \n",
      "Epoch 5 [Loglikelihood -1378.05] [Time 40.41] \n",
      "Epoch 6 [Loglikelihood -1377.01] [Time 40.33] \n",
      "Epoch 7 [Loglikelihood -1376.31] [Time 40.68] \n",
      "Epoch 8 [Loglikelihood -1375.77] [Time 40.67] \n",
      "Epoch 9 [Loglikelihood -1375.43] [Time 40.40] \n",
      "Epoch 10 [Loglikelihood -1375.32] [Time 40.58] \n"
     ]
    }
   ],
   "source": [
    "train(params, x, gradient,\n",
    "      epochs     = n_epochs,\n",
    "      lr         = lr, \n",
    "      batch_size = batch_size,\n",
    "      loggers    = [log_loss, timer()],\n",
    "      log_every  = 1\n",
    "     )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZwAAAEOCAYAAAC976FxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de3xcdZ3/8dcn96ZtkqaEtqRp03Iv5R7aIuANEN1VC8jFG4KC9Ya6uq6i/nSL6++3gq6siqsiqIiXLaIoLAIri7u4Oi2kcmlLuTQpkJSWptM2bdpMmsx8fn+ck3YaJtdO5kyS9/PxmEfOnPM9M58ZSt75fr/nYu6OiIjIaCuIugAREZkYFDgiIpITChwREckJBY6IiOSEAkdERHKiKOoC8tVhhx3m9fX1UZchIjKmrF69epu712TapsDpR319PY2NjVGXISIyppjZi/1t05CaiIjkhAJHRERyQoEjIiI5ocAREZGcUOCIiEhOKHBERCQnFDgiIpITOg9HRCQiqZST6EmS6E6R6E7Sk3SS7iRTTir8mUw57rxqfSrVty0Hb/cD61/d9uD9Un7g9VIp59zjZ3ByXVXWP68CR0QEcHf2JVMkulN0dYch0JMk0Z2kqycIhN5gSHQnSfT0tktb35PeJkVXT//bu7pT7Eumov7YGR1eUabAERHpqyeZoqOrh92J4NHR1UNHV/dBz3cnuulI9LC7qyf4ub9dsG1PV5KuniSpEd6PssCgrLgweBQVUFZcSGlxIWXFBZQVFXLYlKID24sLKC06sJy+T1FhAYUFUGBGYYFRaEZBgYXPX72+cP+23nXsX7Z+1hf02X//dgteq6DAsvsfKI0CR0QikUx5GALdwS//MAgOhEJ3nyA58Dw9PDq7k4O+V2GBMaW0iKllRft/1kwtZd5hk5lSVsTkksL9gVAa/vIvSwuMg8KhT2CUFhVSXBj8gpeBKXBE5JCkUs7uRA87O/exY283O/fuY2fvz87ug5Z37O2mPVxu7+xmsDvcmxEERGkRU8uKmVJWxLTyEuZUlzO1LFxXeiBEgkApDn6WHdivrLhAgZAHFDgiAoTB0dVzIDA608Ojmx1799Eertuxt3v/cntn94BDUVPLiqgqL6ZqUglV5cXMqS6nalIx08qLqZhUfHBwlBVREYbGlLIiyosLR3WIR3JLgSMyjiVTTtvuLrbsSrClPcGW9k627Opi665E2OPYR3tauAwYHKVFVJYXU1VezLTyEmZPm7R/uXJSMVXlJUwLt1dOKtkfKMWFOvtCAgockTEq0Z3klV0JNrcn9v/c0vsIA6ato4tknxQpLjQOn1rGtMlBr+OIqklBUIQ9kKryEqomFR9YLi+mUsEhWaDAEckz7s6uzh627Eqwub0zc6jsSrBzb/er9p1SWsTMyjJmVpRx9tGHMbOijBmVZcyqKAvWV5ZRXV6iYSqJhAJHJIeSKWdbRxdb2g8OkVd2HdwzyXTk1WFTSphZWcbsaZM4fe40ZlWWMaOijFmVk5hZWcqMijKmlhVH8KlEhkaBIzJKunqSPLelgzWb2lmzqZ21m9p5dsvuV53s1zvENauyjAVHVPDG4w5PC5Pg5+EVpZQWFUb0SUSyQ4EjkgVdPUme3bJ7f7CsCcOlOxnMn1SUFXHi7EquOqueuupyZqaFyfTJGuKSiUGBIzJMie5Xh8tzrxwIl8pJxZxYW8nVZ8/nxNpKTqytpK56ks4DkQlPgSMygER3kmd6w6X1QLj0hEd+VZUH4XLNOQfCZfY0hYtIJnkZOGZ2KbAcOB5Y5O6N4fpFwC29zYDl7n53uO1dwBcAB14G3uvu28ysFPgpcDoQBy539xdy92lkrEh0J1m/edf+XsuaTbt4Pi1cppUXs7C2kmXHBuGyUOEiMix5GTjAWuBi4AcZ1je4e4+ZzQKeNLN7w23fAhaEIXMjcC1BaF0N7HD3o8zsncANwOW5+BCSvxLdSZ7uDZew5/L81o7956xUTy5hYW0lbzyuZn+41FYpXEQORV4GjruvB171P7e77017WkbQm4Ggt2PAZDOLAxXAhnDbUoLgAbgLuNnMzH2wqzjJeOHurHt5F6tf3LF/3iU9XKaH4XLe8TNYWFvJibMrOaKyTOEikmV5GTgDMbPFwI+AucAV7t4Trv8IsAbYAzwPfCzcpRZoAQh7Ru3AdGBbjkuXHIt3dHH345u4s7GF517pAIJzWRbWVnL+gjBcaiuZpXARyYnIAsfMHgJmZtj0RXf/XX/7ufsq4AQzOx643czuB5LAR4BTgWbgO8Dnga8S9Hxe9TL91LQMWAYwZ86coX8YyRvJlPPI823c+VgLD61/he6kc0pdFf988Ym8/tgaZlYoXESiElnguPt5h7j/ejPbAywkDBV3bwIwszuB68KmrUAd0GpmRUAlsL2f17yF8KCEhoYGDbmNIS/F9/Kr1S3ctbqVze0JqieX8L4z67n8jDqOmTE16vJEhDE2pGZm84CWcGhsLnAs8AJQAiwwsxp3bwPOB9aHu90DXAnEgEuAhzV/Mz4kupM8sHYLdza28JemOAUGrz2mhi+/dQHnHj+DkiJdbFIkn+Rl4JjZRQTDYjXAfWb2hLtfAJwNXGdm3UAK+Ki7bwv3uR54JNz2InBV+HK3AXeY2QaCns07c/phJOvWbmpnxWMt/O6JTexK9FBXPYm/P/8YLmmYzazKSVGXJyL9MP2xn1lDQ4M3NjZGXYaE2vd289snNrHisRae3ryLkqIC3rJwJpc31LFk/nRdGkYkT5jZandvyLQtL3s4IhDcgfIvTXFWNLbw4Lot7OtJsbC2gn9aegJvP7mWynJdGVlkLFHgSN55eWcnv2ps5VerW2jd0UlFWRHvOqOOy86o44QjKqMuT0RGSIEjeaGrJ8lDT29lRWMLf3q+DXc466jp/MMFx3LBCTMpK9al+UXGOgWOROrZLbtZ8VgLdz/eyo693cyqLOPjbziKSxvqqKsuj7o8EckiBY7k3O5EN/c+uZkVjS082bKT4kLj/AUzuKyhjnOOrqFQBwCIjEsKHMkJd+fRjdtZ0djC79dsJtGd4tgZU/nSWxdw0am1VE8uibpEERllChwZVds6urizsYVfNbaycdseppQWcdGps7n8jDpOnl2py8yITCAKHBk1TW0dXP6DGNs69rGovpqPveEo/ubEmZSX6J+dyESk//NlVLwU38t7frgKd7jvE2frcGYRUeBI9r28s5N337qSzu4k/75sCcfPqoi6JBHJA7q6oWTV1t0J3nvrKtr3dnPH1YsUNiKyn3o4kjXb9+zjilsfZXN7gjuuXsRJs6uiLklE8ogCR7KivbOb9/1oFRvje/jJVWfQUF8ddUkikmc0pCaHrKOrh6t+/CjPbtnND957Oq856rCoSxKRPKQejhySRHeSa25/jKda2/nuu0/lDccdHnVJIpKn1MOREevqSfKhO1azauN2/uXSk3nzwllRlyQieUyBIyPSnUzx8V88zv8818bXLj6RC0+tjbokEclzChwZtmTK+fSdT/KfT7/C8rct4PIz5kRdkoiMAQocGZZUyvncr5/i3idf5rq3HMdVZ82LuiQRGSMUODJk7s4/3rOOu1a38slzj+bDrzsy6pJEZAxR4MiQuDv/fP8z3LHyRZa9dj5/d97RUZckImOMAkeG5F8fep5bHmnmiiVz+fxbjtNtBURk2BQ4Mqjv/XcT3/qv57n09Nlc//YTFDYiMiIKHBnQT/68kRseeIa3nXwEX3vHSRTo9s8iMkIKHOnXisdeYvm9T/OmBTP45mUnU6iwEZFDoMCRjH73xCau+80aXndMDd9596kUF+qfiogcmrz8LWJml5rZOjNLmVlD2vpFZvZE+HjSzC5K23a5mT0V7ndj2vpSM1thZhvMbJWZ1ef204w9D6zdzKfvfJIl86bzgytOp7SoMOqSRGQcyMvAAdYCFwOPZFjf4O6nAG8GfmBmRWY2Hfg6cK67nwDMMLNzw32uBna4+1HATcANOfkEY9Qfn9nKx3/5OCfPruTWKxsoK1bYiEh25GXguPt6d382w/q97t4TPi0DPFyeDzzn7m3h84eAd4TLS4Hbw+W7gHNNh1ll9OcN2/jQz1Zz7Myp/Pj9i5hcqouJi0j25GXgDMTMFpvZOmAN8OEwgDYAx5lZvZkVARcCdeEutUALQNi2HZjez2svM7NGM2tsa2vL1GTcanxhO9fc3si86ZO54wOLqZxUHHVJIjLORBY4ZvaQma3N8Fg60H7uviocNjsD+LyZlbn7DuAjwArgT8ALQG9PKFNvxjOsw91vcfcGd2+oqakZ6Ucbc55q3cn7f/wYsyrLuOOaRUybXBJ1SSIyDkU2ZuLu5x3i/uvNbA+wEGh093uBeyHoqQDJsGkrQW+nNez9VALbD+W9x5P1m3dxxW2PUjW5mJ9/cDGHTy2LuiQRGafG1JCamc0LQwMzmwscS9CbwcwOD39OAz4K3Brudg9wZbh8CfCwu2fs4Uw0G7Z2cMVtq5hUXMgvrlnCrMpJUZckIuNYXs4Kh4c7fweoAe4zsyfc/QLgbOA6M+sGUsBH3X1buNu3zOzkcPkr7v5cuHwbcIeZbSDo2bwzZx8kj70Y38N7bl0JwM8/uJi66vKIKxKR8c70x35mDQ0N3tjYGHUZo+LlnZ1c+v0Ye/b18O/LlnDczIqoSxKRccLMVrt7Q6ZtY2pITQ7d1t0J3nPrKnZ1dnPHBxYrbEQkZ/JySE1Gx/Y9+3jvrat4ZVeCO65exImzK6MuSUQmEPVwJoj2zm6uuG0VL8b3cuuVDZw+tzrqkkRkglHgTAAdXT1c9eNHee6V3fzgitN5zZGHRV2SiExAGlIb5zr3Jbn6J4/xVGs7//ae03j9sYdHXZKITFDq4YxjXT1Jlt3RyKMvbOebl53MBSfMjLokEZnAFDjjVHcyxbW/eJw/Pb+NGy4+iaWn1EZdkohMcAqcceqbf3iOPzz9Cl9ZegKXnVE3+A4iIqNMgTNOPbB2C68/tob3nVkfdSkiIoACZ1za0p5g47Y9nH2UjkYTkfyhwBmHYs3B5eWWzM942x8RkUgocMahWFOcyknFLJily9aISP4Y8nk4ZjYbOAdYAPSO1WwDngb+191bsl+ejESsOc7iedUUFOhO2iKSPwYMHDMrBd4HXAM0kPnumQBuZquBHwJ3uHsiq1XKkLXu2EvL9k4+cNa8qEsRETlIv0NqZvYB4Hng34B24LPAa4HZwCRgcrj8OuA6YFfY9vlwX4lArCkOwJlHav5GRPLLQD2cG4FvA9939639tOkEXgb+BHw9vOvmR4AbgB9ls1AZmlhznOrJJRxz+NSoSxEROchAgTPX3fcM58XCYLrezL5xaGXJSLg7K5viLJmv+RsRyT/9DqkNN2yyta+M3Evb9/Jye4IzdTi0iOShIR8WbWZvGEKbaw+tHDkUmr8RkXw2nPNwHjKzG82suO8GM5tpZg8C38peaTJcseY4NVNLObJmStSliIi8ynAC5wfAZ4BHzez43pVm9g5gDfAa4MPZLU+Gyt2JNcVZMn86Zpq/EZH8M+TAcfePAm8FZgGrzexTZvYT4FdAE3Cqu/9wVKqUQTVv28PW3V2avxGRvDWsO366++/N7ATg90DvkWhfB77g7slsFydDp/kbEcl3w7qWmpkVAp8ETgM2ASngovC5RCjWHGdmRRn108ujLkVEJKPhHKV2NPAX4P8AtwHHAq8n6CX92cy+bGa6GGgE3J1VzXHOPFLzNyKSv4YTEI8D9cBSd/+wu3e6+5+Bk4CfA8uB/816hTKo57d2sK1jn+ZvRCSvDSdw/hs40d3vTV/p7h3u/n7gEuDobBRlZpea2TozS5lZQ4btc8ysw8w+k7budDNbY2YbzOzbFv6pb2alZrYiXL/KzOqzUWM+0fyNiIwFwzlK7a0DXFMNd/8NcGJWqoK1wMXAI/1svwm4v8+67wHLCELvaODN4fqrgR3uflS43w1ZqjFvxJri1FZNoq5a8zcikr+yOufi7luy9Drr3f3ZTNvM7EKgGViXtm4WUOHuMXd34KfAheHmpcDt4fJdwLk2jiY6Uiln5ca4ejcikvcGuj3BF8xs2JccNrOpZvbFQyur39eeDHwOuL7PplqgNe15a7iud1sLgLv3ENxqIeNvZzNbZmaNZtbY1taWzdJHzTNbdrNzb7fmb0Qk7w3Uw3k/8KKZfcvMFg/2QmZ2ppl9G3gRuHII7R8ys7UZHksH2O164CZ37+j7chna+hC2HbzS/RZ3b3D3hpqamsE+Ql6INWv+RkTGhoFO/Dwe+BDwD8C1ZrYbWE0wnLUjbDMNOBI4HZhC0LP4EsFlcAbk7ueNoN7FwCVmdiNQBaTMLAH8muBmcL1mE9ynh7CmOqDVzIqASmD7CN47L8Wa4sydXs4RVZOiLkVEZED9Bk44/PRdM/s34ALgHcDZBOfe9PYaHHgOWAH8BngwnEMZFe5+Tu+ymS0HOtz95vD5bjNbAqwiuC32d8Km9xD0uGIER9I9PJo15lIy5azaGOdvT5wVdSkiIoMa9NI24S/nB8IH4cmdveM3cXdPZbsoM7uIIDBqgPvM7Al3v2CQ3T4C/ITg9tf3c+AottuAO8xsA0HP5p3ZrjcqT7+8i92JHg2niciYMKxrqQGEATOqM+rufjdw9yBtlvd53ggszNAuAVyazfryRax5G4AOGBCRMWHQwDGz44CPE8zVbAN+4e6/H+3CZHCxpjjzayZzeEVZ1KWIiAxqwMAxs5OBPxEcENDrXWZ2rbt/b1QrkwF1J1M8unE7F55aO3hjEZE8MNiJn18mOEDgYmAqwVWhnwb+KbxytERkzaZ29uxLav5GRMaMwQLnbOD77v5bd9/j7k8AnyI4HPr4gXeV0dR7/bQlmr8RkTFisMCZTnBds3RrCXo9+k0XoZXNcY6ZMYXDppRGXYqIyJAMFjgFQHefdb3PNaQWkX09KRpf2KGj00RkTBnKYdELzOyNac8rwp+nZLoGprs/nI3CpH9Ptu6ks1vzNyIytgwlcD4fPvr6Bgdfk8zC5+r5jLJYUxwzWDxPgSMiY8dggfP+nFQhwxJrinPczAqmTS6JuhQRkSEbMHDc/faBtkvuJbqTrH5pB+9dPDfqUkREhiWrN2CT0ff4SzvZ15PS/I2IjDkKnDEm1hynwGDRvOqoSxERGRYFzhizsinOwtpKKicVR12KiMiwKHDGkM59SR5v0fk3IjI2KXDGkNUv7qA76SzR/I2IjEEKnDEk1ryNwgLjjHrN34jI2KPAGUNiTXFOml3JlNJh3zdPRCRyQ/7NZWaDXbLGgQTQAvwX8Bt3Tx5CbZJmT1cPT7W2s+y186MuRURkRIbzp3IBUEtw58+dwEaCy9nUA1XABqAdWAx8EFhtZm9y953ZLHiieuyF7fSkXOffiMiYNZwhtc8Q3AfnaqDG3U9399OAGoKAqQY+DBwOfIjgZm1fzW65E1esOU5xodEwV/M3IjI2DSdw/gX4mbv/OH2ozN2T7n4b8DPgX9w95e63Aj8F3p7dcieulU1xTqmrYlKJro0qImPTcAKnAXhmgO3Phm16PQrMGElRcrBdiW7WbGrX+TciMqYNJ3B2Am8aYPubgF1pzyuAHSMpSg722MbtpBydfyMiY9pwAuenwIVmdoeZnWFmVWZWGS7/jGD4LP3q0m8C1mSz2Ikq1hSnpKiA0+ZMi7oUEZERG85Ral8iGCK7Cnh3n21GEEhfAjCzMuA/gVWHXqLEmuOcNqeKsmLN34jI2DXkwHH3HuADZvZN4K3AXIKg2Qj83t3XpLVNADdmudYJaefefTy9eRd/d+4xUZciInJIhn2lAXdf6+5fc/ePuPuH3f2G9LDJBjO71MzWmVnKzBoybJ9jZh1m9pm0df/XzFrMrKNP21IzW2FmG8xslZnVZ7PW0bZq43bc0fk3IjLmDTtwzKzQzJaY2bvM7J3hcrbHetYCFwOP9LP9JuD+PuvuBRZlaHs1sMPdjwr3uyFbReZCrClOWXEBJ9dVRl2KiMghGdZFuczsEuBbwEyC4TQILmmz2cw+6e6/zkZR7r4+fL9MNVwINAN7+uyzsp99lgLLw+W7gJvNzNzds1HraFvZHKdhbjWlRZq/EZGxbcg9HDN7M7AC6Aa+CFwIXBQudwMrzOyC0SgyrYbJwOeA64exWy3B9d1656HagYzjU2a2zMwazayxra3tUMs9ZPGOLp7ZslvDaSIyLgynh/N/gKeBs9w9/Xyb35nZd4EYQfg8OJQXM7OHCHpKfX3R3X/Xz27XAze5e0em3k9/b5VhXcbejbvfAtwC0NDQEHkPaNXG7QAs0QmfIjIODCdwTgG+3CdsAHD33Wb2I+ArQ30xdz9vGO/dazFwiZndSHDB0JSZJdz95gH2aQXqgFYzKwIqge0jeO+cizXFKS8p5KTZmr8RkbFvOIGTBMoG2F4Wthk17n5O77KZLQc6BgkbgHuAKwl6YJcAD4+V+ZtYc5wz6qspLtRti0Rk7BvOb7JVwIfNrLbvBjM7AvgIsDIbRZnZRWbWCpwJ3Gdmgw7TmdmN4T7lZtYaBhLAbcB0M9sAfBq4Lhs1jratuxNs2Nqh+RsRGTeG08P5MvBH4Bkz+wXBhTwdWAC8M3ytL2WjKHe/G7h7kDbL+zz/LPDZDO0SwKXZqCuXVjYHo366YKeIjBfDudLASjM7j+Bclg/22dwIfMrdH81mcRNZrCnO1NIiTjiiIupSRESyYljn4bj7n4FFZnY4MC9c/YK7v5L1yia4lc1xFs2rpkjzNyIyTgwrcHq5+1Zga5ZrkdCW9gQbt+3hPYvnRF2KiEjW9Bs4Zjai33bu/tLIyxGAWPM2QOffiMj4MlAP5wX6OUFyELoGyyGKNcWpnFTMglmavxGR8WOgwPkAIwscOUSx5jiL51VTUDDkqymIiOS9fgPH3X+Swzok1LpjLy3bO/nAWfMGbywiMoboEKg8E2uKA7r/jYiMPwqcPBNrjlM9uYRjDp8adSkiIlmlwMkj7s7KpjhL5mv+RkTGHwVOHnlp+15ebk/ocjYiMi4pcPKI5m9EZDxT4OSRvzTFqZlaypE1U6IuRUQk6xQ4ecLdiTXHWTJ/OsO4m6mIyJihwMkTTW17aNvdpfkbERm3FDh5Itas+RsRGd8UOHliZVOcmRVl1E8vj7oUEZFRocDJA+7OyuY4Zx6p+RsRGb8UOHnguVc6iO/Zp/kbERnXFDh5INYU3P9G8zciMp4pcPJArDnO7GmTqKvW/I2IjF8KnIilUs6qjds1nCYi454CJ2Lrt+xi595uDaeJyLinwImYrp8mIhOFAidiK5vj1E8vZ1blpKhLEREZVQqcCCV752/UuxGRCSAvA8fMLjWzdWaWMrOGDNvnmFmHmX0mfF5uZveZ2TPhfl9La1tqZivMbIOZrTKz+tx9koGte7md3YkeluiAARGZAPIycIC1wMXAI/1svwm4v8+6b7j7ccCpwFlm9pZw/dXADnc/KtzvhlGod0T2z98ocERkAsjLwHH39e7+bKZtZnYh0AysS2u/193/GC7vA/4KzA43LwVuD5fvAs61PLl+TKw5zpE1kzm8oizqUkRERl1eBk5/zGwy8Dng+gHaVAFvA/4rXFULtAC4ew/QDmTsUpjZMjNrNLPGtra2bJb+Kt3JFI9p/kZEJpDIAsfMHjKztRkeSwfY7XrgJnfv6Oc1i4BfAt929+be1Rmaeqb93f0Wd29w94aamprhfJxhW7OpnT37kpw5/7BRfR8RkXxRFNUbu/t5I9htMXCJmd0IVAEpM0u4+83h9luA5939X9P2aQXqgNYwkCqB7YdQelb0zt8smV8dcSUiIrkRWeCMhLuf07tsZsuBjt6wMbOvEoTJNX12uwe4EogBlwAPu3vGHk4urWyOc+yMqUyfUhp1KSIiOZGXczhmdpGZtQJnAveZ2YODtJ8NfBFYAPzVzJ4ws97guQ2YbmYbgE8D141i6UOyrydF4ws7NH8jIhNKXvZw3P1u4O5B2ixPW24l81wN7p4ALs1mfYfqydaddHYndf6NiEwoednDGe9iTXHMNH8jIhOLAicCsaY4x8+soKq8JOpSRERyRoGTY4nuJKtf0vyNiEw8Cpwce/ylnezrSelyNiIy4ShwcizWHKfAYJHmb0RkglHg5NjKpjgLayupKCuOuhQRkZxS4ORQ574kj7fs0HCaiExICpwcWv3iDrqTzhIdMCAiE5ACJ4dizdsoLDDOqNf8jYhMPAqcHIo1xTlpdiVTSvPyAg8iIqNKgZMje7p6eKq1XfM3IjJhKXBy5LEXttOTcp3wKSITlgInR2LNcYoLjYa5mr8RkYlJgZMjK5vinFJXxaSSwqhLERGJhAInB3YlulmzSfM3IjKxKXBy4LGN20k5Ov9GRCY0BU4OxJrilBQVcNqcaVGXIiISGQVODsSa45w2p4qyYs3fiMjEpcAZZTv37uPpzbs4c/5hUZciIhIpBc4oW7VxO+7o/BsRmfAUOKMs1hSnrLiAk+sqoy5FRCRSCpxRtrI5TsPcakqLNH8jIhObAmcUxTu6eGbLbg2niYigwBlVK5u3A7BEJ3yKiChwRlOseRvlJYWcNFvzNyIiCpxRFGuKc0Z9NcWF+ppFRPLyN6GZXWpm68wsZWYNGbbPMbMOM/tM2roHzOzJcL/vm1lhuL7UzFaY2QYzW2Vm9bn4DFt3JWhq26P5GxGRUF4GDrAWuBh4pJ/tNwH391l3mbufDCwEaoBLw/VXAzvc/ahwvxuyX+6rxZrjALpgp4hIKC8Dx93Xu/uzmbaZ2YVAM7Cuzz67wsUioATw8PlS4PZw+S7gXDOzrBfdx8rmOFNLizjhiIrRfisRkTEhLwOnP2Y2GfgccH0/2x8EtgK7CcIFoBZoAXD3HqAdyNjtMLNlZtZoZo1tbW2HVGusKc7i+dUUaf5GRASIMHDM7CEzW5vhsXSA3a4HbnL3jkwb3f0CYBZQCryx960yNe1n/1vcvcHdG2pqaobxaQ62ub2TF+J7dTi0iEiaoqje2N3PG8Fui4FLzOxGoApImVnC3W9Oe92Emd1DMJT2B6AVqANazawIqAS2H/IHGECsKZy/0QEDIiL7RRY4I+Hu5/Qum9lyoI4t1CQAAAeKSURBVMPdbzazKcBUd98chsrfAH8Km94DXAnEgEuAh909Yw8nW2JNcarKizl+puZvRER65eUEg5ldZGatwJnAfeHczEAmA/eY2VPAkwTzON8Pt90GTDezDcCngetGqez9Ys1xFs+rpqBg1I9NEBEZM/Kyh+PudwN3D9JmedryK8AZ/bRLcOAQ6VHXsn0vrTs6uebsebl6SxGRMSEvezhj2f7zb47UDddERNIpcLJsWnkJ5y+YwTEzpkRdiohIXsnLIbWx7PwFMzh/wYyoyxARyTvq4YiISE4ocEREJCcUOCIikhMKHBERyQkFjoiI5IQCR0REckKBIyIiOaHAERGRnLBRvnDymGVmbcCLI9z9MGBbFssZ6/R9HEzfxwH6Lg42Hr6Pue6e8YZiCpxRYGaN7t4QdR35Qt/HwfR9HKDv4mDj/fvQkJqIiOSEAkdERHJCgTM6bom6gDyj7+Ng+j4O0HdxsHH9fWgOR0REckI9HBERyQkFjoiI5IQCJ8vM7M1m9qyZbTCz66KuJypmVmdmfzSz9Wa2zsw+GXVN+cDMCs3scTP7j6hriZqZVZnZXWb2TPjv5Myoa4qKmX0q/P9krZn90szKoq5pNChwssjMCoHvAm8BFgDvMrMF0VYVmR7g7939eGAJ8LEJ/F2k+ySwPuoi8sS3gAfc/TjgZCbo92JmtcAngAZ3XwgUAu+MtqrRocDJrkXABndvdvd9wL8DSyOuKRLuvtnd/xou7yb4ZVIbbVXRMrPZwN8Ct0ZdS9TMrAJ4LXAbgLvvc/ed0VYVqSJgkpkVAeXAyxHXMyoUONlVC7SkPW9lgv+SBTCzeuBUYFW0lUTuX4HPAqmoC8kD84E24MfhEOOtZjY56qKi4O6bgG8ALwGbgXZ3/89oqxodCpzssgzrJvRx52Y2Bfg18HfuvivqeqJiZm8Ftrr76qhryRNFwGnA99z9VGAPMCHnPM1sGsFIyDzgCGCymb032qpGhwInu1qBurTnsxmnXeOhMLNigrD5ubv/Jup6InYW8HYze4FgqPWNZvazaEuKVCvQ6u69vd67CAJoIjoP2Ojube7eDfwGeE3ENY0KBU52PQYcbWbzzKyEYOLvnohrioSZGcH4/Hp3/2bU9UTN3T/v7rPdvZ7g38XD7j4u/4odCnffArSY2bHhqnOBpyMsKUovAUvMrDz8/+ZcxukBFEVRFzCeuHuPmV0LPEhwpMmP3H1dxGVF5SzgCmCNmT0RrvuCu/8+wpokv3wc+Hn4x1kz8P6I64mEu68ys7uAvxIc3fk44/QSN7q0jYiI5ISG1EREJCcUOCIikhMKHBERyQkFjoiI5IQCR0REckKBIzLOmVm9mbmZfTXqWmRiU+CIHAIze334y7y/x0S+moDIQXTip0h23Ab8d4b1zTmuQyRvKXBEsmOlu6s3IzIADamJ5ICZLQ+H2E41s1vMbJuZ7TGz/zCz+Rna15rZT8zsFTPrMrOnw7tCvuqK5GZ2opndaWZbzSxhZs3he0zN0PYdZrYmbPecmV2Woc2HzOxJM+sws/aw/fXZ+zZkolIPRyQ7ppjZYRnW73b3rrTnPwZ2AV8huFfSx4FHzOwkd98OYGbTgb8AMwnuINsMvBX4JnAkcG3vi5nZOQTX7usEfhi2nQ1cDEwHdqe995sJrlf2fWAnsAz4pZk94e7Pha/Xu/134U+AY4HXDf8rEenD3fXQQ48RPoDXE9zzqL/HVWG75eHzPwNFafu/LVx/Q9q6G8N170hbZwSXrXfgxHBdAfAcsB2oy1Bb77US68P9OoDZadtnAV3AjWnr7gbWRf296jE+HxpSE8mObwLnZ3g82Kfdze7e0/vE3e8FnicInl5vJ7hV+a/T2jnw9fBpb9tTgKOB77p7+p1m0/dJd4+7t6Zt3ww8Q9Br6tUO1JrZkgE/rcgIaEhNJDvWu/tDQ2j3bD/rzkt7Xg/8IUO73vvFzAt/Hh3+fGooBQIvZli3A6hOe/414I1AzMw2Ehx591vg3gwBJjIs6uGI5NZQf2kP1K7vtqG+ZrKf9fsPRHD3ZwjmbC4CHiCYu/kd8ICZFQ7xfUQyUuCI5NZxGdYdC2xMe/4CcHyGdsenbYdgKA7gpGwU1svdO939t+7+UeAo4AbgTQR3ohQZMQWOSG5da2b7h7LN7G0EQ2P/kdbmXuAoM7sorZ0Bn0nbDvAEQeh8zMxq+75RpkOoBxMeIbdfOIzWe8fW6lfvITJ0msMRyY4lZpbIsD7u7venPZ8MPGxmvyI4LPoTwGaCI9N6fQ24jOCQ5d7Dov8WeAvBAQJrANw9ZWYfJBj6etLMeg+LPoLgsOilHOgNDdUfzKyN4Gi6TUAd8DGgjczzSiJDpsARyY6rw0dfq4H0wHk/8FGCw6TLCCblP+Hu23obuHvczF4D/D/gfUAFQZD8PXBT+ou7+/+Ebb9McF7NZIKg+AOwjeH7HvAugnN9KoFXgPuAf3L3+AheT2Q/04EnIqPPzJYD/wgc7e4bIi5HJBKawxERkZxQ4IiISE4ocEREJCc0hyMiIjmhHo6IiOSEAkdERHJCgSMiIjmhwBERkZxQ4IiISE78f7HUAFrwQr0NAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "ax.set_ylabel('log P(x)', fontsize='xx-large')\n",
    "ax.set_xlabel('Epochs', fontsize='xx-large')\n",
    "plt.plot(get_loss_vs_time())\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}