examples/example_produce.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f2aa08b6-0ed3-4078-a387-ce32da492a9f",
   "metadata": {},
   "source": [
    "# Embeddings for expanding vocabularies"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "567e043f-621e-494a-bf75-9f3cd8545064",
   "metadata": {},
   "source": [
    "This notebook demonstrates how to construct embeddings for vocabularies that change incrementally."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a08b2ae-8ede-4883-9e00-e0653cab378d",
   "metadata": {},
   "source": [
    "A vocabulary is a set of elements that are relevant to a subject domain. For a natural language like English, the word 'vocabulary' is typically used to refer to the set of all words in the language. But the concept is also applicable in other settings. For example, a vocabulary may refer to the set of produce stocked in a shop. A finite vocabulary is useful for data management. However, a vocabulary by itself does not assign meaning to any of its elements, nor does it convey relationships (similarities and dissimilarities) between any of the elements."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8354224d-1335-45ba-9eaa-aa43c7bdbcdd",
   "metadata": {},
   "source": [
    "An embedding is a mapping of vocabulary elements into a low-dimensional space. The position of the elements in the low-dimensional space is intended to convey semantic meaning. For example, consider mapping elements \"apple\" and \"banana\" to nearby positions in a two-dimensional space, and mapping an element \"carrot\" to a position that is further away. These mappings can convey that \"apple\" and \"banana\" are more similar to each other (they are both fruits) than to \"carrot\" (a vegetable). The coordinates can also be used in calculations such as clustering, regressions, and other machine-learning models."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd80c9e7-9e10-40bd-9161-582f80d7f014",
   "metadata": {},
   "source": [
    "PyTorch's module `Embedding` implements mappings from integers to coordinates. This module can be trained to create embeddings for a fixed vocabulary, and then apply those mappings in more complex neural networks. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59df8051-5045-455c-a4e4-92b5f3f9e627",
   "metadata": {},
   "source": [
    "In long-running projects, it is possible that an initial vocabulary becomes too restrictive over time. In a natural language like English, it might become necessary to expand the set of words to include new words or phrases. Similarly, a vocabulary of produce may need to grow as a shop adds inventory."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d05f5ebf-b6b0-4633-9dbf-c3798e39c65c",
   "metadata": {},
   "source": [
    "A straightforward approach to respond to an expansion of a vocabulary is to train a new `Embedding` module. However, this is problematic because re-training from scratch is bound to produce coordinates that are incompatible with previous versions. If an embedding is used in any downstream applications, for example in a machine-learning classifier, then those applications would also require re-training, triggering a cascade of updates."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e0a0a1b-dd48-497a-8d15-3f0cfd02537b",
   "metadata": {},
   "source": [
    "To counter this problem, this notebook shows how to constrain training of a new embedding so that the new embedding remains compatible with previous versions. The notebook relies on [pytorch](https://pytorch.org/) and [pytorch-lightning](https://pytorchlightning.ai/) to build embeddings, and on a class `EmbeddingConstraint` to constrain training of new embeddings. We begin by loading the required libraries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7b255d37-9496-43f2-a682-8fee0bde44df",
   "metadata": {},
   "outputs": [],
   "source": [
    "# general purpose packages\n",
    "import os\n",
    "import sys\n",
    "import warnings\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# machine-learning with pytorch and pytorch-lightning\n",
    "import torch\n",
    "from torch import nn\n",
    "from torch.utils.data import Dataset, DataLoader\n",
    "import pytorch_lightning as pl\n",
    "from pytorch_lightning.callbacks.progress import TQDMProgressBar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5ce08f25-a72b-498f-abfb-750ce85a33e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# adjustment to sys,path to allow import from a local directory\n",
    "callbacks_path = os.path.abspath(os.path.join(\"..\", \"callbacks\"))\n",
    "if callbacks_path not in sys.path:\n",
    "    sys.path.append(callbacks_path)\n",
    "\n",
    "# callback used to constrain embeddings\n",
    "from embedding_constraint import EmbeddingConstraint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b81878ff-5c07-4c68-92a3-45e6489f2276",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Global seed set to 123\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "123"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# settings used by pytorch and/or pytorch-lightning\n",
    "device = \"cpu\"\n",
    "n_cpus = os.cpu_count()\n",
    "warnings.filterwarnings('ignore', 'CUDA')\n",
    "\n",
    "# random number generation\n",
    "pl.seed_everything(123)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8dc303a6-fdaa-4d86-8c49-d4d3ce2779a6",
   "metadata": {},
   "source": [
    "## Embedding for an initial vocabulary\n",
    "\n",
    "\n",
    "\n",
    "A small dataset `produce_1.txt` contains 3-tuples of strings. This is a synthetic dataset with strings representing items of produce, and 3-tuples holding permutations of the constituent frutis and vegetables. Let's load this data and preview the first entry."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4a19f636-2144-47aa-89bc-f8e63809d512",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First data row:\t['apple' 'orange' 'banana']\n",
      "Dataset shape:\t(240, 3)\n"
     ]
    }
   ],
   "source": [
    "def read_produce_data(file_path):\n",
    "    with open(file_path, \"rt\") as f:\n",
    "        result = f.readlines()\n",
    "    result = [_.strip().split(\",\") for _ in result]\n",
    "    return np.array(result)    \n",
    "\n",
    "\n",
    "produce_data = read_produce_data(\"produce_1.txt\")\n",
    "print(f\"First data row:\\t{produce_data[0]}\")\n",
    "print(f\"Dataset shape:\\t{produce_data.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b3ed95d-cde2-466c-8392-0c887e8526d8",
   "metadata": {},
   "source": [
    "Scanning the entire dataset, we can construct a vocabulary of all fruits and vegetables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "25236bbe-0960-42a9-8355-5df3068c94a6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary:\t['apple', 'banana', 'broccoli', 'cabbage', 'carrot', 'cauliflower', 'mango', 'orange', 'parsnip', 'pear', 'spinach', 'strawberry']\n",
      "Vocabulary size:\t12\n"
     ]
    }
   ],
   "source": [
    "produce_vocabulary = sorted(set(produce_data.flatten()))\n",
    "print(f\"Vocabulary:\\t{produce_vocabulary}\")\n",
    "print(f\"Vocabulary size:\\t{len(produce_vocabulary)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70d7ede5-33ae-46ba-9a7f-ccad7bc63353",
   "metadata": {},
   "source": [
    "We would like to produce concise numeric representations for these strings in a low-dimensional space. To this end, we can use the 3-tuples to train a neural network. We can provide two of the elements in each 3-tuple to the network, and ask the network to predict the third. \n",
    "\n",
    "It will be convenient to extend the `Dataset` class to present tuple data as tensors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c84f9f94-c79a-4f34-866b-a6d2eaab7cb1",
   "metadata": {},
   "outputs": [],
   "source": [
    "class VocabDataset(Dataset):    \n",
    "\n",
    "    def __init__(self, vocab, data):\n",
    "        \"\"\"\n",
    "        :param vocab: list with all vocabulary elements\n",
    "        :param data: matrix of strings\n",
    "        \"\"\"\n",
    "        super().__init__()\n",
    "        self.vocab = vocab\n",
    "        vocab_map = {k: i for i, k in enumerate(vocab)}\n",
    "        data = np.array(data)\n",
    "        encoded = np.array([vocab_map[_] for _ in data.flatten()])\n",
    "        encoded.shape = data.shape\n",
    "        # use first element in each matrix row as a label\n",
    "        self.label = torch.tensor(encoded[:, 0])\n",
    "        self.data = torch.tensor(encoded[:, 1:]).int()\n",
    "\n",
    "    def __len__(self):\n",
    "        return self.label.shape[0]\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        return self.data[idx], self.label[idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4388873-2ff5-4cb7-861f-e80531e69d54",
   "metadata": {},
   "source": [
    "We can now create a `Dataset` object to present the data as tensors, and a `DataLoader` to fetch the data in batches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "3a6617d2-a4ed-4b4e-a358-e45e829c22e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "produce_dataset = VocabDataset(produce_vocabulary, produce_data)\n",
    "produce_loader = DataLoader(produce_dataset, \n",
    "                            batch_size=4,\n",
    "                            num_workers=n_cpus,\n",
    "                            shuffle=True, \n",
    "                            drop_last=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57ee42a9-d18c-48e1-af61-2c040c71a73c",
   "metadata": {},
   "source": [
    "To train a neural network, we need an `Embedding` component that will convert vocabulary elements into a low-dimensional space. We also need a layer that will attempt to convert a pair of embeddings back to a vocabulary items, i.e. to predict a vocabulary items given a pair of two other items. This approach is implemented in the following pytorch-lightning module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "f7cfca95-946f-4a63-86a8-b1d8852192dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "class VocabModule(pl.LightningModule):    \n",
    "\n",
    "    def __init__(self, vocab_size, embedding_dim=2, context_size=2, lr=0.01):\n",
    "        \"\"\"\n",
    "        :param vocab_size: number of elements in the vocabulary\n",
    "        :param embedding_dim: number of embedding dimensions\n",
    "        :param context_size: number of elements provided as a context for training\n",
    "        :param lr: base learning rate\n",
    "        \"\"\"\n",
    "        super().__init__()\n",
    "        self.save_hyperparameters()\n",
    "        self.lr = lr\n",
    "        self.embedding = nn.Embedding(vocab_size, embedding_dim)        \n",
    "        self.output = nn.Linear(embedding_dim * context_size, vocab_size)\n",
    "        self.criterion = nn.CrossEntropyLoss()\n",
    "\n",
    "    def forward(self, x):\n",
    "        \"\"\"\n",
    "        :param x: data batch of shape (batch_size, context_size)\n",
    "        :return: tensor of shape (batch_size, vocab_size)\n",
    "        \"\"\"\n",
    "        batch_size, context_size = x.shape[0], x.shape[1]        \n",
    "        x = self.embedding(x.int())              \n",
    "        x = x.view(batch_size, context_size*self.hparams.embedding_dim)            \n",
    "        return self.output(x)        \n",
    "\n",
    "    @torch.no_grad()\n",
    "    def embed(self, x):        \n",
    "        return self.embedding(x.int())\n",
    "\n",
    "    def configure_optimizers(self):\n",
    "        optimizer = torch.optim.Adam(self.parameters(), lr=self.hparams.lr)\n",
    "        return [optimizer], []\n",
    "\n",
    "    def training_step(self, batch, batch_idx):\n",
    "        data, labels = batch        \n",
    "        pred = self.forward(data)        \n",
    "        loss = self.criterion(pred, labels)        \n",
    "        self.log(f\"train_loss\", loss)\n",
    "        return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "478547fc-169a-40c0-917f-11993275e162",
   "metadata": {},
   "source": [
    "Training can be carried out using a pytorch-lightning `Trainer`. For simplicity, we train for a fixed number of epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2772f344-5d86-48f4-b9b9-0e21a9d86c63",
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_vocab_embedding(train_loader=None, epochs=10, **kwargs):\n",
    "    model = VocabModule(**kwargs)\n",
    "    trainer = pl.Trainer(gpus=0, \n",
    "                         max_epochs=epochs,\n",
    "                         log_every_n_steps=1,\n",
    "                         callbacks = [],\n",
    "                         gradient_clip_val=5,\n",
    "                         enable_progress_bar=False)    \n",
    "    trainer.fit(model, train_loader)\n",
    "    model.to(device)\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99accd31-74c1-4a6e-b809-b06b2eb358ab",
   "metadata": {},
   "source": [
    "We can finally train a model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f2ab4405-fdc4-4da6-b5da-e14bef23b67d",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: False, used: False\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "HPU available: False, using: 0 HPUs\n",
      "\n",
      "  | Name      | Type             | Params\n",
      "-----------------------------------------------\n",
      "0 | embedding | Embedding        | 24    \n",
      "1 | output    | Linear           | 60    \n",
      "2 | criterion | CrossEntropyLoss | 0     \n",
      "-----------------------------------------------\n",
      "84        Trainable params\n",
      "0         Non-trainable params\n",
      "84        Total params\n",
      "0.000     Total estimated model params size (MB)\n"
     ]
    }
   ],
   "source": [
    "model = train_vocab_embedding(produce_loader,                               \n",
    "                              vocab_size=len(produce_dataset.vocab))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b1b57dd-4fb4-42be-aae2-d83779fcba6e",
   "metadata": {},
   "source": [
    "Once the model is trained, we can apply the `Embedding` component withing the model to extract the low-dimensional representation for each vocabulary item, and visualize the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "ca4313b9-dca6-410f-a600-8f9ee1cb8d6a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>x</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>apple</td>\n",
       "      <td>1.405304</td>\n",
       "      <td>0.635577</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>banana</td>\n",
       "      <td>1.001177</td>\n",
       "      <td>0.466538</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       id         x         y\n",
       "0   apple  1.405304  0.635577\n",
       "1  banana  1.001177  0.466538"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def make_embedding_table(vocab, model):\n",
    "    n = len(vocab)\n",
    "    indexes = torch.tensor(range(n))    \n",
    "    result = model.embedding(indexes).detach().numpy()\n",
    "    result = pd.DataFrame(result)\n",
    "    result.rename({0: \"x\", 1: \"y\"}, axis=1, inplace=True)\n",
    "    result[\"id\"] = vocab    \n",
    "    result = result[[\"id\", \"x\", \"y\"]]\n",
    "    return result\n",
    "\n",
    "embedding = make_embedding_table(produce_vocabulary, model)\n",
    "embedding.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "278577e3-a82a-4f3e-9c87-b1f4d503e5d0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbgAAAG+CAYAAADhkxGuAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAA35ElEQVR4nO3deXhV5bn+8e9jEAEZZBBFREHFMGUnIQEJgowCUtqCI2hBkFOkah2qVtRWWqutVmsHaxVHJtGjokexgMh0DJMSJBDCoALqD0fKICAyJDy/P7KzT8QkbCA7e2dxf65rXdlrr+lZEbl511rvu8zdERERCZrj4l2AiIhILCjgREQkkBRwIiISSAo4EREJJAWciIgEkgJOREQCKWEDzsxqmNl7ZrbCzPLN7PfxrklERKoOS9R+cGZmwInuvsvMjgcWADe5+5I4lyYiIlVAtXgXUBYvSt5d4dnjw1NiprGIiCSchA04ADNLApYB5wCPufu7pawzChgFcOKJJ2a0atWqcosUEZFKt2zZsv+4+8nlrZOwlyhLMrOTgNeAX7r7qrLWy8zM9JycnEqrS0RE4sPMlrl7ZnnrJOxDJiW5+3ZgHtAvzqWIiEgVkbABZ2Ynh1tumFlN4EJgbVyLEhGRKiOR78E1ASaE78MdB7zk7m/GuSYREakiEjbg3H0lkB7vOkREpGpK2EuUIiIiR0MBJyIigaSAExGRQFLAiYhIICngREQkkBRwIiISSAo4EREJJAWciIgEkgJOREQCSQEnIiKBpIATEZFAUsCJiEggKeBERCSQFHAiIhJICjgREQkkBZyIiASSAk5ERAJJASciIoGkgBMRkUBSwImISCAp4EREJJAUcCIiEkgKOBERCSQFnIiIBJICTkREAkkBJyIigaSAExGRQFLAiYhIICngREQkkBRwIiISSAo4EREJJAWciIgEkgJOREQCSQEnIiKBpIATEZFAUsCJiEggKeBERCSQFHAiIhJICjgREQkkBZyIiASSAk5ERAJJASciIoGkgBMRkUBSwImISCAp4EREJJAUcCIiEkgKOBERCSQFnIiIBJICTkREAkkBJyIigaSAExGRQFLAiYhIICngREQkkBRwIiISSAo4EREJJAWciIgEUsIGnJk1M7N5ZrbazPLN7KZ41yQiIlVHtXgXUI4C4FZ3f9/M6gDLzOxtd18d78JERCTxJWwLzt2/cPf3w593AmuApvGtSkREqoqEDbiSzKw5kA68W8qyUWaWY2Y5mzdvrvTaREQkMSV8wJlZbWAqcLO77zh4ubs/6e6Z7p558sknV36BIiKSkBI64MzseIrC7Xl3fzXe9YiISNWRsAFnZgY8A6xx90fiXY+IiFQtCRtwwPnAUKCnmeWGp/7xLkpERKqGhO0m4O4LAIt3HSIiUjUlcgtORETkiCngREQkkBRwIiISSAo4EREJJAWciIgEkgJOREQCSQEnIiKBpIATEZFAUsCJiEggKeBERCSQFHAiIhJICjgREQkkBZyIiASSAk5ERAJJASciIoGkgBMRkUBSwImISCAp4EREJJAUcCIiEkgKOBERCSQFnIiIBJICTkREAkkBJyIigaSAE5EqpbCwMN4lSBWhgBORCmVmjB07lrS0NJKTk5k6dWpk2bvvvkuPHj3IyMggIyODf//73wAUFBTQt29fMjMzadu2LSNGjGDfvn0AjB8/nt69ezNo0CDatWtHXl5eXM5Lqp5q8S5ARIInKSmJ3Nxc1q1bR+fOnenatSvVq1dn9OjRTJ8+nSZNmvDFF1/QoUMHVq1aRb169ZgyZQoNGzbE3bn66qt59tlnGT16NABLlixhxYoVnH322XE+M6lKFHAiUuFGjhwJQHJyMu3bt2fJkiVUq1aNjRs3ctFFF0XWMzM++ugj0tPTefjhh5kxYwaFhYVs27aNWrVqRdbr0qWLwk0OmwJORCqFuxMKhXjnnXd+sGzSpEksWLCA7Oxs6tSpwx//+Ec++OCDyPLatWtXZqkSELoHJyIV7rnnngPgww8/ZPny5XTq1InOnTvz4YcfMm/evMh6S5cuxd3Zvn07jRo1ok6dOnzzzTdMmTIlXqVLgKgFJyIVrqCggPT0dHbv3s24ceNo3LgxAG+88Qa33347N998M/v27eOss85i2rRpDBs2jNdff51WrVrRuHFjunbtynfffRfns5Cqztw93jVUmMzMTM/JyYl3GSLHNDNj586duqwoMWVmy9w9s7x1dIlSREQCSZcoRaRCBemqkFRtasGJiEggKeBERCSQFHAiIhJICjgREQkkBZyIiASSAk5ERAJJASciIoGkgBMRkUBSwImISCAp4EREJJAUcCIiEkgKOBERCSQFnIiIBJICTkREAkkBJyIigaSAExGRQFLAiYhIICngREQkkBRwIiISSAo4EREJJAWciIgEkgJOREQCSQEnIiKBlNABZ2bPmtnXZrYq3rWIiEjVktABB4wH+sW7CBERqXoSOuDc/R1ga7zrEBGRqiehAy4aZjbKzHLMLGfz5s3xLkdERBJElQ84d3/S3TPdPfPkk0+OdzkiIpIgqnzAiYiIlEYBJyIigZTQAWdmLwCLgWQz22RmI+Ndk4iIVA3V4l1Aedx9SLxrEBGRqumQLTgzO76U7xrFphwREZGKUWbAmVkPM9sEfGFms8yseYnFs2JemYiIyFEorwX3Z6CvuzcCngTeNrNO4WUW88pERESOQnn34Kq7ez6Au79iZmuAV83sDsArpToREZEjVF7A7TezU939SwB3zzezXsCbwNmVUp2IiMgRKu8S5RjglJJfuPsmoBvwQCyLEhEROVpltuDcfXYZ338D3B+zikRERCpAQnf0FhEROVIKOBERCSQFnIiIBNIhh+oys3OB24EzS67v7j1jWJeIiMhRiaYF9zLwPvAbioKueBIROWbNnz+fzMzMeJch5YhmsOUCd3885pWIiIhUoGhacNPM7Doza2JmDYqnmFcmIlLBrrrqKjIzM0lJSWHQoEFs27aN+fPnk5qayrBhw2jbti0dO3Zk9erVAOUuO9j06dM5//zzycjIICsriyVLllTmqUlp3L3cCdhYyrThUNvFY8rIyHARkbJs3rw58vnuu+/2O+64w+fNm+eAz58/393dx48f78V/lxxqWfHnjz76yDt16uTffPONu7uvWrXKmzVrVqG1A37fffd5Zmamt2jRwmfPnu1jxozxtLQ0b9u2ra9evdrd3b/44gvv3r27t2/f3tu0aeO33357ZB9jx471wYMH+0UXXeTJycnev39///bbb93dffv27X7xxRd7cnKy9+zZ04cOHeq33nqru7vv3LnThw8f7m3btvW2bdv6gw8+WKHndiSAHD9EJhyyBefuLUqZzopZ4oqIxMjEiRPJyMggJSWFKVOmkJubC8A555xDt27dABg6dCh5eXns2LHjkMuKvfXWW6xfv54LLriAtLQ0rrrqKgoKCvjqq68qtP6TTjqJpUuX8uCDD/LTn/6U888/n+XLlzNs2DDuv//+yDrTpk1j2bJl5ObmkpOTw8yZMyP7yMnJYcqUKaxZs4b9+/fz/PPPA3DvvfdSv3591q5dy8svv0x2dnZkmz/84Q8cOHCAvLw8Fi1axIQJE5gxY0aFnlssRPU+ODO70cxeCU83lPaOOBGRRJadnc3jjz/OzJkzycvL47777mPPnj0Vsm93p1+/fuTm5kamzz//nFNOOeXQGx+GK664AoD27dtjZgwYMACAjIwMPvroIwAKCwu5/fbbSU1NJSMjg1WrVkWCHKBv376cdNJJmBnnnXce69evB2DevHmMGDECgAYNGjBw4MDINrNnz+bnP/85ZkbdunUZMmQIs2eXOthVQonmHtzjQAbwr/CUEf5ORKTK2L59O/Xq1aNhw4bs3buXZ599NrJs/fr1kRbLlClTSElJoW7duodcVqxPnz7MnDmT/Pz8yHdLly6t8HOoUaMGAElJSZxwwgmR75OSkigoKADgkUceYdu2bbz77rusXLmSgQMHfi/Ii/dx8HZBFE3AdXD3q919bngaAXSIdWEiIhWpX79+nH322Zx77rl069aN9u3bR5alpKTw9NNP065dO/7xj38wceLEqJYVa9myJZMnT2bkyJGkpqbSunVrxo0bVynndbDt27fTpEkTatSowWeffcbrr78e1Xbdu3ePnNv27du/t13v3r155plncHd27tzJiy++yIUXXhiT+itSNN0ECs3sbHdfD2BmZwGFsS1LRKRiHX/88fz3f//3D76fP38+xx9/PBMmTChzu9KWde/enZycnMh8nz596NOnT8UVfIRuvPFGLrvsMtq1a8fpp59Or169otrunnvuYcSIEbRq1YomTZqQmZlJvXr1APjtb3/LDTfcQEpKClB0L7Jfv34xO4eKYkUPo5SzQtE74J4DNlD0Ju8zgRHuPi/25R2ezMxML/kHTkTkUObPn89tt91GaX93lLcsaPbv309hYSE1atRgx44ddOnShUceeYTevXvHu7RSmdkydy+3p/0hAy68oxOA5PDsOnffWwH1VTgFnIjIkfn666+56KKLKCwsZM+ePVx55ZXcc8898S6rTNEEXJmXKM2sp7vPNbOLD1p0jpnh7q9WSJUicswwM3bu3Ent2rXjXYocpHHjxixbtizeZVSo8u7BdQPmAj8uZZkDCjgREUlYZT5F6e5jwz9HlDJdU3klikiQPPTQQ6SlpZGcnMzUqVMj35c2jBYU3QdLS0vj2muvJRQKkZqaypo1awD48ssv6dGjBxkZGbRt25Zf//rXkf397ne/Y8iQIfTv359WrVrxox/9iN27dwMwZ84csrKySE9PJyUlhRdffLESfwNSWaLp6H2TmdW1Ik+b2ftmFv9HhUSkSkpKSiI3N5c33niDUaNG8fXXXwPw97//nZycHPLy8mjbti0PPvhgZJv8/HxGjx7NypUrufzyy7nvvvuAIx+1o3379ixYsIDly5cze/ZsbrvttkigSnBE0w/uGnffAfQBGgJDgQdiWpWIBNbIkSMBSE5Opn379pFBicsaRqt43fT0dAA6deoUGX3jSEft2Lx5M5deeint2rWjb9++bN26lXXr1lXC2UtliibgLPyzPzDR3fNLfCcictQONYxWWaNvHOmoHb/4xS/o3r07eXl55Obmcvrpp1fYsF2SOKIJuGVmNouigHvLzOoAB2JblogE1XPPPQfAhx9+yPLly+nUqVO5w2iV50hH7di+fTvNmzfHzHj77bcj4zhKsEQzkslIII2iV+TsNrOGwIiYViUigVVQUEB6ejq7d+9m3LhxNG7cmH79+jF58mTOPfdcGjVqxAUXXMB77713yH0d6agdDzzwANdddx1jx46lQ4cOhEKhoz0tSUDRdvRuStEIJpFAdPd3YljXEVFHbxGRIh9//DGzZs1i1KhRMTvG7373O3bt2sXDDz8cs2OUJZqO3tE8RfkgsBD4DXB7eLqtQioUEZGY+Pjjj3nyySfLXJ4obxEorY7CwooZ7jiae3ADgWR37+/uPw5PP6mQo4uIyFHbvXs3l112GW3atCE1NZXLL7+c66+/ntWrV5OWlsall14KQPPmzRkzZgwdO3bk2muvLbcfYdOmTSNdOPr378+PfvQjoGhIr9NPPz2y3qeffkrPnj1p1aoVl1xyCd988w0A+/bt4/bbb6djx46kpqYydOhQdu3aBcDw4cP5r//6L7p27UpmZibz588nFAoxYsQI0tLSeP7552nSpMn3Hvz5yU9+wpQpUw7r9xJNwG0A9IJTEZEE9dZbb7Fjxw5Wr17NihUrGDduHI899hht2rQhNzeXV155JbLujh07eO+993jmmWfK7UfYo0cP5s6dy/79+9m4cSMbN25k//79zJkzhx49ekT2l52dzQsvvMDatWupV68ef/jDHwD485//TL169XjvvfdYsWIFp512Gn/6058i2+Xm5jJz5sxIt478/HxGjRpFbm4uw4YNo1u3bpG3P3z88cfk5OREgjpa0TxkshvINbM5QGSQZXe/8bCOJCIiMVE8usv1119P9+7dI62t0gwbNizyubgf4aJFi3B3vvzyS3Jzc+nXrx+9evVi9uzZNG3alE6dOuHuvPvuu8yePZuePXtG9jFgwIDIm8tHjhzJL3/5SwDeeOMNduzYEQnXvXv3kpqaGtnu0ksv5cQTT4zMt2zZkqysrMj8jTfeyC233MLVV1/NE088wTXXXEP16tUP6/cSTcC9EZ5ERCQBnXXWWeTn5zNnzhxmzJjBXXfdxaOPPlrquiUHui7Zj7BGjRqMGjUqclmwZ8+e3HvvvZGnU92dOXPmMGfOHMaOHXvImtydf/3rX98Lw7LqKG2+c+fOFBYWsnDhQsaPH39Eb0g/5CVKd58AvAQscfcJxdNhH0lERGJi06ZNJCUlMXDgQP7617+yefNm6tatG7kfVpby+hGeeeaZJCUlMWHCBHr16kWvXr0YP348xx9/PGeccUZkvX//+99s3rwZKOrjWBxoP/nJT3jkkUf47rvvANi5c2dkDNFo/fKXv2Tw4MF07tyZZs2aHda2EN1TlD8GcoGZ4fk0M1OLTkQkQeTl5ZGVlUVqaiodO3bkzjvvpGPHjiQnJ9OuXbsy713deOONLFy4kHbt2jFy5Mgf9CPs1asXtWrVokmTJpx22mnUrFnzBy2yrl27MnjwYFq1asXWrVv57W9/C8CYMWNITU2N9DPs0qXLYQfc4MGD2bZtG9ddd91hbVcsmjd6LwN6AvPdPT383Sp3b3dER4wh9YMTEQmOBQsWMHr0aPLy8jD7/giRR/XC0xL2u/s3B+1cQ3WJiEjMjBw5krfffpuJEyf+INyiFU3A5ZvZlUCSmbUEbgQWHdHRREREovDMM88c9T6i6Qf3S6AtRV0EXgB2ADcf9ZFFRERi6JAtOHffDdwdnkRERKqEQwacmWUCdwHN+f5gyxp+W0REElY09+Cep2iA5Tz0cImIiFQR0QTcZndXvzcREalSogm4sWb2NHDwWJSvxqwqERGRoxRNwI0AWlH0RoHiS5QOKOBERCRhRRNwHdw9OeaViIiIVKBo+sEtMrM2Ma9ERESkAkXTgutE0fvgNlJ0D84AVzcBERFJZNEEXL+YVyEiIlLBygw4M6vr7juAnZVYj4iISIUorwU3BRgALKPoqcmSwzk7cFYM6xIRETkqZQacuw8I/2xReeWIiIhUjPIuUbYvb0N3f7/iyxEREakY5V2i/Ev4Zw0gE1hB0WXKEJADZMW2NDCzfsDfgSTgaXd/INbHFBGRYCizH5y793D3HsAXQHt3z3T3DCAd+CzWhZlZEvAYcBHQBhii/ngiIhKtaDp6J7t7XvGMu68CWseupIiOwEfuvsHd9wEvAj+thOOKiEgARNMPbmV4sOXJ4fmrgJWxKymiKfD/SsxvAs47eCUzGwWMAjjjjDMqoSwREakKomnBjQDygZvC0+rwdwnB3Z8MXz7NPPnkk+NdjoiIJIhDtuDcfQ/w1/BUmT4DmpWYP51KuPcnIiLBEE0LLl6WAi3NrIWZVQcGA3rxqoiIRCWae3Bx4e4FZnYD8BZF3QSedff8OJclIiJVRMIGHIC7Twemx7sOERGpeg4ZcGY2jaKxJ0v6hqLO3uPC9+hEREQSSjT34DYAu4CnwlPxGwbODc+LiIgknGguUXZ29w4l5qeZ2VJ372BmuicmIiIJKZoWXG0zi/SgDn+uHZ7dF5OqREREjlI0LbhbgQVmtp6iwZZbANeZ2YnAhFgWJyIicqQO2YILP8nYEriZopFMkt393+7+rbv/LbblSUUzM3bt2hXvMr6nefPmrFq1CoD/+q//Ijs7O84ViUgQRNtNIANoHl4/1cxw94kxq0rirqCggGrVKr8XydNPP13pxxSRYDpkC87MJgEPA12ADuEpM8Z1SQw99NBDpKWlkZyczNSpUyPfmxm/+93v6NChA7///e/56quvGDRoEKFQiJSUFCZO/L9/06xZs4Y+ffpElk2YUHS1+rPPPuOSSy4hFAoRCoX405/+BFDuvkrq3r07b775ZgzPXkSOFdH8Ez0TaOPuB/eFkyoqKSmJ3Nxc1q1bR+fOnenatSuNGzcGoGbNmixduhSAK664gnbt2vHaa6/xxRdfkJGRQfv27WnVqhU//elPuf/++7nssssA2LJlCwA/+9nP6N+/fyQ4//Of/wBw4403lrqvdu3aVfbpi8gxIpqnKFcBp8a6EKk8I0eOBCA5OZn27duzZMmSyLKrr7468nn27Nlce+21ADRp0oT+/fszb9481q1bR0FBQSTcABo2bMiuXbtYtGgRt9xyS+T7Ro0albsvEZFYiaYF1whYbWbvAXuLv3T3n8SsKomb2rVrH3olEZEqIJoW3O+AgcAfgb+UmKSKeu655wD48MMPWb58OZ06dSp1vd69e/PUU0WD1Xz55ZdMnz6dnj17kpycTLVq1Xj55Zcj627ZsoXatWvTuXNn/vrX/3uzUvElyrL2JSISK9F0E/jf0qbKKE5io6CggPT0dAYMGMC4ceMi998O9o9//IMVK1YQCoW48MILeeCBB2jbti3VqlXj9ddf54knniAlJYXU1FSmTy8aE3vy5MksXLiQdu3akZqayjPPPFPuvkREYsXKenbEzBa4excz28n3B1s2wN29bmUUeDgyMzM9Jycn3mWIiEiMmdkydy/3if4y78G5e5fwzzoVXZiIiEislRlwZtagvA3dfWvFlyMiIlIxynuKchlFlyYNOAPYFv58EvApRWNSioiIJKQyHzJx9xbufhYwG/ixuzdy94bAAGBWZRUoIiJyJKLpJtApPOAyAO4+A+gcu5JERESOXjQdvT83s98Ak8PzVwGfx64kERGRoxdNC24IcDLwWnhqHP5OREQkYR2yBRd+WvKmSqhFRESkwpTXTWAa3+/g/T0ai1JERBJZeS24h8M/L6bobQLF9+CGAF/FsigREZGjVd5IJv8LYGZ/OWg4lGlmpvGwREQkoUXzkMmJZnZW8YyZtQBOjF1JIiIiRy+abgK3APPNbANFI5mcCVwb06pERESOUjRPUc40s5ZAq/BXa919b3nbiIiIxNshL1GaWS3gduAGd18BnGFmA2JemYiIyFGI5h7cc8A+ICs8/xlwX8wqEhERqQDRBNzZ7v5nYD+Au++m6F6ciIhIwoom4PaZWU3Cnb7N7GxA9+BERCShRfMU5VhgJtDMzJ4HzgeGx7IoERGRoxXNU5Rvm9n7QCeKLk3e5O7/iXllIiIiRyGaFhxAN6ALRZcpj6forQIiIiIJK5puAv8CRgN5wCrgWjN7LNaFiYiIHI1oWnA9gdbuXvyQyQQgP6ZViYiIHKVonqL8CDijxHyz8HciIiIJK5r3wdUB1pjZe+H584D3Kqc8ERGRIxPN++BERESqnEO+D66YmdUtb30REZFEcsjAMrNRwL3AHuAARX3hHDirvO1ERETiKZoW2e1AO3XuFhGRqiSapyjXA7tjXYiIiEhFiqYFdyewyMzepcQgy+5+Y8yqEhEROUrRBNw4YC5FI5kciG05IiIiFSOagDve3X8V80pEREQqUDT34GaY2Sgza2JmDYqnmFcmIiJyFKJpwQ0J/7yzxHfqJiAiElCFhYUkJSXFu4yjdsgWnLu3KGVSuImIVEEzZ84kPT2dUChEr169+Oijj5g/fz6hUIgRI0aQlpbGjBkzmDJlCueddx7p6emkp6czZ86cyD6aN2/OPffcQ1ZWFs2bN+ef//xnZFl2djYpKSmEQiFuuukmzjzzTFatWgXAunXruOiii+jQoQOpqak899xzsT1Zdy91An5d4vNlBy37Y1nbxXPKyMhwEREp3VdffeWNGjXy/Px8d3d/+umnvWPHjj5v3jw/7rjjfNGiRZF1//Of//iBAwfc3X3t2rXetGnTyLIzzzzTb731Vnd337hxo5944om+c+dO37Nnjzdt2tTfeecdd3d/9dVXHfC8vDzfv3+/t2/f3tesWePu7jt27PBzzz03Mn+4gBw/RCaU14IbXOLznQct61eBGSsiIpXg3XffJTU1lTZt2gAwYsQIcnNz2blzJy1btiQrKyuy7vr16+nbty9t27bliiuu4Msvv+TLL7+MLB88uCgimjdvTv369dm0aRPr1q2jZs2adO3aFYBBgwZx0kknAfDBBx+wZs0aBg8eTFpaGl27dmXv3r2sWbMmZudb3j04K+NzafMiIlKF1a5d+3vzQ4YM4S9/+QsDBw7kwIED1KpViz179kSW16hRI/I5KSmJgoKCcvfv7jRq1Ijc3NwKrbs85bXgvIzPpc2LHFNycnK46qqrYrLv+fPnk5mZGZN9y7GtU6dOrFixgrVr1wIwYcIE0tPTqVOnzg/W3b59Oy1atADg2WefZe/evT9Y52DJycns3r2bhQsXAvD666+zffv2yLJatWoxadKkyPpr165lx44dR3taZSqvBZdqZjsoaq3VDH8mPF+j7M1Egi8zM5Pnn38+3mWIHJaTTz6ZSZMmceWVV1JQUMDJJ5/M5MmT2bRp0w/W/dvf/sbAgQOpX78+/fr1o2HDhofc/wknnMCUKVMYPXo0Zka3bt1o3Lgx9erVo1q1akybNo2bb76Zhx56iMLCQk455RReeumlWJxqkUPdpKtKkx4yqTo2btzoDRs2POxlsfLtt9/6pZde6q1bt/ZQKOSXXXaZz5s3z0OhkA8dOtTbtGnjHTp0iNycnzdvnhf/eSuu96677vK0tDQ/99xzPTs7293d9+/f73369PGMjAxv06aNDx8+3Pfu3Rs57h//+Edv166dh0Ihz8rK8sLCQp83b56npqb6qFGjPCUlxUOhkK9evbpSfx8iR2rHjh2Rz3PnzvUzzjjDCwsLK/w4HOVDJnFjZpeZWb6ZHTAzXauRmHvrrbfYsWMHq1evZsWKFYwbNw6AlStXMnLkSPLz87n++usZNmxYqdtv2bKFrKwsli9fzj333MMdd9wBFN2bmDJlCjk5OaxatYrCwkKeffZZoOjy0BtvvMGiRYtYsWIF06ZN47jjiv6XzM/PZ/To0axcuZLLL7+c++67rxJ+CyJHb+rUqaSmppKSksKvf/1rpkyZEvlzXdkSMuCAVcDFwDvxLkSit3jxYrp06UJqaiqpqanMmjWL2267LdLnpVevXnzyySff2+bWW28lFAqRkpJCdnb2IZcVFBTQt29fMjMzadu2LSNGjGDfvn0A7Nu3j1GjRnHuuefSpUsXbrjhBi699NLI/h588EE6duxI+/bt+fGPf/y9J8JSU1NZs2YN119/PS+//DInnHACAOeccw7dunUDYOjQoeTl5ZV6z6B27doMGDAAKLrPsX79egAOHDjAww8/TFpaGqFQiLlz50Zusr/55pv84he/iNz/KHkJKDk5mfT09B/sTyTRDR8+nBUrVpCXl8fSpUs5//zz41ZLQgacu69x93XxrkOit3XrVgYNGsSf//xnVqxYwfvvv0+HDh0YM2YMS5cuZcWKFQwZMiTSsoGiVk9qaiorV67k0UcfZciQIZEb2WUtK69FNG7cOJYuXcr48eOZPXs2OTk5kWNNnjyZ9evXs2TJEt5//3369+/PrbfeGll+1llnkZ+fz4UXXsjs2bNJTU393hNjh1IciPD9J8qmTJnCggULyM7OJi8vj+uuuy6q/R7uE2oi8kPRDNWV0MJvHB8FcMYZZ8S5mmPX4sWLadOmDZ07dwaK/lKuX78+kyZN4rHHHmPXrl0/+Eu6evXq/OxnPwOge/fu1KxZk3Xr1lG3bt3IsoKCgu8tS05O5uGHH2bGjBkUFhaybds2atWqBcC8efOoU6cO77//Pp07d2bIkCGRlt8bb7xBTk4O7du3B4pagvXq1YvUsmnTJho0aMDAgQPp06cPp512Glu3bmX9+vVkZ2fTtWtXpkyZQkpKCnXr1o3697J9+3YaNWpEnTp1+Oabb5gyZUrkCckBAwbw+OOPM2jQIOrUqcOWLVuiupEvItGJW8CZ2Wzg1FIW3e3ur0e7H3d/EngSIDMzU90XEsgnn3zCLbfcwrhx4/jrX//KV199xSeffMKsWbN4+eWX2b9/P+np6TRq1CjSCvvss8/o3r07BQUFZGZmMmrUKF588UU2b97Mz372M8yMOnXqcMEFFzB37lz2799PdnY2hYWFfPXVV6xYsYINGzbw9NNP07Zt20gt7s5vfvMbrrnmmlJrzcvLY8yYMUDROHx33nknp512GikpKTz99NP84he/oFatWkycOPGwfgfDhg3j9ddfp1WrVjRu3JiuXbvy3XffRZZ99tlndOrUieOPP57atWvzzju6Ki9SYQ71FEo8J2A+kBnt+nqKMn62bNnip5xySmSon4KCAn/nnXf8lFNO8caNG3t2drYPHTrUzzjjDN+6dasvW7bMAZ80aZI/9dRT3rNnT2/atKmvW7fOKepn6ZMmTXJ397S0NK9Ro4Z/++23/o9//MNTU1O9V69e/vXXX3vr1q391FNP9X/961/+97//3Zs2bep/+9vf/LvvvvOsrCy/5JJL3N194sSJnpGR4Vu3bnV39z179nhubm6551TySUkRSSxU1acopepp0KABr776Kr/61a8IhUJkZGSwd+9eOnbsyI4dO7jlllto0aIFZkb9+vWZP38+1apV4+abb+aGG25gwYIFvPDCC1SvXp0TTjiBBg0akJubSygU4oMPPuCOO+6gVq1aDBs2jM8//5z8/HwuueQSunXrxjnnnMPs2bMZPXo0NWvW5P7776dnz560bt06chly6NChXHXVVXTr1i1SX3FnVBEJqEMlYDwmYBCwCdgLfAW8Fc12+td24nnzzTe9R48e3/vu448/9oYNG/qGDRvc3X3hwoV+5plnunvpfeC6devm06ZNi8xffPHFkdadu/ukSZP84osvdnf3K6+80h999FHfs2eP9+nTx5966qlYnJaIxBlVtQXn7q+5++nufoK7n+LufeNdkxyZrKwsVq9ezeLFi4Gi+1uffvop1atX59RTT+XAgQM88cQTh7XP3r17M2HCBPbv38/+/fuZMGECF154IQBz5szh/vvvJzU1lZYtWzJ8+PCKPiURqSISMuAkOMq6dHnZZZfRpk0bzjvvvMh4d9EaNWoUoVAo8p6qUCjEz3/+cwCmTZtGgwYNqFGjBh07dqRatSr/oLCIHCEraukFQ2Zmppfs+yQiIsFkZsvcvdyRrtSCExGRQFLAiYhIICngREQkkBRwIiISSAo4EREJJAWciIgEkgJOREQCSQEnIiKBpIATEZFAUsCJiEggKeBERCSQFHAiIhJICjgREQkkBZyIiASSAk5ERAJJASciIoGkgBMRkUBSwImISCAp4EREJJAUcCIiEkgKOBERCSQFnIiIBJICTkREAkkBJyIigaSAk4TTv39/1q9fH+8yRKSKqxbvAiS4CgsLSUpKOuztpk+fHoNqRORYoxaclMrMGDt2LGlpaSQnJzN16tTIsquuuorMzExSUlIYNGgQ27ZtA2D+/PmEQiFGjBhBWloaM2bM4Mknn6R169akpaURCoVYu3YtAM2bN+eee+4hKyuL5s2b889//jOy/+bNm7Nq1SoAunfvzs0330zHjh0555xzuOuuuyrxtyAiVZlacFKmpKQkcnNzWbduHZ07d6Zr1640btyYv//97zRq1AiA3/zmNzz44IM88MADAOTn5zNu3DiysrIAqFevHmvXrqVJkybs3buXwsLCyP53797N4sWL+fjjj2nXrh3Dhw+ndu3aP6hj9erVLFq0iD179pCVlUXnzp0ZMGBAJfwGRKQqUwtOyjRy5EgAkpOTad++PUuWLAFg4sSJZGRkkJKSwpQpU8jNzY1s07Jly0i4AfTs2ZOrr76aRx99lM8++4xatWpFlg0ePBgoarHVr1+fTZs2lVrH1VdfTbVq1ahduzaDBw9m7ty5FX2qIhJACjg5LNnZ2Tz++OPMnDmTvLw87rvvPvbs2RNZfnAL7NVXX+W+++7j22+/pUePHsyYMSOyrEaNGpHPSUlJFBQUxP4EROSYoYCTMj333HMAfPjhhyxfvpxOnTqxfft26tWrR8OGDdm7dy/PPvtsmdsXFBSwYcMGOnbsyJgxY+jTpw/Lly8/7DomT55MQUEB3377LS+99BI9e/Y84nMSkWOH7sFJmQoKCkhPT2f37t2MGzeOxo0b069fPyZPnsy5555Lo0aNuOCCC3jvvfdK3b6wsJDhw4ezfft2jjvuOJo1axa5V3c4WrVqRefOndm6dSuXX3657r+JSFTM3eNdQ4XJzMz0nJyceJcRCGbGzp07S33oozJ1796d2267TaEmIt9jZsvcPbO8dXSJUkREAkmXKKVUidKynz9/frxLEJEqSi04EREJJAWciIgEkgJOREQCSQEnIiKBpIATEZFAUsCJiEggKeBERCSQFHAiIhJICjgREQkkBZyIiASSAk5ERAJJASciIoGkgBMRkUBSwImISCAp4EREJJAUcCIiEkgKOBERCSQFnIiIBJICTkREAikhA87MHjKztWa20sxeM7OT4l2TiIhULQkZcMDbQDt3DwEfAHfGuR4REaliEjLg3H2WuxeEZ5cAp8ezHhERqXoSMuAOcg0wo6yFZjbKzHLMLGfz5s2VWJaIiCSyavE6sJnNBk4tZdHd7v56eJ27gQLg+bL24+5PAk8CZGZmegxKFRGRKihuAefuvctbbmbDgQFAL3dXcImIyGGJW8CVx8z6Ab8Gurn77njXIyIiVU+i3oP7J1AHeNvMcs3siXgXJCIiVUtCtuDc/Zx41yAiIlVborbgREREjooCTkREAkkBJyIigaSAExGRQFLAiYhIICngREQkkBRwIiISSAo4EREJJAWciIgEkgJOREQCSQEnIiKBpIATEZFAUsCJiEggKeBERCSQFHAiIhJICjgREQkkBZyIiASSAk5ERAJJASciIoGkgBMRkUBSwImISCAp4EREJJAUcCIiEkgKOBERCSQFnEgczZ8/n8zMTAA+//xzevToEVn2P//zP7Ru3Zr09HTWrVuHmbFr1654lSpS5SjgRBLEaaedxrx58yLz48aN495772X58uUkJyfHpabCwsK4HFekIijgRI7A4sWL6dKlC6mpqaSmpjJr1ixuu+02OnToQGpqKr169eKTTz4Bvt9KK22+2Mcff0yjRo0AuOWWW8jOzuaOO+74Xquu2NKlS8nKyiIUCpGVlcXSpUsBuPPOO3nooYcAeOmllzjuuOP4+uuvAejfvz+zZs0CYPr06Zx//vlkZGSQlZXFkiVLIrWFQiFGjBhBWloaM2bMqKhfmUjlc/fATBkZGS4Sa1u2bPFTTjnFFy5c6O7uBQUFvnXrVt+8eXNknaeeesqvuOIKd3efN2+el/yzWXK+5OeNGzd6w4YNI+t169bNp02bFpkHfOfOnb53715v1qyZz549293d3377bW/WrJnv3bvX3377be/bt6+7u48aNcqzsrL8hRde8H379nmDBg3822+/9Y8++sg7derk33zzjbu7r1q1yps1axap57jjjvNFixZV7C9NpIIBOX6ITKgW74AVqWoWL15MmzZt6Ny5MwBJSUnUr1+fSZMm8dhjj7Fr1y4KCgpidvx169ZRvXp1evXqBUDv3r2pXr0669at4/zzz+eKK65g3759LFy4kIcffphXXnmFpk2b0q5dO2rVqsVbb73F+vXrueCCCyL7LCgo4KuvvgKgZcuWZGVlxax+kcqigBOpAJ988gm33HILS5cupUWLFixatIgrr7wSgGrVqnHgwIHIunv27IlZHTVr1iQUCvHCCy/QpEkTevTowa233srpp58eCUR3p1+/fkycOPEH269Zs4batWvHrD6RyqR7cCKHKSsri9WrV7N48WKg6EGMTz/9lOrVq3Pqqady4MABnnjiicj6Z511Fhs2bGDbtm24Oy+88MJRHT85OZl9+/ZFHkiZO3cu+/fvjzyI0qtXL8aOHUuvXr044YQTOP300xk/fnwk4Pr06cPMmTPJz8+P7LP4Hp5IkCjgRA5TgwYNePXVV/nVr35FKBQiIyODvXv3ctlll9GmTRvOO+88WrRoEVn/tNNO49ZbbyUjI4POnTvTpEmTozp+9erVmTp1KnfddRehUIi7776bV155herVqwNEHnApDrRevXqxZcsWOnbsCBRdgpw8eTIjR44kNTWV1q1bM27cuKOqSSQRWdG9umDIzMz0nJyceJchIiIxZmbL3P2HjyOXoBaciIgEkgJOREQCSQEnIiKBpIATEZFAUsCJiEggKeBERCSQFHAiIhJICjgREQkkBZyIiASSAk5ERAJJASciIoGkgBMRkUBSwImISCAp4EREJJAUcCIiEkgKOBERCSQFnIiIBJICTkREAkkBJyIigaSAExGRQFLAiYhIICngREQkkBRwIiISSAkZcGb2BzNbaWa5ZjbLzE6Ld00iIlK1JGTAAQ+5e8jd04A3gXviXI+IiFQxCRlw7r6jxOyJgMerFhERqZqqxbuAspjZ/cAw4BugRznrjQJGhWf3mtmqSigvETUC/hPvIuLkWD53OLbPX+d+7Eo+1ArmHp/GkZnNBk4tZdHd7v56ifXuBGq4+9go9pnj7pkVWGaVoXM/Ns8dju3z17kfm+cO0Z1/3Fpw7t47ylWfB6YDhww4ERGRYgl5D87MWpaY/SmwNl61iIhI1ZSo9+AeMLNk4ADwCTA6yu2ejF1JCU/nfuw6ls9f537sOuT5x+0enIiISCwl5CVKERGRo6WAExGRQApcwB3Lw3yZ2UNmtjZ8/q+Z2UnxrqmymNllZpZvZgfM7Jh4dNrM+pnZOjP7yMzGxLueymRmz5rZ18div1cza2Zm88xsdfjP/E3xrqmymFkNM3vPzFaEz/335a4ftHtwZla3eCQUM7sRaOPu0T6kUqWZWR9grrsXmNmDAO5+R5zLqhRm1pqih5LGAbe5e06cS4opM0sCPgAuBDYBS4Eh7r46roVVEjO7ANgFTHT3dvGupzKZWROgibu/b2Z1gGXAwGPhv72ZGXCiu+8ys+OBBcBN7r6ktPUD14I7lof5cvdZ7l4Qnl0CnB7PeiqTu69x93XxrqMSdQQ+cvcN7r4PeJGiLjXHBHd/B9ga7zriwd2/cPf3w593AmuApvGtqnJ4kV3h2ePDU5l/xwcu4KBomC8z+3/AVRy7AzVfA8yIdxESM02B/1difhPHyF9y8n/MrDmQDrwb51IqjZklmVku8DXwtruXee5VMuDMbLaZrSpl+imAu9/t7s0oGgXlhvhWW7EOde7hde4GCig6/8CI5txFjhVmVhuYCtx80JWrQHP3wvCbZk4HOppZmZeoE7Wjd7mO5WG+DnXuZjYcGAD08oDdYD2M/+7Hgs+AZiXmTw9/J8eA8P2nqcDz7v5qvOuJB3ffbmbzgH5AqQ8bVckWXHmO5WG+zKwf8GvgJ+6+O971SEwtBVqaWQszqw4MBt6Ic01SCcIPWjwDrHH3R+JdT2Uys5OLnw43s5oUPWRV5t/xQXyKcipFr1GIDPPl7sfEv2zN7CPgBGBL+Kslx9ATpIOAR4GTge1Arrv3jWtRMWZm/YG/AUnAs+5+f3wrqjxm9gLQnaJXxnwFjHX3Z+JaVCUxsy5ANpBH0d9zAHe5+/T4VVU5zCwETKDoz/xxwEvufm+Z6wct4ERERCCAlyhFRERAASciIgGlgBMRkUBSwImISCAp4EREJJAUcCIiEkgKOBERCSQFnIiIBJICTkREAkkBJyIigaSAExGRQFLAiYhIICngREQkkBRwEjhmVmhmuSWmMYexbXcze/Mojl3m9mb2sZk1Cn9edKTHOMTx55tZZvjz9OJ3Z1UmM7vXzI765bRm1tDM5pnZLjP7Z0XUJseWKvlGb5FD+C78SvuE5e6dK+EY/WN9jDKOe08F7WoP8FugXXgSOSxqwckxI9yC+lO4VZdjZu3N7C0zW29mJV8MW9fM/m1m68zsCTM7Lrx9HzNbbGbvm9nLZlY7/H0/M1trZu8DF5c4XkMzm2Vm+Wb2NGAllu0K/+webnW9Et7H8+E3NmNm/cPfLTOzf5TWMjSzmmb2opmtMbPXgJoHnW8jM2se3s94M/sgfIzeZrbQzD40s47h9U80s2fN7D0zW25mPw1/P9zMXjWzmeH1/xz+Pim8z1Vmlmdmt4S/H29ml4Y/9wrvKy+87xNK1Pb78O8yz8xaHXxu7v6tuy+gKOhEDpsCToKo5kGXKK8osezTcOsuGxgPXAp0An5fYp2OwC+BNsDZwMXhS4u/AXq7e3sgB/iVmdUAngJ+DGQAp5bYz1hggbu3BV4Dziij3nTg5vDxzgLOD+93HHCRu2dQ9Kby0vwC2O3urcPHyyhjvXOAvwCtwtOVQBfgNuCu8Dp3A3PdvSPQA3jIzE4ML0sDrgBSgCvMrFn4u6bu3s7dU4DnSh4wfA7jgSvCy6uF6y32n/Dv8vFwHSIVSgEnQfSdu6eVmP67xLI3wj/zgHfdfae7bwb2lrhf9Z67b3D3QuAFioKgE0UBtNDMcoGrgTMpCouN7v6huzswucSxLiied/d/A9vKqPc9d9/k7geAXKB5eL8b3H1jeJ0Xyti25DFWAivLWG+ju+eFj5EPzAnXmxc+HkAfYEz4/OYDNfi/UJ7j7t+4+x5gdfjcNwBnmdmjZtYP2HHQMZPDx/0gPD8hXG+xV8M/l5WoQaTC6B6cHGv2hn8eKPG5eL74/wc/aBun6PLi2+4+pOQCM0urwJoAConN/5cHn2vJ30Px8Qy4xN3XldzQzM4rrUZ332ZmqUBfYDRwOXDNEdQUq3OWY5xacCI/1NHMWoTvvV0BLACWUHTp8ByI3K86F1gLNDezs8PblgzAdyi6FIiZXQTUP4wa1lHUOmoenr+ijPVKHqMdEDqMYxzsLeCXJe4Bppe3cviy7XHuPpWiy7ftD1plHUW/m3PC80OB/z2K+kQOi/7VJEFUM3yZrdhMd4+6qwCwFPgnRfet5gGvufsBMxsOvFD8oATwG3f/wMxGAf82s90U3durE17++/D6+cAi4NNoC3D378zsOmCmmX0brqk0jwPPmdkaYA1Fl/uO1B+AvwErw+G+ERhQzvpNw8cu/ofynSUXuvseMxsBvGxm1Sg6hycOpyAz+xioC1Q3s4FAH3dffTj7kGOXFV2GF5FEY2a13X1XuEX1GPChu/813nWJVBW6RCmSuH4ebonmA/UoeqpSRKKkFpyIiASSWnAiIhJICjgREQkkBZyIiASSAk5ERAJJASciIoH0/wG6y7TgIN0u7wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plt_embedding(embedding, ax, xylim=(-3, 3),\n",
    "                  xlab=\"Embedding dimension 1\", \n",
    "                  ylab=\"Embedding dimension 2\",\n",
    "                  title=\"\"):            \n",
    "    sns.scatterplot(data=embedding, x=\"x\", y =\"y\",\n",
    "                    color=\"#ffffff\", ax=ax)\n",
    "    for i in range(embedding.shape[0]):\n",
    "        ax.text(x=embedding[\"x\"][i], y=embedding[\"y\"][i], \n",
    "                s=embedding[\"id\"][i], ha=\"center\",\n",
    "                fontdict=dict(size=11))    \n",
    "    ax.set_xlabel(xlab, labelpad=12)\n",
    "    ax.set_ylabel(ylab, labelpad=12)    \n",
    "    ax.set_title(title)\n",
    "    ax.set(xlim=xylim)\n",
    "    ax.set(ylim=xylim)\n",
    "\n",
    "\n",
    "fig, ax = plt.subplots(1, 1, figsize = (6, 6))\n",
    "fig.tight_layout()\n",
    "plt_embedding(embedding, ax)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b334036-ee29-46f7-ba66-a12f0a256993",
   "metadata": {},
   "source": [
    "The layout suggests a partition of the vocabulary into two primary groups. The labels reveal that the groups represent fruits and vegetables. Thus, the two-dimensional embedding captures semantic meaning of the vocabulary items."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7eafaa3c-7eb9-4c58-9e29-8b61224aed18",
   "metadata": {},
   "source": [
    "## Expanding the vocabulary\n",
    "\n",
    "Suppose that after some time, the dataset is updated to `produce_2.txt`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "ebf6b480-3491-4814-8168-aa3f16b4d89c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First data row:\t['apple' 'orange' 'banana']\n",
      "Dataset shape:\t(672, 3)\n"
     ]
    }
   ],
   "source": [
    "produce_data2 = read_produce_data(\"produce_2.txt\")\n",
    "print(f\"First data row:\\t{produce_data2[0]}\")\n",
    "print(f\"Dataset shape:\\t{produce_data2.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18c9c7a6-d28c-418b-bf85-0fcc353be8e7",
   "metadata": {},
   "source": [
    "The dataset contains more tuples. It also contains a larger vocabulary of fruits and vegetables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "7d3edbc6-0bfc-40d9-8b63-267e87745146",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New vocabulary items:\t['kiwi', 'lemon', 'pepper', 'pumpkin']\n",
      "New vocabulary:\t['apple', 'banana', 'broccoli', 'cabbage', 'carrot', 'cauliflower', 'mango', 'orange', 'parsnip', 'pear', 'spinach', 'strawberry', 'kiwi', 'lemon', 'pepper', 'pumpkin']\n",
      "New vocabulary size:\t16\n"
     ]
    }
   ],
   "source": [
    "produce_vocabulary2 = sorted(set(produce_data2.flatten()))\n",
    "new_produce = sorted(set(produce_vocabulary2).difference(produce_vocabulary))\n",
    "produce_vocabulary2 = produce_vocabulary + new_produce\n",
    "print(f\"New vocabulary items:\\t{new_produce}\")\n",
    "print(f\"New vocabulary:\\t{produce_vocabulary2}\")\n",
    "print(f\"New vocabulary size:\\t{len(produce_vocabulary2)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f09a7814-716d-48e3-8b6f-aad76ceee6e4",
   "metadata": {},
   "source": [
    "The new vocabulary items are 'kiwi', 'lemon', 'pepper', and 'pumpkin'. These elements cannot be encoded into the low-dimensional space by our existing model. We must therefore train a new model. At first, let's try re-using our current training procedure on the new dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "82dd87f4-854b-4888-8da7-19a02b0b6cff",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: False, used: False\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "HPU available: False, using: 0 HPUs\n",
      "\n",
      "  | Name      | Type             | Params\n",
      "-----------------------------------------------\n",
      "0 | embedding | Embedding        | 32    \n",
      "1 | output    | Linear           | 80    \n",
      "2 | criterion | CrossEntropyLoss | 0     \n",
      "-----------------------------------------------\n",
      "112       Trainable params\n",
      "0         Non-trainable params\n",
      "112       Total params\n",
      "0.000     Total estimated model params size (MB)\n"
     ]
    }
   ],
   "source": [
    "# dataset and loader\n",
    "produce_dataset2 = VocabDataset(produce_vocabulary2, produce_data2)\n",
    "produce_loader2 = DataLoader(produce_dataset2, \n",
    "                             batch_size=4,\n",
    "                             num_workers=n_cpus,\n",
    "                             shuffle=True, \n",
    "                             drop_last=True)\n",
    "# model\n",
    "model2 = train_vocab_embedding(produce_loader2,                               \n",
    "                               vocab_size=len(produce_dataset2.vocab))\n",
    "embedding2 = make_embedding_table(produce_vocabulary2, model2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed24db48-e7ba-4ecb-b20c-daaa99eb638a",
   "metadata": {},
   "source": [
    "And we visualize the embedding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "6e2a75de-d7a4-4834-a314-21d8f172374e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbgAAAG+CAYAAADhkxGuAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAA+U0lEQVR4nO3deXxU5dn/8c8FAQFZlEVBZBGFIGQlARMWWYKIuFSrotSKIBZ3qy20Wq1itYpL9dEuilUUBFyq9lFUUEGo7BgkLGEREOoPFRVlXxISrt8fGeYJSMJAtsnJ9/16zStz5mzXGZZv7nPf5xxzd0RERIKmWkUXICIiUhYUcCIiEkgKOBERCSQFnIiIBJICTkREAkkBJyIigRS1AWdmtcxsoZktMbNsM7u/omsSEZHKw6L1OjgzM+B4d99pZjWA2cCv3X1+BZcmIiKVQExFF1AUL0jenaHJGqFXdKaxiIhEnagNOAAzqw4sAs4A/u7uCw6zzHBgOMDxxx+f0r59+/ItUkREyt2iRYs2u3uT4paJ2lOUhZnZCcC/gVvdfXlRy6WmpnpmZma51SUiIhXDzBa5e2pxy0TtIJPC3H0rMAPoX8GliIhIJRG1AWdmTUItN8ysNnAOsKpCixIRkUojmvvgmgHjQv1w1YDX3f3dCq5JREQqiagNOHdfCiRXdB0iIlI5Re0pShERkZJQwImISCAp4EREJJAUcCIiEkgKOBERCSQFnIiIBJICTkREAkkBJyIigaSAExGRQFLAiYhIICngREQkkBRwIiISSAo4EREJJAWciIgEkgJOREQCSQEnIiKBpIATEZFAUsCJiEggKeBERCSQFHAiIhJICjgREQkkBZyIiASSAk5ERAJJASciIoGkgBMRkUBSwImISCAp4EREJJAUcCIiEkgKOBERCSQFnIiIBJICTiTgMjMzueqqq8pk2zNnziQ1NbVMti1SUgo4kYBLTU1l4sSJFV2GSLlTwIlUMrt37+byyy+nQ4cOJCYmMnDgQGbOnEliYiKDBw+mY8eOdOnShRUrVgAHt7I2bNhA48aNufvuu0lOTiY2NpbZs2cDkJeXx7nnnktqaiodO3Zk6NCh5Obmhvf78MMPEx8fT2JiIl27dmX//v3h9a6//noSEhJITExk5cqVRdb+v//7vyxcuPCw8wrX+fXXX9O7d++D1jvzzDNJTk5m9erVmBk7d+4swbcoVYECTqSS+eCDD9i+fTsrVqxgyZIljBkzBoClS5cybNgwsrOzufnmmxk8ePBh1//hhx9IT09n8eLF3Hvvvfz+978HoHr16kyaNInMzEyWL19Ofn4+Y8eOBWDcuHG88847zJ07lyVLljB58mSqVatGfn4+2dnZ3HDDDSxdupTLLruMBx98sMjaiwu4wk455RRmzJgRnh4zZgx/+tOfWLx4MbGxsRF/V6UpPz+/QvYrJeDugXmlpKS4SNCtW7fOW7Ro4TfddJO//vrrvmvXLp8xY4afccYZ4WXy8/O9Zs2avm3bNp8xY4Yf+Lexfv16r1u3bni5tWvXesOGDb1bt24eHx/vTZo08TZt2njjxo29Ro0a3rBhQ+/Tp48PGDDAx40b5+vXr/dGjRr5b3/7W09OTvY77rjDa9Wq5c2aNfM6dep47dq1PTY21lNSUrxWrVp+3HHHecuWLX3dunU+depUr1u3rteoUcMTExN93LhxB9V2aJ2NGjVyd/fbb7/djz/+eD/ttNO8V69e7u4O+I4dO9zdfeHChZ6Wlubx8fGelpbmCxcudHf3O++80x999FF3d3/ttdfczPzbb791d/fzzjvPP/jgA3d3f++997xr167eqVMnT0tL83nz5oXriY+P9yFDhnhiYqJPnjy5DP405VgBmX6ETFALTqSSadOmDdnZ2ZxzzjlMmzaNxMRE9u7dG/H6xx13XPj9jh072LJlC48++igjR46kXbt2zJgxg5UrVzJq1CguvPBCBg0axNKlS8Pr/PDDD3Tu3JnPPvuMXr16kZOTQ5s2bdi2bRuTJ09m165d1KlTh23btrF9+3bq1KnDwIEDOffcc+nWrRtNmzYlKyuryBbmoZ588klSU1N5+umnD2rVAeTm5nLppZfy4IMPsnTpUh544AEuvfRScnNzycjIYPr06QBMnz6dtLQ0Pv74Y/bt28eCBQvo3r0769at44EHHmDKlCksWrSI559/noEDB4a3n52dzfDhw8nKyuKCCy6I+DuW6BBT0QWIyNHZuHEjDRs25OKLL6Zfv36ccsop/Pjjj6xbt45Zs2bRo0cPJk2aRHx8PPXr1y92W5999hkxMTF07dqVRYsW0aRJE1q2bMmYMWN48MEHOe6445g/fz55eXk888wzdOrUiVq1apGRkQEUBEC1atW45ZZbiImJoXr16uzcuZMOHTpw9tlns3PnTrZt28bGjRvL5LtYvXo1NWvWDNfTt29fatasyerVq+nWrRtXXHEFubm5zJkzh8cff5w33niD5s2bExcXR506dfjggw9Yt24dZ599dnibeXl5fPvttwC0bduW9PT0Mqldyp5acFKmonEwQOvWrVm+fDkA1113HbNmzargio7OsmXLSE9PJzExkS5dunDXXXdxyimnEB8fz/PPP09cXBxPP/0048ePP6rtDh48mB07dnD66adzyy23cMkll/Czn/2MsWPHUrNmTS688EJ+/vOfs2/fPi6++OLwIBOAunXrht/v37+fiRMn8sorr7B8+XKuu+668LJmRsHZpQJH0/I8WrVr1yYhIYFXXnmFZs2a0bt3b+bNm8f06dPDgeju9O/fn6ysrPDr66+/5uSTT/7JcUkldKRzmJXppT646EOhvpLi7Nu3rxyqKdCqVStftmxZue2vPBTuvzoaP/zwg5988sk+d+5cd3fPy8vzTz75xJs1a+a7d+/2/Px8v/rqq71Vq1bufnDf2IsvvugXXnih16hRwx977LHw+r/5zW/czHzTpk2+d+9eb9q0qTds2NDd3YcOHeq1atXyH3/80ffv3++DBw8+Yh+cu3vPnj0P6gM78PcqJyfHW7Ro4R9//LG7u0+fPt1btmzpOTk57u7+wAMPeKtWrfzhhx92d/d+/fp5q1atfPbs2e7u/vnnn3uTJk18+fLl4W0f6MM71u9Uygfqg5No8Nhjj5GUlERsbCxvvvlm+HMzY9SoUXTu3Jn777+fb7/9lksuuYSEhATi4+MPaoGsXLmSfv36heeNGzcOgK+++opLL72UhIQEEhISePjhhwGK3VZhvXr14t133y3Do49uDRs25K233uI3v/kNCQkJpKSkkJOTE74M4ayzzuK0004rcv2aNWvSsWNHnn/++fD6/fr1IykpiRYtWnDCCSfQunVr6tWrB8CNN95IvXr1aNq0KW3btqVZs2Ylqr9mzZq8+eab/OEPfyAhIYG7776bN954g5o1awKQkZHBf//733CLLSMjgx9++IEuXboABacgJ0yYwLBhw0hMTOTMM88Mj0qVADhSAlaml37bij6A33///e7uvmrVKm/YsGF4JBvgo0ePDi87cOBAv+eee9zd/euvv/ZmzZr5smXLfN++fd62bVt//fXXw8tu3rzZ3d179eoVHinn7v79998Xuy33g1twh7YMKhvA7733Xk9MTPR27dr5G2+84e4/bQEVnj7w/s477/SkpCSPjY31zMxMv+666zw+Pt67dOni33zzjbsXtNL69u3rF154oZ955pneu3dv37hxY3jepZde6u7uX375paekpPhrr70WrutAy71Vq1b+xz/+0dPS0rxVq1b+17/+tXy+HAk01IKTaDBs2DAAYmNj6dSpE/Pnzw/Pu+aaa8Lvp02bxvXXXw9As2bNGDBgADNmzGD16tXk5eVx+eWXh5dt1KgRO3fuZO7cudxxxx3hzxs3blzstoKoevXqZGVl8c477zB8+HC+++67I67zww8/0L17dxYvXsywYcPIyMjg5ptvZunSpaSkpPC3v/0tvOzs2bN57LHHWLFiBT179uTXv/71QdtasmQJAwYM4MknnzxoBGJhu3fvZt68ecycOZM777wz6vplJZgUcFKh1IlfcsX9AlGUunXrcv755wPQqVMnTj31VJKSkgBISUlh7dq14WW7d+8evrj6uuuu4+OPPw7PW7p0KT//+c95/fXX6dGjR5H7u/LKK4GCAT4nnnhimY2qFClMASdl7sUXXwRgzZo1LF68mLS0tMMu17dvX/75z38CsGnTJt5//3369OlDbGwsMTEx/Otf/wov+8MPP1C3bl26du3Kk08+Gf588+bNxW6rqoiJiTlolOOhoxULXwtXvXp1atWqddB0Xl5eRPs59dRTadiwITNnzix2uWPdvkhJKOCkzOXl5ZGcnMwFF1zAmDFjOOmkkw673NNPP82SJUtISEjgnHPOYfTo0XTs2JGYmBjefvttnn322fC9EN9//30AJkyYwJw5c4iLiyMxMZEXXnih2G0F0eF+gWjatCn79u0Lt8QmTZp0zNufM2cOa9asCe+r8C8KDRs2ZNq0abz88ss88cQTJTiKgw0YMIB169aV2vakatKF3lKmPHTN06hRo4qcd8DJJ5/M//7v/x52O2eeeWb4rhSFNW/e/LDrFLetDRs2hN8fqeVRGRz4BWL37t0H/QLx1FNPcc4559CkSZPw6chj0a1bN0aMGMGaNWto2rQpL7/88kHzGzRowIcffsgFF1zArl27+OMf/xied6z3bzzwC4xISdih/8lUZqmpqZ6ZmVnRZYiUGzNjx44dZdKXaWZcdNFF/Oc//+Hkk0/moYce4tJLLwXgqquuYvXq1eTk5HDGGWcwduxYTjzxRGbOnMltt91GSkoKixcv5sEHH+Trr7/mySef5LjjjmP//v28/vrrtG/fntatWzN48GA++ugjvvnmG0aMGMEtt9wCFPTVvfvuu8TFxdGrVy+SkpKYO3cuP/74IwMHDuShhx4q9eOVysXMFrl7sQ8jVAtORIpUrVo1+vbty5///Ge6du1Kjx49OOmkk3jqqafCI1bvueceHnnkEUaPHg0U3L5rzJgx4VtcNWjQgFWrVtGsWTNycnIOatUdGF25YcMG4uLiGDJkyGHDesWKFcydO5e9e/eSnp5O165ddW9IOSL1wYlUYu5epiNR//a3v/HGG2/8ZITm+PHjSUlJIT4+nkmTJpGVlRVe59D7N/bp04drrrmGv/71r3z11VfUqVMnPC/S0ZXXXHMNMTEx1K1blyuvvPKgkZwiRVHAichRmTVrFs888wxTp05l2bJlPPjggweN0jw0cN966y0efPBBdu3aRe/evZkyZUp4nkZXSllSwIlIkQ43QnPr1q00aNCARo0akZOTE34o6uHk5eXxxRdf0KVLF+6880769evH4sWLj7qOCRMmkJeXx65du3j99der1CUfcuzUByciRTrcCM3+/fszYcIE2rVrR+PGjTn77LOLfEp3fn4+Q4YMYevWrVSrVo0WLVqE++qORvv27enatWt4kIn63yQSGkUpIodVliM0j0avXr0YMWKEQk0OEskoSp2iFBGRQFLAiVRxGzZsCA/5L8zd2bx582HnlaeZM2eq9SbHRAEnIiKBFLUBZ2YtzGyGma0ws2wz+/WR1xIRgHnz5tG9e3cSExNJTEzkww8/ZMSIEXTu3JnExMTwg0AL++1vfxt+QOysWbOOOC8vL49zzz2X1NRUOnbsyNChQ8nNzQUgNzeX4cOH065dO7p3784tt9zCZZddFt7eI488QpcuXejUqRMXXnghmzZtKuNvRKqkIz0wrqJeQDOgU+h9PeBzoENx6+iBpyLuP/zwg5988sk+Z84cd3fPy8vzH3/8MfwwWHf3f/7zn37FFVe4e8EDUAEfN26cu7vPmDHDmzdv7nv37i123v79+8MPnt2/f79fffXV/swzz7i7+9NPP+3nnnuu79u3z/fs2eNnnXVW+OGoL7/8sv/qV7/y/Px8d3f/xz/+4b/4xS/K4ZuRICGCB55G7WUC7v4N8E3o/Q4zWwk0B1ZUaGEiUW7evHl06NCBrl27AgUXUJ944om8/PLL/P3vf2fnzp0/uaC6Zs2a/PKXvwQKRi3Wrl2b1atXU79+/SLndezYkccff5wpU6aQn5/Pli1bwncpmTFjBldffTUxMTHExMQwaNCgcMvvnXfeITMzk06dOgEFLcEGDRqUy3cjVUvUBlxhZtYaSAYWHGbecGA4QMuWLcu3MJFK4r///S933HEHn376Kaeddhpz587lF7/4RYm2OWnSJGbPns2sWbOoV68eDz30EJ9//vkR13N37rnnHq699toS7V/kSKK2D+4AM6sLvAnc7u7bD53v7s+5e6q7pzZp0qT8CxSJMunp6axYsYJ58+YBBRdbf/nll9SsWZOmTZuyf/9+nn322YPWyc3NDT8zbtasWezZs4f27dsXO2/r1q00btyYevXqsW3btoOeOderVy8mTpxIXl4ee/fu5bXXXgvPu+iii/jHP/7Bli1bAMjJyWHJkiVl94VIlRXVLTgzq0FBuE1097cquh6RyqBhw4a89dZb/OY3v2HXrl1Uq1aNxx9/nMsvv5wOHTrQuHFjBgwYwCeffBJep1GjRmRlZfHoo4/i7rzyyivUrFmz2HmDBw/m7bffpn379px00kn06NGDPXv2AHDDDTewZMmS8P7OPPPM8L6uvvpqNm/eTM+ePQHYv38/N910E4mJieX4LUlVELV3MjEzA8YBP7r77ZGsozuZiESPHTt2UK9ePXJycrjooou4/PLLue666yq6LAmIyv48uG7A1cAyM8sKffYHd9ejfkUqgb59+5KTk8PevXvp27cvQ4YMqeiSpIqJ2oBz99mAVXQdInJsFiz4yZgwkXIV9YNMREREjoUCTkREAkkBJyIigaSAExGRQFLAiYhIICngREQkkBRwIiISSAo4EREJJAWciIgEkgJOREQCSQEnIiKBpIATEZFAUsCJiEggKeBERCSQFHAiIhJICjgREQkkBZyIiASSAk5ERAJJASciIoGkgBMRkUBSwImISCAp4EREJJAUcCIiEkgKOBERCSQFnIiIBJICTkREAkkBJyIigaSAExGRQFLAiYhIICngREQkkBRwIiISSAo4EREJJAWciIgEkgJOREQCSQEnIiKBpIATEZFAUsCJiEggKeBERCSQFHAiIhJICjgREQkkBZyIiASSAk5ERAJJASdSCeXn51d0CSJRTwEnUgwz47777iMpKYnY2FjefPPN8LwFCxbQu3dvUlJSSElJ4b333gNgw4YNNG7cmN/+9rckJCQQHx/PrFmzjjgP4P3336dbt26kpKSQnp7O/PnzAZg5cyYJCQkMHTqUpKQkpkyZUo7fgkgl5e6BeaWkpLhIaQL8/vvvd3f3VatWecOGDf3bb7/1LVu2eFJSkn/99dfu7v7111978+bNfcuWLb5+/XoHfNy4ce7uPmPGDG/evLnv3bu32Hlr1671tLQ037Ztm7u7L1++3Fu0aBFerlq1aj537tzy/gpEohKQ6UfIhJiKjVeR6Dds2DAAYmNj6dSpE/PnzycmJob169dz3nnnhZczM9auXUvjxo2pWbMmv/zlLwHo1asXtWvXZvXq1dSvX7/IebNnz2bdunWcffbZ4W3m5eXx7bffAtC2bVvS09PL67BFKj0FnMgxcHcSEhL45JNPfjJvw4YNx7zN/v37M378+J/MW7lyJXXr1j2m7YpUVeqDEzmCF198EYA1a9awePFi0tLS6Nq1K2vWrGHGjBnh5T799FMKzpxAbm4ukyZNAmDWrFns2bOH9u3bFzuvX79+TJ06lezs7IO2KSLHRi04kSPIy8sjOTmZ3bt3M2bMGE466SQA3nnnHUaOHMntt99Obm4ubdq0YfLkyQA0atSIrKwsHn30UdydV155hZo1axY7r23btkyYMIFhw4axZ88ecnNz6datG507d66wYxepzOzAb5xBkJqa6pmZmRVdhgSImbFjx46jOj24YcMGUlNT2bx581HNE5HImdkid08tbhmdohQRkUDSKUqRYhzLGY7WrVsX2UIrbp6IlC614EREJJAUcCIiEkgKOBERCSQFnIiIBFJUB5yZjTWz78xseUXXIiIilUtUBxzwEtC/oosQEZHKJ6oDzt0/AX6s6DpERKTyieqAi4SZDTezTDPL/P777yu6HBERiRKVPuDc/Tl3T3X31CZNmlR0OSIiEiUqfcCJiIgcjgJOREQCKaoDzsxeAeYBsWa20cyGVXRNIiJSOUT1zZbdfVBF1yAiIpXTEVtwZlbjMJ81LptyRERESkeRAWdmvc1sI/CNmX1oZq0Lzf6wzCsTEREpgeJacI8C57p7Y+A54CMzSwvNszKvTEREpASK64Or6e7ZAO7+hpmtBN4ys98DR/8USBERkXJUXMDtM7Om7r4JwN2zzSwDeBc4vVyqExEROUbFnaK8Ezi58AfuvhHoCYwuy6JERERKqsgWnLtPK+LzbcCfy6wiERGRUhDVF3qLiIgcKwWciIgEkgJOREQC6Yi36jKzdsBIoFXh5d29TxnWJSIiUiKR3IvyX8CzwD+B/LItR0REpHREEnB57v5MmVciIiJSiiLpg5tsZjeZWTMza3jgVeaViYiIlEAkLbhrQj9HFvrMgTalX46IiEjpOGLAuftp5VGIiIhIaYpkFGUN4Ebg7NBHM4Ex7r6vDOsSEREpkUhOUT4D1AD+EZq+OvTZdWVVlIiISElFEnCd3T2x0PTHZrakrAoSEREpDZGMosw3s/DjccysDboeTkREolwkLbiRwAwz+4KCJ3m3AoaWaVUiIiIlFMkoyulm1haIDX202t1zyrYsERGRkiky4Mysj7t/bGY/P2TWGWaGu79VxrWJiIgcs+JacD2Bj4ELDzPPAQWciIhEreKe6H1f6Kf620REpNI54ihKM/u1mdW3As+b2Wdm1q88ihMRETlWkVwmcK27bwf6AY0ouNB7dJlWJSIiUkKRBJyFfg4Axrt7dqHPREREolIkAbfIzD6kIOA+MLN6wP6yLUtERKRkIrnQexiQBHzh7rvNrBG60FtERKJcJBd67zezb4EOZhZJIIqIiFS4SB6X8whwBbCC/7sHpQOflGFdIiIiJRJJi+xiIFa35xIRkcokkkEmX1DwPDgREZFKI5IW3G4gy8ymA+FWnLvfVmZViYiIlFAkAfdO6CUiIlJpRDKKcpyZ1QZauvvqcqhJRESkxCK5F+WFQBYwNTSdZGZq0YmISFSLZJDJKKALsBXA3bOANmVWkYiISCmIJOD2ufu2Qz7TrbpERCSqRTLIJNvMfgFUN7O2wG3A3LItS0REpGQiacHdCnSk4BKBV4DtwO1lWJOIiEiJRTKKcjdwd+glIiJSKURyL8pU4A9A68LLu3tC2ZUlIiJSMpH0wU0ERgLL0OASERGpJCIJuO/dXde9iYhIpRJJwN1nZs8Dh96L8q0yq0pERKSEIgm4oUB7Cp4ocOAUpQMKOBERiVqRBFxnd48t80pERERKUSTXwc01sw5lXomIiEgpiqQFl0bB8+DWU9AHZ4DrMgEREYlmkQRc/zKvQkREpJQVGXBmVt/dtwM7yrEeERGRUlFcC24ScAGwiIJRk1ZonqNH5oiISBQrMuDc/YLQz9PKrxwREZHSUdwpyk7Frejun5V+OSIiIqWjuFOUfwn9rAWkAksoOE2ZAGQC6WVbGphZf+ApoDrwvLuPLut9iohIMBR5HZy793b33sA3QCd3T3X3FCAZ+KqsCzOz6sDfgfOADsAgXY8nIiKRiuRC71h3X3Zgwt2XA2eWXUlhXYC17v6Fu+cCrwI/K4f9iohIAERyHdzS0M2WJ4SmrwKWll1JYc2B/1doeiNw1qELmdlwYDhAy5Yty6EsERGpDCJpwQ0FsoFfh14rQp9FBXd/LnT6NLVJkyYVXY6IiESJI7bg3H0v8GToVZ6+AloUmj6Vcuj7ExGRYIikBVdRPgXamtlpZlYTuBLQg1dFRCQikfTBVQh3zzOzW4APKLhMYKy7Z1dwWSIiUklEbcABuPv7wPsVXYeIiFQ+Rww4M5tMwb0nC9tGwcXeY0J9dCIiIlElkj64L4CdwD9DrwNPGGgXmhYREYk6kZyi7OrunQtNTzazT929s5mpT0xERKJSJC24umYWvoI69L5uaDK3TKoSEREpoUhacL8FZpvZOgputnwacJOZHQ+MK8viREREjlUkF3q/b2Ztgfahj1YXGljyP2VVmIiISElEeplACtA6tHyimeHu48usKhERkRKK5DKBl4HTgSwgP/SxAwo4ERGJWpG04FKBDu5+6LVwIiIiUSuSUZTLgaZlXYiIiEhpiqQF1xhYYWYLgZwDH7r7RWVWlYiISAlFEnCjyroIERGR0hbJZQL/KY9CRERESlORAWdms929u5nt4OCbLRvg7l6/zKsTERE5RkUGnLt3D/2sV37liIiIlI7iWnANi1vR3X8s/XJERERKR3F9cIsoODVpQEtgS+j9CcCXFNyTUkREJCoVeR2cu5/m7m2AacCF7t7Y3RsBFwAflleBIiIixyKSC73T3P39AxPuPgXoWnYliYiIlFwk18F9bWb3ABNC01cBX5ddSSIiIiUXSQtuENAE+HfodVLoMxERkagVyYXePwK/LodaRERESk1xlwlM5uALvA+ie1GKiEg0K64F93jo588peJrAgT64QcC3ZVmUiIhISRV3J5P/AJjZX9w9tdCsyWaWWeaViYiIlEAkg0yON7M2BybM7DTg+LIrSUREpOQiuUzgDmCmmX1BwZ1MWgHXl2lVIiIiJRTJKMqpZtYWaB/6aJW75xS3joiISEU74ilKM6sDjARucfclQEszu6DMKxMRESmBSPrgXgRygfTQ9FfAg2VWkYiISCmIJOBOd/dHgX0A7r6bgr44ERGRqBVJwOWaWW1CF32b2emA+uBERCSqRTKK8j5gKtDCzCYC3YAhZVmUiIhISUUyivIjM/sMSKPg1OSv3X1zmVcmIiJSApG04AB6At0pOE1Zg4KnCoiIiEStSC4T+AdwA7AMWA5cb2Z/L+vCRERESiKSFlwf4Ex3PzDIZByQXaZViYiIlFAkoyjXAi0LTbcIfSYiIhK1InkeXD1gpZktDE2fBSwsn/JERESOTSTPgxMREal0jvg8uAPMrH5xy4uIiESTIwaWmQ0H/gTsBfZTcC2cA22KW09ERKQiRdIiGwnE6eJuERGpTCIZRbkO2F3WhYiIiJSmSFpwdwFzzWwBhW6y7O63lVlVIiIiJRRJwI0BPqbgTib7y7YcERGR0hFJwNVw99+UeSUiIiKlKJI+uClmNtzMmplZwwOvMq9MRESkBCJpwQ0K/byr0Ge6TEBERKJaJM+DO608ChERESlNRZ6iNLPfFXp/+SHzHirLokREREqquD64Kwu9v+uQef3LoBYREZFSU1zAWRHvDzctIiISVYoLOC/i/eGmRaLChg0beO6558p0H6NGjWLEiBFlug8RKbniAi7RzLab2Q4gIfT+wHR8OdUnclSOFHB5eXnlWE3RDldHfn5+BVQiElxFBpy7V3f3+u5ez91jQu8PTNcozyJFDmf37t1cfvnldOjQgcTERAYOHMjNN9/MihUrSEpK4rLLLgOgdevW3HnnnXTp0oXrr7+eTZs20bt3b1JSUujYsSO/+114PBXNmzfnu+++A2DAgAGcf/75AHz33Xeceuqp4eW+/PJL+vTpQ/v27bn00kvZtm0bALm5uYwcOZIuXbqQmJjI1Vdfzc6dOwEYMmQI1113HT169CA1NZWZM2eSkJDA0KFDSUpKYuLEiTRr1oy9e/eG93PRRRcxadKksv0iRQIqkgu9y52ZXW5m2Wa238xSK7oeiU4ffPAB27dvZ8WKFSxZsoQxY8bw97//nQ4dOpCVlcUbb7wRXnb79u0sXLiQF154gRNOOIHJkyezaNEisrKyyMzMZOrUqQD07t2bjz/+mH379rF+/XrWr1/Pvn37mD59Or179w5vb9asWbzyyiusWrWKBg0a8MADDwDw6KOP0qBBAxYuXMiSJUs45ZRTePjhh8PrZWVlMXXqVLKysgDIzs5m+PDhZGVlMXjwYHr27Mlrr70GFLRGMzMzw0EtIkcnWh9guhz4OQX3wRQ5rMTERFauXMnNN99Mr169wq2twxk8eHD4fX5+PiNHjmTu3Lm4O5s2bSIrK4v+/fuTkZHBtGnTaN68OWlpabg7CxYsYNq0afTp0ye8jQsuuICTTz4ZgGHDhnHrrbcC8M4777B9+/ZwuObk5JCYmBhe77LLLuP4448PT7dt25b09PTw9G233cYdd9zBNddcw7PPPsu1115LzZo1S/hNiVRNURlw7r4SwEyDNaVobdq0ITs7m+nTpzNlyhT+8Ic/8Ne//vWwy9atWzf8/oknnmDLli0sWLCAWrVqMXz48PBpwT59+vCnP/2JU089lYyMDNyd6dOnM336dO67774j1uTu/OMf/zgoDIuq43DTXbt2JT8/nzlz5vDSSy/x6aefHnGfInJ4UXmK8miE7pOZaWaZ33//fUWXI+Vo48aNVK9enYsvvpgnn3yS77//nvr164f7w4qydetWmjVrRq1atfjqq694++23w/NatWpF9erVGTduHBkZGWRkZPDSSy9Ro0YNWrZsGV7uvffe48DftxdffDEcaBdddBFPPPEEe/bsAWDHjh2sXLnyqI7r1ltv5corr6Rr1660aNHiqNYVkf9TYQFnZtPMbPlhXj87mu24+3PunuruqU2aNCmrciUKLVu2jPT0dBITE+nSpQt33XUXXbp0ITY2lri4uCL7rm677TbmzJlDXFwcw4YNIyMj46D5GRkZ1KlTh2bNmnHKKadQu3btn7TIevTowZVXXkn79u358ccf+eMf/wjAnXfeSWJiIp07dyYhIYHu3bsfdcBdeeWVbNmyhZtuuumo1hORg5l79F7SZmYzgRHunhnJ8qmpqZ6ZGdGiIlFr9uzZ3HDDDSxbtkyn6UWKYGaL3L3YQYhR2QcnUlUNGzaMjz76iPHjxyvcREooKgPOzC4B/go0Ad4zsyx3P7eCyxIpcy+88EJFlyASGFEZcO7+b+DfFV2HiIhUXpV+FKWIiMjhKOBEqgjd61KqGgWcSJQwM+677z6SkpKIjY3lzTffDM9bsGBB+P6ZKSkpvPfee0DBTZvPPfdcUlNT6dixI0OHDiU3NxeAl156ib59+3LJJZcQFxfHsmXLKuS4RCpKVPbBiVRV1atXJysri9WrV9O1a1d69OhBzZo1ueGGG3j//fdp1qwZ33zzDZ07d2b58uU0aNCASZMm0ahRI9yda665hrFjx3LDDTcAMH/+fJYsWcLpp59ewUcmUv4UcCJRZNiwYQDExsbSqVMn5s+fT0xMDOvXr+e8884LL2dmrF27luTkZB5//HGmTJlCfn4+W7ZsoU6dOuHlunfvrnCTKksBJxLl3J2EhAQ++eSTn8x7+eWXmT17NrNmzaJevXo89NBDfP755+H5h97rUqQqUR+cSBR58cUXAVizZg2LFy8mLS2Nrl27smbNGmbMmBFe7tNPP8Xd2bp1K40bN6ZevXps27ZNz44TKUQtOJEokpeXR3JyMrt372bMmDGcdNJJQMFjeEaOHMntt99Obm4ubdq0YfLkyQwePJi3336b9u3bc9JJJ9GjR4/wjZ5Fqrqovhfl0dK9KKUyMzN27Nih04oiEYjkXpQ6RSkiIoGkU5QiUSJIZ1NEooFacCIiEkgKOBERCSQFnIiIBJICTkREAkkBJyIigaSAExGRQFLAiYhIICngREQkkBRwIiISSAo4EREJJAWciIgEkgJOREQCSQEnIiKBpIATEZFAUsCJiEggKeBERCSQFHAiIhJICjgREQkkBZyIiASSAk5ERAJJASciIoGkgBMRkUBSwImISCAp4EREJJAUcCIiEkgKOBGJWjNnziQ1NbWiy5BKSgEnIiKBpIATkVJ11VVXkZqaSnx8PJdccglbtmxh5syZJCYmMnjwYDp27EiXLl1YsWIFQLHzDvX+++/TrVs3UlJSSE9PZ/78+eV5aFLJKOBEpFQ99dRTZGZmsmzZMjp27MgjjzwCwNKlSxk2bBjZ2dncfPPNDB48OLxOcfMOWLduHQ888ABTpkxh0aJFPP/88wwcOLDcjksqn5iKLkBEgmX8+PFMnDiR3Nxcdu3aRbt27ejfvz9nnHEGPXv2BODqq69m+PDhbN++HaDYeQd88MEHrFu3jrPPPjv8WV5eHt9++y0nn3xyOR2dVCYKOBEpNbNmzeKZZ55h7ty5NGnShEmTJvHcc8+Vyrbdnf79+zN+/PhS2Z4En05Rikip2bp1Kw0aNKBRo0bk5OQwduzY8Lx169Yxa9YsACZNmkR8fDz169c/4rwD+vXrx9SpU8nOzg5/9umnn5b1IUklphaciJSa/v37M2HCBNq1a0fjxo05++yzWbhwIQDx8fE8//zz3HjjjdSpU+egllhx8w5o27YtEyZMYNiwYezZs4fc3Fy6detG586dy+34pHIxd6/oGkpNamqqZ2ZmVnQZInKImTNnMmLECA7377O4eSJFMbNF7l7sRZI6RSkiIoGkFpyIiFQ6asGJRMDM+POf/0znzp1p06YN06dP56677iI5OZm4uDhWrlwJwKZNm+jduzcpKSl07NiR3/3ud+FtjBo1ikGDBjFgwADat2/P+eefz+7duwHYtm0bl156Ke3btycjI4PBgwczYsQIAHbu3MnQoUOJi4sjLi6ORx99tPy/AJGAUsCJACeccAKffvopjzzyCD/72c/o1q0bixcvZvDgwfz5z38OLzN58mQWLVpEVlYWmZmZTJ06NbyNzMxMJk2axMqVK9m3bx8TJ04E4E9/+hMnnngiq1at4l//+ld4tCDAAw88wP79+1m2bBlz585l3LhxTJkypXwPXiSgFHAiwBVXXAFAp06dMDMuuOACAH7/+9+zevVqAPLz8xk5ciSJiYmkpKSwfPlysrKywts499xzOeGEEzAzzjrrLNatWwfAjBkzGDp0KAANGzbk4osvDq8zbdo0fvWrX2Fm1K9fn0GDBjFt2rRyOGKR4NNlAiJArVq1AKhevTrHHXfcQfPy8vIAeOKJJ9iyZQsLFiygVq1aDB8+nL179/5kGwe2s2fPnnKoXESKohacSIS++OIL5s+fT48ePejQoQOvvvpqeN7999/P/Pnzw/14X3zxBbNmzSI5OZkvv/ySp556Cii4EHrcuHG89NJLxMXFkZuby7PPPou7c9ddd/HYY4/xySef/KQfT0SOngJOJAJ5eXlkZmZSr1499uzZwymnnALA5s2bw8scd9xx4X68V199lVNOOYXFixdz6623Mm/ePNq3b8/ZZ59Nfn4+119/PcuWLSM+Pp4lS5YQHx/PCy+8QI0aNfjoo49+0o8nIkdPASdVnrtTt25dAFq3bn1QaEHB/RU///xz1q1bR/Xq1YmJiWHz5s00bNiQjIyM8HKvvfYaUNCPd9xxx/Hvf/8bgLS0NJo1a8aqVavo2bMntWvXpnfv3pgZN910EzVr1mT58uXcdNNNXHnllYftxxORo6c+OJEIuDuNGzc+aFDJoYrqx9u9ezfZ2dkkJSXx5ZdfkpycTN++fYvdxoHtqB9P5NipBScSgdjYWOrUqcPLL78c/mzVqlU/eaTL4Zx44omceeaZZGVlMXHiRL777jt27NiBu/P8889zzjnnlGXpIlVWVAacmT1mZqvMbKmZ/dvMTqjomqRqi4mJYfLkybz66qskJCTQsWNHbrrpJnJzc49qO+eddx6//OUvSU9PJz4+HoB77rmnLEoWqfKi8lZdZtYP+Njd88zsEQB3//2R1tOtukREqoZKe6sud//Q3fNCk/OBUyuyHhERqXyiMuAOcS1Q5L2LzGy4mWWaWeb3339fjmWJiEg0q7BRlGY2DWh6mFl3u/vboWXuBvKAIi8GcvfngOeg4BRlGZQqIiKVUIUFnLsffpx0iJkNAS4AMjwaOwpFRCSqReV1cGbWH/gd0NPdda8iERE5atHaB/c3oB7wkZllmdmzFV2QiIhULlHZgnP3Myq6BhERqdyitQUnIiJSIgo4kVJmZuzcufMnn997773hGzJH4tlnn+XJJ58szdJEqpSovJPJsdKdTCQamBk7duwIP6FAREpfpb2TiUgQ7N+/nzvuuINBgwaRk5PDkCFD+Nvf/gZA8+bN+e677wAYMGAA559/PgDfffcdp55acOOeUaNGMWLEiIj3V1TLUaSqUsCJlIG9e/cycOBAYmJimDRp0kGPzwHo3bs3H3/8Mfv27WP9+vWsX7+effv2MX36dHr37l1BVYsEiwJOJCQ/P7/UttW/f3/S0tJ47LHHMLOfzM/IyGDatGnMnz+ftLQ0unTpwoIFC5g2bRp9+vQ55v0+9thjJCUlERsby5tvvhn+/KqrriI1NZX4+HguueQStmzZAsDMmTNJSkri+uuvJyEhgcTERFauXAnApk2b6N27NykpKXTs2JHf/e534e2NGjWKQYMGMWDAANq3b8/555/P7t0Fl6xOnz6d9PR0kpOTiY+P59VXXz3m4xEpCQWcVAlTp04lOTmZhIQEMjIyWLt2LTNnziQhIYGhQ4eSlJTElClTmDRpEmeddRbJyckkJyczffr08DZat27NvffeS3p6Oq1btw6fboSCp37Hx8eTkJAAwOeff84bb7zB7t27Wb16Needdx7vvvsuDz/8MC+++CJ9+vRh+vTpTJ8+nYyMDDIyMg6aPlbVq1cnKyuLd955h+HDh4dPgz711FNkZmaybNkyOnbsyCOPPBJeJzs7mxtuuIGlS5cycOBAHnzwQQBOOOEEJk+ezKJFi8jKyiIzM5OpU6eG18vMzGTSpEmsXLmSffv2MXFiwR31OnXqxOzZs1m8eDHTpk1jxIgR4UAVKU9ReR2cSGn67rvvuPrqq/nPf/5Dhw4deOGFF7jqqqt45JFHyM7OZsyYMaSnpwPwww8/MGjQIMyM1atXk5GRwcaNG8Pb2r17N/PmzWPDhg3ExcUxZMgQatSowaBBg3jllVfo0aNHeJBJeno6/fr1Y+fOnbz66quMHj2auLg4Ro8eTXp6OtWrV2fcuHHMnTsXd+fee++lRo0atGzZ8piPddiwYUDBA1o7derE/Pnzueiiixg/fjwTJ04kNzeXXbt20a5du/A6sbGxJCcnA5CWlsbkyZOBghbtyJEjw/Vt2rSJrKws+vfvD8C5557LCSecAMBZZ53FunXrAPj++++59tprWbNmDTExMfz444+sXr2atLS0Yz4ukWOhgJPAW7BgAYmJiXTo0AGAoUOHctNNN7Fjxw7atm0bDjeAdevWMWjQIL766itq1KjBpk2b2LRpE02bFtwX/MorrwQKWnMnnngiGzduJDc3l9q1a9OjR4/wdho0aMCwYcOoXbs2o0eP5rLLLuPrr79m2rRpxMTEsHLlSjIyMpg9ezbNmjUD+Mk2SsusWbN45plnmDt3Lk2aNGHSpEk899xz4fm1atUKv69evTp5eQVPqnriiSfYsmULCxYsoFatWgwfPpy9e/cWud6ePXsAuPHGG7nooot46623MDPatWt30Hoi5UUBJ1XaoUP5Bw0axF/+8hcuvvhi9u/fT506dYr9T/1AGBTm7px44olAQd/XhAkTWL58+U+Wu+SSSw6aXrFixUHTo0aNOurjefHFF7nnnntYs2YNixcvJi0tjQULFtCgQQMaNWpETk4OY8eOjWhbW7dupVmzZtSqVYuvvvqKt99+mxtvvDGi9Vq3bo2Z8dFHH7F27dqjPg6R0qA+OAm8tLQ0lixZwqpVqwAYN24cycnJ1KtX7yfLbt26ldNOOw2AsWPHkpOTc8Ttx8bGsnv3bubMmQPA22+/zdatW8Pz6tSpw8svvxxeftWqVWzfvr2kh3VYeXl5JCcnc8EFFzBmzBhOOukk+vfvz+mnn067du3o2bMnnTp1imhbt912G3PmzCEuLo5hw4ZF3Dc4evRoRowYQVJSEq+//nq4X1Kk3Ll7YF4pKSkucjhTpkzx5ORkj4+P9z59+viaNWt8xowZfujfmfHjx3vr1q09OTnZ77rrLm/UqJGvX7/e3d1btWrly5YtCy9beHrmzJkeFxfn8fHxfsstt/hJJ53kX375pbu7f/755z5gwACPj4/3Dh06eO/evf37778vnwMXCSgg04+QCbqTiUgp2LFjR7hFOGPGDIYMGcL69eupVk0nSUTKQiR3MlEfnEgpePPNN3nyySfZv38/tWrVYtKkSQo3kQqmFpyIiFQ6uheliIhUWQo4EREJJAWciIgEkgJOREQCSQEnIiKBpIATEZFAUsCJiEggKeBERCSQFHAiIhJICjgREQkkBZyIiASSAk5ERAJJASciIoGkgBMRkUBSwImISCAp4EREJJAUcCIiEkgKOBERCSQFnIiIBJICTkREAkkBJyIigaSAExGRQFLAiYhIICngREQkkBRwIiISSAo4EREJJAWciIgEkgJOREQCSQEnIiKBpIATEZFAUsCJiEggKeBERCSQFHAiIhJICjgREQkkBZyIiASSAk5ERAJJASciIoGkgBMRkUBSwImISCAp4EREJJAUcCIiEkhRGXBm9oCZLTWzLDP70MxOqeiaRESkconKgAMec/cEd08C3gXureB6RESkkonKgHP37YUmjwe8omoREZHKKaaiCyiKmf0ZGAxsA3oXs9xwYHhoMsfMlpdDedGoMbC5oouoIFX52KFqH7+OveqKPdIC5l4xjSMzmwY0Pcysu9397ULL3QXUcvf7IthmprunlmKZlYaOvWoeO1Tt49exV81jh8iOv8JacO7eN8JFJwLvA0cMOBERkQOisg/OzNoWmvwZsKqiahERkcopWvvgRptZLLAf+C9wQ4TrPVd2JUU9HXvVVZWPX8dedR3x+CusD05ERKQsReUpShERkZJSwImISCAFLuCq8m2+zOwxM1sVOv5/m9kJFV1TeTGzy80s28z2m1mVGDptZv3NbLWZrTWzOyu6nvJkZmPN7LuqeN2rmbUwsxlmtiL0d/7XFV1TeTGzWma20MyWhI79/mKXD1ofnJnVP3AnFDO7Dejg7pEOUqnUzKwf8LG755nZIwDu/vsKLqtcmNmZFAxKGgOMcPfMCi6pTJlZdeBz4BxgI/ApMMjdV1RoYeXEzM4GdgLj3T2uouspT2bWDGjm7p+ZWT1gEXBxVfizNzMDjnf3nWZWA5gN/Nrd5x9u+cC14Krybb7c/UN3zwtNzgdOrch6ypO7r3T31RVdRznqAqx19y/cPRd4lYJLaqoEd/8E+LGi66gI7v6Nu38Wer8DWAk0r9iqyocX2BmarBF6Ffl/fOACDgpu82Vm/w+4iqp7o+ZrgSkVXYSUmebA/ys0vZEq8p+c/B8zaw0kAwsquJRyY2bVzSwL+A74yN2LPPZKGXBmNs3Mlh/m9TMAd7/b3VtQcBeUWyq22tJ1pGMPLXM3kEfB8QdGJMcuUlWYWV3gTeD2Q85cBZq754eeNHMq0MXMijxFHa0XeherKt/m60jHbmZDgAuADA9YB+tR/LlXBV8BLQpNnxr6TKqAUP/Tm8BEd3+rouupCO6+1cxmAP2Bww42qpQtuOJU5dt8mVl/4HfARe6+u6LrkTL1KdDWzE4zs5rAlcA7FVyTlIPQQIsXgJXu/kRF11OezKzJgdHhZlabgkFWRf4fH8RRlG9S8BiF8G2+3L1K/GZrZmuB44AfQh/Nr0IjSC8B/go0AbYCWe5+boUWVcbMbADwP0B1YKy7/7liKyo/ZvYK0IuCR8Z8C9zn7i9UaFHlxMy6A7OAZRT8PwfwB3d/v+KqKh9mlgCMo+DvfDXgdXf/U5HLBy3gREREIICnKEVEREABJyIiAaWAExGRQFLAiYhIICngREQkkBRwIiISSAo4EREJJAWciIgEkgJOREQCSQEnIiKBpIATEZFAUsCJiEggKeBERCSQFHASOGaWb2ZZhV53HsW6vczs3RLsu8j1zWyDmTUOvZ97rPs4wv5nmllq6P37B56dVZ7M7E9mVuKH05pZIzObYWY7zexvpVGbVC2V8oneIkewJ/RI+6jl7l3LYR8DynofRez33lLa1F7gj0Bc6CVyVNSCkyoj1IJ6ONSqyzSzTmb2gZmtM7PCD4atb2bvmdlqM3vWzKqF1u9nZvPM7DMz+5eZ1Q193t/MVpnZZ8DPC+2vkZl9aGbZZvY8YIXm7Qz97BVqdb0R2sbE0BObMbMBoc8WmdnTh2sZmlltM3vVzFaa2b+B2occb2Mzax3azktm9nloH33NbI6ZrTGzLqHljzezsWa20MwWm9nPQp8PMbO3zGxqaPlHQ59XD21zuZktM7M7Qp+/ZGaXhd5nhLa1LLTt4wrVdn/ou1xmZu0PPTZ33+XusykIOpGjpoCTIKp9yCnKKwrN+zLUupsFvARcBqQB9xdapgtwK9ABOB34eejU4j1AX3fvBGQCvzGzWsA/gQuBFKBpoe3cB8x2947Av4GWRdSbDNwe2l8boFtou2OA89w9hYInlR/OjcBudz8ztL+UIpY7A/gL0D70+gXQHRgB/CG0zN3Ax+7eBegNPGZmx4fmJQFXAPHAFWbWIvRZc3ePc/d44MXCOwwdw0vAFaH5MaF6D9gc+i6fCdUhUqoUcBJEe9w9qdDrtULz3gn9XAYscPcd7v49kFOov2qhu3/h7vnAKxQEQRoFATTHzLKAa4BWFITFendf4+4OTCi0r7MPTLv7e8CWIupd6O4b3X0/kAW0Dm33C3dfH1rmlSLWLbyPpcDSIpZb7+7LQvvIBqaH6l0W2h9AP+DO0PHNBGrxf6E83d23ufteYEXo2L8A2pjZX82sP7D9kH3Ghvb7eWh6XKjeA94K/VxUqAaRUqM+OKlqckI/9xd6f2D6wL8HP2Qdp+D04kfuPqjwDDNLKsWaAPIpm3+Xhx5r4e/hwP4MuNTdVxde0czOOlyN7r7FzBKBc4EbgIHAtcdQU1kds1RxasGJ/FQXMzst1Pd2BTAbmE/BqcMzINxf1Q5YBbQ2s9ND6xYOwE8oOBWImZ0HnHgUNaymoHXUOjR9RRHLFd5HHJBwFPs41AfArYX6AJOLWzh02raau79JwenbTocsspqC7+aM0PTVwH9KUJ/IUdFvTRJEtUOn2Q6Y6u4RXyoAfAr8jYJ+qxnAv919v5kNAV45MFACuMfdPzez4cB7Zrabgr69eqH594eWzwbmAl9GWoC77zGzm4CpZrYrVNPhPAO8aGYrgZUUnO47Vg8A/wMsDYX7euCCYpZvHtr3gV+U7yo80933mtlQ4F9mFkPBMTx7NAWZ2QagPlDTzC4G+rn7iqPZhlRdVnAaXkSijZnVdfedoRbV34E17v5kRdclUlnoFKVI9PpVqCWaDTSgYFSliERILTgREQkkteBERCSQFHAiIhJICjgREQkkBZyIiASSAk5ERALp/wOPJoXhWZvdzgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(1, 1, figsize = (6, 6))\n",
    "fig.tight_layout()\n",
    "plt_embedding(embedding2, ax)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a043f7f-7efb-4212-97a4-6da341e3bcb0",
   "metadata": {},
   "source": [
    "Qualitatively, this embedding again separates the fruits and vegetables. However, the coordinates are all changed. This is problematic because any downstream applications that use the embedding coordinates must now also be updated. In particular, any machine-learning models that rely on the coordinates to make predictions must now be retrained with the new coordinates."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87c6a9cf-c637-47ad-8de4-804e20f89a4e",
   "metadata": {},
   "source": [
    "## Expanding an embedding with constraints"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "463a73dc-5969-43dc-953c-c264859637ad",
   "metadata": {},
   "source": [
    "We would like to train a new embedding for the expanded vocabulary, but ensure that the previously-trained elements remain at the previously-defined coordinates. For this, we introduce a new training function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "8622db9a-8371-4bda-a406-8ebe1268bcb2",
   "metadata": {},
   "outputs": [],
   "source": [
    "def update_vocab_embedding(train_loader=None, epochs=10,\n",
    "                           prev_model=None, **kwargs):\n",
    "    model = VocabModule(**kwargs)\n",
    "    constraint = EmbeddingConstraint(model.embedding, prev_model.embedding)\n",
    "    trainer = pl.Trainer(gpus=0, \n",
    "                         max_epochs=epochs,\n",
    "                         callbacks = [constraint],\n",
    "                         gradient_clip_val=5,\n",
    "                         enable_progress_bar=False)    \n",
    "    trainer.fit(model, train_loader)\n",
    "    model.to(device)\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2efe919-8935-4379-9e3a-c3596cfad82f",
   "metadata": {},
   "source": [
    "The training function defines an `EmbeddingConstraint` object with two arguments. The first argument is the `Embedding` component in the new model. The second argument is a pre-existing embedding. The constraint is provided to the pytorch-lightning trainer as a callback. As the trainer will carry out training, the callback will maintain parts of the model embedding fixed."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea2d1d3e-e7fb-4509-9817-52f8aba465b9",
   "metadata": {},
   "source": [
    "We can now train a model with the new training function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "9058fb1b-096f-4a15-94de-342a327b5220",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GPU available: False, used: False\n",
      "TPU available: False, using: 0 TPU cores\n",
      "IPU available: False, using: 0 IPUs\n",
      "HPU available: False, using: 0 HPUs\n",
      "\n",
      "  | Name      | Type             | Params\n",
      "-----------------------------------------------\n",
      "0 | embedding | Embedding        | 32    \n",
      "1 | output    | Linear           | 80    \n",
      "2 | criterion | CrossEntropyLoss | 0     \n",
      "-----------------------------------------------\n",
      "112       Trainable params\n",
      "0         Non-trainable params\n",
      "112       Total params\n",
      "0.000     Total estimated model params size (MB)\n"
     ]
    }
   ],
   "source": [
    "model3 = update_vocab_embedding(produce_loader2, \n",
    "                                prev_model=model,\n",
    "                                vocab_size=len(produce_dataset2.vocab))\n",
    "embedding3 = make_embedding_table(produce_vocabulary2, model3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cfb01e7a-afed-4dcd-996d-96b4bfe44e6f",
   "metadata": {},
   "source": [
    "Finally, we can visualize the result. For ease of comparison, the first plot below reproduces the first embedding, and the second plot shows the new version."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "c8c44a92-eefe-4504-8bcd-9ccf657bd24a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2gAAAHKCAYAAAB2XWjxAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAABTNElEQVR4nO3deXxU9dXH8e8xgIAgCogiqGjVIGQlAQmCIFFExK1u4IJQWkrVulRs1VrBrY9btda2ilplM1q3PooCKgiPIIIECTuIuIIbsgiILAnn+WNupgGzDJrJXDKf9+t1X8zdzx1xDuf+fvd3zd0FAAAAAEi8fRIdAAAAAAAgggINAAAAAEKCAg0AAAAAQoICDQAAAABCggINAAAAAEKCAg0AAAAAQoICDaiEmd1kZo9X97YxHMvN7OjqONZPZWYDzWzGj9y3TXAtdao7LgBAfJhZDzNbVdP7xoOZjTKzO37kvj86/wE/BQUakkbwQ7vQzLaY2Zdm9rCZHVDZPu7+Z3f/ZSzH35NtAQDJycw+NrPvzWxzmenviY4LQHhQoCEpmNl1ku6WdL2kJpI6SzpC0htmVq+CfWj1SSC+fwC12Bnu3qjMdGWiA0L1Iofhp6BAQ61nZvtLulXSb919krvvcPePJV0gqY2kS4LtRpjZ82Y2zsw2ShoYLBtX5lgDzOwTM1trZn8K7oSeXGb/ccHn0q59l5nZp2b2jZn9scxxOpnZO2a2wcy+MLO/V1Qo7nYtF5pZ4W7LrjWzl4PPTcxsjJmtCeK82cz2KbPtr8xsqZltMrMlZtYhWH6Dma0ss/ycH57a/m5m35rZMjPLL7Mi+h3s/j2UE/+gMuf/0Mx+XWZdDzNbZWZ/MLMvJT1pZovM7Iwy29QNvsvsqr4rANjbBD07Xigzf7eZTbGI0t/Im4LfwY/N7OIy255uZvPMbKOZfWZmI8qsqyonNQi6Aq43syWSOu4W16Fm9kKQWz4ys6ti3bec67tvt2Uvmdnvgs/Hmdm0IDcuNrMzdzvPX4Lc9q2ZzTCzBsG65yzSM+ZbM3vLzNrvdurmZvZGkHv+z8yO2O17qVPmPNPMrNzeMGb2YPDdbjSzuWbWrcy63f8NcYNFeuw0K7NNh+A7rFvRdwRIFGhIDl0k1Zf0YtmF7r5Z0gRJp5RZfJak5yUdIOmpstubWTtJ/5R0saSWirTEtari3F0lpUrKl3SLmR0XLC+RdK2k5pLygvWXx3At4yWlmtkxZZZdJKkg+PxQENdRkrpLGiBpUBD/+ZJGBMv2l3SmpLXBfisldQv2vVXSODNrWeYcxwfbNJc0XNKLZtY0hnh397WkvsH5B0l6oLRIDBwiqakirZtDJI1RUEAH+kj6wt3n/YhzA0DYXScp3SJd8rtJGizpMnf3YP0hivwOt5J0maRHzSw1WPedIr/vB0g6XdJvzOzs3Y5fUU4aLulnwXRqcGxJkkVu8o2XND84b76ka8zs1Kr2LcfTki40MwuOfaCkXpKeCYqW8ZJel9RC0m8lPVXm+u6TlKNITm8q6feSdgbrJko6JtjvPe2WvxXJ27cH311ROetjNUdSVnD+AknPmVn9MuvL/hviL5KmKXIzuNSlkp5x9x0/8vxIEhRoSAbNJX3j7sXlrPsiWF/qHXf/X3ff6e7f77bteZLGu/sMd98u6RZJrsrd6u7fu/t8RZJbpiS5+1x3n+XuxUFr3khFCqpKufsWSS9J6i9JQaHWVtLLZpYiqZ+kG919U3DcvyiSECTpl5Lucfc5HvGBu38SHPc5d/88uO5/S1ohqVOZU38t6a9B6+O/JS1X5B8Ae8TdX3X3lcH5/0+RRNytzCY7JQ13923B9z9OUh+LtIIquJaxe3peAAiZ/w1aiUqnX0nR3/hLJd2vyO/fb9199wE3/hT8Rv6fpFcVFADuPs3dFwa/4wsUKYZ2zyvl5qTgGHe6+zp3/0zS38rs01HSQe5+m7tvd/cPJT2mSL6pat/dTVckb5b+7p+nSN79XJFHDxpJuis4z5uSXpHUPygSfyHpandf7e4l7j7T3bcF1/5EkPe2KXIjMtPMmpQ576vu/law/o+S8szssEriLJe7j3P3tUHu/oukfRUpeEvt/m+I0fpvL50URXI3OQxVokBDMvhGke4N5fUHbxmsL/VZJcc5tOz6IJGurXhzSdKXZT5vUST5yMyONbNXgi4ZGyX9WbsWipUpUFCgKdJ69r9BLM0l1ZX0SZltP9F/W/kOU6QV7Acs0nWzqPQfC5LSdotndZk7uKXHPTTGeMue5zQzm2Vm64Lz9NntPGvcfWvpTJC035Z0rkUGdDlNP/7OJwCExdnufkCZ6bHSFe4+W9KHkkzSs7vtt97dvyszH/0tNrPjzWxq0IXuW0lD9cO8Um5O0m75TbvmkSMkHVq2oJR0k6SDY9h3F0EeeUa75rDS3/RDJX3m7jvL7FKaw5or0hPmBznMzFLM7C6LdNPfKOnjYFXZay+buzdLWqcfl8OGWaSb/rfB99CkovMEXpLUzsyOVKS3zrfu/u6enhfJhwINyeAdSdsk/bzsQjNrpMg/+KeUWVxZi9gXklqX2b+BpGYVb16phyUtk3SMu++vSLKzGPd9Q9JBZpalSJIr7d74jaQdiiTTUodLWh18/kyRLii7CPriPybpSknN3P0ASYt2i6dVaZeUMsf9PPj8naSGZdYdUl7QZravpBcU6aZycHCeCbudp7zvv/QO5PmK3J1cXc42AFArmNkVirTMfK5IN76yDjSz/crMl/0tLpD0sqTD3L2JpEcUe175QpGbeGWPW+ozSR/tVlA2dvc+MexbnqclnRfknuMVyQsKruMwK/PctP6bw76RtFXl5DBFiryzJJ2sSMHUJlhe9tqj8QW5v2lwvtJiN5Yc1k2R/x4XSDowyGHfqpIcFtxwfFaRHEYPEMSMAg21nrt/q8hzVQ+ZWW+LDDTRRpEfzVWK/QfzeUlnmFkXiwzoMUKxJ7/dNZa0UdJmM2sr6Tex7hj0XX9O0r2KJJk3guUlilzTnWbWOEh+v1Okm4wkPS5pmJnlWMTRwTb7KZJU1kiRgTwUaUErq4Wkq4Lv7nxJxylSXEmR/vz9gnW5inRZKU89Rf7RsUZSsZmdpsizB1X5X0kdJF2tyDNpAFArmdmxku7Qf/9B//vgZlxZt5pZvaBg6KtIPpAieWWdu281s06KFC6xelbSjWZ2oJm1VuT5r1LvStpkkQGcGgQtVmlm1jGGfX8geIb4G0Vy0mvuviFYNVuRVr3fB/mkh6QzFHlma6ekJyTdb5EBS1LMLC+48ddYkZuwaxUptP5czmn7mFnXIHffLmmWu3/m7msUKQAvCY75C5VfBCo4T7EiOayOmd2iyPPUVRkjaaAiz31ToCEmFGhICu5+jyKtVPcpUhjNVuSuYH5pH/YYjrFYkcTzjCJ3DDcr8mxWTPvvZpgiyXOTIq1X/97D/QsUuVv43G7P1v1WkTuCH0qaEWz3RBD/c5LuDJZtUqTwaeruSxR5Vu0dSV9JSlekW2FZsxV5APub4BjnuXtp984/KZLQ1itSCBeoHO6+SdJViiTz9cH1v1zVhQb9+F+QdKR2G+gFAPZS423X96D9J+iGP07S3e4+391XKJK3xgaFiBTporhekdafpyQNdfdlwbrLJd1mZpsUeUZ69+6RlblVke6EHynybHC0kAhu/vVVZHCMj/Tf4qpJVftWojSHRfNF8Gz3GYr0bPlGkUG5BpS5vmGSFioyUMc6RV6ds48iBdAnihRaSyTNquB8w4P9crTr4FO/UuQVPGsltZc0s4KYX5M0SdL7wfm2qvLHIkqv621Fnq9+r/S5b6AqtutjJQBiFXST2KBIN8WPEhxOrRbcqTzW3S+pcmMAqIWCFqVx7t66ik0RMmb2pqQCd3880bFg70ALGrAHzOwMM2sYPANwnyJ38z5ObFS1m0WG8x8s6dFExwIAwJ4IuoJ20J73lEESC22BZmb1zexdM5tvkZcV3promABFHkT+PJiOkdTPaYaOm2Do6c8kTXT3txIdD7A7chWAipjZaEmTJV0TdPMHYhLaLo7BiHH7uftmi7y8cIYi778or28xAAA1jlwFAKhu5b0XKhSCVonNwWzdYApnNQkASErkKgBAdQttgSZF37o+V9LRkv4RvLxx922GSBoiSfvtt19O27ZtazZIAEAozJ079xt3P6imz0uuAgDEKpZcFdoujmWZ2QGS/iPpt+6+qKLtcnNzvbCwsMbiAgCEh5nNdffcBJ7/AJGrAACViCVXhXaQkLKClxhOldQ7waEAAFAuchUAoDqEtkAzs4OCu5EyswaSTpG0rNKdAACoQeQqAEB1C/MzaC0ljQ769u8j6Vl3fyXBMQEAUBa5CgBQrUJboLn7AknZiY4DAICKkKsAANUttF0cAQAAACDZUKABAAAAQEhQoAEAAABASFCgAQAAAEBIUKABAAAAQEhQoAEAAABASFCgAQAAAEBIUKABAAAAQEhQoAEAAABASFCgAQAAAEBIUKABAAAAQEhQoAEAAABASFCgAQAAAEBIUKABAAAAQEhQoAEAAABASFCgAQAAAEBIUKABAAAAQEhQoAEAAABASFCgAQAAAEBIUKABAAAAQEhQoAEAAABASFCgAQAAAEBIUKABAAAAQEhQoAEAAABASFCgAQAAAEBIUKABAAAAQEhQoAEAAABASFCgAQAAAEBIUKABAAAAQEhQoAEAAABASFCgAQAAAEBIUKABAAAAQEhQoAEAAABASFCgAQAAAEBIUKABAAAAQEhQoAEAAABASFCgAQAAAEBIUKABAAAAQEhQoAEAAABASFCgAQAAAEBIUKABAAAAQEhQoAEAAABASFCgAQAAAEBIUKABAAAAQEhQoAEAAABASFCgAQAAAEBIUKABAAAAQEhQoAEAAABASFCgAQAAAEBIUKABAAAAQEhQoAEAAABASFCgAQAAAEBIUKABAAAAQEhQoAEAAABASFCgAQAAAEBIUKABAAAAQEiEtkAzs8PMbKqZLTGzxWZ2daJjAgCgLHIVAKC61Ul0AJUolnSdu79nZo0lzTWzN9x9SaIDAwAgQK4CAFSr0LagufsX7v5e8HmTpKWSWiU2KgAA/otcBQCobqEt0MoyszaSsiXNLmfdEDMrNLPCNWvW1HhsAABI5CoAQPUIfYFmZo0kvSDpGnffuPt6d3/U3XPdPfeggw6q+QABAEmPXAUAqC6hLtDMrK4iCe8pd38x0fEAALA7chUAoDqFtkAzM5P0L0lL3f3+RMcDAMDuyFUAgOoW2gJN0gmSLpXU08yKgqlPooMCAKAMchUAoFqFdph9d58hyRIdBwAAFSFXAQCqW5hb0AAAAAAgqVCgAQAAAEBIUKABAAAAQEhQoAEAAABASFCgAQAAAEBIUKABAAAAQEhQoAEAAABASFCgAQAAAEBIUKABAAAAQEhQoAEAAABASFCgAQAAAEBIUKABAAAAQEhQoAEAAABASFCgAQAAAEBIUKABAAAAQEhQoAEAAABASFCgAQAAAEBIUKABAAAAQEhQoAEAAABASFCgAQAAAEBIUKABAAAAQEhQoAEAAABASFCgAQAAAEBIUKAB2GuVlJQkOgQAACpFrsKeokADEDdmpuHDhysrK0upqal64YUXoutmz56tk046STk5OcrJydGrr74qSSouLtapp56q3NxctW/fXoMGDdL27dslSaNGjdLJJ5+sc845R2lpaVq4cGFCrgsAUHuQqxA2dRIdAIDaLSUlRUVFRVq+fLm6dOmibt26qV69eho6dKgmTJigli1b6osvvlDHjh21aNEiNWnSRAUFBWrWrJncXZdddpmeeOIJDR06VJI0a9YszZ8/Xz/72c8SfGUAgNqCXIUwoUADEFeDBw+WJKWmpqpDhw6aNWuW6tSpo48++kinnXZadDsz0wcffKDs7Gzdd999mjhxokpKSrR+/Xo1bNgwul3Xrl1JeACAakWuQphQoAGoce6ujIwMvfXWWz9YN3bsWM2YMUPTp09X48aN9ec//1nvv/9+dH2jRo1qMlQAQJIiV5XPzLRp06YfXOMtt9yi9u3b68ILL4zpOI888oi+//57XXvttfEIc6/GM2gA4urJJ5+UJK1YsULz5s1T586d1aVLF61YsUJTp06Nbjdnzhy5uzZs2KDmzZurcePG+vbbb1VQUJCo0AEASYJc9dPddtttMRdnkjR06FCKswpQoAGIq+LiYmVnZ6tv374aOXKkWrRooQMPPFAvv/yybr31VmVmZuq4447TiBEj5O4aMGCANm3apLZt2+qMM85Qt27dEn0JAIBajly153bu3Klrr71W/fv317Zt2zRw4ED9/e9/lyS1atVKX3/9tSSpT58+Ov300yVJX3/9tVq3bi1JGjFihIYNG5aY4EPO3D3RMVSb3NxcLywsTHQYAAIVdYMA4sHM5rp7bqLjqAq5CgiXqnIVuWxXZqY1a9Zo6NChOvLII3XPPffIzDRw4EDl5ubqyiuv1CWXXKK+ffvq3HPPVUZGhsxM8+fP1/PPP68JEyZo7NixGjFihDZv3qz77rsv0ZdUo2LJVTyDBgAAACBmvXv3Vr9+/SpsAcvPz9fkyZPVqlUrde7cWe6u2bNna/LkyerZs2cNR7v3oYsjgLhxd+44AgBCbU9y1fLly3XaaaepY8eOyszMjD67JkValu6880517NhRRx11lKZMmaIbb7xR2dnZSktL09KlS6Pb3n333UpLS1NaWpoGDRqkzZs3S4p0++vfv7/69Omjtm3b6vTTT9eWLVuq94KrQY8ePTRp0qQKY+vZs6emTJmiKVOmKD8/X/n5+bvMo3IUaAAAAEAViouLddFFF+mBBx7QnDlzNGPGDN11111atmxZdJsDDjhAc+bM0d13362zzjpLJ5xwgubNm6cBAwbozjvvlCRNnDhRY8eO1cyZM7Vw4UKVlJTo9ttvjx6jsLBQBQUFWrp0qXbs2KGnnnqqxq+1KiNGjNApp5yiU089VRs3bvzB+iOOOEIpKSkaPXp0tEAbNWqU6tatq8MPPzwBEe9dKNAAAACAKrz//vtaunSp+vXrp6ysLHXr1k3btm3bpWWsdBTDDh06yMzUt29fSVJOTo4++OADSdLkyZPVr18/7b///jIzDRkyRJMnT44e49RTT9UBBxwgM9Pxxx+vlStX1uBVxu4Pf/iDzj//fJ188slat27dD9bn5+erYcOGatmypQ499FA1aNCA7o0x4hk0AAAAoArurubNm6uoqKjCberXry9JSklJ0b777htdnpKSouLi4pjOU3qM0v2+//77HxdwnJQdYPCqq67SVVddJUkaNWrULtuNHDlyl/klS5bsMj9ixIi4xFcb0IIGAAAAVCE1NVUNGzbU2LFjo8uWLVtWbhe/ypx88sn697//rU2bNsnd9fjjj+uUU06p7nCxF6NAAwAAAKpQp04djR8/Xs8884wyMjLUvn17XX755dq+ffseHee0007TJZdcory8PKWnp0uSbr755niEjL0U70EDANQKvAcNABB2seQqWtAAAAAAICQo0AAAAAAgJCjQAAAAACAkKNAAAAAAICQo0AAAAAAgJCjQAAAAACAkKNAAAAAAICQo0AAAAAAgJCjQAAAAACAkKNAAAAAAICQo0AAAAAAgJCjQAAAAACAkKNAAAAAAICQo0AAAAAAgJCjQAAAAACAkKNAAAAAAICQo0AAAAAAgJEJdoJnZE2b2tZktSnQsAADsjjwFAKhuoS7QJI2S1DvRQQAAUIFRIk8BAKpRqAs0d39L0rpExwEAQHnIUwCA6hbqAi0WZjbEzArNrHDNmjWJDgcAgB8gVwEAYrXXF2ju/qi757p77kEHHZTocAAA+AFyFQAgVnt9gQYAAAAAtQUFGgAAAACERKgLNDN7WtI7klLNbJWZDU50TAAAlCJPAQCqW51EB1AZd++f6BgAAKgIeQoAUN2qbEEzs7rlLGsen3AAAAAAIHlVWKCZ2UlmtkrSF2b2upm1KbP69bhHBgAAAABJprIWtHskneruzSU9KukNM+scrLO4RwYAAAAASaayZ9DquftiSXL3581sqaQXzewPkrxGogMAAACAJFJZgbbDzA5x9y8lyd0Xm1m+pFck/axGogMAAACAJFJZF8cbJB1cdoG7r5LUXdJd8QwKAAAAAJJRhS1o7j65guXfSrozbhEBAAAAQJIK9YuqAQAAACCZUKABAAAAQEhQoAEAAABASFQ2iqMkycyOlXS9pCPKbu/uPeMYFwAAAAAknVha0J6T9J6kmxUp1EonAEBg2rRpys3NTXQYAACUizy196iyBU1Ssbs/HPdIAAAAACDJxdKCNt7MLjezlmbWtHSKe2QAEGcXX3yxcnNzlZ6ernPOOUfr16/XtGnTlJmZqQEDBqh9+/bq1KmTlixZIkmVrtvdhAkTdMIJJygnJ0d5eXmaNWtWTV4aAKAWIE8lp1gKtMsU6dI4U9LcYCqMZ1AAUBMefPBBFRYWauHChWrfvr3uvvtuSdKCBQs0ePBgLV68WFdccYUGDBgQ3aeydaVWrlyp22+/XRMnTtTcuXP1+OOP64ILLqj2+M1Md955pzp27KijjjpKU6ZM0Y033qjs7GylpaVp6dKlkqQvv/xSJ510knJyctS+fXv9/ve/jx5jxIgR6t+/v/r06aO2bdvq9NNP15YtWyRJ3377rc4991y1bdtW+fn5GjBggIYNGyZJ2rx5swYNGqS0tDSlpaXpnnvuqfbrA4BkR55KzjxVZYHm7keWMx1VE8EBQDyNGTNGOTk5Sk9PV0FBgYqKiiRJRx99tLp37y5JuvTSS7Vw4UJt3LixynWlXnvtNa1cuVInnniisrKydPHFF6u4uFhfffVVtV/DAQccoDlz5ujuu+/WWWedpRNOOEHz5s3TgAEDdOedd0a3GT9+vObOnauioiIVFhZq0qRJ0WMUFhaqoKBAS5cu1Y4dO/TUU09Jkm677TYdeOCBWrZsmZ577jlNnz49us/tt9+unTt3auHChZo5c6ZGjx6tiRMnVvv1AUAyI09FJFueqrJAM7O6ZnaVmT0fTFeaWd2aCA4A4mX69Ol6+OGHNWnSJC1cuFB33HGHtm7dWi3Hdnf17t1bRUVF0enzzz/XwQcfXC3HL+vCCy+UJHXo0EFmpr59+0qScnJy9MEHH0iSSkpKdP311yszM1M5OTlatGhRNMlL0qmnnqoDDjhAZqbjjz9eK1eulCRNnTpVgwYNkiQ1bdpUZ599dnSfyZMn61e/+pXMTPvvv7/69++vyZMnV/v1AUCyIk8VRY+RbHkqli6OD0vKkfTPYMoJlgHAXmvDhg1q0qSJmjVrpm3btumJJ56Irlu5cmX0LlxBQYHS09O1//77V7muVK9evTRp0iQtXrw4umzOnDlxuY769etLklJSUrTvvvtGl6ekpKi4uFiSdP/992v9+vWaPXu2FixYoLPPPnuXJF96jN33AwAkDnkqefNULAVaR3e/zN3fDKZBkjrGOzAAiKfevXvrZz/7mY499lh1795dHTp0iK5LT0/X448/rrS0NP3tb3/TmDFjYlpX6phjjtG4ceM0ePBgZWZm6rjjjtPIkSNr5LrKs2HDBrVs2VL169fX6tWr9dJLL8W0X48ePaLXt2HDhl32O/nkk/Wvf/1L7q5NmzbpmWee0SmnnBKX+AEgGZGnqlZb81Qsw+yXmNnP3H2lJJnZUZJK4hsWAMRX3bp19e9///sHy6dNm6a6detq9OjRFe5X3roePXqosPC/4yf16tVLvXr1qr6Af4KrrrpK559/vtLS0tS6dWvl5+fHtN8tt9yiQYMGqW3btmrZsqVyc3PVpEkTSdKf/vQnXXnllUpPT5cUec6hd+/ecbsGAEg25Kmq1dY8Ze5e+QZm+ZKelPShJJN0hKRB7j41/uHtmdzcXC/7Fw8A9tS0adM0bNgwlfdbUtm62mjHjh0qKSlR/fr1tXHjRnXt2lX333+/Tj755ESHVi4zm+vuoX8LK7kKwE9BnvqvvS1PSbHlqioLtOBA+0pKDWaXu/u2aoiv2pH0AKD6fP311zrttNNUUlKirVu36qKLLtItt9yS6LAqRIGG6lRSUqKUlJREhwGgEntbnpJiy1UVPoNmZj2DP38u6XRJRwfT6cEyAPhJzEybN29OdBioQIsWLaJDHi9btiz0SQ97JzPT8OHDlZWVpdTUVL3wwgvRdbNnz46+GyknJ0evvvqqJOnjjz9W8+bNdd111ykjI0Pp6enRQREqWydV/HLeadOmKSMjQ4MGDVJWVtZeMxw34o9cFV61NU9V9gxad0lvSjqjnHUu6cW4RAQAAJJKSkqKioqKtHz5cnXp0kXdunVTvXr1NHToUE2YMEEtW7bUF198oY4dO2rRokWSpLVr1yozM1N/+ctfNG3aNPXv3z869HZF61atWqXbb79dr732mvbff38tXrxYp512mj799FNJ0uLFizVy5Ejl5eUl7LsAgApb0Nx9ePDnoHKmX9RciABqs3vvvbfcO+cXX3yxcnNzlZ6ernPOOUfr16+XFLnLnZWVpV//+tfKyMhQZmamli5dKkn68ssvo3fb27dvr9///vfR440YMUL9+/dXnz591LZtW51++unasmWLJGnKlCnKy8tTdna20tPT9cwzz9TgNwBg8ODBkqTU1FR16NBBs2bN0syZM/XRRx/ptNNOU1ZWlk477TSZWfS9SfXq1dMll1wiKTL4QYMGDbR8+fJK11X1ct5jjjmG4gzlIlehJsXyouqrzWx/i3jczN4zs3AM+QJgr1d65/zll1/WkCFD9PXXX0uSHnzwQRUWFmrhwoVq37697r777ug+ixcv1tChQ7VgwQJdcMEFuuOOOyRJBxxwgMaPHx/t7lBYWKhJkyZF9yssLFRBQYGWLl2qHTt26KmnnpIUeXnmjBkzNG/ePE2ePFnDhg2LJlkAieHuysjI2OVFup999plyc3/8Y4ZVvZy3UaNG1RU+ahlyFWpSLO9B+4W7b5TUS1IzSZdKuiuuUQFIGuXdOZekMWPGKCcnR+np6SooKFBRUVF0n9TUVGVnZ0uSOnfuHO3WVFJSouuvv16ZmZnKycnRokWLdtnv1FNP1QEHHCAz0/HHHx/db82aNTrvvPOUlpamU089VevWrYveiQcQf08++aQkacWKFZo3b546d+6sLl26aMWKFZo69b+DRs+ZM0elg5tt375dBQUFkqTp06fr+++/V9u2bStdV5Mv50XtQq5CTYrlPWgW/NlH0hh3X2xmVtkOAPBTTJ8+XQ8//LBmzpypgw46SAUFBXr00Uej6+vXrx/9nJKSouLiYknS/fffr/Xr12v27NmqX7++hgwZoq1bt1a43/fffy9J+s1vfqMzzzxTL774osxMxx577C77AYiv4uJiZWdna8uWLRo5cqRatGghSXr55Zd1/fXX65prrtH27dt11FFHafz48ZKkZs2aqaioSPfcc4/cXU8//bTq1atX6bqyL+f9/vvvtX37dp1wwgnq2LFjwq4dey9yFeIllgJtrpm9LulISTeaWWNJO+MbFoBk8eSTT+rmm2/e5c757Nmz1aRJEzVr1kzbtm3TE088EdOxNmzYoJYtW6p+/fpavXq1XnrpJf3mN7+Jab82bdrIzPTGG29En3EBUDOGDRumESNG/GB5x44dNW3atAr3u++++/Z4XUUv5939Jb5AWeQq1KRYCrTBkrIkfejuW8ysmaRBcY0KQNIo78557969NW7cOB177LFq3ry5TjzxRL377rtVHuuqq67S+eefr7S0NLVu3Vr5+fkxxXDXXXfp8ssv1/Dhw9WxY0dlZGT81MsCANQi5CrUpFhfVN1K0hEqU9C5+1txjOtH4eWfAFCxjz/+WK+//rqGDBkSt3OMGDFCmzdvrrRlI154UTUA7N1qe56SfuKLqssc5G5Jb0u6WdL1wTSsWiIEANSYjz/+eJfnI3ZX+nxEopUXR0lJSQIiAQDUJPJURCyjOJ4tKdXd+7j7GcF0ZrVFAACodlu2bNH555+vdu3aKTMzUxdccIGuuOIKLVmyRFlZWTrvvPMkSW3atNENN9ygTp066de//nWl7+dp1apVdGjpPn366PTTT5ckff3112rdunV0u08//VQ9e/ZU27Ztde655+rbb7+VFBlZ7/rrr1enTp2UmZmpSy+9VJs3b5YkDRw4UL/85S/VrVs35ebmatq0acrIyNCgQYOUlZWlp556Si1bttzlgfgzzzwzOlIfAGDvQp6qWCwF2oeS6u7xkQEACfPaa69p48aNWrJkiebPn6+RI0fqH//4h9q1a6eioiI9//zz0W03btyod999V//6178qfT/PSSedpDfffFM7duzQRx99pI8++kg7duzQlClTdNJJJ0WPN336dD399NNatmyZmjRpottvv12SdM8996hJkyZ69913NX/+fB166KH6n//5n+h+RUVFmjRpUnS46cWLF2vIkCEqKirSgAED1L17d/373/+WFLnLWlhYGE3gAIC9C3mqYrEMErJFUpGZTZG0rXShu1+1x2cDANSIzMxMLV26VFdccYV69OgRvYtYngEDBkQ/l76fZ+bMmXJ3ffnllyoqKlLv3r2Vn5+vyZMnq1WrVurcubPcXbNnz9bkyZPVs2fP6DH69u0bffHv4MGD9dvf/lZSZMj0jRs3RpPutm3blJmZGd3vvPPO03777RedP+aYY5SXlxedv+qqq3Tttdfqsssu0yOPPKJf/OIX0WHVAQB7F/JUxWIp0F4OJgDAXuKoo47S4sWLNWXKFE2cOFE33XSTHnrooXK3bdSoUfRzZe/n6dmzp2677bboqGPurilTpmjKlCkaPnx4lTG5u/75z3/ukiQriqO8+S5duqikpERvv/22Ro0axQuGAWAvRp6qWJVdHN19tKRnJc1y99Gl0486GwCgRqxatUopKSk6++yz9cADD2jNmjXaf//9o/3sK1Le+3lKHXHEEUpJSdHo0aOVn5+v/Px8jRo1SnXr1tXhhx8e3e7VV1/VmjVrJEXeHVSa6M4880zdf//90Zeubtq0SUuXLt2j6/rtb3+rfv36qUuXLjrssMP2aF8AQHiQpyoWyyiOZ0gqkjQpmM8yM1rUACDEFi5cqLy8PGVmZqpTp0668cYb1alTJ6WmpiotLa3CPvFXXXWV3n77baWlpWnw4ME/eD9Pfn6+GjZsqJYtW+rQQw9VgwYNfnCnsVu3burXr5/atm2rdevW6U9/+pMk6YYbblBmZmb0/T1du3bd48TXr18/rV+/Xpdffvke7QcACBfyVMWqfA+amc2V1FPSNHfPDpYtcve0H33WOOHdMgBQu82YMUNDhw7VwoULZWa7rOM9aACARKssT0mx5apYnkHb4e7f7naCnXsWKgAAP83gwYP1xhtvaMyYMeUmPQAAEqm68lQsBdpiM7tIUoqZHSPpKkkzf/QZAQD4Ef71r38lOgQAACpUXXkqlveg/VZSe0WG2H9a0kZJ11TL2QEAAAAAUVW2oLn7Fkl/DCYAAAAAQJxUWaCZWa6kmyS1Kbu9u2fELywAAAAASD6xPIP2lKTrJS0Ug4MAAAAAQNzEUqCtcXfeewYAAAAAcRZLgTbczB6XNEWRgUIkSe7+YtyiAgAAAIAkFEuBNkhSW0l19d8uji6JAg0AAAAAqlEsBVpHd0+NeyQAAAAAkORieQ/aTDNrF/dIAAAAACDJxdKC1llSkZl9pMgzaCbJGWYfAAAAAKpXLAVa77hHAQAAAACouEAzs/3dfaOkTTUYDwAAAAAkrcpa0Aok9ZU0V5FRG63MOpd0VBzjAgAAAICkU2GB5u59gz+PrLlwAAAAACB5VdbFsUNlO7r7e9UfDgAAAAAkr8q6OP4l+LO+pFxJ8xXp5pghqVBSXnxDk8yst6QHJaVIetzd74r3OQEA2BPkKgBAdarwPWjufpK7nyTpC0kd3D3X3XMkZUtaHe/AzCxF0j8knSapnaT+vI8NABAm5CoAQHWL5UXVqe6+sHTG3RdJOi5+IUV1kvSBu3/o7tslPSPprBo4LwAAsSJXAQCqVSwF2gIze9zMegTTY5IWxDswSa0kfVZmflWwbBdmNsTMCs2scM2aNTUQFgAAUeQqAEC1iqVAGyRpsaSrg2lJsCwU3P3RoPtl7kEHHZTocAAA+AFyFQAgVpUNEiJJcvetkh4Ippq0WtJhZeZbqwaefQMAYA+QqwAA1SqWFrREmSPpGDM70szqSeon6eUExwQAQFnkKgBAtaqyBS1R3L3YzK6U9JoiQxc/4e6LExwWAABR5CoAQHULbYEmSe4+QdKERMcBAEBFyFUAgOpUZYFmZuMl+W6Lv1XkZdUjg2fUAAAAAAA/USzPoH0oabOkx4Jpo6RNko4N5gEAAAAA1SCWLo5d3L1jmfnxZjbH3TuaGf3sAQAAAKCaxNKC1sjMDi+dCT43Cma3xyUqAAAAAEhCsbSgXSdphpmtlGSSjpR0uZntJ2l0PIMDAAAAgGRSZQtaMDrVMZKukXS1pFR3f9Xdv3P3v8Y3PMSLmWnz5s2JDmMXbdq00aJFiyRJv/zlLzV9+vQERwQASBTyFIBkFesw+zmS2gTbZ5qZ3H1M3KJCaBQXF6tOnZp/G8Pjjz9e4+cEAOx9yFMAapsqW9DMbKyk+yR1ldQxmHLjHBdqwL333qusrCylpqbqhRdeiC43M40YMUIdO3bUrbfeqq+++krnnHOOMjIylJ6erjFj/lubL126VL169YquGz060ut19erVOvfcc5WRkaGMjAz9z//8jyRVeqyyevTooVdeeSWOVw8ACDvyFIBkFMstp1xJ7dx993ehYS+XkpKioqIiLV++XF26dFG3bt3UokULSVKDBg00Z84cSdKFF16otLQ0/ec//9EXX3yhnJwcdejQQW3bttVZZ52lO++8U+eff74kae3atZKkSy65RH369Ikm1G+++UaSdNVVV5V7rLS0tJq+fABAyJGnACSjWEZxXCTpkHgHgpo3ePBgSVJqaqo6dOigWbNmRddddtll0c+TJ0/Wr3/9a0lSy5Yt1adPH02dOlXLly9XcXFxNOlJUrNmzbR582bNnDlT1157bXR58+bNKz0WAAC7I08BSEaxtKA1l7TEzN6VtK10obufGbeokHCNGjWqeiMAABKEPAWgtoqlBW2EpLMl/VnSX8pM2Ms9+eSTkqQVK1Zo3rx56ty5c7nbnXzyyXrsscckSV9++aUmTJignj17KjU1VXXq1NFzzz0X3Xbt2rVq1KiRunTpogceeCC6vLTrSEXHAgBgd+QpAMkolmH2/6+8qSaCQ3wVFxcrOztbffv21ciRI6P9+nf3t7/9TfPnz1dGRoZOOeUU3XXXXWrfvr3q1Kmjl156SY888ojS09OVmZmpCRMmSJLGjRunt99+W2lpacrMzNS//vWvSo8FAMDuyFMAkpFVNPaHmc1w965mtklS2Y1Mkrv7/jUR4J7Izc31wsLCRIcBAEgAM5vr7qEfZZhcBQDJK5ZcVeEzaO7eNfizcXUHBgAAAAD4oQoLNDNrWtmO7r6u+sMBAAAAgORV2SiOcxXp2miSDpe0Pvh8gKRPJR0Z7+AAAAAAIJlUOEiIux/p7kdJmizpDHdv7u7NJPWV9HpNBQgAAAAAySKWYfY7u/uE0hl3nyipS/xCAgAAAIDkFMuLqj83s5sljQvmL5b0efxCAgAAAIDkFEsLWn9JB0n6TzC1CJYBAAAAAKpRlS1owWiNV9dALAAAAACQ1CobZn+8dn1B9S7c/cy4RAQAAAAASaqyFrT7gj9/LukQ/fcZtP6SvopnUAAAAACQjCos0Nz9/yTJzP7i7rllVo03s8K4RwYAAAAASSaWQUL2M7OjSmfM7EhJ+8UvJAAAAABITrEMs3+tpGlm9qEkk3SEpF/HNSoAAAAASEKxjOI4ycyOkdQ2WLTM3bfFNywAAAAASD5VdnE0s4aSrpd0pbvPl3S4mfWNe2QAAAAAkGRieQbtSUnbJeUF86sl3RG3iAAAAAAgScVSoP3M3e+RtEOS3H2LIs+iAQAAAACqUSwF2nYza6DgpdVm9jNJPIMGAAAAANUsllEch0uaJOkwM3tK0gmSBsYzKAAAAABIRrGM4viGmb0nqbMiXRuvdvdv4h4ZAAAAACSZWFrQJKm7pK6KdHOsK+k/cYsIAAAAAJJULMPs/1PSUEkLJS2S9Gsz+0e8AwMAAACAZBNLC1pPSce5e+kgIaMlLY5rVAAAAACQhGIZxfEDSYeXmT8sWAYAAAAAqEYVtqCZ2XhFnjlrLGmpmb0bzB8v6d2aCQ8AAAAAkkdlXRzvq7EoAAAAAAAVF2ju/n9l581s/8q2BwAAAAD8NFUWXGY2RNJtkrZK2qnIu9Bc0lHxDQ0AAAAAkkssLWLXS0rj5dQAAAAAEF+xjOK4UtKWeAcCAAAAAMkulha0GyXNNLPZkraVLnT3q+IWFQAAAAAkoVgKtJGS3pS0UJFn0AAAAAAAcRBLgVbX3X8X90gAAAAAIMnF8gzaRDMbYmYtzaxp6RT3yAAAAAAgycTSgtY/+PPGMssYZh8AkkhJSYlSUlISHQYAABWqLbmqyhY0dz+ynIniDABqiUmTJik7O1sZGRnKz8/XBx98oGnTpikjI0ODBg1SVlaWJk6cqIKCAh1//PHKzs5Wdna2pkyZEj1GmzZtdMsttygvL09t2rTR3//+9+i66dOnKz09XRkZGbr66qt1xBFHaNGiRZKk5cuX67TTTlPHjh2VmZmpJ598ssavHwAQfkmVq9y93EnS78t8Pn+3dX+uaL9ETjk5OQ4AiN1XX33lzZs398WLF7u7++OPP+6dOnXyqVOn+j777OMzZ86MbvvNN9/4zp073d192bJl3qpVq+i6I444wq+77jp3d//oo498v/32802bNvnWrVu9VatW/tZbb7m7+4svvuiSfOHChb5jxw7v0KGDL1261N3dN27c6Mcee2x0fk9JKvQQ5KKqJnIVAOyZZMtVlXVx7CfpnuDzjZKeK7Out6SbqrdUBADUtNmzZyszM1Pt2rWTJA0aNEiXX365Nm3apGOOOUZ5eXnRbVeuXKn+/ftr9erVqlu3rr788kt9+eWXOuSQQyRJ/fr1kxS5Q3nggQdq1apV2r59uxo0aKBu3bpJks455xwdcMABkqT3339fS5cuje4nSdu2bdPSpUvVtm3bmrh8AMBeINlyVWUFmlXwubx5AEAt06hRo13m+/fvr7/85S86++yztXPnTjVs2FBbt26Nrq9fv370c0pKioqLiys9vrurefPmKioqqta4AQDJozbmqsqeQfMKPpc3DyStwsJCXXzxxXE59rRp05SbmxuXYwOS1LlzZ82fP1/Lli2TJI0ePVrZ2dlq3LjxD7bdsGGDjjzySEnSE088oW3btlV5/NTUVG3ZskVvv/22JOmll17Shg0bousaNmyosWPHRrdftmyZNm7c+FMvC0AZ5Cns7ZItV1XWgpZpZhsVaS1rEHxWMF+/4t2A5JKbm6unnnoq0WEAP8pBBx2ksWPH6qKLLlJxcbEOOuggjRs3TqtWrfrBtn/961919tln68ADD1Tv3r3VrFmzKo+/7777qqCgQEOHDpWZqXv37mrRooWaNGmiOnXqaPz48brmmmt07733qqSkRAcffLCeffbZeFwqkLTIU9jbJV2uquohtb1p4sHrvctHH33kzZo12+N18fbdd9/5eeed58cdd5xnZGT4+eef71OnTvWMjAy/9NJLvV27dt6xY8fog6pTp0710r97pXHfdNNNnpWV5ccee6xPnz7d3d137NjhvXr18pycHG/Xrp0PHDjQt23bFj3vn//8Z09LS/OMjAzPy8vzkpISnzp1qmdmZvqQIUM8PT3dMzIyfMmSJTX/pQA/wcaNG6Of33zzTT/88MO9pKSk2s8jBglBHIQxV5GngOoXplwVy4uqa5yZnW9mi81sp5nRbo4a9dprr2njxo1asmSJ5s+fr5EjR0qSFixYoMGDB2vx4sW64oorNGDAgHL3X7t2rfLy8jRv3jzdcsst+sMf/iAp0s+5oKBAhYWFWrRokUpKSvTEE09IijTVv/zyy5o5c6bmz5+v8ePHa599Iv97Ll68WEOHDtWCBQt0wQUX6I477qiBbwGoPi+88IIyMzOVnp6u3//+9yooKIj+/d6bkauQKOQpoPqFKVeFNUMukvRzSW8lOhDsmXfeeUddu3ZVZmamMjMz9frrr2vYsGHR90bk5+frk08+2WWf6667ThkZGUpPT9f06dOrXFdcXKxTTz1Vubm5at++vQYNGqTt27dLkrZv364hQ4bo2GOPVdeuXXXllVfqvPPOix7v7rvvVqdOndShQwedccYZ+vLLL39wDZmZmVq6dKmuuOIKPffcc9p3330lSUcffbS6d+8uSbr00ku1cOHCcvsfN2rUSH379pUU6TO9cuVKSdLOnTt13333KSsrSxkZGXrzzTejD5y+8sor+s1vfhPtS122OT41NVXZ2dk/OB6wtxg4cKDmz5+vhQsXas6cOTrhhBMSHVJ1IVftpfb2XEWeAqpfmHJVKAs0d1/q7ssTHQf2zLp163TOOefonnvu0fz58/Xee++pY8eOuuGGGzRnzhzNnz9f/fv3j96pkyJ38TIzM7VgwQI99NBD6t+/f/RhzorWVXaHb+TIkZozZ45GjRqlyZMnq7CwMHqucePGaeXKlZo1a5bee+899enTR9ddd90PruOoo47S4sWLdcopp2jy5MnKzMzcZfSfqpQmSmnX0YEKCgo0Y8YMTZ8+XQsXLtTll18e03H3dLQhADWDXLV3qg25ijwF1G6VDRKyVzCzIZKGSNLhhx+e4GiS2zvvvKN27dqpS5cukiI/0gceeKDGjh2rf/zjH9q8efMPfrTr1aunSy65RJLUo0cPNWjQQMuXL9f+++8fXVdcXLzLutTUVN13332aOHGiSkpKtH79ejVs2FCSNHXqVDVu3FjvvfeeunTpov79+0fvZr788ssqLCxUhw4dJEXubjZp0uQH17Fq1So1bdpUZ599tnr16qVDDz1U69at08qVKzV9+nR169ZNBQUFSk9P1/777x/z97NhwwY1b95cjRs31rfffquCgoLoyFd9+/bVww8/rHPOOUeNGzfW2rVrY3qoFcDegVwVHrUhV5GngNotYQWamU2WdEg5q/7o7i/Fehx3f1TSo5KUm5vL8P8h88knn+jaa6/VyJEj9cADD+irr77SJ598otdff13PPfecduzYoezsbDVv3jx6Z3H16tXq0aOHiouLlZubqyFDhuiZZ57RmjVrdMkll8jM1LhxY5144ol68803tWPHDk2fPl0lJSX66quvNH/+fH344Yd6/PHH1b59+2gs7q6bb75Zv/jFLyqNeeHChbrhhhskSSUlJbrxxht16KGHKj09XY8//rh+85vfqGHDhhozZswefRcDBgzQSy+9pLZt26pFixbq1q2bvv/+++i61atXq3Pnzqpbt64aNWqkt96i1xSQaOSq5LC35SryFFDLVTWKSCInSdMk5ca6PSNjJdbatWv94IMP9pkzZ7q7e3Fxsb/11lt+8MEHe4sWLXz69Ol+6aWX+uGHH+7r1q3zuXPnuiQfO3asP/bYY96zZ09v1aqVL1++3BV5156PHTvW3d2zsrK8fv36/t133/nf/vY3z8zM9Pz8fP/666/9uOOO80MOOcT/+c9/+oMPPuitWrXyv/71r/799997Xl6en3vuue7uPmbMGM/JyfF169a5u/vWrVu9qKgopmsrOwIWgHBSgkZxJFftXWprriJPAXuHWHJVKJ9Bw96padOmevHFF/W73/1OGRkZysnJ0bZt29SpUydt3LhR1157rY488kiZmQ488EBNmzZNderU0TXXXKMrr7xSM2bM0NNPP6169epp3333VdOmTVVUVKSMjAy9//77+sMf/qCGDRtqwIAB+vzzz7V48WKde+656t69u44++mhNnjxZQ4cOVYMGDXTnnXeqZ8+eOu6446JdQy699FJdfPHF6t69ezS+0hcSAgCSA7kKQOhVVcElYpJ0jqRVkrZJ+krSa7Hsx52jcHrllVf8pJNO2mXZxx9/7M2aNfMPP/zQ3d3ffvttP+KII9y9/PfKdO/e3cePHx+d//nPfx69Y+nuPnbsWP/5z3/u7u4XXXSRP/TQQ75161bv1auXP/bYY/G4LAAhoxpuQSNX1S7kKgA1IZZcFcoWNHf/j7u3dvd93f1gdz810THhx8vLy9OSJUv0zjvvSIr0l//0009Vr149HXLIIdq5c6ceeeSRPTrmySefrNGjR2vHjh3asWOHRo8erVNOOUWSNGXKFN15553KzMzUMccco4EDB1b3JQEAuaqWIVcBCItQFmioXSrqTnL++eerXbt2Ov7443XkkUfu0TGHDBmijIwMZWdnKzs7WxkZGfrVr34lSRo/fryaNm2q+vXrq1OnTqpTZ68frBQAEGfkKgBhYZGWttohNzfXy75LBACQPMxsrrvnJjqOqpCrACB5xZKraEEDAAAAgJCgQAMAAACAkKBAAwAAAICQoEADAAAAgJCgQAMAAACAkKBAAwAAAICQoEADAAAAgJCgQAMAAACAkKBAAwAAAICQoEADAAAAgJCgQAMAAACAkKBAAwAAAICQoEADAAAAgJCgQAMAAACAkKBAAwAAtc6oUaN03nnnlbsuKytL33//fQ1HBACxqZPoAAAAAGpSUVFRokMAgArRggYAAOLCzDR8+HBlZWUpNTVVL7zwgiTp448/VvPmzaPblZ0v/XzjjTcqOztbbdu21dy5c/WrX/1KGRkZOv744/Xll19KirSSnXLKKTrzzDPVrl079ezZU6tXr/5BHJ999plyc3P17LPPRuPavHmzJKlNmza65ZZblJeXpzZt2ujvf/97XL8TAKgKBRoAAIiblJQUFRUV6eWXX9aQIUP09ddfV7nP2rVr1bVrV82bN0+DBw9Wfn6+rrjiCi1YsEA5OTm7FFEzZszQvffeqyVLlqh79+66+uqrdznW/Pnz1adPHz3wwAO64IILyj3fli1b9M4772jatGm64YYbosUbACQCBRoAAIibwYMHS5JSU1PVoUMHzZo1q8p9GjVqpNNPP12S1KFDB7Vu3VpZWVmSpJycHH3wwQfRbbt27arU1FRJ0i9/+Uu9+eab0XULFizQz3/+cz377LPq1q1bhefr16+fpEhr2oEHHqhVq1bt2UUCQDWiQAMAADWqTp062rlzZ3R+69atu6zfd999o59TUlJUv379XeaLi4tjOk/r1q3VtGlTTZs2rdLtfuzxASAeKNAAAEDcPPnkk5KkFStWaN68eercubMOOeQQ7dixI9oSVlBQ8KOP//bbb2vFihXRc/Xs2TO6rmnTppo8ebLGjh2r+++//ydcBQDUHEZxBAAAcVNcXKzs7Gxt2bJFI0eOVIsWLSRJDz74oE455RQddNBB0e6MP8YJJ5ygYcOGacWKFTrkkEM0duzYXdY3adJEr7/+uvr27avvvvtOf/rTn37S9QBAvJm7JzqGapObm+uFhYWJDgMAkABmNtfdcxMdR1WSKVeZmTZt2qRGjRrF5fijRo3SK6+8oueffz4uxweA6hZLrqKLIwAAAACEBAUaQqtPnz5auXJlosMAAPxI7h631jNJGjhwYEJbz8hTAOKBZ9AQdyUlJUpJSdnj/SZMmBCHaAAA2BV5CkCY0IKGSpmZhg8frqysLKWmpuqFF16Irrv44ouVm5ur9PR0nXPOOVq/fr0kadq0acrIyNCgQYOUlZWliRMn6tFHH9Vxxx2nrKwsZWRkaNmyZZIi75y55ZZblJeXpzZt2uzy8tE2bdpo0aJFkqQePXrommuuUadOnXT00UfrpptuqsFvAQAQVuQpALUNLWioUkpKioqKirR8+XJ16dJF3bp1U4sWLfTggw+qefPmkqSbb75Zd999t+666y5J0uLFizVy5Ejl5eVJioyitWzZMrVs2VLbtm1TSUlJ9PhbtmzRO++8o48//lhpaWkaOHBguV1ilixZopkzZ2rr1q3Ky8tTly5d1Ldv3xr4BgAAYUaeAlCb0IKGKg0ePFiSlJqaqg4dOmjWrFmSpDFjxignJ0fp6ekqKChQUVFRdJ9jjjkmmvQkqWfPnrrsssv00EMPafXq1WrYsGF0Xb9+/SRF7kQeeOCBWrVqVblxXHbZZapTp44aNWqkfv366c0336zuSwUA7IXIUwBqEwo0/CjTp0/Xww8/rEmTJmnhwoW64447tHXr1uj63e8svvjii7rjjjv03Xff6aSTTtLEiROj6+rXrx/9nJKSouLi4vhfAACgViNPAdhbUaChSk8++aQkacWKFZo3b546d+6sDRs2qEmTJmrWrJm2bdumJ554osL9i4uL9eGHH6pTp0664YYb1KtXL82bN2+P4xg3bpyKi4v13Xff6dlnn1XPnj1/9DUBAGoP8hSA2oRn0FCl4uJiZWdna8uWLRo5cqRatGih3r17a9y4cTr22GPVvHlznXjiiXr33XfL3b+kpEQDBw7Uhg0btM8+++iwww6LPgOwJ9q2basuXbpo3bp1uuCCC+jXDwCQRJ4CULuYuyc6hmqTm5vrhYWFiQ6jVjEzbdq0Ka7vsYlFjx49NGzYMJIdgAqZ2Vx3z010HFUhV1Uv8hSAvUksuYoujgAAAAAQEnRxRKXC0sI6bdq0RIcAAAgh8hSA2oYWNAAAAAAICQo0AAAAAAgJCjQAAAAACAkKNAAAAAAICQo0AAAAAAgJCjQAAAAACAkKNAAAAAAICQo0AAAAAAgJCjQAAAAACAkKNAAAAAAICQo0AAAAAAgJCjQAAAAACAkKNAAAAAAICQo0AAAAAAgJCjQAAAAACAkKNAAAAAAICQo0AAAAAAgJCjQAAAAACAkKNAAAAAAIiVAWaGZ2r5ktM7MFZvYfMzsg0TEBAFAWuQoAEA+hLNAkvSEpzd0zJL0v6cYExwMAwO7IVQCAahfKAs3dX3f34mB2lqTWiYwHAIDdkasAAPEQygJtN7+QNLGilWY2xMwKzaxwzZo1NRgWAABR5CoAQLWok6gTm9lkSYeUs+qP7v5SsM0fJRVLeqqi47j7o5IelaTc3FyPQ6gAgCRFrgIA1LSEFWjufnJl681soKS+kvLdnWQGAKhx5CoAQE1LWIFWGTPrLen3krq7+5ZExwMAwO7IVQCAeAjrM2h/l9RY0htmVmRmjyQ6IAAAdkOuAgBUu1C2oLn70YmOAQCAypCrAADxENYWNAAAAABIOhRoAAAAABASFGgAAAAAEBIUaAAAAAAQEhRoAAAAABASFGgAAAAAEBIUaAAAAAAQEhRoAAAAABASFGgAAAAAEBIUaAAAAAAQEhRoAAAAABASFGgAAAAAEBIUaAAAAAAQEhRoAAAAABASFGgAAAAAEBIUaAAAAAAQEhRoAAAAABASFGgAAAAAEBIUaAAAAAAQEhRoAAAAABASFGgAAAAAEBIUaAAAAAAQEhRoAAAAABASFGgAAAAAEBIUaEBITJs2Tbm5uZKkzz//XCeddFJ03f/+7//quOOOU3Z2tpYvXy4z0+bNmxMVKgAgSZGrgPijQANC6NBDD9XUqVOj8yNHjtRtt92mefPmKTU1NSExlZSUJOS8AIBwIlcB8UGBBvxE77zzjrp27arMzExlZmbq9ddf17Bhw9SxY0dlZmYqPz9fn3zyiaRd7zyWN1/q448/VvPmzSVJ1157raZPn64//OEPu9ypLDVnzhzl5eUpIyNDeXl5mjNnjiTpxhtv1L333itJevbZZ7XPPvvo66+/liT16dNHr7/+uiRpwoQJOuGEE5STk6O8vDzNmjUrGltGRoYGDRqkrKwsTZw4sbq+MgBADSNXAXsRd681U05OjgM1ae3atX7wwQf722+/7e7uxcXFvm7dOl+zZk10m8cee8wvvPBCd3efOnWql/17Wna+7OePPvrImzVrFt2ue/fuPn78+Oi8JN+0aZNv27bNDzvsMJ88ebK7u7/xxht+2GGH+bZt2/yNN97wU0891d3dhwwZ4nl5ef7000/79u3bvWnTpv7dd9/5Bx984J07d/Zvv/3W3d0XLVrkhx12WDSeffbZx2fOnFm9XxoQJ5IKPQS5qKqJXIWaRq4CwiOWXFUnseUhsHd755131K5dO3Xp0kWSlJKSogMPPFBjx47VP/7xD23evFnFxcVxO//y5ctVr1495efnS5JOPvlk1atXT8uXL9cJJ5ygCy+8UNu3b9fbb7+t++67T88//7xatWqltLQ0NWzYUK+99ppWrlypE088MXrM4uJiffXVV5KkY445Rnl5eXGLHwAQf+QqYO9CgQZUs08++UTXXnut5syZoyOPPFIzZ87URRddJEmqU6eOdu7cGd1269atcYujQYMGysjI0NNPP62WLVvqpJNO0nXXXafWrVtHk6S7q3fv3hozZswP9l+6dKkaNWoUt/gAAIlDrgLCi2fQgJ8gLy9PS5Ys0TvvvCMp8nDyp59+qnr16umQQw7Rzp079cgjj0S3P+qoo/Thhx9q/fr1cnc9/fTTP+n8qamp2r59e/Qh7TfffFM7duyIPpydn5+v4cOHKz8/X/vuu69at26tUaNGRZNer169NGnSJC1evDh6zNLnAgAAtQO5Cti7UKABP0HTpk314osv6ne/+50yMjKUk5Ojbdu26fzzz1e7du10/PHH68gjj4xuf+ihh+q6665TTk6OunTpopYtW/6k89erV08vvPCCbrrpJmVkZOiPf/yjnn/+edWrV0+Sog99lya5/Px8rV27Vp06dZIU6RYybtw4DR48WJmZmTruuOM0cuTInxQTACBcyFXA3sUiz6rVDrm5uV5YWJjoMAAACWBmc939h0PNhQy5CgCSVyy5ihY0AAAAAAgJCjQAAAAACAkKNAAAAAAICQo0AAAAAAgJCjQAAAAACAkKNAAAAAAICQo0AAAAAAgJCjQAAAAACAkKNAAAAAAICQo0AAAAAAgJCjQAAAAACAkKNAAAAAAICQo0AAAAAAgJCjQAAAAACAkKNAAAAAAICQo0AAAAAAgJCjQAAAAACAkKNAAAAAAICQo0AAAAAAgJCjQAAAAACAkKNAAAAAAICQo0AAAAAAgJCjQAAAAACIlQFmhmdruZLTCzIjN73cwOTXRMAACURa4CAMRDKAs0Sfe6e4a7Z0l6RdItCY4HAIDdkasAANUulAWau28sM7ufJE9ULAAAlIdcBQCIhzqJDqAiZnanpAGSvpV0UiXbDZE0JJjdZmaLaiC8sGou6ZtEB5FAXH9yX7/Ed5Ds159a0yckV/0oyf73lOvn+rn+5FZlrjL3xNzwM7PJkg4pZ9Uf3f2lMtvdKKm+uw+P4ZiF7p5bjWHuVbh+rj+Zr1/iO+D6q//6yVXVj+vn+rl+rj/RcSRSLN9BwlrQ3P3kGDd9StIESVUmPQAAqhO5CgBQ00L5DJqZHVNm9ixJyxIVCwAA5SFXAQDiIazPoN1lZqmSdkr6RNLQGPd7NH4h7RW4/uSW7Ncv8R1w/TWLXPXjcP3JjetPbsl+/VIM30HCnkEDAAAAAOwqlF0cAQAAACAZUaABAAAAQEjUugLNzG43swVmVmRmr5vZoYmOqSaZ2b1mtiz4Dv5jZgckOqaaZGbnm9liM9tpZkkzjKuZ9Taz5Wb2gZndkOh4apqZPWFmXyfju6XM7DAzm2pmS4K/+1cnOqaaZmb1zexdM5sffAe3JjqmypCnkjtPSeSqZMxVyZynJHLVnuapWvcMmpnt7+4bg89XSWrn7rE+uL3XM7Nekt5092Izu1uS3P0PCQ6rxpjZcYo8sD9S0jB3L0xwSHFnZimS3pd0iqRVkuZI6u/uSxIaWA0ysxMlbZY0xt3TEh1PTTKzlpJauvt7ZtZY0lxJZyfZf3+TtJ+7bzazupJmSLra3WclOLRykaeSO09J5ColYa5K5jwlkav2NE/Vuha00qQX2E9S7apAq+Dur7t7cTA7S1LrRMZT09x9qbsvT3QcNayTpA/c/UN33y7pGUWG/E4a7v6WpHWJjiMR3P0Ld38v+LxJ0lJJrRIbVc3yiM3BbN1gCu1vP3kqufOURK5KxlyVzHlKIlftaZ6qdQWaJJnZnWb2maSLJd2S6HgS6BeSJiY6CMRdK0mflZlfpST60cN/mVkbSdmSZic4lBpnZilmViTpa0lvuHuovwPyVBR5KnmQqyApeXPVnuSpvbJAM7PJZraonOksSXL3P7r7YZKeknRlYqOtflVdf7DNHyUVK/Id1CqxXD+QbMyskaQXJF2zWwtNUnD3EnfPUqQ1ppOZJbQLEXkqufOURK4CypPMuWpP8lRYX1RdKXc/OcZNn5I0QdLwOIZT46q6fjMbKKmvpHyvbQ8Zao/++yeL1ZIOKzPfOliGJBH0Z39B0lPu/mKi40kkd99gZlMl9ZaUsIfxyVPJnackclU5yFVJjlwVEUue2itb0CpjZseUmT1L0rJExZIIZtZb0u8lnenuWxIdD2rEHEnHmNmRZlZPUj9JLyc4JtSQ4MHjf0la6u73JzqeRDCzg0pHAjSzBooMQhDa337yFHkqSZGrkliy56o9zVO1cRTHFySlKjI60ieShrp70tyhMbMPJO0raW2waFaSjQ52jqSHJB0kaYOkInc/NaFB1QAz6yPpr5JSJD3h7ncmNqKaZWZPS+ohqbmkryQNd/d/JTSoGmJmXSVNl7RQkd89SbrJ3SckLqqaZWYZkkYr8vd/H0nPuvttiY2qYuSp5M5TErlKSZirkjlPSeSqPc1Tta5AAwAAAIC9Va3r4ggAAAAAeysKNAAAAAAICQo0AAAAAAgJCjQAAAAACAkKNAAAAAAICQo0AAAAAAgJCjQAAAAACAkKNAAAAAAICQo0AAAAAAgJCjQAAAAACAkKNAAAAAAICQo0AAAAAAgJCjQAAAAACAkKNNRqZlZiZkVlphv2YN8eZvbKTzh3hfub2cdm1jz4PPPHnqOK808zs9zg8wQzOyAe56kihtvM7ORqOE4zM5tqZpvN7O/VERsAhAF5ijwF7K5OogMA4ux7d89KdBCVcfcuNXCOPvE+RwXnvaWaDrVV0p8kpQUTANQW5CmRp4CyaEFDUgruDP5PcLey0Mw6mNlrZrbSzIaW2XR/M3vVzJab2SNmtk+wfy8ze8fM3jOz58ysUbC8t5ktM7P3JP28zPmamdnrZrbYzB6XZGXWbQ7+7BHcTXw+OMZTZmbBuj7Bsrlm9rfy7niaWQMze8bMlprZfyQ12O16m5tZm+A4o8zs/eAcJ5vZ22a2wsw6BdvvZ2ZPmNm7ZjbPzM4Klg80sxfNbFKw/T3B8pTgmIvMbKGZXRssH2Vm5wWf84NjLQyOvW+Z2G4NvsuFZtZ292tz9+/cfYYiCRAAaj3yFHkKyYsCDbVdA9u168iFZdZ9Gty1nC5plKTzJHWWdGuZbTpJ+q2kdpJ+JunnFunycbOkk929g6RCSb8zs/qSHpN0hqQcSYeUOc5wSTPcvb2k/0g6vIJ4syVdE5zvKEknBMcdKek0d8+RdFAF+/5G0hZ3Py44X04F2x0t6S+S2gbTRZK6Shom6aZgmz9KetPdO0k6SdK9ZrZfsC5L0oWS0iVdaGaHBctauXuau6dLerLsCYNrGCXpwmB9nSDeUt8E3+XDQRwAkCzIUz9EnkJSo0BDbfe9u2eVmf5dZt3LwZ8LJc12903uvkbSNvtvP/h33f1Ddy+R9LQiCaKzIonpbTMrknSZpCMUSSIfufsKd3dJ48qc68TSeXd/VdL6CuJ9191XuftOSUWS2gTH/dDdPwq2ebqCfcueY4GkBRVs95G7LwzOsVjSlCDehcH5JKmXpBuC65smqb7+m6ynuPu37r5V0pLg2j+UdJSZPWRmvSVt3O2cqcF53w/mRwfxlnox+HNumRgAIBmQp36IPIWkxjNoSGbbgj93lvlcOl/6/4bvto8r0u3jDXfvX3aFmWVVY0ySVKL4/D+6+7WW/R5Kz2eSznX35WV3NLPjy4vR3debWaakUyUNlXSBpF/8iJjidc0AsDciT5GnkIRoQQMq18nMjgz69F8oaYakWYp06ThaivaDP1bSMkltzOxnwb5lE+NbinTRkJmdJunAPYhhuSJ3/doE8xdWsF3Zc6RJytiDc+zuNUm/LfNsQXZlGwfdafZx9xcU6VbTYbdNlivy3RwdzF8q6f9+QnwAgAjylMhTqF24A4DarkHQ/aHUJHePeQhjSXMk/V2R/vBTJf3H3Xea2UBJT5c+QCzpZnd/38yGSHrVzLYo8sxA42D9rcH2iyXNlPRprAG4+/dmdrmkSWb2XRBTeR6W9KSZLZW0VJFuGD/W7ZL+KmlBkPQ/ktS3ku1bBecuvelzY9mV7r7VzAZJes7M6ihyDY/sSUBm9rGk/SXVM7OzJfVy9yV7cgwACCHy1I9DnkKtZZEuvQDCzMwaufvm4E7hPyStcPcHEh0XAAASeQqoTnRxBPYOvwrusC6W1ESR0bIAAAgL8hRQTWhBAwAAAICQoAUNAAAAAEKCAg0AAAAAQoICDQAAAABCggINAAAAAEKCAg0AAAAAQuL/Ae7jzi3dZiP3AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x432 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(1, 2, figsize = (12, 6))\n",
    "fig.tight_layout()\n",
    "plt_embedding(embedding, ax[0], title=\"Original vocabulary\")\n",
    "plt_embedding(embedding3, ax[1], ylab=\"\", title=\"Expanded vocabulary\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80ad6586-bfb5-4e50-9e1f-1e3ab6120817",
   "metadata": {},
   "source": [
    "The original set of fruits and vegetables are located at exactly the same positions in the new embedding as in the original version. The new elements are placed at reasonable locations within the layout, for example, 'kiwi' is placed nearby other fruits."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65e1698a-6843-4e97-b891-016484ddfeb5",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2195e115-5e86-4450-9106-6ec92a4c87a4",
   "metadata": {},
   "source": [
    "This notebook demonstrated how to adjust an embedding to accomodate an expanding vocabulary. The approach uses an `EmbeddingContraint` callback within a pytorch-lightning trainer."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}