ee046211_tutorial_07_sequential_tasks_rnn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# <img src=\"./assets/course-icon.png\" style=\"height:50px;display:inline\"> ECE 046211 - Technion - Deep Learning\n",
    "---\n",
    "\n",
    "#### <a href=\"https://taldatech.github.io\">Tal Daniel</a>\n",
    "\n",
    "## Tutorial 07 - Sequential Tasks - Recurrent Neural Networks and Transformers\n",
    "---\n",
    "<center><img src=\"./assets/rnn_1.jpg\" style=\"height:200px\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/bubbles/50/000000/checklist.png\" style=\"height:50px;display:inline\"> Agenda\n",
    "---\n",
    "* [Natural Language Processing and Sequences](#-Natural-Language-Processing-and-Sequences)\n",
    "* [Text Preprocessing](#-Text-Preprocessing)\n",
    "* [Evaluation in NLP - Perplexity and BLEU](#-Evaluation-in-NLP---Perplexity-and-BLEU)\n",
    "* [Recurrent Neural Networks (RNNs)](#-Recurrent-Neural-Networks-(RNNs))\n",
    "* [Backpropagation Through Time (BPTT)](#-Backpropagation-Through-Time-(BPTT))\n",
    "* [Long Term Short Memory (LSTM)](#-Long-Term-Short-Memory-(LSTM))\n",
    "* [Gated Recurrent Unit (GRU)](#-Gated-Recurrent-Unit-(GRU))\n",
    "* [Receptence, Weight, Key and Value (RWKV) - New Frontiers in RNNs](#-Receptence,-Weight,-Key-and-Value-(RWKV)---New-Frontiers-in-RNNs)\n",
    "* [xLSTM - Extended Long Short-Term Memory](#-xLSTM---Extended-Long-Short-Term-Memory)\n",
    "* [The Attention Mechanism](#-The-Attention-Mechanism)\n",
    "* [The Transformer](#-The-Transformer)\n",
    "* [Pretrained Models - BERT and GPT](#-Pretrained-Models---BERT-and-GPT)\n",
    "* [Vision Transformer (ViT)](#-Vision-Transformer-(ViT))\n",
    "* [How to Tame Your Transformer](#-How-to-Tame-Your-Transformer?)\n",
    "* [Staying Up-to-Date with Transformers](#-Staying-Up-to-Date-with-Transformers)\n",
    "* [Recommended Videos](#-Recommended-Videos)\n",
    "* [Credits](#-Credits)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# imports for the tutorial\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import time\n",
    "import os\n",
    "\n",
    "# pytorch\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.utils.data import DataLoader\n",
    "from torch.utils.data.dataset import random_split\n",
    "# torchtext\n",
    "import torchtext\n",
    "from torchtext.datasets import IMDB\n",
    "from torchtext.data.utils import get_tokenizer\n",
    "from torchtext.vocab import build_vocab_from_iterator\n",
    "from torchtext.data.functional import to_map_style_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/nolan/64/language.png\" style=\"height:50px;display:inline\"> Natural Language Processing and Sequences\n",
    "---\n",
    "* So far we have dealt with tabular data and images, but what about text or sequences?\n",
    "* Sequence modeling is the field of modeling sequences, e.g., text sentences, videos, stocks rate, trajectories in reinforcement learning or autonomous driving, weather forecast and etc...\n",
    "* Unlike our previous assumption that the data we have is i.i.d., this is not usually the case in sequences (e.g., if you randomly change the words in a sentence, it would be very hard to understand its meaning).\n",
    "* We will focus on text data in the field of natural language processing (NLP)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Language Models\n",
    "---\n",
    "* Language models assign a probability to a text: $p(x_0, \\cdots, x_n)$.\n",
    "* The most popular method is to factorize distribution using the basic probability principles and the Markovian assumption: $$p(x_0, \\cdots, x_n) = p(x_0)p(x_1|x_0)\\cdots p(x_n|x_{n-1}).$$\n",
    "* However, this approach makes many assumptions that are unnecessarily true (e.g. Markovian assumption - dependency only on the previous work and not the entire history)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><img src=\"./assets/lang_model.gif\" style=\"height:250px\"></center>\n",
    "\n",
    "* <a href=\"https://medium.com/perceptronai/recurrent-neural-network-an-introduction-for-beginners-1c13a541c906\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Neural Language Models\n",
    "---\n",
    "* There are classical ways to build language models, but the focus of this course is deep learning, so we will leave the classical approaches to the various NLP courses.\n",
    "* The first question we ask is: how do we convert strings (characters/words) to numbers that can be fed into a neural network?\n",
    "    * This conversion is typically termed **embeddings**. \n",
    "* **Embeddings**: Basically we input the text into a neural network, the neural network will map all this context onto a vector. This vector usually represents the next word (or the next \"token\") and we have some big word embedding matrix (a \"dictionary\" mapping from index to vectors). The word embedding matrix contains a vector for every possible word the model can output.\n",
    "* Embedding layer in PyTorch: <a href=\"https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html\">`nn.Embedding(num_embeddings, embedding_dim)`</a>.\n",
    "* There are several pre-trained models, such as BERT or Word2Vec, that have already trained word embeddings with some objective, and are ready for use in downstream tasks.\n",
    "    * Essentially, we can download these models, use them to convert our text data to numbers, and then just continue as usual as our data is already represented as numbers (just like we have seen in the previous tutorials)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"./assets/embedding_1.png\" style=\"height:250px\"></center>\n",
    "<a href=\"https://www.shanelynn.ie/get-busy-with-word-embeddings-introduction/\">Image Source</a>\n",
    "\n",
    "<center><img src=\"./assets/embedding_2.png\" style=\"height:250px\"></center>\n",
    "<a href=\"https://www.tylercrosse.com/ideas/semantic-search\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"./assets/lookup_table.gif\" style=\"height:250px\"></center>\n",
    "<a href=\"https://lena-voita.github.io/nlp_course/word_embeddings.html\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* The first neural language models were convolutional-based (1D):\n",
    "    * Embed each word as a vector using a lookup table to the embedding matrix, so the word will get the same vector no matter what context it appears in.\n",
    "    * Apply same feed forward network at each time step.\n",
    "    * Unfortunately, fixed length history means it can only condition on bounded context, but these models are very fast!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><img src=\"./assets/lang_model_conv.jpg\" style=\"height:300px\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Forms of Sequence Prediction Tasks\n",
    "---\n",
    "* **One-to-one**: from fixed-sized input to fixed-sized output (e.g. image classification).\n",
    "* **One-to-many**: Sequence output (e.g. image captioning takes an image and outputs a sentence of words).\n",
    "* **Many-to-one**: Sequence input (e.g. sentiment analysis where a given sentence is classified as expressing positive or negative sentiment or given some text predict the next character).\n",
    "* **Many-to-many**: Sequence input and sequence output (e.g. Machine Translation).\n",
    "* **Many-to-many**: Synced sequence input and output (e.g. video classification where we wish to label each frame of the video)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><img src=\"./assets/seq_tasks.jpeg\" style=\"height:250px\"></center>\n",
    "\n",
    "<a href=\"http://karpathy.github.io/2015/05/21/rnn-effectiveness/\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/bubbles/50/000000/connection-sync.png\" style=\"height:50px;display:inline\"> Text Preprocessing\n",
    "---\n",
    "* Before we dive into the specific models, we need to understand how to process text data, as you can't just feed words to neural networks, but you need to give them some numerical representation.\n",
    "    * In classic NLP, the words are sometimes represented as one-hot vectors, where the vector's size is the vocabulary size.\n",
    "* The general steps are:\n",
    "    * Load text as strings into memory.\n",
    "    * **Tokenization**: Split strings into tokens (e.g., words, parts of words and characters).\n",
    "    * **Vocabulary**: Build a table of vocabulary to map the split tokens to numerical indices.\n",
    "    * Convert text into sequences of numerical indices so they can be manipulated by models easily.\n",
    "* We will use <a href=\"https://pytorch.org/text/stable/index.html\">`torchtext`</a>, the official PyTorch library to handle text data.\n",
    "* We use the IMDB dataset: this dataset contains movie reviews which are labeled as `positive` and `negative` (for good and bad reviews, respectively).\n",
    "    * This task is called **sentiment analysis** in NLP, and it is essentially a classification task.\n",
    "* If you want to load other datasets or create a custom dataset from a text file:\n",
    "    * https://pytorch.org/tutorials/beginner/torchtext_custom_dataset_tutorial.html\n",
    "    *  https://pytorch.org/text/stable/datasets.html\n",
    "    *  https://pytorch.org/data/beta/torchdata.datapipes.iter.html#text-datapipes\n",
    "* **Special tokens**:\n",
    "    * `<sos>`/`<bos>` - token that marks the **s**tart/**b**eginning **o**f a **s**entence.\n",
    "    * `<pad>` - token that is used to **pad** sentences that are shorter than the longest sentence in a batch.\n",
    "    * `<eos>` - token that marks the **e**nd **o**f a **s**entence.\n",
    "    * `<unk>` - token that marks **unknown** words (e.g., if the model thinks that no word in the vocabulary is good as next word or you decided to leave out some words, like names)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"./assets/pad_token.jpg\" style=\"height:250px\"></center>\n",
    "\n",
    "<a href=\"https://livebook.manning.com/book/natural-language-processing-in-action/chapter-10/\">Image Source</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# note: downloading datasets requires 'portalocker' -- 'pip install portalocker' or 'conda install -c conda-forge portalocker'\n",
    "ds_train = IMDB(root='./datasets/imdb/aclImdb/', split='train')\n",
    "ds_test = IMDB(root='./datasets/imdb/aclImdb/', split='test')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train examples: 25000, classes: {1, 2}\n"
     ]
    }
   ],
   "source": [
    "# create iterator\n",
    "train_iter = iter(ds_train)\n",
    "labels = [label for (label, text) in train_iter]\n",
    "num_train = len(labels)\n",
    "classes = set(labels)\n",
    "num_class = len(classes)\n",
    "print(f'train examples: {num_train}, classes: {classes}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 'I rented I AM CURIOUS-YELLOW from my video store because of all the controversy that surrounded it when it was first released in 1967. I also heard that at first it was seized by U.S. customs if it ever tried to enter this country, therefore being a fan of films considered \"controversial\" I really had to see this for myself.<br /><br />The plot is centered around a young Swedish drama student named Lena who wants to learn everything she can about life. In particular she wants to focus her attentions to making some sort of documentary on what the average Swede thought about certain political issues such as the Vietnam War and race issues in the United States. In between asking politicians and ordinary denizens of Stockholm about their opinions on politics, she has sex with her drama teacher, classmates, and married men.<br /><br />What kills me about I AM CURIOUS-YELLOW is that 40 years ago, this was considered pornographic. Really, the sex and nudity scenes are few and far between, even then it\\'s not shot like some cheaply made porno. While my countrymen mind find it shocking, in reality sex and nudity are a major staple in Swedish cinema. Even Ingmar Bergman, arguably their answer to good old boy John Ford, had sex scenes in his films.<br /><br />I do commend the filmmakers for the fact that any sex shown in the film is shown for artistic purposes rather than just to shock people and make money to be shown in pornographic theaters in America. I AM CURIOUS-YELLOW is a good film for anyone wanting to study the meat and potatoes (no pun intended) of Swedish cinema. But really, this film doesn\\'t have much of a plot.')\n"
     ]
    }
   ],
   "source": [
    "# let's print an example\n",
    "train_iter = iter(ds_train)\n",
    "print(next(train_iter)) # [label, text], 1 is negative, 2 is positive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialize tokenizer -- there are several types, we use the basic one\n",
    "tokenizer = get_tokenizer(\"basic_english\")\n",
    "# function for tokenization\n",
    "def yield_tokens(data_iter):\n",
    "    for _, text in data_iter:\n",
    "        yield tokenizer(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# build vocabulary\n",
    "max_vocab_size = 55_000\n",
    "vocab = build_vocab_from_iterator(yield_tokens(iter(ds_train)), specials=[\"<unk>\", \"<pad>\"], max_tokens=max_vocab_size)\n",
    "vocab.set_default_index(vocab[\"<unk>\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[132, 10, 41, 465]\n"
     ]
    }
   ],
   "source": [
    "# let's see an example\n",
    "print(vocab(['here', 'is', 'an', 'example']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[54, 39, 4337]\n",
      "['<unk>', '<pad>', 'good', 'how']\n"
     ]
    }
   ],
   "source": [
    "# alternative approach to convert strings to tokens\n",
    "string_to_token = lambda s: vocab.vocab.get_stoi()[s]\n",
    "print([string_to_token(k) for k in ['what', 'they', 'cake']])\n",
    "# convert tokens to string\n",
    "token_to_string = lambda n: vocab.vocab.get_itos()[n]\n",
    "print([token_to_string(i) for i in [0, 1, 57, 95]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# functions to process the raw data strings from the dataset iterators\n",
    "text_pipeline = lambda x: vocab(tokenizer(x))\n",
    "label_pipeline = lambda x: int(x) - 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[132, 10, 2, 41, 465]\n",
      "1\n"
     ]
    }
   ],
   "source": [
    "print(text_pipeline('here is the an example'))\n",
    "print(label_pipeline('2'))  # 1 (negative) -> 0, 2 (positive) -> 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/dusk/64/000000/translate-app.png\" style=\"height:50px;display:inline\"> Evaluation in NLP - Perplexity and BLEU\n",
    "---\n",
    "#### Perplexity\n",
    "---\n",
    "**Perplexity** measures the language model quality. **A better language model should allow us to predict the next token more accurately**. Thus, it should allow us to spend fewer bits in compressing the sequence. We can measure it by the cross-entropy loss averaged over all the $n$ tokens of a sequence: $$ \\frac{1}{n}\\sum_{i=1}^n -\\log P(x_t|x_{t-1}, ..., x_1), $$ where $P$ is given by the language model and $x_t$ is the actual token observed at time step $t$ from the sequence. The **perplexity** is defined as $$ \\exp\\left(\\frac{1}{n}\\sum_{i=1}^n -\\log P(x_t|x_{t-1}, ..., x_1)\\right). $$\n",
    "* Perplexity can be best understood as the harmonic mean of the number of real choices that we have when deciding which token to pick next. \n",
    "* In the **best** case scenario, the model always perfectly estimates the probability of the label token as 1. **In this case the perplexity of the model is 1**.\n",
    "* In the **worst** case scenario, the model always predicts the probability of the label token as 0. In this situation, the perplexity is **positive infinity**.\n",
    "* At the **baseline**, the model predicts a uniform distribution over all the available tokens of the vocabulary. In this case, the perplexity equals the number of unique tokens of the vocabulary.\n",
    "    *  In fact, if we were to store the sequence without any compression, this would be the best we could do to encode it. Hence, this provides a nontrivial upper bound that any useful model must beat."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Bilingual Evaluation Understudy (BLEU) Score\n",
    "---\n",
    "* **BLEU** score is an algorithm for evaluating the quality of text which has been machine-translated from one natural language to another.\n",
    "* Scores are calculated for individual translated segments—generally sentences—by comparing them with a set of good quality reference translations. \n",
    "* Those scores are then averaged over the whole corpus to reach an estimate of the translation's overall quality. \n",
    "* **Intelligibility or grammatical correctness are not taken into account**.\n",
    "* BLEU's output is always a number between 0 and 1. \n",
    "* This value indicates how similar the candidate text is to the reference texts, with values closer to 1 representing more similar texts.\n",
    "    * Few human translations will attain a score of 1, since this would indicate that the candidate is identical to one of the reference translations. \n",
    "    * For this reason, it is not necessary to attain a score of 1. Because there are more opportunities to match, adding additional reference translations will increase the BLEU score.\n",
    "* BLEU uses a modified version of the precision score between a candidate translation and translation ground-truth (it is better to provide more than one reference), and it is based on $n$-grams.\n",
    "* <a href=\"https://en.wikipedia.org/wiki/BLEU\">Read More</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/nolan/64/re-enter-pincode.png\" style=\"height:50px;display:inline\"> Recurrent Neural Networks (RNNs)\n",
    "---\n",
    "* The idea of **Recurrent Neural Networks (RNNs)**: save the output of a particular layer and feed it back to the input in order to predict the output of the layer.\n",
    "* Every time step we maintain some state (received from the previous time step)--**hidden state**, which represents what we’ve read so far. This is combined with current word being read and used at later state. Then we repeat this process for as many time steps as we need.\n",
    "\n",
    "<center><img src=\"./assets/rnn_2.gif\" style=\"height:200px\"></center>\n",
    "\n",
    "* <a href=\"https://medium.com/perceptronai/recurrent-neural-network-an-introduction-for-beginners-1c13a541c906\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Let $x$ denote the input layer, $h$ the hidden layer and $y$ the output layer.\n",
    "* Let $A, B \\text{ and } C$ be some network parameters used to improve the output of the model.\n",
    "* At any given time $t$, the current input is a combination of the input at $x(t)$ and $x(t-1)$ (through $h(t-1)$).\n",
    "\n",
    "<center><img src=\"./assets/rnn_3.gif\" style=\"height:250px\"></center>\n",
    "\n",
    "* <a href=\"https://www.simplilearn.com/tutorials/deep-learning-tutorial/rnn\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### The Hidden State of RNN Cells\n",
    "---\n",
    "* For each element in the input sequence, each layer computes the following function: $$ h_t = tanh\\left(W_{ih}x_t +b_{ih} + W_{hh}h_{(t-1)} + b_{hh}\\right), $$ where $h_t$ is the hidden state at time $t$, $x_t$ is the input at time $t$, and $h_{(t-1)}$ is the hidden state of the previous layer at time $t-1$ or the initial hidden state at time 0.\n",
    "\n",
    "<center><img src=\"./assets/rnn_4.gif\" style=\"height:250px\"></center>\n",
    "\n",
    "* Image by Michael Nguyen\n",
    "\n",
    "* In PyTorch: <a href=\"https://pytorch.org/docs/stable/generated/torch.nn.RNN.html\">`torch.nn.RNN(input_size, hidden_size, num_layers...)`</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Disadvantages of RNNs\n",
    "---\n",
    "* The whole history of the document reading is compressed into a fixed-size vector at each time step, which is the bottleneck of this model.\n",
    "* **Gradients tend to vanish** with long contexts.\n",
    "* Not possible to parallelize over time-steps, so **slow training**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/nolan/64/time-machine.png\" style=\"height:50px;display:inline\"> Backpropagation Through Time (BPTT)\n",
    "---\n",
    "* **Forward propagation** in an RNN is relatively straightforward and is the same as MLPs (but delayed, as we need the previous output).\n",
    "* **Backpropagation through time (BPTT)**: a specific application of backpropagation in RNNs. \n",
    "* BPTT requires us to expand or *unroll* the computational graph of an RNN one time step at a time to obtain the dependencies among model variables and parameters.\n",
    "* Then, based on the chain rule, we apply backpropagation to compute and store gradients. \n",
    "    * Since sequences can be rather long, the dependency can be rather lengthy. \n",
    "    * For instance, for a sequence of 1000 characters, the first token could potentially have significant influence on the token at the final position. This is not really computationally feasible (it takes too long and requires too much memory) and it requires over 1000 matrix products before we would arrive at that very elusive gradient.\n",
    "* High powers of matrices can lead to **divergent or vanishing eigenvalues -- exploding or vanishing gradients**.\n",
    "* For efficient computation, **intermediate values are cached** during backpropagation through time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Computing Gradients in BPTT\n",
    "---\n",
    "* Consider an RNN *without* bias parameters, whose activation function in the hidden layer uses the identity mapping ( $\\phi(x)=x$ ). \n",
    "* For time step $t$ , let the single example input and the label be $x_t\\in \\mathbb{R}^d$ and $y_t$ , respectively.\n",
    "* The hidden state  $h_t\\in \\mathbb{R}^h$  and the output $o_t\\in \\mathbb{R}^q$  are computed as: $$ h_t=W_{hx}x_t+W_{hh}h_{t−1}, $$ $$ o_t = W_{qh}h_t, $$ where $W_{hx}\\in \\mathbb{R}^{h\\times d}$,  $W_{hh}\\in \\mathbb{R}^{h\\times h}$, and $W_{qh}\\in \\mathbb{R}^{q\\times h}$ are the weight parameters.\n",
    "* Denote by $l(o_t,y_t)$ the loss at time step $t$. Our objective function, the loss over $T$ time steps from the beginning of the sequence is thus: $$ L = \\frac{1}{T}\\sum_{t=1}^T l(o_t,y_t).$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Computational graph for 3 time steps:\n",
    "\n",
    "<center><img src=\"./assets/bptt_1.svg\" style=\"height:250px\"></center>\n",
    "\n",
    "* For example, the computation of the hidden states of time step 3,  $h_3$ , depends on the model parameters $W_{hx}$ and $W_{hh}$, the hidden state of the last time step $h_2$, and the input of the current time step $x_3$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* According to the dependencies in the graph, we can traverse in the opposite direction of the arrows to calculate and store the gradients in turn. We can look at this as unrolled backpropagation.\n",
    "* Differentiating the objective function with respect to the model output at any time step $t$ is straightforward: $$ \\frac{\\partial L}{\\partial o_t} = \\frac{\\partial l(o_t, y_t)}{T\\cdot \\partial o_t} \\in \\mathbb{R}^q.$$\n",
    "* Now, we can calculate the gradient of the objective function with respect to the parameter $W_{qh}$ in the output layer. Note that the objective function $L$ depends on $W_{qh}$ via $o_1,…,o_T$: $$ \\frac{\\partial L}{\\partial W_{qh}}=\\sum_{t=1}^T \\text{prod} \\left(\\frac{\\partial L}{\\partial o_t}, \\frac{\\partial o_t}{\\partial W_{qh}} \\right) = \\sum_{t=1}^T \\frac{\\partial L}{\\partial o_t}h_t^T $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* We continue down the graph, and we need the derivatives w.r.t $h_t$.\n",
    "* At the final time step $T$ the objective function $L$ depends on the hidden state $h_T$ only via $o_T$: $$ \\frac{\\partial L}{\\partial h_T} = \\text{prod}\\left(\\frac{\\partial L}{\\partial o_T}, \\frac{\\partial o_T}{\\partial h_T} \\right) = W_{qh}^T \\frac{\\partial L}{\\partial o_T}\\in \\mathbb{R}^h. $$\n",
    "* It gets trickier for any time step $t<T$, where the objective function $L$ depends on $h_t$ via $h_{t+1}$ and $o_t$. According to the chain rule: $$\\frac{\\partial L}{\\partial h_t} = \\text{prod}\\left(\\frac{\\partial L}{\\partial h_{t+1}}, \\frac{\\partial h_{t+1}}{\\partial h_t} \\right) + \\text{prod}\\left(\\frac{\\partial L}{\\partial o_t}, \\frac{\\partial o_t}{\\partial h_t} \\right) = W_{hh}^T\\frac{\\partial L}{\\partial h_{t+1}} + W_{qh}^T\\frac{\\partial L}{\\partial o_t}. $$\n",
    "* (EXERCISE) Expanding the recurrent computation for any time step $1\\leq t \\leq T$ gives: $$ \\frac{\\partial L}{\\partial h_t} = \\sum_{i=t}^T \\left(W_{hh}^T \\right)^{T-i}W_{qh}^T\\frac{\\partial L}{\\partial o_{T+t-i}} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Notice that the simple linear equation already exhibits some key problems of long sequence models: it involves potentially very large powers of $W_{hh}^T$. \n",
    "* In it, eigenvalues smaller than 1 **vanish** and eigenvalues larger than 1 **diverge**. This is numerically unstable, which manifests itself in the form of **vanishing and exploding gradients**. \n",
    "* One way to address this is to **truncate** the time steps at a computationally convenient size. In practice, this truncation is effected by detaching the gradient after a given number of time steps.\n",
    "* GRUs and LSTMs cells can alleviate this better as we will soon see."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Finally, the objective function $L$ depends on model parameters $W_{hx}$ and $W_{hh}$ in the hidden layer via hidden states  $h_1,…,h_T$: $$ \\frac{\\partial L}{\\partial W_{hx}} = \\sum_{i=t}^T \\text{prod}\\left(\\frac{\\partial L}{\\partial h_t}, \\frac{\\partial h_t}{\\partial W_{hx}} \\right) = \\sum_{i=t}^T \\frac{\\partial L}{\\partial h_t}x_t^T, $$\n",
    "$$ \\frac{\\partial L}{\\partial W_{hh}} = \\sum_{i=t}^T \\text{prod}\\left(\\frac{\\partial L}{\\partial h_t}, \\frac{\\partial h_t}{\\partial W_{hh}} \\right) = \\sum_{i=t}^T \\frac{\\partial L}{\\partial h_t}h_{t-1}^T$$\n",
    "\n",
    "* BPTT computes and stores the above gradients in turn. Specifically, stored intermediate values are reused to avoid duplicate calculations, such as storing  $\\frac{\\partial L}{\\partial h_t}$ to be used in computation of both $\\frac{\\partial L}{\\partial W_{hx}}$ and $\\frac{\\partial L}{\\partial W_{hh}}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/dusk/64/000000/memory-slot.png\" style=\"height:50px;display:inline\"> Long Term Short Memory (LSTM)\n",
    "---\n",
    "* As mentioned before, during backpropagation, RNNs suffer from the vanishing gradient problem, which essentially creates a **short memory**.\n",
    "* Long short-term memory (LSTM) is a type of recurrent cell that tries to preserve long term information. The idea of LSTM was presented back in 1997, but flourished in the age of deep learning.\n",
    "* LSTM introduces a memory cell that has the same shape as the hidden state, engineered to record additional information.\n",
    "* The memory is controlled by 3 main gates: \n",
    "    * **Input gate**: decides when to read data into the cell.\n",
    "    * **Output gate**: outputs the entries from the cell.\n",
    "    * **Forget gate**: a mechanism to reset the content of the cell.\n",
    "* These gates learn which information is relevant to forget or remember during the training process. The gates contain a sigmoid activation function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><img src=\"./assets/lstm_1.svg\" style=\"height:250px\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Suppose that there are $h$ hidden units, the batch size is $n$, and the number of inputs is $d$. Thus, the input is $X_t\\in \\mathbb{R}^{n\\times d}$ (number of examples: $n$, number of inputs: $d$) and the hidden state of the previous time step is $H_{t-1}\\in\\mathbb{R}^{n\\times h}$ (number of hidden units: $h$). We define the following at timestep $t$:\n",
    "    * **Input gate**: $$ I_t = \\sigma(X_tW_{xi} +H_{t-1}W_{hi} +b_i) \\in \\mathbb{R}^{n\\times h},$$\n",
    "    * **Forget gate**: $$ F_t = \\sigma(X_tW_{xf} +H_{t-1}W_{hf} +b_f) \\in \\mathbb{R}^{n\\times f}, $$\n",
    "    * **Output gate**: $$ O_t = \\sigma(X_tW_{xo} +H_{t-1}W_{ho} +b_o) \\in \\mathbb{R}^{n\\times o}, $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Memory Cell\n",
    "---\n",
    "* The candidate memory cell $\\tilde{C}_t$ is defined as follows: $$ \\tilde{C}_t = \\text{tanh}(X_tW_{xc} +H_{t-1}W_{hc} +b_c) \\in \\mathbb{R}^{n\\times c}$$\n",
    "    * Note the difference in notations: a candidate memory is denoted with $\\tilde{\\cdot}$ while the actual memory is without the tilde.\n",
    "* The input gate $I_t$ governs how much we take new data into account via $\\tilde{C}_t$ and the forget gate $F_t$ addresses how much of the old memory cell content $C_{t-1}$ we retain. This yields: $$ C_t = F_t \\odot C_{t-1} + I_{t} \\odot \\tilde{C}_t$$\n",
    "    * $\\odot$ is the element-wise product operator (Hadamard).\n",
    "* If the forget gate is always approximately 1 and the input gate is always approximately 0, the past memory cells $C_{t-1}$ will be saved over time and passed to the current time step. \n",
    "* This design is introduced to alleviate the vanishing gradient problem and to better capture long range dependencies within sequences.\n",
    "* Finally, the hidden state at time $t$: $$ H_t = O_t \\odot \\text{tanh}(C_t).$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><img src=\"./assets/lstm_2.gif\" style=\"height:350px\"></center>\n",
    "\n",
    "* <a href=\"https://becominghuman.ai/long-short-term-memory-part-1-3caca9889bbc\">Image Source</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shapes: output - torch.Size([5, 3, 20]), hidden - torch.Size([2, 3, 20]), memory - torch.Size([2, 3, 20])\n"
     ]
    }
   ],
   "source": [
    "rnn = nn.LSTM(input_size=10, hidden_size=20, num_layers=2)  # batch_first=False\n",
    "x = torch.randn(5, 3, 10)  # 5 words per senetence, 3 sentences (batch_size), embedding dimension of each word is 10\n",
    "h0 = torch.randn(2, 3, 20)  # initialize hidden states per layer\n",
    "c0 = torch.randn(2, 3, 20)  # initialize memory per layer\n",
    "output, (hn, cn) = rnn(x, (h0, c0))\n",
    "print(f'shapes: output - {output.shape}, hidden - {hn.shape}, memory - {cn.shape}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/color/96/000000/front-gate-open.png\" style=\"height:50px;display:inline\"> Gated Recurrent Unit (GRU)\n",
    "---\n",
    "* Unlike regular RNNs, Gated Recurrent Units (GRUs) support gating of the hidden state.\n",
    "* GRUs have two mechanism to control when a hidden state should be updated, **update gate** and also when it should be reset, **reset gate**.\n",
    "* **Reset gate**: allows to control how much of the previous state should be remembered, helps capture short-term dependencies in sequences.\n",
    "* **Update gate**: allows to control how much of the new state is just a copy of the old state, help capture long-term dependencies in sequences.\n",
    "* Unlike LSTMS, GRUs don't have a memory component, and thus are much faster to update (which results in faster training), but usually LSTMs perform better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><img src=\"./assets/gru_1.svg\" style=\"height:300px\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Suppose that the input is a mini-batch $X_t \\in \\mathbb{R}^{n\\times d}$ for a given time step $t$ and the hidden state of the previous time step is $H_{t−1}\\in \\mathbb{R}^{n\\times h}$.\n",
    "* The reset gate $R_t \\in \\mathbb{R}^{n \\times h}$ and update gate $Z_t \\in \\mathbb{R}^{n \\times h}$ are computed as follows: $$ R_t = \\sigma(X_tW_{xr} + H_{t-1}W_{hr} +b_r), $$ $$ Z_t = \\sigma(X_tW_{xz} + H_{t-1}W_{hz} +b_z), $$ where $W_{xr}, W_{xz} \\in \\mathbb{R}^{d \\times h}$ and $W_{hr}, W_{hz} \\in \\mathbb{R}^{h \\times h}$ are weight parameters, and $b_r, b_z \\in \\mathbb{R}^{1 \\times h}$ are biases."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### GRUs Hidden State\n",
    "---\n",
    "* The *candidate* hidden state $\\tilde{H}_{t} \\in \\mathbb{R}^{n \\times h}$ at timestep $t$ is defined as: $$ \\tilde{H}_{t} = \\text{tanh}\\left(X_t W_{xh} + (R_t \\odot H_{t-1})W_{hh} \\right) + b_h$$\n",
    "    * The result is a candidate since we still need to incorporate the action of the *update gate*.\n",
    "* Finally, the new hidden state $H_t$ and the final update to the GRU in timestep $t$: $$ H_t = Z_t \\odot H_{t-1} +(1-Z_t) \\odot \\tilde{H}_t. $$\n",
    "* Whenever the update gate $Z_t$ is close to 1, we simply retain the old state. In this case the information from $X_t$ is essentially ignored, effectively skipping time step $t$ in the dependency chain. \n",
    "* In contrast, whenever $Z_t$ is close to 0, the new latent state $H_t$ approaches the candidate latent state $\\tilde{H}_t$.\n",
    "* These designs can help us cope with the vanishing gradient problem in RNNs and better capture dependencies for sequences with large time step distances."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "shapes: output - torch.Size([5, 3, 20]), hidden - torch.Size([2, 3, 20])\n"
     ]
    }
   ],
   "source": [
    "rnn = nn.GRU(input_size=10, hidden_size=20, num_layers=2)  \n",
    "x = torch.randn(5, 3, 10)  # 5 words per senetence, 3 sentences (batch_size), embedding dimension of each word is 10\n",
    "h0 = torch.randn(2, 3, 20)  # initialize hidden states per layer\n",
    "output, hn = rnn(x, h0)\n",
    "print(f'shapes: output - {output.shape}, hidden - {hn.shape}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/bubbles/50/000000/fire-element.png\" style=\"height:50px;display:inline\"> PyTorch RNN Model Example\n",
    "----\n",
    "Following is an example of building a classifier with LSTMs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# https://github.com/FernandoLpz/Text-Classification-LSTMs-PyTorch\n",
    "class TweetClassifier(nn.Module):\n",
    "\n",
    "    def __init__(self, args):\n",
    "        super(TweetClassifier, self).__init__()\n",
    "\n",
    "        self.batch_size = args.batch_size\n",
    "        self.hidden_dim = args.hidden_dim\n",
    "        self.LSTM_layers = args.lstm_layers\n",
    "        self.input_size = args.max_words  # embedding dimension\n",
    "\n",
    "        self.dropout = nn.Dropout(0.5)\n",
    "        self.embedding = nn.Embedding(self.input_size, self.hidden_dim, padding_idx=0)\n",
    "        self.lstm = nn.LSTM(input_size=self.hidden_dim, hidden_size=self.hidden_dim, num_layers=self.LSTM_layers,\n",
    "                            batch_first=True)  # NOTE: batch_first=True:\n",
    "        # input and output tensors are provided as (batch, seq, feature) instead of (seq, batch, feature).\n",
    "        self.fc1 = nn.Linear(in_features=self.hidden_dim, out_features=257)\n",
    "        self.fc2 = nn.Linear(257, 1)\n",
    "\n",
    "    def forward(self, x):\n",
    "        h = torch.zeros((self.LSTM_layers, x.size(0), self.hidden_dim))\n",
    "        c = torch.zeros((self.LSTM_layers, x.size(0), self.hidden_dim))\n",
    "\n",
    "        torch.nn.init.xavier_normal_(h)\n",
    "        torch.nn.init.xavier_normal_(c)\n",
    "\n",
    "        out = self.embedding(x)\n",
    "        out, (hidden, context) = self.lstm(out, (h, c))\n",
    "        out = self.dropout(out)\n",
    "        out = torch.relu_(self.fc1(out[:, -1, :]))  # \"last-token pooling\": (batch, seq, feature) -> (batch, feature)\n",
    "        # can also use other pooling methods like \"mean\" or \"max\", e.g., self.fc1(out.mean(dim=1))\n",
    "        out = self.dropout(out)\n",
    "        out = torch.sigmoid(self.fc2(out))\n",
    "\n",
    "        return out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <img src=\"https://img.icons8.com/bubbles/50/new.png\" style=\"height:50px;display:inline\"> Receptence, Weight, Key and Value (RWKV) - New Frontiers in RNNs\n",
    "----\n",
    "* <a href=\"https://arxiv.org/abs/2405.04517\">RWKV: Reinventing RNNs for the Transformer Era - Peng et al., 2024</a>\n",
    "* Receptence, Weight, Key and Value (RWKV): an attention-free, parallelizable RNN that can reach transformer-level language models performance.\n",
    "* Fast training and inference, lower VRAM footprint than transformers.\n",
    "* Competitive with Tranformers on most language-models benchmarks!\n",
    "* As in RNNs, to compute the state at time-step $t+1$ only the hidden state at time-step $t$ is required.\n",
    "* \"GPT\" mode: quickly compute the hidden state - <a href=\"https://github.com/BlinkDL/ChatRWKV\">ChatRWKV</a>.\n",
    "* <a href=\"https://github.com/BlinkDL/RWKV-LM\">Code on GitHub</a>\n",
    "\n",
    "<center><img src=\"./assets/rwkv_att.png\" style=\"height:200px\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <img src=\"https://img.icons8.com/bubbles/50/x.png\" style=\"height:50px;display:inline\"> xLSTM - Extended Long Short-Term Memory\n",
    "----\n",
    "* <a href=\"https://arxiv.org/abs/2405.04517\">xLSTM - Extended Long Short-Term Memory - Beck et al., 2024</a>\n",
    "* A new LSTM-based neural network, adapted for the attention and Tranformer era.\n",
    "* The new development strives to answer whether it is possible to scale LSTMs to compete in the language-modeling arena.\n",
    "* Introduce new exponential gating with appropriate normalization and stabilization techniques.\n",
    "* Modify the LSTM memory structure:\n",
    "    * **sLSTM**: a scalar memory, a scalar update, and new memory mixing.\n",
    "    * **mLSTM**: fully parallelizable with a matrix memory and a covariance update rule.\n",
    "* **xLSTM** = **sLSTM** + **mLSTM**: Integrating them into residual block backbones yields xLSTM blocks.\n",
    "* <a href=\"https://github.com/NX-AI/xlstm\">Code on GitHub</a>\n",
    "\n",
    "<center><img src=\"./assets/xlstm.PNG\" style=\"height:400px\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/cute-clipart/64/000000/alarm.png\" style=\"height:50px;display:inline\"> The Attention Mechanism\n",
    "---\n",
    "* So far, we have processed the inputs sequentially (recurrence), which has led to several issues. Can we process sequences differently?\n",
    "* **Idea**: we can look at all the different tokens (words) at **the same time** and learn to “pay attention“ (give each one a weight) to the correct ones depending on the task at hand.\n",
    "* **Attention** is a generalized pooling method (extracting features from sequences) with bias alignment over inputs. \n",
    "* An input of the attention layer is called a **query**. \n",
    "* For a query, attention returns an output based on the memory — a set of **key-value** pairs encoded in the attention layer.\n",
    "* There are two main types of attention: **self-attention** and **cross-attention**, and in each type, we can define **hard** attention or **soft** attention.\n",
    "* Our focus will be **self-attention**: computing a weighted average of feature representations with the weights proportional to a similarity score between pairs of representations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><img src=\"./assets/attention_graph.png\" style=\"height:300px\"></center>\n",
    "\n",
    "<a href=\"https://theaisummer.com/attention/\">Image by Nikolas Adaloglou</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "* Consider a set of $t$ input $x$'s, where $x_i \\in \\mathbb{R}^d$. This set can be represented as the matrix $X \\in \\mathbb{R}^{d \\times t}$.\n",
    "* In self-attention, we define the hidden representation $h$ as a linear combination of the inputs (or some features of the inputs): $$ h = \\alpha_1 x_1 +...\\alpha_t x_t. $$\n",
    "* Using the matrix representation described above, and stacking all the $\\alpha$'s as a vector $a \\in \\mathbb{R}^{t}$ we can write the hidden layer as the matrix product: $ h=Xa \\in \\mathbb{R}^d $.\n",
    "* Depending on the constraints we impose on the vector $a$, we can achieve hard or soft attention.\n",
    "* **Hard attention**: we impose the following constraint on the alphas: $||a||_0=1$, which means that $a$ is a one-hot vector. That hidden representation reduces the input $x_i$ corresponding to the element $\\alpha_i=1$.\n",
    "* **Soft attention**: With soft attention, we impose that $||a||_1=1$. The hidden representation is a linear combination of the inputs where the coefficients sum up to 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Obtaining $\\alpha_i$'s\n",
    "---\n",
    "* **Hard attention**: $$a = \\text{argmax}(X^Tx_i) \\in \\mathbb{R}^t $$\n",
    "    * We get a one-hot vector of alphas.\n",
    "* **Soft attention**: $$ a = \\text{soft(arg)max}(X^Tx_i) \\in \\mathbb{R}^t  $$\n",
    "    * The components of the resulting vector $a$ sum to 1.\n",
    "* The components of the vector $a$ are also called **“scores”** because the scalar product between two vectors tells us how **aligned or similar** two vectors are. \n",
    "* Therefore, the elements of $a$ provide information about the similarity of the overall set to a particular $x_i$.\n",
    "* Generating $a$ this way gives a set of them, one for each $x_i$. \n",
    "* Moreover, each $a_i \\in \\mathbb{R}^t$ so we can stack the alphas in a matrix $A \\in \\mathbb{R}^{t \\times t}$.\n",
    "* Since each hidden state is a linear combination of the inputs $X$ and a vector $a$, we obtain a set of $t$ hidden states, which we can stack into a matrix $H \\in \\mathbb{R}^{d \\times t}$: $$ H = XA $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><img src=\"./assets/attn_3.png\" style=\"height:300px\"></center>\n",
    "\n",
    "* Visualization of the outputs upon using two heads of attention with the same query.\n",
    "* We can see that if the Query word is **it**, the first head focuses more on the words **the animal**, and the second head focuses more on the word **tired**. \n",
    "* Hence, the final context representation will be focusing on all the words **the, animal** and **tired**, and thus is a superior representation as compared to the traditional way.\n",
    "* <a href=\"https://blogs.oracle.com/datascience/multi-head-self-attention-in-nlp\">Images Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/nolan/64/key.png\" style=\"height:50px;display:inline\"> Attention with Key-Value Mechanism\n",
    "---\n",
    "* Now that we have a way to calculate scores based on similarity, we apply it in a key-value fashion.\n",
    "* A key-value store is a paradigm designed for storing (saving), retrieving (querying) and managing associative arrays (dictionaries / hash tables).\n",
    "* Imaginative example: **The Lasagna recipe** :\n",
    "    * **Query** $Q$: say we wanted to find a recipe to make lasagna. We have a recipe book and search for “lasagna” - this is the **query**. \n",
    "    * **Key** $K$: this query is checked against all possible **keys** in your dataset - in this case, this could be the titles of all the recipes in the book. \n",
    "    * We check how aligned the query is with each title to find the **maximum matching score** between the query and all the respective keys. \n",
    "    * **Value** $V$: if our output is the argmax function - we retrieve the *single* recipe (value) with the highest score. Otherwise, if we use a soft argmax function, we will get a probability distribution and can retrieve in order from the most similar content to less and less relevant recipes matching the query.\n",
    "    * Summary: keys and queries - titles of recipes, values - the actual recipe."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><img src=\"./assets/attention_kqv.png\" style=\"height:300px\"></center>\n",
    "\n",
    "<a href=\"https://theaisummer.com/transformer/\">Image by Nikolas Adaloglou</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **attention formula** which revolutionized the field:\n",
    "\n",
    "<center><img src=\"./assets/qkv_attention_formula.png\" style=\"height:250px\"></center>\n",
    "\n",
    "<a href=\"https://lena-voita.github.io/nlp_course/seq2seq_and_attention.html\">Image by Lena Voita</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Queries, Keys and Values\n",
    "---\n",
    "* We denote the vectors $q ,k$ and $v$ as the query, key and value vectors in **self-attention**, respectively and their corresponding learnable parameters matrices $W_{q}, W_k$ and $W_v$ that are used to **project** our input embeddings $x \\in \\mathbb{R}^{d_{x}}$: $$ q = W_q x \\in \\mathbb{R}^{d }$$ $$ k = W_k x \\in \\mathbb{R}^{d}$$ $$ v = W_v x \\in \\mathbb{R}^{d_v}$$\n",
    "*  We also usually **don't** include any non-linearities since attention is completely based on **orientation**.\n",
    "* In order to compare the query against all possible keys, $q$ and $k$ must have the same dimensionality, i.e. $q, k \\in \\mathbb{R}^d$.\n",
    "* $v$ can be of any dimension, $v \\in \\mathbb{R}^{d_v}$. \n",
    "    * In the lasagna recipe example - we need the query to have the dimension as the keys, i.e. the titles of the different recipes that we’re searching through. The dimension of the corresponding recipe retrieved, $v$, can be arbitrarily long though.\n",
    "* For simplicity, we will assume that everything has the same dimension $d$ ($d_v=d$), which is what we usually do in practice as well."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Given a set of $x$'s, a set of queries, a set of keys and a set of values, we can stack these sets into matrices each with $t$ columns since we stacked $t$ vectors; each vector has height $d$ (hidden dimension): $$ \\{x_i\\}_{i=1}^t \\to \\{q_i\\}_{i=1}^t, \\{k_i\\}_{i=1}^t, \\{v_i\\}_{i=1}^t \\to Q, K, V \\in \\mathbb{R}^{d \\times t} $$\n",
    "* We compare one query $q$ against the matrix of all keys $K$: $$ a = \\text{[soft]argmax}(K^T q) \\in \\mathbb{R}^t $$\n",
    "* Then the hidden layer is going to be the linear combination of the columns of $V$ weighted by the coefficients in $a$: $$ h=Va \\in \\mathbb{R}^{d} $$\n",
    "* Since we have $t$ queries, we’ll get $t$ corresponding $a$ weights and therefore a matrix $A \\in \\mathbb{R}^{t \\times t}$, which yields: $$ H = VA \\in \\mathbb{R}^{d \\times t}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><img src=\"./assets/attention_kqv_2.png\" style=\"height:300px\"></center>\n",
    "\n",
    "<a href=\"https://theaisummer.com/transformer/\">Image by Nikolas Adaloglou</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><img src=\"./assets/self-attention_qkv.png\" style=\"height:300px\"></center>\n",
    "\n",
    "<a href=\"http://wenqianzhao.cn/2020/12/29/transformer/\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* For implementation, we can speed up computation by stacking all the $W$’s into one tall $W$ and then calculate $q, k, v$ in one go: $$ \\begin{bmatrix} q \\\\ k \\\\ v \\end{bmatrix} = \\begin{bmatrix}W_q \\\\ W_k \\\\ W_v\\end{bmatrix} x \\in \\mathbb{R}^{3d} $$\n",
    "* **Multi-head Attention**: one \"head\" of attention corresponds to **one dictionary** of queries, keys values. For $h$ heads (\"dictionaries\") of attention we have $h$ $q$’s, $h$ $k$’s and $h$ $v$’s and we end up with a vector: $$  \\begin{bmatrix} q^1 \\\\ \\vdots \\\\ q^h \\\\ k^1 \\\\ \\vdots \\\\ k^h \\\\ v^1 \\\\ \\vdots \\\\ v^h \\end{bmatrix} = \\begin{bmatrix}W_q^1 \\\\ \\vdots \\\\ W_q^h \\\\ W_k^1 \\\\ \\vdots \\\\ W_k^h \\\\ W_v^1 \\\\ \\vdots \\\\  W_v^h\\end{bmatrix} x \\in \\mathbb{R}^{3hd} $$\n",
    "* We can still transform the multi-headed values to have the original dimension $\\mathbb{R}^{d}$ by using a linear layer with weights $W_h \\in \\mathbb{R}^{d \\times hd}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><img src=\"./assets/transformer_heads.png\" style=\"height:350px\"></center>\n",
    "\n",
    "<a href=\"https://lena-voita.github.io/nlp_course/seq2seq_and_attention.html#multi_head_attention\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* The idea of multi-head attention is very similar to **group convolution**, where each filter (or \"head\") operates on *part* (=group) of the input channels.\n",
    "\n",
    "<center><img src=\"./assets/group_conv.gif\" style=\"height:300px\"></center>\n",
    "\n",
    "<a href=\"https://animatedai.github.io/\">Image Source</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class MultiHeadAttention(nn.Module):\n",
    "    def __init__(self, d_model, num_heads, dropout, d_input=None):\n",
    "        super().__init__()\n",
    "        self.num_heads = num_heads\n",
    "        self.d_model = d_model\n",
    "        if d_input is None:\n",
    "            d_xq = d_xk = d_xv = d_model\n",
    "        else:\n",
    "            d_xq, d_xk, d_xv = d_input\n",
    "            \n",
    "        # Make sure that the embedding dimension of model is a multiple of number of heads\n",
    "        assert d_model % self.num_heads == 0\n",
    "\n",
    "        self.d_k = d_model // self.num_heads  # here d is divided between the heads\n",
    "        # each head has hidden dimension d\n",
    "        \n",
    "        # These are still of dimension d_model. They will be split into number of heads \n",
    "        self.W_q = nn.Linear(d_xq, d_model, bias=False)\n",
    "        self.W_k = nn.Linear(d_xk, d_model, bias=False)\n",
    "        self.W_v = nn.Linear(d_xv, d_model, bias=False)\n",
    "        \n",
    "        # Outputs of all sub-layers need to be of dimension d_model\n",
    "        self.W_h = nn.Linear(d_model, d_model)\n",
    "        \n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        \n",
    "    def scaled_dot_product_attention(self, Q, K, V):\n",
    "        batch_size = Q.size(0) \n",
    "        k_length = K.size(-2) \n",
    "        \n",
    "        # Scaling by d_k so that the soft(arg)max doesn't saturate\n",
    "        Q = Q / np.sqrt(self.d_k)                         # (bs, n_heads, q_length, dim_per_head)\n",
    "        scores = torch.matmul(Q, K.transpose(2, 3))          # (bs, n_heads, q_length, k_length)\n",
    "        \n",
    "        A = torch.softmax(scores, dim=-1)  # (bs, n_heads, q_length, k_length)\n",
    "        A = self.dropout(A)\n",
    "        \n",
    "        # Get the weighted average of the values\n",
    "        H = torch.matmul(A, V)     # (bs, n_heads, q_length, dim_per_head)\n",
    "\n",
    "        return H, A \n",
    "\n",
    "        \n",
    "    def split_heads(self, x, batch_size):\n",
    "        \"\"\"\n",
    "        Split the last dimension into (heads X depth)\n",
    "        Return after transpose to put in shape (batch_size X num_heads X seq_length X d_k)\n",
    "        \"\"\"\n",
    "        return x.view(batch_size, -1, self.num_heads, self.d_k).transpose(1, 2)\n",
    "\n",
    "    def group_heads(self, x, batch_size):\n",
    "        \"\"\"\n",
    "        Combine the heads again to get (batch_size X seq_length X (num_heads times d_k))\n",
    "        \"\"\"\n",
    "        return x.transpose(1, 2).contiguous().view(batch_size, -1, self.num_heads * self.d_k)\n",
    "    \n",
    "\n",
    "    def forward(self, X_q, X_k, X_v):\n",
    "        batch_size, seq_length, dim = X_q.size() # dim = embedding dimension\n",
    "\n",
    "        # After transforming, split into num_heads \n",
    "        Q = self.split_heads(self.W_q(X_q), batch_size)  # (bs, n_heads, q_length, dim_per_head)\n",
    "        K = self.split_heads(self.W_k(X_k), batch_size)  # (bs, n_heads, k_length, dim_per_head)\n",
    "        V = self.split_heads(self.W_v(X_v), batch_size)  # (bs, n_heads, v_length, dim_per_head)\n",
    "        \n",
    "        # Calculate the attention weights for each of the heads\n",
    "        H_cat, A = self.scaled_dot_product_attention(Q, K, V)\n",
    "        \n",
    "        # Put all the heads back together by concat\n",
    "        H_cat = self.group_heads(H_cat, batch_size)    # (bs, q_length, dim)\n",
    "        \n",
    "        # Final linear layer  \n",
    "        H = self.W_h(H_cat)  # (bs, q_length, dim)\n",
    "        \n",
    "        return H, A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Self-Attention Sanity Check\n",
    "---\n",
    "* If the query matches with one of the key values, it should have all the attention focused there, with the value returned as the value at that index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attention weights are: tensor([3.7266e-06, 9.9999e-01, 3.7266e-06, 3.7266e-06])\n",
      "Output is: tensor([1.0004e+01, 4.0993e-05, 0.0000e+00])\n"
     ]
    }
   ],
   "source": [
    "def print_out(Q, K, V):\n",
    "    temp_out, temp_attn = temp_mha.scaled_dot_product_attention(Q, K, V)\n",
    "    print('Attention weights are:', temp_attn.squeeze())\n",
    "    print('Output is:', temp_out.squeeze())\n",
    "    \n",
    "\n",
    "temp_mha = MultiHeadAttention(d_model=512, num_heads=8, dropout=0)\n",
    "test_K = torch.tensor(\n",
    "    [[10, 0, 0],\n",
    "     [ 0, 10, 0],\n",
    "     [ 0, 0, 10],\n",
    "     [ 0, 0, 10]]\n",
    ").float()[None, None]\n",
    "\n",
    "test_V = torch.tensor(\n",
    "    [[   1, 0, 0],\n",
    "     [  10, 0, 0],\n",
    "     [ 100, 5, 0],\n",
    "     [1000, 6, 0]]\n",
    ").float()[None, None]\n",
    "\n",
    "test_Q = torch.tensor(\n",
    "    [[0, 10, 0]]\n",
    ").float()[None, None]\n",
    "print_out(test_Q, test_K, test_V)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Great! We can see that it focuses on the second key and returns the second value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/color/96/000000/transformer.png\" style=\"height:50px;display:inline\"> The Transformer\n",
    "---\n",
    "* **Transformer**: attention-based encoder-decoder architecture that aims to combine the advantages from both Feed Forward Networks (FFNs, basically MLPs implementing 1-D convolution) and RNNs.\n",
    "* It achieves **parallelization** by capturing recurrence sequence with attention and at the same time encodes each item’s position in the sequence.\n",
    "    * RNNs are replaced with multi-head attention layers, incorporating the position-wise information through **position encoding**, and applying **layer normalization**. \n",
    "* A compatible model with significantly shorter training time.\n",
    "* 3 main stages: input stage, $n$ times transformer blocks (encoding layers) with different parameters, output stage.\n",
    "* **When do we need encoder-decoder and when can we use just an encoder or just a decoder?**\n",
    "    * Classification (e.g., sentiment analysis) may just use a Tranformer encoder.\n",
    "    * Next-token prediction (e.g., language modeling) may just use a Transformer decoder (autoregressive inference).\n",
    "    * Sequence-to-Sequence (seq2seq, e.g., machine translation) models require both an encoder and decoder modules, and **cross-attention** is used between 2 different sequence (e.g., a sentence in English and a sentence in French).\n",
    "* We will now take a look at each component of the Transformer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><img src=\"./assets/transformer_enc_dec.PNG\" style=\"height:550px\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Transformer's Encoder Module\n",
    "---\n",
    "<center><img src=\"./assets/transformer_encoder.png\" style=\"height:350px\"></center>\n",
    "\n",
    "<a href=\"https://jalammar.github.io/illustrated-transformer/\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* The encoder module accepts a set of inputs $\\{x_i\\}_{i=1}^t$, which are simultaneously fed through the **self-attention** block and bypass it to reach the `Add, Norm` block.\n",
    "    *  In tasks that try to model sequential data, **positional encodings** are added prior to self-attention layer.\n",
    "    * The `Add, Norm` block has two components: the add block is a residual connection, and then layer normalization.\n",
    "* At which point, they are again simultaneously passed through the FFN (just an MLP that is shared between all inputs) and another `Add, Norm` block, and consequently outputted as the set of hidden representation $\\{h_i^{Enc}\\}_{i=1}^t$.\n",
    "    * A position-wise feed forward network (1D-convolution): consists of two dense layers. Depending on what values are set, this block allows you to adjust the dimensions of the output $h^{Enc}$.\n",
    "    * Similar to the multi-head attention, the position-wise feed-forward network (1D convolution) will only change the last dimension size of the input—the feature dimension. \n",
    "    * In addition, if two items in the input sequence are identical, the according outputs will be identical as well.\n",
    "* This set of hidden representation is then either sent through an arbitrary number of encoder modules (i.e. more layers), or to the *decoder*.\n",
    "* **Note**: it is now common to put the layer normalization **before** the self-attention layer for a more stabilized training (usually referred to as *Pre-Norm*)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Feed Forward Network (FFN): an MLP with one hidden layer and ReLU activation applied to each and every element in the set.\n",
    "\"\"\"\n",
    "class FFN(nn.Module):\n",
    "    def __init__(self, d_model, hidden_dim_multiplier=4, resid_pdrop=0.1):\n",
    "        super().__init__()\n",
    "        self.fc_1 = nn.Linear(d_model, hidden_dim_multiplier * d_model)\n",
    "        self.act = nn.ReLU(True)  # inplace=True, saves a little bit of memory\n",
    "        self.proj = nn.Linear(hidden_dim_multiplier * d_model, d_model)\n",
    "        self.dropout = nn.Dropout(resid_pdrop)\n",
    "\n",
    "    def forward(self, x):\n",
    "        # x: [batch_size, seq_len, embed_dim]\n",
    "        x = self.dropout(self.proj(self.act(self.fc_1(x))))  #  [batch_size, seq_len, embed_dim]\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.0314, -0.1459,  0.2119, -0.3628],\n",
       "        [-0.0314, -0.1459,  0.2119, -0.3628],\n",
       "        [-0.0314, -0.1459,  0.2119, -0.3628]], grad_fn=<SelectBackward0>)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ffn = FFN(d_model=4, hidden_dim_multiplier=2)\n",
    "ffn.eval()\n",
    "ffn(torch.ones((2, 3, 4)))[0]  # batch_size = 2, seq_len = 3, embed_dim = 4\n",
    "# note that the FFN only operates on the last dimension\n",
    "# PyTorch's nn.Linear knows how to handle tensors of shape [batch_size, seq_len, embed_dim]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Positional Encoding\n",
    "---\n",
    "* Unlike the recurrent layer, both the multi-head attention layer and the position-wise feed-forward network compute the output of each item in the sequence independently. \n",
    "* This feature enables us to **parallelize the computation**, but it fails to model the sequential information for a given sequence. \n",
    "* To better capture the sequential information, the Transformer model uses the positional encoding to **maintain the positional information of the input sequence**.\n",
    "    * The positional encoding adds positional information. This can be implemented in multiple ways, and the original Transformer uses `sin` and `cos` functions to add that information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* Assume that $X\\in \\mathbb{R}^{l\\times d}$ is the embedding of an example, where $l$ is the sequence length and $d$ is the embedding size. \n",
    "* This positional encoding layer encodes $X$’s position in the matrix $P\\in \\mathbb{R}^{l\\times d}$ and outputs $P+X$.\n",
    "* The position $P$ is a 2-D matrix, where $i$ refers to the order in the sentence, and $j$ refers to the position along the embedding vector dimension. \n",
    "* In this way, each value in the origin sequence is then maintained using the equations below: $$ P_{i, 2j} = \\sin(i/10000^{2j/d}), $$ $$ P_{i, 2j+1} = \\cos(i/10000^{2j/d}) $$ for $i=0,…,l−1$ and $j=0,…,\\lfloor(d−1)/2\\rfloor$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><img src=\"./assets/transformer_position_enc.svg\" style=\"height:200px\"></center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class PositionalEncoding(nn.Module):\n",
    "    def __init__(self, num_hiddens, dropout, max_len=1000):\n",
    "        super(PositionalEncoding, self).__init__()\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        # Create a long enough `P`\n",
    "        self.P = torch.zeros((1, max_len, num_hiddens))\n",
    "        X = torch.arange(0, max_len, dtype=torch.float32).reshape(-1, 1)\n",
    "        X = X / torch.pow(10_000, torch.arange(0, num_hiddens, 2, dtype=torch.float32) / num_hiddens)\n",
    "        self.P[:, :, 0::2] = torch.sin(X)\n",
    "        self.P[:, :, 1::2] = torch.cos(X)\n",
    "        # alternative approach - learned positional embeddings, like in GPT:\n",
    "        # self.position_embeddings = nn.Parameter(0.02 * torch.randn(1, max_len, num_hiddens)))  # draw from N(0, 0.02^2)\n",
    "\n",
    "    def forward(self, X):\n",
    "        X = X + self.P[:, :X.shape[1], :].to(X.device)\n",
    "        # if using learned embeddings:\n",
    "        # X = X + self.positional_embeddings[:, :X.shape[1]]  # [bs, seq_len, embed_dim]\n",
    "        return self.dropout(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAArQAAAFfCAYAAAC/VV8GAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOydd3gU1feH39ma3jsJIaH33ougAoJiBzsWkC/qTxFsICjNXhDEXrEiKioWFBCl9450COm9963z+2OyCZAEUrYz7/PkyWR35t6zmZ2Zc+8953MEURRFZGRkZGRkZGRkZFwUhaMNkJGRkZGRkZGRkWkOskMrIyMjIyMjIyPj0sgOrYyMjIyMjIyMjEsjO7QyMjIyMjIyMjIujezQysjIyMjIyMjIuDSyQysjIyMjIyMjI+PSyA6tjIyMjIyMjIyMS6NytAGOwGw2k56ejq+vL4IgONocGRkZGRkZGRmZCxBFkZKSEqKiolAoLj4He1k6tOnp6cTExDjaDBkZGRkZGRkZmUuQkpJCdHT0Rfe5LB1aX19fQPoH+fn52aVPg8HA2rVrGTVqFGq12i59ylgX+Ry6B/J5dA/k8+geyOfRPbDVeSwuLiYmJqbab7sYl6VDawkz8PPzs6tD6+XlhZ+fn3zRuijyOXQP5PPoHsjn0T2Qz6N7YOvz2JDwUDkpTEZGRkZGRkZGxqWRHVoZGRkZGRkZGRmXRnZoZWRkZGRkZGRkXBrZoZWRkZGRkZGRkXFpZIdWRkZGRkZGRkbGpZEdWhkZGRkZGRkZGZdGdmhlZGRkZGRkZGRcGtmhlZGRkZGRkZGRcWls6tBu2rSJcePGERUVhSAI/PLLL5c8ZuPGjfTu3RsPDw/i4+P54IMPau2zcuVKOnXqhFarpVOnTvz88882sF5GRkZGRkZGRsYVsKlDW1ZWRvfu3XnnnXcatP/Zs2cZO3YsQ4cOZf/+/Tz77LM89thjrFy5snqf7du3c9ttt3HPPfdw8OBB7rnnHiZMmMDOnTtt9TGajyg62gIZGRkZGRkZGbfFpqVvx4wZw5gxYxq8/wcffEDLli1ZvHgxAB07dmTPnj288cYb3HLLLQAsXryYkSNHMmvWLABmzZrFxo0bWbx4McuXL6+zXZ1Oh06nq/67uLgYkEq1GQyGpny0RlH87vMU//Ib7bTepP/6A8qQFii8vFF4e6Hw90cd0xJNq1jUMTEIGo3N7bEmpToje5IK2JGQz9nccgbEB3Fd1whCfbWONs3qWL4r1vrOlBpKSS1JJbVU+smryKPSVEmlsfK832qFGh+1Dz4aH+m32gc/jR/RPtG08m9FpFckSoXSKja5HcZKKM1GKMuBsmwoy0EszqRNVhKGop7gH+loC5uF2SxyLLOEogoDOqOZSoMJndGMzmgmyEvDlR1CUSouXTLSFbH29ShjQ8wmhGOrEFJ3gU8EYkBLCIhFDIjFoJbKz7v6ecwv0/PD3jROZZcSH+JNx0hfOkT4EuGnbVDZVmfEVFiIMTMTY3Y2xqxsjNlZmKq2wxbMRxUWVr2vra7HxrRnU4e2sWzfvp1Ro0ad99ro0aP59NNPMRgMqNVqtm/fzvTp02vtY3GC6+Lll19m/vz5tV5fu3YtXl5eVrH9YnTe+S+GTBNQTHnSXmBvnfuJgoAxIAB9SAi6qEgq4uKoaNUKs6enzW1sDGeK4WiBglPFAimlYKbmYv3nRA4v/3mc9v4ifUNFugaJaN3M11q3bl2jjyk2F5NkTCLZmEyqKZU8cx7lYrlV7FGhIlgRTKgylDBlGK2UrYhRxaAWLt+66ILZSIfMn2md/SdK0Vjr/c6A8b1VnA67hjNhYzAqnesaawiZ5fBdgpKzJfU/LFt4idwcZ6KNnx0NszNNuR5l7IMgGonJ30bbrN/w0WXVvY9CSz/fTmxeXUClOtDOFjaftDLYmKFgb66AUax9LXqrRKK8RLoHiwwJF3FW31ZZWoo2NQ2P1FQ80lLxSE1DVTX5VxebflxJZavYWq9b+3osL2/4c9KpHNrMzEzCw8PPey08PByj0Uhubi6RkZH17pOZmVlvu7NmzWLGjBnVfxcXFxMTE8OoUaPw87P9nV4fF4Vu6y8UHfoX/7J0zDoDZoOA2ShgNHiiV8ZhyMzHXFqKuqAAdUEB3qdOwcZNIAho2rfHs3cvPHv3xmvgQBQ+Pja3uS7MZpFX15zksyNJ570eE+jJgPggYoO8+Pt4NgdSijheJHC8CLw1SsZ1j+TZa9rjqXFtz9ZgMLBu3TpGjhyJWn1xZzG/Mp9NaZvYm7WXAzkHSCtLq3O/AG0A0T7RxPjEEOYVhpfKC61Ki4fSAw+VB1qlFoPZQKm+lFKD9FOiL6FIX0RySTLJxcnozXqyzFlkmbOgajCrUWjoFtqNvmF96RPehy7BXVArLxMHN/cUylVTUWQdBEBUasE7FNE7BLxDMXuGUHp6GwEVSXTI/IX2RZswD5qGufcDoHZ+x9ZgMvPx5kTe2XUGg0nEU60gOtATD7USrUqBViX93pNUQFq5kaVHVIztEs7To9vRIsD5P19Dacz1KGNnjJUoDi5Hsf1thKIUAETPIMydb0HQFUNhEkJhEkJJBiqzjsii/UQYkjHd8AFi/AgHG39pRFFk3bFsvtiezK7EgurXO0f5clX7MBLzyjmWWUxCbjllRjhVLHCqGEq9wnn5ps74aB3veol6PeW7d1O+YSNlmzdjTKv7GaUMCkIVHoYyLAxVWDiqsDBU4WG0GjoUVUhI9X62uh6LL+JUX4jj/6sXcOHUvFgVf3ru63Xtc7Epfa1Wi1ZbewlcrVbb5Uao7tYXTccebF7dj7GjrkKdugNO/AHHV0NpJigLEW/9DFPoAPRJSejPnqXi4EHKd+9Bn5iI/vhx9MePU/TNtwgaDT5XDMP3mmvwHTEChR1mmAEq9CYe/34/a45Io+wbekQxtG0oA1sHn/eQ/L+r2nE2t4yf96fxy/40kvPL+W53Kgm55Xx2X1+nuJCbS33fm9yKXNYnrWdd0jp2Z+3GLJqr3xMQaB/Unp5hPekR2oM4/zhifGPw0TRvcGIym0gvS+ds0VnOFp3lSN4RdmfuJrcilz1Ze9iTtQcOg5/Gj5GxIxkbN5be4b3dM0RBFGHPp7BmDhgrwDMQrluM0OkGEITqdQSTwcDGP37n2ngTqo2vIOSdQrl+LspdH8KohdD1Vod+jItxOLWIp1ce4liGdJMf3j6UF2/qWqejml+m5821J1i+K5nV/2Wx/ngOU69ozdQrWrv84PJc7HUfl2kgKbvh+4lQki797R0Ggx5F6PMASu0F9ztDJcb0g5R9Nxn/8mRUy8fDkBkwYjYonfNZYTaLzPzpEN/vSQVAqRC4pksEDwxuRa+Wgef5IpUGE6ezS9l0Koe31p3kryNZnMkp46OJfYgL8ba77caCAkr/+ZfSDf9SunUb4gWzn5q4ODy6dMGzS2c8unTBo2PHRvsY1r4eG9OWU31jIiIias20Zmdno1KpCA4Ovug+F87aOi0qLbS9WvoZ/RL8OAlO/IHw/T2orl+KqvfdePXuTcCt0kPVkJ1Nxd69lO/eTdnWbeiTkihZ9zcl6/5G8PDAZ/hw/K4di+/w4Qg2uqnnlOiY/OUeDqYUolEqeH18N27o0aLe/eNCvJkxsh3Tr27LxpM5PPrtfnadzWfipztZ9kA//Dzc5+FTbihn9dnV/JHwB3uz9iJSkwDYKbgTQ1oMoVdYL7qFdsNX42v1/pUKJTG+McT4xjAsehggDfDOFp9ld8ZudmXuYnfmbgp0Baw8tZKVp1YS6hnK6FajuTb+WjoHd3bZ+K7zKMmCX/8PTq2V/o4fATe+D371xMgKCsSO10HnG+HgctjwChSnwspJkmPcbbzdTG8IJrPIm2tP8OGmBExmkUAvNXPHdeaGHlH1nr8gbw0v3tSVu/rHMv+3I+w8m8+S9adYcyST76cOdKvrUMZJyDoK39wClUXg1wIGPw697ql/5UPtgRjVi03tnmescivKfZ/DlkWQtA1u/RT8o+1q/qUQRZHnf/2P7/ekohBgyrDW3Dsolkj/uj+fh1pJlxb+dGnhT/+4YB76ei+nsku5/p0tLLm9B1d2sL3fIprNlO/YQcEPP1Dy93o4JyZVFRqKz/Dh+IwYjlffvih9rf+MsidO5dAOHDiQ33777bzX1q5dS58+faq99IEDB7Ju3brz4mjXrl3LoEGD7GqrVVB7woQv4fdpsP9rWPUIlOXCkMdrdgkLQz1mDH5jxiCKIroTJyhe/SfFf/6JISWFkr/+ouSvv1BFRBB4550EjL8VVaD14pBOZZVw/7LdpBZUEOCl5uOJfejbKqhBxwqCwPD2YXw9uT/3fLqTfcmF3P3JTr58oB8BXq6V/HYhKSUpfHf8O34+/TMl+pLq17uGdGVk7EhGxo4k2tcxN2NBEIj3jyfeP57bOtyGyWxib9ZeVp9dzbqkdeRU5PD1sa/5+tjXtA9sz8TOExnTaozrhiQUJMEnV0tJX0otjFwA/aaAogEiLkqV9MDtOh7WzoHdH8MvD4FPGMRfYXvbG8iidSd4b8MZAMZ1j2LuuE6E+DQs8bJTlB/fTRnA6sOZzP31P45nlvDIN/v4/L6+qJSyFLmMlShIgq9vlpzZ6H4w8RfQNGwW0qzQYB7zOsrWV8Cvj0HKDvhgCIxfBvHDbWl1gxFFkRf/OMbXO5IRBFg0oQc39qx/YudCescG8vujQ3j4m33sSSpg0hd7mH51O/5vRBsUNkjcNGRnU/TTzxSuXIkhJaX6dW2HDvhedRU+w4fj0bkTQkPuky6CIIq205QqLS3l9OnTAPTs2ZNFixYxYsQIgoKCaNmyJbNmzSItLY0vv/wSkGS7unTpwv/+9z8efPBBtm/fztSpU1m+fHm1ysG2bdsYNmwYL774IjfccAOrVq1izpw5bNmyhf79+zfIruLiYvz9/SkqKrJLDC1I8SWrV69m7NixtafQRRH+ngtbl0h/D3oURi7kYtHjoihS+d8Riv/8k6JVqzDl5QEgeHjgP24cQRPvQdu2bbNs3nYml/99tZeSSiOtgr34/P5+TV4mOZJexN2f7KSg3EDHSD++ntSP4AY+kJ0FnV7H0l+Xcsb/DFvTt1bPxsb4xjC+3XhGtxpNlE+Ug628OAaTga3pW1mdsJp/U/6l0lQJQKhnKHd0uIPx7cYT4BHgWCMbg6ESPhsNGQcgtIP0AAzrePFD6rsWzWZY+QAc+Rm0fnD/nxDRxabmN4Q/DmXwyLf7AHjtlm5M6BvT5LYOpxYx4cPtVBhMTBwYy4IbHP/5mspF76ky9qU0Bz4bBfkJENoR7l8NXg2b+Kh1HvPPwo/3Q/p+0PrD1E0Q2Mq29jeAN9ac4J1/JX/m1Vu6clvflk1qR280s/D3o3y1Q8pFuXdgLPOteB3qzpwh94MPKV69GkwmABQ+PviNu47A8ePx6NTJan2di62ux8b4azZ1aDds2MCIEbUDvO+9916WLVvGfffdR2JiIhs2bKh+b+PGjUyfPp0jR44QFRXFM888w9SpU887/scff2TOnDkkJCTQunVrXnzxRW6++eYG2+V0Dq2FrW/Duuek7d73wbglDWrbrNdT/Mdq8r/6Et3RY9Wvew8ZQui0aXh2bfzFkpRXxpglmynXm+gTG8hHE/sQ5N28WdUTmSXc9clOckt1tAv34evJ/Qnz9WhWm/ZAFEU2p21m6b6lHC84Xv364BaDubPDnQxpMQSF4Hqj3CJdET+c/IHlx5aTXZENgIfSg+tbX8+D3R4kwjvCwRY2gN+mwd5l4BkE/9sEAZd29i56LRoqpVmmpK3gGwWT1zl02fN4ZjE3vbuNCoOJB4fGMfva5j+M/vovk4e+2YsowrxxnbhvcJwVLLU/skPrJFQWw7JrIfMQ+LeESWvAr+ED+zrPo1EntZm6G6J6wQNrQOW4Vb13/jnFG2tPArDghs5MHNiq2W2u2J3MMysPA/DRPb0Z1bl599vKEyfIff8DStasqda+9+zZk4Dx4/G7ZrTN823c3qF1VpzWoQXY/40UCyiapdmmzjc1uA9RFKnYs4f8L7+iZP16acYJ8B01itBpj6Ft3bpB7ZjMIhM+3M7epAL6xQXx5QP98FBbJ4nkTE4pd368g6xiHR0j/Vj1yGA0Kud1Bndm7GTp/qUczJEy5jVouKXdLdzZ6U5a+bdyrHFWwmAy8FfiX3x19CuO5UsDIo1Cw50d72Ry18n4a/0dbGE9HFgOv0wFBLh7JbS5qkGHXfJarCiAz8ZAzjFptumBv8AzwKqmN4TCcj3Xv7OV5PxyhrQJYdn91gsR+GDjGV758zgKAT69ty8jOoRd+iAnQ3ZonQBDJXxzKyRuBq8QmLQWghv2nKluor7zWJgMHwyFykIY8DBc87J1bW8gn2xO4IU/pPvis2M7MGVY4z7fxXhp9TE+2pRAgJeav6YNI8K/8RM8FUeOkPv++5T+vb76NZ+rryJk6kN4dulsNVsvhTM4tM7rSVyu9LwLhj4pbf/xpBRT20AEQcCrb1+il75N6zV/4X/D9SAIlKxdS8K460l/djaG9PRLtvPRpgT2JhXgo1WxaEJ3qzmzAK1Dffj+fwMJ8tZwLKOYDzeesVrb1uRgzkEmr5nM5LWTOZhzEA+lBxM7TuQJvyd4qs9TbuPMAqiVasa1HseK61bw2ejP6B3eG71Zz7IjyxizcgyfHP6ECmOFo808n6wj8HtVHP3wmQ12ZhuEZyDc9QP4RkpO7Xd3STNGdsRoMvPo8v0k55cTE+TJ0jt6WjXe9X/D4pnQJxqzCI8u38/xzIZL48jIAFUhOpMkZ1bjKw0qG+nMXpSAlnDTB9L2jvfg2O/Wa7uBbDudW+3MPjGynVWdWYAnR7WnSws/CssNzPj+ACZzw+cXDVlZpD31NIm33Co5s4KA75hriFv1CzHvvGNXZ9ZZkB1aZ2TYUxDWGcpzYfWTTWpCExND1KuvEvfLL/hcdRWYzRT99BNnRl9D9htvYK5HrPhoejGL1p0AYO64TkQHWn+ZIjbYm7njpKXTpf+c5nR2ySWOsB+5FbnM2jyLu1ffzc7MnagUKu7ocAerb17N4z0fx1thf6kVeyEIAn0j+vL56M9596p3aRvYlhJDCUv2LeG6n67jp1M/nSdF5jAqi2DFPZI0V5urYdjT1u8jIEZyajW+kLQF/nzG+n1chNfXnGDzqVw81Uo+vLsPgc0M97kQQRB44cauDIgPolRnZNKyPeSU2Ndpl3FxDn0Hx3+XEjHvWA5RPazfR/sxMPD/pO1VD0uJZ3ai0mBi9i//AXBn/5Y8elXzclLqQqNS8PbtPfFUK9l2Jo+PNiVc8hizTkfuBx9w5poxFP/2GwgCfuPGEf/7b0S/9RYe7dtb3U5XQXZonRGVBm58DwSllKBy5JcmN+XRvh0x775Dq++W49WvH6LBQN4nn5Jw3ThK/v33vH11RhMzvj+AwSQyslM4t/a2Xezg9d2jGNE+FL3JzMyVhzE3YmRqC0xmE8uPL+f6n6/n94TfERC4qc1N/HHTHzzb/1lCvUIdap89EQSBYdHD+OG6H3hpyEtEekeSXZHN3G1zue+v+zhT6MBZdVGU1EDyz4B/DNz8ccPUDJpCRFeY8IW0vXcZpO2zTT8X8OvBdD6serC9Pr4bnaJsExalUSn44O7exIV4k1ZYwbM/H7ZJPzJuSEUhrK3K97hyNsQNtV1fV82FFn2kgeyP94NRb7u+zuGDjWc4m1tGmK+WmWM62Kyf+FAf5l8vzaa+ufYEB1IK69xPFEWK160j4drryFm8BLGiAs+ePWn1ww+0eP21BocUujOyQ+usRPWAoVXVzf54olGhB3Xh2aMHLb9YRvR776KKisSQnk7qQw+T+uijGKp0fRetO8nxzBKCvTW8fHNXm+qTCoLACzd1xVujZE9SAV/vtN/I+0IO5xzmjj/u4KWdL1FiKKFzcGeWX7ucBYMXOL1qgS1RKpSMaz2O32/6nSd6P4GnypP92fu59bdbWbp/KTqTA2b0tr8Lx34DhRrGf9HgTOom0+Yq6HYbIMKfT1fHpduK/DI9s3+SHMupV7Tmum62/f4FeGn48J7eKBUC645msfV08+4zMpcJ/74orSCGtIf+D9m2L5UGbv0MPPwhbS/8Pc+2/QEJOaW89680cH9+XCebazaP7xPNtd0iMZpFpn23n1Ld+eW69alppEyaTNqjj2FITUUVHk7U668T++03l2VoQX3IDq0zM+wpCOtUFXrwVLObEwQB3yuvpPXvvxM06QFQKilZ9zcJY6/lwKL3+GSjJEny8s1dG6xx2RxaBHjyTNXI99U/j5NWaN84zXJDOS/seIG7Vt/Fsfxj+Kp9mdN/Dt+M/YbOIfJNwoJGqeG+Lvex6oZVDI8ejtFs5KNDH3HzqpvZmbHTfoYUp8M/L0jb17wM0b3t0+/V80HjI2VcH1ph067eXn+KEp2RzlF+PDXaPkuH7cJ9uWeAVJN94e9HMZqcIKxExnnJOAS7P5G2x75uH/WBwFipUArAjnfh9PqL798MRFHkuVX/oTeZuaJdKNd2rac4ixURBIGXbpQq/iXllTN31ZFqWwq+W8HZ66+nbNs2BI2G4Kn/o/XqP/Afd517FMWxIrJD68yotOeEHvwER1dZpVmFlxfhTz1F3E8/4dmzJ+bycrQfLeWVze9zX7ym2fIhjeHu/rH0jg2kTG9izs+HsZfoxsGcg4z/bTwrTqxARGRc/Dh+velXbutwm3uWhbUCkT6RvH3l2ywavohQz1CSS5KZvHYy87bNo9xQd0y2VdnwihQ3GzMA+k62fX8W/CKlwSXAuuclmSIbkJBTytdV2pSzx3ZEaQOx9fqYdlVb/D3VHM8sYcWelEsfIHN5YjZLeR2iGTrfbN/iIx2uhb4PStvr5tpstWTVgXS2ns5Dq1Kw8IYudnMa/b3ULL69BwoBVu5LZcOmQ6RMmkzmvHmYy8vx7N2b+F9XEfb44yi83TeXoznIDq2zE9UThlRlc/8+A8ryrNa0R/t2xH7zNXtu/R/lKi1d8s5y+0fPUvTrr3ZzLBUKgVdv6YpGqeDfEzn8evDSKgzNwWA28O6Bd7n3z3tJLkkm3Cucj0d9zEtDXyLEM8SmfbsDgiAwMnYkq25cxe3tb0dAYOWplUz4fQJHco/YruOck1I1PYCr51206IhNGPAQBLWWqpFtes0mXbz21wmMZpER7UMZ1Ma+38VAbw3Tr5aSXt5ce5LiSsMljpC5LDm4HFJ2gtobRr9o//5HPCsVPck6LE3yWJmicgMv/HEUgMeuakvLYNtqt15I31ZBPDgkjtGJO/F/5F5pVlarJXzWTGK//AJNq1Z2tcfVkB1aV+CKp2tCD/5+3qpNZ5XoWSi25+ERMzB06IJYWkr608+Q/sSTmIqKrNpXfbQJ8+X/rmwDwPzfjpJfZpug/7NFZ5m4eiIfHPwAk2ji2vhr+emGnxgQOcAm/bkzvhpfZg+YzaejPyXMK4yk4iTuXn03nx7+1DZKCP8sANEE7cdC7EDrt38pVFq45hVpe8f7koNtRfYk5vPXkUwUAswae/FKZ7birgGxtA71Jr9Mzzv/nHaIDTJOTEWBtEIBklReI4onWA2vIKmSJkhxvCbrDrxeXXOc3FI9bcJ8eHBovFXbbgimwkJuW/U2jx/4AQ9DJZXtOhH3888E3XsvglJeObwUskPrCqi0cN1iafvgd1aVLnl/w2n0JjOxXdrS5YdvCXnsUVAqKV69moQbb6Js5y6r9XUxpl7RmvbhvuSX6atlw6zJypMrmfDbBP7L+w9fjS+vD3udV4a+gp/GPoU13JW+EX356fqfGBk7EqNoZPG+xUxZO4WssizrdZKyW0oEExRwlXUHdI2i3Shodw2YjfDXzOpqPM1FFEVeXC1pXd7WN4Z24b5WabexqJUK5lwnyel9vvUsibllDrFDxkn555xEsAE2TgS7GAMekoo45CfUrNpYgb1JBXy7MxmAF2/sYveCPxWHDpFw883oNm3EpFLzSefreGroI6jkWdkGIzu0rkLL/hA/XHqYbnnLKk1mFlWyfJcUL/f41W1RqNWEPvwwrb79BnXLlhgzMki+/35yP/7Y5iEIGpWCeVXSJd/vSSW7uNIq7epMOuZum8u87fOoNFXSP7I/P13/E9fEXWOV9mXAX+vPm1e8yfxB8/FUebIzcye3/HYLm1M3N79xUazJau5+J4Q5ZvaymtEvgVIDZ9bDiT+t0uTqw5nsTy7ES6Nk+tXtrNJmUxnRPowr2oViMNU42TIyZByEPZ9K22NfB6UDK7NpfWFYlT77xlfB0PxkYlEUmfurpDl7a+9o+scHN7vNxvSd/+VXJN51N8b0DNQtWxLx9des7z6S03kVrDqQZjdbXB3ZoXUlLALyB76BouZ/yS2zs/3ighh4zgXs2b078T//hP+NN4LZTM6bi0h7fDqmUtvO2AyID6J3bCB6o5lPt5xtdnvppelM/HMiP536CYWgYFqvaXw08iMivO2X9Ha5IAgCN7e9mRXXraBjUEeKdEU8sv4RPj7UzMHQ6b+lwgZKLYyYZT2Dm0pw6xqh9zWzpNKfzUBvNPPqX8cBeHBoPGF+jS99aW3mXNtRlvGSOZ+/nnVMIlh99HlA0qEuyYBdHze7uY0nc/gvrRgvjZJZNtScvRBTSQlp0x4n66WXwGDAd9Qo4lb+SEiPbkwZJoU8LFl/CoOsPNIgZIfWlWg1GGIHg0kPW5c0q6nMokqW766anb2qba1MToW3N5Evv0TE3OdBraZkzRoSb78N3dnmO5r1IQgC/zdCiqX9ekcSheVNj6Xdlr6N236/jaN5RwnQBvDB1R8wuetkFIL8lbclcf5xfD32a25tdysiIm/vf5snNj7RNBUEs7lmdrb/FPC3XaGPRjH0CaksbkGiVHChGXy1I4nk/HJCfbXVDzBH0/YCGa/GlOOUcUNSdkuDSoUaRi10tDUSKq0UxwuwZZFUdKEZfFBVgv2Ofi0JtoNkJUDlyZOcveVWStauBbWa8GefpcWSxSh9pZCj+wa1IsRHQ1JeOSv3ptrFJldHfrq7Ghb5oH1fQEnT4xQ/2HgGvdFMv1ZBDGxd9/KKIAgE3nEHsV98gSo0FP3pMySOn0DJP//Wub81GN4+lI6RfpTpTXyxrfGxwmbRzMeHPmbquqkU6grpFNyJFdetYGCUAxKJLlM0Sg1zB87luQHPoVKoWJe0jrtW30VKcSPloA7/AFn/gdYfhsywjbFNQetTs+S54z0wm5rUTFGFgaX/nAJgxsh2eGtV1rKw2Zwr4/X7Idsqj8g4Odveln53m+A8g0qAbrdDSDspWW3bO01u5kBKITsS8lEpBCYNibOigfVTunEjSXfciSE5GXVUFK2++ZqgifecN7HkpVHx0HBpguft9afQGZt2n7mckB1aVyN+OET3BWMlbF/apCYyiyr5dpcU/P741bVnZy/Eq1dPWq38Ec9evTCXlpL68MPkvPuuTeJqBUHgkRFSCb/Pt52l7IKKKRdDZ9Ixc9NM3t7/NiIiN7e9mS/HfHlZV/tyJBPaT+Cz0Z8R4hnC6cLT3PbHbWxJ29Kwg426miIKQx63fUWwxtL9TvAMgsIkqZ59E/ho0xkKyw20DfNhvA3LTDeFQG8Nk6se7p9uOWs3GT8ZJyM/QUrIhJpQG2dBqYIRs6Xt7e9CaU6Tmvmwanb2+h5RRAV4Wsu6OhFFkfwvviDloYcxl5Xh1a+f9Gzt1q3O/e/q35JwPy3pRZWs2C3rQ18K2aF1NQShJpZ292dN0qW1zM72bRVY7+zshajDwohd9jmBd90FQO7Sd8h4djaiwfp6lWO6RBIX4k1huaE66/RSFFYWMmXtFP5M/BOVoOL5gc8zf9B8tEr7LB/J1E3PsJ6suG4F3UK7UaIv4ZH1j/D9ie8vfeDeL6AoWVra7z/V9oY2Fo0X9J0kbTdhdqhCb+Kbqu/2E6Pao1I63634zv4t0aoUHEotYm9SgaPNkXEE298FRGgzEsI7Odqa2nS6ASJ7gKEMNr/Z6MMTckr564hU+n3qFa2tbNz5iAYDmfPmk/XyK2A2EzD+Vlp+8jGqwMB6j/FQK/m/KyV96KX/nKZCL8/SXgznu4vKXJq2IyGyu3QR73i3UYdmFZ87O9uuUVVQBI2GiOfmEDFvHigUFP38Myn/m4qptLRRNlwKpULgoaqby8ebE6g0XPwiTilO4e4/72Zf9j581D68P/J9xrcbb1WbZJpOmFcYn4/+nBvb3IhZNLNwx0KW7FtS/6yf2Qw7q8pcDn1Cch6dkb4PSooHqbsgpXHydqsOpFFYbiA60JORncJtZGDzCPbRclPPFgB8ttV2sfMyTkpZHuz/Rtq2aL86G4JQI+W359NGh+F9vDkBUYSrOoTZVC7PVFRE8pQpFK5YAYJA2DPPELFgAYLm0mWDb+sTQ4sAT3JKdNWVBGXqRnZoXRFBqIml3fmRFEPUQN7fUDM7O6iBs7MXEnj7bUS/9y6Cpydl27aRdNfdGLKsqDsK3NizBZH+HmSX6Fi5r/6A+APZB7hr9V0kFScR6R3Jl2O+lAslOCEapYYFgxbwcI+HAfjk8CfM3jIbQ13C6Gf+kZY6tf7Q/Q47W9oIfMOh6wRpe3vDZ2lFUWTZtkQA7h3Yyq4lbhvL/YOlsIO//sskJd8O5Y1lnIc9n0qlpiO7Q9wwR1tTP62vhOh+UrL0vi8bfFh2SSUr90pqQVOH22521pCWRuLtd1C+fQcKLy+i332X4Pvva/BkkkalYNpV0izt+xvPXHKC53JGdmhdlfbXStXD9CWw88MGHdKc2dkL8R0+nNgvv0QZEoLuxAkSb7udyhPWq56kUSmqs74/2HgGYx2yJeuS1jF57WQKdAV0DOrIN2O/oW1gW6vZIGNdBEHgoe4PsWDQApSCkt8SfuPh9Q9Tqr9ghn93lQxPz7ukBCxnZuAj0u9jv0mqBw1g59l8jmeW4KlWMqFPjO1sswLtI3wZ2jYEswhfbk90tDky9sJQUfNcGfSY/UtNNwZBgH4PStt7PwdTw/IuPt+aiN5kplfLAPrE1r/s3xx0p0+TeOdd6M+eRRUZSezyb/G9ckSj27m5VwtaBHiSX6bnj0MZNrDUPZAdWldFoTg/07qy+JKHfLMzGb3RTO/Yps/Onotn1y60+u47NPHxGDMzSbrrLqtWFru9b0uCvDWk5Ffw+wUX8Q8nf+CJDU+gM+m4IvoKll2zjFCvUKv1LWM7bmp7E+9c9Q6eKk92ZOzg3r/uraksln8WTq6RtvtOdpyRDSW8kzRDJJphxwcNOmTZ1kQAburVAn8vBwrUN5AHqmZpv9udQmkjkjRlXJiD30lVwfxjoNONjrbm0nS6AbyCoTgNTl664ElJpaF6+X7qFa2bNblTHxUHD5J0190Ys7LQtGlNq++W49G+fZPaUikV3Nm/JQDf7JTDDupDdmhdmU43QnBbSYPvwLcX3dVoMvN9VZbkfYNaWe0C1kS3oNXyb/Hq0wdzaSkpU6ZQunGjVdr21CirZVTe23Aac5Ue5hdHvmDB9gWIiExoN4ElI5bgpXbSOEuZOhnSYgjLrllGsEcwJwtOMvHPiaSUpFRVIxKh9VVSEQNXwJL9vf8rqCi86K6pBeWsPSolodw3qJVt7bISV7QLJT7Um5JKIz/ukTOt3R6zuSaEZsDDkpqAs6PSQq+J0vbuTy65+7c7kympNNImzIerO1o/hr10y1aS7n8AU1ERnt270+rrr1GHN6+f8X2iUSkE9iUXcjT90hNYlyOyQ+vKKJTQb4q0ve+Li9aW33Aih8ziSoK8NYzqbN0LWOnvT8ynn+AzYgSiTkfK/z1K8Zq1Vmn77gGx+GpVnMwqZcPJbN498C5v7HkDgAe6PMCcAXNQKpRW6UvGvnQK7sQ3135DrF8s6WXp3PfnvZw9WJWEYvleuwKtr6wK/ymVrsOL8PWOZMwiDG4TbNMkFGuiUAjVsbSfb0usHljKuCkn/4S801IMe697HG1Nw+l9PyBAwgbIPVXvbjqjqTrJccqweBRWjmEv/vNPUh56CLG8HO/Bg2n5+WcoAwKa3W6Yrweju0hVLuVZ2rqRHVpXp9t4UHlA9lFI21vvbpbY2Vt7R6NVWd8BVGi1RL+9BL+xY8BgIG36dIpWrWp2u/6eam7tEw2IvLrzdT44KC3rTus1jem9p9tkqUjGfrTwacGya5bRJqAN2RU53BfsycmglpKSh6sgCDWxtDs/hLoS3ZCkur7bLV2H9w5sZSfjrMMtvVrg56EiKa+cf45nO9ocGVuyrUrfvO8DoHWNQRcAgbHQ7hppe89n9e626kA6WcU6wv203NDDuhrlBSu+J23GE2Aw4Dd2DDHvv4fCy3qrh3dVhR38sj9NDv+pA9mhdXU8A6X4Iah3diitsIINJ6SH0O19bZeEIqjVRL3+Ov433wxmM+nPzKTgu++a3e6tfaLQRvxEOlJs5ax+s5jc1QXiK2UaRIhnCJ+N+pQOZiX5SiUPBGo5WnDC0WY1jq7jwTtMiuE78nOdu5wr1XWVDZY5bYmXRsUdVQ/TT7fIEl5uS8puSN4ulbnt9z9HW9N4LHH3+78BfVmdu1j0n+8fHGfVyZ2C5cvJnDsXRJGA228j6vXXGyTL1RgGxgcTH+pNmd7EL/vTrNq2OyA7tO5Ar3ul34dXgq6k1tvf707BLFouBttmjQtKJZEvLKwuwJA5bz55n33e5PZMZhNfnnoFTeBuRFHg6pDHuLPjndYyV8ZJCMw9zScpiXTTGSgy65i8ZjIHsg842qyGo9LWhElsW1or/MeVpLrqw2L39oQ8OYbPXdnzqfS72wTwi3SsLU2h9ZUQ2Ap0RXD4x1pvn8oq4WBKISqFwK1WrM6X/+23ZM5fAEDQ/fcTMXcugtL6K6GCIHBX/1hAcszlCn7nYxeH9r333iMuLg4PDw969+7N5s2b6933vvskfbYLfzp37ly9z7Jly+rcp7Ky0h4fx/mIHQTBbaRCC//9dN5bRpOZ76sSOSwzLLZGUCgInzOb4Ael0XL2a681yak1mU08t/U5Vp9djQIllWl38t+JdvJF7I7s+gh/s8hHYVfQO7w3JYYSpqybwp7MPY62rOH0nSSF/2QegvT9573lSlJd9REV4MmYqhi+z+VCC+6HrgSOVoWJ9b7PoaY0GYUC+lRV8Nv9ca2B5Q97JU3zER3CCPGxThXJ/G+/JWvBQgCCHniAsKefsmko3C29WqBVKTiWUcy+5EKb9eOK2NyhXbFiBY8//jizZ89m//79DB06lDFjxpCcXHdJ0yVLlpCRkVH9k5KSQlBQEOPHn1/5yc/P77z9MjIy8PDwsPXHcU4EoSbD84Kwg40nc8goqiTQS81oKyeDXdwkgdAZMwj5PykDPPu118j/+psGH28WzczfPp/fEn5DKSh5YfCrqCq7cyq7lP0phTayWsYhlGbDkV8A8O73EO9f/T4DIwdSYazgkfWPcDDnoGPtayheQdDhOmn74PLz3nI1qa76eKBKdWTVgXSKyq1f9lrGgRz9FQzl0uRIdF9HW9N0et5dNbA8DKk1A2KDycxP+6Rl+vFWmp3N/+abGmd20gOEPfWkzfM6Arw0XNdNiv2Vk8POx+Z6HIsWLWLSpElMnizN1i1evJg1a9bw/vvv8/LLL9fa39/fH39//+q/f/nlFwoKCrj//vvP208QBCIiIhpkg06nQ6fTVf9dXCwtlxkMBgwG+9yULf3YrL9Ot6JavwAhbS+G1AMQLs1of1OltXdzzygUohmDoXaBAlviP+VBTLpKCj7+hKwXXsCsUOA//taLHiOKIi/tfomfT/+MQlDw4qAXGRV7Jf90PszPBzL4bmcSXSPtL7hv83N4maLY/RlKswFzi76YQjujEuHNoW8ybeM0dmftZuq6qXx41Yd0DOpolf5seR6FLuNR/fcj4uEfMV45D5Qa0gsrqqW67u4b7dLfn66RPrQP9+FEVimrDqRwhw1j8i+FfD1aF+X+r1EApq63YTbaL+HI6udR7Yuy000oDi3HvOsjTBE9AFh/PJvcUh3B3hqGtA5sdn+Fy78j96WXAAi4/z4Cp03DaKf/2+19oli5L5XfD2Uwc3RbAr2sG6vbFGx1PTamPUG04fqtXq/Hy8uLH374gZtuuqn69WnTpnHgwAE2NkCvdNy4ceh0OtaurZGBWrZsGZMnT6ZFixaYTCZ69OjBwoUL6dmzZ51tzJs3j/nz59d6/dtvv8XLihmIjqbv2aVEFe4mIXQkh6PvoUAH8/cpERGY3cNImKeDDBNFQv5YTdDmzYiCQNb4Wynu3bueXUX+qPiDHfodCAjc4nULPTQ9ADhdDEuPqNAqRBb2MaGV1bpcHkE0MvLIE3gaCtgTO5W0oEHV7+lFPV+UfkGSKQkvwYtJPpMIVzp3MpUgmhj133Q8jIXsjJtGZkBv1qQKrE5R0sbPzKOd7TugtAX/pAusSlIS5yvyeBe5DKc74KXLYeTRJxARWNt5EZWa5hfecSQBZQlccXIeJkHF2i5L0Kt8+fSEgkP5CoZHmrmpVfOuQ/8dOwn/WUr+zL9iGLljxti1mpoowuuHlKSVC9wYa2JElPuG4ZWXl3PnnXdSVFSEn5/fRfe16Qxtbm4uJpOJ8AsEhcPDw8nMzLzk8RkZGfz55598++35RQM6dOjAsmXL6Nq1K8XFxSxZsoTBgwdz8OBB2ratXfp01qxZzJgxo/rv4uJiYmJiGDVq1CX/QdbCYDCwbt06Ro4ciVptmyVH4YwWvruNuJJdxIz8jKWb0xE5Q/+4QO67xbFLSOLYseS+/ApFy5cT8eNKuvfpg++YMefvI4os2reIHSckZ3begHmMix933vu/LdlKYl45phbdGdurhV0/gz3O4eWGcPIvVAcKEL1D6X77c3RXnR/XdrXhah7+52H+y/uPbwzf8MkVn9DKr1Wz+rT1eVR47IEd79JXcxrjmDkseXsrUM6DV3dlbE/7fmdtQe/iSn57YxNnSwS6DBhOyyDHTArI16P1UGx6DQAxbhhX3mhf7VlbnUfzZ7+gzDjAqNAscrrdxBM7NwIiT90yuFka0CV//knWL78AEHDfvbSeMcMh8pHFYSk8/+sxDpT68tqYwQ6XsLTVebSsqDcEu5QAufAfLYpig/75y5YtIyAggBtvvPG81wcMGMCAAQOq/x48eDC9evVi6dKlvP3227Xa0Wq1aLW1A8DVarXdb4Q27bPdSPCPQShKQXnqT37YFwLAnf1jneKGH/ncHASjkcIffiBr1rOoPDzwGzWq+v2l+5fyzQkpznbuwLnc3O7mWm1M6BvDa3+dYOW+dO7o38pepp+HI743bstRKYlR6HYbas/aYSSB6kA+GPkBk9dO5nj+cab+M5Vl1ywjxrf5S902O48974Yd76I4tZbTyekk5JajVSkY262FW3xvooPVDG4TwuZTufx+OJtpV9eeRLAn8vXYTMxmOLwCAEXPu1E46H9p9fPYbwqsehjlga/5Q7gJo1mkW7Q/naODmtxk6caNZD07W5LmuuN2Ip55xmGO5M29W/LamlMk5pWzO7mYwW1CHGLHhVj7PDamLZsmhYWEhKBUKmvNxmZnZ9eatb0QURT57LPPuOeee9BcQstNoVDQt29fTp2qvzrIZYFCKT1MgaKtn56TDNawWGNbIygURMyfh/+NN4LJRPoTT1K2YwcA3xz7ho8OfQTA7P6zuaXdLXW2cWuvaJQKgT1JBZzOLrWX6TK2QFcCJ6rqrnetP67aX+vPhyM/pLV/a7LLs5m8ZjJZZVl2MrIJhHeCiG5gNpC48UsARnYKx9fDfZyum6pmmn/enyqrjrg6yduhMAk0vjVJje5ApxtA7QX5CRzc+Q/QvGSw8j17SH1sGhiN+F17LRHPPefQWVEfrYobe0rJYd/urDvJ/nLDpg6tRqOhd+/erFu37rzX161bx6BBg+o5SmLjxo2cPn2aSZMmXbIfURQ5cOAAkZEuqJtnbXreDQgEZu+klZDBLb2i8VA7T7CpoFAQ+eIL+I4ejWgwkPrI/7Fu7Ye8susVAP6vx/9xe4fb6z0+zM+DEe1DAfhBrivv2hxfDcYKKas6ssdFdw3yCOLjUR9Xl8md+vdUinRF9rGzKfSQtJJjUyUZpBt7uH6owbmM7hyBp1pJYl65LB3k6hyoCunrfCNo3CenBK0PtJfC2roV/o1GpeD67k27DiuOHCFl6kOIOh0+V1xB1CsvIygcL+N/e19JivPvY1mUVMrJkTY/IzNmzOCTTz7hs88+49ixY0yfPp3k5GSmTp0KSPGtEydOrHXcp59+Sv/+/enSpUut9+bPn8+aNWtISEjgwIEDTJo0iQMHDlS3eVnjH40u7koAblNu4PZ+9tGebQyCUknU66/h1b8/5rIyfGcuJjxf5M4OdzKl25RLHj++Ssdz5b5UDCbXT7K5bDn8vfS764QGJVSEeoXy4cgPCfUM5XThaR775zEqjU6qPd3lVsyCis7iaXp5ZjGsXaijLbIq3loV11Rp0v68P9XB1sg0GX0ZHP1F2u5xl0NNsQldpJWfccrtjO4Y0iTJPF3CWVIenIK5tBSvPn1osWQxgpOEuHSO8iM+xBud0czfx5x41cpO2Nyhve2221i8eDELFiygR48ebNq0idWrVxMbK1W7yMjIqKVJW1RUxMqVK+udnS0sLGTKlCl07NiRUaNGkZaWxqZNm+jXr5+tP45LsM3/WgBuV2+hTbBzavMqNBqK5k8lKVyBfzm8uFLLE3EPNGgJ58oqUezcUr1cV95VKc2BM/9K2xcJN7iQFj4teP/q9/FV+7Ivex8zN8/EZHbCTHufUI56S/ejaaH70KgcP5tjbW6uSsr8/VAGeqM8sHRJjv0G+lIIjIOWAy69v4uhixtBEd6EC4VMbpnR6OMNmZkkT5qEKT8fj06diP7gfRROpHcvCALXdZfCDn472PjP527Y5S778MMPk5iYiE6nY+/evQwbNqz6vWXLlrFhw4bz9vf396e8vJwHH3ywzvbeeustkpKS0Ol0ZGdns2bNGgYOHGjLj+BSfJTRjjzRl0CxABLrr8rmSBIKE3h4x5O8MEGgMMQDv9xyUqdMxVRSu3TvhaiVCm6peph+v1sOO3BJjv4CogmiekFw60Yd2j6oPUuuXIJGoWF98npe3Pmi08VxluuNfFrcH4CBpX+DMzrdzWRQ6xDC/bQUlhv494Q8sHRJLOEGPe60q+yUvfj7RCGrjdLAsmvB34061lRSQsqU/2HMyEATF0fMJx+j9LG//vmluL67FGq56WQOheV6B1vjWNxv2uAyJ7Ookh3JxawxVcl0HfnZsQbVQVZZFlPWTaFIV0TLVt3o9OV3KIOD0R0/Tuoj/4f5nCIY9TGhStB9w8kc8ssu74vYJTlUFW7QbUKTDu8b0ZdXhr2CgMAPJ3/gg4MfWNG45rPuaBZ/6HtQgjea8gynHVg2B6VC4Iaq2OCfqyowybgQhclwdpO03e02x9piI37Ym8KvZilfR3F0FRgb9qwQ9XrSpk1Dd/IkytAQWn7yMaqgpqsj2JI2Yb50jPTDaBb5879Ly6G6M7JD62b8cTgDUYRTIVdLLxz7DUzOEyxeZijj//75P7LKs4jzj+Pdq97FP749LT/+CIW3N+W7dpE+cyai+eJLmK1Dfegc5YfJLLL2yOV9Ebsc+WchdRcICuh806X3r4eRsSOZ3X82AO8dfI/vT3xvLQubzaoD6ehRcyZ8tPTCgeUXP8BFsagdrD+eddnPDrkcB1cAIrQaCoGxjrbG6mQVV7LpZA47zR0xeoVDZSGcWX/J40RRJOO55ynbth3By4uYDz5A3cK5kzrHVc3S/nYw3cGWOBbZoXUzfj8kfaFb9R4FXiFQkV8zCncwRrORpzY+xfH84wR5BPHeVe8R6BEIIMUnvfsOqNWU/PkXOUtq6wlfyNiu0kX8x2E5dsil+G+l9DtuGPg2T1Lutg638b9u/wPgpZ0vsSVtS3OtazZ5pTo2nswBIHjwfdKLx34FnfvJzHWM9KNDhC8Gk8jvh+Tr0GUQRTh4TriBG/LHoQzMIvSMDUbVrSpO//APlzwud+k7FK1aBUol0YvfwrNzZxtb2nzGdZPiaLcn5JFd7KSJsnZAdmjdiNSCcvYnFyIIMKZ7DHS6XnrDCcIORFHklV2vsDltM1qllqVXLiXa93xNQO8BA4hcsACAvA8/pHDlTxdt0+LQbjuTR4EcduAaiGLNQ6Vr08INLuSRHo9wQ+sbMIkmntz4JCcLTlql3abyx+EMTGaRri38iek6DIJag6FccmrdkFt6Sdfxz/vlsAOXIXUP5CeA2hs6Xu9oa2zCn/9JA6zrukVC1ypd8xN/XnRgWfjjj+S+9x4AEXOfx+ecfB9nJibIi54tAxBFWH0ZT/DIDq0b8UfVDEn/uCDC/DxqlnOdIOzgq6NfseLECgQEXhn6Ct1Cu9W5X8BNNxI8VZpxy5g7l7IdO+ttMy7Em46RVWEHR+WwA5cg6z/IOQ5KLXS0joi7IAjMHTiXvhF9pZCW9f9HbkWuVdpuChbH7saeLaREmx53SG8c/M5hNtmSG3pEoRBgb1IBSXlljjZHpiEck/SRaT9G0mt1M7KLK9mTVAAgyctF9ZKUHAzlNcVcLqB08xYy5s4DIHjq/wicYJ0Bt72wzNL+ehmHHcgOrRthWfK7ruqLTexg8A6VYocSNjrMrvVJ63ljzxsAPNHnCa6Ovfqi+4c+9hh+Y8eA0UjqY4+hS0iod99ru0pL1n8clh1al8AyO9tuNHj4W61ZtVLNW8PfopVfKzLKMnjsn8eoMFZYrf2GkpRXxv7kQhRCTVybRQuTxC1Qnm93m2xNmJ9HddlNeZbWBRBFOFq1WtDJPWdn1xzJRBShZ8sAIv09pYFl1/HSm//9WGt/3alTpD3+OJhM+F0/jtBp0+xrsBW4tlskggD7kgtJyS93tDkOQXZo3YTE3DIOpxWhEKgWPEehlMr/gcPCDg7nHGbm5pmIiExoN4GJnWoX0bgQQaEg8uWX8ezZE3NxsSSdkl+3I1AddnA6V05KcXbMZjhcFT/bRHWDi+Gv9ZeSDLX+HM49zOwtszGL9tVH/WW/NDsyuE0IYb5VepVBcRDRVZIpq2d2yNWxJIf9IcfROj+Zh6RStypPaHPxyQVXZXXVBMfYLudUD7XoXZ/++7yBpbGggJSHHsZcVoZXnz5EvfCCQ0vaNpVwPw8GxAUDl29eiezQugmWL/Cg1iGE+Ghr3uh0o/T7+G8NliyxFlllWTz272NUmioZ0mIIs/rPavCNQqHVEv3uO6hjYjCkppL68CN1ynnFh/rQIcIXo1lk7RG5UopTk7wdilNB6w9tRtqki5Z+LVkyYgkqhYp1SetYun+pTfqpC1EUWXVAmqG0OHjVWOIU3TSO9qqO4agUAqeySzmT437Jb26FZXa2zVWg8XasLTYgr1THzrN5wDmTOwCh7aWBpdkIR6WQC1GvJ+2xaRhSU1HHxNBi6dsIGo0jzLYK46qKLPx64PIMO5AdWjfBItdxXbfI89+IHQTeYVBZBGftF3ZQaaxk2r/TyK3IpU1AG9644g1UClWj2lAFBRHz4Qco/PyoOHCAzLnz6hTQv1ZWO3ANLKVuO40Dte2q7fQO782CQVJy4SeHP+GX07/YrK9zOZFVQkJuGRqVglGdL1Bv6DhO+n3mH9BduniIq+HvqWZQVdjBGllGz7k59pv027J652asPZqFWURKygzyOv9NS/jP4R8RRZHMhS9Qvns3Cm9vYt57F1VgoP0NtiJjukSgUggczSjmdPblN7CUHVo34HR2KcczS1AphPNHpOCQsANRFJm7bS5H8o4QoA1g6ZVL8VY3bSZAGx9P9JLFoFRS9MsvFHz5Za19xlY58VvlsAPnxVQzK1Idy2ZDxrUex5RuUwBYsH0BB3MO2rxPywrBsLYh+GgvGLyFdoDgNmDSw6m1NrfFEYzuHA7AGnmlxHnJOQG5J0ChluLY3RBLlv+YrnVIAnapUjtI2krBJ+9S+MMPIAhEvfkG2rZt7WilbQj01jC0rTSwtEh4Xk7IDq0bYPniDmkbQoBXHcsl1WoHv9sl7ODzI5+z+uxqlIKSN694s5Y8V2PxHjiQ8GeeBiDr1dco27btvPdbnxN2sO6o/DB1SpK3QUUBeAVLQu524JEej3BlzJUYzAam/zud7HLblme1zEyO6lTHg1QQamZpLTNkbsbITuEIAhxMKSSjyP4JeTINwBJuED/cqkmZzkJBmZ5tZ6RwgzFdImvvEBADLQdSmqEha5EkzxX21FP4Dh9uRyttiyXs4LeD6U5XEtzWyA6tG/DHheoGF9JyAPhEgK4IEv61qS2bUjexeO9iAJ7p9wz9IvtZpd3Ae+7B/6abwGwmdfoM9ElJ571vSQ67nDX4nJrjq6Xf7cZIqwZ2QCEoeGnoS7QJaENORQ7TN0xHb7LNgC61oJwj6cUoBLiqY1jdO1kc2pNrweB+4udhvh70aikt2crx7E6KRa7LTdUN1h3LwmQW6RjpR1xI3auCuoBhpG0LBFHE/+abCbr/PvsaaWNGdgpHq1JwJqeMoxnFjjbHrsgOrYtzIrOEU9mlaJQKRlUt+dXCTmEHCUUJPLPpGUREbml7C7e3v91qbQuCQMS8uXh074a5qIiURx7BVFqjeTm2anlpy+lciiqcp9SvDJJM0PE/pO0OY+3atbfamyUjluCr8eVQziEW7lhok1kLy8pAn1ZBBJ+blHkuUb3ALxoMZTYfWDqKa6pih+U4Wick/yxkHgZBCe2vdbQ1NuGv/6Tv3ZgLQ++qMJWWkvrhv5gNCjxD9UQ89X8uqWhwMXw91IxoLw2q/7zM5Cxlh9bFsYQbDGsXip+Huv4dLWEHx/8AY221gOZSpCti2j/TKDWU0iusF7P7z7b6jUKh1RK9dCmqsDD0p8+Q/swziGZJlqlNmC/twn0wmOSwA6cj6z8oSpZkguJH2L37ln4teWPYGygEBb+c/oUVJ1dYvQ/LjOSoTvUMKuH8sIOj7ql2MLrKod15Nl+u3udsWEJdWg0G72DH2mIDiisNbD4llZweW0f8rGg2kz5zJvqkVFQ+CqIH56NI/MfeZtqF0V2k+9Dfxy6vZ6Hs0Lo4f1aNSGupG1xITH/wjQRdsZRpbUXMoplZm2eRWJxIhHcEi4YvQq28iHPdDNRhYUS/sxRBo6F0/Xpy33mn+j057MBJsczOtr4SNF4X39dGDGoxiBm9ZwDw5r43STDUX6yjsRSU6dmVKOlajr5Q3eBCLA7tidUOr95nC1oGe9EhwheTWbzsHqZOj0Uyzk1L3a4/loXBJNI2zIc2Yb613s/76CNK/16PoFYT/choVB5m6Tp0Q0a0D0OpEDieWXJZFVmQHVoXJimvjNPZpagUAiM61BO3Z0GhqNGktXLYwYeHPmRz2ma0Si1LRiwh2NO2o3/Pbt2IWDAfgNz33qfkH2n51iLftflUjhx24ExUhxs4dplzYqeJXBd/HSbRxHfl35FeZp0s4PXHs6vj9mrJBF1IywHgFSJV70vcYpX+nY3R1WEHskPrNBSlQepuabuDdUpOOxuWYgpjutae3CndtImcJW8DEP78c3iOrirwc+YfMLhfAmOAl4a+raR49stpYCk7tC7M38ekrO1+cUH4ezZgRtSSCHByjSSjZAU2p27m/QPvA/DcgOfoFNzJKu1eioAbbyTw7rsBSH/mGfTJybQN96VtmBR28LccduAcFCZLlYkEBbS7xqGmCILA3IFz6RjUkXKxnKc3P22VJLG11eoGFwk3sKBQ1jj2bqp2YJEO3Hwqh3K9de4zMs3k+O/S75j+4HeJ1TwXpFRnZONJKdzgwvhZfXIyaU8+BaJIwIQJBI4fD5Hdq+LZyx1aFt6WXN3x8gs7kB1aF8bitF3VsQEPUoDofuAZKM0Opexsdv+pJanVZW3HtxvPDW3sK9Qd/vRTePbogbmkhNTHpmGuqKgOO7CEYsg4GEup15YDnSJuz0PlwetDX8dT8ORo/lFe3fVqs9qr0JvYVBW3V29S5oVYlnyP/y6VA3YzOkT40jLIC53RzMYTOY42RwZqBk9uGm7w7/Fs9EYzcSHedIioCTcwl5eT+uhjmIuL8ejejfA5s6U3BAHaj5G2T/zhAIttz8iqAfbOhPzLZsVSdmhdlKIKA7ur4vaurk8m6EKUKmg7Sto++Vez+q80VjJjwwyK9cV0Ce7CzH4zm9VeUxA0GlosWYwyOBjd8eNkzl/AqE7S/2Lr6VwqDSa72yRzAZaZofb2VTe4GFHeUYz3Go+AwPcnv+fXM01P0Np0KodKg5noQE86Rfo17KC4YVL539IsSN3V5L6dFUEQqoss/CWrHTieslxI2iptW2K43Yw//6sqptAlojoZWRRFMp57Ht2JEyiDg4lesgTFuWVtLYorJ/5yy4FlbLA3bcN8MJpFNpywrQa3syA7tC7KxpM5GM1SAHxscCOqcFmqwzTDoRVFkRd3vsix/GMEagNZNHwRGqVj6l+rw8Np8eaboFBQ9MsvRG7+iwg/DyoMJnYk5DnEJpkqKgogsepBame5rkvRTt2OKV2lSmILty/kRP6JJrVTo24Q0XBVD5UG2leFX7hp2IEljvafqpkzGQdy/HcQzdIye2Cso62xOpUGE/8et4Qb1IRTFHzzLcV//AEqFdGL30IdcUHCZuwQ0PpBWTak7bGnyXbj6k6WsAPZoZVxYhodbmChzdWgUEHuScg706S+V55ayS+nf0EhKHjtiteI9HFsTJb3gP6EzZgOQPaLLzHeRxKT/uf45XEROy0n14JogrBOEBTvaGtq8WCXBxncYjCVpprVhsZgNJlZf7zKoW1ouIGF6qphv0o6vW5Gr5aBhPpqKak0sl0eWDqWo+6tbrAjIY8Kg4lIfw+6tJBWSSoOHybrVSmcKOzJJ/Dq27f2gSoNtB0pbR9377CDDScuj4Gl7NC6IAaTuXoJocHhBhY8/CF2kLR9ck2j+z6Sd4SXdr4EwKM9H2VA5IBGt2ELgiZNwnfk1YgGA6O+X4yfroz1x7Ivu9J/TsUJ51A3qA+FoOCVIa8Q6R1Jckkyc7bMadT3ZVdiPoXlBgK91PSJDWxc562vknR5LUlzboZCIVQ/TP+S49kdh64Ezm6Stt3UobVMXAxvH4YgCJiKikib9jgYDPiOHEnQvffWf7AlFMpN5bt6RAcQ4qOhpNLIrrP5jjbH5sgOrQuyJ7GA4kojQd4aerZs5IMUarLNT/7ZqMOK9cU8seEJDGYDI2JGMKnLpMb3bSMEQSDypZfQxMaiys3mmX3fkl5QxsmsUkebdnliqIRTf0vbThQ/eyEBHgGSbrJCzb8p//LZf581+FhLuMHVHcNRKRt5K9V4QdurpW03nR2yhB2sOyqVI5VxAAkbwGyAwDgIaetoa6yOKIrVDu1VHcKqiifMwpCejjomhsiXXrx4KFDbkaBQSyuWuaftZLX9UCgErupw+agdyA6tC2L5YlrEkxuNxaFN2gaVRQ06RBRF5m2bR1ppGi18WrBw8EKnKxmo9PWlxdK3ETw86JV1gltPbaheEpaxM2c3SSVefaMgqqejrbkoXUJqkhqX7l/Kvqx9lzxGFGsq0o26VDGF+mhXlWV9am3TjndyBsYH4+uhIrdUx/7kAkebc3li+W61Gy1l9rsZp7NLSS2oQKNSMKhNMPmff07pv/9KCcOL30LpW7vAwnl4+EOrIdK2m6odWOJo1x3NcvsVS7s4tO+99x5xcXF4eHjQu3dvNm/eXO++GzZsQBCEWj/Hjx8/b7+VK1fSqVMntFotnTp14uefrVsswFkRRZH1xywzQ40MN7AQ3BpC2oHZCKfXN+iQ5ceXsy5pHSqFiteHvY6/1r9pfdsYj3btiKiSZrn32F+c/nurgy26TLGoG3QY6xIP0vHtxnNt/LWYRBNPbXqKgsqLO2BH0otJK6zAU61kaNuQpnXapmqGNn0/lLjfwEujUnBlVcGX9XI8u/0RRTi1Ttq2qNu4GZbZ2YHxwXD4INmL3gIg/Nln8ezcuWGNWEKijrtn2MGQNiF4qBWkFVZwPLPE0ebYFJs7tCtWrODxxx9n9uzZ7N+/n6FDhzJmzBiSk5MvetyJEyfIyMio/mnbtma5ZPv27dx2223cc889HDx4kHvuuYcJEyawc2fztVWdnTM5ZSTmlaNRKhjaLrTpDVWHHVxa7eBI7hHe2PMGAE/0foKuoV2b3q8d8L/lFtTXjEUpmrnp9/fJTXU/Z8GpMZtr9GedNH72QgRB4LkBz9HKrxXZ5dnM2XrxeFpLMYUr2oXioVY2rVPfcIjsIW2f/rtpbTg5Fof2X9mhtT+Zh6AkA9ReEDvY0dbYBMtAaVSUhrTpM8Bkwm/cOAJum9DwRix6tCk7odT9dJM9NUqGtJF8BXcvOKSydQeLFi1i0qRJTJ48GYDFixezZs0a3n//fV5++eV6jwsLCyMgIKDO9xYvXszIkSOZNWsWALNmzWLjxo0sXryY5cuX19pfp9Oh0+mq/y4ulrKZDQYDBoN9BIct/TS3vzX/SeU6+8cFolWITW5PaH01qm1vI55ai1FXISkf1EGJvqQmbjZ6BBPaTLDb/6w5xMx/nu079hJemEXCE0/h9+XHCIrmjd+sdQ7dHSF1N6qybEStL8YW/cHJ/l/1nUcNGl4Z/AoT10xkU+omPj/8Ofd0vKfONizhBle2D2nW90HR+iqUGQcwn1yDqUsjHsIuwoBWAQgCHM8sITm3hEh/D6u1LV+PF0dx/C+UgLnVMEwone46tNDU81hUYWBvUgGCaKb3N4sxZmejjosjZM5sjMZGVKjzCkcV3hUh6zDGY38g9rirUXa4Ale2D+bvY1msPZrJ1GGtbNKHra7HxrRnU4dWr9ezd+9eZs48X3R/1KhRbNu27aLH9uzZk8rKSjp16sScOXMYMWJE9Xvbt29n+vTp5+0/evRoFi9eXGdbL7/8MvPnz6/1+tq1a/HyukTtdSuzbt26Zh2/8j8lIBBuymb16qYvkQiiiWuU3mgqCtjx41LyfdrX2kcURb4r/440QxoBigAGFQ/izz8bl0jmSHaOvpMJPyzF9+BudsycRcHwK6zSbnPPobvTKW0FbYE0r87sXeO8M4/1ncdrtNfwa8WvLNm/hPJT5cSoYs57v0gPxzKlW6c++QCrMw402YbAMh+GAaaT6/jzj18RBZvPMdidWG8liaUC7/70L4PCrR/DJ1+PdTP05PcEAYcqI0lqxrPCXjT2PO7LFTCZlTyQuAHx4E7MajWnbryRIxs2NLrv9kIbOnCYnC3L2JXehERrJ8ekBwElh9OK+fbn1QRobdeXta/H8vLyBu9r07tnbm4uJpOJ8PDzNRrDw8PJzKxbyiUyMpKPPvqI3r17o9Pp+Oqrr7jqqqvYsGEDw4YNAyAzM7NRbc6aNYsZM2ZU/11cXExMTAyjRo3Cz6+B1X2aicFgYN26dYwcORK1Wt2kNgrK9UzfsQGAR28eTlSAZ7NsUhr/gP9+ZFBIMeYra2eif3fiO47sPYJKoeLtq9+mS0iXZvVnbyK7FvLhiRQeO/AjoWvX0uPOO/Ds0aPJ7VnjHF4OqD5+BYCIK+5nbGfnUzi41HkcI46hYmsF65LX8av4K8uvXo6fpuY+8eO+NNh7hK4t/LjthmbK1plNiEveQV2ex9guwYhuuDSc4HmGJf+cIV8bydixPazWrnw9XoTyPFT7JZ3xzjc+Tme/Fg42qH6aeh7//fEwHfJ3cMthSX4yYs5s2t18c9OMyIyBT38mouw4Y0cOl8I03IyV2Ts5kFKEEN2VsX1jLn1AI7HV9WhZUW8IdpkOuDAbXhTFejPk27dvT/v2NbOFAwcOJCUlhTfeeKPaoW1sm1qtFq229pBErVbb/UbYnD63nMnCLELHSD9iQ63giLcfA//9iPLUWpSjXzjvreP5x3lrvxRg/0TvJ+gZ6dyZ6nXRu1UID3Uayr+5ZxiRup+sp58h7qeVqAKbNwJ3xPfGZShKg+yjIChQtRsFTvx/uth5nD94Psfyj5FamsrCXQt5a/hb1feXLaclPccR7cOs8D1QS8lhh1agOvsPtBnezPacj6s7RbLknzNsO5OHKCjRqKybuiFfj3WQtAkQIbwL6uBWjramQTTmPJrMIruPJPPCnq9RmE34jR1L0IQJTVfeie4J/jEIRSmoU3fUVNR0I0Z2iuBAShH/nshl4iDbFbqx9vXYmLZsmhQWEhKCUqmsNXOanZ1da4b1YgwYMIBTp05V/x0REdHsNl2Rv5urbnAh1VXDTkB+QvXL5YZyntr4FAazgeExw7mro2vGFCkVAsM7hLO0+y2UhERizMgg47nn3F66xKFYkpta9AavIMfa0gx8Nb68MfwN1Ao165PXs/y4FJtvNJnZfEpKHBnewUrXoSUD/ZR7Lp13jvIjxEdDmd7EnkT3F3d3CixFc9xU3eBAcgETty0norwAVXQ0EfPnNU9GUhBqVEfcNEFzVJV819YzeZTpGhFj7ELY1KHVaDT07t27VkzFunXrGDRoUIPb2b9/P5GRNeVVBw4cWKvNtWvXNqpNV0NvNLPpZC4gCblbBc8AaDlQ2j6nathru18jsTiRMM8wFgxa4HR6s43hqo5hVKg9eGfI/aBWU/r3egpXfO9os9wXy8OgzUjH2mEFOgd35ok+TwDw5p43OZF/gv0phRRXGgn0UtM9OsA6HbW+EgSFNLNdmGKdNp0IhULginZVagcnZLUDm2My1lyHbjjTCHD6828Yln4Ik0JJ9KI3L6032xAsZXDddGDZJsyHmCBP9EYz28+4Zzlqm8t2zZgxg08++YTPPvuMY8eOMX36dJKTk5k6dSogxbdOnDixev/Fixfzyy+/cOrUKY4cOcKsWbNYuXIl//d//1e9z7Rp01i7di2vvvoqx48f59VXX+Xvv//m8ccft/XHcRg7z+ZRqjMS6qulawsrasBa5LuqZJbWJK5h5amVCAi8PPRlAj1cO0B+aNsQVAqBTQShnPIIAFmvvILutPtVhXE4JoNUmQhqZjtcnDs73MkV0VegN+t5ZtMz/H1ccjiHtg1tWlGTuvAKguh+0rabFlkY0UGSDdpwwv1kkZyO1N1QWQgeAdCij6OtsTq606fpsPJjAHJvewDPbt2s03DcMGnFsuAs5J2xTptOhCAIXFEl9bnxpHtehzZ3aG+77TYWL17MggUL6NGjB5s2bWL16tXExsYCkJGRcZ4mrV6v58knn6Rbt24MHTqULVu28Mcff3DzOcHegwYN4rvvvuPzzz+nW7duLFu2jBUrVtC/f39bfxyH8e9x6Qt4ZfswFNZ6kEKNBl/SVtLzTjJ/m6QGMbnrZPpF9rNePw7C10NN/3hp6XtTt6vwHjwYsbKStBlPYD5Hyk3GCqTuBl0xeAU7fXWwhiIIAgsGLyDEM4QzRWf4NfkjAIa3b4YGdF24+ezQ0DbSAOBUdikp+Q3PWpZpApZBUZurQeleqhnmykoSp01HYzSwJ6w9XWc8bL3Gtb41K5YNLDjkalhWSjadkh3aJvPwww+TmJiITqdj79695yV3LVu2jA3nyGw8/fTTnD59moqKCvLz89m8eTNjx9bOlL711ls5fvw4er2eY8eOnefwuiOWL+AV1n6QBreG4LYYzUZmbphOiaGEbiHdeKjHQ9btx4FcWVXL+p+TuUS98jLK4GB0J0+S/cabDrbMzbA4Y62vhGZq/joTQR5BvDjkRQBKtZtQ+RxhWHOKmtSFZWn47EYwVFq3bSfA30tNr5YBAGxw09khp+HccrduRvZrr2E+c5p8rS9rr59CsK/1dI0BaHOV9NtN42gHtg5GrRRIyisnMbfM0eZYHfd56rgxaYUVnM4uRakQGNymiWU2L0a70XwY4M/+0mR81D68OuxV1Ar3yRq+qip5Z9fZfMp9A4h6+SUACr76ipImaBbK1MPpKofWDeJnL2RQ1CAGBt8CgHf0T5iEQut2EN4FfCPBUA5JW6zbtpMwvL10HW6Qq4bZjqJUyPoPENwm7MdCyb//UvCtlJz5Zu/b6d+r7SWOaAKWe9fZTW45sPTRqugTK61YumPYgezQugCbqr54PWIC8Pe0vqO5NySWjwIkGbDnBswh2jfa6n04klYh3sSHemM0i2w+mYvPsGEE3SvFbWfMehZDtvyAbTYlmZB5GOlBepWjrbEJysKxmCpaYBbKmL1lNmbRbL3GBcHtww5GVDm0W8/kUmkwOdgaN8Xy3Ynu69IqIxdizM0lY/YcAH5tN5x9Ye0ZYS2VkXMJ7ww+EWCsgOSLF39yVSyrvLJDK+MQLA7tsLZWXuYEivXFzEr4AbMgcENJKWN9bKdP50iGV8UOWSSXQp94Am2HDpgKCsiYOQvRbEXn5HLEEnMW1QO8bbCK4GCMJjNbTxdQkX47GoUHOzN3suzIMut2Ui3f5Z6JYR0jfQn301JpMLPrrCzfZROqww3cR65LFEXSn30WU34+hlat+aT9NUT4edAp0gZFkc6T73LPOFqLH7H9TJ7bDSxlh9bJMZrMbDktyXVZPX4WeGnnS2SUZxKDmll5Be57EbeTnKxNJ3MQRRGFRkOLN99A8PCgbNs2Cr7+2sEWujhuHG4AVMt1+auieLb/LACW7lvKkdwj1uskfjgo1JImdK77qXAIglA9SyvLd9kAQ2WNykhb94mfLfjmW8o2bUbQall30yMYlCpGdAiznZxk2yqH1k1XSjpG+hLqq6XCYGJPYoGjzbEqskPr5BxIKaSk0kiAl9q6cl3A6oTV/JHwB0pBycvR1+ItinDmH6v24Sz0jwtGo1KQXlTJmRwpGF7bujXhM58BIPuNN6k8edKRJrouJiOc+VfabuueDu2GKgdsWNtQbm57E6NiR2EUjczcPJMKY4V1OtH6QmyVlrabztJa1CFk+S4bkLRVisH2jYSIro62xiroTp0i+7XXAAh78kl+L5HKvVtdZeRc4odLutC5J6Aw+ZK7uxrnyne5m9qB7NA6OZZwgyFtQqynewlklGbwwg6p3O2UblPo3vVO6Y2krWCw0gPaifDUKOnXqkq+65zYoYDbbsPniisQ9XrSn34Gs17vKBNdl7S95+he9na0NTbB4oANbx+KIAg8P/B5wjzDSCxO5M09VlTLcPOwg8FtJF3os7llbpll7VAsM4ptR0pL5y6OWa8n7cmnEPV6vIcOpXTsTZzNLUOpEBjUOth2HXsGSjHI4LYrltV6tG42sJQdWifHErhtTZkgs2hmztY51RJdD3Z7EEI7gG8UGCshyT2D4Ye2lcIONp8zKhUEgcgXFqIMDER3/Dg5S5Y4yjzX5fS5cl1Kx9piA7KLKzmSXgzUXIf+Wn8WDlkIwIoTK9icutk6nVkc2qStoCu1TptOhK+Hmr5VA8sNctiBdTlT5Xy5ibpBzqK30J04gTIwkKiXXmTLaam6Va+WAfh62FiFx83L4A5pE4JCgBNZJWQUuc8EluzQOjH5ZXoOpRUBNSMqa/DV0a/YlbkLT5UnLw19SZLoEgRoc6W0g5uGHVickR0J+eiMNcHwqtBQIl+QnJP8zz6nbOcuh9jnslSXu3WPB+mFWHRTu0X7E+KjrX59UNQg7up4FwDPb3uegkorxKOFtIWAWDDpIdE95bssVcP+dbPZIYdSlAq5J6Wl8rgrHG1Nsynbvp38ZcsAiHzxRVShodUTEUNtkBxdC8u9LGEjGN1v1S7QW0O3qtLdm9xI7UB2aJ2YLadzEUXoEOFLuJ91BKRP5J9gyT5pFvKZvs8Q6xdb82Zri6i0ey6zdIioCYbfe0EwvO9VVxEw/lYQRdJnzsRUXOwgK12M0hxI3y9tu6lDa1mWG17HoPLxXo/T2r81uRW5LNi+AFEUm9eZIEDrEdJ2wr/Na8tJsejRbk/Io0LvXlnWDsMSw96iN3gGONSU5mIqLiZ91rOAFBLme+WIKpURKTnastJmUyJ7SBUP9SWQ6p4THO5YBld2aJ2YTVYON9CZdMzcPBOD2cCImBHc3PaC6mqWYPicY1CUZpU+nQlBEKpvhhvrCIYPnzkTdcuWGDMyyFyw0N7muSaW2fyIbuAb7lhbbIDRZK6eGbqifW3dSw+VBy8PfRmVQsXfyX+z6syq5nfa2rJS4p4ObdswH6L8PdAbzexKlOW7rILlOrR8d1yYzBdewJiZiTq2JeHPPA3AobQiiiuN+HmoqmcWbYpCcc4Ej3uGHVhUk7acysVocg/ZStmhdVJEUbS6/uzb+97mdOFpgj2CmTdoXm3ZE68giOolbbtr2EHV/3Lzydxa7ym8vWnx2qugVFL8++8U/f6Hvc1zParlutxzdtYi1xXgpaZHTECd+3QM7sgjPR4B4JVdr5Baktq8TuOG1WRZu+nAckjVwHKLm2VZOwSzuUauy8Ud2uK/1lD862+gUNDi1VdReHkBNffrIW2tmxx9Udw8jrZ7tFSoqbjSyMHUQkebYxVkh9ZJOZ5ZQnaJDk+1kj6tApvd3u7M3Xx19CsAFgxeQJBHPVVkqmeH3DPswPIgPZpRTE6Jrtb7nj16EPK//wGQuWABhqwsu9rnUphNNeEpbirX1VCVkfs730+vsF6UGaQqYiZzM5bSPQNrBpZuGnYwxDKwPFV7YCnTSDIPQkU+aHxdWmXEmJND5ty5AARPeRDPHj2q37Nr/KyFNlcBglQBsSTTfv3aCaXinBVLN4lnlx1aJ8XyIB0QH4SHunmZ46X6Up7b+hwiIre0vYVh0cPq39lStjRhg+SwuBkhPlo6R0kVZracrvsiDnloKh6dO2MuLiZjznPNj4t0V9IPSA9SrV+NzI2bYSlqcqlVEqVCyYtDXsRb7c2+7H18efTL5nVsiaN107CDwVWyS8czS+ocWMo0AstqWtwwUNo4+99WiCLZc+dhKipC26kjoQ8/XP1WcaWB/SmFgDSwtBveIVLlQ3DbvJJhbhZHKzu0Too15bpe3/M6aaVptPBpwVN9n7r4zi36gNYfKgokh8UNsfxPN9URdgAgqNVEvfoKgkZD2ebNFK743p7muQ6WB2n8Fa77IL0IRRUGDloepA1IRIn2jebpvlLM39L9SzlVcKrpnVtWShL+lZaU3YzgcwaWlmQfmSZiGfRYBkEuiP+uXZRv3oyg0dDitdcQNJrq97afycNkFokP8SYmyMu+hrl52IElMexQWhF5pa4/sJQdWiekXG+sLknXXId2Y8pGfjr1EwICCwcvxFvtffEDlCqIr5rBddOwgxo92lzM5rpnX7Vt2hA6fToAWa+9hj7Z/SrGNBtL3F686z5IL8b2M3mYRYgP9SYqwLNBx9zU5iaGRQ/DYDYwe8tsDCZD0zqP7gsaHyjPg6zDTWvDyRlyznUo00T0ZZC8Q9p20fhZQ0oKoVX5CqEzpqNt0+a892vCDew4O2vB4tCe+cctVyzD/TzoEOGLKNasRrkyskPrhOxIyENvMhMd6El8yCUc0ItQUFnA3G1STNI9ne6hb0QDl4XdXL6rd2wgXholuaU6jmXWL88VdO9EvPr0QSwvJ33Ws4gm97uhNRldKaTslLbjhzvUFFthmTlszDKnIAjMGzgPf60/x/KP8dHhj5rWuVINrYZI226aoDm0TVWW9ekcOaynqSRuBbMBAlpCULyjrWk0oslE1rOzUej1ePbtS9DEibX2sQx47Bo/a6FFHyk2ubIQMg/Zv387YFE7cIewA9mhdUIsS+HD2oXWViJoIKIosnDHQvIq84j3j+exXo81/GDLSD91N1QWNal/Z0arUjIgXorhu9jskKBQEPnKyyi8vKjYu5fCr76yl4nOT/J2l36QNoQtTXBoAUK9QpnTfw4AHx/6mP9y/2uaAW4u39WnVSBalYKsYh2ns92vKppdsCQNtr7SJcvd5i9bRuWBA5i0WsJeWIigON8lScorIymvHJVCYIAty93Wh1IFcUOlbTe9Dq84J0HT1QeWskPrhGy0glzX6rOrWZe0DpWg4qWhL6FVai99kIXAWAhuA6JJqpTihgyrWr66VJUUTXQ0YTOfASDv7aVoMmXVA+CccIPhLvkgvRSpBeXVdeOb8iC9Ju4arml1DSbRxLNbnqXSWNl4IyyhHMk7QF/e+OOdHA+1kn5xktqKHHbQRKrj2F0v7Ed3+jQ5S94GIGfcdaijomrtY/le9I4NxEersqt91VhWoCz3PDejV2wgHmoFOSU6Trn4wFJ2aJ2MlPyaB+mgNk0bkWaXZ/PizhcBmNJtCp2DOze+EUvYgbvG0VbFJu9JLKBcb7zovgHjx+M9bCgYDER8/z2ioYlxke6EZbbCBR+kDWFL1YO0R0wAfk2sGz+7/2xCPEM4W3SWpfuXNr6BkLbgFw0mHSRva5INzo5l9tsd4vfsTlEa5ByvKnd7EeUaJ0Q0GkmfOQtRr8dr6FCK+/Spcz9L/Ky1igs1CYtDm7wDDBWOs8NGeKiV9G0lDSy3uPjAUnZonQxL3F7PJj5IRVFk/vb5lOhL6BTcicndJjfNEIt81+l/wMWXIeoiPsSbFgGe6E1mdiZcvFqRIAhELnwBhZ8fHmlp5H/8sZ2sdFJKsiD7CCC4Rd34urA4WIObIRMU4BHA/EHzAfjq6FfsztzduAYEAVoPl7bddLnTkhi2IyEPvdH91BxsimXGMKqXVBTHhcj7+GMq//sPhb8/YfPm1rnKYzSZ2XY6D3BQQpiFkHbgG1k1sNzhODtsiGVg6eqKI7JD62RYHqSDmvggXXVmFZtSN6FWqHlx8IuoFU2UU2o1BJQaKEqGvDNNa8OJEQSBYe2qwg4aUK1IHR5G6JzZABR8/AmVR4/a1D6n5mxVGEpkN/B2QFybjTGbRbadsc6DdFj0MG5uezMiIs9tfY5yQyNDB+LdW4+2Y4QfIT4ayvUm9iUXONoc16K63K1rrZJUHj9OznvvAxAxZzaqsNolpQEOphZSojMS6KWmc5S/PU08H0GouQ7dNOzAMnDfkZCHwYXL4MoOrRNhNotsr3qQNkVAOrMsk1d3vQrAIz0eoU1gm0sccRE03hDTX9o+u6Hp7TgxlhjlS8XRWvC55hpKunQBo1FSPdDrbWme83Ju/KwbcjSjmPwyPd4aZb3lbhvDU32eItI7krTSNN7a+1bjDo4fAQjSjHiJ+8VvKxRC9cPU1Zc77YrZfH5CmIsg6vWkPzMTDAZ8R16N33XX1buvJTl68CWq9NmF6jha9xxYdor0I9BLTZneVK297YrIDq0TcTyzhLwyPV5NeJCKosi8bfMoNZTSLaQb93a+t/kGWZaT3TQxbFDrEBQCnMkpI73w0rFRgiCQfdONKAID0Z04Qe4HH9jBSidDFN3eobUkogyID0atbP4t0kfjUx168N2J79iZsbPhB3sHSzPh4LazQ5bB+2YXX+60K1mHJY1ijY9LVenLef99dCdOoAwMJGLevIuq+FTHzzpCrutC4quehRmHoCzPsbbYAIVCqF4VduV4dtmhdSIs8Sv94oLQqBp3an469RNb07eiUWhYOGQhKoUVMkItDsvZTW4pKu3vpaZrdABA9RLzpTD5+BA2Wwo9yP3wIyr+O2Ir85yT3FNQnAZKLbQc6GhrbEK1/qwV4/YGRg1kfLvxAMzdNrdxoQfV8l1uqkdb5bAcTi2kqFxOuGwQLljutuLwYfI+kvIPIuY+jyq4/nClogoDBxpRpc/m+EZAaEdAhMRNjrbGJrhDHK1dHNr33nuPuLg4PDw86N27N5s3b653359++omRI0cSGhqKn58fAwcOZM2aNefts2zZMgRBqPVTWdkEaRwnYuuZpuleZpRm8Pqe1wF4rNdjxPtbSRc0qido/dxaVNpSU35bIy5in9Gj8B1zDZhMZMyaiflyCj2wzBK2HADqhlXPciUqDSZ2JUpJgtZORHmizxNEeUeRVprGor2LGn7gufF7bpigGeHvQZswH8wibDvjug9Tu+Jicl1mvZ70WbPAZMJv7Bj8rrnmovtvP5OLWYQ2YT4NrtJnc1q7dxytxe/Yn1xIqe7iyj/Ois0d2hUrVvD4448ze/Zs9u/fz9ChQxkzZgzJ9ZQS3bRpEyNHjmT16tXs3buXESNGMG7cOPbv33/efn5+fmRkZJz34+HhYeuPYzP0xpps+8ZkVouiyPPbnqfMUEaP0B7c3fFu6xmlVEHsYGnbTS/iwecsszRGVDri+edRBgejO3Wa3Hffs5V5zkd13J5rPEgby+7EfPRGMxF+HrQO9bFq295qbxYMXgDAihMr2JHRwIzplgNA5QmlmZB9zKo2OQty2EEj0Je7XLnb3HffQ3/6DMrgYMKfe+6S+ze1qIlNcXM92pggL1oGeWE0i+w665phFTZXKl60aBGTJk1i8mRJPmrx4sWsWbOG999/n5dffrnW/osXLz7v75deeolVq1bx22+/0bNnz+rXBUEgIiKiQTbodDp0Ol3138XFUrlTg8GAwU6aopZ+6utvd2I+FQYTQd5q4oM8GmzXj6d+ZEfGDjyUHsztPxezyYzZilmKilZDUZ78E/OZDZj6/5/V2nUWukf5oFUpyC7RcTy9kDZh9Tsx555DtY8Poc/NIfPx6eR9/DGew6/Ao0sXe5ntGMxGVGc3IQCGlkPARfV4L3YtbjyRDcDA1kEYjdafpegV0ovxbcfzw6kfeH7r83w/9nu81Zcqb61A2XIQioT1mE6twxzU1up2OZqB8YEs25bI5pM5Db73Xeqe6q4ICZtQmfSIftEY/Vo6/XVYeeQIeZ98AkDoc3MQfXzOO2d1ncetVXHs/VsFOM/5jeqLSqFCKEjEkH0KAls52iKrMzA+iOT8cjadyGZo68ZJwdnqemxMezZ1aPV6PXv37mXmzJnnvT5q1Ci2bWuYULjZbKakpISgoPP/uaWlpcTGxmIymejRowcLFy48z+E9l5dffpn58+fXen3t2rV4eXk18NNYh3Xr1tX5+uoUBaCglYeOv/76s0FtFZgLWFosCbZfqb6SI1uOcATrxnT6VsCVgDlxG3/9/gtmhcaq7TsDsd4KThYp+OS3zQyLvPQs7bnnMKJHd/wOHOTMtMdJfuxRRLVrxLM1hcCyUwzTl6JXevPnvlQQ0h1tUrOo61r865ASEPApSWH16rpXkZpLB7EDAYoAMsoymLFqBjd43XDJY1rrwugC5O78kR15rWxilyOpNIFCUJJSUMGXP60mpBGLbfXdU92VTmnLaQskq+I58GfDnhWOQjAaabl0KVqTieLu3Tmp08Hq1XXuazmPhTo4m6dCQKTw1B5Wn7WnxRdniGc8wWUnOfLbuySFuN8qlWexAChZcyCJniQ0qQ1rX4/l5Q3PN7CpQ5ubm4vJZCI8PPy818PDw8nMzGxQG2+++SZlZWVMmDCh+rUOHTqwbNkyunbtSnFxMUuWLGHw4MEcPHiQtm1rz17MmjWLGTNmVP9dXFxMTEwMo0aNws/Pr4mfrnEYDAbWrVvHyJEjUdfh9Hzx8S6gkFuHdmFs7+hLtieKIo/8+wj6Yj09Qnuw4OoFKAQbRJCIIuKSxajKshnTJRix1VDr9+Fgkr0TePPv0xR7RjJ2bI9696vrHJoGDSL5xpvQZmfTLzGR4GnT7GS1/VFsfh1Ogqrd1Yy9tn65HWenvmsxr0xP6vYNADx8y5WE+DSiXHQjicmKYcr6KezW7+aBIQ/QP6L/xQ/IbgUfLyes8jRjR18taUS7GT9k7WJPUiHqmK6M7Rtzyf0vdU91V1SfSPkSLYbeSVSXsQ625uLkvfMuBZlZKIMC6b70bXoFBtba58Lz+NP+NNh3hK7R/tx6/QAHWF0/Cp//YPNrdPPJp/NY5/7fN4UBZXq+eHUDGRUCfYdeRahvw++BtroeLSvqDcEuxZEvlOYQRfGich0Wli9fzrx581i1ahVh54gvDxgwgAEDar7ogwcPplevXixdupS33367VjtarRattvaJUavVdr8R1tVnSaWBg6lFAAxtF94gm1aeXMmOzB1olVpeGPICWo3tHr7ED4fD36NK3gptXSNmqzEMbR/Om3+fZsfZfASFEtUlpJrOPYfq0FAi5s0l7dHHKPh8Gf7XjMGzSxNKDbsCiVIyp6LNlSjcwIG48FrclSTJBHWI8CUy0LrxsxcyMHogt7e/ne9OfMfCnQv56YafLh56ENUNvEMRynJQZx6AVoNtap8jGNYujD1JhWxPKGDioIYntjriPu4wyvMh6z8AVG2uBCf+3BVHjlBQFWoQ8fxcPOopoGDBch53ni0EYEibUOc7r22vhs2voUjchEKpBIV7CUWFB6jpHOXHf2nF7Eoq4saeLRrdhrWvx8a0ZdOzERISglKprDUbm52dXWvW9kJWrFjBpEmT+P7777n66qsvuq9CoaBv376cOnWq2TY7gl1n8zGZRWKDvYgJunQIRGZZJm/seQOAR3s+SqxfrG0NdPNg+K4t/PH1UFFSaeS/9IaPBi34jRyJ39gxkurBs25acEFXAqm7pG031Z+1xO3ZKxFleu/ptPBpQXpZOov3Lr74zoIgSTSBJKPnhljkmbadycNsdj81B6uQuBkQJQkp34s/Qx2JqNeT8exsMJnwveYa/K4Z3bDjRLFa7ac5ZadtRoteoPGFigL3Vf5xYT1amzq0Go2G3r1714qpWLduHYMGDar3uOXLl3Pffffx7bffcu21116yH1EUOXDgAJGRkc222RE0pm68KIrM3z5fKqAQ2s26qgb1YRGVTt8HlUW278/OKBUCA+Ml+a6mavCFz5mDMigI3cmT5H74kTXNcw6StoHZKCVCuGEyhCiKNZnVdtK99FJ7MW/QPEAquLA7c/fFD7AUOjnrnoVOurXwx0eroqjCwNGMxg8sLwssRW4s92QnJffDj2oKKDw3p8HHnckpI6tYh0aloHds7fAEh6NUS2XhwW2rhp2rR9sY5R9nwObz5TNmzOCTTz7hs88+49ixY0yfPp3k5GSmTp0KSPGtEydOrN5/+fLlTJw4kTfffJMBAwaQmZlJZmYmRUU1jtT8+fNZs2YNCQkJHDhwgEmTJnHgwIHqNl2NbacliYzBrS/9IP31zK9sSdsiFVAYtBClQmlr88A/GoJag2iGxC22788BWAYTTdXBVAUFVd+4cz/8kMrjx61mm1NQXR3M/RIhABLzykkrrECtFOgX17js3uYwIHIAt7a7FZAKLlQYL1KxzjJDm7obdKV2sM6+qJQK+lf972U92nqwDGYs3wUnpPL4cXI//BCAiOefu2gBhQuxnPc+sYF4qO3wbGsKbr5i2beVVNgpo6iShNwyR5vTKGzu0N52220sXryYBQsW0KNHDzZt2sTq1auJjZWWyTMyMs7TpP3www8xGo088sgjREZGVv9MOyfZprCwkClTptCxY0dGjRpFWloamzZtol+/frb+OFYnu6SSE1klCAIMbH3xCz+7PJtXd78KwEM9HiI+wEoFFBpC9UXsnrNDg9tI//vdiQVUGppWFc33mmvwHTkSjEbSn30W0VnkZqzBmarZCDcNN7A8SHu1DMRLY5fUgmpm9J5BuFc4KSUpLN2/tP4dg+IgoKU0U57cQA1bF2NQ9eyQa+pg2pSiNMg7DYKiRh/cyRANBtKffRaMRnxHjsT3EgUULmRrI1YrHYblHpi0HQyXLpnuaniolfSpmh1vTMEhZ8AuEc0PP/wwiYmJ6HQ69u7dy7BhNaPLZcuWsWHDhuq/N2zYgCiKtX6WLVtWvc9bb71FUlISOp2O7Oxs1qxZw8CBrlmG0zI72znKjyDv+jOXRVFk4Y6FlOhL6Bzcmfs632cnC6uId+/lztahPoT7adEbzexNKmhSG4IgEPH8cyj9/dEdPUbep59a2UoHUZIFOccAwalnhpqD5Toc1IBVEmvjq/Fl7sC5AHx99GsOZB+of+fqsIMNNrfLEQxqbRlYSgUuZM7Bcu+N6gmeAQ41pT7yPv0M3dFjKP39iZj7fIOSvy2YzCLbz1iuw4bP6tqd0PbgGwkmHaTsdLQ1NsFV42jdK0XPBakekV7iQbr67Go2pGxApVCxcPBCVAr7ziLRaiggQM5xKM6wb992QBCE6nPQnFrWqtBQwmc/C0jVcXQumqh4HpYkpMhu4GW/5Xh7YTaLbE+oepC2ccyDdGj0UK5vfT0iIs9tfQ6dSVf3jhaH1k1XStqH+xLsraFcb+JgaqGjzXEuLNdhnHPGz+rOnCH33XcBCJ/9LKqQxg0Oj2YUU1xpxFeromsLf1uYaB0EoWaW9ox7xtHWhODlYXKhBE3ZoXUgoig2aIklryKPV3a9AsD/uv2PtoEOqBTkFQSR3aVtN82yrl7uPNO85U6/cePwGT5cWn6bPQfR1LQQBqehOm7POR+kzeVEVgn5ZXq8NEq6Rwc4zI6n+z5NqGcoicWJvHegnnLKlhnyzMOShJOboVAIDKiandsmhx3UIIpOnRAmmkxkPDsb0WDA+4ph+I0b1+g2tp2Rvs/944MvKZ3ocKpXStzzWXiu8s/hNNdJBHfyb417cza3jPSiSjRKBX1b1T/z9cquVyjUFdIusB2Tuk6yo4UX4ObB8JY42sOphRRVND3+VRAEIubPQ+HrS+WhQ+R/8aW1THQMTj4z1Fwsg0pLMoSj8Nf689wAqc79siPLOJJbR9U/33BJsgmxWhfY3aheKZETw2rIOw0l6aDUQswlinA4gPyvvqLi4EEUPj5Ezp/fqFADC9sTJId2sINWSRqFZWCZcQAqCh1piU1QKoTqsI8tp3IcbE3DkR1aB2KZCewVG4Cnpu6Mzn+S/+GvxL9QCkoWDF6AWuFAoelz42hdTM6jIUT6exIf4o1ZhJ0JzZsdUoeHE/7M0wDkLFmCPinJGiban4JEKEwChQpaOlfVHmthidtzhgfpiJYjGBM3BrNo5rltz2Ew1TGwsjxM3TTswPIg3Z9cQIXexVc3rIVlEiGmH6g9HWrKheiTkshZvASAsKefQh0R0eg2jGbYmyzlLjh1QpgF/xYQ3EZS/kna5mhrbILlPGxv5rPQnsgOrQO5lJB7ka6IF3a8AMC9ne+lc7CDK1DFDJBKbhZXZdu6IYPbND+O1oL/LbfgNXAAok5HxpznEM0umORimZ1t0Qe0tq2e5QiMJjM7z0ozQ45ICKuLmf1mEqgN5FTBKT7575PaO8S793JnbLAXLQI8MZhEdie6X1hFkzjrnOEGotks3dsqK/EaMICA8eOb1M7ZEoFKg5lQXy1tw1zkPuPmhU4sA8s9zVD+sTeyQ+sgzk9EqftB+saeN8ipyKGVXyse6v6QPc2rG41XzXKXm4cdNDeOFqTQg8iFCxE8PSnfvZvC779vdpt2pzrcwD3VDQ6lFVGqM+LvqaZjpJ+jzQEgyCOImf1mAvDRoY84XXDB4DF2sCTdlHcKitMdYKFtEQShWsJQDjsAzGY4WxVeEjfckZbUovD77ynfvRvB05PIhQuaFGoAcLJIOm5Q6+Amt2F33NyhbR3qQ5ivFp3RzP7kQkeb0yBkh9ZBHM0opqjCgI9WRbc6Mjq3pW3jl9O/ICCwYPACPFQeDrCyDiwzBG7q0A6ID0YQ4HR2KVnFlc1uTxMdTdj0xwHIfv0NDBkupBAhijU3ayebGbIWlnCDgfHBKBXO8yAdEzeG4dHDMZqNzN02F5P5nBkSzwCI7CFtu2nYgWVgud0KA0uXJ/MQVBZKJVejejrammoM6elkv/Y6AGHTp6OJiWlyWxaHtiHFhZyGVkOl39lHoNR14kwbiiDUxNFud5GBpezQOgjLjbp/XFCtjM5yQznzt88H4I4Od9AzzHluYtWVohK3gNk1liEaQ4CXhi5R0gDDWtWKAu+6C88ePTCXlZExd67rlBPMPQmlWaDygOi+jrbGJljOsaPkuupDEATmDJiDj9qHQ7mH+PrY1+fv4OZhB5bwj8NpRRSVu1GBkqZgCTdoNRiUdpZrrAdRFMmYNw9zeTmePXsSeNedTW6rpNJIclXhO2e7Di+KdwiEd5G2E937OrTGiqU9kB1aB2F5kNZVHWzxvsWkl6XTwqcF03pNq/W+Q4nsIc0UVBZK0kFuSLWo9CnrXMSCUknkiy8gqNWUbdpM8a+/WqVdm2OZ/Ws5AFRax9piA3QGE3sSpUQUZxRyD/cO58k+TwLwzv53SC6uqahYIxvkngma4X4etA71RhRhx1nXeJjajATnk80r/u03yjZtRlCrpXubsullancnFWBGoGWQJ9GBXla00g64ediBxT85mFJIqc7oYGsujezQOgCDycyuehJR9mXt47vj3wHw/MDn8VI72QWuVEkzBeC2F3HNcmeu1WZTta1bE/LIIwBkvfQyxlwXWMJxgbrxzWF/ShE6o5kwXy2tQ50zEeXmtjfTP6I/laZK5m2fV/N9jOl/ToLmGccaaSOqxd1drFqRVTHqIXm7tO0kYT/GvDyyXnwJgJBHHkEb37wS7OeG/bgcbq5HGxPkRUyQJ0azayRoyg6tAziUWkSZ3kSgl5oOEb7Vr+tMOuZum4uIyI1tbmRQ1CAHWnkRLA6Om+pg9okNQq0USC+qJDm/3GrtBk96AG3HjpiKish84UWrtWsTzCYprAScLhHFWlh0L505EUUQBOYOmounypPdmbv58dSP0hvnJmi6eRncbS6y3GkT0vaAoRy8QyGsk6OtASDrxRcxFRWh7dCB4EkPNLu96usw3gWrEMYOAkEJ+QlQmOJoa2zCoPgq+S4XuA5lh9YBbD8n3EBxTiLKhwc/JLE4kRDPkOqlRqfEEgyftA3q0sl0cTw1Snq2DASs+zAV1GqiXnwBlEpK/vqLkr//tlrbVifzsBRWovWrqRDnZuxwMrmu+ojxjeHRno8CsGjPIjLLMqU33Hx2yJKgeSq7lGwrJGi6JAnnrJI4waCrZP16ilf/CeeEUTWH3FIdJ7KkANr+rujQevjVJOq56QSPJa7ZWjkltkR2aB2AxUkaeM6D9Hj+cT7/73MA5vSfg7/WiWtZh3cBz0DQl0L6fkdbYxMsy1/Wnh3y6NSJ4AekWY3M+QswFRdbtX2rYQk3iHWeRBRrUmmSVkrANRJR7uxwJ91Cu1FqKOXFHS9KoQfViWGbJWknNyPAS0PnKElKzZXE3a2KE4X9mIqLyZwnJSsHP3A/np2br4u+o+q8RnmJBHtrmt2eQ3D3ONqqZ+GR9GIKy/UOtubiyA6tndEZTOxJOj8RxWg28vzW5zGKRkbGjuSq2KscaeKlUShqZmnPuqds0LlyJdZWJQh55GE0rVphzMkh+/XXrdq21XBz/dkzxQJGs0jLIC+XSERRKpTMHzgflULFhtQNrElcI80MaXygIh+y3DRBs7X1Cp24HPoySN0tbTtBQlj2669jzMlBExtbnQ/QXCwTBm39XTix8dzKfW6YoBnm50GbMB8pQTPBueNoZYfWzuxPKUJvNBPupyU+xBuAL49+ybH8Y/hqfHm2/7MOtrCBuPmotEfLADzUCnJL9ZzKLrVq2woPDyJfWAhA4Q8/UrZjh1XbbzZGPSQ5VyKKtTl1jpC7q9AmsA1Tuk0B4OVdL1NgKJVm0MFt9WgHXs5xtMnbwWwE/5YQFOdQU8q2b6fwByl+O/KFhSg8rKOLvqPqvLbzc2FH0JKgWZLutgmarqJHKzu0dqYmESUEQRBIKk7ivQPvAfB036cJ8XTueL5qLDMGyTvB4H7xbVqVkr6tpJguWwTDe/XpQ8AdtwOQ8dzzmCsqrN5Hk0nfB4Yy8AqB0I6OtsYmVFcmcoW68ecwuctk2gS0Ib8yn9d2v1Yz4HDT+L1+cUGoFAKpBRUk51kvQdMlcJJVEnNFBRnPzwUg8M478OprHU3qjKIKEnLLUAjQ2pUdWo0XRPeTtt18xdLZB5ayQ2tnLIkoA+ODMYtm5m2bh86kY2DkQG5ofYODrWsEIW3BJwJMOkjd5WhrbELN7JBtRqVhTzyBKiICQ0oKOW8vtUkfTaI6EWWoFF7iZhSU60krlxxaV5MKUivVLBi0AIWg4PeE39nsXaWS4qYJml4aFT1bBgCXYRlcJ3Foc95eiiElBVVEBKEzZlitXctEQZcoPzxdPUzfzQud9I87J0GzxHknsNzvaeXEnJuIMrB1MD+e/JE9WXvwVHkyd9Bcp5UOqhNBcPuwA4uzsyMhH5PZ+jMISh8fIuZJMx/5X3xBxWEniYN0kgeprdh5VophbxvmTaiv6xWM6Bralbs63gXAghNfUebl3gmaFhUKV5ANshoVhZBxUNqOG+o4Mw4fJv+LLwCImDcXpY/19Jots30DXFHd4ELOlbJ0wwTNQG8NnSKrEjSd+DqUHVo7klCViBIT5IlaW8yivYsAeKznY7TwaeFg65qAmzu0XVv446NVUVRh4FiGbdQIfIcPx2/cODCbyZg9B1Hv4CxSfXnNjLsTJKLYAkvGvKvNzp7L//X4P1r4tCCzPJMlUVXxlW56HZ4bR+syZaObS9I2EM0Q3Ab8ohxigmgwkDHnOTCb8bv2WnyHD7de26JY7Ri5hUMb1QvU3lCeB9lHHW2NTaiJo5UdWhlqElEGxgXz4o4XKTOU0S2kG3d0uMPBljURi0Obthd0JY61xQaolAr6x0k3W1tq8IU/OwtlYCC6kyfJ+/RTm/XTIFJ2gkkPftEQ1LwKQM6KJVN3QJzrPki91F7MHSjN7n9nymW/VuO2Dm3PlgFoVQpyS3WctnKCptPiBKskeZ9+iu7ECZQBAYTPtm6yckp+BWmFFaiVAr2rQkpcGpUGYgdK224bR1tVuU92aGUAThZLDq1fyBE2pG5ApVAxf9B8lIqm18F2KIGxENBSysRNdrJMfSthjyxrVWAg4c9KD4zc995Hd8aBmbJnnUvI3dpkFVeSkFuOgEj/uEBHm9MsBkYN5MY2NyICc0OC0aW4f4KmMz9MrYqDHVpdQgK570rJyuGzn0UVZN3Bn2WCoEdMAF4aVw+grcLNVyz7xgWhVAgk55eTYsUKmtZEdmjtRGG5gbQyQFnGmswPAJjSdQptAts41rDm4uYXsWVUuvtsPgaT7WKj/K67Fu8rhlUv84mOisOqfpA6Lm7PlliWy6K9wc+zeVWOnIEn+zxJiGcIZzVqPvLR1uiWuhkDXWC502qU5UL2EWm7lf2vQ9Fslu5BBgPew4bid911Vu+jruJCLk91HO1WMBkda4sN8NGq6B4tFXxy1kInskNrJ3Yl5iMiENpyDYX6AtoEtGFy18mONqv5uHn5zQ4RvgR6qSnTm/gvzXZVvQRBIHLuXBReXlTs30/Bt8tt1le9VBbVJBa5aUKYZWaonSsLuZ+Dv9a/Wrv6swA/Tpz81cEW2YZqhzYhD7MNEjSdCosEW1hn8La/w1ewfDkV+/ah8PIict48qycri6JY7dC6kg70JYnoBh4BoC9x2wTNwW2cO0FTdmjtxI6EfJTeJ6n02IWAwLxB81ArXX+GqHoGIeMgVBQ41hYboFAIDIi3PExtWyVFHRVF6BOSLE7OokUY0tNt2l8tkrZLiShBrcE/2r5924nqykSurHt5ASNjR3KVX1uMgsDcjPUYze43O9TtnATNozZK0HQaHBhuYMjIIOdNKVk5dMYM1FHWT0g7nV1KbqkOrUpRLcnmFiiU0GqItJ3onhM850pZOmOCpl0c2vfee4+4uDg8PDzo3bs3mzdfXAR848aN9O7dGw8PD+Lj4/nggw9q7bNy5Uo6deqEVqulU6dO/Pzzz7Yy3ypsTcjAI+InAO7qeBfdQ7s72CIr4RcJIe0AUVpqcUMsswgWDWFbEnjHHXj26oW5vJyMefPse9Nw83CDlPxyUgsqUCkE4t3IoQV4tv+z+JrMHBEMfHP4M0ebY3VUSgX94mxX6MSpcJBDK4oiGfPmYS4vx7NXLwLvtE2ysmW5uk+rQLQqF80fqQ83D8Hr1TIQjUpBVrGOhNwyR5tTC5s7tCtWrODxxx9n9uzZ7N+/n6FDhzJmzBiSk5Pr3P/s2bOMHTuWoUOHsn//fp599lkee+wxVq5cWb3P9u3bue2227jnnns4ePAg99xzDxMmTGDnzp22/jhNIrukkjTFzyg0hYR7RvBoz0cdbZJ1cfOL2BLntTe5EIONQ1sFhYLIFxYiqNWUbdpM8e+/27bDc3GCzGpbYgk36B7tj9bNnqNhkb15olK6nb9z6ANSilMcbJH1GXRO2IHbUpwOeadBUEDsIPt2/fsflG3chKBWE7lwAYKNiqpsO20JN3Cj+FkLlntn8g4w6hxriw3wUCvp2yqQDhG+FJQ5WGKyDmyeXrho0SImTZrE5MlSvOjixYtZs2YN77//Pi+//HKt/T/44ANatmzJ4sWLAejYsSN79uzhjTfe4JZbbqluY+TIkcyaNQuAWbNmsXHjRhYvXszy5bVjD3U6HTpdzZeruFhasjIYDBgMtq+us/zAJtSB2wB4ts8s1Kjt0q+9EFoORrX7E8SzGzG60eey0DJAQ5ivluwSHYklgs3PnSImhsD//Y/8d94h88WX0Pbrh9LKWca1KM9DnSUVdjC0GABueB63npIc2n6x/mDMdatrEODG8MGszlnPLk+Yu20uH1z5gWsVa7kEfVpKCSk7E/Ior9SB2QTgVudROP0vKsAc0Q2Tyttu16GpoIDMl14CIPB//0PRsqVN/q9ms8j2BOk67Bvrf94z2C3OY0BrVN6hCGU5GJN2ILa076DEHnx0V080Kmmwc+45s9V5bEx7NnVo9Xo9e/fuZebMmee9PmrUKLZt21bnMdu3b2fUqFHnvTZ69Gg+/fRTDAYDarWa7du3M3369Fr7WJzgC3n55ZeZP39+rdfXrl2Ll5dXIz5R0/g+azmCViRY35OSwyWsPrza5n3aE7WxnLGAkHOc9auWo1P7O9okqxOjVZBdouBUkcC6dets32FkBLEREWgzMzk47XEy77jdtt0V7KIfUOwRzb+b9ti0L0cgirDhmBIQUOYlgD/2OY92JLrYj7m5+dwcHcXurN288MsL9Nb2drRZVsMsgpdSSZnexMc//kWrqqq/7nQeeyQtJxY4Y2rB0dX2e05EfPcdfgUF6CIi2BEZATbqO7UMiipUaJUiqYe2kXFOcUR3OY+91a2JJofT6z7jRGSho82xO9Y+j+XlDZcIs6lDm5ubi8lkIjw8/LzXw8PDyczMrPOYzMzMOvc3Go3k5uYSGRlZ7z71tTlr1ixmnFODuri4mJiYGEaNGoWfn19TPlqj6F44lEW7P6ZVQUtGjhyJWu0GyWAXIGZ/gJB1mKvbaBA7j3W0OVanbG8qe385ysligbfsdA4r4+NJvetu/A4coP2Dk/EeZrtQAMVfGyARvLuOZewo9zt/Z3LKKN6xFY1KwaQbh7Hp33/c71os7oF66Yc8UlDIoqAA/jb+zdRrphLqGepoy6zGn8UHWHs0GyGiAyMHxbBu3Tq3Oo+qd2YDEHflvbRqfaVd+izbvJmM/QdAoaD1W4vo3KWLzfr6dGsiHDrJgNahjLu2FyDNwLnTeRT258LqHbRTZ9J6rPvdS+vDVufRsqLeEOyiaHzhspcoihddCqtr/wtfb0ybWq0WrbZ2zXa1Wm2XC6hVaBBvjpzB6tWr7dan3YkbBlmHUSVvgx62nU10BEPbhQNHSSoFvVnAyw7nUN2zJ0ETJ5K/bBk5L7yI728DUPp426azpC0AKOOvQOmG38/dSYUA9IkNxMfTA7Df9W83gmMhuA335J3mr5jOHC1L4419b7Bo+CJHW2Y1BrcJZe3RbHYmFvDQFVIlO7c5jwWJUJQCChWq+CFgh89kKi0jZ+ELAARNnIhvz5427W/nWUkJZ0ib0FrnzG3OY5sRACjS9qIQ9aCx0T3bSbH2eWxMWzZNCgsJCUGpVNaaOc3Ozq41w2ohIiKizv1VKhXBwcEX3ae+NmXsgJsnhsUEeREd4IFZFNiTZD95stDHHkUdHY0xI4OcRTZyTIozIPckIECrwbbpw8FYEoncSveyLloNRQUs0MSjElSsS1rH+qT1jrbKaljO357EAnRGBxUfsRWWe2eLPnZzgnLeegtjRgbq6GhCH7NtsrLBZGZXlVLMQHe+DgPjwD8GzAa3rKDpjHJdFmzq0Go0Gnr37l0rpmLdunUMGlR3sPTAgQNr7b927Vr69OlT7anXt099bcrYgdhBICih4CwU1q1g4epY9Gh3nLWfQ6vw8iJy4QIACr79lvK9e63fiUXIPbI7eLp2Odi6MJvFaqknt6pMVBdVA8v2qQe4v8v9ALy480WK9e6h3domzIcQHy06o5kDKYWONse62FllpHzfPgq+/RaAyAXzUdg4n+RwWhFlehP+nmo6Rdo+1M9hCEKNPnvixSVKXZGPD3/MvG3znPKeYnPZrhkzZvDJJ5/w2WefcezYMaZPn05ycjJTp04FpPjWiRMnVu8/depUkpKSmDFjBseOHeOzzz7j008/5cknn6zeZ9q0aaxdu5ZXX32V48eP8+qrr/L333/z+OOP2/rjyNSHhx+0kGKiOOt+FzHAgHhJaWCHjQssXIj3wIH433Lz/7N33uFRVF0cfmdLNr13EkhC7yUghBoLRUGsWFBQ9MOKgigiIIhgV7Cg2BUFFGwoKiBFOoTewUBCQjpJSG+bLfP9MdlAIIGU3exmM+/z7JPNZubes3szO+fee87vAJD+8myMWjPLwSRslX7aqVzXfxmF5JbocHFQ0i3E/hIWq2C6kWae4PE2dxPmHkZWaRYL99tH2IEgCBd1oRv5OrQootioDq2xvJz0l2eDKOJx5524NMJiUOWkMsIHhcJ+1DeqxU53LM/mneWzI5/x65lf2Zlqe7rzFndo7733Xj744APmzZtHjx492LZtG2vWrKFVq1YApKenV9GkDQ8PZ82aNWzZsoUePXowf/58Pvroo0rJLoD+/fuzYsUKvv32W7p168aSJUtYuXIlffv2tfTbkbkadnoRm+gXLq1enkgvIL+kcSVmAl58EaWfL+UJCWR/+ql5GzdNQOzUoTXpz/YJ90attPPiiK5+UslUQJO8h7n95wLw65lf2ZNumzrddSWqEQudNBrZZ6DoPKgcIaSP5bv79FPKz55F6etLwPQXLd4fXLwO7TrcwISpOE3aIamkuB1gFI28susVdEYdg1oMYkTYCGubdAWN8u3+1FNPkZiYiFar5cCBAwy+JFt7yZIlbNmypcrxQ4YM4eDBg2i1WhISEipXcy/l7rvv5r///qO8vJxTp05x5513WvptyFyLSx1aG46zqS8B7o4EOImIIsQkNK64u9LDg8CXZwNw4auvKYuNNU/DuYmQdw4UKmjZzzxt2hi77bFu/NUw3UwTthEZEMm97e8FYO6uuZTqS61omHkwjeORlHy0BisbYy5MuySh14Ha0aJdlcXGcuHLrwAInD0bpYfldy3KdAb2J0qhWs3iOvQIkUqIi0Y4V71EaVNjxX8rOJx1GGeVM7P7zbZJjWs7X66QaVRC+4LSAQrT4EK8ta2xCO0qSqZao/ym+/BhuA0dCno96bNeRtTrG96oaXW2RSRo3Breno2hvyQRxS4rE1XHZTslU3pNIcA5gJSiFBYfXmxFw8xDS29nWng6oTOIJBTa3k21Xph2tcIsu0siVnx3oNfjNvQm3IcPu/ZJZuBQUh5avRE/Nw1t/F0bpU+rY0c7lmlFaXx48EMApkROIcg1yMoWVY/s0MqYD7WT5NTCxRUHO6Oth+TQ7ozLtkr/AbNfRuHuTtnx4+R8933DG7TzcrfH0woo1OrxcFLT0Z4TUS6l1QCpdOqFOChIw9XBlTlRcwD4/uT3nMg+YWUDG4YgCJXb1qfz7cChNRovJg9FDLFoVznfL6Xs+HEUbm4EzJ5t0b4uxRRu0L+1j02u7FmESoe2aeeUiKLIvJh5lOhL6Onfs3LHxxaRHVoZ82JHs9LqaOMuIghwJrOIzMKyRu9f7e9PwIvTAMhatIjypAYoSjRyIoo1MN1I+4Z7o7T3RBQTTp6SYgVUju/gkMHcHH4zRtHInF1z0BmbdpnRqArFkTP24NCePwalueDgBsG9LNZNeVISWR99BEDA9BdR+/tbrK/L2dXcwn7gYoLm+WNQ3Pg7eubir7N/sTN1J2qFmrn956IQbNdttF3LZJomJscocbu08mBnuKihY6C0NW+NsAMAj7vuwrlfP8SyMtJnz6m/LmD2GSjKAKUGQq4zr5E2QrOLnzVRzcTypetewlPjyenc03x7/FsrGWYeTCu0ycVQUNq0nfPKMWrVH5SWqXUkiqL0XVFWhnO/fnhckmRtaYq0eo5USKw1m7AfqEjQ7CQ9b6LyXRdKL/DOvncAeKL7E0R4RFjZoqsjO7Qy5iW4F6hdoOQCZJ60tjUWIapCvstaDq0gCATNn4fg6EjJnj3k/fJL/RpKrLiRNkIiijXQ6g3sS6yIn23TjG6kcNGhPbu1MkHT29Gb6ddNB+CzI59xNu+staxrMMGeToT5OCMisC+x8XShLUIj7JLk/fILJXv2IDg6St8djbjtvy8xB71RJNTbiVBvy2rd2hxNfMfy7b1vk6fNo51Xu0pda1tGdmhlzIvKAVpFSc+b6EV8LUx6tLus5NACOISG4jdlMgCZb7+D7vz5ujdSeSO1bNyetTiSnE+ZzoivqwNtm0siiomWUaBQQ0EK5Fx0XEeGj2RQi0HojDrm7JqDwdh0ZQIqr8OmrEdr0F3MgrdQ/KzufCaZ77wLgN/kyTiEhlqkn5rYVZFv0D+imU0qoUk7tFuTt7I2cS0KQcG8/vNQK2y/LLHs0MqYnyZ8EdeG3q28UCkEknJKSM4psZod3uPG4ditG8aiIjJenVe30AOj0e71Z02Je/0imlEiigkHF2nlHaokaAqCwJyoOTirnDmSdYQVsSusZGDD6W+lQidmJfUglBeBs0+lfrA5EUWRjPnzMBYW4titG97jx5m9j2tRGT/bppmF/cAlCZpnpBLjTYTC8kLmxUgVKsd3Gk9nX/P/b1oC2aGVMT8mB+ncTjCYQVrKxnDVqOge6glYL+wAQFAqCXptPqjVFP37L4Xr1tX+5MwTUJojhYe0sFwiijUxJYQNaG7hBiZqmFgGugQyNXIqAB8e/JDUotTGtsws9A2XHNrTmUVkFZq5el5jYZpshA0Chflvx4X//EPRxk2gUhE0fz6CUmn2Pq5GbnE5J9OlEqnNoqDC5VyaoNmE4mgXHlhIZkkmLd1a8lSPp6xtTq2RHVoZ8xPYDRw9QFsA6UesbY1FMCUZmZwma+HYrh2+jz8OQMb819Dn1jKe0OTktOwHStvfSqorJeV6DiXlATCgOSWiXIoplCRh2xUJmmPaj6GXfy9K9aXM213H1X0bwdvFgRbOkt3Wvg7rjQXjZ/W5uWTMfw0A38cew7F9O7P3cS32JFxAFKGtvyv+bvYXp18rTGoHTUTKcm/6Xn45LeVlzO0/FyeVk5Utqj2yQytjfhTKJncR15WoSof2gtWdAd/HJqJp2wZDTg6Zb71Vu5PObpF+Wlj30lrsTZASUVp4OhHq3XS+kM1Ki0hQO1eboKkQFLza/1UcFA7sStvF6vjVVjKyYZh0oa25U1JvdKWQXFGO2AJx7JlvvYXhwgU0bdvg88TjZm+/NuyMa6YqI5diGtuzth+CV6ovZe7uuQDc0+4e+gRavgyzOZEdWhnLYOdxtL1aeuGgUpBZqCU+q9iqtggODgS99hoIAvl/rKZo2zU+80sTUew0IcwUtzegTTOMnzWhcpCkoKDaiWWYR1jlduI7+94hu7TprXK2MxU6aYortEkxYCgH9xbg09qsTRdt3Ur+H6tBoSDotddQODiYtf3aYlo5j2quuyQg7YIpVJCfBDkJ1rbmqnxy6BOSC5MJcA7gucjnrG1OnZEdWhnLYHJok2JA30Tj266Co1pJ71ZeAOy2gZupU/fueI8fD0D6K3MxFBXVfHDqASkRxclLCg+xQ0wJYc02ftbENSaWD3V+iI7eHSkoL+CNPW80omHmobW7iEohkJxTatUEzXpxabiBGSddhqIi0ue+CoD3+PE4de9utrbrwvmCMuKzihGEi4UwmiUaVwipWOm04R3LY1nHWHpqKQBzoubg6tD0lGFkh1bGMvh1ABc/0JdCyn5rW2MR+l8SdmAL+E2ZjDo0FH16OpnvvVfzgWcrvlTDB1skEcXaNPtElEsxrcAnVp+gqVKomDdgHipBxYZzG9hwbkMjG9gwHJXQLcQDaIJxtCbnxsy7JJkLFqBPT0cdGorf5GfN2nZdMIWBdAn2wMPZ/uL060Rl2IFtOrQ6gyTjZxSNjIwYyeCQpql8Y393MxnbQBDsPuzAtI22++wFjEbrJ9UonJwImj8fgLwVKynes7f6Ay10I7UVdp+VElHaBTTjRBQTgV3B0RPKCyHtULWHdPDuUCma/nrM6+SV5TWefWbAVOjEFK/ZJCjLvzge4YPM1mzx3r3k/ShJsQXNn4/CyXrx46ZdkmYdP2siIlr6mbDVJitofnXsK+Ly4qTiK32mW9uceiM7tDKWw84d2u4hHrhqVOSV6CpXBK2NS7++eN57LwDps2djLC2tekB5MSRXOLqmL1k74+KNtJmHG4CUoGlymBK21HiYqazlhbKLpS6bClGXFDqxdoJmrTm3C0QjeLcGjxCzNGksKyN99mwAPO+5B5d+fc3Sbn0QRbFy56rZ75JARYKmqYLmCWtbU4UzuWf44tgXAMy4bgZejl5Wtqj+yA6tjOUwObQp+yRHys5QKRVcF27dMrjV4T/tBVSBgeiSksj68KOqf0zaDUYduIeAt23X5a4vuysTwmSHFqgq31UDDkoH5g2Yh4DAn2f/ZFtK05mE9gj1RKNSkF2k5UzmVWLHbQnT1rMZVUayFi1Cdy4JVUAA/tNeMFu79SE5p5TUvFJUCqHyO7JZc2mCpg2FHeiNembvnI3eqOf60OsZHjbc2iY1CNmhlbEcXuHg0VJyoM7ttrY1FsFW9GgvRenqStCrcwHI+f57So9cogV86Y3UDrP/0/NLOZtdjEJAvpGaMDm0SXskqaga6O7XnXGdpEpS83bPo6i8aTiHGpWCPmGmsAPbuQ6vipn1Z0uPHiXn2yUABL7yCko3N7O0W19MqhM9W3ri7KCyqi02g2nyYkOJYd+f/J4TF07g5uDGy/1ebvKKMLJDK2M5BAEiTGEHW6xqiqUwbaftTchBZ7Cd2CjXIUNwH30rGI2kzZqFsbxc+oOdx8+a4ii7hnji4dTME1FM+LYF10AwaC+Gm9TApJ6TCHUL5XzJeRYeWNhIBjYcU1lVW0nQvCpFWRe3ncMa7tAay8tJmzkTjEbcb70Vtxuub3CbDeViuIG8S1JJlQRNnXVtARLyE/jk0CcAvNjnRfyd/a1sUcORHVoZyxJR8eVqEvK3MzoGuuPlrKa43MCR5Dxrm1OFgBkzUPr4UB4XT/ann0JJDqQflf5ogcpEtsAuk1yXHLd3EUGo9eqQk8qJV/tLkk8/n/6ZvelXd4BtBVM1uJizF9Db0MSyWhIrVmcDuoJLw/9Psz/9lPK4eJQ+PgTMnNHg9hqKKIqVUobydXgJAV3A2Qd0xVZX/jEYDczZOYdyYzkDggdwW+vbrGqPuZAdWhnLYnKcMo5BcRPZDqwDCoVQmXxka1nWKi8vAiuSRC58+RVlm1cCIvi2B/cg6xpnAURRrNzqlONnL6MOCZp9AvtwT7t7AHhl1yuU6Gxf37VLCw/cHFUUluk5nmYbCZo1YsZwg7KTJ7nwxZcABM6Zg8rL+gk9ZzKLyC4qx1GtoEdLT2ubYzsoFJdch9YNO1gRu4LDWYdxUbvwStQrTT7UwITs0MpYFld/aWYKVr+ILYXJedoRl2VlS67EfcRw3IYNA72etAXfIhqx23K38VnFnC/Q4qBSENnK+jd2m8J0I009CGXXdviei3yOQJdAUopSWHRokYWNazhKhUC/CNuLZ68WMyWEiTodabNeBoMBtxEjcB8+zAzGNRxTHHPvVt5oVEorW2Nj2IAebXJhMh8e/BCAqZFTCXK1n8UN2aGVsTwmeSg7DTsYWOHQHkrKo0h7pXi9tQmcMxulpyfatEKyT7rabfysaZuzdysvHNXyjbQKni2lJE3RcLHs8VVwdXBlbtRcAJafWs7B8wctbGDDMW1v77KxnZIq5CVBbgIISmgZ1aCmLnz1FdpTp1B6ehL48iwzGdhwTA7twLbyLskVmCYxVlL+MYpG5u6aS6m+lOsCr+Pudnc3ug2WRHZoZSyPyaGN3wJNRSeyDrT0caaltzN6o8jeBNu7map8fQmY+hQA2SfdKCsPtLJFlsEU8iGHG9RAHbOsB7QYwB1t7kBEZM6uOZTqa1ZIsAX6V4z7vsQcynQGK1tTA6Zwgxa9wNG93s1oz5wha/GnAATMmoXK1zb+53UGIzFnc4CLE32ZS7Cy8s8vp39hb8ZenFROzI2ai0KwLxfQvt6NjG3SMgoUasivWJ2wQyrDDs7YnkML4N5WgWuLUjAKpL/6JqLO+lm25sRgFNl9VhZyvyqmsIM6bHe+0OcF/J39OVdwjo8PfWwhw8xDW39X/Nw0aPVGDiXlWduc6onfLP1sQFETUa8nbeYs0OlwveEG3EeNNI9tZuBIsrRL5eWsplNQ/R12u8WKyj/pRemVyiXP9nyWUPfQRu2/MbCoQ5ubm8u4cePw8PDAw8ODcePGkZeXV+PxOp2O6dOn07VrV1xcXAgODmb8+PGkpaVVOS46OhpBEKo87rvvPku+FZmGoHGFkD7ScxsSlTYnptUIW9XBFBK2Edg7H4Wzg5RI8vU31jbJrJxMKyC/VIebRkW3Fh7WNsc2MYWaZJ6AosxaneLu4M4rUa8AsPTkUg5nHraQcQ1HEASb1IWuxGi8GHYVUX9prQvffkvZsWMo3N0JfMW2Enq2n6mo0tfGF4XCduyyKayg/COKIq/seoViXTE9/Xtyf4f7G63vxsSiDu3YsWM5fPgw69atY926dRw+fJhx48bVeHxJSQkHDx5k9uzZHDx4kN9++43Tp08zevToK46dOHEi6enplY/PP//ckm9FpqHYeRxtVGsfBAFizxeSWVhmbXOqIoqQsBW1k5HAJ6UvsuxPPkF75oyVDTMfJnWDvhHeqJTyxlO1uPhCYFfpeR0mloNDBjO69WhERGbvnE2Z3sb+vy/B5NDa5MQy8wSUZEslUE0T/DqijYsj+yMpSS/gpZdQB9iWdmhl/KwcblAzVZR/GmdH75czv7A7fTcapYb5A+ajVNhnjoHFSnicOnWKdevWERMTQ9++Uk3pL7/8kqioKGJjY2nfvv0V53h4eLBhw4Yqry1atIjrrruOpKQkWrZsWfm6s7MzgYG1iwXUarVotdrK3wsKpCxfnU6HrpG2Xk39NFZ/tobQaiAqQEzYir5cC00wdudqY+jmINApyI0TaYVsi83ktu42lDmafQZ1YTqiUoPT2Cdx3pdAybZtpM6YScjS7xFUTb+Sz44zksJE33Cva15jzflaVIQNQZlxDGPcJgwdb6/1eVN7TmVX2i4SCxJZdHARU3pOsZiNtaW6cewb5gnAkZR8cgpLcHO0neIaijMbUQLGVv0xiALU8f9P1OtJfWkGok6H8+DBOI8aaVP/w4Vleg5VaHH3C/OstW3N7nrUeKHy64iQdQp9/GbEjpbVgE0rTuO9fe8B8HT3pwl2CrbIZ22pcaxLexa7k+3evRsPD49KZxagX79+eHh4sGvXrmod2urIz89HEAQ8PT2rvL58+XKWLVtGQEAAN998M6+88gpuNZT7e/PNN3n11VeveH39+vU4OzvX/k2Zgcsd9uaCIOq5WeGIujSXnb9+Rr5zmLVNqjc1jWEgCk6g4KetR1CnHmpkq2omPGsj3YBspwh2bdyCauAAWu3di/b4cWJmziQ3OtraJjYIvRH2nFUCAvrUE6xZc6JW5zXHa9GvwJn+gPbUOtYr/q5T+eMRihEsYxlLTy3FKdmJUJVtxOBdPo5+jkqyyuCTXzbSzdt2klCj4n7BHzhR6sfZNWvqfL7X5i34HT+OwdGR4wMHcHjtWvMb2QCO5wgYjEp8NSJHd2/maB3Pb07XYxdCac0pkrcu42iC5SZdoiiypHgJJfoSWipb4hnvyZqzdf/fqwvmHseSktrrYFvMoc3IyMDf/8rtEH9/fzIyMmrVRllZGS+99BJjx47F3f1igPkDDzxAeHg4gYGBHD9+nBkzZnDkyJEaP8gZM2YwderUyt8LCgoIDQ1l2LBhVdq1JDqdjg0bNjB06FDUattZNWhMlMUr4cw/DGqhxxh1i7XNqTPXGkOP+AtsWnKAJK0TN9882GZi25S/rATAO/IObhkofe4Fjk5kzpmD/6Z/iXz8cRxat7amiQ1iT0IOuj378XV14NG7hl7zc2/W16LuesQFH+Kky+WWvm3Bt12tT72FW8jZlcOaxDWsV6znh+E/oFFqLGjs1alpHPcaTrF8bzJlHmHccktHq9lXBX0ZqmOPAdBh5FN08OtQp9O1cXEkvywVSQl6eRbtb7O9yk77//4PSGJot1BuuaVTrc9rjtejcFoBP68nzJhIyC2Wuxf+Gvcr8Xvj0Sg1fHTzR7R0b3ntk+qJpcbRtKNeG+rs0M6dO7fa1c5L2bdvH0C1NxZRFGt1o9fpdNx3330YjUYWL15c5W8TJ06sfN6lSxfatm1L7969OXjwIL169bqiLY1Gg0Zz5RevWq1u9AvIGn3aDK1vgDP/oEzcjnLw89a2pt7UNIb9WvvhoFKQUaAlOb+c1n6uVrDuMowGOLcDAGWbG1BW2O095m6KN22keOs2Ml+eTdiKH5ts6MHuhFxAittzcHCo9XnN8lpUq6FVFJzdgvrcdgjqXKfTZ/abyd7ze0koSOCLE18wNXLqtU+yMJeP45D2/izfm8yuszm2M77Ju0BfBm5BqIO61GllXNTrSZk9R1I1GDIE77vuspnJ8qXsrpDrGtzOv16fe7O6HlsPAUGJkJuAujhd0ok2M2lFabx/8H1AUjVo7dM4ixbmHse6tFXnQMZJkyZx6tSpqz66dOlCYGAg58+fv+L8rKwsAgICrtqHTqfjnnvuISEhgQ0bNlxzFbVXr16o1WrO2FGSi11iSgxL2g06200sqS+OaiV9wqQKVTvO2EhSSvphKMsHjTsE96x8WRAEgubNQ+HuTtnx41z46mvr2dhATJnVg9r6WdmSJkJllvXmOp/qofGoVD347sR3Nql60K+1D0qFQEJ2Mck5NlK29+wlcl11dEYvfP0NZcePS6oG8+bZpDObkV9GXGYRgkBlKXCZq+DoDi0ipecWUP4RRZG5u+ZSoi+hp39PHuj4gNn7sEXq7ND6+vrSoUOHqz4cHR2JiooiPz+fvXv3Vp67Z88e8vPz6d+/f43tm5zZM2fOsHHjRnx8rq0peeLECXQ6HUFBNpSII3Mlfu3BNVBaqUjeY21rLMLFMrg24tDG/yv9DB8MyqorsOqAAAJnzQQg65NPKIs93djWNZjc4nKOpeYDMEiuTFQ7Wlc4tIk7wFD3BI7o0GhGtx6NUTTy8s6Xba7ggrujmh6hnoAtXYf1058tO32a7I8l/d+AmTNsTtXAhOlz7tbCAw/nZrLK2lBMhU4soPxzqarBvP7z7FbV4HIslmresWNHRowYwcSJE4mJiSEmJoaJEycyatSoKglhHTp0YNWqVQDo9Xruvvtu9u/fz/LlyzEYDGRkZJCRkUF5eTkA8fHxzJs3j/3795OYmMiaNWsYM2YMPXv2ZMCAAZZ6OzLmQBDsXr7LJFcTE38BvcFoZWu4eCNtfUO1f3YfPRrXG24AnY60GS81uYILO+OzEUVoH+CGv7ujtc1pGgR0BWdfKC+SSnDWg+nXTa8suPDRwY/MbGDDMU1ubGKnpCQH0o9Iz+vg0Io6HekzZyHqdLhGR+Nhg3GzJuRyt/XgUj1ao/nuFSmFKZWqBs/2fJYwjzCztW3rWFQ7afny5XTt2pVhw4YxbNgwunXrxtKlS6scExsbS36+tMKSkpLC6tWrSUlJoUePHgQFBVU+du2S6o87ODiwadMmhg8fTvv27Xn22WcZNmwYGzduRKlsHrOQJo0FZ6W2QOdgDzyc1BRq9RytWDm0GtrCiyvhNTi0giAQ9OpclB4eaE+eIvuLLxrRwIaz/bQp3EC+kdYaheLidRhf97ADkAouvNpfyqVYfmo5+zP2m8s6s2D6f9gZn43BaGWlg7NbABH8O4Fb7ctOZ3/55cVQg1dftclQA5C2t00rtHLZ6ToQ0gccXCVt4oy6akJUj1E0MnvnbEr0JfTy79VsQg1MWNSh9fb2ZtmyZRQUFFBQUMCyZcuukN8SRZGHH34YgLCwMERRrPYRXSEtFBoaytatW7lw4QJarZa4uDg+/PBDvL29LflWZMyFqVpR2iEozbWuLRZAqbhYrWintVeHEneAUS/VD/cOr/EwlZ8fAXOkDOrsTz+j7OTJxrKwQYiiyPYK/dlB7eT42TrRgDhaEwNbDOSutndVFlwo0dlIvCrQPcQTN42KvBIdx609saxHdbCykyfJXvwpAIGzZ9tsqAHA6fNFZBVqcVQriGzlZW1zmg4qh4tFFkyhYQ3kh1M/sP/8fpxUTrw24LVmE2pgoump28s0bTxaVEgFiZLDZYeYtt2sHr8Xt0n6WcPq7KW433ILbsOGgV5P2kszECtCfGyZs9nFpOWX4aBScF2YPKGtE6Y42tQDUJpX72Ze6P0CQS5BpBSlVNaJtwVUSgVRFRNLq16Honhx0tC6dg6tsbyctOkvgV6P27BhuI8aaUEDG45pUnlduA8aVfNyoBqM6bvZDA5tQn4CHxz8AJCuy1B329CJbkxkh1am8WkmcbQHk3IpKddbzxDTl2QtHFpBEAh8ZQ5KLy+0p0+T9fEnFjau4Ww/Ld1I+4R54eQg30jrhEcI+LQF0QiJ2+vdjKuDK/MGzANgZexKYtJjzGVhgzGFHZgcLquQcxbykkChhlY1J0NfSvaiRWjPnEHp40Pg3FdsNtTAhCl+dpAcblB3TN/NSTGgLap3M3qjnpd3vIzWoCUqKIox7caYycCmhezQyjQ+Joe2nvF7tk5Lb2dCvJzQGUT2JuRYx4jcRMiJB0EJ4YNqdYrKx4fAuXMBuPDVV5Qcsp1qZ9Uhy3U1ENOKYQOvw35B/bi3/b0AzN45m8LywoZaZhYGVvxfHDiXS7HWShNL0+psaF9wcLnm4SUHD3Hh628ACHp1LiobD6Ur1xvZU/EdJ8fP1gPvCPBsBUYdnNtZ72aWnFjC0eyjuKndmDfANqXdGgPZoZVpfMIGSo5WTjzkJFjbGrMjCELlKu1Oa213mpyUkD7g6FHr09yHD8PjttFgNJL20ksY61B2sDEp1xuJOXsBkBPC6o0Z4mhNTI2cSqhbKBnFGby1960Gt2cOwnxsYGJZqTISfc1DjSUlpM14CYxGPG67DbebbrKsbWbgUFIuJeUGfF0d6BBYfel5masgCA0OO4jNieWTw9KO2kt9XyLQpfaJh/aG7NDKND6OHtKKBUD8JuvaYiFMqxXbrZUYVodwg8sJmDULVWAgunNJZL63wMyGmYdDSbkUlxvwcXGgY2DjlK+2Oyonlmch91yDmnJWO/PGwDdQCApWx69m47mNZjKy/giCcEnYgRWuQ6MBEirCOSKufR1mLnwf3bkkVIGBBFToQ9s6pvjk/q19USia56pgg2lzo/Qzru73Qp1Bx6wds9Ab9USHRnNrxK1mNq5pITu0Mtah8iI2T3anrTGwjS+CAP9lFHK+oJGrohn0kFBRfcb0OdcBpbs7wW+8DkDuDz9QtKP+W2GWwuSgDGwr30jrjaO7tIIPZlml7eHfg0e6PALAvN3zyC61vgasKRxlR5wV4mjTDoE2X5rAB/e46qHFMTHkLlsGQNBrr6G8RnVMW2GHrD/bcMIHSxPLC2ekeOs68OmRT4nNjcVT48krUbYfb21pZIdWxjq0qdhOS9gKetvPqK8rXi4OdA/xBGDr6Ua+maYdksrdOnpUKXdbF1z698frwQcBSJ81C0O+laWPLqNSrkuOn20YZoqjNfFU96do79WeXG0ur+56FVG0rgZs/9Y+CIIkLZWR38gTS9NnGj4YriKfZCgoIG2GtCLref99uA5sGgWC8kt1HEnOA+T42Qbh6HFxYlmHsINDmYf4+rhUsnx2v9n4OsljIDu0MtYhsNvFakV2WgZ3SIU2aqM7tKYvxYjoq95Ir4X/81NxCAtDf/48Ga+9bh7bzEBeSXll0Qo5fraBmOJoE7ZKW+QNRK1U88agN1Ar1GxJ2cLvcb83uM2G4OnsQLcWUgx5o8t3mVa9r6E/mzH/NfTp6ahbtSTghRcawTDzsDMuG6MIEX4utPB0srY5TZs6xtEW64qZuX0mRtHI6NajGRY2zILGNR1kh1bGOigUF7fD7TSOdkh7yaHdfjqrccvgNiB+9lIUTk4Ev/M2KJUU/PknBevWmcG4hrMz7gKiCO0CXAmQy902jBaRoHGXipyYyrM2kHZe7Xim5zMAvLX3LVIKU8zSbn2pDDtoTPkubREk75WeX0V/tmDNGgr+/BOUSlq88w4Kl2srIdgKW2OlzzO6ne0WfWgymL6rz26p1cTy3X3vklKUQpBLEC9d95JlbWtCyA6tjPUwhR3EWT+BxBJ0D/HEw0lNQZmeIyl5jdNpWT6k7JOe16EyUU04deuG7+OPAZDxylx05zMb3GZDkcMNzIhSBWEVsm5miKM1Mb7TeHr596JEX8KsHbMwmGH1t75cWujE2FhlcBO2SlJMXmFSpb5q0J0/T/qrkoav7+OP49S9e+PYZgZEUazceYpuL1+HDaZFLyn0oCwfUg9e9dDNSZv59cyvCAi8PvB13BxkdQkTskMrYz1MDlfGMSg8b11bLIBScTHL2rSaYXEStoNoAJ824NXKLE36Pvkkjp06YcjPJ33mTERjI642X4ZU7takPyuHG5gF0wqiGRM0lQolrw18DWeVMwczD/L9ye/N1nZd6dXSC2cHJdlF5fyX0UgauWc2SD/bDpOkmS5DNBpJnzETY34+jl274vvkE41jl5n4L6OQjIIyHNUKrgu3ba3cJoFCeYk+e83X4YXSC8zdPReAhzo/RJ/APpa3rQkhO7Qy1sPVD4J6SM/NVMva1ohuL23HNVocrZnCDS5FUKsJfu9dBEdHinfuJHfZcrO1XVcSsotJzSvFQamgb7iP1eywK9oOlX4m7ZZWiMxEqFso06+bDsBHhz7i1IVTZmu7LjioFPSLkP5XGqVqmChe3HVqM7TaQ3KXLad41y4ER0eC334bQa22vF1mZEvFBD0qwgdHtVylzyxcI45WFEXm7p5LTlkObTzbMKnnpEY0rmkgO7Qy1sXOww4GV6wiHk3N50KR1vIdWsChBdBERBAw/UUAMt97j7LTp83afm0xrc72lsvdmg+vMPBtJ63sm7l63x1t7uDGljeiN+p5aftLlOpLzdp+bTFdh40yscyKhfxkUGokrd/L0MbFkblA0nf2f3EamojqQxJsma2npdAj04RdxgyYvrNT9lU7sVwVt4otyVtQKVS8NegtNEpN49rXBJAdWhnrUpkY9q9ZsqxtDX93RzoFuSOKjZBlnXMWchOkuvFhtSt3Wxc877sPlyGDEcvLSZv2Isbyxpdbk+NnLUTbiizpuA1mbVYQBF6JegU/Jz/O5p9l4f6FZm2/tpgcr32JORSW6Szb2Zn10s+wgeDgXOVPYnk5aS9OR9RqcRk0CK/777esLRagsEzH/sRcQI6fNSueLcGnrTSxTNhW5U9JBUm8vfdtAJ7p+Qztvdtbw0KbR3ZoZaxLSJ+KLOscSDtsbWssgkntYIul42hNq7OhfUHjavbmBUEg+PXXUXp7o42NJev9D8zex9XQ6g3sipfL3VoE007JmQ3SlrkZ8XL04rUBrwGwInYF21K2XeMM8xPm60KErws6g8jOuAuW7cw0KWh7ZbhB1qKPKTt5EqWHB0Gvv9YkhfB3xl1AbxQJ93WhlU/TUWVoElRTNUxn1DF923RK9CVEBkTyUKeHrGSc7aOytgG2jMFgQKczz2xep9OhUqkoKyvDYLC/lci6oFarUSortouVaogYAqf+lOS7QiKta5wFGNLOj0+3xLPtdBZGo2i5ylaVdeMbrm5QEypfX4Jee42Up54i59tvcR08CJeoKIv1dyl7E3IoKTfg76ahc3DTqKTUZGjVH9QuUHQeMo5CkHkz7vu36M+DHR9k2allzN45m99G/4aPU+PGQEe39+dsdgJbYjMZ0cVC9e61hXBut/S8bVVt0OKYGC589RUAgfPnofZvmtv1prANk862jBlpfQPs+Uy6F4oiCAKfHv6U4xeO4+bgxluD3kLZAG1xe0d2aKtBFEUyMjLIy8sza5uBgYEkJyc3yVm5ufH09CQwMFD6LNrcJDm0cRthyIvWNs3sRLbywlWj4kJxOSfSCuga4mH+TvRaScMQ6lXuti643XA9nvfeS97KlaS9NIOIP35H6elp0T4B/v1Pitu7vr2/fA2ZG5VGyrKO/VtapTWzQwswJXIKMekxxOXFMWfXHD6+4eNGHcfrO/jxzc4ENsdmIoqiZfpO2FYh1xUOPq0rX9bn5pL24nQQRTzHjMF9WNMUwhdFka2x0nU4RA43MD+tBkghY3lJkHOWfbocvjomTYLmRM0h0MVCEzE7QXZoq8HkzPr7++Ps7GyWLz6j0UhRURGurq4oFM030kMURUpKSsjMlL4Ug4KCoHWFA5ayTxJ4d/KyooXmR61UMKCND/+cOM/W05mWcWjP7ZKqrrkGQKDl9SwDpr9IyZ49lCcmkv7KXFp88L7FnZPNJoe2Q9Nc2bJ52t500aEdbP6KVRqlhrcGvcX9f9/PtpRt/BT7E/d2uNfs/dTEdeHeOKmVnC/QcjK9gM7BFrgOTfGzl4QbiKJIxpw56DMzcQgPJ2BG0xXCP5NZRFp+GRqVgqgIWWXE7GhcoWU/SNxO/n9/MiPtL0REbmt9GyPCRljbOptHdmgvw2AwVDqzPj7mu2CNRiPl5eU4Ojo2a4cWwMlJKpOYmZmJv78/Ss9Q8G0P2bFwdit0vt26BlqAIe38KxzaLCbd0Nb8HZz+R/rZdqhUhc3CKJydCX73XRLvv5/Cf/4h75df8BozxmL9nc0qIvFCCWqlUCmUL2NmTBJTKXstNrFs792eKb2m8O7+d3lv/3v0DuxNa8/W1z7RDGhUSga08WXjqfNsic0yv0MrinDmSrmuvJ9+pnDDRlCrabHgPRTOzjU0YPtsqVid7SfLdVmOdsMRE7czL24l5yki1C2UGX1nWNuqJkHz9qyqwRQz69yEv3SaAqbPtzJG2d7lu9pJTtjBpDzyS82cZS2KcLqiLG27xpvFO3Xtgv+UyQCcf/0NtPHxFuvLFG5wXbg3rhp5Hm4RPEPBvxOIRovqQj/Y6UGigqIoM5Qxbds0yvRlFuvrcq7vIG2Tm1b7zUrWf1CQAirHSrkubXw85998EwD/557DsVMn8/fbiMjxs41AuxH84erCeopQCUreHvQ2Lmo5+a42yA5tDcgxepblis+3TYUGX9wms2dZ2wIhXs608XfFYBTZZW75rgtxF+W6TNVmGgnvRx7BpX9/xLIyUqc+j1FrGa3dzbEX42dlLMilagcWQiEoeGPQG3g7enMm9wzv7X/PYn1djun/52BSLnklZpadM31mFXJdxvJyUp9/AbGsDJf+/fF+uGlnpxdr9exLkOW6LE2S2oE3fKXd4ada3EhXv65WtqjpIDu0MrZBqwHSykZhmrTSYYeYVjXMLt9lCjcIGwiaxq3rLSgUBL/9VqWUV+a75ndOirR69ibkAHCDHD9rWSr1aDeCBUsc+zr58vrA1wFYGbuSTUmbrnGGeQj2dKJDoBtG0QJFFkzxsxXhBlkL30f7338ovbwIeutNhCYearYr/gLlBiMtvZ0J95VXDC2BzqDjxe3TKRWgd2kZjxQ23u6FPdC0rzAZ+0HtdLGqjgVXh6yJyaHdejoL0Zyr0JXhBsPN12YdUPn5EfyWtK2au2wZhf+at9rUjjPZ6AwiYT7ORPiZX19X5hJa9gMHNyjOgvTDFu1qYIuBlZqac3bOIaM4w6L9mTAVWTDrxFJbCEkx0vO2QynaupWcJUsACHr99SYr0XUppvjZ6PZ+8g6mhXj/4PucuHACd5Uzb2ZdQBm3waITS3tDdmjtnOjoaKZMmVL5e1hYGB988IHV7Lkqldud661rh4W4LtwbR7WCjIIyTp8vMk+jZfmQVKF7aSWHFsB18GC8H34YgPSZM9GdP2+2tmV1g0ZEqYbW0dLzRphYTu41mc4+nSkoL2D6tunojXqL93l9+4sTS4PRTBPLs1sluS7vCHQ6F9KmS0oGXg88gNsNltOFbixEUZTjZy3M1uStLD25FIDXBrxGoNJJ0oW28MTSnrCoQ5ubm8u4cePw8PDAw8ODcePGXVPb9eGHH0YQhCqPfv36VTlGq9XyzDPP4Ovri4uLC6NHjyYlJcWC78R+2LdvH4899pjF2tdqtfTo0QNBEDh8+HDdTjYlNJ3bBSU5ZrfN2jiqlZVSN6bVjgYT/y8Y9VLJRO8I87RZT/ymSkkvhrw8qbynGQqIiKJYGT8rhxs0EhYqg1sdaqWadwa/g7PKmYOZB/ny6JcW77NXKy/cHFXkFJdzNCXPPI1WfFZi+I2kvTANQ14emk4d8X9xmnnatzLxWcWk5JbioFQQ1VqW6zI3GcUZzNo5C4AHOj7A9WFDpSILcDGkTOaaWNShHTt2LIcPH2bdunWsW7eOw4cPM27cuGueN2LECNLT0ysfa9asqfL3KVOmsGrVKlasWMGOHTsoKipi1KhRzb4CV23w8/OzqILDiy++SHBwcP1O9g6vyLI22K3agWmVceMpM61gnq5Yzbbi6qwJhYMDwQveQ3B2pmTPHrI//7zBbZ5IKyCzUIuzg5Lrwr3NYKXMNTHtlKTsh2ILl4kFWrq3ZHbUbAA+O/oZ+zP2W7Q/tVLB4LYVagfmCDsQxcrV7OwDBkr270fh7EzIwoUoNJqGt28DmCbgfSO8cXaQVUbMid6oZ/q26eRr8+no3ZGpkVOlP5gWeEwhZTLXxGL/madOnWLdunXExMTQt29fAL788kuioqKIjY2lffv2NZ6r0WgIDKy+IkZ+fj5ff/01S5cu5aabpC/eZcuWERoaysaNGxk+/Mobu1arRXtJ9nVBQQEgSUZdXtpWp9MhiiJGoxFjReyKKIqU6hrmLIuiSGm5AaVWV6f4Iye1stbHFxcX89RTT7Fq1Src3Nx4/vnnK/s2vZeIiAgmT57M5MmS3JJSqWTx4sX89ddfbN68mVatWvHVV1/h5+fHY489xr59++jWrRvff/89rVtfXS9y7dq1rF+/np9//pm1a9dW+Qwvx2g0IooiOp3uYhlcQNFmOMrMkxhP/YWh4x21et+Nhel/pSHlkKPbSqsbB87lkpFXjI+LQ/0NEo2ozqxHAPQRNyKaqUxzQ1CEhOA3ayaZs14m++NPcOjeHefrrqt3extOpAPQP8IbhWhEp2t4PJk5xtGucfJD5d8FIfM4+tP/IHaxnL6wieGhw9kZvpO/Ev5i+rbp/Hjzj3g5Xl0HtyHjOKiNN38fS2fzf+d5Jjq8XjZXknkKdUEqRdmuZG+SVtP8XpmD0KKF3fyPmcJ+BrXxMft7au7X46dHP+Vg5kGcVc682f9NBKOAzqiDsGhUCAjph9HlJIFbkLVNvSqWGse6tGcxh3b37t14eHhUOrMA/fr1w8PDg127dl3Vod2yZQv+/v54enoyZMgQXn/9dfwrguoPHDiATqdj2CWlA4ODg+nSpQu7du2q1qF98803efXVV694ff369VesVqpUKgIDAykqKqK8XJJ1KS03ELUwpm4fgJnYPbUfTg61E7B+/vnn+ffff1m6dCn+/v7Mnz+fAwcO0LFjx0on3mg0UlZWVvk7wGuvvcZrr73Gq6++yty5cxk7dixhYWE8++yzhISE8Mwzz/Dkk0/yyy+/1Nh3ZmYmjz32GMuWLat0YouLi6v0cynl5eWUlpaybds29PqLcXNexe4MBgyx/7D2r9WICttbDdiwoWFbsSEuSlKKBT76eRN9/esfw+dVHM/gkmx0CifWnshFPLnm2ic1BioVAb0j8dh/gKQpz3Fu8rMY3OqnvvD7MSUg4FOeccVOTUNp6DjaMx2FcNpxnPRtSzmY1DgZ7T3FnsQoYsgszeSJ1U8wzmUcCuHam4j1GUddOYCKY6kFrPh9De4NmFe2Of837csUpOz0AFEkv08fTgOY+f/VWpTqIeasdB2SfoI1a05YpJ/meD3G6+JZUrwEgJEOIzm+4zjHOV7590HOEXiXxHN81fsk+UZbx8g6Yu5xLCkpqfWxFvMWMjIyKp3QS/H39ycjo+Zs1ptvvpkxY8bQqlUrEhISmD17NjfccAMHDhxAo9GQkZGBg4MDXl5VZ+8BAQE1tjtjxgymTp1a+XtBQQGhoaEMGzYMd3f3KseWlZWRnJyMq6srjo6OAKjKLZ+oUBNu7m612uIpKipi2bJlLFmyhNtuuw2QVq5btmyJg4ND5ftUKBQ4OjpWed8TJkzgoYekbOOZM2cyYMAAZs+ezR13SCukU6ZM4dFHH73iszIhiiL33XcfTzzxBEOGDCExMREAFxeXGs8pKyvDycmJwYMHV37OUmNGxA8/RV2cyS2d3BEbWVf1auh0OjZs2MDQoUNRq9X1bifeMZ6PNseT5RDELbf0qHc7ii1vwmlQth/GzSNH17sdS2C8/npSxo6FuHi6bNhA8OefIyjrVlnoQnE5STFbAJh01/UEujte/YRaYq5xtGeEJC9Y+ichZf8ROGI4KBqnKlSXvC489M9DnNGfIaNVBv/r8r8aj23oOP6UEcOx1ALULbtzS68W9bZZsfQzUmM8EUtFHNq0pucnH6OoqIZoD/x9LAPDvqNE+Low4a4BZm+/uV6POWU5fLDmA6m0bcRtzOh3ZTUwhfsp2Pom3Z3S6XLLLVawsvZYahxrWhSrjjo7tHPnzq12tfNS9u3bB1RfnEAUxatuod9778Xa3l26dKF37960atWKv//+mzvvvLPG867WrkajQVNNLJNarb7igzcYDAiCgEKhqCxR66JRc3Jew2IUjUYjhQWFuLm71an0bW1DDhISEigvL2fAgAGV7fv6+tK+ffvK92Pi8t+7d+9e+XtQUFC1r5WVlVFUVFStg/rRRx9RWFjIzJkzq3xulz6/HIVCgSAI1Y4B7W+Gg9+hil8P7YdWe741qdbmOjC8axAfbY5nR9wFDCjqX0IyXpoJK9qPQGFrNwK1mpAPPyRhzD2U7tlL/ldf4zfp6To1sTP+PKIInYLcCfUxv75uQ8fRrgnrDxoPhNIc1JnHILRPo3Tbya8TM/vOZM6uOXx27DMiAyO5LujqISv1HcfrOwRwLLWA7XE53Nc3rH4Gl+SQ/c8xijPcEDQaQj74AE0Nk/imyqZYqRDMsM6BFr1emtP1aDAamLV7Ftll2UR4RDCz38zq33uHW2DrmygStqLAAGrzTOotibnHsS5t1TkpbNKkSZw6deqqjy5duhAYGMj5aqR7srKyCAgIqHV/QUFBtGrVijNnzgAQGBhIeXk5ubm5VY7LzMysU7t1QRAEnB1UDX44OSjrfE5t42cbomt66T+Mqb/qXqspHvbff/8lJiYGjUaDSqWiTZs2APTu3bty5bdOtK+YicautcuqYZ2C3Gnh6USpzsDO+lYNK0iDjKOAUKVuvC2had2aoLmvAJD9yScU795dp/NldQMrolRBmxul57F/N2rXd7S9g9ta34ZRNDJ9+3SyS81cWa8Ck3zXtjNZ6Az1i80uXvUpWUclbeTA2S+jqfjusxe0ekNl/Oywzpa5vzZHPj3yKXvS9+CkcmLBkAU4q2tI1A7sCm7BoCuBczsa18gmSJ0dWl9fXzp06HDVh6OjI1FRUeTn57N3797Kc/fs2UN+fj79+/evdX8XLlwgOTm5cuUwMjIStVpdJU4jPT2d48eP16lde6NNmzao1WpiYi7G+ubm5nL69GmL9/3RRx9x5MgRDh8+zOHDhytjHVeuXMnrr79e9wYjhoDKCfKT4fzxax/fxBAEgZs6Sk7ahpP1VDswafW2iARX29WF9Bg9Go+77wJRJPWFaegyaydXpjcY2Vaheynrz1qJjqOkn6f+bPSJ5ax+s2jj2Ybs0mymb5uOwWh+BZtuIZ54uzhQWKZnf2LutU+4DN35TFIXrgBRwCOqNR533WV2G61NzNkcirR6/Nw09AjxtLY5dsH2lO18flRSgJkTNYc2XleZBAkCtKvIF5Llu66JxWS7OnbsyIgRI5g4cSIxMTHExMQwceJERo0aVSUhrEOHDqxatQqQ4kBfeOEFdu/eTWJiIlu2bOHWW2/F19e3Mp7Tw8ODRx99lOeff55NmzZx6NAhHnzwQbp27VqpetAccXV15dFHH2XatGls2rSJ48eP8/DDD9cpvKG+tGzZki5dulQ+2rVrB0Dr1q0JCQmpe4Nqp4safLFrzWip7TC0k6TisfFUJsb6iLvbkFzXtQh8+WU07dphuHCBtBemIeqvHZN+4FwuBWV6vJzV9Aj1tLyRMlfSZigoHeBCHGTFNmrXppUrJ5UTezP28tnRz8zeh1IhcGPFZOmfE3WrUibqdKROmYyhWI/GU0fgK3PtsnrWhpPS5zK0UwAKhf29v8YmrSiNGTukWNl729/LqIhR1z7pUvkuO9yxNCcW9XaWL19O165dGTZsGMOGDaNbt24sXbq0yjGxsbHk5+cDkoTUsWPHuO2222jXrh0PPfQQ7dq1Y/fu3bhdkiX9/vvvc/vtt3PPPfcwYMAAnJ2d+fPPP6vIPzVH3n33XQYPHszo0aO56aabGDhwIJGRkdY2q360v1n6GWsfmcKX0zfCGzdHFdlFWg4l59XtZF0ZnK0oL9sEHFqFoyMtPvhA0qfdu5esDz+85jn/VmxzDmnnh1K+kVoHR3cwJWX+92ejdx/hGcErUVLIyudHPmdn6k6z9zGiizSx/OdERp3CtjIXLKT00GEUaiMhNzuhaNVEv2evgtEoVu4gDe0khxs0lHJDOS9sfYF8bT6dfTrzYp8Xa3di+BBQOUJeEmT9Z1kjmzgW1UTy9vZm2bJlVz3m0i8RJycn/vnn2svqjo6OLFq0iEWLFjXYRnvC1dWVpUuXVpk0TJtWtVKNSYHAxOVf4mFhYVe8Fh0dXacv++raqDPthgMCpB2S4kXd61mswUZRKxVc396f1UfS2HjqPJGtrq65WYVzO6SYKrcgCOxmOSPNiCYinODXXyP1ualc+PIrHLt2xf0S6b1LEUWRtcdNK0PV61HLNBIdRknhLaf+gsGNX/VqZMRI9p/fzy+nf2H69umsHLWSFq71VyS4nAFtfHFxUJKeX8bRlHy612I3oOCf9eQsWQJAUN88HPo+Km0N2xlHU/M5X6DFxUFJf7k6WIN5b/97HMs+hruDOwuiF+CgrKVWnIMzhA+WrsPT68C/o2UNbcJYfj9aRqY+uPpDSEVmtZ1WSrmpYtWjznG0/1WsWrcd2qRupO4334z3ww8DkP7SDLRnz1Z73Mn0ApJyStCoFES3t9344GZB+1tAUEj15POSrWLCS9e9RBefLuRr83lu83OU6cvM1rajWkl0RdjBulqEHWgTEkifORMA7y463EPKLsYa2xmmcIPoDv5oVM1797OhrE1Yy4///QjAm4PerPukzLQTJ8fRXhXZoZWxXSrDDuwzjja6vR9qpUBcZhEJ2cW1O8lokJJ0ADreZjnjLIT/C8/j3KcPxpISUp55FkPRle97XcXq7JB2frhobK+wRrPC1Q9C+0nP/2tctQMTGqWG969/Hy+NF6dyTjE/Zn7Dd4AuYURnaRdg3fGrhx0YS0pIfXYyxuJinLu0xb9TFjj7QmjfGs9pyqw/IU20h8nhBg3iTO4ZXtklhc5M7DqRwSGD695I2wqHNnkPFFtG9cMekB1aGdvFJN91ditoi6xriwVwd1TTL0LayjOthlyTpBgozgRHD2kbqokhqFS0eH8hKn9/yuPjSZ816wonwhRucHNXOdzAJjCtQP73l9VMCHQJ5N0h76IQFKyOX81PsT+Zre3rO/jjoFSQkF3Mmczqv2dEUSR99hy0Z86g9PUl+O5WCAqkSXcjFZ1oTEyfhUohEN1eVhmpL/nafCZvnkypvpS+gX15ukfdtLgr8QyFoO4gGi8uaMhcgezQytgufu3BKxwM2otJUHaGKdli48nayVlx8g/pZ/uRoGpAvU4rovL1pcWHH4BaTeE//5DzzbeVf4vLLCQuswi1UuCGDvLKkE3QocKhPbcTii9YzYy+QX2Z0msKAG/te4sjWUfM0q6rRsWgtr7Axd2By8n55lsK/v4bVCpCFi5EnbZJ+kMH+w43iGrtg4dT8yh2YG4MRgPTt08nuTCZYJdg3h3yLsqGTH463S79NN0DZK5AdmhlbBdBqFpkwQ65saPktO0/l0NOcfnVDzYa4dRq6XmnphducCnOPXsSMOMlADIXLKA4Zg8Aa49JN9IBbXzlG6mt4NVKEngXjVZXHXm488MMbTUUvVHPiztepNBYaJZ2h3e5GHZwOUXbd5C5YAEAATNn4NxCCQWpoHa5qAJhZ5jCDWR1g/rz8eGP2Zm6E0elIx/e8CFejnVI/K0O03d+wjarTixtGdmhlbFtTHG0p9dJ8aN2RgtPJzoHu2MUYdOpaySHpeyDwnRwcIPW1zeOgRbE6/778bjtNjAaSX3uOcpTUi+GG3SRww1sio6jpZ9WDDsAqSjJ/AHzae3RmqzSLFYUr0Bn1DW43Zs6BqBUCFJC4oWSytfLz50j9fnnwWjE4+678Lr//oufQdubmkQp0rqSVajlQJJUaOKmjrJDWx/+SfyHr459BcCr/V+lg3eHhjfq07piYmmw+nVoq8gOrYxt07IfOHpCyQUpIN4OGVpbtYPKcIObQaWxsFWWRxAEAl+di2OnThhyczn7+BOcTc5CqRBkuS5bw7S1Hr8ZtOZZFa0vLmoX3r/+fVxULpwznOOd/e80OEnM28WBvuHewMUiC8biYlImTcJYUIBT9+4EzpkjFU8wJcfZabjBv/+dRxShW4gHwZ5O1janyXE69zSzd84G4KFOD3FLxC3ma1wOO7gqskMrY9so1RfDDo7/Zl1bLITJod1+JpsyXQ2r0KJ48UusiYcbXIrC0ZGQTz5G6euLGB/HtAM/0jdMKkkqY0P4dwTvCCmePW6jta0h3COc1we8joDAr3G/siJ2RYPbNBVZWFdRZCHtpRloz8Sh9POlxUcfoXBwgOw4SdxeoYK21esoN3Uqww3k1dk6k6/NZ/K/FUlgQX2ZEjnFvB10liqmkrAVSnLM27YdIDu0MrZPl4oa6Sd/B8O1y6Y2NToFudPC04lSnYEtsVnVH5R6EApSpLi9Njc2roEWRh0URMiij9ArVPRPP86jNuAwyVyGIFxckTxlG9udg1sMZpij5FS+vfdtYtJjGtTesIpdgQPncjn34ScUbtgAajUhH32EOqAi09+01Rs2CJw8G9SfLVKs1bM9TpKFGtZZ3iWpC3qjnmlbp5FSlEIL1xa8N/g9VAozyw76tIaArmDUW01Gz5aRHVo7Jzo6milTplT+HhYWxgcffGA1e+pFxBBw8obiLEjcbm1rzI4gCIzsFgTAn0fSqj/o5O/Sz3bDQW1/24D5ER34sIc0cQle/SMFa+0zCbBJ0/FW6eeZ9aC/RgJjIzFQM5CRYSMxiAae3/I85wrO1butQA9Herb0JCrtOCWfL5ZemzMb5549Lx5kcmjttJjCttNZlOuNtPJxpl2Aq7XNaVK8tfctdqfvxknlxAfXf4Cno6dlOjLt0MlhB1cgO7TNjH379vHYY4+Zvd2wsDAEQajyeOmll8zTuFJ98SI+/qt52rQxRneXSvtuPHWeIu1lq9B2Gm5wKeuOZ7CxZR929JJW3NJmzKT0xAkrWyVThRa9wTUQtAVSprUNIAgCL/d9mW5+3SgoL+CZf5+hsLz+Mb53uRXx4oHlCKKI19ixeI0Zc/GPhRlSYiZcDIOyM/4+lg5IxRSEJlSF0Nr8cOoHVsauREDgzUFvmicJrCY63y79PLsFSnMt108TRHZomxl+fn44OztbpO158+aRnp5e+Xj55ZfN17gp7ODUaptZHTInnYPdifBzQas3sv7yEpwZRyHvHKicpHK3dohJ3UB4fBIugwYhlpWR8vQk9Fk1hGDIND4KBXSocOT+sx1xd41Sw4fXf0iAcwAJ+QlM2zYNQz0UUXTnzxP5+es4GnQc8G+PZsrzVQ8wrc626A3uwWaw3LYo0urZWKG0Mrp7HUuzNmN2pO7g7X1vAzAlcgo3trRwSJhvW/DvDEbdxTLoMoDs0NYOUYTy4oY/dCV1P6cO2bvFxcWMHz8eV1dXgoKCWFChnXgpl4ccCILA559/zqhRo3B2dqZjx47s3r2buLg4oqOjcXFxISoqivj4+Gv27+bmRmBgYOXD1dWMW1at+kurQ2X5EP+v+dq1EQRBqFylXX152IFpdbbtUHBwaWTLLE9WoZZ9iVKCw4huLWix4D0cwsPRZ2SQ/NTTGEtLrWyhTCWmONr//rYpGT1fJ18+uuEjHJWO7EzdyYIDV373XQ1jcTHJTz4J2VlkeAXxRp8H2XTmsqSboz9LP+10l+Sf4xmU6YxE+LrQpYW7tc1pEsTlxjFt6zSMopHb29zOhM4TGqdjOeygWuRC6bVBVwJvNGxGrgA863PizLRaOzHTpk1j8+bNrFq1isDAQGbOnMmBAwfo0aPHVc+bP38+CxcuZOHChUyfPp2xY8cSERHBjBkzaNmyJY888giTJk1i7TXiGt9++23mz59PaGgoY8aMYdq0aTg4mClbXaGUMjz3fCqFHbQfYZ52bYjR3YP5YOMZdpzJJqe4XMr0F0U48bt0gJ3eSNefzEAUoXuIByFe0u5B6KeLSbzvfsqOHSN12jRCPvwQQWl/JUabHGGDwMlLimc/u8WmEhQ7+XTitYGv8cLWF1h6cimhbqHc3+H+a54nGo2kvjgd7clTKL29iX3mVUqOFLDueAZ3R4ZIB+UmQnIMIEDXuy36PqzFHxUT6dt6tJDDDWpBTlkOk/6dRJGuiMiASOb0m9N4n1vn22HLG9LiTmmeXSYo1gd5hdZOKCoq4uuvv+a9995j6NChdO3ale+++w6D4dqrKBMmTOCee+6hXbt2TJ8+ncTERB544AGGDx9Ox44dmTx5Mlu2bLlqG5MnT2bFihVs3ryZSZMm8cEHH/DUU0+Z6d1VYAo7iF0D5SVXP7YJEuHnSpcW7uiNImsqYtnIPAk58aDUSAlhdoipOtOILkGVrzmEhRHyyccIajVFGzeR+c471jJP5lJUDtD5Tun50ZXWtaUahocN59mezwJSks6W5C3XPCdzwQKKNm1CcHAg5JOPGRLdHYBtZ7LIL60o2nCsYnU2fLBdhhtkFWrZcUYK77mth/29P3NTbihnyuYppBalEuIawvvR76NWNmJlQ7/24NdRCjuw0yqa9UFeoa0NamdppbQBGI1GCgoLcXdzQ6GowzxCXbt41/j4eMrLy4mKiqp8zdvbm/bt21/z3G7dulU+DwiQtAe7du1a5bWysjIKCgpwd69+K+q5556r0p6Xlxd33303b7/9Nj4+PrV6D9ckpDd4tIT8JCnT2hQcb0eM7h7M8dQCVh9J48F+rS5uKbW5CTRu1jXOAuSVlLM7XirjeHl1MOfISILeepO0518g57vvUYeE4j3uQWuYKXMp3e+D/V/DqT9BWwQa28qG/1/X/5FalMqvZ37lxW0v8s3wb+ji26XaY3N//pmcr78BIOiNN3Du2ZMOokj7ADdizxey5lg69/cJhaM/SSd0u7ex3kaj8vfRNIwidA/1JMzX/sKazIlRNPLyjpc5lHkIN7Ubn9z4ScPL2taHTrfB1lPSPaLHtXcimgPyCm1tEARp27+hD7Vz3c+p5RZGQyrlqNUXZ5amLZPqXjMajbVus1+/fgDExcXV264rEAToUrE6ZKdqB6O6SasjexNySMsrtXt1gzXHMtAbRToEulV7I/UYORK/qVMBOP/mmxT+a3/x002OkD5SkQVdiU1qYQqCwKx+sxjQYgCl+lKe3vQ0KYUpVxxXtHUrGXNfBcD36afxGDWy8vw7e0lJUb8eSIH0w5B9GlSOF6XL7IzfD0sLNrfLq7PXZOH+haxNXItKULEgegERnhHWMcS0oBO/ScotkZEdWnuhTZs2qNVqYmIuiovn5uZy+vRpq9hz6NAhAIKCgq5xZB0xhR2cWQ9lBeZt2wYI9nTiujCpBOfOXdsqqhKp7TJmGOCXA8kA3NUrpMZjfCb+D88xY8BoJPX5Fyg9dryxzJOpDkG4uFJ5tOEVuiyBWqFmwZAFdPDuQE5ZDk9teop87cWbfumRI6RMeQ4MBjxuuw3fSU9XOf/2ni1QCLD/XC4Fe5dLL7a/BRztL1kqMbuYw8l5KAQq9bBlquf7E9/z3cnvAJg3YB5RwVHXOMOC+HcE3/ZgKIfYddazw4aQHVo7wdXVlUcffZRp06axadMmjh8/zsMPP1y38IZ6snv3bt5//30OHz5MQkICP/30E48//jijR4+mZcuW5u0ssCv4tAV9md3GDt1asUqiPPqj9EK74eDoYUWLLEN8VhEHk/JQKgRu61nzypAgCATOmY3LgAGIpaUkP/kk5cnJjWipzBV0u0f6eXaLpM9qg7ioXfjkxk8q5bymbJ5CuaEcbUICyY8/gVhaisugQQS9Nv+KZJ4Ad0cGtPFFiQHliYrdIDsNNzCpqgxo44u/m6OVrbFd1iWs49397wIwpdcUbm1tA6v1pp27E/ZZFr6uyA6tHfHuu+8yePBgRo8ezU033cTAgQOJjIy0eL8ajYaVK1cSHR1Np06dmDNnDhMnTuTHH380f2eCcHGV1k7DDkZ2DcJRYWBQ6SbphR4PWNcgC/HrAWkbOLqd3zVvpIJaTYsPP0DTvj2G7GySHv2frFFrTbwjILQviMaLCVM2iL+zP4tvWoyr2pX95/cz76+pJP1vIoa8PBy7dCHkg/cR1NUn89zVK4QBiuO46HIQnbxtStHBXIiiyO+HUwG4vYesPVsTe9P3MnPHTADu73A/j3R5xMoWVWBS3DizAQrPW9cWG0BOCrMjXF1dWbp0KUuXLq18bdq0aVWOSUxMrPL75bG3YWFhV7wWHR191RjdXr16VQl1sDhd7oStb0mxQyU54OzdeH03At4uDjzZ4ix+WQUUq71xscNiCgajyG8HpRtppTTSNVC6uhL6xRece+ABdElJJE18jFbff4eyhkRFGQvT7R5I3gNHVkL/Z6xtTY2082rH+9e/z9S/n6D/wk3oz4O6ZUtCP/8MhUvNCVDDOwei+H0XAJmtbiGgMbPYG4kTaQWczSpGo1IwrHOAtc2xSU7nnmby5snojDpuankT0/tMtx1ZM7/2EHIdpOyFIz/CwCnWtsiqyCu0Mk0Pv/YQ0BWMeinT2g4Zo9wKwF8MRlTY37xzZ1w2GQVleDqruaGjf63PUwf40/Lrr1D6+qL97z+Sn3oKY1mZBS2VqZHOd0rx3eePwXnbLlPc16cXn2xsRfh5yHOGTc8NQHUN9RUnyhiu3A/Ab7oBjWFmo/P7IWlSeVOnANwc7c9hbyipRak8ufFJinRF9PLvxZuD3kSpsDE97F7jpJ+HltWpEJM9Iju0Mk2TSrWDX6xrhyUoyiTovOTQflnUnxNp9pf89ktFuMFt3YPRqOp2g3Bo1YqWX36BwtWV0v0HSH1uKqJOZwkzZa6Gs/dFbeQjtpkcBiDq9aROexHHw6cxODnw5j1KPs78mSXHl1z9xP/WoDGWcs7oz+J4b8p0tlMZzRwYjGJl/KwcbnAlmSWZ/O+f/5FZkkmER4RUiU5lgzHGne8AtQtcOCPtmDRjZIdWpmlicmgTtkPuOevaYm6OrkQQDSQ4diRODLmyFG4TJ79Uxz8npESiuyND69WGY8eOhH66GEGjoWjzZtJfno1YB1k5GTNhSpQ69otNlcI1IRqNpM+aReE//yCo1YR9vJjbR0kycAsOLGDVmVU1n1xROGKTOprCMgMbT9lXjOKesxfILNTi4aRmSDs/a5tjU+SU5fDY+sdIKUqhhWsLvhj6BR4aG03M1bhJTi3AwaVXP9bOsahDm5uby7hx4/Dw8MDDw4Nx48aRl5d31XMEQaj28e6771YeEx0dfcXf77vvPku+FRlbwysMIqIBEQ5+Z2VjzIgoSltHQEknyVn443AqeoP9OGt/H01HqzfSPsCtQTXjnfv0ocX774NSSf4ff5D59tsN0mOWqQfthoOjJxSmQeJ2a1tTBVEUyZg3j/w/VoNSSYsP3sd1wAAe6fIIE7pMAGDu7rlsPLfxypOLMqWyooDYdQxwMYnRXjAlg93SNQgHlby2ZaKgvIAnNjxBfH48/s7+fDXsKwJcbDy+uGdFwZkTq0BbaF1brIhF/4vHjh3L4cOHWbduHevWrePw4cOMGzfuquekp6dXeXzzzTcIgsBdd91V5biJEydWOe7zzz+35FuRsUUipZsSB5eCwU62nFMPStqzKkfa3PAw3i4OnC/Qsum/TGtbZjZM2rN3R4Y0OLnC7YbrCXr9NQByvvuerAULZKe2MVFpLq4OHbGdUriiKJL59jvkrVgJgkDwO2/jduNFlYLnej3HXW3vwigaeXHbi+xI3VG1geO/gWiA4F5ED+gPwLYz2WQW2ke8dpnOwNqKktNyMYWLlOhKeHrj05zKOYW3ozdfDvuSELfaJa1alZb9wKcN6Iolp7aZYrFsk1OnTrFu3TpiYmLo27cvAF9++SVRUVHExsbWWJI1MLBq+cs//viD66+/noiIqtU4nJ2drzi2JrRaLVqttvL3ggIpJlGn06G7LPZOp9MhiiJGo7FOlbGuhekma2q7uWM0GhFFEZ1Oh1JZzyD71sNQufgjFGeiP7EaseNo8xp5Gab/lcv/Z8yJ4uD3KAFjh1EoNK7c3SuYL7Yn8v2uRG5oZ6YSwlbkbFZxpfbsyC7+ZvksXUaOxK+oiKzXXufCV19jBLyffbZGZ7kxxrE5IXS+C9WBbxFP/YF++Fu1LtfdUK42jhc+/oTcJUsA8H/1VZyHDbviuJciX6JAW8CGpA1M/ncyCwcvpH+w5Lwqj6xAARi63E1LTw09Qj04nJzP7wdTmNC/lUXfV2Pw15F0Csv0BHk40qOFm1WvBVu5HrUGLZO3TOZw1mGppO31nxDqHGp1u2qLottYlJvnYTy4FEPXxi+Fa6lxrEt7gmih5YxvvvmGqVOnXhFi4Onpyfvvv8+ECROu2cb58+cJCQnhu+++Y+zYsZWvR0dHc+LECURRJCAggJtvvplXXnkFN7fqa93PnTuXV1999YrXf/jhB5ydq375qlQqAgMDCQ0NxcHBoRbvVKY+lJeXk5ycTEZGBnq9vt7tdEj7hfbnV5Pl2oldbV8yo4WNj8JYzojjz6I2lLCzzXSy3TpzoQzmH1IiIjCrhx5/J2tb2TD+TFKwMVVBJ08jj3c078TOc+dO/FdLqhcXbryBC8OGmbV9mRoQRW46+QIu5Vnsb/UEqd79rWqO15Yt+K2VKidl3jaavP4122MQDawsWclJ3UlUqHjA5QF66Z244b+ZGFGwvsuHaNUe7MgQ+DlBSQtnkRe7216scF15/5iSxCKBW0INDA+RdzT0op4fi38kVh+LAw5McJ1AqKp+8f3WQqPLY9jxKSgwsqnjmxQ52keiX0lJCWPHjiU/Px/3a0g0WmyFNiMjA3//K+V4/P39ycioXWWZ7777Djc3N+68884qrz/wwAOEh4cTGBjI8ePHmTFjBkeOHGHDhg3VtjNjxgymVtSDB2mFNjQ0lGHDhl3xAZWVlZGcnIyrqyuOjubLaBRFkcLCQtzc3GxHw86KlJWV4eTkxODBgxv2Oed3Rfz4T/yKTnJL33bStouF0Ol0bNiwgaFDh6KuQYy9IQjHf0F1pATRI5Tr7nkeBCkiaFvxQbaczibNuTUP31z9zkZTwGAUeWPBNkDLE8N7cHOX2u2w1JpbbiGvQwey33kXn03/0rZ9B7yffOKKwyw9js0RhetR2LGAXsJJut/yWqP0Wd045n79DRcqnFmfKVNo8+i1BfBHGEbw0s6X2JyymR9Lf6SvU0fpD+1GcONt0kpX/xIdv7+zhdQSiOg1iA6B1S+eNAWOpeaTuHsPaqXAnLE34OOqsao91r4ey/RlvLD9BWLzY9EoNXwU/RF9Avo0uh1mQfs3nPmHaI9UjDdObNSuLTWOph312lBnh7am1c5L2bdvH0C1jpsoirV26L755hseeOCBKxyeiRMvDlSXLl1o27YtvXv35uDBg/Tq1euKdjQaDRrNlRetWq2+4oM3GAwIgoBCoTBr2VhTmIGp7eaOQqFAEIRqx6BO+EZA22Fw5h/UR5bB8NfNZ2QNNNjmmjgmSR8JPR5A7XDx//Wh/uFsOZ3Nb4fSeHFER5wcbEwHsZbsPp3F+QIpq3p412DUdZTrqg1+jzyCAoHMd94hZ/FilA5qfJ+40qkFC45jcyTyIdj5PorEbShyzkBAp0brWq1Wo1KpyF68mAuLPgbAd9Ik/J54vNbnL4hewPNbn2dz8maeKzzMIkcNUVFPoaj4//DzUHNjhwDWncjg9yMZzA5tusVclu+VksFGdQsm0MvVytZcxBrXY6m+lKnbpxKTHoOj0pFFNy6iX1C/RrXBrPQaD2f+QXlsJcqhc8EKxUDMPY51aavOntWkSZM4derUVR9dunQhMDCQ8+evlDnJysoiIODaGYPbt28nNjaW//3vf9c8tlevXqjVas6cOVPXt2P3REdHM2XKlMrfw8LC+OCDD6xmj0XoXbEKc3g56Jpo0kZeEpyVtGfpUTX+aXA7P0K9ncgv1fHn0aYr4VWpPduj7tqzdcHnkQn4PS/tyGR98CFZixfLiWKWxqsVdBglPd/zWaN2LYoiWR98SHaFM+v33HP4TXq6Tm2olWoWDFlAtHNLtAqBZwID2O1Qdb3n3j7SFvRP+5Ip0tY/TMqaXCjSVn6HjI9q+rHADaFEV8LTm54mJj0GJ5UTi29a3LSdWZBUR1z8oDgLzqy3tjWNTp0dWl9fXzp06HDVh6OjI1FRUeTn57N3797Kc/fs2UN+fj79rxLTZOLrr78mMjKS7t27X/PYEydOoNPpCAoKquvbaXbs27ePxx57zCJt//333/Tt2xcnJyd8fX2vCBWxGG2HgnsIlObCyT8ap09zc/hHQISwQZIk2SUoFQIP9JVuPstimqbmblahlnUV2rN39bJ81rDvxIn4VUzksj9aROa778lOraXp95T08+hKKL7QOH2KIhfeW8CFCpUb/5em4/t4/b7f1IKCBSlJDCkpRSvApH8nsTlpc+Xfh7TzI8LPhUKtnp/2JZvF/MZm5f5kyvVGuoV40CPU09rmWI1iXTFPbnySfRn7cFG78PnQz+kT2ETDDC5FqYbuFRKmzVCT1mJ73x07dmTEiBFMnDiRmJgYYmJimDhxIqNGjaqicNChQwdWraoqM1FQUMDPP/9c7epsfHw88+bNY//+/SQmJrJmzRrGjBlDz549GTDAPssTmhM/P78rEuHMwa+//sq4ceOYMGECR44cYefOnVUS+SyKQgmRD0vP93/TOH2aE4MODlV8+fSsXtZuTGQIDkoFR1PyOZKc13i2mYnvdiVSrjfSs6Un3UIaR6Dc94nH8X9pOgA533xDxpw5iIamn9Bjs7TsB0HdQV8GB761eHei0Yj/H3+Q9/33AATMmY3Pww/Xv8HYtTjkJ7GwQE90i8GUG8t5bstz/BkvJRoqFAKPDgwH4JudCRiMTWuCpDcYWR6TBMD4qLBmm8tRWF7I4xse52DmQdzUbnwx9At6+ve0tlnmw3QPObMeCmuXr2QvWDSYc/ny5XTt2pVhw4YxbNgwunXrxtKlVWcNsbGx5OfnV3ltxYoViKLI/fdfKT3h4ODApk2bGD58OO3bt+fZZ59l2LBhbNy4sf7yT9dAFEVKdCUNfpTqS+t8Tl1WlYqLixk/fjyurq4EBQWxYMGCK465PORAEAQ+//xzRo0ahbOzMx07dmT37t3ExcURHR2Ni4sLUVFRxMfH19ivXq9n8uTJvPvuuzzxxBO0a9eO9u3bc/fdd9fpc24QPR8EQQnJMTZfV/4KTqyC/GRw9oVO1UuP+bhqGNlN2oFoaqu0xVo9SytsfnxwRKPeSH0efljSqVUoyPv5F1JfeEEuk2spBOHiKu2+ryyqDS3q9WTOfRXP3TEgCAS9Nh/vhk6gK0IlHCIf5v0bPuTWiFsxiAZm7pjJ8lPLAbizZwhezmpScktZf6JpOQub/sskNa8UbxcHRnVrnruZWSVZTFg3gSNZR3B3cOfLYV/Sza+btc0yL37tIeQ6SUf5wBJrW9OoWEzlAMDb25tly5Zd9ZjqHLbHHnusxm3x0NBQtm7dahb7akupvpS+P/Rt1D5N7Bm7B+da6jpOmzaNzZs3s2rVKgIDA5k5cyYHDhygR48eVz1v/vz5LFy4kIULFzJ9+nTGjh1LREQEM2bMoGXLljzyyCNMmjSJtWvXVnv+wYMHSU1NRaFQ0LNnTzIyMujRowfvvfcenTt3rutbrh/uQdDhFjj1J+z/Fka+1zj9NhRRhB0fSM/7PQHqmnW5HuzXilWHUll9JI1ZIzvi6dw0ZOV+2p9MfqmOMB9nhnYys7JBLfC86y4ULq6kTptG4dp1GAqLEIbLkl4WofMdsGEOFKZL4T9dzT+pNZaUkDr1eYq2bEEUBAJffx3PO+9oWKMZx6VKZ4IS+vwPlULFawNfw13jzvJTy3lr71tSBaluT/Bgv1Ys+jeOr3YkcHPXpuMYfr87EYD7+oTiqG6aiaUNITE/kSc2PkFqUSrejt58PvRzOnh3sLZZlqHv45CyV5qkRU0Cje0k/1kSOd3eTigqKuLrr7/mvffeY+jQoXTt2pXvvvsOQy22WCdMmMA999xDu3btmD59OomJiTzwwAMMHz6cjh07MnnyZLZs2VLj+WfPngUkBYyXX36Zv/76Cy8vL4YMGUJOTo653uK1MSWHHV0J5cWN129DOLMBMk+Agyv0uXoCZK+WnnQKckerN1YmWNk6eoORr3ckAPC/QREoFdbZ5nQfMZzQxYsRHB0p2bGDFt98g6EOcjAytUSlufh/vPsTacJmRvQ5OZybMIGiLVsQNBrSxj2I262jGt7w3opKkx1vBQ8pxlshKJjeZzpP9ZBWnRcfXsw7+97hgX6hOCgVHDiXy8Gk3Ib33QicOV/IzrgLKAR4oF/zSwY7lnWM8WvHk1qUSku3liy7eZn9OrMAnW4Hr3Apr8SeSsNfA4uu0NoLTion9ozd06A2jEZjpQ5tXWS7nFS1U9KPj4+nvLycqKioyte8vb1rrMh2Kd26XdxyMSlQdO3atcprZWVlFBQUVCtsbJIkmzVrVmWJ4m+//ZaQkBB+/vlnHn+8dvI5DSY8WrqIcxPg2C+SlJCts+N96Wfkw+DkddVDBUFgXFQrZvx2jGUx53hkQDgKKzmItWXN8QxScqVtzrsjrVtC0nXQQFp+/RXJjz+Bc0IiKePG0/KLz3EIaQKlLZsSkRNg23uQdhBS9kHodWZptjw5meT/TaT83DmUHh4EfryI2DQzqH6U5MDRn6TnfatKvAmCwJPdn8TdwZ239r7FslPLuFB2gZHdx7DqYCZf70ig19irX7e2wPe7pZCfoZ0CaOHZxKuz1JHtKdt5fuvzlOpL6eTTicU3LsbHqelXXbwqShUMmAx/TYFdH0uTTJV19YYbA3mFthYIgoCz2rnBDyeVU53PqW28YUMyuC/VeTP1V91rNZXsNalLdOp0UXtSo9EQERFBUlJSve2qMwoF9K6oQLdrERhtPAEoaQ8k7QKFGqJqJzN0W49g3DQqEi+UsDM+28IGNgxRFPlimxR7PT6qlU1sczpHRtJiybfo3N3RnT1L4j33Unr4sLXNsi9c/aDrGOl5zGKzNFl64gSJ94+l/Nw51MHBtPrxB5yuEUpVaw5+JyWyBXaTEtuq4YGOD/DGwDdQCSrWJqwlWfM+grKItcfSSc4pMY8dFqKgTMevB6UdnYeiwqxrTCPzR9wfPPPvM5TqS+kf3J9vh39r/86siR5jwTUQCtOkXctmgOzQ2glt2rRBrVYTExNT+Vpubi6nT5+2eN+RkZFoNBpiY2MrX9PpdCQmJtKqVSNvb0VOkFY6L5yBYz83bt91ZecH0s/u94J7cK1OcXZQcVfFSuenW2pO1LMFdsdf4HhqAY5qBeNt6Eaqad+e5ElPo+nYAUNODuceepiCdf9Y2yz7ol/FSufJ1ZDfsPCYoq1bSRo3HkN2NpoOHWj1449oIiLMYCRg0MPer6TnfZ+QEttq4NbWt/LZ0M9wc3AjNu8Y3m0/A/V5vtuVaB5bLMRvB1IoKTfQ1t+VqNbNw5kzikYWHVrEyztfxiAaGBkxko9v+LjW+Sh2gUoD/SdJz3d8YPsLPGZAdmjtBFdXVx599FGmTZvGpk2bOH78OA8//HCjVCVzd3fniSee4JVXXmH9+vXExsby5JNPAjBmzBiL918FR3fo/6z0fMtbFs20bhCZ/0HsGkCA/pPrdOr/BoWjVgrsir/AzjjbXaX9fJsUWz0mMhRvF9tKYNN7eNBiyRJco6MRtVpSp0wh+8svZa1acxHYVdJUFg2w98t6NSGKItlffknyE09iLCnBOaofrZYtRR1wZUn1evPfX1CQIimMdLnrmof3DerLsluWEeoWSrmQjXPYYlYc30RhmW1+z+gMRpZUONzj+zcPqa5iXTFTNk/hi6NfADChywTeGPgGaitUzbI6kRPA0RNy4puuRnsdkB1aO+Ldd99l8ODBjB49mptuuomBAwcSGRnZaH3fd999jBs3jj59+nDu3Dn+/fdfvLysEF923WPSDSo3AY782Pj914ZdH0k/O4wEv3Z1OjXEy7my0MK7/8TapBP2X0YBW09noRAkB9wWUTg7E/LJx3iNk3QbsxYsJP3llzFqtVa2zE4wSXgdWFLnJE1jaSlpz79A1oKFIIp43nMPLT//HKWrGbO1RRF2S9XF6D0B1I5XP76CCI8Ilt+ynF7+vRCUWgj8ilmbvjCfXWbkp/3JJF4owcfFgTt7trC2ORYnuTCZB9c8yObkzagVal4f+DpTI6eiEJqpq6NxvRgXvn2h2ZM0bY1mOsr2iaurK0uXLqW4uJiMjAymTZvGli1bqujOJiYmVimFK4oit99+e+XvYWFhiKJYReorOjoaURTx9PSssW+1Ws17773H+fPnKSgoYMOGDY0n2XU5GlcY+Jz0fOu7oC+3jh01kZ9yMabJZGcdeer61jiplRxOzmPjqUwzGmcevqhYnR3RJZBWPi5WtqZmBKWSwFkzCZg1CxQK8n/9jXMPPIguNdXapjV92g2XkjTL8iQpvVqiS00lcewDFKxZAyoVgXNfIWjeqwgOZl7lP/WnlLSmcrqmwsjleDl68eWwL+nqcQOCYGRz9me8snMuWoPtTIZKyvV8sFEqB//MDW1w0dh3Dvje9L3c//f9xOXF4evky7cjvmV06+p1vZsVfR8HtQucPwZxG61tjUWRHVoZ+6T3I+AaAPlJcOh7a1tTld2LwaiXtmRDeterCX83RyYMCAPgvX9iMdpQ1aL0/FJWH5ayzx8b3NrK1tQO73EPEvrFFyg9PSk7fpyEO++iaPt2a5vVtFEoYdDz0vNt79SqHG7x3r0k3D0G7alTKL29afXtN3jdd5/5bdOXw8ZXpOf9nwG3uusjOygd+PqWBSjzbkEUBX6L+5Vxa8aRXGgbZXG/3p5AVqGWlt7OjO1rv1Jdoiiy/NRyHtvwGPnafDr7dGbFyBV09+tubdNsA2fvi8nS268stmRPyA6tjH3i4HzJzXQB6Mqsa4+JkpyL1VsGTmlQU48Pbo2bo4rY84X8edQM8kVm4ottZ9EbRa4L925S9eJdBw4g/NdfcOzaFUN+PsmPPU7Wx58g1qDuIVMLeoyV4mnL8mHLmzUeJhqNZH/xJUmPPIohNxdNp46E//Izzn36WMau/d9Azllw8YcBz9a7GScHFU/2eIzS5EfA4MKpnFPc++e9bE7abEZj686FIm1lDPsLw9vjoLLPW32+Np+pW6by1t63KpO/loxYQoBLgLVNsy2ingalAyTthnO7rW2NxbDP/3IZGYBeD4F7C0m2xFZKAO7+BHTF0k2+9Y0NasrDWc0TQ6QV0IUbTqMzWN/xis0orNS8nHR9GytbU3fULVrQavkyPO+7F0SR7I8/JvnxJ9DnNg0BfZtDoYThb0jP938jJUNehu58JkmPPkrWwoWg1+M+ciRhy5ejDq6d8kedKc2DrW9Jz6+fCRq3BjX38IAwWjr1oOjsM/iq2lGoK+TZzc+y8MBC9EZ9w+2tB4v+jaNIq6drCw9GNaFqZnXhcOZhxvw5ho1JG1EpVEzrPY03B76Jo6p2sdDNCvdg6H6/9HzHQuvaYkFkh1bGflE7wuAXpOfbF0C5lfUis05fTAYb/OJVJYJqy8P9w/B1deDchRJ+3m/d6mGiKDL7j+MYjCLDOgUwuJ2fVe2pLwoHB4LmziXorTcRNBqKt28nYfRtcghCfQkfDB1GSYoH61+u8qfCzZtJuP12SnbHIDg5EfTafILfexeFkwXF/7e/J1VQ8usAPcc1uDmNSsnsUR0R9Z6knHyIW8PuBeDb49/y6D+PNnoIQtKFEpbvkSaVL93cweaLr9QVo2jkq2Nf8fC6h0kvTifULZRlNy9jfOfxzULFod4MmAyCAs6shwT7/C6THVoZ+6bHg+DZEoozYd9X1rNDFOGv58BQDm2HSSU2zYCLRsXTFSuhH246TZnOelqDqw6lsjchB0e1gjm3drr2CTaO5+23E7ZyBQ7h4eizskie+Bjpc+diLLFtIX2bZOg8qYBI3AaI24hRqyXj9TdIefIpKcSgY0fCf/0Fz7vvtqxTkpsIeyrK3A6dL1VUMgM3dAjg+vZ+6AxKMhKGs2DIAlzULhzMPMhdq+/ip9ifGk2N5L31segMIoPa+jKgjW+j9NlYZJVk8fiGx/nw4IcYRAM3h9/MT6N+orOvlRKQmxI+raWKlAB/TgZdqVXNsQSyQytj36gcYMh06fnOD0BbaB07Dv8A53ZIGdW3vGeW1VkTY/u2JNjDkfMFWpZWbPc3NvmlOt5YcwqAZ25oS4iXfQiYO3boQPhvv1ZKe+WtWMnZ2++g5OAhK1vWxPBpLWVbA6XfvUTimDHkLl0KgPdD4wlbucJ8xRKuxqZ50qQyfAi0HWrWpmeP6oRaKfDvf5moy3rw860/ExkQSam+lPkx83li4xNkFGeYtc/LOZaSz+ojUjz9Szd3sGhfjYkoiqw6s4rb/riNmPQYnFROzOs/j7cHvY2rgxml3Oydm+aCW5CkS7v1HWtbY3Zkh1bG/ul2H3i3hpILsHFu4/dffOHiVuv1M8DLvBnHGpWSKTdJWraLt8RRYAWR94XrY8kuKifCz4WJgxrBMWlEFE5OBM6aSctvv0EVGIguKYlzDz5I5sL3Zc3aOmDs/TQZR/1J/LkI7ekzKL29Cf38MwJmzEBhbkmu6kjZD8d/BQQY9ppZJ5UAEX6uTBggaS7P/+skAU4t+Gb4N7zY50U0Sg270nZx5x938mf8nxZbrX17nRSjfHuPYDoHe1ikj8YmuTCZxzY8xpxdcygsL6STTydWjFzBHW3vkEMM6oqjh7SgArDzQ8g4Zl17zIzs0MrYP0oVjKy4iPd9BbHrGrf/DbOhNAf8O18Umzczd/ZqQYSfC7klOl5edbxRiy0cT81naYy0Mjz/ti52m1HtEhVFxOo/8LhtNBiNXPjiC87eOpqirVutbZrNU7R1K/FjHiT3pAoQ8GhtIOKX5bgOGdI4BojixUllj7EQ1M0i3TxzQxt8XTWczS5mya4EFIKCcZ3G8dOtP9HVtyuFukJm7pjJU5ue4lyBeXdTtp7OYkdcNg5KBc8Pa2/Wtq2BwWjguxPfcdfqu4hJj0Gj1PB85PMsv2U5EZ72NWluVDqOgo6jpZj21c/YVUlc+7zzyFQSHR1dpZBCWFhYlUILzYbWN0BURV3rP56CwvON02/Cdji8HBDg1g/AQuUXVUoF797dDaVCYPWRNH7a3ziJKEajlAhmFGFUtyC7i9m7HKW7O8Fvv02Ljz5E5eeHLimJ5MefIPnpSZSnyMUYLkeflUXq8y9IShFp6ahDWhA62ongPudRHatfSdx6cfxXSbJI5QTXz7JYN26OaqaPkJzJjzbFkVkoyQVGeETw/c3f82zPZ1EpVOxI3cEdf9zBBwc+oETX8JjsrEItL/5yBIAH+7Ui1Ltph/wczTrKuLXjeG//e5TqS7ku8Dp+G/0bD3d5GJXCvgtENAq3vAsaD0g7BHs+s7Y1ZkN2aJsZ+/bt47HHHjNrm1u2bEEQhGof+/btM2tfDeLGORDQVQo9+P1JsLS+qF4rJYKBVOgh9DqLdhfZypsXKlZmXll9gtgMy8cL/3wgmUNJebg4KHl5ZNNPBKst7sOGEbF2Ld6PPAIqFUWbNnF25EiyFi+WwxAAY3ExWZ98QtzwERT8/TcoFHhPmEDE6tW4TnxbOmjPZ5C40/LGZJ6SkmBA0pz1sGwJ2Lt6hdA91JMirZ6318ZWvq5SqJjYbSK/jf6NAS0GoDPq+Pr419z6+62sObum3rsqeoORZ348yPkCLa39XJg6rG6ltG2JlMIUXtz6Ig+seYBj2cdwU7sxN2ouXw37ipbuLa1tnv3gFgjD5knP/30Ncq2Te2FuZIe2meHn54ezs3ln7/379yc9Pb3K43//+x9hYWH07l2/SlgWQaWBu74ClSPEb7L8zHTHB3DhjFSx7MY5lu2rgscHRzC4nR9lOiOTfjhIabnltpOyi7S8tVaK2XtuaDsCPZqX/qPS1YWAF6cR8fsqnPv2RdRqyf5oEWdvGUneb6sQ9dbRILUmol5P7sqfiBsxguxFHyOWlODYrRthP/1EwPQXUTg7S8lYXe6SquWtfBByEixnUEkO/Hg/lBdJlfkGT7NcXxUoFAJzK1Q+fj2Ywo97k6r8PdwjnE9v/JSPrv+IENcQMksymb59OhP+mcDhzMN17u/d9bHEnM3BxUHJ5+MicW2CJW5LjaV8cOgDRv8+mrWJaxEQuL3N7fx+++/c1e4uOVbWEvQcD60Ggq5EWnhpxDA1SyE7tHZEcXEx48ePx9XVlaCgIBYsuLLM3eUhB4Ig8PnnnzNq1CicnZ3p2LEju3fvJi4ujujoaFxcXIiKiiI+Pr7Gfh0cHAgMDKx8+Pj4sHr1ah555BHb+yLy7wDDX5eeb3wFMo5bpp+U/ZLeJcCIt8DJ0zL9XIZCIbDwnu74u2k4k1nE3NUnLNJPYZmOCd/uI7dER/sANx7qH2aRfpoCmjZtaLnkW4IXvIfK3x9dairpM2cSP3Ik+atXIxrsJ0atJkRRpHDTJs6Ovo2MV17BkJWNumVLWnzwPmErV+DU5TJZpdEfQ3BPKbb8x/ukSmLmxqCHXx+F3ATwaAljvrNYyM/l9GzpxbM3tgXg5d+Pszk2s8rfBUHg+pbX8/vtvzOpxyQclY4cOH+AcWvH8dj6x2rt2K47nsHnW6WKYO/c3Z02/g0rEtHYlOhKWP7fct4vfJ/vT32Pzqijb1Bffrr1J+YPmI+/s7+1TbRfFAq49UNQaqQFdwZTEAAAG51JREFUnqM/WduiBiM7tLVAFEWMJSUNf5SW1vmcumxDTZs2jc2bN7Nq1SrWr1/Pli1bOHDgwDXPmz9/PuPHj+fw4cN06NCBsWPH8vjjjzNjxgz2798PwKRJk2ptx+rVq8nOzubhhx+u9TmNSu9Hod3NknzPr4+aX48v7RAsvVNqv93N0PkO87Z/DXxdNXxwXw8EAVbuT+b3Q+aN7SzTGfjfd/s5lpqPt4sDix/shVrZvL9KBEHAY+RIWq9bi/+0aSi9vNCdSyLtxemcvXU0+X//bZcldMXycvJW/U7C6NtIeXoS5WfPovTyImDWLFr/9SfuI0ZUP6l1cIb7fpQkhLL+g18ekRxQc7JpLsT/C2pnuG85uPiYt/1r8NxNbbmzVwsMRpGnlx/keOqVTrtGqeHx7o+z+vbV3NX2LlSCit3puysd20OZNcvDnc0q4oWfpbjZ/w0MZ2S3plMRLLcsl8WHFzP81+EsOLiAErGECI8IPrnxE74c+iUdvO1Hcsym8W0DQ16Unv/5LMRtsq49DaTp7U1YAbG0lNhekWZpq66pSO0PHkCoRYhAUVERX3/9Nd9//z1Dh0r6it999x0hISHXPHfChAncc889AEyfPp2oqChmz57N8OHDAZg8eTITJkyotc1ff/01w4cPJzQ0tNbnNCqCALd9DJ/2l26m/8yEkQvNI+OTfhS+vx20+dAySgpxsMIqdf/Wvjx7Q1s+3HSGWauO0T3Uk3Bflwa3qzNIoQx7EnJw1aj4/pHraO0n60CaUDg74/PoI3jddy85y38g5+uvKT97lrTnXyDrw4/wuvdePO68A5WXl7VNbRCGggJyV64kd+ky9JnS6qPC2RmvBx/EZ+L/ULrVYqXQPQju/xG+uRniNkoqBDe/ZR4Dj6yEXYuk57cvtpiqwdUQBIG37uxGZoGWHXHZTFiyj1VP9a9WoznINYi5/ecysdtEvjz6JX/E/cHu9N3sTt/NdYHXcW/7e7m+5fWoFdIKc0m5nieWHaBIq+e6MG+mNxHN2ZTCFL4/+T2rzqyizCAlzIW4hhBpiGTWzbNw0liwQpxM9QyYDCn74PQ6KTznvuVm12huLJr3soodER8fT3l5OVFRUZWveXt70779teVbunW7+GUfEBAAQNeuXau8VlZWRkFBwTXbSklJ4Z9//uHRRx+ti/mNj4sv3P6p9Hz/N7Dq8Yav1J4/Cd/fBmV5ENIHHvgZNNZz9p69sS39IrwpLjcw8fv9JGQXN6g9o1Fk+i9H2XgqE41KwVcP9aZLC/vQujQ3ChcXfB+bSOtNG/Gb/CwKd3d0SUlkvvsucUOiSXtpBqXHmpYGpCiKlB47TsZrrxMXfT1ZCxaiz8xE5eeH3/NTabNlM/5Tn6udM2siuCfcWVG5a8+n0rXYUFIPSnJEAIOeb/QdkktxUClY/GAvOgS6kVWo5eFv95FfUrNOdAvXFsztP5e/7vyLu9vdjUpQsTdjL89vfZ5hvwzjo4MfkVqYyku/HuP0+SL83TR8/EBPm94h0Rl0bEraxOR/JzNq1Sh+/O9HygxldPLpxHtD3mPVqFX00fSR1QushVIN9yyF9iPBoIUVYyF2rbWtqhfyf1AtEJycaH/w2lv3V8NoNFJQWIi7mxsKRe2/fIRa1jRviO6oWn0xrsy0PVjda8ZabJl+++23+Pj4MHr06Hrb02i0uRFGLoA1L8LRlZAVK81OPa69qn0FWbHw/WgpJjC4Jzz4K2isG8+mVAh8eF9Pbl20g7jMIm5dtIO37+pWr61JURSZ99dJfjuUilIh8MnYXvSLaNwt3KaI0tUV3yefxPuhh8j/+29yf/gR7alT5P/+O/m//45j5864jxyJ27ChONRiN8UalKekUPDXX+Sv/pPys2crX9e0bYv3I4/gMfIWhIYURuh0G9zwspRt/fcL4N4C2g2vX1vndleEL2ih3Qi4/uX622Um3B3VfDuhD3d8sou4zCIeX7af7x65Do1KWeM5LVxb8ErUKzzW9TF+Pv0zq+JWkV2azZfHvuTLo1+hL26HxiOS98Y8iL+b7SVjiqLIyQsn+SP+D9YmrCVPm1f5t/7B/ZnQZQJ9A/siCAI6XeMXgpG5DJUD3POddO2cWg0rx8GYJZJmbRNCdmhrgSAItdr2vypGIwq9HoWzc50c2trSpk0b1Go1MTExtGwpyZvk5uZy+vRphjSSeLkoinz77beMHz++ikNs0/T5H/i2g58egvTD8EW0NFttFXWtMy+SHQff3QrFWRDYFcatkiqy2AAB7o78+cxAnvnhEHsTc3j6h4PsTWjFzJEdr3pDvZSScj0fbDzDkl2JALw3phs3dQqwoNX2h8LZGa8xY/C8+27Kjhwh98cfKVizlrITJyg7cYLMd95B07EjbkNvwn3oUBzatLFaQqVoNKI9fZri3TEUbtxI6SVx+IJGg9uNN+Bxx524DBxgPhsHvQBZp+HYT/DDPdD1Hhg6TwpLqA1l+VIVQNMKr287uPMLKfHFBgjycOLbCX0Y89luYs7mcN8XMcy8pSN9wryvfp5rEM/2epYnezzJu9t/48dTK8HpDCrXWHCNZfLun7gu4TquD72e6NBoAl0CG+kdXYnWoOVQ5iF2pe1ia/JWzuZfnPz4OfkxKmIUt7a+lbZeba1mo8xVUKrh7m/gt8fgxG/w80PS751us7ZltcaiDu3rr7/O33//zeHDh3FwcCAvL++a54iiyKuvvsoXX3xBbm4uffv25ZNPPqFz54tZslqtlhdeeIEff/yR0tJSbrzxRhYvXlyreFF7xdXVlUcffZRp06bh4+NDQEAAs2bNsojzXBP//vsvCQkJth9ucDnhg+GxLbDiATh/THJOb3lH0o69GmmHYd+XcOwX0JdJlcDGrwYn24qPDHB35IeJfVm44TSLt8Tz3e5zHErO45Oxva4qwJ5fouO73Yl8uzOB3Ipt0rm3duKOns33OmsogiDg1KMHTj164D99OgVr1lK4cSMle/eiPXUK7alTZH+0CHVICE69euLUowfOPXuiadsWQWWZr2tRFNGlpFAcE0PJ7t0Ux+zBkJNzqdE49+uLx62jcRs2FKWrBcJoBAFGL5JCdPZ/Kzm2sWtgyHTo+4S0glTTqf/9Bf+8BEUZ0gs9x0nOsI1MKk10DHLnswcjmfj9fg4l5THms93c1DGAF0e0p11Azbs5eSXlzP7jBH8ecQUepUOIln7d4zl4YTsJ+QnsStvFrrRdvL7ndTp6dyQyIJKuvl3p6teVENcQi02Myg3lxOfFsy9jH7vSd3Eg40BlXCxICW83hN7A6Daj6RfUTw4paAoo1XDnl6BQwrGf4ecJ0PNB6PGApKNua6pFl2HR/7Dy8nLGjBlDVFQUX3/9da3Oeeedd1i4cCFLliyhXbt2vPbaawwdOpTY2FjcKmKzpkyZwp9//smKFSvw8fHh+eefZ9SoURw4cAClsnarTvbIu+++S1FREaNHj8bNzY3nn3+e/HwLyOHUwNdff03//v3p2LFjo/VpNrxawaP/wB9Pw4lVki7f/m+k8IGgHhDcA7zbozDqEI79BAe/lQLpTYT0kbK2na++4mItVEoFL47oQJ8wb5776TBHU/IZ+dF2RvcIJszHhXBfF8J8XQj1cia3pJyvdySwPOYcxRU6ti29nXluaFvZmTUjKm9vvB98AO8HH0Cfm0vRv5sp3LCB4p070aWkoEtJoWD1nwAIzs44de2Kpl071C2CUbdogUOLFqhDQlC6u1+zL9FoxFhYiCEvj/KkZLRxcWjjzlAeF482Ph5jUVGV4wVnZ5x7R+LSvz/uN9+MOqARVuTVjjDqfeg1Xgo9SN0vlY0+tBSGvQZeYWDQSeohRj1CWRHXnf0Q1aGKFWTv1pIMUfggy9taTwa29WXLtGg+3HSGlfuS2XjqPP/+d567I0N45oa2aNQKcot15JaUk1tczvmCMhZviSezUItSITDp+jZMuqFNRczsNBLyE9icvJnNSZs5knWEUzmnOJVzqrI/L40XXXy70MG7A8GuwQS6BBLkEkSgSyAu6msniZYbysnT5pFblktaURpn8s5wJld6JBYkYhCrStL5O/kTFRxFVHAUg0MG4+bQtGTEZJBKxd/xOSgdpCqXB7+THj5tpLLR3e6zeHGS+iKIjVD0fcmSJUyZMuWaK7SiKBIcHMyUKVOYPn06IK3GBgQE8Pbbb/P444+Tn5+Pn58fS5cu5d577wUgLS2N0NBQ1qxZU5mZfylarRbtJdV7CgoKCA0NJTs7G/fLbgZlZWUkJycTFhaGo6P5YpNEUaSwsBA3Nzfb02a1AmVlZSQmJhIaGmrWz7nBiCKKXR+i2PoGglg1ZlhUqNALDqgNJRW/qxE73oox8lHEENufvZpIyytlyk9HOZR85WRHIYBCENAbpa+FDgGuPD44nBGdA1DZcOJJXdDpdGzYsIGhQ4faZGiMsbiY0sOHKTtyhLLDR9AeO3aFw3kpgosLCicnBAeHKg9EEWNBAYb8fIyFhVcXTlepcOzaFad+fXHu2xfHbt0QrPnZiEaEoytR/vsqQkn21Q9VqDBGTcY48DmpaEoTIT6rmIUbz7D+ZOY1j43wdebdu7rSLaTmVecLpReIyYjhxIUTHL9wnNjcWHTGmuNT3dRuuDq4ohJUKBVKVAoVKkGFIAgUlheSW5ZLsf7qiaRuaje6+HYhKjCKqKAoIjwi6nx/s/XrsdkiighJO1EcXYFwajVCRYlmUVAghkdjHDIDMbhn5eGWGseCggJ8fX3Jz8+/wl+7HJtyaM+ePUvr1q05ePAgPXte/KBuu+02PD09+e677/j333+58cYbycnJwesS6Zvu3btz++238+qrr17R7ty5c6t9/YcffriiapZKpSIwMJDQ0FAcGpLoIHNVysvLSU5OJiMjA70NVlRyLM/BqyQez5JEPEsS8ChJRGOQnIpStTeJvtdzzmcIWrWndQ2tJwYjHMkRSCsRyCqDrFLpZ7lRuhlFuInc1MJIJ0+xqfjp9ovRiENmJo5JSThcuIA6JxdVbi7q3FxUV3F0q21KrUbn5UV5QADl/v5oAwKk574+YKGQhoag0hfTIWMVITm7EBAxCipEQYlRUGIUVBRr/DkZfC+FTk135yCxEFafUxJfKCAg4qwCFxW4qMFFJRLiInJjsIhDHTcf9aKeDEMGyYZksgxZ5BvzyTPmkS/mUyaWXbuBCgQEnAVn3AQ3ApQBBCgDCFQGEqAMwF1wlxdomgEqQynBefsIzdmOb5FUznlL+3nkO4dZvO+SkhLGjh1bK4fWpr7BMjKkGKiAy7a3AgICOHfuXOUxDg4OVZxZ0zGm8y9nxowZTJ06tfJ30wrtsGHDalyhdXV1lVdoLUhZWRlOTk4MHjzYtlZoa0IUKb2QwL7Nf9N79P9oo3GijbVtaiC3Xva7KIpkFZVTqjPQ6iqxtU0de1oRMpaUoM/MRNRqpUd5eeUDQUDh7o7SwwOFuwdKD3frrrrWmzGVzy716XQ6HXvsZByfAoq1ehzVSpQKy98finRFnC8+T6m+FL2oR2/UYxANlT/dHdzx0njhqfHEzcENhWC53Rl7uh7tm7sA0OUmoIjbwIA+j1X5qyVXaGtLnR3amlY7L2Xfvn307t27rk1XcrnDJ4riNZ3Aqx2j0WjQaDRXvK5Wq6/44A0GA4IgoFAozJpQZZK8MrXd3FEoFAiCUO0Y2Cy+EeS6tEatcWo6NteRFt7NZ1eiSf3v1YSHBxoP20p+amzsYhwBz0Z8D15qL7ycbSt51V7G0e7xbwf+7ahpw8Dc41iXturs0E6aNIn77rvvqseEhYXVtVkAAgMlyZGMjAyCgi7KtWRmZlau2gYGBlJeXk5ubm6VVdrMzEz69+9fr35lZGRkZGRkZGSaLnV2aH19ffH19bWELYSHhxMYGMiGDRsqY2jLy8vZunUrb7/9NgCRkZGo1Wo2bNhQWa41PT2d48eP884771jELhkZGRkZGRkZGdvFojG0SUlJ5OTkkJSUhMFg4PDhw4BUBMC1QsuwQ4cOvPnmm9xxxx0IgsCUKVN44403aNu2LW3btuWNN97A2dmZsWPHAuDh4cGjjz7K888/j4+PD97e3rzwwgt07dqVm266yWy216Yqlkz9kT9fGRkZGRkZGXNhUYd2zpw5fPfdd5W/m1ZdN2/eTHR0NACxsbFVtFJffPFFSktLeeqppyoLK6xfv75Sgxbg/fffR6VScc8991QWVliyZIlZNGgdHBxQKBSkpaXh5+eHg4ODWZK4jEYj5eXllJWVNesYWlEUKS8vJysrC4VCIStJyMjIyMjIyDQYizq0S5YsYcmSJVc95nLVMEEQmDt3LnPnzq3xHEdHRxYtWsSiRYvMYGVVFAoF4eHhpKenk5aWZrZ2RVGktLQUJycnWeUAcHZ2pmXLls3auZeRkZGRkZExDzYl22UrODg40LJlS/R6PQaD4don1AKdTse2bdsYPHhws8/kVCqVqFQq2bGXkZGRkZGRMQuyQ1sD5paUUiqV6PV6HB0dm71DKyMjIyMjIyNjTuT9XhkZGRkZGRkZmSaN7NDKyMjIyMjIyMg0aWSHVkZGRkZGRkZGpknTLGNoTcoKdakR3FB0Oh0lJSUUFBTIMbRNFHkM7QN5HO0DeRztA3kc7QNLjaPJT7tcEas6mqVDW1hYCEBoaKiVLZGRkZGRkZGRkbkahYWFeHh4XPUYQayN22tnGI1G0tLScHNzazTpqIKCAkJDQ0lOTsbd3b1R+pQxL/IY2gfyONoH8jjaB/I42geWGkdRFCksLCQ4OPiauvXNcoVWoVAQEhJilb7d3d3li7aJI4+hfSCPo30gj6N9II+jfWCJcbzWyqwJOSlMRkZGRkZGRkamSSM7tDIyMjIyMjIyMk0a2aFtJDQaDa+88goajcbapsjUE3kM7QN5HO0DeRztA3kc7QNbGMdmmRQmIyMjIyMjIyNjP8grtDIyMjIyMjIyMk0a2aGVkZGRkZGRkZFp0sgOrYyMjIyMjIyMTJNGdmhlZGRkZGRkZGSaNLJDKyMjIyMjIyMj06SRHdpGYPHixYSHh+Po6EhkZCTbt2+3tkkyV+HNN9+kT58+uLm54e/vz+23305sbGyVY0RRZO7cuQQHB+Pk5ER0dDQnTpywksUy1+LNN99EEASmTJlS+Zo8hk2D1NRUHnzwQXx8fHB2dqZHjx4cOHCg8u/yONo+er2el19+mfDwcJycnIiIiGDevHkYjcbKY+RxtD22bdvGrbfeSnBwMIIg8Pvvv1f5e23GTKvV8swzz+Dr64uLiwujR48mJSXFIvbKDq2FWblyJVOmTGHWrFkcOnSIQYMGcfPNN5OUlGRt02RqYOvWrTz99NPExMSwYcMG9Ho9w4YNo7i4uPKYd955h4ULF/Lxxx+zb98+AgMDGTp0KIWFhVa0XKY69u3bxxdffEG3bt2qvC6Poe2Tm5vLgAEDUKvVrF27lpMnT7JgwQI8PT0rj5HH0fZ5++23+eyzz/j44485deoU77zzDu+++y6LFi2qPEYeR9ujuLiY7t278/HHH1f799qM2ZQpU1i1ahUrVqxgx44dFBUVMWrUKAwGg/kNFmUsynXXXSc+8cQTVV7r0KGD+NJLL1nJIpm6kpmZKQLi1q1bRVEURaPRKAYGBopvvfVW5TFlZWWih4eH+Nlnn1nLTJlqKCwsFNu2bStu2LBBHDJkiDh58mRRFOUxbCpMnz5dHDhwYI1/l8exaTBy5EjxkUceqfLanXfeKT744IOiKMrj2BQAxFWrVlX+Xpsxy8vLE9VqtbhixYrKY1JTU0WFQiGuW7fO7DbKK7QWpLy8nAMHDjBs2LAqrw8bNoxdu3ZZySqZupKfnw+At7c3AAkJCWRkZFQZV41Gw5AhQ+RxtTGefvppRo4cyU033VTldXkMmwarV6+md+/ejBkzBn9/f3r27MmXX35Z+Xd5HJsGAwcOZNOmTZw+fRqAI0eOsGPHDm655RZAHsemSG3G7MCBA+h0uirHBAcH06VLF4uMq8rsLcpUkp2djcFgICAgoMrrAQEBZGRkWMkqmbogiiJTp05l4MCBdOnSBaBy7Kob13PnzjW6jTLVs2LFCg4cOMD+/fuv+Js8hk2Ds2fP8umnnzJ16lRmzpzJ3r17efbZZ9FoNIwfP14exybC9OnTyc/Pp0OHDiiVSgwGA6+//jr3338/IF+PTZHajFlGRgYODg54eXldcYwlfCDZoW0EBEGo8rsoile8JmObTJo0iaNHj7Jjx44r/iaPq+2SnJzM5MmTWb9+PY6OjjUeJ4+hbWM0GunduzdvvPEGAD179uTEiRN8+umnjB8/vvI4eRxtm5UrV7Js2TJ++OEHOnfuzOHDh5kyZQrBwcE89NBDlcfJ49j0qM+YWWpc5ZADC+Lr64tSqbxiJpKZmXnFrEbG9njmmWdYvXo1mzdvJiQkpPL1wMBAAHlcbZgDBw6QmZlJZGQkKpUKlUrF1q1b+eijj1CpVJXjJI+hbRMUFESnTp2qvNaxY8fKpFr5WmwaTJs2jZdeeon77ruPrl27Mm7cOJ577jnefPNNQB7HpkhtxiwwMJDy8nJyc3NrPMacyA6tBXFwcCAyMpINGzZUeX3Dhg3079/fSlbJXAtRFJk0aRK//fYb//77L+Hh4VX+Hh4eTmBgYJVxLS8vZ+vWrfK42gg33ngjx44d4/Dhw5WP3r1788ADD3D48GEiIiLkMWwCDBgw4ArJvNOnT9OqVStAvhabCiUlJSgUVd0NpVJZKdslj2PTozZjFvn/du7XNbU4DuP4ueDPGQQxbEwmLC0IgusGs//AklVhsCWLwTQw2VYEsSiYLBqFmQ2eoFgM/iinKygD8Vk73F12713Ypl94v+Cbzid84Ann4XDOub21vF7vuxnHcazJZPI9uX75Z2Z4p91uy+v1ql6vazqd6vHxUaFQSIvF4tir4S8KhYLC4bAGg4Ecx3HPdrt1ZyqVisLhsDqdjsbjse7u7nRxcaH1en3EzfEvv//lQCJDEwyHQ3k8Hj09PWk2m6nVauns7EzNZtOdIcfTl8vldHl5qV6vp/l8rk6no2g0qmKx6M6Q4+nZbDaybVu2bcuyLFWrVdm2reVyKelzmeXzecViMfX7fY1GI2UyGSWTSe33+y/fl0L7A56fnxWPx+Xz+ZRKpdzfP+E0WZb14Wk0Gu7M4XBQuVzW+fm5/H6/0um0xuPx8ZbGf/1ZaMnQDN1uV4lEQn6/Xzc3N6rVau+uk+PpW6/Xenh40NXVlQKBgK6vr1UqlfT6+urOkOPpeXl5+fBemMvlJH0us91up/v7e0UiEQWDQWWzWa1Wq2/Z95ckff1zXwAAAOBn8A4tAAAAjEahBQAAgNEotAAAADAahRYAAABGo9ACAADAaBRaAAAAGI1CCwAAAKNRaAEAAGA0Ci0AAACMRqEFAACA0Si0AAAAMNobsOCkRUGCa1MAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 800x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\"\"\"\n",
    "Test the PositionalEncoding class with a toy model for 4 dimensions. \n",
    "The  4th  dimension has the same frequency as the 5th but with different offset (i.e. phase)\n",
    "because one is produced by a sine function and the other is produced by a cosine function. \n",
    "The  6th  and  7th  dimensions have lower frequency.\n",
    "\"\"\"\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "pe = PositionalEncoding(num_hiddens=20, dropout=0)\n",
    "pe.eval()\n",
    "Y = pe(torch.zeros((1, 100, 20))).data.cpu().numpy()  # 1 example, 100 words with embedding dim of 20\n",
    "fig = plt.figure(figsize=(8, 4))\n",
    "ax = fig.add_subplot(111)\n",
    "for p in [4, 5, 6, 7]:\n",
    "    ax.plot(np.arange(100), Y[0, :, p].T, label=f'dim {p}')\n",
    "ax.legend()\n",
    "ax.grid()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Embeddings class: sequences -> features\n",
    "\n",
    "class Embeddings(nn.Module):\n",
    "    def __init__(self, d_model, vocab_size, max_position_embeddings, dropout=0):\n",
    "        super().__init__()\n",
    "        self.dropout = dropout\n",
    "        self.word_embeddings = nn.Embedding(vocab_size, d_model, padding_idx=1)\n",
    "        self.position_embeddings = PositionalEncoding(num_hiddens=d_model, dropout=self.dropout,\n",
    "                                                      max_len=max_position_embeddings)\n",
    "        self.LayerNorm = nn.LayerNorm(d_model, eps=1e-12)\n",
    "        self.d_model = d_model\n",
    "\n",
    "    def forward(self, input_ids):\n",
    "        seq_length = input_ids.size(1)\n",
    "        \n",
    "        # Get word embeddings for each input id\n",
    "        word_embeddings = self.word_embeddings(input_ids)                   # (bs, max_seq_length, dim)\n",
    "        # Get position embeddings for the word embeddings and add them     \n",
    "        embeddings = self.position_embeddings(word_embeddings) # (bs, max_seq_length, dim)\n",
    "        \n",
    "        # Layer norm \n",
    "        embeddings = self.LayerNorm(embeddings)             # (bs, max_seq_length, dim)\n",
    "        return embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Transformer encoder\n",
    "class EncoderLayer(nn.Module):\n",
    "    def __init__(self, d_model, num_heads, hidden_dim_mult=4, dropout=0.1):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.dropout = dropout\n",
    "        self.mha = MultiHeadAttention(d_model, num_heads, dropout=dropout)\n",
    "        self.ffn = FFN(d_model, hidden_dim_mult, dropout)\n",
    "\n",
    "        self.layernorm1 = nn.LayerNorm(normalized_shape=d_model, eps=1e-6)\n",
    "        self.layernorm2 = nn.LayerNorm(normalized_shape=d_model, eps=1e-6)\n",
    "    \n",
    "    def forward(self, x):\n",
    "        \n",
    "        # Multi-head attention \n",
    "        attn_output, _ = self.mha(x, x, x)  # (batch_size, input_seq_len, d_model)\n",
    "        \n",
    "        # Layer norm after adding the residual connection \n",
    "        out1 = self.layernorm1(x + attn_output)  # (batch_size, input_seq_len, d_model)\n",
    "        \n",
    "        # Feed forward \n",
    "        ffn_output = self.ffn(out1)  # (batch_size, input_seq_len, d_model)\n",
    "        \n",
    "        # Second layer norm after adding residual connection \n",
    "        out2 = self.layernorm2(out1 + ffn_output)  # (batch_size, input_seq_len, d_model)\n",
    "\n",
    "        return out2\n",
    "\n",
    "class TransformerEncoder(nn.Module):\n",
    "    def __init__(self, num_layers, d_model, num_heads, ff_hidden_dim_mult, input_vocab_size,\n",
    "               maximum_position_encoding, dropout=0.1):\n",
    "        super().__init__()\n",
    "\n",
    "        self.d_model = d_model\n",
    "        self.num_layers = num_layers\n",
    "        self.dropout = dropout\n",
    "\n",
    "        self.embedding = Embeddings(d_model, input_vocab_size, maximum_position_encoding, dropout)\n",
    "\n",
    "        self.enc_layers = nn.ModuleList()\n",
    "        for _ in range(num_layers):\n",
    "            self.enc_layers.append(EncoderLayer(d_model, num_heads, ff_hidden_dim_mult, self.dropout))\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x = self.embedding(x) # Transform to (batch_size, input_seq_length, d_model)\n",
    "\n",
    "        for i in range(self.num_layers):\n",
    "            x = self.enc_layers[i](x)\n",
    "\n",
    "        return x  # (batch_size, input_seq_len, d_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Transormer classifier for sentiment analysis\n",
    "class TransformerClassifier(nn.Module):\n",
    "    def __init__(self, num_layers, d_model, num_heads, ff_hidden_dim_mult, input_vocab_size, num_answers):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.encoder = TransformerEncoder(num_layers, d_model, num_heads, ff_hidden_dim_mult, input_vocab_size,\n",
    "                                          maximum_position_encoding=10000)\n",
    "        self.dense = nn.Linear(d_model, num_answers)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.encoder(x)  # [batch_size, seq_len, d_model]\n",
    "        # pooling\n",
    "        x, _ = torch.max(x, dim=1)  # [batch_size, d_model], can also use torch.mean(dim=1) or just x[:, -1]\n",
    "        x = self.dense(x)  # [batch_size, num_answers]\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TransformerClassifier(\n",
       "  (encoder): TransformerEncoder(\n",
       "    (embedding): Embeddings(\n",
       "      (word_embeddings): Embedding(55000, 32, padding_idx=1)\n",
       "      (position_embeddings): PositionalEncoding(\n",
       "        (dropout): Dropout(p=0.1, inplace=False)\n",
       "      )\n",
       "      (LayerNorm): LayerNorm((32,), eps=1e-12, elementwise_affine=True)\n",
       "    )\n",
       "    (enc_layers): ModuleList(\n",
       "      (0): EncoderLayer(\n",
       "        (mha): MultiHeadAttention(\n",
       "          (W_q): Linear(in_features=32, out_features=32, bias=False)\n",
       "          (W_k): Linear(in_features=32, out_features=32, bias=False)\n",
       "          (W_v): Linear(in_features=32, out_features=32, bias=False)\n",
       "          (W_h): Linear(in_features=32, out_features=32, bias=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "        (ffn): FFN(\n",
       "          (fc_1): Linear(in_features=32, out_features=128, bias=True)\n",
       "          (act): ReLU(inplace=True)\n",
       "          (proj): Linear(in_features=128, out_features=32, bias=True)\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "        (layernorm1): LayerNorm((32,), eps=1e-06, elementwise_affine=True)\n",
       "        (layernorm2): LayerNorm((32,), eps=1e-06, elementwise_affine=True)\n",
       "      )\n",
       "    )\n",
       "  )\n",
       "  (dense): Linear(in_features=32, out_features=2, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "vocab_size = len(vocab)\n",
    "model = TransformerClassifier(num_layers=1, d_model=32, num_heads=2, \n",
    "                              ff_hidden_dim_mult=4, input_vocab_size=vocab_size, num_answers=2)\n",
    "model.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# collate_fn processes sample from DataLoader according to the data processing pipelines declared previously. \n",
    "# label is a tensor saving the labels of individual text entries.\n",
    "max_seq_len = 200\n",
    "\n",
    "def collate_batch(batch):\n",
    "    label_list, text_list = [], []\n",
    "    for _label, _text in batch:\n",
    "        label_list.append(label_pipeline(_label))\n",
    "        processed_text = torch.tensor(text_pipeline(_text)[:max_seq_len], dtype=torch.int64)\n",
    "        if processed_text.shape[0] < max_seq_len:\n",
    "            pad = vocab(['<pad>'])[0] * torch.ones(max_seq_len - len(processed_text), dtype=torch.int64, device=processed_text.device)\n",
    "            processed_text = torch.cat([processed_text, pad])\n",
    "        text_list.append(processed_text)\n",
    "    label_list = torch.tensor(label_list, dtype=torch.int64)\n",
    "    text_list = torch.stack(text_list, dim=0)\n",
    "    return label_list.to(device), text_list.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# functions for training and evaluation\n",
    "def train(dataloader):\n",
    "    model.train()\n",
    "    total_acc, total_count = 0, 0\n",
    "    log_interval = 500\n",
    "    start_time = time.time()\n",
    "\n",
    "    for idx, (label, text) in enumerate(dataloader): \n",
    "        predicted_label = model(text)\n",
    "        # print(f'predicted_label: {predicted_label.shape}, label: {label.shape}')\n",
    "        loss = criterion(predicted_label, label)\n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        # torch.nn.utils.clip_grad_norm_(model.parameters(), 0.1)\n",
    "        optimizer.step()\n",
    "        total_acc += (predicted_label.argmax(1) == label).sum().item()\n",
    "        total_count += label.size(0)\n",
    "        if idx % log_interval == 0 and idx > 0:\n",
    "            elapsed = time.time() - start_time\n",
    "            print(f\"| epoch {epoch:3d} | {idx:5d}/{len(dataloader):5d} batches| accuracy {total_acc / total_count:8.3f}\")\n",
    "            total_acc, total_count = 0, 0\n",
    "            start_time = time.time()\n",
    "\n",
    "\n",
    "def evaluate(dataloader):\n",
    "    model.eval()\n",
    "    total_acc, total_count = 0, 0\n",
    "\n",
    "    with torch.no_grad():\n",
    "        for idx, (label, text) in enumerate(dataloader):\n",
    "            predicted_label = model(text)\n",
    "            loss = criterion(predicted_label, label)\n",
    "            total_acc += (predicted_label.argmax(1) == label).sum().item()\n",
    "            total_count += label.size(0)\n",
    "    return total_acc / total_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# hyper-parameters\n",
    "batch_size = 128\n",
    "epochs = 10\n",
    "lr = 1e-3\n",
    "optimizer = torch.optim.AdamW(model.parameters(), lr=lr)\n",
    "criterion = torch.nn.CrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# split and prepare dataset\n",
    "train_iter, test_iter = IMDB(root='./datasets/imdb/aclImdb/')\n",
    "train_dataset = to_map_style_dataset(train_iter)\n",
    "test_dataset = to_map_style_dataset(test_iter)\n",
    "num_train = int(len(train_dataset) * 0.95)\n",
    "split_train_, split_valid_ = random_split(train_dataset, [num_train, len(train_dataset) - num_train])\n",
    "\n",
    "train_dataloader = DataLoader(split_train_, batch_size=batch_size, shuffle=True, collate_fn=collate_batch)\n",
    "valid_dataloader = DataLoader(split_valid_, batch_size=batch_size, shuffle=False, collate_fn=collate_batch)\n",
    "test_dataloader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False, collate_fn=collate_batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-----------------------------------------------------------\n",
      "| end of epoch   0 | time: 11.51s | valid accuracy    0.835\n",
      "-----------------------------------------------------------\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   1 | time: 11.08s | valid accuracy    0.834\n",
      "-----------------------------------------------------------\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   2 | time: 10.96s | valid accuracy    0.843\n",
      "-----------------------------------------------------------\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   3 | time: 12.04s | valid accuracy    0.866\n",
      "-----------------------------------------------------------\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   4 | time: 10.96s | valid accuracy    0.877\n",
      "-----------------------------------------------------------\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   5 | time: 10.95s | valid accuracy    0.879\n",
      "-----------------------------------------------------------\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   6 | time: 10.96s | valid accuracy    0.881\n",
      "-----------------------------------------------------------\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   7 | time: 11.04s | valid accuracy    0.881\n",
      "-----------------------------------------------------------\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   8 | time: 10.97s | valid accuracy    0.882\n",
      "-----------------------------------------------------------\n",
      "-----------------------------------------------------------\n",
      "| end of epoch   9 | time: 11.05s | valid accuracy    0.869\n",
      "-----------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "# train loop\n",
    "total_accu = None\n",
    "for epoch in range(epochs):\n",
    "    epoch_start_time = time.time()\n",
    "    train(train_dataloader)\n",
    "    accu_val = evaluate(valid_dataloader)\n",
    "    total_accu = accu_val\n",
    "    print(\"-\" * 59)\n",
    "    print(f\"| end of epoch {epoch:3d} | time: {time.time() - epoch_start_time:5.2f}s | valid accuracy {accu_val:8.3f}\")\n",
    "    print(\"-\" * 59)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test accuracy    0.829\n"
     ]
    }
   ],
   "source": [
    "# evaluation on test set\n",
    "accu_test = evaluate(test_dataloader)\n",
    "print(f\"test accuracy {accu_test:8.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* We can usually plot the attention matrices and analyze the learned correlations.\n",
    "* Hopefully, different heads will learn different features/relations (just like filters in CNNs).\n",
    "* The common trend is that earlier layer learn more global features, making the attention matrices *almost* diagonal, and as we go deeper, finer relations are learned and the attention matrices can have unique structures.\n",
    "\n",
    "<center><img src=\"./assets/attention_matrix.PNG\" style=\"height:500px\"></center>\n",
    "\n",
    "<a href=\"https://jalammar.github.io/illustrated-transformer/\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Transformer's Decoder Module\n",
    "---\n",
    "<center><img src=\"./assets/transformer_decoder.png\" style=\"height:350px\"></center>\n",
    "\n",
    "<a href=\"https://jalammar.github.io/illustrated-transformer/\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* The Transformer decoder block looks similar to the Transformer encoder block. \n",
    "* However, besides the two sub-layers (the multi-head attention layer and the positional encoding network), the decoder Transformer block contains a third sub-layer, which applies multi-head attention on the output of the encoder stack.\n",
    "* **Cross-attention**: The cross attention follows the query, key, and value setup used for the self-attention blocks. However, the inputs are a little more complicated. \n",
    "    * The input to the decoder is a data point $y_i$, which is then passed through the self-attention and add-norm blocks, and finally ends up at the cross-attention block. \n",
    "    * This serves as the **query** for cross-attention, where the **key and value** pairs are the output $h^{Enc}$, where this output is calculated with all past inputs $x_1, ..., x_t$.\n",
    "* During training, the output for the $t$-query could observe all the previous key-value pairs. \n",
    "* It results in a different behavior from prediction. Thus, during *prediction* we can eliminate the unnecessary information by specifying the valid length to be $t$ for the $t^{th}$ query.\n",
    "* The output probabilities predict the next token in the output sentence.\n",
    "* During training, we can use \"Teacher Forcing\" which allows us to use the labels from the data; however, during inference, the decoding part is done *iteratively*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/plasticine/100/000000/teacher.png\" style=\"height:50px;display:inline\"> Teacher Forcing\n",
    "---\n",
    "* Teacher forcing is a strategy for training sequential neural networks that use model output from a prior time step as an input.\n",
    "* Teacher forcing works by using the actual or expected output from the training dataset at the current time step $y_t$ as input in the next time step $X_{t+1}$, rather than the output generated by the network.\n",
    "* This startegy allows for faster training, especially in RNNs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<center><img src=\"./assets/transformer_decoding_2.gif\" style=\"height:500px\"></center>\n",
    "\n",
    "<a href=\"https://jalammar.github.io/illustrated-transformer/\">Animation by Jay Alammar</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Transformer Architecture Summary\n",
    "---\n",
    "\n",
    "<center><img src=\"./assets/transformer_arch_sum.png\" style=\"height:500px\"></center>\n",
    "\n",
    "<a href=\"https://lena-voita.github.io/nlp_course/seq2seq_and_attention.html#transformer_model_architecture\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/cotton/64/000000/torch.png\" style=\"height:50px;display:inline\"> Native Transformer in PyTorch\n",
    "---\n",
    "* Transformer is implmented natively in PyTorch:<br>`torch.nn.Transformer(d_model=512, nhead=8, num_encoder_layers=6, num_decoder_layers=6, dim_feedforward=2048, dropout=0.1, activation='relu', custom_encoder=None, custom_decoder=None, layer_norm_eps=1e-05, batch_first=False, norm_first=False, device=None, dtype=None)`\n",
    "* <a href=\"https://pytorch.org/docs/stable/generated/torch.nn.Transformer.html\">Documentation</a>\n",
    "* <a href=\"https://github.com/pytorch/examples/blob/main/word_language_model/main.py\">Code example usage</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/emoji/96/000000/woman-lifting-weights.png\" style=\"height:50px;display:inline\"> Pretrained Models - BERT and GPT\n",
    "---\n",
    "* Large-scale pretrained models have gained popularity over the past years, as big companies can train very large models, which are then published for the public to use as-is or to use with fine-tuning for the users' custom datasets.\n",
    "* **Bidirectional Encoder Representations from Transformers (BERT), Google** - a Transformer-based machine learning technique for natural language processing (NLP) pre-training developed by Google. The idea is to mask certain words and then try to predict them. The original English-language BERT model comes with two pre-trained general types:\n",
    "    * (1) the $BERT_{BASE}$ model, a 12-layer, 768-hidden, 12-heads, 110M parameter neural network architecture.\n",
    "    * (2) the $BERT_{LARGE}$ model, a 24-layer, 1024-hidden, 16-heads, 340M parameter neural network architecture. \n",
    "    * Both of which were trained on the BooksCorpus dataset with 800M words, and a version of the English Wikipedia with 2,500M words.\n",
    "    * Extensions: RoBERTa (Facebook), DistillBERT (HuggingFace)\n",
    "* **Generative Pre-trained Transformer (GPT), OpenAI** - an autoregressive language model that uses deep learning to produce human-like text. GPT was trained with a causal language modeling (CLM) objective and is therefore powerful at predicting the next token in a sequence. The proposed method utilizes generative pre-training of a language model on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each specific task. Unlike BERT, GPT is a generative model, while BERT is an effective pretrained model for embeddings of words/sentences.\n",
    "    * GPT Demo - <a href=\"https://transformer.huggingface.co/doc/gpt\">Write With Transformer</a>.\n",
    "* <a href=\"https://huggingface.co/\">HuggingFace</a> is a company that is dedicated to publishing all of the available pretrained models and it works in PyTorch as well - <a href=\"https://github.com/huggingface/transformers\">HuggingFace Transformers</a>.\n",
    "* <a href=\"https://pytorch.org/hub/huggingface_pytorch-transformers/\">Examples with PyTorch</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"./assets/gpt_3_train.gif\" style=\"height:400px\"></center>\n",
    "\n",
    "<a href=\"https://jalammar.github.io/how-gpt3-works-visualizations-animations/\">Animation by Jay Alammar</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<a href=\"https://huggingface.co/models\">HF Models Hub</a>\n",
    "\n",
    "<center><img src=\"./assets/huggingface.PNG\" style=\"height:500px\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/bubbles/50/null/picture.png\" style=\"height:50px;display:inline\"> Vision Transformer (ViT)\n",
    "---\n",
    "* Instead of word tokens, we can think of image patches as our \"words\", i.e., we treat image patches as tokens.\n",
    "* This enables employing a Transformer architecture for vision tasks!\n",
    "* First, an image is split into fixed-size patches, each of them are then linearly embedded. Then, 2D position embeddings are added, and the resulting sequence of vectors is fed to a standard Transformer encoder. \n",
    "* In order to perform classification, an extra learnable “classification token” is added to the sequence, similarly to Transformer-based NLP tasks.\n",
    "* Transformers require a large amount of data for high accuracy, thus, in the case of having less data, CNNs generally perform better than Transformers.\n",
    "* To reach high performance of ViT, usuall pre-training using a large-size dataset is employed, as its dependence on a large dataset is interpreted as due to low locality inductive bias, an important property of CNNs.\n",
    "\n",
    "\n",
    "* <a href=\"https://pytorch.org/vision/main/models/vision_transformer.html\">Official ViT Pre-trained Models in PyTorch</a>.\n",
    "* <a href=\"https://github.com/lucidrains/vit-pytorch\">ViT Models and Examples with PyTorch</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"./assets/vit_gif_wiki.gif\" style=\"height:400px\"></center>\n",
    "\n",
    "<a href=\"https://en.wikipedia.org/wiki/Vision_transformer\">Image Source</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# code skeleton from: https://lightning.ai/docs/pytorch/latest/notebooks/course_UvA-DL/11-vision-transformer.html\n",
    "\n",
    "def img_to_patch(x, patch_size, flatten_channels=True):\n",
    "    \"\"\"\n",
    "    Inputs:\n",
    "        x - Tensor representing the image of shape [B, C, H, W]\n",
    "        patch_size - Number of pixels per dimension of the patches (integer)\n",
    "        flatten_channels - If True, the patches will be returned in a flattened format\n",
    "                           as a feature vector instead of a image grid.\n",
    "    \"\"\"\n",
    "    B, C, H, W = x.shape\n",
    "    x = x.reshape(B, C, H // patch_size, patch_size, W // patch_size, patch_size)\n",
    "    x = x.permute(0, 2, 4, 1, 3, 5)  # [B, H', W', C, p_H, p_W]\n",
    "    x = x.flatten(1, 2)  # [B, H'*W', C, p_H, p_W]\n",
    "    if flatten_channels:\n",
    "        x = x.flatten(2, 4)  # [B, H'*W', C*p_H*p_W]\n",
    "    return x\n",
    "\n",
    "class VisionTransformer(nn.Module):\n",
    "    def __init__(\n",
    "        self,\n",
    "        embed_dim,\n",
    "        hidden_dim,\n",
    "        num_channels,\n",
    "        num_heads,\n",
    "        num_layers,\n",
    "        num_classes,\n",
    "        patch_size,\n",
    "        num_patches,\n",
    "        dropout=0.0,\n",
    "    ):\n",
    "        \"\"\"\n",
    "        Inputs:\n",
    "            embed_dim - Dimensionality of the input feature vectors to the Transformer\n",
    "            hidden_dim - Dimensionality of the hidden layer in the feed-forward networks\n",
    "                         within the Transformer\n",
    "            num_channels - Number of channels of the input (3 for RGB)\n",
    "            num_heads - Number of heads to use in the Multi-Head Attention block\n",
    "            num_layers - Number of layers to use in the Transformer\n",
    "            num_classes - Number of classes to predict\n",
    "            patch_size - Number of pixels that the patches have per dimension\n",
    "            num_patches - Maximum number of patches an image can have\n",
    "            dropout - Amount of dropout to apply in the feed-forward network and\n",
    "                      on the input encoding\n",
    "        \"\"\"\n",
    "        super().__init__()\n",
    "\n",
    "        self.patch_size = patch_size\n",
    "\n",
    "        # Layers/Networks\n",
    "        self.input_layer = nn.Linear(num_channels * (patch_size**2), embed_dim)\n",
    "        self.transformer = nn.Sequential(\n",
    "            *(AttentionBlock(embed_dim, hidden_dim, num_heads, dropout=dropout) for _ in range(num_layers))\n",
    "        )\n",
    "        self.mlp_head = nn.Sequential(nn.LayerNorm(embed_dim), nn.Linear(embed_dim, num_classes))\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "\n",
    "        # Parameters/Embeddings\n",
    "        self.cls_token = nn.Parameter(torch.randn(1, 1, embed_dim))\n",
    "        self.pos_embedding = nn.Parameter(torch.randn(1, 1 + num_patches, embed_dim))\n",
    "\n",
    "    def forward(self, x):\n",
    "        # Preprocess input\n",
    "        x = img_to_patch(x, self.patch_size)\n",
    "        B, T, _ = x.shape\n",
    "        x = self.input_layer(x)\n",
    "\n",
    "        # Add CLS token and positional encoding\n",
    "        cls_token = self.cls_token.repeat(B, 1, 1)\n",
    "        x = torch.cat([cls_token, x], dim=1)\n",
    "        x = x + self.pos_embedding[:, : T + 1]\n",
    "\n",
    "        # Apply Transforrmer\n",
    "        x = self.dropout(x)\n",
    "        x = x.transpose(0, 1)\n",
    "        x = self.transformer(x)\n",
    "\n",
    "        # Perform classification prediction\n",
    "        cls = x[0]\n",
    "        out = self.mlp_head(cls)\n",
    "        return out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/external-others-maxicons/62/null/external-magic-medieval-others-maxicons.png\" style=\"height:50px;display:inline\"> How to Tame Your Transformer?\n",
    "---\n",
    "* Transformers are notoriously hard to train as they are sensitive to the size of your dataset and the choice of hyperparameters including the learning rate, batch size and optimizer.\n",
    "* Following is a collection of tips and tricks that make Transformers much more stable and converge faster.\n",
    "* For a more detailed analysis, check out <a href=\"https://www.borealisai.com/research-blogs/tutorial-17-transformers-iii-training/\">Tricks For Training Transformers - Borealis AI - P. Xu, S. Prince</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/clouds/100/rocket.png\" style=\"height:50px;display:inline\"> Initialization\n",
    "---\n",
    "* Initialization matters in LLMs, not only for stability, but also for the final performance!\n",
    "    * <a href=\"https://github.com/bigscience-workshop/bigscience/blob/master/train/lessons-learned.md\">BLOOM: Lessons Learned in Training LLMs</a>\n",
    "* **Transformer initialization**: there are many approaches, and papers, but the ground rule is using low standard deviations for the initialization distribution (which is usually normal/Gaussian).\n",
    "    * GPT and T5 have similar initializations (up to small differences in the STDs).\n",
    "* Using PyTorch’s attention and transformer modules: it is better to initialize them manually as the default initialization is not good enough.\n",
    "* LLMs require more carful initialization which is depth-dependent (i.e., how deeper the layer is in the network):\n",
    "    * **T-Fixup**: <a href=\"https://proceedings.mlr.press/v119/huang20f.html\">“Improving Transformer Optimization Through Better Initialization”. Huang et al., ICML 2020</a>.\n",
    "    * **DT-Fixup**: <a href=\"https://arxiv.org/abs/2012.15355\">\"Optimizing deeper transformers on small datasets.“ Peng et al, ACL 2021</a>.\n",
    "    * **Admin**: <a href=\"https://arxiv.org/abs/2004.08249\">\"Understanding the difficulty of training transformers.“. Liu et al., EMNLP 2020</a>.\n",
    "    * **GradInit**: <a href=\"https://arxiv.org/abs/2102.08098\">\"Gradinit: Learning to initialize neural networks for stable and efficient training.\" Chen et al., NeurIPS 2021</a>.\n",
    "    * **DS-Init**: <a href=\"https://arxiv.org/abs/1908.11365\">\"Improving deep transformer with depth-scaled initialization and merged attention.“. Biao et al., 2019</a>.\n",
    "    * **Mimetic-Init** (ViTs): <a href=\"https://arxiv.org/abs/2305.09828\">\"Mimetic Initialization of Self-Attention Layers\", Trockman and Kolter., 2023</a>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# GPT initialization example\n",
    "# https://github.com/karpathy/minGPT/blob/master/mingpt/model.py\n",
    "def _init_weights(self, module):\n",
    "    if isinstance(module, nn.Linear):\n",
    "        torch.nn.init.normal_(module.weight, mean=0.0, std=0.02)\n",
    "        if module.bias is not None:\n",
    "            torch.nn.init.zeros_(module.bias)\n",
    "    elif isinstance(module, nn.Embedding):\n",
    "        torch.nn.init.normal_(module.weight, mean=0.0, std=0.02)\n",
    "    elif isinstance(module, nn.LayerNorm):\n",
    "        torch.nn.init.zeros_(module.bias)\n",
    "        torch.nn.init.ones_(module.weight)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/external-flaticons-lineal-color-flat-icons/64/null/external-workout-running-flaticons-lineal-color-flat-icons-4.png\" style=\"height:50px;display:inline\"> Learning Rate Warm-Up\n",
    "---\n",
    "* **Learning rate warm-up**: the learning rate is gradually increased during the early stages of training.\n",
    "* While this is not typically required for most deep learning architectures, for Transformers training fails if we just start with a typical learning rate. \n",
    "* If we *start with a very small learning rate*, then the training is stable, but then it takes an impractically long time.\n",
    "* <a href=\"https://arxiv.org/abs/2002.04745\">Xiong et al., 2020</a> conducted several experiments with different optimizers and learning rate schedules. Their results show that **learning rate warm-up is essential for both Adam and SGD**, and that the training process is sensitive to the warm-up steps."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"./assets/transformers_warmup.PNG\" style=\"height:400px\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Although learning rate warm-up works, it has some obvious disadvantages -- it introduces an extra hyper-parameter - the number of warm-up steps, and it initializes the learning rate to zero which slows the training down.\n",
    "* <a href=\"https://www.cs.toronto.edu/~mvolkovs/ICML2020_tfixup.pdf\">Huang et al., 2020</a> found that without warm-up, the gradients vanish very quickly, and the Adam updates also rapidly become much smaller.\n",
    "*  <a href=\"https://arxiv.org/abs/2004.08249\">Liu et al., 2020</a> observed that differentiating through the self-attention mechanism creates unbalanced gradients. \n",
    "    * In particular, the gradients for the query $W_q$ and key $W_k$ parameters were much smaller than those for the value parameters $W_v$, and so the former parameters change much more slowly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **Gradient Shrinkage Effect**: <a href=\"https://arxiv.org/abs/2002.04745\">Xiong et al., 2020</a> found that the magnitude of the gradients through layer normalization is inversely proportional to magnitude of the input. Specifically, the gradient has the following property: $$   \\left\\lVert \\frac{\\partial \\bf Layernorm[\\mathbf{X}]}{\\partial \\mathbf{X}} \\right\\rVert=\\mathcal{O}\\left(\\frac{\\sqrt{D}}{\\lVert\\mathbf{X}\\rVert}\\right), $$ where $X$ is the input to layer normalization and $D$ is the embedding dimension.\n",
    "* If the input norm $||X||$ is larger than $\\sqrt{D}$ then back-propagating through layer normalization *reduces* the gradient magnitude in lower layers. As this effect compounds through multiple layers, it causes **the gradient to vanish at lower layers for deep models**.\n",
    "* Moreover, using adaptive optimizers like Adam aggravates the gradient shrinkage effect as the variance of the Adam updates is unbounded at the start of training, and these updates are also known to exhibit high variance in the early stages of training.\n",
    "* This can lead to *problematic large updates* early on, which can make the input norm $||X||$ to each layer increase as we move through the network and thus the increased gradient shrinkage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Finally, residual connections are required in the Transformer architecture for the ease of optimization, which further requires layer normalization to avoid gradient explosion and adaptive optimizers like Adam to address unbalanced gradients in the self-attention blocks. \n",
    "* On the other hand, the use of layer normalization causes the gradients to shrink in the early layers and also amplifies the output perturbations. \n",
    "* Moreover, the instability of Adam in the early stages of training exacerbates both of these effects.\n",
    "* **Learning rate warm-up** effectively stabilizes the Adam updates during the early stages of training by making the parameter changes much smaller. Consequently, Adam no longer aggravates gradient shrinkage and amplification of output perturbations and training becomes relatively stable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"./assets/transformer_warmup_d.PNG\" style=\"height:500px\"></center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# learning rate warmup scheduler example\n",
    "class CosineWarmupScheduler(torch.optim.lr_scheduler._LRScheduler):\n",
    "    # https://uvadlc-notebooks.readthedocs.io/en/latest/tutorial_notebooks/tutorial6/Transformers_and_MHAttention.html\n",
    "    def __init__(self, optimizer, warmup, max_iters):\n",
    "        self.warmup = warmup\n",
    "        self.max_num_iters = max_iters\n",
    "        super().__init__(optimizer)\n",
    "\n",
    "    def get_lr(self):\n",
    "        lr_factor = self.get_lr_factor(epoch=self.last_epoch)\n",
    "        return [base_lr * lr_factor for base_lr in self.base_lrs]\n",
    "\n",
    "    def get_lr_factor(self, epoch):\n",
    "        lr_factor = 0.5 * (1 + np.cos(np.pi * epoch / self.max_num_iters))\n",
    "        if epoch <= self.warmup:\n",
    "            lr_factor *= epoch * 1.0 / self.warmup\n",
    "        return lr_factor\n",
    "    \n",
    "# usage - similar to all other scheduelrs\n",
    "warmup = 3 * len(dataloader)\n",
    "max_iter = 50 * len(dataloader)\n",
    "scheduler = CosineWarmupScheduler(optimizer, warmup=warmup, max_iters=max_iter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/external-flaticons-lineal-color-flat-icons/64/external-activation-media-agency-flaticons-lineal-color-flat-icons.png\" style=\"height:50px;display:inline\"> GLU Variants Activations\n",
    "---\n",
    "* <a href=\"https://arxiv.org/abs/2002.05202\">GLU Variants Improve Transformer</a> - Noam Shazeer, 2020.\n",
    "* The ReLU activation in the FFN (MLP) part of the Transformer can be replaced with variants from the Gated Linear Units (GLU) family for improved performance.\n",
    "* Gated Linear Units consist of the component-wise product of two linear projections, one of which is first passed through a sigmoid function.\n",
    "\n",
    "<center><img src=\"./assets/ffn_glu.PNG\" style=\"height:150px\"></center>\n",
    "\n",
    "* The reason?\n",
    "    * *\"We offer no explanation as to why these \n",
    "architectures seem to work; we attribute their success, as all else, to divine benevolence\"* (from the Conclusion section of the paper).\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.nn import functional as F\n",
    "# replace FFN with FFNSwiglu\n",
    "class FFNSwiglu(nn.Module):\n",
    "    def __init__(self, d_model, hidden_dim_multiplier=4, resid_pdrop=0.1):\n",
    "        super().__init__()\n",
    "        self.w1 = nn.Linear(d_model, hidden_dim_multiplier * d_model, bias=False)\n",
    "        self.w2 = nn.Linear(hidden_dim_multiplier * d_model, d_model, bias=False)\n",
    "        self.w3 = nn.Linear(d_model, hidden_dim_multiplier * d_model, bias=False)\n",
    "\n",
    "        self.dropout = nn.Dropout(resid_pdrop)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.dropout(self.w2(F.silu((self.w1(x))) * self.w3(x)))\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"./assets/glu_variants_benchmark.PNG\" style=\"height:250px\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/nolan/64/replace.png\" style=\"height:50px;display:inline\"> Alternatives to (Post) Layer Normalization\n",
    "---\n",
    "* As the problems we introduced above are directly connected to layer normalization, it is natural to question whether we can train deep transformer models without it. \n",
    "* Indeed, it is possible! sometimes we can achieve even better generalization without layer normalization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **Pre-LN Transformers**: a simple solution to balance the residual dependencies which will limit the output perturbations and mitigate the problem of gradient vanishing.\n",
    "* Pre-LN changes the location of layer normalization inside the transformer layer so that it occurs inside the residual blocks and before the self-attention or MLP. This simple change can help control the gradient magnitude and balance the residual dependencies.\n",
    "* Pre-LN transformer models can be trained **without learning rate warm-up**. However, they sometimes lead to inferior empirical performance. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"./assets/transformers_preln.PNG\" style=\"height:400px\"></center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EncoderLayerPreLN(nn.Module):\n",
    "    def __init__(self, d_model, num_heads, hidden_dim_mult=4, dropout=0.1):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.dropout = dropout\n",
    "        self.mha = MultiHeadAttention(d_model, num_heads, dropout=dropout)\n",
    "        self.ffn = FFN(d_model, hidden_dim_mult, dropout)\n",
    "\n",
    "        self.layernorm1 = nn.LayerNorm(normalized_shape=d_model, eps=1e-6)\n",
    "        self.layernorm2 = nn.LayerNorm(normalized_shape=d_model, eps=1e-6)\n",
    "    \n",
    "    def forward(self, x):\n",
    "        \n",
    "        # pre-ln\n",
    "        x = self.layernorm1(x)\n",
    "        \n",
    "        # Multi-head attention \n",
    "        attn_output, _ = self.mha(x, x, x)  # (batch_size, input_seq_len, d_model)\n",
    "        \n",
    "        # the first residual connection \n",
    "        out1 = x + attn_output  # (batch_size, input_seq_len, d_model)\n",
    "        \n",
    "        # Feed forward + pre-ln \n",
    "        ffn_output = self.ffn(self.layernorm2(out1))  # (batch_size, input_seq_len, d_model)\n",
    "        \n",
    "        # the second residual connection \n",
    "        out2 = out1 + ffn_output  # (batch_size, input_seq_len, d_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# in pytorch set `norm_first=True`\n",
    "model = torch.nn.Transformer(d_model=512,\n",
    "                             nhead=8, num_encoder_layers=6,\n",
    "                             num_decoder_layers=6,\n",
    "                             dim_feedforward=2048,\n",
    "                             dropout=0.1,\n",
    "                             activation='silu',\n",
    "                             batch_first=True,\n",
    "                             norm_first=True)  # pre-ln: norm_first=True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **ReZero**: <a href=\"https://arxiv.org/abs/2003.04887\">Bachlechner et al., 2020</a> propose to remove the layer normalization and introduces a single trainable parameter $\\alpha$ per residual layer so that the self-attention block residual layer becomes, $\\mathbf{X} + \\alpha\\bf MhSa[\\mathbf{X}]$, where $\\alpha$ is initialized to zero.\n",
    "* The result of this is that the entire network is initialized just to compute the identity function, and the contributions of the self-attention and MLP layers are gradually and adaptively introduced."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"./assets/transformer_rezero.PNG\" style=\"height:150px\"></center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for more examples, check out https://github.com/majumderb/rezero, \n",
    "# https://github.com/tbachlechner/ReZero-examples/blob/master/ReZero-Deep_Fast_Transformer.ipynb\n",
    "class EncoderLayerReZero(nn.Module):\n",
    "    def __init__(self, d_model, num_heads, hidden_dim_mult=4, dropout=0.1):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.dropout = dropout\n",
    "        self.mha = MultiHeadAttention(d_model, num_heads, dropout=dropout)\n",
    "        self.ffn = FFN(d_model, hidden_dim_mult, dropout)\n",
    "\n",
    "        # instead of LN, we use a learnable alpha parameter initialized to zero\n",
    "        self.resweight = nn.Parameter(torch.tensor([0.0]), requires_grad=True)\n",
    "    \n",
    "    def forward(self, x):\n",
    "        # Multi-head attention \n",
    "        attn_output, _ = self.mha(x, x, x)  # (batch_size, input_seq_len, d_model)\n",
    "        \n",
    "        # the first residual connection + rezero\n",
    "        out1 = x + attn_output * self.resweight  # (batch_size, input_seq_len, d_model)\n",
    "        \n",
    "        # Feed forward\n",
    "        ffn_output = self.ffn(out1)  # (batch_size, input_seq_len, d_model)\n",
    "        \n",
    "        # the second residual connection + rezero\n",
    "        out2 = out1 + ffn_output * self.resweight  # (batch_size, input_seq_len, d_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **SandwichNorm**: why not both?\n",
    "    * This technique first made an appearance in the CoqView paper, a Chinese version of the famous text-to-image transformer DALL-E. \n",
    "    * They propose, when using Pre-LN, to add an extra LN to all the branch outputs. \n",
    "    * Some people found this to be very effective when facing instability during training.\n",
    "    \n",
    "<center><img src=\"./assets/sandwich_norm.png\" style=\"height:200px\"></center>\n",
    "\n",
    "<a href=\"https://arxiv.org/abs/2105.13290\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **RMSNorm**: LN, but without mean centering and learned bias.\n",
    "    * Faster than LN.\n",
    "    * <a href=\"https://arxiv.org/abs/2102.11972\">An investigative paper</a> found this to be the best performing normalization variant.\n",
    "    * Extensive experiments on several tasks using diverse network architectures show that RMSNorm achieves comparable performance against LayerNorm but reduces the running time by 7%∼64% on different models.\n",
    "    * Usually in post-LN configuration.\n",
    "    \n",
    "$$ y_i = \\text{RMSNorm}(x_i)=\\gamma_i \\hat{x}_i \\in \\mathbb{R}^d $$\n",
    "$$ \\hat{x}_i = \\frac{x_i}{\\sqrt{\\frac{1}{d}\\sum_{l=1}^d x_{i,l}^2}} $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://github.com/lucidrains/x-transformers\n",
    "class RMSNorm(nn.Module):\n",
    "    def __init__(self, dim, eps = 1e-8):\n",
    "        super().__init__()\n",
    "        self.scale = dim ** -0.5\n",
    "        self.eps = eps\n",
    "        self.g = nn.Parameter(torch.ones(dim))\n",
    "\n",
    "    def forward(self, x):\n",
    "        norm = torch.norm(x, dim=-1, keepdim=True) * self.scale\n",
    "        return self.g * x / norm.clamp(min=self.eps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"./assets/transformer_norm_comparison.png\" style=\"height:300px\"></center>\n",
    "\n",
    "<a href=\"https://arxiv.org/abs/2102.11972\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/clouds/100/null/support.png\" style=\"height:50px;display:inline\"> Rectified Adam (RAdam) - Reducing Adam's Variance\n",
    "---\n",
    "* <a href=\"https://arxiv.org/abs/1908.03265\">Liu et al., (2019)</a> argue that the high variance of learning rates in the Adam optimizer at the early stages of training is due to the lack of samples in the early stages of learning.\n",
    "* They base their argument on an experiment in which they do not change the model parameters or momentum term of Adam for the first 2000 learning steps, but only adapt the learning rate.\n",
    "    * After this, warm-up is no longer required!\n",
    "* They propose **Rectified Adam or RAdam** which gradually changes the momentum term over time in a way that helps avoid high variance. \n",
    "    * One way to think of this is that we have effectively incorporated learning rate warm-up into the Adam algorithm, but in a principled way.\n",
    "* <a href=\"https://nn.labml.ai/optimizers/radam.html\">Step-by-step algorithm and implementation</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src=\"./assets/transformer_radam.PNG\" style=\"height:300px\"></center>\n",
    "\n",
    "* Training loss v.s. # of iterations of Transformers on the De-En IWSLT’14 dataset (machine translation)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# RAdam in pytorch: https://pytorch.org/docs/stable/generated/torch.optim.RAdam.html#torch.optim.RAdam\n",
    "optimizer = torch.optim.RAdam(model.parameters(), lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/nolan/64/123.png\" style=\"height:50px;display:inline\"> Positional Encodings/Embeddings/Bias\n",
    "---\n",
    "* Transformers are stripped of any inductive bias (e.g, locality of CNNs).\n",
    "* Positional embeddings are crucial for sequence modeling with transformers.\n",
    "* Positional encodings can be *learned* or *constant*, and they can **absolute** or **relative**.\n",
    "* **Vanilla transformers**: constant absolute positional encodings (sine and cosine).\n",
    "* In addition, instead of adding the positional encodings before the input the transformer, they can be **directly injected to the attention matrix**, which usually results in better performance.\n",
    "* For example, GPT-3 uses *learned* **absolute** positional encodings, while T5 uses *learned* **relative** positional bias.\n",
    "\n",
    "<center><img src=\"./assets/att_pos_enc_table.png\" style=\"height:200px\"></center>\n",
    "\n",
    "<a href=\"https://arxiv.org/abs/2102.11090\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **Relative Positional Encoding (RPE)**: Relative positional encoding (RPE) is added directly to the attention matrix!\n",
    "    * Also termed “relative positional bias”.\n",
    "* Currently, RPE is superior to absolute positional encoding (APE) and it has become the standard in all recent LLMs.\n",
    "* There are several approaches to calculate the relative positional bias matrix, and the values can be learned or pre-determined.\n",
    "* Some popular recent positional encodings:\n",
    "    * Simple Relative Positional Bias (used in T5).\n",
    "    * Attention with Linear Biases (ALiBi).\n",
    "    * Rotary Positional Embeddings (RoPE, used in PaLM).\n",
    "* <a href=\"https://github.com/lucidrains/x-transformers/blob/52bcac25437064757d8c4e5bd9e77b9598b462bb/x_transformers/x_transformers.py#L227\">Code Examples</a>\n",
    "\n",
    "<center><img src=\"./assets/att_rpe.png\" style=\"height:300px\"></center>\n",
    "\n",
    "<a href=\"https://paperswithcode.com/method/relative-position-encodings\">Image Source</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/bubbles/50/refresh.png\" style=\"height:50px;display:inline\"> Staying Up-to-Date with Transformers\n",
    "---\n",
    "* The field is moving very fast!\n",
    "* How can we keep track of all the new improvements?\n",
    "* Recommended repository: https://github.com/lucidrains/x-transformers\n",
    "* Other recommended repositories:\n",
    "    * https://github.com/facebookresearch/fairseq\n",
    "    * https://github.com/microsoft/unilm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### <img src=\"https://img.icons8.com/bubbles/50/000000/video-playlist.png\" style=\"height:50px;display:inline\"> Recommended Videos\n",
    "---\n",
    "#### <img src=\"https://img.icons8.com/cute-clipart/64/000000/warning-shield.png\" style=\"height:30px;display:inline\"> Warning!\n",
    "* These videos do not replace the lectures and tutorials.\n",
    "* Please use these to get a better understanding of the material, and not as an alternative to the written material.\n",
    "\n",
    "#### Video By Subject\n",
    "\n",
    "* Deep Learning for Natural Language Processing (NLP) -  <a href=\"https://youtu.be/6D4EWKJgNn0\"> Deep Learning for Natural Language Processing (NLP) </a>\n",
    "    * Attention and the Transformer <a href=\"https://www.youtube.com/watch?v=f01J0Dri-6k&feature=youtu.be\">Practicum: Attention and the Transformer</a>\n",
    "\n",
    "* Recurrent Neural Networks - <a href=\"https://www.youtube.com/watch?v=SEnXr6v2ifU\"> Recurrent Neural Networks | MIT 6.S191 </a>\n",
    "\n",
    "* LSTM & GRU - <a href=\"https://www.youtube.com/watch?v=8HyCNIVRbSU\"> Illustrated Guide to LSTM's and GRU's: A step by step explanation </a>\n",
    "\n",
    "* Transformers - <a href=\"https://www.youtube.com/watch?v=S27pHKBEp30\">LSTM is dead. Long Live Transformers! </a>\n",
    "* BERT - <a href=\"https://www.youtube.com/watch?v=OR0wfP2FD3c\">BERT Explained!</a>\n",
    "* GPT - <a href=\"https://www.youtube.com/watch?v=9ebPNEHRwXU\">GPT Explained!</a>\n",
    "    * GPT-3 - <a href=\"https://www.youtube.com/watch?v=_x9AwxfjxvE\">OpenAI GPT-3 - Good At Almost Everything!</a>\n",
    "* Vision Transformer (ViT) - <a href=\"https://www.youtube.com/watch?v=TrdevFK_am4\">An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale</a>\n",
    "* Relative Positional Encoding - <a href=\"https://www.youtube.com/watch?v=7XHucAvHNKs\">Relative Positional Encoding for Transformers with Linear Complexity</a>\n",
    "* Rotary Positional Embeddings - <a href=\"https://www.youtube.com/watch?v=o29P0Kpobz0\">Rotary Positional Embeddings: Combining Absolute and Relative</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## <img src=\"https://img.icons8.com/dusk/64/000000/prize.png\" style=\"height:50px;display:inline\"> Credits\n",
    "---\n",
    "* Icons made by <a href=\"https://www.flaticon.com/authors/becris\" title=\"Becris\">Becris</a> from <a href=\"https://www.flaticon.com/\" title=\"Flaticon\">www.flaticon.com</a>\n",
    "* Icons from <a href=\"https://icons8.com/\">Icons8.com</a> - https://icons8.com\n",
    "* <a href=\"https://d2l.ai/chapter_recurrent-neural-networks/index.html\">Dive Into Deep Learning - Recurrent Neural Networks</a>\n",
    "* <a href=\"https://atcold.github.io/pytorch-Deep-Learning/en/week12/12-1/\">DS-GA 1008 - NYU CENTER FOR DATA SCIENCE - Deep Sequence Modeling</a>\n",
    "* <a href=\"https://pytorch.org/tutorials/beginner/text_sentiment_ngrams_tutorial.html\">Text classification with the torchtext library\r\n",
    "</a>\n",
    "* <a href=\"https://www.borealisai.com/research-blogs/tutorial-17-transformers-iii-training/\">Tricks For Training Transformers - Borealis AI - P. Xu, S. Prince</a>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}