1-BiLSTM-NER-CONLL.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4SZ5VlnTaFmo"
   },
   "source": [
    "# 1 - BiLSTM pour Named Entity Recognition\n",
    "\n",
    "## Introduction\n",
    "\n",
    "Dans ce notebook, nous allons implémenter un modèle LSTM multi-couche et bidirectionnel pour faire de la reconnaissance d'entités nommés (ou NER) en utilisant le dataset CONLL2003.\n",
    "\n",
    "## Préparer les données\n",
    "\n",
    "On importe tout d'abord les librairies utiles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "7PIIl2KcaFmq"
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "\n",
    "from torchtext import data\n",
    "from torchtext import datasets\n",
    "\n",
    "import spacy\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import time\n",
    "import random\n",
    "import string\n",
    "from itertools import chain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Les données sont téléchargeable [ici](https://github.com/davidsbatista/NER-datasets/tree/master/CONLL2003) au format .txt et sont sous la forme : \n",
    "\n",
    "```\n",
    "-DOCSTART- -X- -X- O\n",
    "\n",
    "SOCCER NN B-NP O\n",
    "- : O O\n",
    "JAPAN NNP B-NP B-LOC\n",
    "GET VB B-VP O\n",
    "LUCKY NNP B-NP O\n",
    "WIN NNP I-NP O\n",
    ", , O O\n",
    "CHINA NNP B-NP B-PER\n",
    "IN IN B-PP O\n",
    "SURPRISE DT B-NP O\n",
    "DEFEAT NN I-NP O\n",
    ". . O O\n",
    "\n",
    "Nadim NNP B-NP B-PER\n",
    "Ladki NNP I-NP I-PER\n",
    "\n",
    "AL-AIN NNP B-NP B-LOC\n",
    ", , O O\n",
    "United NNP B-NP B-LOC\n",
    "Arab NNP I-NP I-LOC\n",
    "Emirates NNPS I-NP I-LOC\n",
    "1996-12-06 CD I-NP O\n",
    "```\n",
    "\n",
    "où les tags qui nous intéressent pour le named entity recognition sont ceux de la dernière colonne de chaque ligne. Par exemple, 0, B-LOC, B-PER ...\n",
    "\n",
    "Nous allons arranger les données pour avoir une phrase par ligne et les convertir au format csv, pour qu'on puisse utiliser TorchText."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_doc(filename):\n",
    "    data_test = []\n",
    "    tag_test = []\n",
    "    t = []\n",
    "    sent = []\n",
    "    label = []\n",
    "    vocab = {}\n",
    "    f1  = open(filename, \"r\") \n",
    "    for i, line in enumerate(f1):  \n",
    "        if line.split(): #on ne prend pas en compte les listes vides\n",
    "            vocab[line.split()[0]] = i\n",
    "            sent.append(line.split()[0])\n",
    "            tag = [s for s in line.split()[1:] if s not in string.punctuation]\n",
    "            tag_test.append(tag[-1])\n",
    "            t.append(tag[-1])\n",
    "\n",
    "        else:\n",
    "            if tag_test:\n",
    "                label.append(\" \".join(tag_test))\n",
    "                tag_test = []\n",
    "            if sent:\n",
    "                data_test.append(\" \".join(sent))\n",
    "                sent = []\n",
    "             \n",
    "    return data_test, label\n",
    "\n",
    "word_train, tag_train = process_doc('data_ner/train.txt')\n",
    "word_val, tag_val = process_doc('data_ner/valid.txt')\n",
    "word_test, tag_test = process_doc('data_ner/test.txt')\n",
    "\n",
    "\n",
    "train = pd.DataFrame({'text':word_train, 'tag': tag_train})\n",
    "validation = pd.DataFrame({'text':word_val, 'tag': tag_val})\n",
    "test = pd.DataFrame({'text':word_test, 'tag': tag_test})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nos données sont maintenant sous cette forme : "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>tag</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-DOCSTART-</td>\n",
       "      <td>O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>EU rejects German call to boycott British lamb .</td>\n",
       "      <td>B-ORG O B-MISC O O O B-MISC O O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Peter Blackburn</td>\n",
       "      <td>B-PER I-PER</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>BRUSSELS 1996-08-22</td>\n",
       "      <td>B-LOC O</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>The European Commission said on Thursday it di...</td>\n",
       "      <td>O B-ORG I-ORG O O O O O O B-MISC O O O O O B-M...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Germany 's representative to the European Unio...</td>\n",
       "      <td>B-LOC O O O O B-ORG I-ORG O O O B-PER I-PER O ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>\" We do n't support any such recommendation be...</td>\n",
       "      <td>O O O O O O O O O O O O O O O O O O O O B-ORG ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text  \\\n",
       "0                                         -DOCSTART-   \n",
       "1   EU rejects German call to boycott British lamb .   \n",
       "2                                    Peter Blackburn   \n",
       "3                                BRUSSELS 1996-08-22   \n",
       "4  The European Commission said on Thursday it di...   \n",
       "5  Germany 's representative to the European Unio...   \n",
       "6  \" We do n't support any such recommendation be...   \n",
       "\n",
       "                                                 tag  \n",
       "0                                                  O  \n",
       "1                    B-ORG O B-MISC O O O B-MISC O O  \n",
       "2                                        B-PER I-PER  \n",
       "3                                            B-LOC O  \n",
       "4  O B-ORG I-ORG O O O O O O B-MISC O O O O O B-M...  \n",
       "5  B-LOC O O O O B-ORG I-ORG O O O B-PER I-PER O ...  \n",
       "6  O O O O O O O O O O O O O O O O O O O O B-ORG ...  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.head(7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On les convertis en csv."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "train.to_csv (r'data_ner/train.csv', index = False, header=True)\n",
    "test.to_csv (r'data_ner/test.csv', index = False, header=True)\n",
    "validation.to_csv (r'data_ner/valid.csv', index = False, header=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On a alors des données en forme pour pouvoir utiliser les méthodes de TorchText."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "NEF9O3m1aFm8"
   },
   "outputs": [],
   "source": [
    "# pour la reproductibilité\n",
    "SEED = 1234\n",
    "\n",
    "random.seed(SEED)\n",
    "np.random.seed(SEED)\n",
    "torch.manual_seed(SEED)\n",
    "torch.backends.cudnn.deterministic = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VgGjIbQjaFnE"
   },
   "source": [
    "On utilise la méthode `Field` de TorchText pour pré-traiter nos données. On utilise seulement `lower = True` pour mettre en minuscule le texte.\n",
    "\n",
    "Pour les tags, on définit également un `Field`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zOgQnsbiaFnG"
   },
   "outputs": [],
   "source": [
    "TEXT = data.Field(lower = True) \n",
    "TAG = data.Field(unk_token = None) # les tags sont tous connus on a alors unk_token = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 69
    },
    "colab_type": "code",
    "id": "gUMdVt03aFnW",
    "outputId": "66801e7c-b944-48ca-bae9-8a90f95047f0"
   },
   "outputs": [],
   "source": [
    "my_path = \"data_ner/\"\n",
    "train_data, valid_data, test_data = data.TabularDataset.splits(\n",
    "        path= my_path ,\n",
    "        train=\"train.csv\",\n",
    "        validation=\"valid.csv\",\n",
    "        test=\"test.csv\", format='csv', skip_header=True,\n",
    "        fields=((\"text\", TEXT), (\"tag\", TAG))\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DLfP1SMNaFng"
   },
   "source": [
    "On affiche le nombre de phrases dans chaque dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 69
    },
    "colab_type": "code",
    "id": "ssqiO7ViaFnh",
    "outputId": "e9e7b493-818a-4490-f9cb-79523ebf6b8d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of training examples: 14986\n",
      "Number of validation examples: 3465\n",
      "Number of testing examples: 3683\n"
     ]
    }
   ],
   "source": [
    "print(f\"Number of training examples: {len(train_data)}\")\n",
    "print(f\"Number of validation examples: {len(valid_data)}\")\n",
    "print(f\"Number of testing examples: {len(test_data)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kB0kIl5daFno"
   },
   "source": [
    "Affichons un exemple."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 54
    },
    "colab_type": "code",
    "id": "G6vqkBQxaFnq",
    "outputId": "41a9ba8d-494d-4397-eabb-bc8f7edd060f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'text': ['brussels', '1996-08-22'], 'tag': ['B-LOC', 'O']}\n"
     ]
    }
   ],
   "source": [
    "print(vars(train_data.examples[3]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "W7opZSKVaFoI"
   },
   "source": [
    "Ensuite, nous construisons le vocabulaire.\n",
    "\n",
    "On importe les embeddings pré-entrainés de [GloVe](https://nlp.stanford.edu/projects/glove/).  \n",
    "\n",
    "\n",
    "`unk_init` est utilisé pour initialiser les embeddings qui ne sont pas dans le vocabulaire des embeddings pré-entraîné, on les initialise en utilisant une distribution Gaussienne.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 52
    },
    "colab_type": "code",
    "id": "YgKDQ4QEaFoI",
    "outputId": "0d53a99f-8db1-4ee2-8ecb-703e4b8cc772"
   },
   "outputs": [],
   "source": [
    "MIN_FREQ = 0\n",
    "\n",
    "TEXT.build_vocab(train_data, \n",
    "                 min_freq = MIN_FREQ, # les mots qui apparaissent moins que MIN_FREQ fois seront ignorés du vocabulaire\n",
    "                 vectors = \"glove.6B.100d\",\n",
    "                 unk_init = torch.Tensor.normal_)\n",
    "\n",
    "\n",
    "TAG.build_vocab(train_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 69
    },
    "colab_type": "code",
    "id": "Ws9tBVJXaFoR",
    "outputId": "32ae587e-fb43-416d-ae70-108c64984a00"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unique tokens dans le vocabulaire de TEXT vocabulary: 21012\n",
      "Unique tokens dans le vocabulaire de TAG : 10\n"
     ]
    }
   ],
   "source": [
    "print(f\"Unique tokens dans le vocabulaire de TEXT vocabulary: {len(TEXT.vocab)}\")\n",
    "print(f\"Unique tokens dans le vocabulaire de TAG : {len(TAG.vocab)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On peut afficher les tags les plus fréquents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "f1KHpSqjaFoz"
   },
   "outputs": [],
   "source": [
    "def tag_percentage(tag_counts):\n",
    "    \n",
    "    total_count = sum([count for tag, count in tag_counts])\n",
    "    \n",
    "    tag_counts_percentages = [(tag, count, count/total_count) for tag, count in tag_counts]\n",
    "        \n",
    "    return tag_counts_percentages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bpbgscFOaFo4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tag\t\tCount\t\tPercentage\n",
      "\n",
      "O\t\t170522\t\t83.4%\n",
      "B-LOC\t\t7140\t\t 3.5%\n",
      "B-PER\t\t6600\t\t 3.2%\n",
      "B-ORG\t\t6319\t\t 3.1%\n",
      "I-PER\t\t4528\t\t 2.2%\n",
      "I-ORG\t\t3704\t\t 1.8%\n",
      "B-MISC\t\t3438\t\t 1.7%\n",
      "I-LOC\t\t1157\t\t 0.6%\n",
      "I-MISC\t\t1155\t\t 0.6%\n"
     ]
    }
   ],
   "source": [
    "print(\"Tag\\t\\tCount\\t\\tPercentage\\n\")\n",
    "\n",
    "for tag, count, percent in tag_percentage(TAG.vocab.freqs.most_common()):\n",
    "    print(f\"{tag}\\t\\t{count}\\t\\t{percent*100:4.1f}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oojww7BfaFpC"
   },
   "source": [
    "On remarque que les tags ne sont pas équilibrés.\n",
    "\n",
    "Enfin, la dernière étape de préparation des données et de créer des itérateurs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Tz436GNKaFpC"
   },
   "outputs": [],
   "source": [
    "BATCH_SIZE = 32\n",
    "\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "#device = 'cpu'\n",
    "train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits(\n",
    "    (train_data, valid_data, test_data), \n",
    "    batch_size = BATCH_SIZE,\n",
    "    device = device, sort=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "EMkvInHsaFpI"
   },
   "source": [
    "## Construire le modèle\n",
    "\n",
    "On définit un modèle LSTM multi-couche et bi-directionnel. L'image ci-dessous illustre l'architecture de ce modèle de manière simplifiée.\n",
    "\n",
    "\n",
    "![](https://github.com/bentrevett/pytorch-pos-tagging/blob/master/assets/pos-bidirectional-lstm.png?raw=1)\n",
    "\n",
    "L'explication détaillée est décrite [ici](https://github.com/bentrevett/pytorch-pos-tagging/blob/master/1%20-%20BiLSTM%20for%20PoS%20Tagging.ipynb) ou dans [cet article](https://colah.github.io/posts/2015-08-Understanding-LSTMs/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "BdZjPt1FaFpJ"
   },
   "outputs": [],
   "source": [
    "class BiLSTMNER(nn.Module):\n",
    "    def __init__(self, \n",
    "                 input_dim, \n",
    "                 embedding_dim, \n",
    "                 hidden_dim, \n",
    "                 output_dim, \n",
    "                 n_layers, \n",
    "                 bidirectional, \n",
    "                 dropout, \n",
    "                 pad_idx):\n",
    "        \n",
    "        super().__init__()\n",
    "        \n",
    "        self.embedding = nn.Embedding(input_dim, embedding_dim, padding_idx = pad_idx)\n",
    "        \n",
    "        self.lstm = nn.LSTM(embedding_dim, \n",
    "                            hidden_dim, \n",
    "                            num_layers = n_layers, \n",
    "                            bidirectional = bidirectional,\n",
    "                            dropout = dropout if n_layers > 1 else 0)\n",
    "        \n",
    "        self.fc = nn.Linear(hidden_dim * 2 if bidirectional else hidden_dim, output_dim)\n",
    "        \n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        \n",
    "    def forward(self, text):\n",
    "\n",
    "        #text = [sent len, batch size]\n",
    "        \n",
    "        embedded = self.dropout(self.embedding(text)) \n",
    "        #embedded = [sent len, batch size, emb dim]\n",
    "        \n",
    "        outputs, (hidden, cell) = self.lstm(embedded)\n",
    "        #output = [sent len, batch size, hid dim * n directions]\n",
    "        #hidden/cell = [n layers * n directions, batch size, hid dim]\n",
    "   \n",
    "        predictions = self.fc(self.dropout(outputs))\n",
    "        #predictions = [sent len, batch size, output dim]\n",
    "        \n",
    "        return predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "l95vHkamaFpN"
   },
   "source": [
    "## Entraînement"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gG0gwBwPaFpO"
   },
   "outputs": [],
   "source": [
    "INPUT_DIM = len(TEXT.vocab)\n",
    "EMBEDDING_DIM = 100 # doit être le même que la dimension du GloVe embeddings\n",
    "HIDDEN_DIM = 128\n",
    "OUTPUT_DIM = len(TAG.vocab)\n",
    "N_LAYERS = 2\n",
    "BIDIRECTIONAL = True\n",
    "DROPOUT = 0.5\n",
    "PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]\n",
    "\n",
    "model = BiLSTMNER(INPUT_DIM, \n",
    "                        EMBEDDING_DIM, \n",
    "                        HIDDEN_DIM, \n",
    "                        OUTPUT_DIM, \n",
    "                        N_LAYERS, \n",
    "                        BIDIRECTIONAL, \n",
    "                        DROPOUT, \n",
    "                        PAD_IDX)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RtEA5KgEaFpT"
   },
   "source": [
    "On initialise les poids avec une disribution Gaussienne."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "RnCGrUG3aFpU"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BiLSTMNER(\n",
       "  (embedding): Embedding(21012, 100, padding_idx=1)\n",
       "  (lstm): LSTM(100, 128, num_layers=2, dropout=0.5, bidirectional=True)\n",
       "  (fc): Linear(in_features=256, out_features=10, bias=True)\n",
       "  (dropout): Dropout(p=0.5, inplace=False)\n",
       ")"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def init_weights(m):\n",
    "    for name, param in m.named_parameters():\n",
    "        nn.init.normal_(param.data, mean = 0, std = 0.1)\n",
    "        \n",
    "model.apply(init_weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "R0jDNLJ0aFpY"
   },
   "source": [
    "On affiche combien ce modèle a de paramètres."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "BaJF29JSaFpY",
    "outputId": "2b662b54-7593-45b0-e63a-8078092cc6bf"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Le modèle a 2,734,554 paramètres à entraîner\n"
     ]
    }
   ],
   "source": [
    "def count_parameters(model):\n",
    "    return sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
    "\n",
    "print(f'Le modèle a {count_parameters(model):,} paramètres à entraîner')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-XlHSDuwaFpd"
   },
   "source": [
    "On initialise la couche embedding du modèle avec les valeurs des embedding pré-entraînées importées précédemment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "TXt4JhmLaFpf",
    "outputId": "4708724b-2d74-4eb6-e085-2c947d39d7af"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.1117, -0.4966,  0.1631,  ...,  1.2647, -0.2753, -0.1325],\n",
       "        [-0.8555, -0.7208,  1.3755,  ...,  0.0825, -1.1314,  0.3997],\n",
       "        [-0.0382, -0.2449,  0.7281,  ..., -0.1459,  0.8278,  0.2706],\n",
       "        ...,\n",
       "        [-0.6350, -1.0381, -0.0065,  ..., -1.8253,  0.3252, -0.3508],\n",
       "        [-0.5832, -0.5807,  0.3504,  ...,  0.4043, -0.0192,  0.0945],\n",
       "        [ 0.5985, -0.3346,  0.6709,  ..., -0.0541, -0.0713, -0.1815]])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pretrained_embeddings = TEXT.vocab.vectors\n",
    "model.embedding.weight.data.copy_(pretrained_embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "H8e-eWPnaFps"
   },
   "source": [
    "On initialise également les embedding des tokens pad à zéro."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ZEljX3lMaFpt"
   },
   "outputs": [],
   "source": [
    "model.embedding.weight.data[PAD_IDX] = torch.zeros(EMBEDDING_DIM)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eV7ojgvAaFpx"
   },
   "source": [
    "On définit ensuite l'optimiseur. On utilise Adam."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "49hR6CitaFpx"
   },
   "outputs": [],
   "source": [
    "def optimiseur(model, lr=1e-4, eps=1e-6, weight_decay_rate=0.001, second_weight_decay_rate=0.0):\n",
    "    param_optimizer = list(model.named_parameters())\n",
    "    no_decay = ['bias', 'gamma', 'beta']\n",
    "    optimizer_grouped_parameters = [\n",
    "        {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],\n",
    "            'weight_decay_rate': weight_decay_rate},\n",
    "        {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],\n",
    "            'weight_decay_rate': second_weight_decay_rate}]\n",
    "    return optim.Adam(\n",
    "        optimizer_grouped_parameters,\n",
    "        lr=lr,\n",
    "        eps=eps\n",
    "    )\n",
    "\n",
    "optimizer = optimiseur(model, lr=1e-4, eps=1e-6, weight_decay_rate=0.001, second_weight_decay_rate=0.0)\n",
    "scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=10, gamma=0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qAe9E1BjaFp1"
   },
   "source": [
    "La fonction de coût est la cross-entropy. Dans le vocabulaire de TAG, on a des tokens `<pad>` car dans un batch les phrases doivent avoir la même taille. En revanche, nous ne voulons par calculer la perte quand le tag vaut  `<pad>`, c'est pour cela qu'on doit les ignorer avec l'argument `ignore_index`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "uezSIvMsaFp1"
   },
   "outputs": [],
   "source": [
    "TAG_PAD_IDX = TAG.vocab.stoi[TAG.pad_token]\n",
    "criterion = nn.CrossEntropyLoss(ignore_index = TAG_PAD_IDX)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dwkIn6LnaFp6"
   },
   "source": [
    "On place notre modèle et la fonction de coût sur GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UGW86nezaFp7"
   },
   "outputs": [],
   "source": [
    "model = model.to(device)\n",
    "criterion = criterion.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DRpt7qE1aFp_"
   },
   "source": [
    "La fonction suivante calcule le f1 score en ne prenant pas en compte les tokens `<pad>`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3eCX3l6raFqA"
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import f1_score, classification_report\n",
    "\n",
    "def f1_loss( preds, y, tag_pad_idx, full_report=False):\n",
    "    index_o = TAG.vocab.stoi[\"O\"]\n",
    "\n",
    "    positive_labels = [i for i in range(len(TAG.vocab.itos))\n",
    "                       if i not in (tag_pad_idx, index_o)]\n",
    "    _, pred = torch.max(preds, 1)\n",
    "    flatten_preds = pred.data.cpu().numpy() \n",
    "    flatten_y = y.data.cpu().numpy()\n",
    "    if full_report:\n",
    "  \n",
    "        positive_names = [TAG.vocab.itos[i]\n",
    "                              for i in range(len(TAG.vocab.itos))\n",
    "                              if i not in (tag_pad_idx, index_o)]\n",
    "        print(classification_report(\n",
    "                y_true=flatten_y,\n",
    "                y_pred=flatten_preds,\n",
    "                labels=positive_labels,\n",
    "                target_names=positive_names\n",
    "            ))\n",
    "    return f1_score(\n",
    "            y_true=flatten_y,\n",
    "            y_pred=flatten_preds,\n",
    "            labels=positive_labels,\n",
    "            average=\"micro\"\n",
    "        ), flatten_preds, flatten_y\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "11kQmhTXaFqD"
   },
   "source": [
    "Vient ensuite la fonction qui gère l'entraînement de notre modèle.\n",
    "\n",
    "Nous avons d'abord mis le modèle en mode `train` pour activer le dropout / batch-normarlization (si utilisé). Ensuite, nous itérons sur notre itérateur, qui renvoie un batch d'exemples.\n",
    "\n",
    "Pour chaque batch:\n",
    "- on remet à zéro les gradients sur les paramètres du dernier calcul de gradient\n",
    "- insérez le batch de texte dans le modèle pour obtenir des prédictions\n",
    "- comme les fonctions de perte de PyTorch ne peuvent pas gérer les prédictions en 3 dimensions, nous redimensionnons nos prédictions\n",
    "- calculer la perte et la précision entre les tags prédits et les tags réels\n",
    "- appeler `backward` pour calculer les gradients des paramètres w.r.t. la perte\n",
    "- effectuez une «step» d'optimisation pour mettre à jour les paramètres"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XVTIS6jfaFqD"
   },
   "outputs": [],
   "source": [
    "def train(model, iterator, optimizer, criterion, tag_pad_idx):\n",
    "     \n",
    "    epoch_loss = 0\n",
    "    epoch_f1 = 0\n",
    "      \n",
    "    model.train()\n",
    "    \n",
    "    for batch in iterator:\n",
    "        \n",
    "        text = batch.text\n",
    "        tags = batch.tag\n",
    "\n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        #text = [sent len, batch size]\n",
    "        \n",
    "        predictions = model(text)\n",
    "        \n",
    "        #predictions = [sent len, batch size, output dim]\n",
    "        #tags = [sent len, batch size]\n",
    "        \n",
    "        predictions = predictions.view(-1, predictions.shape[-1])\n",
    "        tags = tags.view(-1)\n",
    "        \n",
    "        #predictions = [sent len * batch size, output dim]\n",
    "        #tags = [sent len * batch size]\n",
    "        \n",
    "        loss = criterion(predictions, tags)\n",
    "\n",
    "        \n",
    "        f1,_,_ = f1_loss(predictions, tags, tag_pad_idx, full_report=False)\n",
    "        \n",
    "        loss.backward()\n",
    "        \n",
    "        optimizer.step()\n",
    "\n",
    "        epoch_loss += loss.item()\n",
    "        epoch_f1 += f1.item()\n",
    "        \n",
    "    return epoch_loss / len(iterator), epoch_f1 / len(iterator)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cCI_HQHsaFqI"
   },
   "source": [
    "La fonction `evaluer` est similaire à la fonction` train`, sauf avec les modifications apportées afin de ne pas mettre à jour les paramètres du modèle.\n",
    "\n",
    "`model.eval ()` est utilisé pour mettre le modèle en mode évaluation, donc dropout / batch-norm / etc. sont désactivés.\n",
    "\n",
    "La boucle d'itération est également enveloppée dans `torch.no_grad` pour nous assurer que nous ne calculons aucun gradient. Nous n'avons pas non plus besoin d'appeler `optimizer.zero_grad ()` et `optimizer.step ()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "n-ZN-CKraFqI"
   },
   "outputs": [],
   "source": [
    "def evaluate(model, iterator, criterion, tag_pad_idx, full_report):\n",
    "    \n",
    "    epoch_loss = 0\n",
    "    epoch_f1 = 0\n",
    "    \n",
    "    model.eval()\n",
    "    preds = []\n",
    "    labels = []\n",
    "    with torch.no_grad():\n",
    "    \n",
    "        for batch in iterator:\n",
    "\n",
    "            text = batch.text\n",
    "            tags = batch.tag\n",
    "            \n",
    "            predictions = model(text)\n",
    "            \n",
    "            predictions = predictions.view(-1, predictions.shape[-1])\n",
    "            tags = tags.view(-1)\n",
    "            \n",
    "            loss = criterion(predictions, tags)\n",
    "            \n",
    "            \n",
    "            f1, _, _ = f1_loss(predictions, tags, tag_pad_idx, full_report)\n",
    "            _, pred = torch.max(predictions, 1)\n",
    "            flatten_preds = pred.data.cpu().numpy() \n",
    "            flatten_y = tags.data.cpu().numpy()\n",
    "            preds.append(flatten_preds)\n",
    "            labels.append(flatten_y)\n",
    "            epoch_loss += loss.item()\n",
    "            epoch_f1 += f1.item()\n",
    "        \n",
    "    return epoch_loss / len(iterator), epoch_f1 / len(iterator), preds, labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "KJWClTfkaFqM"
   },
   "source": [
    "On affiche la perte et le f1 score à chaque époque, ainsi que le temps d'exécution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 869
    },
    "colab_type": "code",
    "id": "NRzu_9Y5aFqV",
    "outputId": "a16a0610-53c2-4962-a739-962da7e3aeb9",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 01 | Epoch Time: 0m 11s\n",
      "\tTrain Loss: 0.795 | Train F1 score: 1.17%\n",
      "\t Val. Loss: 0.689 |  Val. F1 score: 1.43%\n",
      "Epoch: 11 | Epoch Time: 0m 5s\n",
      "\tTrain Loss: 0.224 | Train F1 score: 71.39%\n",
      "\t Val. Loss: 0.214 |  Val. F1 score: 71.38%\n",
      "Epoch: 21 | Epoch Time: 0m 6s\n",
      "\tTrain Loss: 0.185 | Train F1 score: 75.38%\n",
      "\t Val. Loss: 0.187 |  Val. F1 score: 74.78%\n",
      "Epoch: 31 | Epoch Time: 0m 10s\n",
      "\tTrain Loss: 0.173 | Train F1 score: 76.51%\n",
      "\t Val. Loss: 0.180 |  Val. F1 score: 75.39%\n",
      "Epoch: 41 | Epoch Time: 0m 10s\n",
      "\tTrain Loss: 0.165 | Train F1 score: 77.38%\n",
      "\t Val. Loss: 0.177 |  Val. F1 score: 75.65%\n"
     ]
    }
   ],
   "source": [
    "def epoch_time(start_time, end_time):\n",
    "    elapsed_time = end_time - start_time\n",
    "    elapsed_mins = int(elapsed_time / 60)\n",
    "    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
    "    return elapsed_mins, elapsed_secs\n",
    "\n",
    "N_EPOCHS = 50\n",
    "\n",
    "t_loss = []\n",
    "t_f1 = []\n",
    "v_loss = []\n",
    "v_f1 = []\n",
    "\n",
    "best_valid_loss = float('inf')\n",
    "\n",
    "for epoch in range(N_EPOCHS):\n",
    "\n",
    "    start_time = time.time()\n",
    "   \n",
    "    \n",
    "    train_loss, train_f1 = train(model, train_iterator, optimizer, criterion, TAG_PAD_IDX)\n",
    "    t_loss.append(train_loss)\n",
    "    t_f1.append(train_f1) \n",
    "    \n",
    "    valid_loss, valid_f1,_,_ = evaluate(model, valid_iterator, criterion, TAG_PAD_IDX, full_report= False)\n",
    "    v_loss.append(valid_loss)\n",
    "    v_f1.append(valid_f1)\n",
    "    \n",
    "    scheduler.step()\n",
    "    end_time = time.time()\n",
    "\n",
    "    epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
    "    \n",
    "    if valid_loss < best_valid_loss:\n",
    "        best_valid_loss = valid_loss\n",
    "        torch.save(model.state_dict(), 'tut1-model_conll.pt')\n",
    "    \n",
    "\n",
    "    if epoch%10 == 0: \n",
    "        print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')\n",
    "        print(f'\\tTrain Loss: {train_loss:.3f} | Train F1 score: {train_f1*100:.2f}%')\n",
    "        print(f'\\t Val. Loss: {valid_loss:.3f} |  Val. F1 score: {valid_f1*100:.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f1c175e0650>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAELCAYAAADTK53JAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXwU9f348dfM7J3dXJvNknDIJZgqXuBRryrUX6wGgl9taaPWVovtF4/Wto+K1gJW7bex354K9attVYq1VqmiEa1WW6sWqSIKGlGOcC85Cbn2nJnfH5sEQgI5yDn7fj4e+8gen5l9f9jlPbPv+cxnFNM0TYQQQliOOtQBCCGEGBiS4IUQwqIkwQshhEVJghdCCIuSBC+EEBYlCV4IISxKErwQQliUJHiRcmbOnMm///3voQ5DiAEnCV4IISxKErwQQCwW49577+W8887jvPPO49577yUWiwFQV1fHN7/5TWbMmMGZZ55JSUkJhmEA8NBDD3H++edz2mmnUVhYyJo1a4ayG0J0YBvqAIQYDn7729/ywQcfsGrVKhRFYcGCBSxbtozvfOc7PPLIIwSDwfbk/cEHH6AoCtu2bePxxx/n6aefJhgMsnv37vbEL8RwIHvwQgDPP/88N954I36/n+zsbG688Uaee+45AGw2G9XV1ezduxe73c6MGTNQFAVN04jFYmzdupV4PM6YMWMYN27cEPdEiIMkwQsBVFVVkZ+f3/44Pz+fqqoqAK6//nqOO+44rrvuOmbNmsVDDz0EwHHHHccdd9zB/fffzznnnMOtt95KZWXlkMQvRFckwQsB5Obmsnfv3vbHoVCI3NxcALxeLwsXLuTVV1/lwQcf5JFHHmkv18yePZsnnniCf/zjHyiKwv/+7/8OSfxCdEUSvEhJ8XicaDTafrvsssv47W9/S11dHXV1dSxdupTZs2cD8I9//IMdO3ZgmiZerxdN01BVlW3btrFmzRpisRgOhwOn04mmaUPcMyEOkoOsIiXdcMMNHR5fd911nHTSScyZMweASy65hAULFgCwY8cO7r77burq6khPT+crX/kKZ511Fps2beLnP/85W7duxW63c9ppp/HjH/940PsixJEocsEPIYSwJinRCCGERUmCF0IIi5IEL4QQFiUJXgghLEoSvBBCWJQkeCGEsKhhNQ5+//5mDKP3ozb9fi+1tU0DENHwJX1ODdLn1NDXPquqQlZW2hFfH1YJ3jDMPiX4tmVTjfQ5NUifU8NA9FlKNEIIYVGS4IUQwqJ6VKKpqKhg4cKF1NfXk5mZSWlpKePHj+/Qpra2lttvv51QKEQ8Hufss8/mzjvvxGYbVlUgIUQ/Mk2T/furicUiQP+UGKqq1JS7cMrR+6zgcLjIygqgKEqv1tuj7Lt48WJKSkooLi5m1apVLFq0iOXLl3do8+CDDzJp0iQeeugh4vE4JSUlvPzyy1x66aW9CkgIMXI0NR1AURSCwTEoSv8UBGw2lUQitRL80fpsmgb19TU0NR3A58vs1Xq7/URqa2spLy+nqKgIgKKiIsrLy6mrq+vQTlEUmpubMQyDWCxGPB4nGAz2KhghxMgSDjfh82X2W3IXnSmKis+XRTjc+1E23e7Bh0IhgsFg+zzXmqaRm5tLKBQiOzu7vd2CBQu4+eabOe+88wiHw1x11VVMnz69V8H4/d5ehn9QIODr87IjlfQ5NQznPldVmTidjl6XDrpjs6XeBuNofdY0B2D2+rvQbwXyl156ialTp/LYY4/R3NzM/Pnzeemll7jkkkt6vI7a2qZeDxXasLWGVW9t5/arTsempc6XIhDwUV3dONRhDCrp8/BjGAa6btJf9XeQEs2RGIbR6bugqspRd4y7TfB5eXlUVlai6zqapqHrOlVVVeTl5XVot2LFCn7yk5+gqio+n4+ZM2eydu3aXiX4vqhrjFKxt4HGljhZPueAvpcQYviaP/9a4vE4iUScXbt2MmHCJACmTJnKHXcs7tE6nn32aaLRKPPmXdWr977yytncd98vmThxcq/jHkjdJni/309BQQFlZWUUFxdTVlZGQUFBh/IMwJgxY/jXv/7FySefTCwWY82aNVx88cUDFngbn9sOQGNLTBK8ECns4YcfAyAU2ss3vnENjz76p05tEonEUUf2zZ175YDFNxR6VKJZsmQJCxcuZNmyZaSnp1NaWgrA/PnzueWWW5g2bRp33HEHixcvZvbs2ei6zllnncWXvvSlAQ0ewOdxANAYjg/4ewkhRp4rr5xNUVEx69a9Q37+aG64YQFLlvyQ5uZmYrEY55xzLgsWfBuA3//+/wiHw9x003dYvfp5XnnlJXy+dLZt24rP5+Wee+7D78856vvt3r2Ln/3sJ9TX70fTNG644UbOPvscIpEI99yzmO3bt6FpNsaNO4677/4pO3du5yc/uYtwOIJh6HzhC7MpKbmmX/reowQ/adIknnrqqU7PP/zww+33x40bxyOPPNIvQfWGt3UPvqlFErwQQ+2tjSHe3BA6pnUoCnR1IdHzTs7j3Gl5nV/ogZqaGu6///8AiEajlJb+Eo/HQyKR4LvfvYm33/43Z599TqflPv64nMcee4JgcBSlpffw9NNP8s1v3njU97rrrjspLr6coqK5VFRs46ab5rNixdNs2PA+jY2NrFiRzKUNDQ0A/PWvT3POOefx1a9e3+H5/jDiz0LyeQ6WaIQQoiuXXHJZ+33DMFi27Nds3LgBMKmtrWXz5k+7TPAnn3wKweAoAE488STeeWftUd+npaWZLVs+5dJLkxdvnzBhIpMnT+WjjzYyefLx7Ny5nZ//vJTTTpvOOeecB8Cpp57G0qW/JhqNcfrpMzj99Bn91GsLJPg0lx1FgUbZgxdiyJ07re972W0GYhSNx+Nuv//kk4/T2NjAQw89itPppLT0XmKxaJfLORyO9vuqmhxkcjRmVz89SJ4nNHr0GB5//Cneffcd3n77LR56aCmPPfZnLrxwFqeccipr1vybFSse5YUXnmPRorv70MvORvy4QlVV8HkcNEkNXgjRA42Njfj9OTidTqqrq3jzzdf7bd1paV4mT57Ciy+WAbBjx3a2bv2Uz3zmJKqqKlFVjQsuuJBbbvke9fX7aWxsYPfuXfj9fi69dDZf//p8yss/6rd4RvwePECG1yElGiFEj3zxi1/mRz+6ja9/vYTc3CDTp5/Rr+tfvPgefvazn/CXv/wJTdO4884fk5WVxZo1b/Hggw8AYBg6V1/9NXJyAixf/gdeeeUlbDYbiqLw7W9/r99iUcwj/aYYAn050Qng53/5AD2h84OS0wcgquFpuJ8AMxCkz8PPvn07GDXquH5dp5zo1LWu/q27O9FpxJdoANLTHFKDF0KIw1gowUuJRgghDmWJBJ/hddIUTmAMn2qTEEIMOWsk+DQHhmnSEkkMdShCCDFsWCLBp6clx6rKUEkhhDjIGgnem5xkTOrwQghxkDUSfOsevIykEUKIgyyR4DPSknvwUqIRInV997s38+yzKzs8Z5omX/ziHN5//70jLnfvvUtYufJJIDkf/JNPPt5lu9Wrn+fOO3/Q5WtXXjmbbdu29DHygWOJBJ/ubduDlxKNEKnqssvmsHr18x2eW79+HZqmceqpPTsJcu7cK3t9sY/hzBJTFTjtGk67JiUaIYZY/NO3iH/yr2Nah6IoXU7aZZ96AfYp5x5xuQsuuJBf/OKnVFRsY8KEiQC88MJzXHrpbLZu3cLPf/5TIpEwsViMOXMu50tfKum0jkPng4/H4/zyl/exfv06AoFcxo0b36P4R9x88COBz2OXBC9ECrPb7Vx88SW8+OLzLFjwbVpamnnjjdf51rduIi0tjV/9ahkOh4OWlhZuuOFazjzzs4wfP+GI61u1aiWh0F7++Me/kEgkuPHG+Z0uVdoVmQ9+AHjddhrDUqIRYijZp5x71L3snjiWuWguu6yY73//Zm644UZeffUVTj75FAKBXOrqannggZ+yZcunKIpKTU01W7Z8etQE/9576/jCF4qw2WzYbDYKC7/Ahg3vH/X9h9t88JaowUPy0n1yVSchUtvxx0/B789h7do1rF79HJddlky0//d/S8nO9vOHPzzOY489QUHBicRiR98h7Ms8jD2ZD/6MM87i3XfX8rWvfYVoNMqFF87ioYceYfToMaxY8Sh3372o1+97JJZJ8F63lGiEEMmDrX/4w0Ps2rWT8877HABNTY3k5gax2Wxs27aFDz44+p44wIwZZ/DSS6tJJBJEoxFeeeWlbpeR+eAHiM9jl2GSQgguvvgLLF36G4qL/wu7PXlJz2uvvZ67717Eyy+/yOjRozn11NO6Xc+cOf/Fli1buOaaL5GbG+TUU6cTCu3pdjmZD/4I+joffCDg49HnNrLy9W08+L3P4bBrAxDd8DLc5wkfCNLn4Ufmg+8fMh98N3weOZtVCCEO1aMSTUVFBQsXLqS+vp7MzExKS0sZP358hzY/+MEP+OSTT9off/LJJyxdupRZs2b1a8BH4nMnf4o1heP4M1yD8p5CCDGc9SjBL168mJKSEoqLi1m1ahWLFi1i+fLlHdrcd9997fc3bdrEtddey/nnn9+/0R6F15NM8HI2qxCDyzRNFEUZ6jAsra+V9G5LNLW1tZSXl1NUVARAUVER5eXl1NXVHXGZp59+mtmzZ+NwOPoUVF+0l2jkQKsQg0ZVNXRdrsMw0HQ9gar2/thit3vwoVCIYDCIpiVXrmkaubm5hEIhsrOzO7WPxWI8//zzPProo70O5mgHC7ozYWwWAKaqEgj4+ryekSRV+nko6fPwYpp+GhsPkJWVg6L03yE9m80yhwd77Eh9Nk2DAwcOEAj4e/1d6Pdhkn//+9/Jz8+noKCg18seyyia5qYIqqIQqmoc1qMO+stwH10xEKTPw5GLRKKRPXt2Av0zIE9VVQwjtUbRHL3PCg6HC3B1+i50N4qm2wSfl5dHZWUluq6jaRq6rlNVVXXEORlWrlzJFVdc0d1q+52qKHhlLLwQg0pRFLKzc/t1ncN/o9b/BqrP3f4O8vv9FBQUUFaWPDOrrKyMgoKCLssz+/btY926de31+sGQ2LGeXQ/egmkk8MnZrEII0a5Hha4lS5awYsUKCgsLWbFiBXfddRcA8+fPZ+PGje3tnnnmGS666CIyMzMHJtouGC0HiNfuwWw5kDybVUbRCCEE0MMa/KRJk3jqqac6Pf/www93ePzf//3f/RNVL6juDADMcANej4M91U2DHoMQQgxHI/5QteJJB8AMH5ASjRBCHGLkJ3h3a4JvacDnsdMcjvdpJI4QQliNBRJ8skRjhBvwuu2YQFNE9uKFEGLkJ3ibA8XhTpZoWs9mlQt/CCGEBRI8gJaW0XqQVeajEUKINhZJ8JmY4Yb2GSXlQKsQQlgmwWd0LNHI2axCCGGNBG9Ly8RsSR5kBSnRCCEEWCTBa2mZmNEmbKqB26nJlMFCCIFlEnzb2ayNeN12GUUjhBBYJsEn574xww34PA4p0QghBFZJ8N62PfjW6QqkRCOEEBZJ8IfswXs9Mh+NEEKAZRJ863QFLckSTVM43ueL1AohhFVYIsErdhdojvYSTTxhEI3rQx2WEEIMKWskeEVB8aQfNl2BlGmEEKnNEgkektMGt42iATmbVQghLJPgVXdGe4kG5GxWIYSwTII/uAcvJRohhACrJfhII2nO5GVmJcELIVKdhRJ8BpgmLsJoqkJjWEo0QojUZp0E33rxbSLJkTQyH40QItX1KMFXVFQwb948CgsLmTdvHtu3b++y3erVq5k9ezZFRUXMnj2bmpqa/oz1qNquzWq2NOBzO6REI4RIebaeNFq8eDElJSUUFxezatUqFi1axPLlyzu02bhxIw888ACPPfYYgUCAxsZGHA7HgATdFdWd3INPXvjDLcMkhRApr9s9+NraWsrLyykqKgKgqKiI8vJy6urqOrR79NFHue666wgEAgD4fD6cTucAhNw1pT3BJ0fSyDBJIUSq6zbBh0IhgsEgmqYBoGkaubm5hEKhDu22bt3Krl27uOqqq7j88stZtmzZ4M4H4/CAamu9NquUaIQQokclmp7QdZ1PPvmERx55hFgsxje+8Q3y8/OZO3duj9fh93v7/P65uemEvZk4zBaCOWm0RBNkZadh0yxzHLmTQMA31CEMOulzapA+949uE3xeXh6VlZXouo6maei6TlVVFXl5eR3a5efnc8kll+BwOHA4HMyaNYsNGzb0KsHX1jZhGL3f6w8EfFRXN2I6fYT316IGk+vYvrOODO/glYkGU1ufU4n0OTVIn3tOVZWj7hh3u3vr9/spKCigrKwMgLKyMgoKCsjOzu7QrqioiDfffBPTNInH47z99tuccMIJvQ74WCju9OQomtb5aOTCH0KIVNaj+sWSJUtYsWIFhYWFrFixgrvuuguA+fPns3HjRgAuu+wy/H4/l156KXPnzmXy5MlceeWVAxd5F1R3Omb4AF63TFcghBA9qsFPmjSJp556qtPzDz/8cPt9VVW5/fbbuf322/svul5KzkfTiM+dPCAsQyWFEKnMUkcgFXc6mDo+LZnYZaikECKVWSzBJ89mdZstgJRohBCpzWIJPnmykxprIs1lk/lohBApzWIJvm0+muSBVplRUgiRyqyV4D2HTlcgZ7MKIVKbtRK8Mw0UNXnxbbddErwQIqVZK8ErautQyQPJCcekRCOESGGWSvCQPNBqtJZomlrigzvhmRBCDCOWTPBtJRrdMAlH9aEOSQghhoQFE3xG+5zwAE1SphFCpCgLJvjWGrw7OQuDHGgVQqQqyyV41ZMOegK/J/m4an94aAMSQoghYrkE33ayU8AVR1MVdlc3DXFEQggxNCyY4FunK4g2kudPY5ckeCFEirJggm+driDcwNjcNPZUNw9xREIIMTQsmODbpis4wJiAl/2NUZojcqBVCJF6rJfgXV5AwQw3MDqQvFbh7iop0wghUo/1Eryqobi8mC0NjAmkAbBbyjRCiBRkuQQPbSc7HSDL58TjtLFHDrQKIVKQNRO8JzkfjaIojMn1ykgaIURKsmaCb52PBmBMIDmSRiYdE0KkGosm+GSJBmBMwEskplN7IDLEUQkhxOCy9aRRRUUFCxcupL6+nszMTEpLSxk/fnyHNvfffz9/+tOfyM3NBeD0009n8eLF/R5wTyjudEjEMOMRxrSNpKluJifTPSTxCCHEUOhRgl+8eDElJSUUFxezatUqFi1axPLlyzu1mzt3Lrfddlu/B9lbqvvgpftGB7IB2F3dxKnH5wxlWEIIMai6LdHU1tZSXl5OUVERAEVFRZSXl1NXVzfgwfXVoRffdjtt5GS4ZE4aIUTK6TbBh0IhgsEgmqYBoGkaubm5hEKhTm1feOEFZs+ezXXXXcf69ev7P9oearv4ttF+oNUrY+GFECmnRyWanvjyl7/Mt771Lex2O2+99RYLFixg9erVZGVl9Xgdfr+3z+8fCPja7yec+ewEvFqU9ICP44/LYsO2WjKzPNhtWp/fY7g5tM+pQvqcGqTP/aPbBJ+Xl0dlZSW6rqNpGrquU1VVRV5e3mHBBdrvn3vuueTl5bF582bOPPPMHgdTW9uEYfR+OGMg4KO6urH9sWkkf5g0VFcRrW4kO82BYZhs2FTJuKA1vjiH9zkVSJ9Tg/S551RVOeqOcbclGr/fT0FBAWVlZQCUlZVRUFBAdnZ2h3aVlZXt9z/++GP27NnDhAkTeh1wf1BUGzjTOoyFB2RmSSFESulRiWbJkiUsXLiQZcuWkZ6eTmlpKQDz58/nlltuYdq0afziF7/go48+QlVV7HY79913X4e9+sGmtl6bFSCY7cGmKeyqbuKzQxaREEIMrh4l+EmTJvHUU091ev7hhx9uv9+W9IeLQ89mtWkqef40GUkjhEgpljyTFZIJ3mg9mxUOTlkghBCpwroJ3pOB2dLQ/lgu/iGESDXWTfAuH8TDmIkYgFz8QwiRcqyb4D0Hr80KMDb34Jw0QgiRCiyb4A+djwYg0+sgzWWTA61CiJRh2QSv+JITixkH9iUfKwqjA15J8EKIlGHZBK9m5oFqQ6/d2f6cXPxDCJFKLJvgFdWGmj0ao3ZX+3Ny8Q8hRCqxbIIH0PzjMGp3tu+xH3rxDyGEsDpLJ3jVPw4z0ojZUg/A6NY5aeQi3EKIVGD5BA9gtNbh2y7+sUcSvBAiBVg6wWv+sQCHHWiVi38IIVKDpRO84vCg+ALte/CQLNPsq20hnjCGMDIhhBh4lk7wkDzQevgevGGahGplL14IYW2WT/CqfxzmgSrMWBiAMe1TFkgdXghhbZZP8Jp/HGBi1O0GYFS2G7dTY9PO+qENTAghBpjlE7yakxxJo9clT3jSVJVTJuXw/uYadEPq8EII67J8glfSssGZhlFzsA5/+pQATeE4m3cdOMqSQggxslk/wStKpwOtJ03Mxm5Tee/T6iGMTAghBpblEzwkD7QadbswDR0Al8PGieOzeW9ztUw8JoSwrJRI8Jp/HOhxjAOV7c9NnxqgriHK9n2NQxiZEEIMnJRI8IdPWQBwyuQcVEWRMo0QwrJ6lOArKiqYN28ehYWFzJs3j+3btx+x7bZt2zjllFMoLS3trxiPWXJueK1Dgve67UwdlykJXghhWT1K8IsXL6akpIS//e1vlJSUsGjRoi7b6brO4sWL+fznP9+vQR4rRbOhZo3ucKAVkqNpQrUt7K2Rs1qFENbTbYKvra2lvLycoqIiAIqKiigvL6eurq5T24ceeogLL7yQ8ePH93ugx0ptnRv+UKdPCQDIXrwQwpK6TfChUIhgMIimaQBomkZubi6hUKhDu02bNvHmm2/yta99bUACPVaafxxmuAGj5eAZrFk+JxPz0yXBCyEsydYfK4nH4/zoRz/if/7nf9o3BH3h93v7vGwg4Dvq6+FJJxBaA75EDZ7A2PbnLzhtDI++UI5p08jN8vT5/YdCd322IulzapA+949uE3xeXh6VlZXouo6maei6TlVVFXl5ee1tqqur2blzJzfccAMADQ0NmKZJU1MTd999d4+Dqa1twjB6Py49EPBRXX304Y6m5gegbtsmmtMntz8/dXQ6AH9/ezsXzxjb5bLDUU/6bDXS59Qgfe45VVWOumPcbYL3+/0UFBRQVlZGcXExZWVlFBQUkJ2d3d4mPz+ftWvXtj++//77aWlp4bbbbut1wANFcaah+HI6TFkAEMz2MDqQxnufVI+oBC+EEN3p0SiaJUuWsGLFCgoLC1mxYgV33XUXAPPnz2fjxo0DGmB/0lrPaD3c6ccH+HR3PQ0tsSGISgghBkaPavCTJk3iqaee6vT8ww8/3GX7m2+++diiGiCqfxyJ7esx41EUu7P9+elTAzz/7+28v7mGC07JH8IIhRCi/6TEmaxtVP9YwMTYv7vD82NzveRkuGQ0jRDCUlIqwWutUxboh9XhFUXh9CkByrfXEY4mhiI0IYTodymV4BVvDjjcnU54guRJTwndZOO22iGITAgh+l9qJfgu5oZvM3l0BulpDtaWV3axpBBCjDwpleDh0LnhO16uT1UVLjgln/c318jcNEIIS0i5BK/5x0EihtlQ1em1i2eMwW5XefHtHUMQmRBC9K+US/Btc8N3VabxeRx87pTRrPmokpr68GCHJoQQ/Sr1EnzWaHC4SWxZ0+XrhWeORVHgpf903gAIIcRIknIJXtFsOE6+hMSO9ejVFZ1ez053ce60UfzrgxAHmqJDEKEQQvSPlEvwAI6T/h8404i++0yXr3/hrOPQDYOX3+08rYEQQowUKZngFYcbxymXou/agL5vc6fXg9kezjghl3+8t4fmSHwIIhRCiGOXkgkewHHi51Hc6UTf/WuXr1/22fFEYjqvrdvd5etCCDHcpWyCV+xOHKdehr73YxJ7P+70+thcL6dM8vPKu7uJxvQhiFAIIY5NyiZ4AHvBRSieTGLvPoNpdr7QyGXnjKcpHOf1D/YOQXRCCHFsUjrBKzYHjtPnoO/7FH33h51enzw6gxPGZfK3/+wknjC6WIMQQgxfKZ3gAexTL0Dx+om++9eu9+I/O579jVHWfLRvCKITQoi+S/kEr2g2HKfPwaiuQN/5fqfXPzM+i/GjfDz/1nYiMZlKWAgxcqR8ggewTzkXJT3YuhffsRSjKApfnnU8dQ0R/vzqliGKUAghek8SPKCoNpzTizFqd5GoWNfp9SljM7nk7HH864O9vL+5ZggiFEKI3pME38o26WzUzHyi76zENDqXYi4/fyJjc708+uLHNDTLxbmFEMOfJPhWiqriPPOLmAf2Ed/0r06v2zSV+bM/Q0tU59EXN3V5QFYIIYYTSfCH0I47FS1vKrF1z2LGOk8XPCbg5crPTeT9LTW8sSE0BBEKIUTPSYI/hKIoOM+ahxluIPbB6i7bfP6MsZwwLpMn/r6Zqv0tgxyhEEL0XI8SfEVFBfPmzaOwsJB58+axffv2Tm1WrlzJ7NmzKS4uZvbs2Sxfvry/Yx0UWu5EbJPOIrbhbxjN+zu9rioK11/2GVRV4XdlH2MYUqoRQgxPPUrwixcvpqSkhL/97W+UlJSwaNGiTm0KCwt57rnnWLVqFU888QSPPPIImzZt6veAB4PzjCvB1IkdYSIyf4aLq//fFLbsOcCLa+XyfkKI4anbBF9bW0t5eTlFRUUAFBUVUV5eTl1dXYd2Xq8XRVEAiEQixOPx9scjjZoewH7i54l/8iZ6Xddzwp/9mSBnFuTy7BsVfLqrfpAjFEKI7tm6axAKhQgGg2iaBoCmaeTm5hIKhcjOzu7Q9tVXX+UXv/gFO3fu5Hvf+x5Tp07tVTB+v7dX7Q8VCPj6vGxX9Iu/wq7Nb2K+91cCX7mzyza3XjWD7//6dX676kN+8e3PkZvt6dcYutPffR4JpM+pQfrcP7pN8L0xa9YsZs2axd69e7nxxhu54IILmDhxYo+Xr61t6lNNOxDwUV3d2OvlumM/pYjw2icJrX8b25gTu2yzYO5J3LN8HXc9vIbbr56O06H1exxdGag+D2fS59Qgfe45VVWOumPcbYkmLy+PyspKdD05J7qu61RVVZGXl3fEZfLz85k2bRr//Oc/ex3wcGI/cVZyIrK1T3aawqBNnj+Nb9mwoNgAABaLSURBVM45kV1VTfz+hXIZHy+EGDa6TfB+v5+CggLKysoAKCsro6CgoFN5ZuvWre336+rqWLt2LVOmTOnncAeXYnPgPPNKjNqdJDavOWK7kyf5+eJFk3n3k2qe//f2wQtQCCGOokclmiVLlrBw4UKWLVtGeno6paWlAMyfP59bbrmFadOm8eSTT/LWW29hs9kwTZOrr76a8847b0CDHwy2SWehbvgb0bVPouVNQfUFumxXeOZYdlU18uwbFYzO8TJ9atfthBBisCjmMKopDLcafBu9bg8tz92L4k7HM+cOVHd6l+3iCZ2fPr6evTXN/PCa6YzJ7ftB4+5InTI1SJ9Tw5DV4AVo2aNxX/IdzKZawi/9EjMe6bKd3aZx039Nw+XU+M3KDRxoig5ypEIIcZAk+B6yjZqCe9YCjJodhF95AFPv+uIfWT4nN//XyTS0xLj3j+vYVyfTGQghhoYk+F6wjT8N1/lfQ9/9IZHXf3fEkTUT89O5reR0onGdn/xxHVv2HBjkSIUQQhJ8r9lPuADHGVeS2PI20TV/PuKwyAl56dxxzXQ8Lhs/e2I96z+tHuRIhRCpThJ8HzhOvQz7SRcT//DlI846CRDM8nDHNdMZE/DywDMb+cd7uwcxSiFEqpME3weKouD87FewTTqb2H+eIvreqiPuyad7HPzgK6dx8kQ/f3z5U1a+vlVOhhJCDApJ8H2kKCqui76B7fhzib37DNE3H8M0uq7JOx0aN10xjc+dms8La3bwu7KPSehdtxVCiP7Sr3PRpBpFteG68BvE0jKJvf8CZssBXLP+G8Xm6NRWU1W+WjiVbJ+TZ96ooKE5yoLLp+F2ykcghBgYsgd/jBRFwXnmF3GeczWJHe/T8sJ9mJGmI7adfe4Evn7pCXy8o57SP71HvYyVF0IMEEnw/cRx0udxfX4BRvV2Wp67F6Ox5ohtzz85n1uuPJnKujD3Ll9HqLZ5ECMVQqQKSfD9yD7xDNyXfh+jpZ6WVfeQ2Lf5iG1PnuTnByWnEU+0jpXfLWPlhRD9SxJ8P7Pln4Bnzg9BsxN+7idE3v4zZiLWZdsJeenc8dUZeN12fvbn9az5cJ+MsBFC9BtJ8ANAyx5D2pV3Yy+4kPiGl2j562L0qq1dts3NdHP7NdMZF/TycFk59/5xnVwCUAjRLyTBDxDF7sJ1/rW4L/0+ZjxKy6p7iP7naUw93qltusfB7VdN52tfOIG6hgg/ffw97l+5QWrzQohjoi1ZsmTJUAfRJhyO0ZcKRVqak5aWrssgQ01Nz8V+wvmY4QbiH/2dxPb1KGkZqN4cFPXg5f0UReG4UT4uPHU0DrvGmo/28fd3d3OgOcb4vHRch10KcDj3eaBIn1OD9LnnFEXB4+k8LLuNDMIeBIrDg+tz12ObMIPIG48Sefl+sLuwjTsV28QZ2Mae3D523unQmH3OeD53Sj7PvVXB6+/v5Y0Neznt+ADnTsvjxAlZaKr88BJCdE8u+DHITCOBvncTiW3vkNj+HmakEWxObONOwf6Zi7DlF3RoX1nXwqvrdvN2eSVN4TgZXgfnnDiKogsm4daUIerF0BhJn3N/kT6nhoG64Ick+CFkGjp66BMS2/5DomIdZqQR+9TzcZ79ZRRnWoe2Cd3ggy21vLUxxMZtteiGyYQ8H2ecEGTG1AA5me4h6sXgGamf87GQPqcGSfBHYYUvhJmIEXtvFbEPXkRx+XCe91XsE6Z32bahOcaHO+t55e0d7KhM9nv8KB8zTshlxtQAuVmewQx90Fjhc+4t6XNqkAR/FFb6Qug124m8/geM2p3YJp6B85yrUT0Zndq19bmqPsy6T6p4d1M1FaEGAMblejlulI9gtofcTDe5WW6CWR6chx2oHWms9Dn3lPQ5NUiCPwqrfSFMI0HsgxeJrVsFdieOaYUomi15mUA9jqkncDsUIgkF+7TC9ouA1xwIs+6Tat7fXEOotpmGlo5DMjO8DibnZ3DK5BxOnuwn/ShH34cjq33OPSF9Tg2S4I/Cql8IvX5vcm++csshzyqg2VHtdoxoBMWVhuvC+djGTuu0fDiaoGp/mMr9LVTtD7OvroXy7XXUN8VQgImj0zl1cg6nTs4hPycNRRneB22t+jkfjfQ5NQxpgq+oqGDhwoXU19eTmZlJaWkp48eP79Bm6dKlrF69Gk3TsNls3HrrrZx//vm9ClYSfGemaUK0GTRb8qZoKIpCIOBj3yflRF59EGP/Huwn/T+cZ17Z5VTFh69vZ2UT72+p4f0tNezYl/x3y0hzMC7oY2yul3FBL2NzvQSzPKjq8En6Vv6cj0T6nBqGNMF/9atf5YorrqC4uJhVq1axcuVKli9f3qHNG2+8wYwZM3C73WzatImrr76aN998E5fL1eNgJcH3XFufzUSM6Nq/EP/o76jZY3HN/BZa9ugObc1IE3rVFvSqCpS0LGwTpqO6fADsb4zywdYatu4+wM6qJvbWNKO3fgYOu8ronDSC2R6CWR6CWe7W+248LvuQ9TmVSJ9Tw5Al+NraWgoLC1m7di2apqHrOmeddRYvv/wy2dnZXS5jmiYzZszghRdeYNSoUT0OVhJ8zx3e58TOD4i8/nvMWBjnGVegODzolZvRK7dg1IdaWymACYqKln8CtglnJJN9aw0fIJ4wCNU2s6uqiZ2VTeyubqJqfwt1DVEO/WTSPXbG5noZG/QxLujluKBvwPf45XNODdLnnusuwXd7JmsoFCIYDKJpyREYmqaRm5tLKBQ6YoJ/9tlnGTduXK+Suzg2tnGn4LnibiKv/57o239OPulMQwtOxnH8uWjByWiBCRgH9pGoeJf4tneIvvkY0beWo+WdgDZqCorDg+Jwk+dwk5/l4eygG8U1GiUti4Spttbzw1TWtRCqbWFnVSOvvLOrwx7/2FwvE/MyOH5MBpPHZJDpdQ7hv4oQqa3fpyr4z3/+w69//Wv+8Ic/9HrZo22JuhMI+Pq87EjVuc8+zGsWEdlVjpaWiT07v/OB0/wcKDgJ07yWWNUOmjetoXnT28TeW3XU99LSMgmk5zAq3Y8tI4CtIAfbWX7wjKYq5mRbncK2UCNbd9fz+vt7eOXdXQAEsz0UjM+mYEI2E0dnkJ/jJT2t76N35HNODdLn/tGvJZr169fzne98h2XLlnHiiSf2Ohgp0fRcf/fZNAyIhzFjLZixMGYsDLEWzEgTRlMdZnNt8m9TLUZTLRw+x72iongyUdKyQNWIxHQi0TiRaIJILIFuGOxLZLI+Np69tjHkZHkJZifH5+dmuslOd5Kd7iLL58SmdT3XjnzOqUH63HPHXKLx+/0UFBRQVlZGcXExZWVlFBQUdEruGzZs4NZbb+U3v/lNn5K7GFqKqoIzrdMUCV1pG9ljNNdhNte1bgD2J/+27AdDx+2043a17qkrCrG4znG12/msawtR1c1WYxLv7h7H8x9lYxwya7UCpHsd5Hjt5GWojA+4OS7gYnS2k2jChV5Tj2noKM40FLcPxeVFUfv2Q9RoqsOMNKJm5nU7+kiIkahHo2i2bt3KwoULaWhoID09ndLSUiZOnMj8+fO55ZZbmDZtGldccQV79uwhGAy2L3ffffcxderUHgcje/A9NxL7bCZiJHZvJLH1PyR2vA+JKLjTSWSMJRFpwYw2o8TD2PQINrPzvPlHXK/dDS4fqtuHlh5AzchDzRyFmjEKNXMUis2JaRgY+3ej79ucPPi8bzNmU23rGhSU9ABa1mjUrNGoWfmomfkonozkRqSPG5D+MBI/52Mlfe45OdHJokZ6n81ElMTODSS2rsVoqEZxelAcHnB4kvedHhS7m7CuUt0YZ199nKpGnZ3VYSJxE7cSw6tGSFOieNUIXiWCT42QozWSqTRz6GCeRsWHkygOM1lWitl8tKQfRyJ7EqonA3tLJbamSuxN+9Caq1BMo2OwzjRUdwaKOx3F5QVDx4xHMONRSCT/mvEIisuLmh5ETQ8k/2bkoqTnonr9KLa+HWwe6Z9zX0ife+6YSzRCDATF5sQ+8QzsE884ajsHkAFMJvmfoLKyger6MC3RBOFognBUJxJLEInphKIJKmI68UgYe7gGZ7gaT6wWX6KOsGFjayyHj8N+anUvVB168DnYejsZDZ2A1kiu2oBPDZNpi5JDjEw9iq+lEQ81aHY7Npcbh9uD6s0GuwvF5sCMNGIcqCK+71OIRzp2RLOjuFpLSi4vitObfOxOby01Je+r7nSwuzBb6jGaajlQ0Uxk397ksY+WehS7C8WTierJTP7CaL1vGjpmpDF5CzdiRhqSfxOx5IVlFAVUDRQVVDX5q8TuSq7P7my/j2aHeAQz2tx6S/6yMuPh5AbMF0D1BVB8Ocm/aZkoipos2xkJSMRaN3zRZL+dacmN9RD+Ckpl8q8uRhRVVQhm9222zM8BhmkSieo0R+I0R+JEojqGaSZvRvJ10zCJJQzqm6LUNUT5qDFCXWOU/Y1R6pui7Vcds2kq+TkexgS8jAl4cXs1DD8YuoEab8YZqcEZqcNjNpOuRfEoERxGBDPahNG0AzPcCLGWo8YcAbA5kr8CPJmY0WaM/XtItByAw39pHMrhSW44bA5MwwTTwDR1MIzkcno8eTH4wzdEh1KU1l9UaSh2N0btLhLN/4ZDz4hQW8+wTsSOHo/NmVyP0wN2Fxg66HHQE5hGAvRE8jnTpFlVkjF3WN6e3AC1b5SS91HV5HJGsm+mcXA9KEryxsG/iqK0buQObuxQ1OQxKNo2+srBu+3LdsE0k7/mTP2wGPTWfwsz+e+NSduXJnnsKL11Y+9LfkZp2Zg5px353+4YSIIXKUVVFDwuGx6XjQC9n0O/7USwPdXN7KpuYndVEx9tr+PfH+47whLe1luSokC2z0lOhptsv5M0h0q6PUq6GsWrRPAQxkGMiOalUfGhe7LZ1wDhmE40ppOWYSMj30mm14bfkSBDC+OlBc1mQ7d7SdjTSNjSiBsKcd1AAdxOGy6Hhtth63QimmkarXvdkeSeeyKO4nAlD7bbXShKxxFNph7HbKzFaKxO3hqqwUgkS1B2Z/Jgtc2Z/FVgmslfALGDvwSINic3LKqGotnaNxBK24YCBbfbTjh86DEYEzMRby2HRSAWwYjWYMYiyUSqaslfKaoGStvf1pP6TFr/Gq15tnUDZxrJkWOm0bpBMNr+QeiwAeuugt32vh1iUA+5Kcl/Q0UB08TYvwdz76bkv8Uh7xP23Anpk4/+Xn0gCV6IXrDbVMYFfYwL+vjsIc83hePE4jqqqqAqSuvf5DUzWyIJag6Eqa6PUF0fTt4/EOHTXQeSZaZY4pA84mi9ATQCjSiAy2nDaVdpjiSIJ460p1zdbfwOu4rLYcNl17DZVGyqgqap2DQFm6aiqQq6YZLQDRK6iW4Y6LpJwjDJTHMcHNqaNYpgYAK5x7tx2DV0wyCeMIglkn/jCQNFAZ/Hgcuh9Woiu5wUqMEnS2pNySu6JWK4J06juaa5399HErwQ/cDrtoO76/l53E4b/gwXU8d1vaxpmkRieusxhQTRuIHToeFx2hiTn0FjQxi1NUGapkk4mmB/U4wDTcmSUX1TDMMwsdtUbJqK3dZ609RkSSqmt56PoCffJ5YgGtNJGCaJhEGiLYnrBtG4iaYmk73ToWBTk8lfURT2N0V5f3NNp2mo2zYKR2LTFLxuOz6PA5/HTprLjqYpaGrbTUVVFWyagttpI5jjxUjopLlseFx2PE4b8YTRWlZL0BxOlteawwlMTNxOG26nDU/rze20Yber7Tvwpmlimof8Tf5D0r6vboLZ2jjZPvlkW1uvy06Wz0mmz9Fv10NWVA3FkwGt13o4/JdSf5EEL8QQUxSlPUkdzuOy09wY6dDW47LjcdkZndP9OQsDoSWSoHJ/S3Ia6rowcd3ArqnY7cmNisOuYddUdMOkKRynMRyjsSVOU0vyfl1DFMNI/jpIGGbyfusGJnbEXyed2W0qCvRqmWOhKJDpdZLlc5Ltc+JxHfy8Dq3kmIBpmBitGxWjdcMC4GwtlbkcWrJ05tTwuR1cnD0wn6UkeCFEr3hcNibkpTMhL737xr2U0A3caS527tlPSyRBcyRBSzSOw6aR5rKR5k7+Akhz2XDYtfZlwoeMqmqJJogndOBgmQyF5Ol0ioLCweOmbaWjZBPl4PFVhfZfTY0tcfY3RqhrSB5or2uMsLu6mXAswaGFp0PLUG3vqyoKSmu5zjQhGm8d9RXVO0zeFwx4GZPd/9dVlgQvhBg2bJpKps9J3N/zPVqbpraWf0bO2ciGaRJtLZkZhskJkwMDctxBErwQQgwy9ShluX59nwFduxBCiCEjCV4IISxKErwQQliUJHghhLAoSfBCCGFRkuCFEMKihtUwycMnQhqsZUcq6XNqkD6nhr70ubtlhtUFP4QQQvQfKdEIIYRFSYIXQgiLkgQvhBAWJQleCCEsShK8EEJYlCR4IYSwKEnwQghhUZLghRDCoiTBCyGERY34BF9RUcG8efMoLCxk3rx5bN++fahD6lelpaXMnDmTqVOn8umnn7Y/b+V+79+/n/nz51NYWMjs2bO56aabqKurA6zd7wULFjBnzhzmzp1LSUkJH3/8MWDtPgM88MADHb7fVu7vzJkzueSSSyguLqa4uJg33ngDGMA+myPcNddcYz777LOmaZrms88+a15zzTVDHFH/euedd8y9e/eaF110kfnJJ5+0P2/lfu/fv998++232x//9Kc/NW+//XbTNK3d74aGhvb7r7zyijl37lzTNK3d5w8//NC8/vrrzQsvvLD9+23l/h7+/7jNQPV5RCf4mpoac/r06WYikTBN0zQTiYQ5ffp0s7a2dogj63+HfjFSqd+maZovvfSSee2116ZUv5955hnz8ssvt3Sfo9Go+aUvfcncuXNn+/fbyv01za4T/ED2eVjNJtlboVCIYDCIpmkAaJpGbm4uoVCI7OzsIY5u4KRSvw3D4IknnmDmzJkp0e8f/vCHvPXWW5imye9+9ztL9/nXv/41c+bMYezYse3PWbm/bb7//e9jmibTp0/nu9/97oD2ecTX4IW13X333Xg8Hq6++uqhDmVQ3Hvvvfzzn//k1ltv5b777hvqcAbM+vXr2bhxIyUlJUMdyqB6/PHHee6551i5ciWmafLjH/94QN9vRCf4vLw8Kisr0XUdAF3XqaqqIi8vb4gjG1ip0u/S0lJ27NjBr371K1RVTZl+A8ydO5e1a9cyatQoS/b5nXfeYdu2bcyaNYuZM2eyb98+rr/+enbu3GnJ/rZp64fD4aCkpIT33ntvQL/XIzrB+/1+CgoKKCsrA6CsrIyCggLL/JQ7klTo9y9/+Us+/PBDli5disPhAKzd7+bmZkKhUPvj1157jYyMDMv2+YYbbuDNN9/ktdde47XXXmPUqFH8/ve/59JLL7VkfwFaWlpobGwEwDRNVq9eTUFBwYB+xiP+gh9bt25l4cKFNDQ0kJ6eTmlpKRMnThzqsPrNPffcw8svv0xNTQ1ZWVlkZmbywgsvWLrfmzdvpqioiPHjx+NyuQAYM2YMS5cutWy/a2pqWLBgAeFwGFVVycjI4LbbbuPEE0+0bJ8PNXPmTB588EGmTJli2f7u2rWLm2++GV3XMQyDSZMmceedd5KbmztgfR7xCV4IIUTXRnSJRgghxJFJghdCCIuSBC+EEBYlCV4IISxKErwQQliUJHghhLAoSfBCCGFRkuCFEMKi/j8nFsDQf5DxrQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt \n",
    "import numpy as np \n",
    "sns.set()\n",
    "\n",
    "x = np.linspace(0, N_EPOCHS,N_EPOCHS)\n",
    "\n",
    "plt.plot(x,t_loss)\n",
    "plt.plot(x,v_loss)\n",
    "plt.title(\"Loss\")\n",
    "plt.legend([\"Train loss\", \"Valid loss\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f1c17510dd0>"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAELCAYAAADTK53JAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXzU1b3/8df3+50l+0oShs0ACgYrimhbCm6ojYVgqFbpj1q7KN4Wl1brvVLbstTWFnrrWm1vaavlxnv1KlVq5OJ6XbCK1hWNoEBYJEOWmYRkksks3+/5/TEhELJNQkKS73yeD/KYycz5Ts5JwntOzvd8z9GUUgohhBC2ow91BYQQQgwOCXghhLApCXghhLApCXghhLApCXghhLApCXghhLApCXghhLApx1BXQIhjNXfuXOrq6jAMo/2xTZs2UVBQwM9+9jPefPNN9uzZwx133MGll146hDUV4viSgBe28Ic//IEvfelLnR4/+eSTmTdvHr/5zW+GoFYdRaNRHA75LyeOHxmiEbb2jW98g1mzZuF2u3st+/LLLzNv3jxmzJjB2WefzZ///Of2555//nlKS0s544wzuPDCC3nllVcAqK6u5nvf+x6f//znueiii/if//mf9mPuu+8+brzxRm655RbOOOMMnnjiCZqamrjtttuYM2cOZ599NnfddRemaQ58w4VAevBCtPvJT37C3XffzZlnnsnBgwf57LPPAPjggw+49dZbuffee5k1axa1tbUEAgEAfvSjH3HiiSfy6quvsmvXLr7zne8wfvx4Zs2aBcALL7zAPffcw5o1awiHw9x8882MGjWKZ599lmAwyL/8y7/g8Xj4+te/PmTtFvYlPXhhC9dddx1nnnkmZ555JkuXLu3XazgcDnbs2EEgECAzM5NTTjkFgMcff5zLLruM2bNno+s6BQUFTJ48Ga/Xy9tvv80tt9yC2+2mqKiIyy+/nA0bNrS/5umnn86FF16IrusEAgFeeeUVbrvtNlJSUsjNzeXb3/42Tz/99IB8D4Q4mvTghS3cf//9XY7B98W9997L73//e377298ydepUfvSjHzFjxgy8Xi/nnntup/I1NTVkZmaSlpbW/tiYMWP48MMP2z8fPXp0+/2qqiqi0Shz5sxpf8yyLDwezzHVW4juSMAL0Wb69On8/ve/JxKJ8PDDD/PDH/6Ql19+GY/Hw969ezuVz8/P5+DBgwQCgfaQ93q9FBQUtJfRNK39/ujRo3G5XLzxxhtyslUcFzJEI2wtHA4TCoVQShGNRgmFQliW1WW5v//97zQ1NeF0OklNTW2fdvm1r32Nv/3tb7z++utYlkV1dTU7d+7E4/EwY8YM7rzzTkKhENu2bePxxx9nwYIFXdYlPz+f2bNn8+tf/5pAIIBlWezdu5c333xzUL8HInFJwAtbu/rqq5k+fTrvvvsuP/vZz5g+fTpvvfVWl2U3bNjA3LlzOeOMM3jkkUdYs2YNEOvZ/+pXv+KOO+5g5syZXHnllVRVVQFw5513sn//fs4++2yuv/56brjhBmbPnt1tfdasWUMkEmHevHmcddZZ3HjjjdTW1g58w4UANNnwQwgh7El68EIIYVMS8EIIYVMS8EIIYVMS8EIIYVMS8EIIYVNxXW1RWVnJsmXLaGhoICsri9WrV1NYWNihjM/n48c//jFer5dIJMIXv/hFfvrTn8oFHUIIMUTimiZ51VVXcdlll1FaWsqGDRtYv34969at61Dml7/8JQ6Hg1tvvZVIJMLixYv5zne+w7x58+KuTH19M5bV91mbublp+HyBPh83kkmbE4O0OTH0t826rpGdndrt8712r30+HxUVFTz44IMAlJSUcPvtt+P3+8nJyWkvp2kazc3NWJZFOBwmEol0uGQ7Hpal+hXwh45NNNLmxCBtTgyD0eZex+APra1x6LJtwzDIz8/H6/V2KLd06VIqKyuZM2dO+8fMmTMHvMJCCCHiM2AD5Js2bWLq1Kn89a9/pbm5mSVLlrBp0yYuvvjiuF8jNzet90LdyMtL7/exI5W0OTFImxPDYLS514D3eDxUV1djmiaGYWCaJjU1NZ2WOC0rK+OOO+5A13XS09OZO3cuW7Zs6VPA+3yBfv2ZkpeXTm1tU5+PG8mkzYlB2pwY+ttmXdd67Bj3OkSTm5tLUVER5eXlAJSXl1NUVNRh/B1g3Lhx7duYhcNhXn/9dU466aQ+V1gIIcTAiGse/MqVKykrK6O4uJiysjJWrVoFwJIlS9i6dSsAt912G2+//TYLFixg4cKFFBYWcsUVVwxezYUQQvRoWK0mKUM08ZM2JwZp8/CllOqwocuxGKwhGrkKSQgheqGUoqqume37Gti+t4FP9jXQ3BolI9VJeoqLjBQXGSlO0lNdJLsdKBWb8q0UWCp2q1C4HAYup47babTfT012cs4xTDDpiQS8EAIA07KobwzhdBqkuA2cDqPbskopoqZFMGQSNWM7ZLUHWdttKGLS2BzmYHP48G1LmOZgNFaGw2UPvWYkaqHQaGmNEImahKMWkaiFoWs4HDpOQ8fpiH24HAYKRTSqiFoW0ahF1LSImLEXTHIauF2xjySXQZIz1qaIGSsbPuL1o6aFy2GQ7DZIdjtIdjlIcseO89a1sH1fA4FgBIDsdDdFhdlkpblpagnT1BKhsTnM/roAjc2R9u8HgEbsGiG9bTA8anY9QpGRnsz43ORj+wF2QQJeiCGilKKxOUztwVbqGoLUNgSpPdhKoCVCXlYyY0alcMqJ+aQ4ICXJ2evrRU2L1rBJOGLSGjYJRUyUAvehHqPTaLvViUQt9tUG2FcdYG91E3uqA3xWGyASPRxODkMjxe0g2e0gye2IvX7IpDUcpTVsYvZxONXl0MlIdZGa7ETXNDSN2Acabf9wuwzSUtwoy42rLcgdDh1lQcQ0ibQFciRqEY5aaBqkJuk4DB2HoeE02sor2r8PreEozcEIvoOthCNW25uDjtMZe5NIcTtwGDrhiEkwHKUhECYYitIajhIMmYzKTOK0yblMmZDF1AnZ5GUmdTs0o5TCtBSaRlsbO5azLBV7Y4lYhCKxn5UCTp+aR13dwF+9KwEvxACIRE2CYbOtZ3gogGKfB4IR6ptC1AdCNDSF2u6HqW+KBc6RMlNdpKU4qdjtJxy1gO0AZKW58OSmomm0BcPhgAhFLMKRvgWuBhwqnex2cEJBGufPGIsnN4WoqQiGou0fLaFY0DkMLRb2LqP9NsnlwGHEguxQqEHs1unQyUxzkZEaG8JIchlxjVkPpzH4vo6za5qGw+i+vK5rJLkcJLk6HzcYJOCF6EUkauJvDOFrbMXX2BoL6LYPf2OIhkCo/c/3nhi6Rlaam+x0N+Pz0zhtci55WcmMykxiVNut2xkbFrGUwnewlUDEYtvOOvbXNVPtb0HTNNxOg4wUFy6ngcuh43LGhhJcTuPwsETbrUbsDeHIN4VQ2ETTYHx+GhMK0hnVQ490oCgzilW3B7NuN6rZj549FmPUCWgZ+Wha3xe1VUpBJIiKho94tO1PAkDTDXC60fT+R5wyo6hwSyzkdQPaPxxoet/qrJSCaDj2eqFmVLgFQs2oUAvKimJlXNDvevZEAl6MWEopgiGTxpbYGG9jczj257tpYZoWUbNtbNZUh8dnj7iNmNbhWVua1jZeGvvUNBX+plZ8jSEam8OdvnZ6ipPsdDe5GW5OHJdJdpqLlCRn+/iw09BxOWO3KUmxsmkpzvYebrdtsiyItqLpDkZlJjEtP4OJeYcXk1JKoZrrsQ4ewGqowmrwYh2sRjPScRQUYYwpQs/I68f30oJoGMwouFO6DV2lFFZ9Fab3Y8yqbZgHPkGZEfTUXLS0HPTUnNhtWg7KMrFqd2PW7cby7QMr2vkFnUkYuRPQR52AnjMOTdNp/MxJ+GAzWCZY0VjQtgZQLQ2olgastluinX8unRgONEcSuJJit04XaHqsfZoGmh77AFQ0BKGWthBuAbOn19fAcIDDhWY4225d4HDGXi8aQkVCR9yGOfw309EvpRE6YRIkj++9PX0k0yRHKDu0ORI1aWqJnZRKajux5XToR5WxqKlvwetroTEUZefeBqrrWzgYCNHYEukwZtwbpyM2Vus0tPb7un44cA/NdMhV9YzXqslOssh0WaQ7TVKNKEl6FLcWxZmUjJGUhuZORXOnxgLRnQpoYEbAjKDMCETbbi0TlNX+oay2+2YEFWxEtTbFPoJNqFCA9rOOmo7mdIPhivVGDQdWwA+R1iMalYyeWYBq9qOCjbHD0nIxxpyMY0wRWlpuLBibDwejammIfa1oCCKh2K15xF8gmoGWkomWmo2ekoWWmoWWlIHl34fp3Y5qjf3eaak5GGNORnMmo5r9WAF/rB6tR/xeupIxRhVi5E1EzyvEGFWIlpqNVV8V69H79mDV7cX07YVoqIcfXhJaShZ6SiZaSjZaSiZ6ShY4k2gPTqUO3zfNtva1oto+2tuqrLYfduxWtf1cNGcymisZzZ0CrhS0tg90PfbGZ5koy+zwxoMZbvs5hzv8vDWnGxxuNIc79rNzuGJtcKW0/d60vb47BS0pnfxxowdlmqQE/Ag1ktq8v66ZV9+vwutroaklTDAYJCfk5QRtPyc6qsnRm2MzKtBQHOpVaUQxaIi6abKSaLSSabKSUEnpuNMyyXGbZDlCpOtBUgmSZDXjijajW1G0Q/9h1aFgNdFTsjAKTsTIn4xRMBkto6B9WEKZEUzvdqJ73ye65z1UU23HBjiT0JxJaK5kcLhiPbJQMyrUHHv9eB3qLWo66Dqa4URLSkdLTo/dHrrvSokFRSREklMRbGpCRcJghtFSc9CzPOjZY9CzPGjJmWiaFutZN1RhVrX1rKu2xd4sjuRwoaVkx0IyOaMteNyxxx3uWCjpBirYhNVc3/bGUI/VUg/hYFugF+EYczKG52S09FFdDu2oaBjVXB9rckZeXEMwyrJQzT5QkJuXia+htcOwiGb0fpJ5JBusefAS8CPUcGizUhaEg7E/n0OBtttmUApTwS5vExW766nyBdF0jSkZLRRqXgqiXgxisweCKWMIpeRjWgrTNLFMC8uysEwTQ0VJ01tJtppxRJpjgX00TUdLzogFXXJ6LLB0I9br0ozYWKmmYzXVYdbsbO/9au409ILJaLqD6P6PYo8bToyx03BMOA3H2GmxEHQkdTveGhsHbo2NqYaaY8M8hhPaPjRH233d6Nc4M/T/56yUheXfjwoF2nq72bE3qn6OtatoONamQR6rh+Hxu328yYVOYsio1gCm/zOs+s+w/Pux6vdjNXg7Did0YXzbB4cWyTM19NwJGGMujPUCR08hw939ZgUd6qAUuekatfv2o0LNaK5UtJSM2J+7cYansqxYL7d6B1bNTszqnVjREM4Tv4hjwukYY4tibxBx0jQNXLE/60kfFfdxx4Om6Ri5AzemqzlcvRcSw44EvM2YDVW0/t+f0LPH4px8FsbYaXHNJFBKoVqbYuFdXxU7gVcf+1DBg4cLulIwcsahTZhBs0rGH3JQ3aJR1QR76hUNYSearjHthGy+MC2PyZ50NABloaflto1V952maRjJ6RjZY/t1PICm6xg54zByxkHRef1+HSFGCgl4G7EOVhMsX4MyI1gNXqKfvAruVJwTZ+KY9IXYCTHdiPXI6/dj+T+LBbr/M8z6/RBqPvxiDjdm+mhask6iIXcU1SqbfeFMPmty4NsXm899qO+e5DJiU+6mpTO7IDb9LzMt/p6wEGJwSMDbhNVUR8vTa8CMkrLgx+gZ+ZiffURk1xYiO98ksu0VtKT02Em0lob245QzmZakfHzuKVQ7s9gXSmNHUyr7W5xQc3i81dA1cjIscjN0phVmMyozmXF5qYxvm0fd2/Q/IcTxJwE/jJl1ewi9/t9YjTW4ZpTgPPm8Lk/4WQE/LeWrUeEgKSW3xoYgAEfhDByFM1DRMNF9HxDe9U+aWy0OZGaxI5DKu7Vu9vudgIbD0MnNTGJUZhInepL4YmZS2+fJ5GYkkZnmkhAXYoSRgB+GrGAj4X/+jci2l2MnE7NGE9q8jkjFi7hnLcYxdtrhsi0NtDy9BtXaRMr8f8MYdQIHAyGq6po54G/B62+h2h/kgD9M3cEp7edE87OTOXFSJheMy+SksZl4RqVKgAthMxLww4iyokQ+eoHQ209CJITzlAtxz1wIrhSilf8ktOVRgk+vwVF4BpF5V2MFTYJPr0E117P3lKt5590IFXveoNrf0v6aLqfO6OwUCkdn8MVpo5lQkM6J4zLJTJVZEULYnQT8EFGWhWptPHz5dcBP5MPnsBq8GOM+h3vW/+swY8Q56SwcE04jvPUZwu+Ws+8PPyCop+KINvMfjXP59P8CuF1Bpo7P4tzTxjC+IA1PTgpZ6W7pmQuRoOIK+MrKSpYtW0ZDQwNZWVmsXr2awsLCDmX+7d/+je3bt7d/vn37du6//34uuGBwFtEZiVSomdaX/4xZW4lqOdjpKkgto4Dk4h9gTDi9ywtKNIeLmrHn82JlHmP3P8d01142Jc3nlFmnc/kJORR60nEY/bugRghhP3FdyXrVVVdx2WWXUVpayoYNG1i/fj3r1q3rtvy2bdv41re+xauvvorLFf9QgJ2vZFXhIC0bf4NVtwfHiV+MLcqUkhVbXyM1q+1+drdXTe6sOsjT/9jDezvqSHIZzJ89kTmnFCTUdMSR8HMeaNLmxDBkV7L6fD4qKip48MEHASgpKeH222/H7/eTk5PT5TGPP/44CxYs6FO425kKB2n5399i1e4h6aLrcBaeEddxUdPiw11+nn97HxW760lNcrBwzkQuOHMcheNzEu4/gRCib3oNeK/XS0FBAYYRW6faMAzy8/Pxer1dBnw4HOapp57ioYceGvDKjkQq0kpw011YNbtIuuD7vYa7UoqdVY28/tEB3vq4hkAwQkaqiyvOP5FzTx9DsltOmwgh4jPgafH8888zZswYioqK+nxsT39q9CYvL733QseZFQlx4NHfYFbvIH/hD0mbNrvbsrX1QZ7dsoeX3tnHAV8LLofOFz7n4byZ4zhjan6XY+vDsc2DTdqcGKTNA6PXgPd4PFRXV2OaJoZhYJomNTU1eDyeLsuvX7+eyy67rF+VsdMYvIqGCT5zN2bVxySdfy3BvOkEu6mj19fMr8reoTkYoagwm/lfPIEzpuS199br/c2djhmObR5s0ubEIG2OX29j8L1OucjNzaWoqIjy8nIAysvLKSoq6nJ45sCBA7z99tuUlJT0uaJ2oiKtBJ+9F3P/xySdew3OE2d1W7buYJB/f+Q9dA1+fs0XuOXrM5h9qkeGYoQQxyyuOXUrV66krKyM4uJiysrKWLVqFQBLlixh69at7eWeeOIJzj//fLKysgantiOAWV9Fy5M/x/zsI9znfBvnlO6HZQ4GQvz7I+8RCpvcvOh0xo7q30qLQgjRFdnwYwBFdrxB6ysPojlcJM39Ho5xp3RbNhCMsOa/3qGmIcgtX5/BiWMz+/S1hkubjydpc2KQNsdPNvw4DpQZIfT6I0QqXsAoOImkC5eip2Z3W741HOXux97ngL+FH1x+Wp/DXQgh4iEBf4yspjqCzz+AVbsL56nFuL9weY8bbESiJvet38pubxPfX/g5Tins+loCIYQ4VhLwx8Cs3kHLprvAski66HqcE8/ssXw4YvKHDR/x8Z56rp5fxMypeceppkKIRCQB309WUx3BZ+5Bc6eS8pWb0TNH91i+IRDivvUfsNvbxDe/PIXZp3Y9zVQIIQaKBHw/qHCQ4Ka7UVaU1Itv6jXc9xxo4t71H9DSGuX6y05lxknScxdCDD4J+D5SlkXwxT9gNVSR/JUfoWf13BN/e3sta8s/Ii3ZyY+vPIMJBYl3hZ4QYmhIwPdRaMujmHvfxz3nqh6nQSql2PjGHta/vItJYzK44dJTE2rlRyHE0JOA74PwtpeJbH0G5ykX4po2t9typmXx4MZt/OPDA3xhWgHfnXcyTodxHGsqhBAS8HGLVn1M6NV17bst9aT8H3v4x4cHKJ0zkUtmF3a5eYcQQgw2Cfg4WAerCT73O/SsApIvXIqmd98b31l1kKde282sUwoonTPxONZSCCE6kv3d4tC6ObZ7VXLxD9FcKd2XC0dZ+1QF2ekuvnHR1ONVPSGE6JIEfC/M2t2Y+z/Cddp89Iz8Hss+8sIOauuDXFMyjZQk+eNICDG0JOB7EX5/IziTcU07r8dy735ayyvvV3HxFyYwdUL369AIIcTxIgHfA+vgAaKVb+E6ZW6PQzMHm8M89L/bmJCfxsKzJx3HGgohRPck4HsQfn8T6AbOz13UbRmlFA9u/JhgyGTJgmk4HfItFUIMD5JG3bBaGoh8shnnlLPRU7rfwOTl96r4YKePy8+fzNi8/u8pK4QQA00CvhuRrc+CMnGd9pVuy1TXt/DIi59ySmE2F8wcdxxrJ4QQvYsr4CsrK1m0aBHFxcUsWrSI3bt3d1lu48aNLFiwgJKSEhYsWEBdXd1A1vW4UeEWwhX/h2PiWT3OnHn8pZ3omsZ3509Dl4uZhBDDTFxz+VasWMHixYspLS1lw4YNLF++nHXr1nUos3XrVn73u9/x17/+lby8PJqamnC5XINS6cEWrvg/iARxnT6/2zKV3kbe3l5L6ZyJZKfLGjNCiOGn1x68z+ejoqKCkpISAEpKSqioqMDv93co99BDD/Hd736XvLzYUrjp6em43SMv+FQ0TGTrMxjjPocx6oRuyz3xyi7Skp18+azxx7F2QggRv14D3uv1UlBQgGHELs83DIP8/Hy8Xm+Hcjt37mTfvn184xvf4Ktf/SoPPPAAw2g/77hFPnkNFWzssfe+fW89H1b6mffFE0h2ywVNQojhacDSyTRNtm/fzoMPPkg4HOaaa65hzJgxLFy4MO7X6Gl38N7k5R37OuvKMtn30TO4x5zE6OlndblImFKK3zzyHjkZSVxRfDJu59CtEjkQbR5ppM2JQdo8MHoNeI/HQ3V1NaZpYhgGpmlSU1ODx9Nxo4sxY8Zw8cUX43K5cLlcXHDBBXzwwQd9CnifL4Bl9b3Xn5eXTm1tU5+PO1pk55tE6w/gOPNr1NUFuizzwc46Pt7t56riqTQ2tBzz1+yvgWrzSCJtTgzS5vjputZjx7jXIZrc3FyKioooLy8HoLy8nKKiInJycjqUKykpYfPmzSiliEQivPHGG5x88sl9rvBQUUoRfn8jeuZoHIVndFnGUoq/vbyLvKwk5kyXPVWFEMNbXNMkV65cSVlZGcXFxZSVlbFq1SoAlixZwtatWwGYP38+ubm5zJs3j4ULF3LiiSfyta99bfBqPsCs2kqsut04P3cRmtb1t+Wf22rYWxNg4dmTcBhyCYEQYnjT1DA6EzqUQzTBl9YSrXybtG/cheZK7vS8aVn8dO0WHA6dVd/5PLo+tPPe5c/YxCBtTgxDNkSTCFRrgOjON3Ge9KUuwx3gta0HqK4PcunZk4Y83IUQIh4S8EDkk1fBjOCcdn7Xz0dN/v5aJRM9GZx+0qjjXDshhOifhA94pSzCFf+HMXoKRk7XFy299F4V/sYQl507SfZXFUKMGAkf8Ob+ClRjDc5pc7t83rIUz721j5PGZTKtMKfLMkIIMRwlfMBHPnoBLSkdx8SZXT7//o466g62ctGZsiSBEGJkSeiAtwI+onvfw3nyuWiGs8syz7/9GdnpbmZMkbF3IcTIktABH/n4JVDgLDq3y+f31zXz8Z565p4xFkNP6G+VEGIEStjUUmaUyLZXMCZMR0/P67LMi29/hsPQOee0Mce5dkIIcewSNuCju99BBQ/i6ubkaktrlH98eIAvTMsnPWVkrmsvhEhsCRvwkYoX0NLzMMad2uXzm7d6CUVMLpwpJ1eFECNTQga8Wb8f07sdZ9H5aF2MrVtK8eI7n3Hi2ExOGJ14y5YKIewhIQM+UvEi6A6cU+d0+fyHu3zU1AdlI20hxIiWcAGvrCiRT/6BY9JZ6MkZXZZ5/p+fkZnmYubUrk++CiHESJB4Ad9cD5Egxpiu16r3+pr5sNLP+TPGypLAQogRLeESzArENgvX03K7fP7Fd/bjMDTOPX3s8ayWEEIMuIQLeBXwAV0HfDAU5bWtXs46OZ/MVJkaKYQY2RIu4A/14LW0zguH/ePDA7SGTS6QqZFCCBvoddNtgMrKSpYtW0ZDQwNZWVmsXr2awsLCDmXuu+8+/uu//ov8/HwAzjjjDFasWDHgFT5WKuBDc6ehOdydnnv1gyometKZNKbrk69CCDGSxBXwK1asYPHixZSWlrJhwwaWL1/OunXrOpVbuHAht95664BXciBZAR9aF8MzoYjJvpoA82cVHv9KCSHEIOh1iMbn81FRUUFJSQkAJSUlVFRU4Pf7B71yg0E1+9G7GJ7ZVxNAKSiUC5uEEDbRaw/e6/VSUFCAYRgAGIZBfn4+Xq+XnJyOQfn000+zefNm8vLyuOGGG5gxY0afKtPT5rG9ycuLL5ibm/2kTJrOqKPKb9leC8AZ0zzkZXe9L+twE2+b7UTanBikzQMjriGaeHz961/ne9/7Hk6nk9dee42lS5eyceNGsrOz434Nny+AZak+f+14dyRX4RasUAshI61T+Q931JKe4kRFItTWRvtch+NNdp5PDNLmxNDfNuu61mPHuNchGo/HQ3V1NaZpAmCaJjU1NXg8nqMqmIfTGds0Y/bs2Xg8Hj799NM+V3gwHZ5B03kMfs+BJgpHZ8ieq0II2+g14HNzcykqKqK8vByA8vJyioqKOg3PVFdXt9//+OOP2b9/PxMnThzg6h4b1c1FTuGISVVdiywsJoSwlbiGaFauXMmyZct44IEHyMjIYPXq1QAsWbKEG2+8kVNPPZU777yTjz76CF3XcTqdrFmzhry84bWWi9V2kdPRPfh9NQEspeQEqxDCVuIK+MmTJ/PYY491enzt2rXt9w+F/nCmAj7QDLTkzA6P7z4QG/uSgBdC2ElCXclqBXxoqVmd1oDffaCR9BQn2emdL34SQoiRKqECPjYHvusTrCeMTpcTrEIIW0mogLcC/k7j74dOsBaOluUJhBD2kjABryyry6tY5QSrEMKuEifggwfBMjv14OUEqxDCrhIn4GxGTkcAABj9SURBVJsPzYHv2IPfc6BJTrAKIWwpYQK+uznwuw80yglWIYQtJUzAt+/klHq4B3/4BKsMzwgh7CdhAt4K+MGZBK6U9scOn2CVGTRCCPtJmIBXgdgc+COHYuQEqxDCzhIm4LvayUlOsAoh7CxhAl4FfB3G3yHWg5cTrEIIu0qIgFfRMKq1CS3t6BOszTI8I4SwrcQI+ObO68DLCVYhhN0lRMB3tZOTnGAVQthdQgR8+xz4I4Zo5ASrEMLuEiLgY1examiphzcAlxOsQgi7iyvgKysrWbRoEcXFxSxatIjdu3d3W3bXrl2cdtppw2qHJxXwo6VkohmxTcHlBKsQIhHEFfArVqxg8eLFPPPMMyxevJjly5d3Wc40TVasWMGFF144oJU8VrE58IeHZ/bVxk6wnlAgJ1iFEPbVa8D7fD4qKiooKSkBoKSkhIqKCvx+f6eyf/zjHznvvPMoLCwc8Ioei6PnwO9pO8E60SM9eCGEffW66bbX66WgoADDMAAwDIP8/Hy8Xi85OYdDc9u2bWzevJl169bxwAMP9Ksyublp/ToOIC+v67BWShFo9pN68lnktpWpbmglM83FlEmjRvQYfHdttjNpc2KQNg+MXgM+HpFIhJ/97Gf86le/an8j6A+fL4BlqT4fl5eXTm1tU5fPqdYAKhqmVT9c5uNKP+Pz06irC/S7rkOtpzbblbQ5MUib46frWo8d414D3uPxUF1djWmaGIaBaZrU1NTg8Xjay9TW1rJ3716uvfZaABobG2M950CA22+/vc+VHkhHrwMfNS2q6po57cTOm28LIYSd9Brwubm5FBUVUV5eTmlpKeXl5RQVFXUYnhkzZgxbtmxp//y+++6jpaWFW2+9dXBq3QfWUXPg/U0hLKUoyE7p6TAhhBjx4ppFs3LlSsrKyiguLqasrIxVq1YBsGTJErZu3TqoFTxW6qgevP9gKwA5GXKBkxDC3uIag588eTKPPfZYp8fXrl3bZfkbbrjh2Go1gKyAHwwHWlLsBIa/6VDAJw1ltYQQYtDZ/kpW1bYO/KHZMv7GEAA5skSBEMLmbB/wVrO/wxx4f2MraclOXM7+z/YRQoiRwPYBr47aycnfFJLxdyFEQrB1wCsrimpp6LAOvL+xlVwZfxdCJAB7B3xzAyjVYR0aX2OInHQJeCGE/dk64A/PgY/14IOhKMFQVIZohBAJwdYBf3gO/OGLnECmSAohEoOtA946tBdrattFTo1ykZMQInHYOuBVwI/mTkNzxgK9PeBlDF4IkQBsHfBHb/ThbwyhaZCV7hrCWgkhxPFh64BXAV+nKZJZaW4M3dbNFkIIwOYBbwX8HXvwcpGTECKB2DbgVTgI4ZZOPXgZfxdCJArbBrwViM2g0drWoVFK4W8KyVWsQoiEYduAVy31AGip2QA0BSNEohbZMkQjhEgQ9g341th+q1pybB34+vZlgqUHL4RIDHFt+FFZWcmyZctoaGggKyuL1atXU1hY2KHM+vXreeihh9B1HcuyuPzyy7nqqqsGo85xUa2xDWzbN/pomwOfmyk9eCFEYogr4FesWMHixYspLS1lw4YNLF++nHXr1nUoU1xczKWXXoqmaQQCARYsWMDnP/95Tj755EGpeG9iPXgNzZUKgE8uchJCJJheh2h8Ph8VFRWUlJQAUFJSQkVFBX6/v0O5tLS09l2TWltbiUQi7Z8PBdXahOZORWub8+5vCuEwdNJTnENWJyGEOJ56DXiv10tBQQGGEdsByTAM8vPz8Xq9ncq+8MILzJ8/n/PPP59rrrmGqVOnDnyN46RaA2hJae2f+xtbyclwD+mbjhBCHE9xDdHE64ILLuCCCy6gqqqK6667jnPOOYdJkybFfXxublrvhbqRl5fe4fMqK4hKz2p/vCkYZXRuaqdyI5md2hIvaXNikDYPjF4D3uPxUF1djWmaGIaBaZrU1NTg8Xi6PWbMmDGceuqpvPTSS30KeJ8vgGWpuMsfkpeXTm1tU4fHwo0N6Omj2h+v9jdz8oTsTuVGqq7abHfS5sQgbY6frms9dox7HaLJzc2lqKiI8vJyAMrLyykqKiInJ6dDuZ07d7bf9/v9bNmyhSlTpvS5wgNFtTa1z6AxLYuGprCsAy+ESChxDdGsXLmSZcuW8cADD5CRkcHq1asBWLJkCTfeeCOnnnoqjz76KK+99hoOhwOlFFdeeSVz5swZ1Mp3RynVYQz+YCCMpZSsQyOESChxBfzkyZN57LHHOj2+du3a9vu33XbbwNXqWEVawYoeMQdeLnISQiQeW17J2n4Va1sP3t8kOzkJIRKPTQO+41Wshy5ykoXGhBCJxKYBf1QPvjFEstsg2T2gs0KFEGJYs2nAd16HRsbfhRCJxqYB37kHL1MkhRCJxqYB3wSaDq4UIHaSVU6wCiESjU0DPjYHXtM0whGTppYIOekS8EKIxGLTgD98FWt9U9sceBmiEUIkGHsGfChwxPj7oTnwEvBCiMRiz4A/ogfvb+/ByxCNECKx2DTgD/fgD+/kJAEvhEgstgt4pay2gD+8Dk1GihOnwxjimgkhxPFlu4AnHARldViHJlvG34UQCch2Ad/5KtaQDM8IIRKSDQP+6KtYW2WRMSFEQrJhwB/uwbe0RmkNmzJFUgiRkGwY8Id78IfnwMsQjRAi8cS1fm5lZSXLli2joaGBrKwsVq9eTWFhYYcy999/Pxs3bsQwDBwOBzfddBNnn332YNS5R0f24P01sbCXHrwQIhHFFfArVqxg8eLFlJaWsmHDBpYvX866des6lJk+fTrf/e53SU5OZtu2bVx55ZVs3ryZpKTjG66qNQCGAxxu/I0+QObACyESU69DND6fj4qKCkpKSgAoKSmhoqICv9/fodzZZ59NcnIyAFOnTkUpRUNDwyBUuWeHrmLVNA1fYyu6ppGVJgEvhEg8vQa81+uloKAAw4hdKGQYBvn5+Xi93m6PefLJJ5kwYQKjR48euJrG6cirWP2NIbLTXei6dtzrIYQQQ23A97B78803ueeee/jLX/7S52Nzc9P6/XXz8mLz3vebLTjTs8jLSyfQGiU/J7X9Obuxa7t6Im1ODNLmgdFrwHs8HqqrqzFNE8MwME2TmpoaPB5Pp7Lvvvsu//qv/8oDDzzApEmT+lwZny+AZak+H5eXl05tbezkarjpIEbuBGprm6j2NVPoOfycnRzZ5kQhbU4M0ub46brWY8e41yGa3NxcioqKKC8vB6C8vJyioiJycnI6lPvggw+46aabuPfeeznllFP6XNGBcmgM3lIKf5Nc5CSESFxxzYNfuXIlZWVlFBcXU1ZWxqpVqwBYsmQJW7duBWDVqlW0trayfPlySktLKS0tZfv27YNX8y4oy4RQC1pSGk0tEaKmkimSQoiEFdcY/OTJk3nsscc6Pb527dr2++vXrx+4WvWTCjUDKjYHXpYJFkIkOFtdyXrkVay1DUEARmUlD2WVhBBiyNgs4A9fxVpdHwv4fAl4IUSCslnAH+7B1/hbyEpz4XbJRh9CiMRks4A/ogffEKQgO2WIaySEEEPHZgHfsQefny3DM0KIxGWzgG8Ch5tWU6exJUJBjvTghRCJy2YBH1uHpkZOsAohhN0CvqltBk0LgPTghRAJzWYBH+vByxRJIYSwXcDHevAyRVIIIWwX8G09eJkiKYQQ9gl4ZUYhEmzvwcsUSSFEorNPwIdic+CjjhSZIimEENgp4NuuYj0YdQFyglUIIQZ8y76hcugqVn/IAUSlBy/EMGCaUerra4lGw3EfU1OjY1nWINZq+OmtzQ6Hi+zsPAyjb5Fto4CP9eBrgrGZM9KDF2Lo1dfXkpSUQmrqaDRNi+sYh0MnGk2sgO+pzUopmpsbqa+vZdSozlul9sRGQzSxHry3SZMpkkIME9FomNTUjLjDXXSmaRqpqRl9+ivokLgCvrKykkWLFlFcXMyiRYvYvXt3pzKbN2/m0ksv5XOf+xyrV6/uc0WO1aEe/L5GZIqkEMOIhPux6+/3MK4hmhUrVrB48WJKS0vZsGEDy5cvZ926dR3KjB8/nl/84hc888wzhMN9f6c5Vqo1AK5kDtSHOO3E9OP+9YUQw9uSJd8iEokQjUbYt28vEydOBmDKlKncdtuKuF7jyScfJxQKsWjRN/r0ta+//lqqq6tJTU0FYMKEE/j5z3/Fp59u5847V/PJJ58wa9aX+MUv1vStUb3oNeB9Ph8VFRU8+OCDAJSUlHD77bfj9/vJyclpL3fCCScA8MILLwxRwDeBO02mSAohurR27V8B8HqruOaab/LQQ//VqUw0GsXh6D4WFy78Wr+//g9/eAuzZ5/d4bHs7Byuv/5mdu36hC1b3uj3a3en14D3er0UFBRgGLExbcMwyM/Px+v1dgj4oaZaA0QdsXdHOcEqhIjX1762gJKSUt5++y3GjBnLtdcuZeXKn9Dc3Ew4HOZLX5rN0qU/AODPf/4PgsEg11//QzZufIrnnttEenoGu3btJD09jV/8Yg25uaPi/tqjRuUxalQe+/btHpS2DatZNLm5af0+1oi2EHXGAv7kyaPIy7P/ME0itPFo0uaRpaZGx+E4fKpv8wdVvPJe1aB8rXNOH8Oc6WN6LWcYOqB1qJff7+P3v18LQCgU4re/vYeUlBSi0Qg/+MF1vPXW68yaNRtd19D12LG6rrFtWwVlZY9SUDCaO+64nb/97X/4/vev7/Q1NU3jnnv+nT/96fcALFr0/ygpKe1U5sg6HU3X9T7/LvQa8B6Ph+rqakzTxDAMTNOkpqYGj6dv03Xi4fMFsCzV5+Py8tKJBA7S4Ig13qkUtbVNA129YSUvL932bTyatHnksSyrw/Q/01SoXv6Laxq9lumKaaq4pleapgV0LPvlL89r/zwcjnL//XezdesHgMLn87Ft23bOOmsWlqWwrNixlqU49dTTyM3NJxq1mDbtFN56a0uXdVBK8YMfdByiObqcUj3X37KsTr8Luq712DHuNeBzc3MpKiqivLyc0tJSysvLKSoqGlbDMxAbgz/odskUSSGGsdmneph9as+dw6GYB5+ScnhY99FHH6apqZE//vEh3G43q1f/knA41OVxLper/b6uxzrAw0lc0yRXrlxJWVkZxcXFlJWVsWrVKgCWLFnC1q1bAfjnP//JOeecw4MPPsgjjzzCOeecw6uvvjp4NT+CFQlBNIwv7JApkkKIY9LU1ERu7ijcbje1tTVs3vzyUFep3+Iag588eTKPPfZYp8fXrl3bfv/MM8/klVdeGbia9YEVbLuKtcUgv0BOsAoh+u/yy7/Oz352K9/5zmLy8wuYOfOsQftaXm8VS5deQyjUSigU4qtfncfVV19LScnCAXl9Tan+jHYNjv6OwWeYtez/87/yp6bzmDr7fOZ98YRBqN3wMtLHZvtD2jzyHDiwh9Gj+/b/UZYq6FpX38vexuBtsVSB2RL7D9Cs3DJFUggh2tgi4K1gIwABK0kuchJCiDa2CHjpwQshRGc2CfhGFOBKTZMpkkII0cYWAW8Fm2glifzs/l8JK4QQdmOLgDdbGglYbtloWwghjmCLgI80H6TJdMkJViGEOIItAj7UdJCASpITrEKIbt188w08+eT6Do8ppbj88kt47713uj3ul79cyfr1jwKx9eAfffThLstt3PgUP/3pv3X53PXXX8vll5fy7W8v5tvfXszy5T8G4NNPt/P973+Xc8/9UrfHHothtZpkf1ktTTRbeUyQHrwQohvz51/Co48+zMKFl7U/9u67b2MYBqeffkZcr2G79eCHO6UUWjhAsxonPXghRLfOOec87rzz11RW7mLixEkAPP3035k3bwE7d+7gt7/9Na2tQcLhMJdc8lWuuGJxp9c4cj34SCTCXXet4d133yYvL58JEwr7XKeEWg++XyKt6MrEdKbKFEkhhrnIJ68R2d7zmlWaptGfFVScU8/BOWV29887nVx00cX87/8+xdKlP6ClpZlXX32Z733velJTU7n77gdwuVy0tLRw7bXf4vOfn0Vh4cRuX2/DhvV4vVX853/+D9FolOuuW9LjMup33/3vrF0bWw/+8su/zvz5l/S5jX014gNetQYAcKRkDHFNhBDD3fz5pdxyyw1ce+11vPDCc0yffhp5efn4/T5+97tfs2PHJ2iaTl1dLTt2fNJjwL/zztt85SslOBwOHA4HxcVf4YMP3uu2fFdDNIPNBgEfu4o1KT1ziGsihOiNc8rsHnvZMLiLjZ100hRyc0exZcvrbNz49/ZhmP/4j/vJycnlL395GIfDwU03Xdfr3tLDaJ3Gbo34WTStTQcBSM3KHuKaCCFGgvnzL+Evf/kj+/btZc6ccwEIBJrIzy/A4XCwa9cO3n+/+574IWeeeRabNm0kGo0SCrXy3HObBrvqfTbie/CNfj9pQMYw22FKCDE8XXTRV7j//nspLb0Up9MJwLe+dTW3376cZ5/9X8aOHcvpp8/o9XUuueRSduzYwTe/eQX5+QWcfvpMvN79faqLrAffi09feILROzfQMP83jB+bN0g1G35G+jrh/SFtHnlkPfj4DOl68JWVlSxatIji4mIWLVrE7t27O5UxTZNVq1Zx4YUXctFFF3W5A9RgaMg9lYdDF5CfLz14IYQ4UlxDNCtWrGDx4sWUlpayYcMGli9fzrp16zqUeeqpp9i7dy/PPvssDQ0NLFy4kFmzZjFu3LhBqfghZ542mfPPOZVAY3BQv44QQow0vfbgfT4fFRUVlJSUAFBSUkJFRQV+v79DuY0bN3L55Zej6zo5OTlceOGFbNo0+CcdNE0j2T3iTyUIIcSA6zUZvV4vBQUFGEbsIiLDMMjPz8fr9ZJzxIlNr9fLmDFj2j/3eDwcOHCgT5XpaSypN3l56f0+dqSSNieGkdzmmhodw9DQNK1PxzkcI36CX5/11GalFLqu9/l3YVh1ffu76fZIPxHVH9LmxDDS26zrDg4ebCA1NSPukJeTrB0ppWhubkTXHZ1+F3o7ydprwHs8HqqrqzFNE8MwME2TmpqaTpfkejweqqqqmD59OtC5Ry+ESDzZ2XnU19cSCDTEfYyu61hWYgV8b212OFxkZ/d9lmCvAZ+bm0tRURHl5eWUlpZSXl5OUVFRh+EZgIsvvpjHHnuML3/5yzQ0NPD888/z8MNdL6sphEgMhuFg1Kju12fpykj/q6U/BqvNcQ10rVy5krKyMoqLiykrK2PVqlUALFmyhK1btwJQWlrKuHHj+PKXv8wVV1zBddddx/jx4we8wkIIIeIz4i90AnnHTxTS5sQgbY7fMY/BH0+63rcz7QN17EglbU4M0ubE0J8293bMsOrBCyGEGDiJN9lUCCEShAS8EELYlAS8EELYlAS8EELYlAS8EELYlAS8EELYlAS8EELYlAS8EELYlAS8EELY1IgP+Hj2ix3JVq9ezdy5c5k6dSqffPJJ++N2bnd9fT1LliyhuLiYBQsWcP3117fvIGbndi9dupRLLrmEhQsXsnjxYj7++GPA3m0G+N3vftfh99vO7Z07dy4XX3wxpaWllJaW8uqrrwKD2GY1wn3zm99UTz75pFJKqSeffFJ985vfHOIaDay33npLVVVVqfPPP19t3769/XE7t7u+vl698cYb7Z//+te/Vj/+8Y+VUvZud2NjY/v95557Ti1cuFApZe82f/jhh+rqq69W5513Xvvvt53be/T/40MGq80jOuDr6urUzJkzVTQaVUopFY1G1cyZM5XP5xvimg28I38xEqndSim1adMm9a1vfSuh2v3EE0+or371q7ZucygUUldccYXau3dv+++3ndurVNcBP5htHlarSfZVvPvF2k0itduyLP77v/+buXPnJkS7f/KTn/Daa6+hlOJPf/qTrdt8zz33cMkll3TYN8LO7T3klltuQSnFzJkzufnmmwe1zSN+DF7Y2+23305KSgpXXnnlUFfluPjlL3/JSy+9xE033cSaNWuGujqD5t1332Xr1q0sXrx4qKtyXD388MP8/e9/Z/369Sil+PnPfz6oX29EB/yR+8UC3e4XazeJ0u7Vq1ezZ88e7r77bnRdT5h2AyxcuJAtW7YwevRoW7b5rbfeYteuXVxwwQXMnTuXAwcOcPXVV7N3715btveQQ+1wuVwsXryYd955Z1B/r0d0wB+5XyzQ7X6xdpMI7b7rrrv48MMPuf/++3G5XIC9293c3IzX623//MUXXyQzM9O2bb722mvZvHkzL774Ii+++CKjR4/mz3/+M/PmzbNlewFaWlpoaort2qSUYuPGjRQVFQ3qz3jEb/ixc+dOli1bRmNjIxkZGaxevZpJkyYNdbUGzC9+8QueffZZ6urqyM7OJisri6efftrW7f70008pKSmhsLCQpKQkAMaNG8f9999v23bX1dWxdOlSgsEguq6TmZnJrbfeyimnnGLbNh9p7ty5/OEPf2DKlCm2be++ffu44YYbME0Ty7KYPHkyP/3pT8nPzx+0No/4gBdCCNG1ET1EI4QQonsS8EIIYVMS8EIIYVMS8EIIYVMS8EIIYVMS8EIIYVMS8EIIYVMS8EIIYVP/H1SIB48YfXL6AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.linspace(0, N_EPOCHS,N_EPOCHS)\n",
    "\n",
    "plt.plot(x,t_f1)\n",
    "plt.plot(x,v_f1)\n",
    "plt.title(\"F1 score\")\n",
    "plt.legend([\"Train F1\", \"Valid F1\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jUsyVpt3aFqZ"
   },
   "source": [
    "Nous chargeons nos meilleurs paramètres et évaluons sur les données test. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Loss: 0.242 |  Test F1 score: 69.38%\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix, precision_recall_fscore_support\n",
    "model.load_state_dict(torch.load('tut1-model_conll.pt'))\n",
    "\n",
    "test_loss, test_f1, preds, labels = evaluate(model, test_iterator, criterion, TAG_PAD_IDX,full_report=False)\n",
    "print(f'Test Loss: {test_loss:.3f} |  Test F1 score: {test_f1*100:.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "predict =  [item for sublist in preds for item in sublist]\n",
    "true =  [item for sublist in labels for item in sublist]\n",
    "confusion = confusion_matrix(true, predict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Matrice de confusion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>O</th>\n",
       "      <th>LOC</th>\n",
       "      <th>PER</th>\n",
       "      <th>ORG</th>\n",
       "      <th>MISC</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>O</th>\n",
       "      <td>37834</td>\n",
       "      <td>57</td>\n",
       "      <td>39</td>\n",
       "      <td>480</td>\n",
       "      <td>110</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LOC</th>\n",
       "      <td>193</td>\n",
       "      <td>1517</td>\n",
       "      <td>6</td>\n",
       "      <td>179</td>\n",
       "      <td>29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PER</th>\n",
       "      <td>505</td>\n",
       "      <td>14</td>\n",
       "      <td>1758</td>\n",
       "      <td>494</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ORG</th>\n",
       "      <td>520</td>\n",
       "      <td>116</td>\n",
       "      <td>9</td>\n",
       "      <td>1811</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MISC</th>\n",
       "      <td>261</td>\n",
       "      <td>18</td>\n",
       "      <td>2</td>\n",
       "      <td>49</td>\n",
       "      <td>585</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          O   LOC   PER   ORG  MISC\n",
       "O     37834    57    39   480   110\n",
       "LOC     193  1517     6   179    29\n",
       "PER     505    14  1758   494     1\n",
       "ORG     520   116     9  1811    38\n",
       "MISC    261    18     2    49   585"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "confusion_df =pd.DataFrame(confusion)\n",
    "\n",
    "confusion_df.columns=[i for i in TAG.vocab.itos]\n",
    "s = pd.Series([i for i in TAG.vocab.itos])\n",
    "confusion_df = confusion_df.set_index([s])\n",
    "\n",
    "confusion_df['LOC'] = confusion_df['B-LOC'] + confusion_df['I-LOC']\n",
    "confusion_df['PER'] = confusion_df['B-PER'] + confusion_df['I-PER']\n",
    "confusion_df['ORG'] = confusion_df['B-ORG'] + confusion_df['I-ORG']\n",
    "confusion_df['MISC'] = confusion_df['B-MISC'] + confusion_df['I-MISC']\n",
    "\n",
    "\n",
    "confusion_df = confusion_df.drop(columns=[ i for i in TAG.vocab.itos if i != 'O'])\n",
    "\n",
    "confusion_df.loc['LOC'] = confusion_df.loc['B-LOC'] + confusion_df.loc['I-LOC']\n",
    "confusion_df.loc['PER'] = confusion_df.loc['B-PER'] + confusion_df.loc['I-PER']\n",
    "confusion_df.loc['ORG'] = confusion_df.loc['B-ORG'] + confusion_df.loc['I-ORG']\n",
    "confusion_df.loc['MISC'] = confusion_df.loc['B-MISC'] + confusion_df.loc['I-MISC']\n",
    "\n",
    "confusion_df = confusion_df.drop([i for i in TAG.vocab.itos if i != 'O' ])\n",
    "\n",
    "confusion_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Precision-Recall-F1Score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>F1-score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>O</th>\n",
       "      <td>0.962379</td>\n",
       "      <td>0.982191</td>\n",
       "      <td>0.972184</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LOC</th>\n",
       "      <td>0.880952</td>\n",
       "      <td>0.788462</td>\n",
       "      <td>0.832145</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PER</th>\n",
       "      <td>0.969129</td>\n",
       "      <td>0.634199</td>\n",
       "      <td>0.766681</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ORG</th>\n",
       "      <td>0.601062</td>\n",
       "      <td>0.726143</td>\n",
       "      <td>0.657708</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MISC</th>\n",
       "      <td>0.766710</td>\n",
       "      <td>0.639344</td>\n",
       "      <td>0.697259</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Precision    Recall  F1-score\n",
       "O      0.962379  0.982191  0.972184\n",
       "LOC    0.880952  0.788462  0.832145\n",
       "PER    0.969129  0.634199  0.766681\n",
       "ORG    0.601062  0.726143  0.657708\n",
       "MISC   0.766710  0.639344  0.697259"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm = confusion_df.to_numpy()\n",
    "\n",
    "TP = np.diag(cm)\n",
    "FP = np.sum(cm, axis=0) - TP\n",
    "FN = np.sum(cm, axis=1) - TP\n",
    "\n",
    "num_classes = 4\n",
    "TN = []\n",
    "for i in range(num_classes):\n",
    "    temp = np.delete(cm, i, 0)    \n",
    "    temp = np.delete(temp, i, 1) \n",
    "    TN.append(sum(sum(temp)))\n",
    "    \n",
    "precision = TP/(TP+FP)\n",
    "recall = TP/(TP+FN)\n",
    "f1 = (2*precision*recall)/(precision+recall)\n",
    "\n",
    "prf_df =pd.DataFrame()\n",
    "prf_df['Precision'] = precision\n",
    "prf_df['Recall'] = recall\n",
    "prf_df['F1-score'] = f1\n",
    "\n",
    "s = pd.Series([i for i in confusion_df.index])\n",
    "prf_df = prf_df.set_index([s])\n",
    "\n",
    "prf_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "05snyKgFaFqc"
   },
   "source": [
    "## Inférence\n",
    "\n",
    "Nous définissons une fonction `tag_sentence` qui va:\n",
    "- mettre le modèle en mode évaluation\n",
    "- tokenize la phrase avec spaCy si ce n'est pas une liste\n",
    "- mettre en minuscule des tokens si le `Field` l'a fait\n",
    "- numériser les tokens en utilisant le vocabulaire\n",
    "- découvrir quels tokens ne sont pas dans le vocabulaire, c'est-à-dire sont des tokens `<unk>`\n",
    "- convertir les tokens numérisés en un tenseur et ajouter une dimension de batch\n",
    "- introduire le tenseur dans le modèle\n",
    "- obtenir les prédictions sur la phrase\n",
    "- convertir les prédictions en tags lisibles\n",
    "\n",
    "En plus de renvoyer les tokens et les tags, il renvoie également les tokens qui étaient des jetons `<unk>`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "oYf_SRwFaFqd"
   },
   "outputs": [],
   "source": [
    "def tag_sentence(model, device, sentence, text_field, tag_field):\n",
    "    \n",
    "    model.eval()\n",
    "    \n",
    "    if isinstance(sentence, str):\n",
    "        nlp = spacy.load('en')\n",
    "        tokens = [token.text for token in nlp(sentence)]\n",
    "    else:\n",
    "        tokens = [token for token in sentence]\n",
    "\n",
    "    if text_field.lower:\n",
    "        tokens = [t.lower() for t in tokens]\n",
    "        \n",
    "    numericalized_tokens = [text_field.vocab.stoi[t] for t in tokens]\n",
    "\n",
    "    unk_idx = text_field.vocab.stoi[text_field.unk_token]\n",
    "    \n",
    "    unks = [t for t, n in zip(tokens, numericalized_tokens) if n == unk_idx]\n",
    "    \n",
    "    token_tensor = torch.LongTensor(numericalized_tokens)\n",
    "    \n",
    "    token_tensor = token_tensor.unsqueeze(-1).to(device)\n",
    "         \n",
    "    predictions = model(token_tensor)\n",
    "    \n",
    "    top_predictions = predictions.argmax(-1)\n",
    "    \n",
    "    predicted_tags = [tag_field.vocab.itos[t.item()] for t in top_predictions]\n",
    "    \n",
    "    return tokens, predicted_tags, unks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oReGRyxiaFqg"
   },
   "source": [
    "On utilise un exemple du dataset train."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 54
    },
    "colab_type": "code",
    "id": "PCwGFz4kaFqh",
    "outputId": "a5af1dc6-44b3-43a7-b1fd-d3d56dfea5fe"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['two', 'goals', 'in', 'the', 'last', 'six', 'minutes', 'gave', 'holders', 'japan', 'an', 'uninspiring', '2-1', 'asian', 'cup', 'victory', 'over', 'syria', 'on', 'friday', '.']\n",
      "['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-LOC', 'O', 'O', 'O', 'B-MISC', 'I-MISC', 'O', 'O', 'B-LOC', 'O', 'O', 'O']\n"
     ]
    }
   ],
   "source": [
    "example_index = 39\n",
    "\n",
    "sentence = vars(test_data.examples[example_index])['text']\n",
    "actual_tags = vars(test_data.examples[example_index])['tag']\n",
    "\n",
    "print(sentence)\n",
    "print(actual_tags)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JKDzhOh9aFqk"
   },
   "source": [
    "On utilise la fonction `tag_sentence` pour trouver les tags."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "FjPwbxnaaFqk",
    "outputId": "caff0d00-4ef9-42ed-9d1b-85e1f1f021f8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-LOC', 'O', 'O', 'O', 'B-MISC', 'I-MISC', 'O', 'O', 'B-LOC', 'O', 'O', 'O']\n",
      "['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-LOC', 'O', 'O', 'O', 'B-MISC', 'I-MISC', 'O', 'O', 'B-LOC', 'O', 'O', 'O']\n"
     ]
    }
   ],
   "source": [
    "tokens, pred_tags, unks = tag_sentence(model, \n",
    "                                       device, \n",
    "                                       sentence, \n",
    "                                       TEXT, \n",
    "                                       TAG)\n",
    "print(pred_tags)\n",
    "print(actual_tags)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On vérifie alors si le modèle a correctement prédit les tags."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 677
    },
    "colab_type": "code",
    "id": "ezJE_m3RaFqp",
    "outputId": "aa5b06f3-b1c6-43de-99cb-7cd55b8537c1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pred. Tag\t\t\t\tActual Tag\t\t\tCorrect?\t\t\tToken\n",
      "\n",
      "O\t\t\t\tO\t\t\t✔\t\t\ttwo\n",
      "O\t\t\t\tO\t\t\t✔\t\t\tgoals\n",
      "O\t\t\t\tO\t\t\t✔\t\t\tin\n",
      "O\t\t\t\tO\t\t\t✔\t\t\tthe\n",
      "O\t\t\t\tO\t\t\t✔\t\t\tlast\n",
      "O\t\t\t\tO\t\t\t✔\t\t\tsix\n",
      "O\t\t\t\tO\t\t\t✔\t\t\tminutes\n",
      "O\t\t\t\tO\t\t\t✔\t\t\tgave\n",
      "O\t\t\t\tO\t\t\t✔\t\t\tholders\n",
      "B-LOC\t\t\t\tB-LOC\t\t\t✔\t\t\tjapan\n",
      "O\t\t\t\tO\t\t\t✔\t\t\tan\n",
      "O\t\t\t\tO\t\t\t✔\t\t\tuninspiring\n",
      "O\t\t\t\tO\t\t\t✔\t\t\t2-1\n",
      "B-MISC\t\t\t\tB-MISC\t\t\t✔\t\t\tasian\n",
      "I-MISC\t\t\t\tI-MISC\t\t\t✔\t\t\tcup\n",
      "O\t\t\t\tO\t\t\t✔\t\t\tvictory\n",
      "O\t\t\t\tO\t\t\t✔\t\t\tover\n",
      "B-LOC\t\t\t\tB-LOC\t\t\t✔\t\t\tsyria\n",
      "O\t\t\t\tO\t\t\t✔\t\t\ton\n",
      "O\t\t\t\tO\t\t\t✔\t\t\tfriday\n",
      "O\t\t\t\tO\t\t\t✔\t\t\t.\n"
     ]
    }
   ],
   "source": [
    "print(\"Pred. Tag\\t\\t\\t\\tActual Tag\\t\\t\\tCorrect?\\t\\t\\tToken\\n\")\n",
    "\n",
    "for token, pred_tag, actual_tag in zip(tokens, pred_tags, actual_tags):\n",
    "    correct = '✔' if pred_tag == actual_tag else '✘'\n",
    "    print(f\"{pred_tag}\\t\\t\\t\\t{actual_tag}\\t\\t\\t{correct}\\t\\t\\t{token}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nMCLE1txaFqs"
   },
   "source": [
    "Essayons avec notre propre phrase. La liste unks est nulle cela signifie que tous les mots de cette phrase sont dans le vocabulaire."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "07FscjbZaFqt",
    "outputId": "e1f7a079-f519-4101-fd7a-3f7ff77c7bce"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[]\n"
     ]
    }
   ],
   "source": [
    "sentence = 'The will deliver a speech about the conflict in North Korea tomorrow in New York with my friend Mary Kate.'\n",
    "\n",
    "tokens, tags, unks = tag_sentence(model, \n",
    "                                  device, \n",
    "                                  sentence, \n",
    "                                  TEXT, \n",
    "                                  TAG)\n",
    "\n",
    "print(unks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 295
    },
    "colab_type": "code",
    "id": "Z_oRSprfaFqw",
    "outputId": "2335413b-4c1c-4703-f23f-b3a7d7d88216"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pred. Tag\tToken\n",
      "\n",
      "O\t\tthe\n",
      "O\t\twill\n",
      "O\t\tdeliver\n",
      "O\t\ta\n",
      "O\t\tspeech\n",
      "O\t\tabout\n",
      "O\t\tthe\n",
      "O\t\tconflict\n",
      "O\t\tin\n",
      "B-LOC\t\tnorth\n",
      "B-LOC\t\tkorea\n",
      "O\t\ttomorrow\n",
      "O\t\tin\n",
      "B-LOC\t\tnew\n",
      "I-LOC\t\tyork\n",
      "O\t\twith\n",
      "O\t\tmy\n",
      "O\t\tfriend\n",
      "B-PER\t\tmary\n",
      "I-PER\t\tkate\n",
      "O\t\t.\n"
     ]
    }
   ],
   "source": [
    "print(\"Pred. Tag\\tToken\\n\")\n",
    "\n",
    "for token, tag in zip(tokens, tags):\n",
    "    print(f\"{tag}\\t\\t{token}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Références :\n",
    "\n",
    " - [pytorch-pos-tagging](https://github.com/bentrevett/pytorch-pos-tagging/blob/master/1%20-%20BiLSTM%20for%20PoS%20Tagging.ipynb)\n",
    " - [Understanding LSTM Networks](https://colah.github.io/posts/2015-08-Understanding-LSTMs/)."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Copie de 1 - BiLSTM for PoS Tagging.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}