2-ChaEmbedding-NER-CONLL.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JClDF-ebiYLT"
   },
   "source": [
    "# Capture d'informations à partir des lettres majuscules avec l'embedding de caractères\n",
    "\n",
    "\n",
    "Une autre amélioration potentielle consiste à appliquer l'embedding de caractères. Une représentation d'embeddings de mots pré-entraînée, telle que word2vec, peut nécessiter de transformer les mots en minuscules. Pourtant, comme le font les humains, les lettres majuscules contiennent des informations utiles pour déterminer si un mot est une entité nommée ou non. Par conséquent, dans ce notebook, nous incluons l'embedding au niveau des caractères avec le Convolutional Neural Network (CNN). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 423
    },
    "colab_type": "code",
    "id": "nFc-Ky0Wp7L7",
    "outputId": "fd64c78a-d0b9-4274-ca81-7993db590e5a"
   },
   "outputs": [],
   "source": [
    "#!pip install torchtext==0.6.0\n",
    "\n",
    "import time\n",
    "import torch\n",
    "from torch import nn\n",
    "from torch.optim import Adam\n",
    "from torchtext.data import Field, NestedField, BucketIterator\n",
    "from torchtext.datasets import SequenceTaggingDataset\n",
    "from torchtext.vocab import Vocab\n",
    "from collections import Counter\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "\n",
    "from torchtext import data\n",
    "from torchtext import datasets\n",
    "\n",
    "import spacy\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import time\n",
    "import random\n",
    "import string\n",
    "from itertools import chain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "h4rz26nAjq1z"
   },
   "source": [
    "## Préparer les données\n",
    "\n",
    "Torchtext fournit une classe pratique pour gérer une entrée basée sur des caractères appelée `NestedField`. Pour inclure les caractères en tant que `NestedField`, nous devons créer une sorte de conteneur ` char_nesting_field`, puis spécifier cet objet comme paramètre pour le `NestedField`. \n",
    "Pour la tokenisation, on utilise la fonction intégrée `list` pour transformer le mot en caractères. La fonction de tokenisation peut être spécifiée dans le conteneur `Field`. \n",
    "\n",
    "On construit également le vocabulaire pour le `char_field`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# pour la reproductibilité\n",
    "SEED = 1234\n",
    "\n",
    "random.seed(SEED)\n",
    "np.random.seed(SEED)\n",
    "torch.manual_seed(SEED)\n",
    "torch.backends.cudnn.deterministic = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "TEXT = data.Field(lower = True) \n",
    "TAG = data.Field(unk_token = None) # les tags sont tous connus on a alors unk_token = None\n",
    "CHAR_NESTING= Field(tokenize=list)\n",
    "CHAR = NestedField(CHAR_NESTING) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_path = \"data_ner/\"\n",
    "\n",
    "train_data, valid_data, test_data = data.TabularDataset.splits(\n",
    "        path= my_path,\n",
    "        train=\"train.csv\",\n",
    "        validation=\"valid.csv\",\n",
    "        test=\"test.csv\", format='csv', skip_header=True,\n",
    "        fields=(\n",
    "            ((\"text\", \"char\"), (TEXT, CHAR)), \n",
    "            (\"tag\", TAG)\n",
    "        )\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of training examples: 14986\n",
      "Number of validation examples: 3465\n",
      "Number of testing examples: 3683\n"
     ]
    }
   ],
   "source": [
    "print(f\"Number of training examples: {len(train_data)}\")\n",
    "print(f\"Number of validation examples: {len(valid_data)}\")\n",
    "print(f\"Number of testing examples: {len(test_data)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "MIN_FREQ = 0\n",
    "\n",
    "TEXT.build_vocab(train_data, \n",
    "                 min_freq = MIN_FREQ, # les mots qui apparaissent moins que MIN_FREQ fois seront ignorés du vocabulaire\n",
    "                 vectors = \"glove.6B.100d\",\n",
    "                 unk_init = torch.Tensor.normal_)\n",
    "\n",
    "\n",
    "TAG.build_vocab(train_data)\n",
    "CHAR.build_vocab(train_data) \n",
    "\n",
    "# préparer les index de padding qui devront être ignorés pendant le training et l'évaluation\n",
    "TEXT_PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]\n",
    "CHAR_PAD_IDX = CHAR.vocab.stoi[CHAR.pad_token]  \n",
    "TAG_PAD_IDX = TAG.vocab.stoi[TAG.pad_token]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unique tokens dans le vocabulaire de TEXT vocabulary: 21012\n",
      "Unique tokens dans le vocabulaire de TAG : 10\n",
      "Unique tokens dans le vocabulaire de CHAR : 86\n"
     ]
    }
   ],
   "source": [
    "print(f\"Unique tokens dans le vocabulaire de TEXT vocabulary: {len(TEXT.vocab)}\")\n",
    "print(f\"Unique tokens dans le vocabulaire de TAG : {len(TAG.vocab)}\")\n",
    "print(f\"Unique tokens dans le vocabulaire de CHAR : {len(CHAR.vocab)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On affiche un exemple : "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'text': ['eu', 'rejects', 'german', 'call', 'to', 'boycott', 'british', 'lamb', '.'], 'char': [['E', 'U'], ['r', 'e', 'j', 'e', 'c', 't', 's'], ['G', 'e', 'r', 'm', 'a', 'n'], ['c', 'a', 'l', 'l'], ['t', 'o'], ['b', 'o', 'y', 'c', 'o', 't', 't'], ['B', 'r', 'i', 't', 'i', 's', 'h'], ['l', 'a', 'm', 'b'], ['.']], 'tag': ['B-ORG', 'O', 'B-MISC', 'O', 'O', 'O', 'B-MISC', 'O', 'O']}\n"
     ]
    }
   ],
   "source": [
    "print(vars(train_data.examples[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "BATCH_SIZE = 32\n",
    "\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "#device = 'cpu'\n",
    "train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits(\n",
    "    (train_data, valid_data, test_data), \n",
    "    batch_size = BATCH_SIZE,\n",
    "    device = device, sort=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oJtPUvimmxpv"
   },
   "source": [
    "# Construire le modèle\n",
    "\n",
    "Il y a deux nouvelles couches à ajouter basées sur le modèle de [Ma & Hovy (2016)](https://arxiv.org/abs/1603.01354):\n",
    "\n",
    "1. embedding de caractères\n",
    "2. CNN de l'embedding de caractères"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "W6hqBRQ3rZpY"
   },
   "outputs": [],
   "source": [
    "class BiLSTM(nn.Module):\n",
    "\n",
    "    def __init__(self,\n",
    "                 input_dim,\n",
    "                 embedding_dim,\n",
    "                 char_emb_dim,  \n",
    "                 char_input_dim,  \n",
    "                 char_cnn_filter_num,  \n",
    "                 char_cnn_kernel_size, \n",
    "                 hidden_dim,\n",
    "                 output_dim,\n",
    "                 lstm_layers,\n",
    "                 emb_dropout,\n",
    "                 cnn_dropout,  \n",
    "                 lstm_dropout,\n",
    "                 fc_dropout,\n",
    "                 word_pad_idx,\n",
    "                 char_pad_idx):  \n",
    "        super().__init__()\n",
    "        \n",
    "        # Couche 1A: Word Embedding\n",
    "        self.embedding_dim = embedding_dim\n",
    "        self.embedding = nn.Embedding(\n",
    "            num_embeddings=input_dim,\n",
    "            embedding_dim=embedding_dim,\n",
    "            padding_idx=word_pad_idx\n",
    "        )\n",
    "        self.emb_dropout = nn.Dropout(emb_dropout) \n",
    "        \n",
    "        # Couche 1B: Char Embedding-CNN\n",
    "        self.char_emb_dim = char_emb_dim\n",
    "        self.char_emb = nn.Embedding(\n",
    "            num_embeddings=char_input_dim,\n",
    "            embedding_dim=char_emb_dim,\n",
    "            padding_idx=char_pad_idx\n",
    "        )\n",
    "        self.char_cnn = nn.Conv1d(\n",
    "            in_channels=char_emb_dim,\n",
    "            out_channels=char_emb_dim * char_cnn_filter_num,\n",
    "            kernel_size=char_cnn_kernel_size,\n",
    "            groups=char_emb_dim  # different 1d conv for each embedding dim\n",
    "        )\n",
    "        self.cnn_dropout = nn.Dropout(cnn_dropout)\n",
    "        \n",
    "        # Couche 2: BiLSTM\n",
    "        self.lstm = nn.LSTM(\n",
    "            input_size= embedding_dim + (char_emb_dim * char_cnn_filter_num),\n",
    "            hidden_size=hidden_dim,\n",
    "            num_layers=lstm_layers,\n",
    "            bidirectional=True,\n",
    "            dropout=lstm_dropout if lstm_layers > 1 else 0\n",
    "        )\n",
    "        \n",
    "        # Couche 3: Fully-connected\n",
    "        self.fc_dropout = nn.Dropout(fc_dropout)\n",
    "        self.fc = nn.Linear(hidden_dim * 2, output_dim)  # fois 2 car bidirectionnel\n",
    "        \n",
    "        for name, param in self.named_parameters():\n",
    "            nn.init.normal_(param.data, mean=0, std=0.1)\n",
    "\n",
    "    def forward(self, words, chars):\n",
    "        \n",
    "        # words = [sentence length, batch size]\n",
    "        # chars = [batch size, sentence length, word length)\n",
    "        # embedding_out = [sentence length, batch size, embedding dim]\n",
    "        embedding_out = self.emb_dropout(self.embedding(words))\n",
    "\n",
    "        # char_emb_out = [batch size, sentence length, word length, char emb dim]\n",
    "        char_emb_out = self.emb_dropout(self.char_emb(chars))\n",
    "        batch_size, sent_len, word_len, char_emb_dim = char_emb_out.shape\n",
    "        char_cnn_max_out = torch.zeros(batch_size, sent_len, self.char_cnn.out_channels)\n",
    "        \n",
    "        # pour l'embedding des caractères, on doit itérer sur les phrases       \n",
    "        for sent_i in range(sent_len):\n",
    "            # sent_char_emb = [batch size, word length, char emb dim]\n",
    "            sent_char_emb = char_emb_out[:, sent_i, :, :]  # get the character field of sent i\n",
    "            # sent_char_emb_p = [batch size, char emb dim, word length]\n",
    "            sent_char_emb_p = sent_char_emb.permute(0, 2, 1)  # the channel (char emb dim) has to be the last dimension\n",
    "            # char_cnn_sent_out = [batch size, out channels * char emb dim, word length - kernel size + 1]\n",
    "            char_cnn_sent_out = self.char_cnn(sent_char_emb_p)\n",
    "            char_cnn_max_out[:, sent_i, :], _ = torch.max(char_cnn_sent_out, dim=2)  # max pooling over the word length dimension\n",
    "        char_cnn = self.cnn_dropout(char_cnn_max_out)\n",
    "\n",
    "        # char_cnn_p = [sentence length, batch size, char emb dim * num filter]\n",
    "        char_cnn_p = char_cnn.permute(1, 0, 2).to(device)\n",
    "        word_features = torch.cat((embedding_out, char_cnn_p), dim=2)\n",
    "\n",
    "        # lstm_out = [sentence length, batch size, hidden dim * 2]\n",
    "        lstm_out, _ = self.lstm(word_features)\n",
    "        # ner_out = [sentence length, batch size, output dim]\n",
    "        ner_out = self.fc(self.fc_dropout(lstm_out))\n",
    "        return ner_out\n",
    "\n",
    "    def init_embeddings(self, char_pad_idx, word_pad_idx, pretrained=None, freeze=True):\n",
    "        # initialise l'embedding pour les tokens pad à zéro\n",
    "        self.embedding.weight.data[word_pad_idx] = torch.zeros(self.embedding_dim)\n",
    "        self.char_emb.weight.data[char_pad_idx] = torch.zeros(self.char_emb_dim)\n",
    "        if pretrained is not None:\n",
    "            self.embedding = nn.Embedding.from_pretrained(\n",
    "                embeddings=torch.as_tensor(pretrained),\n",
    "                padding_idx=word_pad_idx,\n",
    "                freeze=freeze\n",
    "            )\n",
    "\n",
    "    def count_parameters(self):\n",
    "        return sum(p.numel() for p in self.parameters() if p.requires_grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 219
    },
    "colab_type": "code",
    "id": "F50qLmWTrwG7",
    "outputId": "095b951f-bc17-4faa-bef9-ec3e6dc875f7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Le modèle a 2,353,460 paramètres à entraîner.\n"
     ]
    }
   ],
   "source": [
    "model = BiLSTM(\n",
    "    input_dim=len(TEXT.vocab),\n",
    "    embedding_dim=100,\n",
    "    char_emb_dim=25,\n",
    "    char_input_dim=len(CHAR.vocab),\n",
    "    char_cnn_filter_num=5,\n",
    "    char_cnn_kernel_size=3,\n",
    "    hidden_dim=64,\n",
    "    output_dim=len(TAG.vocab),\n",
    "    lstm_layers=2,\n",
    "    emb_dropout=0.6,\n",
    "    cnn_dropout=0.6,\n",
    "    lstm_dropout=0.6,\n",
    "    fc_dropout=0.7,\n",
    "    word_pad_idx=TEXT_PAD_IDX,\n",
    "    char_pad_idx=CHAR_PAD_IDX\n",
    ")\n",
    "model.init_embeddings(\n",
    "    char_pad_idx=CHAR_PAD_IDX,\n",
    "    word_pad_idx=TEXT_PAD_IDX ,\n",
    "    freeze=True\n",
    ")\n",
    "print(f\"Le modèle a {model.count_parameters():,} paramètres à entraîner.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "NIb_h9XXreOv"
   },
   "source": [
    "# Entraînement\n",
    "\n",
    " - Optimiseur et fonction loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def optimiseur(model, lr=1e-4, eps=1e-6, weight_decay_rate=0.001, second_weight_decay_rate=0.0):\n",
    "    param_optimizer = list(model.named_parameters())\n",
    "    no_decay = ['bias', 'gamma', 'beta']\n",
    "    optimizer_grouped_parameters = [\n",
    "        {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],\n",
    "            'weight_decay_rate': weight_decay_rate},\n",
    "        {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],\n",
    "            'weight_decay_rate': second_weight_decay_rate}]\n",
    "    return optim.Adam(\n",
    "        optimizer_grouped_parameters,\n",
    "        lr=lr,\n",
    "        eps=eps\n",
    "    )\n",
    "\n",
    "optimizer = optimiseur(model, lr=1e-4, eps=1e-6, weight_decay_rate=0.001, second_weight_decay_rate=0.0)\n",
    "scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=10, gamma=0.5)\n",
    "\n",
    "criterion = nn.CrossEntropyLoss(ignore_index = TAG_PAD_IDX)\n",
    "\n",
    "model = model.to(device)\n",
    "criterion = criterion.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " - Métriques"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import f1_score, classification_report\n",
    "\n",
    "def f1_loss( preds, y, tag_pad_idx, full_report=False):\n",
    "    index_o = TAG.vocab.stoi[\"O\"]\n",
    "\n",
    "    positive_labels = [i for i in range(len(TAG.vocab.itos))\n",
    "                       if i not in (tag_pad_idx, index_o)]\n",
    "    _, pred = torch.max(preds, 1)\n",
    "    flatten_preds = pred.data.cpu().numpy() \n",
    "    flatten_y = y.data.cpu().numpy()\n",
    "    if full_report:\n",
    "  \n",
    "        positive_names = [TAG.vocab.itos[i]\n",
    "                              for i in range(len(TAG.vocab.itos))\n",
    "                              if i not in (tag_pad_idx, index_o)]\n",
    "        print(classification_report(\n",
    "                y_true=flatten_y,\n",
    "                y_pred=flatten_preds,\n",
    "                labels=positive_labels,\n",
    "                target_names=positive_names\n",
    "            ))\n",
    "    return f1_score(\n",
    "            y_true=flatten_y,\n",
    "            y_pred=flatten_preds,\n",
    "            labels=positive_labels,\n",
    "            average=\"micro\"\n",
    "        ), flatten_preds, flatten_y\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 01 | Epoch Time: 1m 38s\n",
      "\tTrain Loss: 0.998 | Train F1 score: 0.13%\n",
      "\t Val. Loss: 0.865 |  Val. F1 score: 0.00%\n",
      "Epoch: 11 | Epoch Time: 0m 24s\n",
      "\tTrain Loss: 0.340 | Train F1 score: 44.01%\n",
      "\t Val. Loss: 0.313 |  Val. F1 score: 51.56%\n",
      "Epoch: 21 | Epoch Time: 0m 24s\n",
      "\tTrain Loss: 0.258 | Train F1 score: 55.21%\n",
      "\t Val. Loss: 0.241 |  Val. F1 score: 63.45%\n",
      "Epoch: 31 | Epoch Time: 0m 38s\n",
      "\tTrain Loss: 0.230 | Train F1 score: 60.22%\n",
      "\t Val. Loss: 0.216 |  Val. F1 score: 67.70%\n",
      "Epoch: 41 | Epoch Time: 0m 48s\n",
      "\tTrain Loss: 0.218 | Train F1 score: 62.81%\n",
      "\t Val. Loss: 0.204 |  Val. F1 score: 69.76%\n"
     ]
    }
   ],
   "source": [
    "def train(model, iterator, optimizer, criterion, tag_pad_idx):\n",
    "    \n",
    "    epoch_loss = 0\n",
    "    epoch_f1 = 0\n",
    "    model.train()\n",
    "    \n",
    "    for batch in iterator:\n",
    "        \n",
    "        text = batch.text\n",
    "        tags = batch.tag\n",
    "        chars = batch.char \n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        #text = [sent len, batch size]\n",
    "        \n",
    "        predictions = model(text,chars)\n",
    "        \n",
    "        #predictions = [sent len, batch size, output dim]\n",
    "        #tags = [sent len, batch size]\n",
    "        \n",
    "        predictions = predictions.view(-1, predictions.shape[-1])\n",
    "        tags = tags.view(-1)\n",
    "        \n",
    "        #predictions = [sent len * batch size, output dim]\n",
    "        #tags = [sent len * batch size]\n",
    "        \n",
    "        loss = criterion(predictions, tags)\n",
    "                \n",
    "        f1,_,_ = f1_loss(predictions, tags, tag_pad_idx)\n",
    "        \n",
    "        loss.backward()\n",
    "        \n",
    "        optimizer.step()\n",
    "        \n",
    "        epoch_loss += loss.item()\n",
    "        epoch_f1 += f1.item()\n",
    "        \n",
    "    return epoch_loss / len(iterator), epoch_f1 / len(iterator)\n",
    "\n",
    "def evaluate(model, iterator, criterion, tag_pad_idx, full_report):\n",
    "    \n",
    "    epoch_loss = 0\n",
    "    epoch_f1 = 0\n",
    "    \n",
    "    model.eval()\n",
    "    preds = []\n",
    "    labels = []\n",
    "    with torch.no_grad():\n",
    "    \n",
    "        for batch in iterator:\n",
    "\n",
    "            text = batch.text\n",
    "            tags = batch.tag\n",
    "            chars = batch.char \n",
    "            \n",
    "            predictions = model(text,chars)\n",
    "            \n",
    "            predictions = predictions.view(-1, predictions.shape[-1])\n",
    "            tags = tags.view(-1)\n",
    "            \n",
    "            loss = criterion(predictions, tags)\n",
    "            \n",
    "            \n",
    "            f1, _, _ = f1_loss(predictions, tags, tag_pad_idx, full_report)\n",
    "            _, pred = torch.max(predictions, 1)\n",
    "            flatten_preds = pred.data.cpu().numpy() \n",
    "            flatten_y = tags.data.cpu().numpy()\n",
    "            preds.append(flatten_preds)\n",
    "            labels.append(flatten_y)\n",
    "            epoch_loss += loss.item()\n",
    "            epoch_f1 += f1.item()\n",
    "        \n",
    "    return epoch_loss / len(iterator), epoch_f1 / len(iterator), preds, labels\n",
    "\n",
    "def epoch_time(start_time, end_time):\n",
    "    elapsed_time = end_time - start_time\n",
    "    elapsed_mins = int(elapsed_time / 60)\n",
    "    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
    "    return elapsed_mins, elapsed_secs\n",
    "\n",
    "N_EPOCHS = 50\n",
    "\n",
    "t_loss = []\n",
    "t_f1 = []\n",
    "v_loss = []\n",
    "v_f1 = []\n",
    "\n",
    "best_valid_loss = float('inf')\n",
    "\n",
    "for epoch in range(N_EPOCHS):\n",
    "\n",
    "    start_time = time.time()\n",
    "   \n",
    "    \n",
    "    train_loss, train_f1 = train(model, train_iterator, optimizer, criterion, TAG_PAD_IDX)\n",
    "    t_loss.append(train_loss)\n",
    "    t_f1.append(train_f1) \n",
    "    \n",
    "    valid_loss, valid_f1,_,_ = evaluate(model, valid_iterator, criterion, TAG_PAD_IDX, full_report= False)\n",
    "    v_loss.append(valid_loss)\n",
    "    v_f1.append(valid_f1)\n",
    "    \n",
    "    scheduler.step()\n",
    "    end_time = time.time()\n",
    "\n",
    "    epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
    "    \n",
    "    if valid_loss < best_valid_loss:\n",
    "        best_valid_loss = valid_loss\n",
    "        torch.save(model.state_dict(), 'tut2-model_conll.pt')\n",
    "    \n",
    "\n",
    "    if epoch%10 == 0: \n",
    "        print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')\n",
    "        print(f'\\tTrain Loss: {train_loss:.3f} | Train F1 score: {train_f1*100:.2f}%')\n",
    "        print(f'\\t Val. Loss: {valid_loss:.3f} |  Val. F1 score: {valid_f1*100:.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f3e5c0fa110>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAELCAYAAADTK53JAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxU5b348c85Z7bsG0mYBDAQthB2cEdFXIIaCG1VenPdqoW22Gtr7yK318tS1Cu21Wst6tXbot5w+/NWW5GAiFVbFQFRkC2sIYQlQ5bJvs12zu+PCYHIkoUJk8x8369XXpNMzjnz/TLhO895znOeRzEMw0AIIUTIUYMdgBBCiN4hBV4IIUKUFHghhAhRUuCFECJESYEXQogQJQVeCCFClBR4IYQIUVLgRdiZMWMGn3/+ebDDEKLXSYEXQogQJQVeCMDtdvPkk08ybdo0pk2bxpNPPonb7QagurqaH/zgB0ydOpUrrriC/Px8dF0H4JVXXuG6665j0qRJ5OTksGnTpmCmIUQHpmAHIERf8NJLL7Fjxw5Wr16NoigsWLCAF198kZ/+9KesXLmS1NTU9uK9Y8cOFEXh8OHDrFq1irfeeovU1FSOHz/eXviF6AukBS8EsGbNGh5++GGSkpJITEzk4Ycf5t133wXAZDJRWVlJWVkZZrOZqVOnoigKmqbhdrspLi7G4/EwaNAghgwZEuRMhDhNCrwQQEVFBWlpae0/p6WlUVFRAcBDDz3EZZddxoMPPshNN93EK6+8AsBll13Gz3/+c1544QWuueYaHn30UcrLy4MSvxDnIgVeCCAlJYWysrL2nx0OBykpKQBER0ezcOFCPvzwQ15++WVWrlzZ3l0za9Ys/vCHP/Dxxx+jKAq/+tWvghK/EOciBV6EJY/Hg8vlav+64447eOmll6iurqa6upoVK1Ywa9YsAD7++GNKS0sxDIPo6Gg0TUNVVQ4fPsymTZtwu91YLBasViuapgU5MyFOk4usIizNnz+/w88PPvggY8eOZfbs2QDMnDmTBQsWAFBaWsqyZcuorq4mNjaWv/u7v+PKK69k3759/PrXv6a4uBiz2cykSZP4xS9+cclzEeJ8FFnwQwghQpN00QghRIiSAi+EECFKCrwQQoQoKfBCCBGipMALIUSIkgIvhBAhqk+Ng6+paULXuz9qMykpGqezsRci6rsk5/AgOYeHnuasqgoJCVHn/X2fKvC6bvSowJ/aN9xIzuFBcg4PvZGzdNEIIUSIkgIvhBAhqk910Qgh+hfDMKipqcTtbgUC08VQUaGG3cIpF85ZwWKxkZCQjKIo3TpupwV++fLlvP/++5w4cYI1a9YwcuTIs7bx+Xw88cQTfPrppyiKwvz587nrrru6FYgQov9pbKxDURRSUwehKIHpEDCZVLze8CrwF8rZMHRqa6tobKwjJia+W8ft9B256aabWLVqFenp6efdZs2aNRw9epQNGzbw5ptv8sILL3D8+PFuBSKE6H9aWhqJiYkPWHEXZ1MUlZiYBFpaejDKprMNpk6dit1uv+A269at46677kJVVRITE7n55ptZv359t4MRQvQvuu5D06Snt7dpmgld93V7v4C8Mw6Ho8NyZ3a7nZMnT3b7OElJ0d3e58u95fzi9S959qc3YDaFVysiOTkm2CFccpJz31JRoWI2B36RE1OY/V+GznNWVbXbfwt96qPX6Wzs9ljQY2V1HHHUc+BwJSkJkb0UWd+TnBxDZWVDsMO4pCTnvkfX9YD3l/e0D37evPvxeDx4vR6OHTvK0KGZAIwcOYqf/3xxl47xzjtv4XK5mDv377v12nfeOYtnnnmOYcOGdztu6FrOuq6f9begqsoFG8YBKfB2u52ysjLGjx8PnN2i701JsVYAnPWusCrwQoiOXn31dQAcjjK+//17ee21/z1rG6/Xi8l0/rI3Z86dvRZfMASkwM+cOZM//vGP3HrrrdTW1vKXv/yFVatWBeLQnUqMswFQXd96SV5PCNG/3HnnLHJz8/jqq62kpaUzf/4Cliz5N5qamnC73VxzzbUsWPATAH73u/+ipaWFH//4p6xbt4YPPlhPTEwshw8XExMTzRNPPENS0oALvt7x48f45S+fora2Bk3TmD//Ya666hpaW1t54onFHDlyGE0zMWTIZSxb9jRHjx7hqaeW0tLSiq77uO22WeTn3xuQ3Dst8E888QQbNmygqqqK733ve8THx7N27VrmzZvHI488wrhx48jLy2PHjh3ceuutADz88MMMHjw4IAF2JjGmrQVfJwVeiGDbuMvBZzsdF3UMRYFzLSQ6bbyda8ddeMDH+VRVVfHCC/8FgMvlYvny54iMjMTr9fKzn/2YzZs/56qrrjlrv717i3j99T+QmjqQ5cuf4K233uQHP3j4gq+1dOnj5OV9i9zcOZSUHObHP55HQcFb7Nz5NQ0NDRQU/BGA+vp6AP70p7e45ppp3HffQx2eD4ROC/zjjz/O448/ftbzr776avv3mqaxdOnSgAXVHWaTRnyMFae04IUQ5zFz5h3t3+u6zosvPs+uXTsBA6fTycGDB85Z4MePn0Bq6kAAsrPHsnXrlgu+TnNzE4cOHeD22/2Ltw8dOozhw0exZ88uhg8fwdGjR/j1r5czadIUrrlmGgATJ05ixYrncbncTJ48lcmTpwYo6z52kbWnUhIipItGiD7g2nE9b2Wf0hs3OkVGRrR//+abq2hoqOeVV17DarWyfPmTuN2uc+5nsVjav1dVDZ/vwkMVjXOdegCKopCePohVq/7Il19uZfPmjbzyygpef/3/MX36TUyYMJFNmz6noOA11q59l0WLlvUgy7OFxFik5PhInPXnfoOEEOJMDQ0NJCUNwGq1UllZwWef/S1gx46Kimb48JG8914hAKWlRyguPsCYMWOpqChHVTWuv346jzzyj9TW1tDQUM/x48dISkri9ttn8b3vzaOoaE/A4gmJFnxyQgRbi05iGEa352oQQoSXu+76Lv/+74/xve/lk5KSypQplwf0+IsXP8Evf/kU//d//4umaTz++C9ISEhg06aNvPzybwH/DWL33PMAAwYk88Ybv+eDD9ZjMplQFIWf/OQfAxaLYpzvnCIIejIOHmDT3gpeXb2b5x+ZRkykpfMdQkBfHx/dGyTnvufkyVIGDrwsoMeUuWjO7Vz/1p2Ngw+NLpoEf/+aXGgVQojTQqTA+29wctZJP7wQQpwSGgU+3t+Cl5E0QghxWkgU+NgoCxaTKl00QghxhpAo8IqikBhrkxa8EEKcISQKPPgnHZMWvBBCnBY6BT7OJjc7CSHEGUKmwCfG2qhvcuPxdn/VEyFE//ezn/0D77zzdofnDMPgrrtm8/XX286735NPLuHtt98E/PPBv/nmuWfCXbduDY8//i/n/N2dd87i8OFDPYy894RMgU+KbZs2uEFa8UKEozvumM26dWs6PLd9+1domsbEiZO7dIw5c+7s9mIffVlITFUA/hY8QHVdK6my8IcQQeE5sBHP/k8u6hiKopxz0i7zqOsxj7z2vPtdf/10nn32aUpKDjN06DAA1q59l9tvn0Vx8SF+/eunaW1twe12M3v2t7j77vyzjnHmfPAej4fnnnuG7du/Ijk5hSFDMroUf7+aD76/OHNlJyFE+DGbzdxyy0zee28NCxb8hObmJj799G/88Ic/Jioqiv/8zxexWCw0Nzczf/79XHHF1WRkDD3v8VavfhuHo4z/+Z//w+v18vDD87DbO58ps1/NBw9QUlLCwoULqa2tJT4+nuXLl5ORkdFhm8rKShYtWsTx48fxer388Ic/JC8vL2CBdiYhxoaCTFcgRDCZR157wVZ2V1zMXDR33JHHP/3TPzB//sN8+OEHjB8/geTkFKqrnfz2t09z6NABFEWlqqqSQ4cOXLDAb9v2FbfdlovJZMJkMpGTcxs7d359wdfva/PBd6kPfvHixeTn5/P++++Tn5/PokWLztrm6aefZuzYsaxZs4ZVq1bx3HPP4XBc3Mou3WE2qcRGW6TACxHGRowYSVLSALZs2cS6de9yxx3+Qvtf/7WCxMQkfv/7Vbz++h/IysrG7XZf8Fg9mYexK/PBX375lXz55RYeeODvcLlcTJ9+E6+8spL09EEUFLzGsmVn19ee6rTAO51OioqKyM3NBSA3N5eioiKqq6s7bLdv3z6uu+46ABITExk9ejTvvfdewALtiiS52UmIsHfHHbP5/e9f4dixo0ybdgMAjY0NpKSkYjKZOHz4EDt2XLglDjB16uWsX78Or9eLy9XKBx+s73SffjcfvMPhIDU1FU3TAP/yfCkpKTgcDhITE9u3y87OZt26dYwbN47jx4+zfft2Bg0aFLBAuyIx1saxisZL+ppCiL7llltuY8WK35CX923MZjMA99//EMuWLWLDhvdIT09n4sRJnR5n9uxvc+jQIe69925SUlKZOHEKDseJTvfrV/PB7969m8cee4y1a9e2P3f77bfzy1/+kuzs7Pbnqqureeqppzhw4ABpaWlYrVbsdjsLFy4MWLCd+d27u1m3sYS3ns6VhT+EuAT27CkiLS2w88GLcysrKyU7e0y39um0BW+32ykvL8fn86Fp/jUJKyoqzrqanJiYyK9+9av2n+fNm0dmZma3gunpgh+nFkWIMKu4vTqHS6uJjQrthT/6+kIQvUFy7nt0XQ/44hyy4Me56bp+1t/CRS/4kZSURFZWFoWF/j6lwsJCsrKyOnTPANTU1OD1egHYtGkTBw4caO+3v1QGtI2FlwutQgjRxWGSS5YsYeHChbz44ovExsayfPlywN9Kf+SRRxg3bhw7d+7kySefRFVVEhISePnll4mIiOjkyIHVfrNTfStD7bGX9LWFCFeyFnLv6+nKql0q8JmZmfzxj3886/lXX321/fsbbriBG264oUdBBEpS3KkWvNzsJMSloKoaPp8Xk8kc7FBCms/nRVW1bu8XMnPRAETZTFjMqgyVFOISiYiIpqGhFsMIrz7zS8kwdBoaaoiIOH9f+/mEzFQF4L+ZICnWJn3wQlwi0dFx1NRUUl5+HOhZN8I3qaqKrofXB8aFc1awWGxER8d1+7ghUeAN/fQUwUmxNpx1UuCFuBQURSExMSWgx+zrI4d6Q2/l3O+7aLylX3P0hR+gN9cByNJ9QgjRpt8XeDVuIL6mOtw7/dMiJMVaqW/24PbIwh9CiPDW/wt8/ECis6fh2fMRekt9+1DJGln4QwgR5vp9gQeIn3Yn6B7cO9a1r+wkF1qFEOEuJAq8JSkdU+ZVePZ8RJLVPwWoXGgVQoS7kCjwANbJs0H3EF3ysSz8IYQQhFCBV+PtmDKvwrv3I9KifVTL3axCiDAXMgUewDJ5Fvg8zIjYKy14IUTYC6kCr8WnYcq8kom+nbTW1wQ7HCGECKqQKvAAlsmz0fAy3r2txzOwCSFEKAi5Aq/Fp+FMHM+1ln3Uf2PdWCGECCchV+ABmobfihkvrTsu7aLfQgjRl4RkgY+1X8Z2dwa2w5+gt4bXpEVCCHFKlwp8SUkJc+fOJScnh7lz53LkyJGztnE6ncyfP59Zs2Yxc+ZMlixZ0r6E36WWGGvjL61jUXU33pKvghKDEEIEW5cK/OLFi8nPz+f9998nPz+fRYsWnbXNyy+/TGZmJmvWrGHNmjXs2bOHDRs2BDzgroiymXBqA2gyxeMt3R6UGIQQItg6LfBOp5OioqL2BbRzc3MpKiqi+hsXMBVFoampCV3XcbvdeDweUlNTeyfqTvgX/ojgqGkovhNFGF656UkIEX46LfAOh4PU1FQ0zb8eoKZppKSk4HA4Omy3YMECSkpKmDZtWvvXlClTeifqLkiMtbLbOxh8HnzHi4IWhxBCBEvAVnRav349o0aN4vXXX6epqYl58+axfv16Zs6c2eVjJCV1f83BU5KTYzr8nJ4Sw9aKJJSYSEzlu0m+/PoeH7uv+mbO4UByDg+Sc2B0WuDtdjvl5eX4fD40TcPn81FRUYHdbu+wXUFBAU899RSqqhITE8OMGTPYsmVLtwq809mIrnf/5qRzLXcVadGobvShjMym8cBWjIo6FCV0Bg3JsmbhQXIODz3NWVWVCzaMO614SUlJZGVlUVhYCEBhYSFZWVkkJiZ22G7QoEF88sknALjdbjZt2sSIESO6HXCgDGibF745JRujpR698kjQYhFCiGDoUpN2yZIlFBQUkJOTQ0FBAUuXLgVg3rx57Nq1C4Cf//znfPXVV8yaNYs5c+aQkZHB3Xff3XuRdyI5IQKAIwwBRZXRNEKIsKMYfWjClkB20eiGwb/+1yYSYmw8Evs+hquZqDuXBSrUoJPT2PAgOYeHoHXR9FeqojB9YjoHjtXSlDQGvfoYekNVsMMSQohLJmQLPMC14+2YNIWNNSkAeEu/DnJEQghx6YR0gY+NtDB1dAofHHBDbCreo1LghRDhI6QLPMCNk9JpcfkojxyBr2wfhrsl2CEJIcQlEfIFfnh6HIOSo/i4cgDoXrzHdwc7JCGEuCRCvsArisKNk9LZUhmNbo6UfnghRNgI+QIPcFX2QMwWM0dNGfiO7cTQ9WCHJIQQvS4sCnyE1cTV2QP51JmM0dqAr6I42CEJIUSvC4sCD/6LrbtddnRUfHJXqxAiDIRNgR+cEs2gtGRKDTse6YcXQoSBsCnw4G/Fb2tOw6gtQ68rD3Y4QgjRq8KqwE8dnUyJNhQAb+m2IEcjhBC9K6wKvNmkkT1uJMe8ibQe3BLscIQQoleFVYEHmD4xjW3uDBTnEfT6imCHI4QQvSbsCnxKQiS1ieMB8BySVrwQInSFXYEHyB47nMOeZFoObAp2KEII0Wu6tOh2SUkJCxcupLa2lvj4eJYvX05GRkaHbf7lX/6F/fv3t/+8f/9+VqxYwU033RTQgANhyugU/vTJUIbVf4Gv+gRaYnqwQxJCiIDrUoFfvHgx+fn55OXlsXr1ahYtWsQbb7zRYZtnnnmm/ft9+/Zx//33c9111wU22gCJjbTQMnACet1WPMWb0RK/E+yQhBAi4DrtonE6nRQVFZGbmwtAbm4uRUVFVFdXn3eft956i1mzZmGxWAIXaYCNzx7GQe9AWvZvog+tWiiEEAHTaYF3OBykpqaiaRoAmqaRkpKCw+E45/Zut5s1a9bwne/07Vbx5JHJfO0Ziqm5Cr3qSLDDEUKIgOtSF013/OUvfyEtLY2srKxu73uhxWM7k5wc0+19TJmX4y3bjOnENgaMGd/j1w6WnuTc30nO4UFyDoxOC7zdbqe8vByfz4emafh8PioqKrDb7efc/u233+5x693pbETXu99d0tMVyceOSGdvaRpZOz5Fn/AtFKX/DCqSlefDg+QcHnqas6oqF2wYd1rRkpKSyMrKorCwEIDCwkKysrJITEw8a9uTJ0/y1VdftffX93XjM5PYrQ/D5KrFd/JgsMMRQoiA6lKTdcmSJRQUFJCTk0NBQQFLly4FYN68eezatat9uz//+c/ceOONxMfH9060AWYxa5gvm4zb0HAf3BzscIQQIqAUow8NIbnUXTQAO4ud1Kz7DeOjq4i//zcoqtaj41xqchobHiTn8BC0LppQNyYjgb0Mx+Rpwle2N9jhCCFEwIR9gTdpKtEjJtNimGmVqQuEECEk7As8wOVj0tjpHoK35CsMnyfY4QghREBIgQdGDI7noDYCzdeK99iuzncQQoh+QAo8oCoKSaMm0qhbaT0go2mEEKFBCnybK7LT2O+x4zlRJHPTCCFCghT4NpelxlBuGYzZ04ghC3ILIUKAFPg2iqIQM3QsAE2le4IcjRBCXDwp8GcYlT2KOj2C2uKdwQ5FCCEumhT4MwxNi+OoYcdSXSz98EKIfk8K/BlURcGXPIJIvRGX89zz3QshRH8hBf4bUrImAnB897YgRyKEEBdHCvw3DBs1kgYjgpZjRcEORQghLooU+G8wmzRqIoaQ0HQEj9cX7HCEEKLHpMCfg3XwGOLUZkoOHAp2KEII0WNS4M9hUPZkAMr37ghyJEII0XNdKvAlJSXMnTuXnJwc5s6dy5EjR8653bp165g1axa5ubnMmjWLqqqqQMZ6yViTB9GiRKJUHpDhkkKIfqvTRbcBFi9eTH5+Pnl5eaxevZpFixbxxhtvdNhm165d/Pa3v+X1118nOTmZhoYGLBZLrwTd2xRFwZWYyZCKYo446hmaFhfskIQQots6bcE7nU6KioraF9LOzc2lqKiI6urqDtu99tprPPjggyQnJwMQExOD1WrthZAvjfjMcSRozezdI4txCyH6p05b8A6Hg9TUVDTNv1appmmkpKTgcDhITExs3664uJhBgwbx93//9zQ3N3PLLbfwox/9CEVRuhzMhdYW7ExyckyP9z2XuIlXcPyL/6WxdA/JyTcG9NiBEuic+wPJOTxIzoHRpS6arvD5fOzfv5+VK1fidrv5/ve/T1paGnPmzOnyMYKx6Pb5GEYcHlMUSU2l7Nx3EntSVECPf7FkYeLwIDmHh6Atum232ykvL8fn848J9/l8VFRUYLfbO2yXlpbGzJkzsVgsREdHc9NNN7FzZ/+dtEtRFDT7KIabytm2vyLY4QghRLd1WuCTkpLIysqisLAQgMLCQrKysjp0z4C/b/6zzz7DMAw8Hg+bN29m9OjRvRP1JRIxZAyJWhOHDhwOdihCCNFtXRomuWTJEgoKCsjJyaGgoIClS5cCMG/ePHbt8q9hescdd5CUlMTtt9/OnDlzGD58OHfeeWfvRX4JaHb/B5S1upjq+tYgRyOEEN2jGH1ooHdf6oMHMAyd+tcfYWtDKso13+OmKYMC/ho9Jf2U4UFyDg9B64MPZ4qiYkkfzShrOdsOVAY7HCGE6BYp8J3Q7KOIp5GK48ekm0YI0a9Ige+Elubvhx+mlbNx98kgRyOEEF0nBb4TakI6ijWaqfHVfLazDL3vXLIQQogLkgLfCUVR0eyjyFRPUFXbzP6jtcEOSQghukQKfBeYhl+F2V3P5EgHn+0sC3Y4QgjRJVLgu8CUMRklKpGZ8cV8ub+S5lZPsEMSQohOSYHvAkXVMI+5kZTWIyTq1WwpKg92SEII0Skp8F1kHn0DaCZuSzjMpzsdwQ5HCCE6JQW+i9SIWEyZVzJeOcDJcifHKhqDHZIQQlyQFPhusGTfgqa7udpWzKc75GKrEKJvkwLfDVpyBmrqcGZEH2TzHgcerx7skIQQ4rykwHeTJftmYn21DPYdZftBmZ9GCNF3SYHvJtPQqSgRcdwUdUAutgoh+jQp8N2kaCbMWdMZoR6jovQIzjqZgEwI0Td1qcCXlJQwd+5ccnJymDt3LkeOHDlrmxdeeIGrr76avLw88vLy2hcFCUXmMTdiKCrTbPvZuEta8UKIvqlLBX7x4sXk5+fz/vvvk5+fz6JFi8653Zw5c1i9ejWrV69m8eLFAQ20L1Ej4zEPu4KrbcVs3nlUJiATQvRJnRZ4p9NJUVERubm5gH/t1aKiIqqrq3s9uL7MMvZmrLjJdBXJna1CiD6p0wLvcDhITU1F0zQANE0jJSUFh+Psrom1a9cya9YsHnzwQbZv3x74aPsQNSUTdcBlzIg6wJ8/KZYhk0KIPscUqAN997vf5Yc//CFms5mNGzeyYMEC1q1bR0JCQpePcaG1BTuTnBzT4317quGqXPTCFcQ3lfLlwRHMvj7zkr5+MHIONsk5PEjOgdFpgbfb7ZSXl+Pz+dA0DZ/PR0VFBXa7/RvBJbd/f+2112K32zl48CBXXHFFl4Ppa4tud8ZImQCWSG5LPMLKDfuZMDSRSFvAPjMvSBYmDg+Sc3gI2qLbSUlJZGVlUVhYCEBhYSFZWVkkJiZ22K68/HQ/9N69ezlx4gRDhw7tdsD9iWKyYB45jUxfMUprPeu/OBrskIQQol2XmptLlixh4cKFvPjii8TGxrJ8+XIA5s2bxyOPPMK4ceN49tln2bNnD6qqYjabeeaZZzq06kOVecx0PLs38J3BJ/nfrVHMmJxOfLQ12GEJIQSKYfSdMX79rYvmlOY1T+Opq+Sfy+5g2oR07ssZ1euvGeycg0FyDg+Sc9dddBeN6Jx5zI2ozU7uHOnik6/LOFndHOyQhBBCCnwgmDKmoNhiuMqyD7NZ5U9/Kw52SEIIIQU+EBTNhHn09SgndpI3KZ4v91dSXFYX7LCEEGFOCnyAmEffAAZMizxEbKSZtz4upg9d3hBChCEp8AGixqagDR4LBz9l1jVD2H+slm0HqoIdlhAijEmBDyBz1o0YzbVMS6hkcEo0BR/sp6nVE+ywhBBhSgp8AJmGTECJSsC3/688eHsWDU0e3vzwULDDEkKEKSnwAaSoGubRN+A7vpvBkS3cdtUQPtvlYPdhZ7BDE0KEISnwAWYefQMoKp69f2X2tRnYkyJ5bf0+WlzeYIcmhAgzUuADTI1KwHTZRDz7P8Wk6Dx4exY19S7e+quMjRdCXFpS4HuBOetGjNYGPPv+RmZ6HLdcPpiPt59gX2lNsEMTQoQRKfC9QBs0Fi09G9cXb6E3OvnW9cNIiY9g5Xt7cbl9wQ5PCBEmpMD3AkVRsF3/ABg6rZ++jsWk8r3bR1NZ28qfPz0c7PCEEGFCCnwvUWOSsV5+J75jO/EWb2bUkARunJzOB1uPcfB4bbDDE0KEASnwvcicfTNqyjBcG1eht9Rz5w2ZJMXZeOXdPTQ0u4MdnhAixEmB70WKqmK7/iEMTwuuz/+XCKuJH80ZS12Tm1fXFPVo7nshhOiqLhX4kpIS5s6dS05ODnPnzuXIkSPn3fbw4cNMmDChfdWncKclpmOZNBtv8Wa8pV8z1B5L/s0j2V1SzZrPjwQ7PCFECOtSgV+8eDH5+fm8//775Ofns2jRonNu5/P5WLx4MTfffHNAg+zvLBPvQE0YROtnr2O4W7hhYhpXZw/k3c9K5C5XIUSv6bTAO51OioqKyM3NBSA3N5eioiKqq6vP2vaVV15h+vTpZGRkBDzQ/kzRTNhueBCjuRbXlv9DURTumzmKtOQoXllThLOuNdghCiFCUKcF3uFwkJqaiqZpAGiaRkpKCg6Ho8N2+/bt47PPPuOBBx7olUD7Oy1lGOaxt+LZ+zHesn1YzRoPf2scXp/OS6t34/XpwQ5RCLh41KYAABp6SURBVBFiTIE4iMfj4d///d/5j//4j/YPgp640OKxnUlOjunxvpeKPvM+jh/bjmfja6TOe5bk5Bh++t3JPP3GVt79vJQffHt8t47XH3IONMk5PEjOgdFpgbfb7ZSXl+Pz+dA0DZ/PR0VFBXa7vX2byspKjh49yvz58wGor6/HMAwaGxtZtmxZl4NxOht7NLKkP63Cbr72AVrWPsOJ9W9gu+q7jEyL4dbLB1O4sYS0pAiuGjOwS8fpTzkHiuQcHiTnrlNV5YIN404LfFJSEllZWRQWFpKXl0dhYSFZWVkkJia2b5OWlsaWLVvaf37hhRdobm7mscce63bAoc6UPgbz6Ol4dr2PedjlaCmZ3Dk9kxJHPSvX7WNAbATDB8UFO0whRAjo0iiaJUuWUFBQQE5ODgUFBSxduhSAefPmsWvXrl4NMBRZr7obJTKB1r/9DsPnwaSp/Pjb40iMsfKbt3dysro52CEKIUKAYvShlaHDoYvmFO/RHbSsfw7L5NlYp34bgIqaZp78n6+wWTT+7d6pxEZZzrt/f8z5YknO4UFy7rrOumjkTtYgMQ2ZgGnENbi3r8XnPApASkIkP7lzAnWNbp5/a4fMPCmEuChS4IPIdnU+ii2K1r/+DkP3r/g0LC2WH+Rlc+RkAy+v3o1Pl+GTQoiekQIfRIotGuu0+9Cdpbh3rG9/ftKIZO65ZSQ7ip2s+uAgfagXTQjRj0iBDzLz0KmYhk7Fve0dfDVl7c/fOHkQt101hL9uP8HaTaVBjFAI0V9Jge8DrNfeCyYrrR++hOFqan/+OzdkctWYVP70yWHe+msxurTkhRDdIAW+D1Aj44i46UfotWW0vP88htflf15ReCg3ixsmprFucyn/XVgkUxoIIbpMCnwfYRo0FtuMH+ArP0jLB7/F8Pkvumqqyn05o/j29cPYvKecZ9/8muZWT5CjFUL0B1Lg+xDzsCuwXvcAvmO7aP34FYy2ETSKopB7TQbfz83i4PE6/mPVNiprWoIcrRCir5MC38dYRt+A9cq5eA9/geuzNzqMoLlmrJ2f3j0BZ10r//zCJxyraAxipEKIvk4KfB9kmXAblom5ePb9FfcXf+zwu+yMRP71nikAPFXwFf/30SGq6qQ1L4Q4W0CmCxaBZ7n8OxjuZtw71oElEuuk3PbfDU6J5lePXM+Lf/yaDVuP8f7Wo0wZmcwtlw9meHociqIEMXIhRF8hBb6PUhQF67X3+Iv81rcwmqqxXpOPovrfsgHxEfxozliq61v5cNtxPvm6jC/3V5IxMIZbLh/M1FHJmE09n5tfCNH/SYHvwxRFxTZ9Pq7IBDw730OvdRBx88MottOTCyXG2rhr+nBmXzOUz/ec5IOtx3h1TREFG0xcPjqZq7MHMmJwPKq06oUIO1Lg+zhFVbFdNRctcRCtn6yk6Z1fEJHzE0ge3WE7q0Xjxknp3DAxjb2lNWzafZItRRV8ssNBYqyVq8YM5OrsVNKTe75qlhCif5HpgvsRX/khWjb8BsPrZuC3fkZT/MgLbu9y+9h+qJLNe8rZfbga3TBIT47i8lEpTBmdQvqAqEsUeWCEy/t8Jsk5PPTWdMFS4PsZvdFJy/u/QXcexTL1W5gzr0CJSUFRLzwgqq7JzRd7y/lyXwWHjtdhAGkDopg6KpmpbcW+r1+cDaf3+RTJOTwEtcCXlJSwcOFCamtriY+PZ/ny5WRkZHTY5u233+a1115DVVV0Xeeuu+7ivvvu61awUuC7xvC4MDa/RtPeTf4nNBNqnB013o6akI6akIYpfQyK9dwt9JoGF9sOVPLV/gr2H6vFMCA9OYrpE9O5ZuxAIqx9s+cu3N5nkJzDRVAL/H333cd3vvMd8vLyWL16NW+//TZvvPFGh20aGxuJivK3AhsbG5k1axYvvfQSo0ePPs9RzyYFvusGDIimvGgnes0JfDUn0Gsd6DVlGA1VgAGKhjZoDOahl2PKmNzhwuyZ6prcbNtfwSc7HZSebMBq1rgqO5UbJ6UzJLVvrWwfju+z5BwegrbottPppKioiJUrVwKQm5vLsmXLqK6u7rDwdnT06RdpbW3F4/H0+VP+/kxRFLSUYWgpwzCf8bzhdaE7j+E9sg3P4a20fvJ7+PR1tPQsTMMuxzzsChRLRPv2cVEWbpw8iBsnD6LEUc/H207w+e6T/O3rMjLTYrl+QhrjM5OIi7Ze+iSFEBel0wLvcDhITU1F0/xjqjVNIyUlBYfD0aHAA3z44Yc8++yzHD16lH/8x39k1KhRvRO1OC/FZEVLHY6WOhzLFXehV5XiPfwFnsNbcX2yEvdXq7Fd9wCmIePP2neoPZahd8Qy96bhbNx1ko+3n2Dle/sAGJQczdihiWQPS2TkoDgZYy9EP9BpF83u3bt57LHHWLt2bftzt99+O7/85S/Jzs4+5z5lZWU8/PDD/PrXv2bYsGGBjVj0iGEYtB4rouq9V/BUHSd63HSSbnkALeL83TCGYVB8oo6vD1SyfX8FRSVOvD4Di0ll7PABXDchnWvG24m0mc97DCFE8HRa4J1OJzk5OWzZsgVN0/D5fFx55ZVs2LDhrBb8mRYtWkRGRgYPPvhgl4ORPviu62nOhs+De9u7uL9ei2KLwXrdfZgzpnRpX5fbx/5jNew+XM3Xh6qoqmvFYlKZNDKZq7NTyR6aiNbJaJ6LIe9zeJCcu+6i++CTkpLIysqisLCQvLw8CgsLycrKOqu4FxcXk5mZCUB1dTVbtmzh1ltv7XbAoncpmhnr5d/BNHQqrX/7b1o3vIB32BVYr8lHjYy/4L5Wi8b4zAGMzxzA3908guIT9Xy+5yRb95azpaic2EgzV4xJZdTgeAYlR5McH4GqynUYIYKlS6NoiouLWbhwIfX19cTGxrJ8+XKGDRvGvHnzeOSRRxg3bhxPPfUUGzduxGQyYRgGd911F/fee2+3gpEWfNcFImdD9+L+eh3ubatB96HEDURLzURL8ffhqwnpnY6vB/D6dHYWO9m0+yQ7iqvw+vzvocWskj4givTkaAYlRzPMHkuGPQaT1rNWvrzP4UFy7jq50SlEBTJnvdaB58hX6OXF+MoPYbS2HddsQ0vJRBs4Am3gSLTUTBTThUfTuDw+yqqaOF7RyPHKJo5XNnKispH6Zv8qVBazyoj0OEYNSWD0kIRuFXx5n8OD5Nx1F91FI0KfGm/HOtE/HbFhGBgNlfjKD7V9HcT91WpOja1XkzPQBo7EZB+Flp51VsG3mjX/aBx7bIfn6xpdHDpRz/6jNew7WsufPjkM+Av+6CEJXD46hckjk/vsTVZC9EfSgu+nLmXOhqvJX+xPHsDr2I9eWQK6DzQLpkHZmDImo102EdXW9RujGprdHDhWx76jNXx9sApnfSsmTWV8ZhJXjkllfGYSVnPHoZjyPocHybnrpAUvLppijcI0ZAKmIROwAobX7S/2pdvxHtmOt3Q7KIq/ZX/ZZLTBY1Hj0y54o1tMpIUpo5KZMiqZ/JtHcLisni1F5WzdV8G2A5VYzRoThicxaUQy44YlylBMIXpAWvD9VF/J2TAMdGcp3iPb8B7Zjl59DAAlIs5/92zaGLT0LNSY5C4dT9cN9h+r5Yu95Xy1v5LGFg+aqjBycDzXTkxn+MBoUhIiezOlPqWvvM+XkuTcdXKRNUT11Zz1hkq8J4rwndiLr6wIo6UeACUmGVP6GLT0MWhpWagRsZ0cyV/sD5fV8/WhKr4+VEVZVRMAAxMjGZQcRUpCJCkJESTHR5CaEEF8jDXkFjbpq+9zb5Kcu04KfIjqDzkbhoFeU4avrAjfiSK8jn3g9i8QriYNRksbgyk9G23giA7z45yPT1X5aEspRUeqOVnTQlVtC74z/l5MmkpclIXYKDMxkRZiIy3ERJmJjbQQF20hMcZGYqyV+Ghrj4dqXmr94X0ONMm566TAh6j+mLOh+9CrjrS18IvwnTwIuhdQUOJS0ZKGoA64DG3AZahJQ85q5X8zZ103qK5vpaK2hYqaFipqW6hrdNPQ7Ka+2U1Ds4f6JneHDwEARYH4aCuJMVbiY6xtHwb+D4WYtse4KAvJ8bagz7nTH9/niyU5d51cZBV9hqJq/nH1KZkwaRaG14Xv5EF85cXozlJ8FcV4D39xevuoRDT7KLS00ZjsozEGdPxDVlWFAfERDIiPYEzGuV/TMAyaXV5qG1xUN7iorm+lut5FdYP/sayqiYbmWppaPHyzaaEASXE2UhMjGZgQSWpiBAMTIxmYFElirC3kuoNE6JECL4JGMVkxDRqLadDY9ueM1kZ8zqNtBf8wvhN78B7ahAs4GpOIkjoKbeBI1NgUlKh41KgEMEecd8SOoihE2cxE2cwXXI9W1w0aWzw0tLX8axtdlNe0UF7dzMnqZj4vc9Di8rVvbzGr2BOjsA+IxJ4URVpSJMnxEQyIiyDSJv+tRN8gf4miT1Fs0ZjSx0D6GKCtH7/Oga9sH6bqYppLduE9tKnjTiarv9hHJqDEJKHGpqDGJPs/BGKSUSJiO12bQFUVYqMsxEZZzvl7wzCob/Zw0tmEo7oZR1UzDmcTB4/VsnlPeYdtI6wmBsTZSIq1MSDORkKslfgoK7HRFuKjLMRFW4mymWS9BNHrpMCLPk1RFLT4NLT4NJKT86ioqMdoqEJvqsZoqsForkFvqvV/31Tjv5h7YGPHg5gsKFEJKLYYVFsMii0GJaLt0RaDYo3yL2/Y9qhYo1A001lxxEX5++ZHDUno8LtWt5eT1c1U1bZSVddKVV0LzrpWKuta2Hu0BpfbxzdpqkJ8tIWEWBuJMVYSY20kxFjbLwRHR/jPOvrQJTLRD0mBF/2KoigoscmosecfV2943eiNVRj1lej1lej1FRjNtRitDegNlRgVhzFaG8E4u/C2M0egxg5AjUlpe72U9jMDJSoBxXS6pW+zmMgYGEvGwLOHfhqGQavbR12Tm7pGF3VNbmob3dQ1uahtcFHT4OKIo4FtB6rw+vSz9ldVhUiriagIM1E2E1azhklTMZvavjQVk0kl0moiOsLcdpHYTHSE/4JxlM2EpW0fEX6kwIuQo5gsaPFpEJ923m0MwwB3M0ZrI4ar6Yyvtp9b6tEbqtBry9CP7QCft+MBrFGokfEokfEoUQn+qZYtEf6Wv2ZG0czQ9mWJiic1cTADE89/g5ZhGDS0eKip9xf9plYPjS0eDFWl0tlEY4uHplYPbq9Oi8uLx6fj9ep4fDoer05zq/es0UJn0lQFq1nDatGwmjVsFo3oSDMxEf4Pg+gIE9GRlvauo1NnDoYBRtvlZ5vZRKTNRKS17bHtA0e6mvouKfAiLCmK0t4l0xnD0DGaav1nAg2V6E01/jOC5lr0plr0Wgfe5lowzm6Bn35BFTU+7fQw0AGXoSUOAkuk/6xEUYhtG7t/2cDTc/p0dfjcqTOFhmY3DS0eGpo9NDZ7aG714PL4cHl0XG5f2/c+WlxeGps9nHQ209jiofUc3UhdoSoK0ZHm9u6ruCgLsdEW4qKsxEdbSIqzMSAugthIs3wQBIEUeCE6oSgqSnQianQiMPqc2xiG7m/l+zwYPs/pR68HvbEKvaoUX1UpvuO78R484xqBavJfBD7zyxaDYrGhmKzUJ8ThaTX8F5LNVvyDN88Ro8WG1RaLLSa2R1M5eLw6jS3+DwTj1KsoCor/AYBWt49ml5eWVi9NrR6aXV6aW73t9xvUNbkoczZR13j2vQdmk0pi20XnxBgr0ZFmom1moiLMRLd92SwaDc0evCXVlJbVUdPgorreRW2jC6tZIyHGSkKM/0a1U99bzCper+E/oznjrMYwON2VpSmYTRomk4JZU7GYNf+XScViVnt1FbJgkwIvRAAoigomi/+C7jd+pw24DM5YFlFvrkWvOoJe68BoaUBvqcNoqcdoqUOvPobR0tB2Axi4ehKMyXr6w8ISAYoKioqiav5qrahgMqPYYlEjY1Ei4lAiYomNiCMuqu0D5iKKnmEYNLV6qWlw4axrxVnfivPUxef6Vo5XNNLY4rlglxJAbKSZhBj/B4LL4+NoeQM7iqtwey5wptQDp7qvTp2JxJ46Ezk1qsqAFreXVpfP/+j20eryvz+RNnPHbiurCZvFhElT0DQVTVUwtT1qqoJPN/DqOj6fgc+n49MNFFXhugvcrHQxulTgS0pKWLhwIbW1tcTHx7N8+XIyMjI6bLNixQrWrVuHpmmYTCYeffRRrrvuut6IWYh+TY2MRx0yEYZMPO82hu4Fr5vEWDPOk04MrwvD64azbsfC31HuaTnjw6IB49Sjq9nfdWToGHrbo6GD1+2fJ0j3nn08RfV/OETGoUTGo0bGoUTE+a8pqCqg+rtbFNX/gaGZ/WccZhuYbSjmCCLNNiIjzaRHAANtgBWMuLZ/ABUskbgME02tXppavDS2eGhxeYmJNDM8Iwnd7cVsOvtDxjAMWlxeqhv8F6k9Xh2TST2jta5i0vxdXl6ffvpaxRnXK9weHY/X323l9vrwePX27q36JjdlVU3sK62hqbXjv40C2KwaNosJm8V/h/OpMxq39+I+dGJjIxic2Pl0Hd3VpQK/ePFi8vPzycvLY/Xq1SxatIg33nijwzbjx4/nwQcfJCIign379nHPPffw2WefYbPZAh60EKFOUU1gMWGKiUFt7Z2pktsvNLfUo7edQRjNbV8tdejNdRhNNXgrj2C01vs/SAJJNWGzRRNhjSbZFoVijQazFa08Go9XQTdZ/GcjJkvbBwuAgklRSAFSTh3HS1tsRtvnnwGqhtL2YYPFhhIZ4f/ebPMfz2QG5cIXiD1enYZmN4qiEGH1d+uc7+5lT9vF72aXlxaX/4K3z6fjbXv0+Qx8uuFvybe17k2q/9Fm0Zg0KqVXpmfotMA7nU6KiopYuXIlALm5uSxbtozq6uoOC2+f2VofNWoUhmFQW1vLwIEDAx60EOLinXmhWY23X3Bbw9Ch7QzAP7RGBwzQdQyfB8PTAu5WDK/L/+hpwfB5UFBOd+L7XxRD92G0NoGr0T9qqdU/ekmvK8fwumjW3fjcLvC4OOcZS+D+AUCz+At+24gn/yio0yOhbJoJFA1D1XApCqia/8yl7UzmdHwGZgPigDhFaT9mh0eTBQwVdAXFq7YfR/HYMPSU88d5ETot8A6Hg9TUVDTNf0qiaRopKSk4HI4OBf5M77zzDkOGDOl2cb/QpDmdSU7u+mpCoUJyDg/hmDO0LR/p82C4Xe0jlE7f+NXWWvdfDQY43Ro/9SHibkF3taC7mv2P7mZ0VyuG1+3/8rjavvegn3rO58XwnrpI7sXwtmDoPjB8/i4uXff/rPv8sbRdiD71uqBgGLr/eB7/MS84uqpNS1IcycMmBO4fr03AL7J+8cUXPP/88/z+97/v9r4ym2TXSc7hQXI+pbsXfU1ADJhi/N92MhpW4Xzjky6OYRj+5S19/g+SU9dDTp8N6aBqRA7LDM5skna7nfLycnw+H5qm4fP5qKiowG4/+5Ru+/bt/PM//zMvvvgiw4YN63awQggRShRFaevyMaGce5qjXtXpx2JSUhJZWVkUFhYCUFhYSFZW1lndMzt37uTRRx/lN7/5DdnZ2b0TrRBCiC7r0nnPkiVLKCgoICcnh4KCApYuXQrAvHnz2LVrFwBLly6ltbWVRYsWkZeXR15eHvv37++9yIUQQlyQrOjUT0nO4UFyDg+9taJT6N6jK4QQYU4KvBBChCgp8EIIEaL61GRjqtrzkagXs29/JTmHB8k5PPQk58726VMXWYUQQgSOdNEIIUSIkgIvhBAhSgq8EEKEKCnwQggRoqTACyFEiJICL4QQIUoKvBBChCgp8EIIEaKkwAshRIjq9wW+pKSEuXPnkpOTw9y5czly5EiwQwqo5cuXM2PGDEaNGsWBAwfanw/lvGtqapg3bx45OTnMmjWLH//4x1RXVwOhnfeCBQuYPXs2c+bMIT8/n7179wKhnTPAb3/72w5/36Gc74wZM5g5c2b7mhmffvop0Is5G/3cvffea7zzzjuGYRjGO++8Y9x7771Bjiiwtm7dapSVlRk33nijsX///vbnQznvmpoaY/Pmze0/P/3008a//uu/GoYR2nnX19e3f//BBx8Yc+bMMQwjtHPevXu38dBDDxnTp09v//sO5Xy/+f/4lN7KuV8X+KqqKmPKlCmG1+s1DMMwvF6vMWXKFMPpdAY5ssA78w8jnPI2DMNYv369cf/994dV3n/+85+Nb33rWyGds8vlMu6++27j6NGj7X/foZyvYZy7wPdmzn1qNsnucjgcpKamomkaAJqmkZKSgsPhOGvN2FASTnnrus4f/vAHZsyYERZ5/9u//RsbN27EMAz++7//O6Rzfv7555k9ezaDBw9ufy6U8z3ln/7pnzAMgylTpvCzn/2sV3Pu933wIrQtW7aMyMhI7rnnnmCHckk8+eST/PWvf+XRRx/lmWeeCXY4vWb79u3s2rWL/Pz8YIdySa1atYp3332Xt99+G8Mw+MUvftGrr9evC7zdbqe8vByfzweAz+ejoqICu90e5Mh6V7jkvXz5ckpLS/nP//xPVFUNm7wB5syZw5YtWxg4cGBI5rx161YOHz7MTTfdxIwZMzh58iQPPfQQR48eDcl8TzmVh8ViIT8/n23btvXq33W/LvBJSUlkZWVRWFgIQGFhIVlZWSFzKnc+4ZD3c889x+7du1mxYgUWiwUI7bybmppwOBztP3/00UfExcWFbM7z58/ns88+46OPPuKjjz5i4MCB/O53v+P2228PyXwBmpubaWjwL6xtGAbr1q0jKyurV9/jfr/gR3FxMQsXLqS+vp7Y2FiWL1/OsGHDgh1WwDzxxBNs2LCBqqoqEhISiI+PZ+3atSGd98GDB8nNzSUjIwObzQbAoEGDWLFiRcjmXVVVxYIFC2hpaUFVVeLi4njsscfIzs4O2ZzPNGPGDF5++WVGjhwZsvkeO3aMf/iHf8Dn86HrOpmZmTz++OOkpKT0Ws79vsALIYQ4t37dRSOEEOL8pMALIUSIkgIvhBAhSgq8EEKEKCnwQggRoqTACyFEiJICL4QQIUoKvBBChKj/Dx3yNiChWFEGAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt \n",
    "import numpy as np \n",
    "sns.set()\n",
    "\n",
    "x = np.linspace(0, N_EPOCHS,N_EPOCHS)\n",
    "\n",
    "plt.plot(x,t_loss)\n",
    "plt.plot(x,v_loss)\n",
    "plt.title(\"Loss\")\n",
    "plt.legend([\"Train loss\", \"Valid loss\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f3e5c073550>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAELCAYAAADTK53JAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxU9b3/8dc5Z2ayr5NtwhaIAmGTTSqyqCyNYjTUWrFU21sttnW7bfVeufUW8GqXeH+1VotdvC2WS28XaxUbqbVqVUBFRZRASFgSCCSTbbKvM3PO+f0xEJsSyCRMMpPJ5/l45JHJ5MzM55vlnW++53u+X8U0TRMhhBBhRw12AUIIIYaGBLwQQoQpCXghhAhTEvBCCBGmJOCFECJMScALIUSYkoAXQogwZQl2AUJcqGXLllFfX4+maT33vfzyy6Snp/Od73yH9957jxMnTvC9732PG264IYiVCjG8JOBFWPjZz37G5Zdfftb9U6dOZdWqVfz3f/93EKrqzev1YrHIr5wYPjJEI8LaF77wBRYuXEhERES/x7755pusWrWKOXPmsGTJEn75y1/2fO7VV18lPz+fuXPnsmLFCt566y0Aampq+NrXvsaCBQtYuXIlf/jDH3oe8+STT3Lvvfdy//33M3fuXJ5//nlaW1v59re/zeLFi1myZAk/+tGP0HU98A0XAunBC9HjwQcf5PHHH2f+/Pk0Nzdz6tQpAPbv388DDzzAE088wcKFC6mrq6OtrQ2A++67j4suuoidO3dSVlbGl7/8ZcaNG8fChQsBeO211/jxj3/Mo48+itvt5lvf+hYpKSm88sordHZ28tWvfhWHw8HNN98ctHaL8CU9eBEW7rrrLubPn8/8+fO58847B/UcFouFo0eP0tbWRkJCAtOnTwfgj3/8I5/97GdZtGgRqqqSnp5OdnY2TqeTvXv3cv/99xMREUFOTg6f+9zn2L59e89zzp49mxUrVqCqKm1tbbz11lt8+9vfJjo6Grvdzr/8y7/w0ksvBeRrIMQ/kx68CAubN2/ucwx+IJ544gl++tOf8sMf/pApU6Zw3333MWfOHJxOJ1dcccVZx9fW1pKQkEBsbGzPfZmZmRw4cKDn44yMjJ7bVVVVeL1eFi9e3HOfYRg4HI4LqluIc5GAF+K0WbNm8dOf/hSPx8NvfvMbvvGNb/Dmm2/icDioqKg46/i0tDSam5tpa2vrCXmn00l6enrPMYqi9NzOyMjAZrPx7rvvyslWMSxkiEaENbfbTXd3N6Zp4vV66e7uxjCMPo978cUXaW1txWq1EhMT0zPt8sYbb+RPf/oT77zzDoZhUFNTw7Fjx3A4HMyZM4fHHnuM7u5uSkpK+OMf/8h1113XZy1paWksWrSIH/zgB7S1tWEYBhUVFbz33ntD+jUQo5cEvAhrt99+O7NmzWLfvn185zvfYdasWbz//vt9Hrt9+3aWLVvG3Llz+d3vfsejjz4K+Hr23//+9/ne977HvHnzuOWWW6iqqgLgscceo7KykiVLlnD33Xdzzz33sGjRonPW8+ijj+LxeFi1ahWXXnop9957L3V1dYFvuBCAIht+CCFEeJIevBBChCkJeCGECFMS8EIIEaYk4IUQIkxJwAshRJiSgBdCiDAVUpfTNTa2YxgDn7Vpt8ficrUNQUWhS9o8OkibR4fBtllVFZKSYs75+ZAKeMMwBxXwZx472kibRwdp8+gwFG2WIRohhAhTEvBCCBGmQmqIpi+madLYWIfb3QX0/S9Mba3a5wJS4WwgbdY0C7GxiURFnXusTggRfkI+4NvamlEUhfT0sShK3/9wWCwqXu/oCnh/22yaJh6Pm6Ym34JWEvJCjB5+BXx5eTnr16+nqamJxMRECgoKyMrK6nXMv//7v1NaWtrzcWlpKZs3b2b58uUXVGBnZxvJyennDHdxfoqiYLNFkJiYSnNzvQS8EKOIXwG/ceNG1q5dS35+Ptu3b2fDhg1s3bq11zFnllYFKCkp4Utf+hJLliy54AINQ0fTQv4fjZBntdrQdW+wyxBi1DJNs9cGMMOh3+R0uVwUFxezZcsWAPLy8nj44YdpaGggOTm5z8ec2fTAZrMFpMjh/qKEI/kaCnF+pmmAuxOzqw2zux2z+/R7TzfoHtA9mLoHvKffKwqKJQLFFgmWSBRrBIo1ChMDs60Bo82F2ebyvW91YXa2gKqBxYqi2cBiQ7HYUCJicK++G4gPeJv6DfgzW5Cd2d1G0zTS0tJwOp19Brzb7ebPf/4zzzzzTMCLDbZ1676Ex+PB6/Vw8mQFEydmAzB58hS+/e2Nfj3HCy/8ke7ubtas+cKAXvvuu++gpqaGmBjfEMuECRN46KHvc+RIKY89VsDhw6UsXLiIRx55tJ9nEmJkMXUv6G4wTV8ImyaYBhgGKAqoGoqqgWoBTQNF84Xx6XA1Wut9t1vrMTuafOF8JrC9/xDc3R2cayJHL4oGmsVXh+4+93GaFSXWjhprRxt/CUp0Ahj66T8SbkyvG7xu3x8KVYMhOI0Y8LGPV199lczMTHJycgb8WLs99qz7amtVLJb+x9/9OeZCbdnyv4Bv8+Qvf/kWtm373VnHeL3e8+63eeONNw3qtRVF4b77/o3Fi5f2uj8lJYV//df7OHKklPfe23Per4OqqqSmxg3q9UPFSK9/MMK9zabXg7vuJO7a43hbXNS970JvbcDb2oC31YXR0XLhL6KoWOLtWOKSUSNiUCzW0282FM13W42MQYuKQ42KRYuMQ42OQ42MQbVG+o478xhV+6R2Q8f0dGO4uzDcXZinZ/tZ4lNQo+MH9J9z6oW38iz9BrzD4aCmpgZd19E0DV3Xqa2tPedO8M899xyf/exnB1WMy9V21tVchmH0O1tkuGfR6LoBmD2veeON15GXl8/eve+TmTmGO+64k02bHqS9vR23283lly/izjv/FYBf/vLndHZ2cvfd32DHjj/zt7+9TFxcPGVlx4iLi+WRRx7Fbk856zVN00TXP3nNM21OSrKTlGSnrKzs9L6j5/46GIZBXV1r4L8gwyQ1NW5E1z8YodxmU/diNFZitNT2HpJob8Bsc2HqXtS4FNS4FJS4VN/t2BSwRWK4TqK7TmDUV2A0VoGp9zyvGh0PUUkoMYloyVlYYhJRLBGgqL4e+z++N00wdF/P2NDB8Po+VjXUWDvKmdePTuwVzGe1BdBPv/ViAN2n3/CcfuuL1fdmOf3HuAPo8H/pgcF+n1VV6bNjfEa/AW+328nJyaGwsJD8/HwKCwvJycnpc3imurqavXv38sMf/nDAhfpjd5GTXfudZ92vKL7v84VYPMvBopl9/9HyR319PU8++XMAuru7KSj4EdHR0Xi9Xr71rbt59923ueyyy8963KFDxfz6178lPT2DgoJH+OMff89Xv3pXn6/x+OP/j6ef/ikAN9/8ea6+uu/NnYW4UOaZIRBTB8PANLwYjVUYrgr0+goM1wmMxkpfmJ5hjUSNTUaJSUazjwdVw2itx2iuxjh1wDcc8Q+UqHjUlAnYxl+CmjIeLXk8SpydtIzkkP2jNtL4NUSzadMm1q9fz1NPPUV8fDwFBQUArFu3jnvvvZeZM2cC8Pzzz3PVVVeRmJg4dBWHqKuvvrbntmEYPPXUjykq2g+YuFwujhw53GfAz5p1CenpGQBMnz6D99/fc87X+MY37mfRIt/MpNE4918Enql70evK0CsPoVcVo9eW+8anzzMWrUTFo9rHYxs3E9U+ATUxAzXWDrbocw5JmKaJ2dWK2VqP6e5ATR6LGj36cmK4+RXw2dnZPPvss2fd//TTT/f6+Otf/3pgqjqHRTP77mWHQthFR0f13P79739Da2sLv/jFM0RERFBQ8F3c7u4+H/ePM41U1TcEJsSFMrvb0RtOYXY2nz4pefrE5On3RmczelUJevXh0z1rBTVlPNacK1BsUb4TiT0nMFVQNNSEdFT7eN9wxwBnZSmKghIVD1GBnykizk0mmA+B1tZW7PYUIiIiqKurZdeuN1m9enDnJYQwDa9vjLu90Tckcmb8GUBRUcA3W6ThJHrDKQzXScz2hn6fV00ag3XKUrTMHCyOKSiR5x7LFSOTBPwQ+NznbuY733mAL395LWlp6cybd+mQvZbTWcWdd36Frq4u3O5uPvOZVdx++x3k5a0estcUF840DMy2et88a3fn6fcdPfOwz5ywPNHuQm9r8O8kk6qhJjrQHFNQk8eiJY9DiU3y/RH4pxOUijVKAn0UUEzzQk9PBk5fs2iqq0+QkTHhvI8LhSGa4TaYNvvztQxloTyjxF+muxNP6Vu4D7yK2VrX90GKihKThBqXQlSKA7c1wTcjJDYZNOsnQy3Qc1uJSURNcKCEwVXf4fB9HqigzaIRQlw4o6UO98FX8ZS8CZ4utPSLscy+FjUqASKiUWzRKLYolIhosEb2rL00GsNOBI4EvBABYHq6Tl/W3gWeLkxPN6anE9xdeI9/iPfEh4CKJftSbDM+jZY2Kdgli1FAAl4IP5meLgzXSYyWGoyWWoyWOt9FPi21mF3n6WVHxGC7ZBXWactRY/tev0mIoSABL0QfTG83Rn0Fev1x9LpyjLrjGE1OeuaHKwpKTDJqfBpa1hyUuDSUqDgUaySKNdI3zGI9vQBVTDKKJTAL7wkxEBLwQuC7EMdoOIX35MfoFfvRa476TmACSlQCamoWtuwFaCkTUBMdKLEpYXFCU4Q3+QkVo5bp6UavKsZb8THeiv09c8fVlAnYLlmFlpaNmpqFGpMU5EqFGBwJeDFqmKaJ2VyD9+THeE8WoTtLQPeCNRLLmOlo8/KxjJslgS4A6Oz2UlbVQkpiJOlJ0cEuZ1Ak4AfgW9+6h6VLr+x1Vappmtx0Uz4PPriJ2bPn9vm47353E1On5vDZz64573rwO3b8mbff3tnnmu6yHnz/TNP0rbPt7sDs7jh94VA7ZncHeu0xXy/99NxzNdGBddpyLONmoTkmo2jWIFcvgs2rGxx3tnLweAPFxxsoq2pBP31dzsVjE1g808H8qWlERZwdm91unUMVjRwsa+BkXRuGaWIaJoZ5umNhgomJRVOxaipWi+/NoqlERVi4LX/GkLRJAn4Arr32en7/+9/0Cvh9+/aiado5w/2frV5946Bfv6/FxpKSkrn77m9x5EgpH3xw7oXKwpFpmhhNVeiVxXhPHfT1yD1dfR9ssfkuyb/kGizjZqLGDcXq22Ik0Q2Dipo2SiuaKK1opPRkE11uHQWYkBFH7oLxTBmfyMnaNnbtd7LlLyX85tXDzJ+SxqKZDuKirRwoa6CozMWRU014dRObVWVCehxWTUW1gKIqqIrCmZV7vIaJx2vQ0e3F027g8RooCrS0u4m1Bn5PixEV8J7Du/GUvnXW/YqicKEX5FqnLMU6edF5j1m69Eoee+wHlJeXMXGibx7zSy+9yKpV13Hs2FF++MMf0NXVidvt5vrrP8NNN6096zn+cT14j8fDj370KPv27SU1NY3x47MGXHdKSiopKamcOFE+4MeORKanC2/5XryVB9ErizE7mgBQ4tOwZl+GmpDmW9Ww5+Kh02+xMpMllDW2dnO8uoUJ6XED2uCky+2luqEDp8v3Vu1qx9nQQWe3l+S4SOwJkdjjI7HHR2BPiCTCqnG0spnSk00cPdVMl9u3uF96UhSXTc9g2oQkpk5IIjbqk//oZk6yc82nxlNW1cKuIifvHarh7QPVPZ8fkxrDinnjmDEpmYvHJmIdxOZDQ3VB24gK+GCzWq2sXHk1f/nLn7nzzn+lo6OdnTvf5Gtfu5uYmBgef/wpbDYbHR0d3HHHl1iwYCFZWRPP+Xzbtz+H01nF//7vH/B6vdx117pzbqQCo3s9eL2xivoPd9L28d/B04USGYc2ZhramGlYxkyTHvkI4/HqHD7VzMGyBg6UuzhV197zuSxHPDkTEpk1yU72mAQsmi8wdcPgVG07RyubOVbVzLHKZuqaPvmPTVEgLTEKhz2GqAgLja1dHKts5oOS2p6hljPGpMSwcHoGU8YnMnlcIomxEeetV1EUssckkD0mgZuXX8xHR+rp9ujMmJhMcnxkAL8ygTWiAt46eVGfvezhXIvm2mvzuf/+e7jjjrt47bW/MWvWJaSmptHQ4OInP/kBR48eRlFU6uvrOHr08HkD/sMP93LNNXlYLBYsFgu5udewf/9H5zx+tK0HbxpevMf34Sl+Hb3qEGgWLBMvxTrtKrT0i3ou5xehp8vtpaahk9ZON22dHto7vbR2uGnv9FLT1MHhiibcXgOLpnDx2ERuvDKDSY54jle3UnKyiVfeO8lf3q0gKkIjZ0Iy7Z0eyqtbcHt8P/MJsTYuyvSNizvsMThSYkhLjOqz92wYJs3tblwtXXR0eclyxBEfPfj/5iKsGp+alj7oxw+nERXwoeDiiydjt6ewZ8877NjxYs8wzM9/vpnkZDu/+tVvsFgsfPObd+F2n2dDXrjgYaVwZXa34y7+O56Dr2J2NKHE2rEtuBHH5ato6JBQD0Vuj87RymYOnWikpKKR8qpWjD5+vmMiLSTGRrDkkkxmTExm6vgkImyfbKU3dUISt+ZNp+JUI8XHGykqq6f4eCOxUVaWzMoke0w8F41JwB4f6fea9KqqkBQXQVLc+Xvp4UgCfhCuvfZ6fvWrX1BT42Tx4isAaGtrJTv7YiwWC2VlR/n4449YufLq8z7P/PmX8vLLO1i2bCW67uVvf3u5Z3en0chob8Rd9Fc8h97wLcg1dga2Jf+CNm4WiqqixcRBhyy8NdwM06Sjy0tbp4fWDjdtHR5aT99u7fBwvLqVsqpmvLqJqihMdMRxzWXjycqIIy7aRmyUldhoKzGRFjTVvz/QUREW5k1JZd4UGXq7EBLwg7By5TVs3vwE+fk3YLX6TsZ86Uu38/DDG3jllb8wZswYZs+e0+/zXH/9DRw9epRbb72JtLR0Zs+eh9NZOaBawmE9eL2xCvfHf8F79G0wTSzZC7DNugYtZeQubTwSdHt0mlq7aWjpoqG1m4bWbppau3uCuyfQO7199sYBbBYVR4rvJOPUCUlcPDahz2mEIjhkPfgRaqSvB2+aJrqzBPf+v6JXfASaDevUJdhmXo0a33evbTQunRvINnt1gwPlDbxzoJpDJxpp6/ScdUxMpIX4GF+vOy7aRly0tdftuNO98bgoG7HRViKsWh+vdGHk++y/gKwHX15ezvr162lqaiIxMZGCggKysrLOOm7Hjh389Kc/xTRNFEVhy5YtpKSkDLhoEb5M3Yu37D3c+/+K4TqBEhmHbW4+1hkrUCP9nx4n/GOaJserW3nnQDV7DtXQ2uEhNsrK7ItSSE2KIjkuwvcWH0liXMSQBLYIHr8CfuPGjaxdu5b8/Hy2b9/Ohg0b2Lp1a69jioqK+MlPfsKvf/1rUlNTaW1t7bWhtBjdzK423CVv4DngO3GqJmYSsfTLWC9aKPPTA6jbrVPlaqeqvp3KunY+PlaP09WBRVO45KIULp+RwcxJ9p6phyK89RvwLpeL4uJitmzZAkBeXh4PP/wwDQ0NJCd/srb1M888w2233UZqqu/f67i4wPXGzvxHIAYvWCNxpteN+8DfcO8rBE8n2pjp2K64DW3sTPmenodpmri9Bq7mTirr2ujo9tLR5aXbo9Pt1n3vT791uXXqm7qorG+jvqnrzILGWDSVSY44Vl49hUunphETKcsxjDb9BrzT6SQ9PR1N8/3rpmkaaWlpOJ3OXgF/7Ngxxo4dyxe+8AU6OjpYuXIlX//61y/4l1hVNXTdi8UiP5wXwuNxow3j8ramaeA9+i7d7z+H2eZCGz+biEs/i2YfN2w1jCSGaXLkZBO7D1RTVOaircNz1sU5fVEAm03DHh9JVkY8i2Y4GJMaw5jUWFITI/2etSLCU8B+43Vdp7S0lC1btuB2u/nKV75CZmYmq1f7P5ujr5MFpmmntbWZpKSU817YYhnE5cEjnT9tNk0Tt7ub1lYXY8Y4SEgY+nHuzhMHcL26FXf1MWwZk7Dn30NU1syAPPdALmMfCZz17fx970le/+AkNQ0dREVoLJjmIDUpipgo3wnOmDNvkRaiIixE2ixE2DQiIyzYLGpY/icUbt9nfwxFm/sNeIfDQU1NDbquo2kauq5TW1t71iX1mZmZXH311dhsNmw2G8uXL2f//v0DCvi+ZtFAJF5vK5WVFfTspvNPVFXFMEbXLJqBtFnTLMTGJuJ2q0M6O0GvPUb3h39Gr/gIJSaZyKvuwHLRZbQpKm0BeN1wmV1R09DB/mMuPiit5cipZhRgWlYS1y/KYu7Fqb0u/OmzzbqOu1PH3Xn+C+lGqnD5Pg9E0GbR2O12cnJyKCwsJD8/n8LCQnJycnoNz4BvbP7NN98kPz8fr9fLu+++S25u7oAL/meKopCcnHbeY+QHInhM00Q/WYT745fQnaW+/UcX3Ihtxqfl5Olp3R6d0opGio75Vh6sbeoEIDMlhs9eMYmF0zNCej0TMXL5NUSzadMm1q9fz1NPPUV8fDwFBQUArFu3jnvvvZeZM2dy7bXXcuDAAVatWoWqqixevJgbbxz80rgitJmGF++x93B/vAOj4RRKTDIRl30ea84Vvj1JRzmvbvDx0XrePlDNgfIGPF4Dm0Vl6oQkVl46jpnZdtISo4JdpghzIX+hkz9CpTc7nILVZqO9EU/pTjwlb2K2uVCTMrFdsgpL9mVDvkdpqH+fTdPkRE0ru/f75py3dXpIiLVx6ZQ0Zl1kZ8q4RKyWgc0zD/U2DwVps/8CcqGTGN1M3Yu34iM8JW+hnyoC00TLzMG26Ba08ZeM6lUdvbrBiZpWSk408u7BGirr27FoKnMnp3D5DAfTJybJTBYRNBLw4pyMpmrcJW/gPbwbs6sVJToR2yXXYp26FDX+/OdFwtWZbd1KTzZSUuHbNKLb49s0YlJmPLfmTmFBjsw5F6FBAl70YhqGr7d+8DX0yoOgaFgmzMY6dYnv4iR19F3K3tntZf8xF3tLaykqa+gJ9DGpMSyamcGU8UlMHpdIQoycVBahRQJeAGB0tuApeQvPob9jtrlQYpKxzb/B11uPTgx2ecOutcPNR0fq2Xu4juLjDXh1k4QYGwtnZDA9yxfocRewaYQQw0ECfpQzTQP3e3/EXfQKGF60zBysCz+PZcKcUdVb73J7OXKqmZKKRkpONHG8ugXTBHt8JMvmjmXelFSyxySghuFFRSJ8ScCPYqah0/XWr/Ae3o3l4kXYZl+LlpQZ7LKGxJlNK9o7feucn3mrbuigpKKR485WdMNEUxUmZsaTtzCLuZNTGZ8eG5ZXiorRQQJ+lDJ1D12v/Qzv8b3Y5n8G25zrwybI2jo9nKhu5Xh1C8erWzlR3YqrpYu+JgRrqkKWI46rPzWeqeOTuGhMQq8rSYUYySTgRyHT003n355EP3WAiIVrsc38dLBL8ptpmlTUtNHQ0uXrhXd5aOvZfcjDqbo26pu7eo5PS4xioiOey6anExtp7Vnf5cxbYmyEBLoIWxLwo4zZ3U7ny4+j1x4l8orbsU5ZEuyS/Ob26PxqxyF2F1X3ut+iKT3BneWI58o5Y8jKiGNCRpxMVxSjmgT8KGJ0ttC54/9hNFYSufzrWCctCHZJfmts7abg//ZRWtHItQsnMG9Kak+PPNKmhc3wkhCBJAE/CpimiffEPrrf+S1mRzNRuf+KZdysYJflt2OVzfzk+SK63Tp3fWYG86aMzoushBgoCfgwp7tO0v3O/6FXHUJNzCTq2n9Dy7g42GX5bef+Kv73r6UkxkbwyL2LiLFIT10If0nAhymjswX3+3/CU/om2KKJuPwWrNOuRFFHxrfcqxv84e9HefWDU+RMSOLrq2eQ5YgfdYtQCXEhRsZvu/Cbaeh4il6h+8Pt4PVgnb6SiHn5KBExwS6tX2dmyOwucvJusW81xpXzx3HTsmxZsEuIQZCADyNGSx2df/85Rs1RtPGXEHnZzaiJjv4fGGQt7W7ePVjNrqJqTtW1YdEU5lycytLZmUzPSu7/CYQQfZKADxOeI2/TtWsrKAqRy76G9aLLgl3SeXV2e/noSD17DtVwsLwB3TCZ6Ijn1k9P5tKcdGKjZHqjEBdKAn6EM90ddO3aivfou2gZk4m86g7UuJRgl9Wnbo/O/mMu3iuu4eNjLry6gT0+gk8vGMeiGQ4yU0J/GEmIkUQCfgTzVh+m6/WfY7Y3Ypt/A7bZeSghMlbd2e3F6erA6WrH6eqgqr6dQyca6fboJMTYuHJ2JgumpTMpM14W8BJiiEjAj1Btxbvp/POPUGJTiM5/EC0tO6j1GIbJR0freevjKipqWmlqc/d8TlMV0pKi+NS0dD41LZ0p4xJRVQl1IYaaXwFfXl7O+vXraWpqIjExkYKCArKysnod8+STT/J///d/pKX5LkKZO3cuGzduDHjBwje3va5wM2paNtHX3IdiC97mzR1dHnbud/La3lPUN3eRHB/BtKxkHPZoHPYYHPZoUhOjsGih8Z+FEKOJXwG/ceNG1q5dS35+Ptu3b2fDhg1s3br1rONWr17NAw88EPAixSfMrjY6X3kCNSKayJV3By3cqxs6ePWDk+wuqqbbozN5bAI3XXURcyanyJRGIUJEvwHvcrkoLi5my5YtAOTl5fHwww/T0NBAcrJMYRtOpmHQ+fefY7Y3kH7rw7RGDO9OS11uL3tL69i130npySYsmsKnctJZMX8cEzLihrUWIUT/+g14p9NJeno6muZbUlXTNNLS0nA6nWcF/EsvvcSuXbtITU3lnnvuYc6cOUNT9Sjl/uBP6CeLiFj8JSLHTqF1GK7qNE2Twyeb2F1UzfsltXR7dNKSorhh6SSWXJIp+5AKEcICdpL15ptv5mtf+xpWq5Xdu3dz5513smPHDpKSkvx+Drs9dtCvn5oa3j3I9pJ3af2okLjZK0hZeh0w9G3e/XEVz7x0kGpXB1ERFpbOGcOKBePJyUoO2uqN4f597ou0eXQYijb3G/AOh4Oamhp0XUfTNHRdp7a2Foej9xWSqampPbcXLVqEw+HgyJEjLFjg/5K0LlcbhtHHtjv9SE2NC+s1SvTGSjpefBI1dRLmvDXU17cNaZt1w+C5N8t4eU8FE9Lj+EpeDvMmp45lNb4AABuwSURBVPVsjFFf3zYkr9ufcP8+90XaPDoMts2qqpy3Y9zv2TC73U5OTg6FhYUAFBYWkpOTc9bwTE1NTc/tQ4cOUVlZycSJEwdcsOjNdHfQ+cqTKBYbUSvvRtGG9grP1g43j/3+Y17eU8FVc8fw4BfncfkMh+x6JMQI5NcQzaZNm1i/fj1PPfUU8fHxFBQUALBu3TruvfdeZs6cyWOPPcbBgwdRVRWr1cqjjz7aq1cvBqfr7d9gttQRlffvqLFDe1L7eHULm/9URHO7h9tW5bB4VuivYyOEODfFNPvaijg4ZIimN2/1ETpf/C622XlELLix1+cC3ebdRU5+/XIp8TFW7vrMTCY64gP23IESrt/n85E2jw5DNUQjV7KGKNPQ6d69FSUmGduc64bsdSrr2yl8+zh7imvImZDEV/OnEx8tM2OECAcS8CHKU/w6huskkSvuQrFGBPz5j1U2s+PdE+w7Uo/NopJ3+QTyF0+Ui5SECCMS8CHI6Gim+4M/oY2ZjmXi/IA9r2maHCxvYMe7JyipaCIm0sL1i7JYPm8scdJrFyLsSMCHoO73ngWvm4hFXwjYfPNyZwvbXiml3NlKUlwENy+7iKWzM4m0yY+AEOFKfrtDjF59BO/hXdguWYWWmHnBz9fR5eX5t8p4/cNTxMfa+JdrprJwegZWiwzFCBHuJOBDiGkYdO3+X9+J1bnXX9hzmSbvl9Ty21eP0NLuZtm8sXxmySSiI+VbLsRoIb/tIcRz6HUMVwWRy+9EsUYO+nlqGjvY9sphDpY3MCEjjntvnBWS0x6FEENLAj5EGJ0tdL//J7TMHCyTLh3cc5gmr75/kufeKkNTFb6wcjJXzRkjm2sIMUpJwIcI9/t/Ak83EYtuHdSJ1bqmTn750iEOn2xi9kUp3Jo7haS4wE+vFEKMHBLwIcDsasNzZBfWqUvRkgZ2YtU0Td76uIrfvX4UVYHbVuWwaGZG0FZ7FEKEDgn4EOA5sht0L9ZpVw3ocY2t3TzzlxKKylzkTEjitlU52BMGP3YvhAgvEvBBZpomnkNvoqZNQrOP9/txR082seGXe/B4Dd9Y+9wxqNJrF0L8Awn4INNrjmA0VRG59Da/H+PVDX70uw+xWTUe/OJ8MpKjh7BCIcRIJVe7BJnn0BtgjcKS/Sm/H/PSOyeoqG7li7lTJNyFEOckAR9EZlcb3rL3sF680O8FxSrr2ih8+zhL54zhkotShrhCIcRIJgEfRJ6j7/hOruZc6dfxhmGy5S8lREVYuGP1zKEtTggx4knAB4nv5OobqKn+n1x9be8pyqpa+PyKi0mIlTnuQojzk4APEr3mKEZjJdacK/w6vq6pk+feOsbMSXYum5Y+xNUJIcKBBHyQ+E6uRmL14+SqaZpsfbkERVH4Yu4UuYhJCOEXCfggMLvbT59cvdyvRcV2F1Vz8HgjN16RLRcyCSH85lfAl5eXs2bNGnJzc1mzZg3Hjx8/57FlZWVccsklFBQUBKrGsOM58jboHqxT+x+eaW7r5vevH+GisQlcNXfMMFQnhAgXfgX8xo0bWbt2LX/9619Zu3YtGzZs6PM4XdfZuHEjK1asCGiR4aTnytXUiWgpE/o9/o9vHqPbo/Pla6bKlapCiAHpN+BdLhfFxcXk5eUBkJeXR3FxMQ0NDWcd+4tf/IIrr7ySrKysgBcaLozaYxiNp/yaGul0tfP2gWqWzR2Lwx4z9MUJIcJKv0sVOJ1O0tPT0TQNAE3TSEtLw+l0kpyc3HNcSUkJu3btYuvWrTz11FODKsZujx3U4wBSU+MG/djhVPvubhRbJI5PLUe1RZ332Gf+WorNqnHrtdNJ7GPp35HS5kCSNo8O0ubACMhaNB6Ph+985zt8//vf7/lDMBguVxuGYQ74campcdTVtQ76dYeL0VxN+8GdWCcvxtXsBc5d86m6Nnbuq+Tqy8bj6XJT1+Xu9fmR0uZAkjaPDtJm/6mqct6Ocb8B73A4qKmpQdd1NE1D13Vqa2txOBw9x9TV1VFRUcEdd9wBQEtLC6Zp0tbWxsMPPzzgosORaRp0vfkr0GzY5ub3e/z2neVE2DSu+VT/4/RCCNGXfgPebreTk5NDYWEh+fn5FBYWkpOT02t4JjMzkz179vR8/OSTT9LR0cEDDzwwNFWPQJ6Dr6FXHybyyq+gxiSd99gT1a3sPVzH9YuyiI2yDlOFQohw49csmk2bNrFt2zZyc3PZtm0bDz30EADr1q2jqKhoSAsMB0ZLLd17nkUbNwvLxYv6Pf6FnWVER1j49KXjhqE6IUS48msMPjs7m2efffas+59++uk+j7/nnnsurKow4hua+SVoGpFLv9zvVajHqpr5+JiLzyydRHSk9N6FEIMnV7IOMU/x6+jOUiIv+3y/QzMAL+wsJzbKyop5Y4ehOiFEOJOAH0I9QzNjZ2CZsqTf4w+fbOJgeQPXXDaeqAjZbEsIcWEk4IeIaRp0vbUFFMWvoRnTNHn+rTLiY2wsmyu9dyHEhZOAHyKeQ2+gVx0i4rKbUWPt/R5/6EQjpSebuHbhBCKsg7+WQAghzpCAHwJGZwvde/6ANma6XwuKAby4q5ykuAiunJ05xNUJIUYLCfghYNQeA08Xtnn5fq3dfqK6lcOnmvn0peOwWqT3LoQIDAn4IaA3VgKgJfm3vO+re08SYdVYMsvR/8FCCOEnCfghYDRUosQkoUT0vwJkS7ubPcU1XD4zQ+a9CyECSgJ+CBiNVah+9t7f/KgSr26yXGbOCCECTAI+wEzDwGjyL+C9usHr+yqZPjGZzBRZ710IEVgS8AFmttWD7kFN6n82zN7SOprb3HLVqhBiSEjAB5jR4P8J1lf3niQtKYqZ2f3PkxdCiIGSgA+wMzNo+huiKXe2cKyyheXzxspeq0KIISEBH2BGYyVKTDJKP9vxvfrBSSJtGotnytRIIcTQkIAPMKOxEjX5/L335rZu3jtUy+KZDllUTAgxZCTgA8g3g8bZ7/DMGx9VoRsmy+XkqhBiCEnAB5DZWgu657wnWD1eg7/vq2RWtp305OhhrE4IMdpIwAeQPydYPyippaVdpkYKIYaeBHwAnZkieb458K/uPYnDHs30icnnPEYIIQLBrzN85eXlrF+/nqamJhITEykoKCArK6vXMc899xzPPPMMqqpiGAaf+9zn+OIXvzgUNYcso7EKJdaOYo3s8/Mna9sod7by+RUX+7XKpBBCXAi/An7jxo2sXbuW/Px8tm/fzoYNG9i6dWuvY3Jzc7nhhhtQFIW2tjauu+46FixYwNSpU4ek8FBkNFaed3jmnQPVaKrCZdPSh7EqIcRo1e8Qjcvlori4mLy8PADy8vIoLi6moaGh13GxsbE9vdKuri48Hs+o6qWahn7eGTSGYfJOcTUzJ9mJi7YNc3VCiNGo3x680+kkPT0dTfNtRKFpGmlpaTidTpKTe48jv/baazz22GNUVFRw3333MWXKlAEVY7fHDuj4f5SaGjfoxwaC21VJm+Elcfwk4vqo5cPSWprb3Fy9aGLAag12m4NB2jw6SJsDI6BX2Sxfvpzly5dTVVXFXXfdxdKlS5k0aZLfj3e52jAMc8Cvm5oaR11d64AfF0ie8sMAtFvsdPVRy192lxEdYWFiakxAag2FNg83afPoIG32n6oq5+0Y9ztE43A4qKmpQdd1AHRdp7a2Fofj3JfYZ2ZmMnPmTN54440BFzxSGY3nnkHT2e3lw9I6FuSkYbXIxCUhxPDoN23sdjs5OTkUFhYCUFhYSE5OzlnDM8eOHeu53dDQwJ49e5g8eXKAyw1dRmMVSlxKnzNoPjxch9trcPkMWXdGCDF8/Bqi2bRpE+vXr+epp54iPj6egoICANatW8e9997LzJkz+f3vf8/u3buxWCyYpsktt9zC4sWLh7T4UHK+GTRvH6gmLTGK7DHxw1yVEGI08yvgs7OzefbZZ8+6/+mnn+65/e1vfztwVY0wvhk01djGzTrrc67mLkpONJK/eOKomlUkhAg+GRAOAKOlBgxvnz34d4urMYHLZmQMf2FCiFFNAj4APlmioHfAm6bJ2wequXhsAmmJ518fXgghAk0CPgCMxipAQU3qfRL1eHUrTlcHl0vvXQgRBBLwAWA0Vvpm0Fgiet3/9oFqLJrKpVPTglSZEGI0k4APgL5m0Hh1gz3FNcy+OIXoSGuQKhNCjGYS8BfINLwYzdVo/7RN34GyBto6PTI8I4QIGgn4C2Q014Chn9WDf/uAk7hoKzNk3XchRJBIwF8go49dnDq6PHx0tJ5PTUvHosmXWAgRHJI+F8hoqARFQU38ZAZN8fFGvLopJ1eFEEElAX+BfDNo0lAsn6zxfqiikQirxkSHLE0ghAgeCfgLZDRWof3TCpIlJxq5eGyCDM8IIYJKEugCmLoXo7mm1/h7c1s3TlcHUyckBbEyIYSQgL8gRnMNmHqvNeBLKpoAmDpeAl4IEVwS8Begrxk0JRWNRNo0JmQMfvtBIYQIBAn4C2A0OQF6zaApqWhi8rhENFW+tEKI4JIUugBGczVKrL1nBk1jazc1DR0yPCOECAkS8BfAaK5BTfhkKYKSikYAcuQEqxAiBEjAD5JpmhjN1agJ6T33lZxoJDrCwrg0GX8XQgSfBPwgmV2t4O7oHfAVjUwZn4iqytZ8Qojg8yvgy8vLWbNmDbm5uaxZs4bjx4+fdczmzZu59tpruf7667nhhhvYuXNnoGsNKWZzDUDPEI2ruYu6pi4ZfxdChAy/Nt3euHEja9euJT8/n+3bt7Nhwwa2bt3a65hZs2Zx2223ERUVRUlJCbfccgu7du0iMjJySAoPNqO5Gvgk4M+Mv8sFTkKIUNFvD97lclFcXExeXh4AeXl5FBcX09DQ0Ou4JUuWEBXl23d0ypQpmKZJU1PTEJQcGozmGlA0lLgUwDf+HhtlZUxqTJArE0IIn3578E6nk/T0dDRNA0DTNNLS0nA6nSQn973W+QsvvMD48ePJyBjYZhd2++BPTqamxg36sYNR01WPmZxOWnoipmlyuLKZWRenkJ42fAuMDXebQ4G0eXSQNgeGX0M0A/Hee+/x4x//mF/96lcDfqzL1YZhmAN+XGpqHHV1rQN+3IXorD2FEptKXV0rtU2d1DV28un544atjmC0OdikzaODtNl/qqqct2Pc7xCNw+GgpqYGXdcB0HWd2tpaHA7HWcfu27ePf/u3f2Pz5s1MmjRpwMWOFKZpYDTXfjL+fkLG34UQoaffgLfb7eTk5FBYWAhAYWEhOTk5Zw3P7N+/n29+85s88cQTTJ8+fWiqDRFmeyPo7l4nWOOjrWTao4NcmRBCfMKvaZKbNm1i27Zt5Obmsm3bNh566CEA1q1bR1FREQAPPfQQXV1dbNiwgfz8fPLz8yktLR26yoPIODNFMjED0zQpOdHIlPFJKIrMfxdChA6/xuCzs7N59tlnz7r/6aef7rn93HPPBa6qENczRTI+nZrGTpra3DI8I4QIOXIl6yAYTdVgsaHEJH4y/j4+MchVCSFEbxLwg2C01KAmpKMoKiUVjSTE2shIlvF3IURokYAfBN8iY6fH3yuayJHxdyFECJKAHyDT8GK21KHGp+N0ddDSLuPvQojQJAE/QGZLPZgGamIGpSd9SzFMkfF3IUQIkoAfIKPlk0XGjlU2Ex9tJS0xKshVCSHE2STgB8ho+mSZ4LKqFiZlJsj4uxAiJEnAD5DRXA0RMbSbEVQ3dJA9ZvgWFxNCiIGQgB+gM9v0lTtbAJjkkIAXQoQmCfgBOrPRdllVCwqQJQEvhAhREvADYHq7MdsbfCdYq5oZkxpDVETAV1wWQoiAkIAfAKO5FgAlPo3y0ydYhRAiVEnAD8CZRcYalUTau7xMypThGSFE6JKAH4AzAV/W6ttIPFsCXggRwiTgB8BorkaJTuRoTTdRERqOFNlgWwgRuiTgB8A3gyadY1XNTHTEo8oFTkKIECYBPwBmcw1mXDqnattl/F0IEfJkjp+fzK42zK5WmpVEDNOUGTRCiJAnPXg/GS2+NWhOdfvG3aUHL4QIdX4FfHl5OWvWrCE3N5c1a9Zw/Pjxs47ZtWsXN9xwAzNmzKCgoCDQdQbdmY22jzRFkJYYRXy0LcgVCSHE+fkV8Bs3bmTt2rX89a9/Ze3atWzYsOGsY8aNG8cjjzzC7bffHvAiQ4HRXA2Kwv46VXrvQogRod+Ad7lcFBcXk5eXB0BeXh7FxcU0NDT0Om7ChAlMmzYNiyU8h/WNpmrMaDv1rXKBkxBiZOg3jZ1OJ+np6WiaBoCmaaSlpeF0OklOTg5oMXZ77KAfm5oaF8BKznaqvY7u6FQA5k13DPnr+SMUahhu0ubRQdocGCHV3Xa52jAMc8CPS02No66udQgq8jFNE7eriuqYWVg0lTibOqSv54+hbnMokjaPDtJm/6mqct6Ocb9DNA6Hg5qaGnRdB0DXdWpra3E4HAMuZqQyO5vB282JjigmZMRi0WTykRAi9PWbVHa7nZycHAoLCwEoLCwkJycn4MMzocxo8q1BU9oUQbbMfxdCjBB+dUU3bdrEtm3byM3NZdu2bTz00EMArFu3jqKiIgA++OADli5dypYtW/jd737H0qVL2blz59BVPozOLDLm9MTJCVYhxIjh1xh8dnY2zz777Fn3P/300z2358+fz1tvvRW4ykKI0VyNoWg0GtES8EKIEUMGk/2g1xyl0ZJKfEwk9vjIYJcjhBB+kYDvh9ndjlFbRonbwaTMeBRZQVIIMUJIwPfDW1kMpsEHrWlkj5ETrEKIkUMCvh/6qQMYWgTHvalMcsj4uxBi5JCAPw/TNPGeOkB9VBamopLlGH1X1wkhRi4J+PMwmp2YbS4+bEsjOzOBSFtIXfgrhBDnJQF/HvrJAwDsaUrhitmZQa5GCCEGRgL+PLynDtCiJdFlTeLSqWnBLkcIIQZEAv4cTK8bb9UhPu5I5/KZGdisWrBLEkKIAZGAPwe9+giK7qHYncmVs8cEuxwhhBgwCfhz8J4qwmuqKOlTyEyJCXY5QggxYDIt5Bzayz7iuDeNxXOzgl2KEEIMivTg+2C0N2Jrq6aMccydnBrscoQQYlAk4PvQcnQfAHHZs7Fa5EskhBiZZIimD/WH9hJpRDF3wexglyKEEIMm3dN/4vV6iWs+SnVEFmnJcnJVCDFyScD/kyMf7yda6Sb+Ium9CyFGNgn4f1Jz8H0MEybOvSzYpQghxAWRgP8H9c2dJLUepTUqE2uMrP0uhBjZ/Ar48vJy1qxZQ25uLmvWrOH48eNnHaPrOg899BArVqxg5cqVfe7hGsp0w+D1PUeYYKkndtIlwS5HCCEumF+zaDZu3MjatWvJz89n+/btbNiwga1bt/Y65s9//jMVFRW88sorNDU1sXr1ahYuXMjYsWOHpPBA6Oz2crC8gX1H6tl/rJ5s/RhanElUtoy/CyFGvn4D3uVyUVxczJYtWwDIy8vj4YcfpqGhgeTk5J7jduzYwec+9zlUVSU5OZkVK1bw8ssv85WvfGXoqgecdc385bkXQfdis6pEWDVsFg2bTcVmUcEEwwTDMDEwMQ3w6gaVde2cqmvDMEwibBrXZsZxiVoLrVFo6dlDWrMQQgyHfgPe6XSSnp6OpvlWU9Q0jbS0NJxOZ6+AdzqdZGZ+sma6w+Ggurp6QMXY7bEDOh6g9cj7LHb9acCPmwrwj7Mgm3zvYqYtIi09acDPFwypqaNvhylp8+ggbQ6MkLrQyeXy9agHIu7iS0mc+iSuuhbAt82ex6vT6dZxe3QURUFVFTRFQel5D1E2C4qinPV8SnwqdXWtAWnPUEpNjRsRdQaStHl0kDb7T1WV83aM+w14h8NBTU0Nuq6jaRq6rlNbW4vD4TjruKqqKmbNmgWc3aMfStbkTDT9k79+FiBqWF5ZCCFCV7+zaOx2Ozk5ORQWFgJQWFhITk5Or+EZgKuvvppnn30WwzBoaGjg1VdfJTc3d2iqFkII0S+/pklu2rSJbdu2kZuby7Zt23jooYcAWLduHUVFRQDk5+czduxYPv3pT3PTTTdx1113MW7cuKGrXAghxHkppmkObNB7CA1mDB5kzG60kDaPDtJm//U3Bi9XsgohRJiSgBdCiDAlAS+EEGEqpObBq+rZ89KH47EjlbR5dJA2jw6DaXN/jwmpk6xCCCECR4ZohBAiTEnACyFEmJKAF0KIMCUBL4QQYUoCXgghwpQEvBBChCkJeCGECFMS8EIIEaYk4IUQIkyN+IAvLy9nzZo15ObmsmbNGo4fPx7skgKqoKCAZcuWMWXKFA4fPtxzfzi3u7GxkXXr1pGbm8t1113H3XffTUNDAxDe7b7zzju5/vrrWb16NWvXruXQoUNAeLcZ4Cc/+Umvn+9wbu+yZcu4+uqryc/PJz8/n507dwJD2GZzhLv11lvNF154wTRN03zhhRfMW2+9NcgVBdb7779vVlVVmVdddZVZWlrac384t7uxsdF89913ez7+wQ9+YP7Hf/yHaZrh3e6Wlpae23/729/M1atXm6YZ3m0+cOCAefvtt5tXXnllz893OLf3n3+PzxiqNo/ogK+vrzfnzZtner1e0zRN0+v1mvPmzTNdLleQKwu8f/zBGE3tNk3TfPnll80vfelLo6rdzz//vPmZz3wmrNvc3d1t3nTTTWZFRUXPz3c4t9c0+w74oWxzSK0mOVBOp5P09HQ0TQNA0zTS0tJwOp1n7RkbTkZTuw3D4Le//S3Lli0bFe1+8MEH2b17N6Zp8j//8z9h3eYf//jHXH/99b229gzn9p5x//33Y5om8+bN41vf+taQtnnEj8GL8Pbwww8THR3NLbfcEuxShsV3v/td3njjDb75zW/y6KOPBrucIbNv3z6KiopYu3ZtsEsZVr/5zW948cUXee655zBNk//6r/8a0tcb0QHvcDioqalB13UAdF2ntrYWh8MR5MqG1mhpd0FBASdOnODxxx9HVdVR026A1atXs2fPHjIyMsKyze+//z5lZWUsX76cZcuWUV1dze23305FRUVYtveMM+2w2WysXbuWDz/8cEh/rkd0wNvtdnJycigsLASgsLCQnJycsPlX7lxGQ7t/9KMfceDAATZv3ozNZgPCu93t7e04nc6ej19//XUSEhLCts133HEHu3bt4vXXX+f1118nIyODX/7yl6xatSos2wvQ0dFBa6tvY23TNNmxYwc5OTlD+j0e8Rt+HDt2jPXr19PS0kJ8fDwFBQVMmjQp2GUFzCOPPMIrr7xCfX09SUlJJCYm8tJLL4V1u48cOUJeXh5ZWVlERkYCMHbsWDZv3hy27a6vr+fOO++ks7MTVVVJSEjggQceYPr06WHb5n+0bNkyfvaznzF58uSwbe/Jkye555570HUdwzDIzs7mP//zP0lLSxuyNo/4gBdCCNG3ET1EI4QQ4twk4IUQIkxJwAshRJiSgBdCiDAlAS+EEGFKAl4IIcKUBLwQQoQpCXghhAhT/x/bfnY93M6LGgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.linspace(0, N_EPOCHS,N_EPOCHS)\n",
    "\n",
    "plt.plot(x,t_f1)\n",
    "plt.plot(x,v_f1)\n",
    "plt.title(\"F1 score\")\n",
    "plt.legend([\"Train F1\", \"Valid F1\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Loss: 0.223 |  Test F1 score: 67.21%\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix, precision_recall_fscore_support\n",
    "model.load_state_dict(torch.load('tut2-model_conll.pt'))\n",
    "\n",
    "test_loss, test_f1, preds, labels = evaluate(model, test_iterator, criterion, TAG_PAD_IDX, full_report=False)\n",
    "print(f'Test Loss: {test_loss:.3f} |  Test F1 score: {test_f1*100:.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "predict =  [item for sublist in preds for item in sublist]\n",
    "true =  [item for sublist in labels for item in sublist]\n",
    "confusion = confusion_matrix(true, predict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Matrice de confusion"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>O</th>\n",
       "      <th>LOC</th>\n",
       "      <th>PER</th>\n",
       "      <th>ORG</th>\n",
       "      <th>MISC</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>O</th>\n",
       "      <td>38061</td>\n",
       "      <td>20</td>\n",
       "      <td>270</td>\n",
       "      <td>143</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LOC</th>\n",
       "      <td>67</td>\n",
       "      <td>1307</td>\n",
       "      <td>259</td>\n",
       "      <td>244</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PER</th>\n",
       "      <td>113</td>\n",
       "      <td>39</td>\n",
       "      <td>2510</td>\n",
       "      <td>100</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ORG</th>\n",
       "      <td>421</td>\n",
       "      <td>210</td>\n",
       "      <td>358</td>\n",
       "      <td>1438</td>\n",
       "      <td>67</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MISC</th>\n",
       "      <td>233</td>\n",
       "      <td>68</td>\n",
       "      <td>103</td>\n",
       "      <td>160</td>\n",
       "      <td>351</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          O   LOC   PER   ORG  MISC\n",
       "O     38061    20   270   143    26\n",
       "LOC      67  1307   259   244    47\n",
       "PER     113    39  2510   100    10\n",
       "ORG     421   210   358  1438    67\n",
       "MISC    233    68   103   160   351"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "confusion_df =pd.DataFrame(confusion)\n",
    "\n",
    "confusion_df.columns=[i for i in TAG.vocab.itos]\n",
    "s = pd.Series([i for i in TAG.vocab.itos])\n",
    "confusion_df = confusion_df.set_index([s])\n",
    "\n",
    "confusion_df['LOC'] = confusion_df['B-LOC'] + confusion_df['I-LOC']\n",
    "confusion_df['PER'] = confusion_df['B-PER'] + confusion_df['I-PER']\n",
    "confusion_df['ORG'] = confusion_df['B-ORG'] + confusion_df['I-ORG']\n",
    "confusion_df['MISC'] = confusion_df['B-MISC'] + confusion_df['I-MISC']\n",
    "\n",
    "\n",
    "confusion_df = confusion_df.drop(columns=[ i for i in TAG.vocab.itos if i != 'O'])\n",
    "\n",
    "confusion_df.loc['LOC'] = confusion_df.loc['B-LOC'] + confusion_df.loc['I-LOC']\n",
    "confusion_df.loc['PER'] = confusion_df.loc['B-PER'] + confusion_df.loc['I-PER']\n",
    "confusion_df.loc['ORG'] = confusion_df.loc['B-ORG'] + confusion_df.loc['I-ORG']\n",
    "confusion_df.loc['MISC'] = confusion_df.loc['B-MISC'] + confusion_df.loc['I-MISC']\n",
    "\n",
    "confusion_df = confusion_df.drop([i for i in TAG.vocab.itos if i != 'O' ])\n",
    "\n",
    "confusion_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Precision-Recall-F1Score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>F1-score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>O</th>\n",
       "      <td>0.978558</td>\n",
       "      <td>0.988084</td>\n",
       "      <td>0.983298</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LOC</th>\n",
       "      <td>0.795012</td>\n",
       "      <td>0.679314</td>\n",
       "      <td>0.732623</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PER</th>\n",
       "      <td>0.717143</td>\n",
       "      <td>0.905483</td>\n",
       "      <td>0.800383</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ORG</th>\n",
       "      <td>0.689688</td>\n",
       "      <td>0.576584</td>\n",
       "      <td>0.628085</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MISC</th>\n",
       "      <td>0.700599</td>\n",
       "      <td>0.383607</td>\n",
       "      <td>0.495763</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Precision    Recall  F1-score\n",
       "O      0.978558  0.988084  0.983298\n",
       "LOC    0.795012  0.679314  0.732623\n",
       "PER    0.717143  0.905483  0.800383\n",
       "ORG    0.689688  0.576584  0.628085\n",
       "MISC   0.700599  0.383607  0.495763"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm = confusion_df.to_numpy()\n",
    "\n",
    "TP = np.diag(cm)\n",
    "FP = np.sum(cm, axis=0) - TP\n",
    "FN = np.sum(cm, axis=1) - TP\n",
    "\n",
    "num_classes = 4\n",
    "TN = []\n",
    "for i in range(num_classes):\n",
    "    temp = np.delete(cm, i, 0)    \n",
    "    temp = np.delete(temp, i, 1) \n",
    "    TN.append(sum(sum(temp)))\n",
    "    \n",
    "precision = TP/(TP+FP)\n",
    "recall = TP/(TP+FN)\n",
    "f1 = (2*precision*recall)/(precision+recall)\n",
    "\n",
    "prf_df =pd.DataFrame()\n",
    "prf_df['Precision'] = precision\n",
    "prf_df['Recall'] = recall\n",
    "prf_df['F1-score'] = f1\n",
    "\n",
    "s = pd.Series([i for i in confusion_df.index])\n",
    "prf_df = prf_df.set_index([s])\n",
    "\n",
    "prf_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tag_sentence(model, device, sentence, text_field, tag_field, char_field):\n",
    "    \n",
    "    model.eval()\n",
    "    \n",
    "    if isinstance(sentence, str):\n",
    "        nlp = spacy.load('en')\n",
    "        tokens = [token.text for token in nlp(sentence)]\n",
    "    else:\n",
    "        tokens = [token for token in sentence]\n",
    "\n",
    "    if text_field.lower:\n",
    "        tokens = [t.lower() for t in tokens]\n",
    "        \n",
    "    max_word_len = max([len(token) for token in tokens])\n",
    "    numericalized_chars = []\n",
    "    char_pad_id = char_field.vocab.stoi[CHAR.pad_token] \n",
    "    for token in tokens:\n",
    "        numericalized_chars.append(\n",
    "                [char_field.vocab.stoi[char] for char in token]\n",
    "                + [char_pad_id for _ in range(max_word_len - len(token))]\n",
    "                )\n",
    "    numericalized_tokens = [text_field.vocab.stoi[t] for t in tokens]\n",
    "\n",
    "    unk_idx = text_field.vocab.stoi[text_field.unk_token]\n",
    "    \n",
    "    unks = [t for t, n in zip(tokens, numericalized_tokens) if n == unk_idx]\n",
    "    \n",
    "    token_tensor = torch.LongTensor(numericalized_tokens)\n",
    "    \n",
    "    token_tensor = token_tensor.unsqueeze(-1).to(device)\n",
    "    char_tensor = torch.as_tensor(numericalized_chars)\n",
    "    char_tensor = char_tensor.unsqueeze(0).to(device) \n",
    "    \n",
    "    predictions = model(token_tensor, char_tensor)\n",
    "    \n",
    "    top_predictions = predictions.argmax(-1)\n",
    "    \n",
    "    predicted_tags = [tag_field.vocab.itos[t.item()] for t in top_predictions]\n",
    "    \n",
    "    return tokens, predicted_tags, unks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['two', 'goals', 'in', 'the', 'last', 'six', 'minutes', 'gave', 'holders', 'japan', 'an', 'uninspiring', '2-1', 'asian', 'cup', 'victory', 'over', 'syria', 'on', 'friday', '.']\n",
      "['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-LOC', 'O', 'O', 'O', 'B-MISC', 'I-MISC', 'O', 'O', 'B-LOC', 'O', 'O', 'O']\n"
     ]
    }
   ],
   "source": [
    "example_index = 39\n",
    "\n",
    "sentence = vars(test_data.examples[example_index])['text']\n",
    "actual_tags = vars(test_data.examples[example_index])['tag']\n",
    "\n",
    "print(sentence)\n",
    "print(actual_tags)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-LOC', 'O', 'O', 'O', 'B-ORG', 'I-ORG', 'O', 'O', 'B-LOC', 'O', 'O', 'O']\n",
      "['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-LOC', 'O', 'O', 'O', 'B-MISC', 'I-MISC', 'O', 'O', 'B-LOC', 'O', 'O', 'O']\n"
     ]
    }
   ],
   "source": [
    "tokens, pred_tags, unks = tag_sentence(model, \n",
    "                                       device, \n",
    "                                       sentence, \n",
    "                                       TEXT, \n",
    "                                       TAG,\n",
    "                               \n",
    "                                       CHAR)\n",
    "\n",
    "print(pred_tags)\n",
    "print(actual_tags)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pred. Tag\tActual Tag\tCorrect?\tToken\n",
      "\n",
      "O\t\tO\t\t✔\t\ttwo\n",
      "O\t\tO\t\t✔\t\tgoals\n",
      "O\t\tO\t\t✔\t\tin\n",
      "O\t\tO\t\t✔\t\tthe\n",
      "O\t\tO\t\t✔\t\tlast\n",
      "O\t\tO\t\t✔\t\tsix\n",
      "O\t\tO\t\t✔\t\tminutes\n",
      "O\t\tO\t\t✔\t\tgave\n",
      "O\t\tO\t\t✔\t\tholders\n",
      "B-LOC\t\tB-LOC\t\t✔\t\tjapan\n",
      "O\t\tO\t\t✔\t\tan\n",
      "O\t\tO\t\t✔\t\tuninspiring\n",
      "O\t\tO\t\t✔\t\t2-1\n",
      "B-ORG\t\tB-MISC\t\t✘\t\tasian\n",
      "I-ORG\t\tI-MISC\t\t✘\t\tcup\n",
      "O\t\tO\t\t✔\t\tvictory\n",
      "O\t\tO\t\t✔\t\tover\n",
      "B-LOC\t\tB-LOC\t\t✔\t\tsyria\n",
      "O\t\tO\t\t✔\t\ton\n",
      "O\t\tO\t\t✔\t\tfriday\n",
      "O\t\tO\t\t✔\t\t.\n"
     ]
    }
   ],
   "source": [
    "print(\"Pred. Tag\\tActual Tag\\tCorrect?\\tToken\\n\")\n",
    "\n",
    "for token, pred_tag, actual_tag in zip(tokens, pred_tags, actual_tags):\n",
    "    correct = '✔' if pred_tag == actual_tag else '✘'\n",
    "    print(f\"{pred_tag}\\t\\t{actual_tag}\\t\\t{correct}\\t\\t{token}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[]\n"
     ]
    }
   ],
   "source": [
    "sentence = 'The will deliver a speech about the conflict in North Korea tomorrow in New York with my friend Mary Kate.'\n",
    "\n",
    "tokens, tags, unks = tag_sentence(model, \n",
    "                                  device, \n",
    "                                  sentence, \n",
    "                                  TEXT, \n",
    "                                  TAG,\n",
    "                                  CHAR)\n",
    "\n",
    "print(unks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pred. Tag\tToken\n",
      "\n",
      "O\t\tthe\n",
      "O\t\twill\n",
      "O\t\tdeliver\n",
      "O\t\ta\n",
      "O\t\tspeech\n",
      "O\t\tabout\n",
      "O\t\tthe\n",
      "O\t\tconflict\n",
      "O\t\tin\n",
      "O\t\tnorth\n",
      "B-LOC\t\tkorea\n",
      "O\t\ttomorrow\n",
      "O\t\tin\n",
      "B-LOC\t\tnew\n",
      "I-LOC\t\tyork\n",
      "O\t\twith\n",
      "O\t\tmy\n",
      "O\t\tfriend\n",
      "O\t\tmary\n",
      "I-PER\t\tkate\n",
      "O\t\t.\n"
     ]
    }
   ],
   "source": [
    "print(\"Pred. Tag\\tToken\\n\")\n",
    "\n",
    "\n",
    "for token, tag in zip(tokens, tags):\n",
    "    print(f\"{tag}\\t\\t{token}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On remarque que ce modèle ne donne pas de bons résultats."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Référence : \n",
    "\n",
    " - [Step-by-step NER Model for Bahasa Indonesia with PyTorch and Torchtext](https://medium.com/@yoseflaw/step-by-step-ner-model-for-bahasa-indonesia-with-pytorch-and-torchtext-6f94fca08406)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "3 - Character-level embedding and CNN for Extra Features",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}