Models/qa-with-squad-2.0-dataset-and-bert-base.ipynb

{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.7.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"colab":{"name":"qa-with-squad-2-0-dataset-and-bert-base.ipynb","provenance":[],"collapsed_sections":[]}},"nbformat_minor":0,"nbformat":4,"cells":[{"cell_type":"markdown","source":["# Question Answering with SQuAD 2.0 dataset and BERT-Base\n","\n","In this assignment, our task was to  develop a model for question answering on\n","the dataset [SQuAD 2.0](https://rajpurkar.github.io/SQuAD-explorer/).\n","\n","**Note:** due to limitations on the free version of google colab, the experiments on this assignment were executed on [Kaggle](https://www.kaggle.com/)."],"metadata":{"id":"is-BQ_NxNsO0"}},{"cell_type":"markdown","source":["## First Steps"],"metadata":{"id":"jTfBhfsHN7-a"}},{"cell_type":"markdown","source":["**GPU Info**\n","\n","To get the same results (best ones) you should have a Tesla P100 GPU. This GPU is available for free usage on [Kaggle](https://www.kaggle.com/)."],"metadata":{"id":"8GvtY-GhN-vi"}},{"cell_type":"code","source":["!nvidia-smi"],"metadata":{"id":"p2mc9E6eNa0R","outputId":"562617ae-c729-438b-97fd-e989fc77de0b","executionInfo":{"status":"ok","timestamp":1646593693702,"user_tz":-120,"elapsed":245,"user":{"displayName":"Βασίλης Κυριακόπουλος","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjLJ843-mdA-XYMgrr2K7G5eg-3pPcT6fAhtDjZiw=s64","userId":"10189451843812858163"}},"execution":{"iopub.status.busy":"2022-03-06T19:38:44.314227Z","iopub.execute_input":"2022-03-06T19:38:44.314935Z","iopub.status.idle":"2022-03-06T19:38:45.229456Z","shell.execute_reply.started":"2022-03-06T19:38:44.314823Z","shell.execute_reply":"2022-03-06T19:38:45.228329Z"},"trusted":true},"execution_count":null,"outputs":[{"name":"stdout","text":"Sun Mar  6 19:38:45 2022       \n+-----------------------------------------------------------------------------+\n| NVIDIA-SMI 450.119.04   Driver Version: 450.119.04   CUDA Version: 11.0     |\n|-------------------------------+----------------------+----------------------+\n| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n|                               |                      |               MIG M. |\n|===============================+======================+======================|\n|   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |\n| N/A   33C    P0    26W / 250W |      0MiB / 16280MiB |      0%      Default |\n|                               |                      |                  N/A |\n+-------------------------------+----------------------+----------------------+\n                                                                               \n+-----------------------------------------------------------------------------+\n| Processes:                                                                  |\n|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n|        ID   ID                                                   Usage      |\n|=============================================================================|\n|  No running processes found                                                 |\n+-----------------------------------------------------------------------------+\n","output_type":"stream"}]},{"cell_type":"markdown","source":["**Install pytorch interface for pre-trained BERT**\n","\n","Then, we will install the interface to use the pre-trained BERT model."],"metadata":{"id":"_pSfqxmEPx88"}},{"cell_type":"code","source":["!pip install transformers"],"metadata":{"id":"0rZyAPuhPySz","outputId":"3fe2671b-41ae-4fd3-f652-03fb21a9c2f4","executionInfo":{"status":"ok","timestamp":1646593712064,"user_tz":-120,"elapsed":16066,"user":{"displayName":"Βασίλης Κυριακόπουλος","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjLJ843-mdA-XYMgrr2K7G5eg-3pPcT6fAhtDjZiw=s64","userId":"10189451843812858163"}},"execution":{"iopub.status.busy":"2022-03-06T19:38:48.906434Z","iopub.execute_input":"2022-03-06T19:38:48.907138Z","iopub.status.idle":"2022-03-06T19:38:57.503398Z","shell.execute_reply.started":"2022-03-06T19:38:48.907100Z","shell.execute_reply":"2022-03-06T19:38:57.502574Z"},"trusted":true},"execution_count":null,"outputs":[{"name":"stdout","text":"Requirement already satisfied: transformers in /opt/conda/lib/python3.7/site-packages (4.15.0)\nRequirement already satisfied: importlib-metadata in /opt/conda/lib/python3.7/site-packages (from transformers) (4.11.2)\nRequirement already satisfied: tokenizers<0.11,>=0.10.1 in /opt/conda/lib/python3.7/site-packages (from transformers) (0.10.3)\nRequirement already satisfied: regex!=2019.12.17 in /opt/conda/lib/python3.7/site-packages (from transformers) (2021.11.10)\nRequirement already satisfied: sacremoses in /opt/conda/lib/python3.7/site-packages (from transformers) (0.0.47)\nRequirement already satisfied: pyyaml>=5.1 in /opt/conda/lib/python3.7/site-packages (from transformers) (6.0)\nRequirement already satisfied: filelock in /opt/conda/lib/python3.7/site-packages (from transformers) (3.4.2)\nRequirement already satisfied: huggingface-hub<1.0,>=0.1.0 in /opt/conda/lib/python3.7/site-packages (from transformers) (0.2.1)\nRequirement already satisfied: numpy>=1.17 in /opt/conda/lib/python3.7/site-packages (from transformers) (1.20.3)\nRequirement already satisfied: requests in /opt/conda/lib/python3.7/site-packages (from transformers) (2.26.0)\nRequirement already satisfied: tqdm>=4.27 in /opt/conda/lib/python3.7/site-packages (from transformers) (4.62.3)\nRequirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.7/site-packages (from transformers) (21.3)\nRequirement already satisfied: typing-extensions>=3.7.4.3 in /opt/conda/lib/python3.7/site-packages (from huggingface-hub<1.0,>=0.1.0->transformers) (4.1.1)\nRequirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging>=20.0->transformers) (3.0.6)\nRequirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata->transformers) (3.6.0)\nRequirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests->transformers) (2021.10.8)\nRequirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests->transformers) (1.26.7)\nRequirement already satisfied: charset-normalizer~=2.0.0 in /opt/conda/lib/python3.7/site-packages (from requests->transformers) (2.0.9)\nRequirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests->transformers) (3.1)\nRequirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from sacremoses->transformers) (1.16.0)\nRequirement already satisfied: joblib in /opt/conda/lib/python3.7/site-packages (from sacremoses->transformers) (1.1.0)\nRequirement already satisfied: click in /opt/conda/lib/python3.7/site-packages (from sacremoses->transformers) (8.0.3)\n\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\n","output_type":"stream"}]},{"cell_type":"markdown","source":["**Import necessary libraries**\n","\n","Now, it's time to import the necessary libraries for this notebook."],"metadata":{"id":"VRmYivZrQPrR"}},{"cell_type":"code","source":["import torch\n","import json\n","import pandas as pd\n","import numpy as np\n","import random\n","import matplotlib.pyplot as plt\n","import re\n","import string\n","import collections\n","from transformers import BertTokenizerFast, BertForQuestionAnswering\n","from transformers.tokenization_utils_base import BatchEncoding\n","from torch.utils.data import Dataset, DataLoader\n","from tqdm import tqdm\n","\n","pd.set_option('max_colwidth', 500)\n","%matplotlib inline"],"metadata":{"id":"V9xaR0zaQP_g","execution":{"iopub.status.busy":"2022-03-06T19:39:01.715909Z","iopub.execute_input":"2022-03-06T19:39:01.716239Z","iopub.status.idle":"2022-03-06T19:39:03.408926Z","shell.execute_reply.started":"2022-03-06T19:39:01.716205Z","shell.execute_reply":"2022-03-06T19:39:03.408257Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["**Enable CUDA**\n","\n","Enable CUDA for GPU utilization by our model. This makes calculations and thus the training of our models faster."],"metadata":{"id":"xUz3L_c8QVXR"}},{"cell_type":"code","source":["device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')\n","device"],"metadata":{"id":"doKXFdk8QVoD","outputId":"8f33ad68-3224-4e14-bf58-989696447dc4","executionInfo":{"status":"ok","timestamp":1646593723427,"user_tz":-120,"elapsed":249,"user":{"displayName":"Βασίλης Κυριακόπουλος","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjLJ843-mdA-XYMgrr2K7G5eg-3pPcT6fAhtDjZiw=s64","userId":"10189451843812858163"}},"execution":{"iopub.status.busy":"2022-03-06T19:39:05.929989Z","iopub.execute_input":"2022-03-06T19:39:05.930533Z","iopub.status.idle":"2022-03-06T19:39:05.979669Z","shell.execute_reply.started":"2022-03-06T19:39:05.930495Z","shell.execute_reply":"2022-03-06T19:39:05.978859Z"},"trusted":true},"execution_count":null,"outputs":[{"execution_count":4,"output_type":"execute_result","data":{"text/plain":"device(type='cuda', index=0)"},"metadata":{}}]},{"cell_type":"markdown","source":["**Set random seed**\n","\n","Set a constant random seed in order to get the same (deterministic) outputs every time we run our models.\n","\n"],"metadata":{"id":"HKOXPf5XQX3Z"}},{"cell_type":"code","source":["seed = 17064\n","def reset_seed():\n","  random.seed(seed)\n","  np.random.seed(seed)\n","  torch.manual_seed(seed)\n","  torch.cuda.manual_seed_all(seed)\n","reset_seed()"],"metadata":{"id":"QnruhHSWQcGO","execution":{"iopub.status.busy":"2022-03-06T19:39:08.266623Z","iopub.execute_input":"2022-03-06T19:39:08.267490Z","iopub.status.idle":"2022-03-06T19:39:08.275213Z","shell.execute_reply.started":"2022-03-06T19:39:08.267430Z","shell.execute_reply":"2022-03-06T19:39:08.274531Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["**BERT model definition**\n","\n","And here, we define the name of the pre-trained bert model from [Hugging Face](https://huggingface.co/models) which we'll use."],"metadata":{"id":"Y-Hb3sovQwHa"}},{"cell_type":"code","source":["BERT_MODEL_NAME = 'bert-base-uncased'"],"metadata":{"id":"wqtitofaQdb3","execution":{"iopub.status.busy":"2022-03-06T19:39:13.265916Z","iopub.execute_input":"2022-03-06T19:39:13.266167Z","iopub.status.idle":"2022-03-06T19:39:13.270974Z","shell.execute_reply.started":"2022-03-06T19:39:13.266137Z","shell.execute_reply":"2022-03-06T19:39:13.270158Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["## Dataset loading\n","\n","Now, let's load our datasets:"],"metadata":{"id":"mtvSPZfoQ5f4"}},{"cell_type":"markdown","source":["**Define json preprocessing function**\n","\n","First, we will define a function that takes a squad json dataset file path as argument and returns a dataframe with the questions, their answers in the form (start_position, end_position, answer_text) and the corpus that contains the answer to the corresponding question."],"metadata":{"id":"Jnby8mS-gXoz"}},{"cell_type":"code","source":["def squad_load_from_json(json_file_path: str):\n","  with open(json_file_path, \"r\") as f:\n","    json_data = json.load(f)['data']\n","    questions = []\n","    answers = []\n","    corpuses = []\n","    for category in json_data:\n","      for paragraph in category['paragraphs']:\n","        context = paragraph['context']\n","        for qa in paragraph['qas']:\n","          corpuses.append(context)\n","          question = qa['question']\n","          questions.append(question)\n","          if qa['is_impossible']:\n","            ans_list = qa['plausible_answers']\n","          else:\n","            ans_list = qa['answers']\n","          ans_set = set()\n","          if len(ans_list) == 0:\n","            print(\"Question \", question, \" has no answers\")\n","          for idx, ans in enumerate(ans_list):\n","            ans_set.add((ans['answer_start'], ans['answer_start']+len(ans['text']), ans['text']))\n","          answers.append(list(ans_set))\n","    return pd.DataFrame(data={'question':questions, 'answer':answers, 'corpus':corpuses})"],"metadata":{"id":"mWhOlZ69gcKr","execution":{"iopub.status.busy":"2022-03-06T19:39:16.074197Z","iopub.execute_input":"2022-03-06T19:39:16.074761Z","iopub.status.idle":"2022-03-06T19:39:16.083958Z","shell.execute_reply.started":"2022-03-06T19:39:16.074723Z","shell.execute_reply":"2022-03-06T19:39:16.082686Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["**Download the datasets**\n","\n","Then we will download the train and validation datasets from the squad website and then convert them to pandas dataframes with the script above."],"metadata":{"id":"8hXgUQBFZgzE"}},{"cell_type":"code","source":["!wget https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v2.0.json\n","!wget https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v2.0.json"],"metadata":{"id":"AZcygV99Q52F","outputId":"4b247b4b-2df2-4ca3-d7ff-f532998e57a8","executionInfo":{"status":"ok","timestamp":1646593728185,"user_tz":-120,"elapsed":838,"user":{"displayName":"Βασίλης Κυριακόπουλος","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjLJ843-mdA-XYMgrr2K7G5eg-3pPcT6fAhtDjZiw=s64","userId":"10189451843812858163"}},"execution":{"iopub.status.busy":"2022-03-06T19:39:18.297163Z","iopub.execute_input":"2022-03-06T19:39:18.297867Z","iopub.status.idle":"2022-03-06T19:39:20.424525Z","shell.execute_reply.started":"2022-03-06T19:39:18.297831Z","shell.execute_reply":"2022-03-06T19:39:20.423653Z"},"trusted":true},"execution_count":null,"outputs":[{"name":"stdout","text":"--2022-03-06 19:39:18--  https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v2.0.json\nResolving rajpurkar.github.io (rajpurkar.github.io)... 185.199.110.153, 185.199.108.153, 185.199.109.153, ...\nConnecting to rajpurkar.github.io (rajpurkar.github.io)|185.199.110.153|:443... connected.\nHTTP request sent, awaiting response... 200 OK\nLength: 42123633 (40M) [application/json]\nSaving to: ‘train-v2.0.json’\n\ntrain-v2.0.json     100%[===================>]  40.17M   197MB/s    in 0.2s    \n\n2022-03-06 19:39:19 (197 MB/s) - ‘train-v2.0.json’ saved [42123633/42123633]\n\n--2022-03-06 19:39:20--  https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v2.0.json\nResolving rajpurkar.github.io (rajpurkar.github.io)... 185.199.111.153, 185.199.109.153, 185.199.108.153, ...\nConnecting to rajpurkar.github.io (rajpurkar.github.io)|185.199.111.153|:443... connected.\nHTTP request sent, awaiting response... 200 OK\nLength: 4370528 (4.2M) [application/json]\nSaving to: ‘dev-v2.0.json’\n\ndev-v2.0.json       100%[===================>]   4.17M  --.-KB/s    in 0.07s   \n\n2022-03-06 19:39:20 (56.8 MB/s) - ‘dev-v2.0.json’ saved [4370528/4370528]\n\n","output_type":"stream"}]},{"cell_type":"markdown","source":["**Train Dataset**"],"metadata":{"id":"ObZ_q6UXQ8v3"}},{"cell_type":"code","source":["train_dataset = squad_load_from_json(\"train-v2.0.json\")\n","train_dataset"],"metadata":{"id":"YIUz68o5g78D","outputId":"69da4e90-8a7f-46c9-e99e-262342eaa377","executionInfo":{"status":"ok","timestamp":1646593731851,"user_tz":-120,"elapsed":1484,"user":{"displayName":"Βασίλης Κυριακόπουλος","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjLJ843-mdA-XYMgrr2K7G5eg-3pPcT6fAhtDjZiw=s64","userId":"10189451843812858163"}},"execution":{"iopub.status.busy":"2022-03-06T19:39:22.986155Z","iopub.execute_input":"2022-03-06T19:39:22.986858Z","iopub.status.idle":"2022-03-06T19:39:24.590518Z","shell.execute_reply.started":"2022-03-06T19:39:22.986818Z","shell.execute_reply":"2022-03-06T19:39:24.589776Z"},"trusted":true},"execution_count":null,"outputs":[{"execution_count":9,"output_type":"execute_result","data":{"text/plain":"                                                                 question  \\\n0                                When did Beyonce start becoming popular?   \n1              What areas did Beyonce compete in when she was growing up?   \n2        When did Beyonce leave Destiny's Child and become a solo singer?   \n3                           In what city and state did Beyonce  grow up?    \n4                              In which decade did Beyonce become famous?   \n...                                                                   ...   \n130314              Physics has broadly agreed on the definition of what?   \n130315                               Who coined the term partonic matter?   \n130316                              What is another name for anti-matter?   \n130317  Matter usually does not need to be used in conjunction with what?   \n130318             What field of study has a variety of unusual contexts?   \n\n                                     answer  \\\n0           [(269, 286, in the late 1990s)]   \n1         [(207, 226, singing and dancing)]   \n2                        [(526, 530, 2003)]   \n3              [(166, 180, Houston, Texas)]   \n4                  [(276, 286, late 1990s)]   \n...                                     ...   \n130314                 [(485, 491, matter)]   \n130315                 [(327, 333, Alfvén)]   \n130316      [(350, 367, Gk. common matter)]   \n130317  [(529, 550, a specifying modifier)]   \n130318                  [(37, 44, physics)]   \n\n                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     corpus  \n0       Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debu...  \n1       Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debu...  \n2       Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debu...  \n3       Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debu...  \n4       Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debu...  \n...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ...  \n130314  The term \"matter\" is used throughout physics in a bewildering variety of contexts: for example, one refers to \"condensed matter physics\", \"elementary matter\", \"partonic\" matter, \"dark\" matter, \"anti\"-matter, \"strange\" matter, and \"nuclear\" matter. In discussions of matter and antimatter, normal matter has been referred to by Alfvén as koinomatter (Gk. common matter). It is fair to say that in physics, there is no broad consensus as to a general definition of matter, and the term \"matter\" usu...  \n130315  The term \"matter\" is used throughout physics in a bewildering variety of contexts: for example, one refers to \"condensed matter physics\", \"elementary matter\", \"partonic\" matter, \"dark\" matter, \"anti\"-matter, \"strange\" matter, and \"nuclear\" matter. In discussions of matter and antimatter, normal matter has been referred to by Alfvén as koinomatter (Gk. common matter). It is fair to say that in physics, there is no broad consensus as to a general definition of matter, and the term \"matter\" usu...  \n130316  The term \"matter\" is used throughout physics in a bewildering variety of contexts: for example, one refers to \"condensed matter physics\", \"elementary matter\", \"partonic\" matter, \"dark\" matter, \"anti\"-matter, \"strange\" matter, and \"nuclear\" matter. In discussions of matter and antimatter, normal matter has been referred to by Alfvén as koinomatter (Gk. common matter). It is fair to say that in physics, there is no broad consensus as to a general definition of matter, and the term \"matter\" usu...  \n130317  The term \"matter\" is used throughout physics in a bewildering variety of contexts: for example, one refers to \"condensed matter physics\", \"elementary matter\", \"partonic\" matter, \"dark\" matter, \"anti\"-matter, \"strange\" matter, and \"nuclear\" matter. In discussions of matter and antimatter, normal matter has been referred to by Alfvén as koinomatter (Gk. common matter). It is fair to say that in physics, there is no broad consensus as to a general definition of matter, and the term \"matter\" usu...  \n130318  The term \"matter\" is used throughout physics in a bewildering variety of contexts: for example, one refers to \"condensed matter physics\", \"elementary matter\", \"partonic\" matter, \"dark\" matter, \"anti\"-matter, \"strange\" matter, and \"nuclear\" matter. In discussions of matter and antimatter, normal matter has been referred to by Alfvén as koinomatter (Gk. common matter). It is fair to say that in physics, there is no broad consensus as to a general definition of matter, and the term \"matter\" usu...  \n\n[130319 rows x 3 columns]","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>question</th>\n      <th>answer</th>\n      <th>corpus</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>When did Beyonce start becoming popular?</td>\n      <td>[(269, 286, in the late 1990s)]</td>\n      <td>Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&amp;B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debu...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>What areas did Beyonce compete in when she was growing up?</td>\n      <td>[(207, 226, singing and dancing)]</td>\n      <td>Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&amp;B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debu...</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>When did Beyonce leave Destiny's Child and become a solo singer?</td>\n      <td>[(526, 530, 2003)]</td>\n      <td>Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&amp;B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debu...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>In what city and state did Beyonce  grow up?</td>\n      <td>[(166, 180, Houston, Texas)]</td>\n      <td>Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&amp;B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debu...</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>In which decade did Beyonce become famous?</td>\n      <td>[(276, 286, late 1990s)]</td>\n      <td>Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&amp;B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debu...</td>\n    </tr>\n    <tr>\n      <th>...</th>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>130314</th>\n      <td>Physics has broadly agreed on the definition of what?</td>\n      <td>[(485, 491, matter)]</td>\n      <td>The term \"matter\" is used throughout physics in a bewildering variety of contexts: for example, one refers to \"condensed matter physics\", \"elementary matter\", \"partonic\" matter, \"dark\" matter, \"anti\"-matter, \"strange\" matter, and \"nuclear\" matter. In discussions of matter and antimatter, normal matter has been referred to by Alfvén as koinomatter (Gk. common matter). It is fair to say that in physics, there is no broad consensus as to a general definition of matter, and the term \"matter\" usu...</td>\n    </tr>\n    <tr>\n      <th>130315</th>\n      <td>Who coined the term partonic matter?</td>\n      <td>[(327, 333, Alfvén)]</td>\n      <td>The term \"matter\" is used throughout physics in a bewildering variety of contexts: for example, one refers to \"condensed matter physics\", \"elementary matter\", \"partonic\" matter, \"dark\" matter, \"anti\"-matter, \"strange\" matter, and \"nuclear\" matter. In discussions of matter and antimatter, normal matter has been referred to by Alfvén as koinomatter (Gk. common matter). It is fair to say that in physics, there is no broad consensus as to a general definition of matter, and the term \"matter\" usu...</td>\n    </tr>\n    <tr>\n      <th>130316</th>\n      <td>What is another name for anti-matter?</td>\n      <td>[(350, 367, Gk. common matter)]</td>\n      <td>The term \"matter\" is used throughout physics in a bewildering variety of contexts: for example, one refers to \"condensed matter physics\", \"elementary matter\", \"partonic\" matter, \"dark\" matter, \"anti\"-matter, \"strange\" matter, and \"nuclear\" matter. In discussions of matter and antimatter, normal matter has been referred to by Alfvén as koinomatter (Gk. common matter). It is fair to say that in physics, there is no broad consensus as to a general definition of matter, and the term \"matter\" usu...</td>\n    </tr>\n    <tr>\n      <th>130317</th>\n      <td>Matter usually does not need to be used in conjunction with what?</td>\n      <td>[(529, 550, a specifying modifier)]</td>\n      <td>The term \"matter\" is used throughout physics in a bewildering variety of contexts: for example, one refers to \"condensed matter physics\", \"elementary matter\", \"partonic\" matter, \"dark\" matter, \"anti\"-matter, \"strange\" matter, and \"nuclear\" matter. In discussions of matter and antimatter, normal matter has been referred to by Alfvén as koinomatter (Gk. common matter). It is fair to say that in physics, there is no broad consensus as to a general definition of matter, and the term \"matter\" usu...</td>\n    </tr>\n    <tr>\n      <th>130318</th>\n      <td>What field of study has a variety of unusual contexts?</td>\n      <td>[(37, 44, physics)]</td>\n      <td>The term \"matter\" is used throughout physics in a bewildering variety of contexts: for example, one refers to \"condensed matter physics\", \"elementary matter\", \"partonic\" matter, \"dark\" matter, \"anti\"-matter, \"strange\" matter, and \"nuclear\" matter. In discussions of matter and antimatter, normal matter has been referred to by Alfvén as koinomatter (Gk. common matter). It is fair to say that in physics, there is no broad consensus as to a general definition of matter, and the term \"matter\" usu...</td>\n    </tr>\n  </tbody>\n</table>\n<p>130319 rows × 3 columns</p>\n</div>"},"metadata":{}}]},{"cell_type":"markdown","source":["**Validation Dataset**"],"metadata":{"id":"YnNmSit1RoDl"}},{"cell_type":"code","source":["validation_dataset = squad_load_from_json(\"dev-v2.0.json\")\n","validation_dataset"],"metadata":{"id":"UHM-6PXxRr3t","outputId":"0a38841c-33b9-41ef-fff5-356b67a35be8","executionInfo":{"status":"ok","timestamp":1646593734549,"user_tz":-120,"elapsed":252,"user":{"displayName":"Βασίλης Κυριακόπουλος","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjLJ843-mdA-XYMgrr2K7G5eg-3pPcT6fAhtDjZiw=s64","userId":"10189451843812858163"}},"execution":{"iopub.status.busy":"2022-03-06T19:39:27.769114Z","iopub.execute_input":"2022-03-06T19:39:27.769618Z","iopub.status.idle":"2022-03-06T19:39:28.009356Z","shell.execute_reply.started":"2022-03-06T19:39:27.769579Z","shell.execute_reply":"2022-03-06T19:39:28.008639Z"},"trusted":true},"execution_count":null,"outputs":[{"name":"stdout","text":"Question  What was proven in 2001 in regard to the solid oxygen phase?  has no answers\nQuestion  What was discovered in 2006 in regard to O4?  has no answers\nQuestion  What does air in equilibrium with water contain?  has no answers\nQuestion  What is paired oxygen?  has no answers\nQuestion  Why are O molecules paramagnetic?  has no answers\nQuestion  What is formula for the reactive oxygen ion?  has no answers\nQuestion  What are products of oxygen use in organisms?   has no answers\nQuestion  What began to accumulate 5.2 billion years ago?  has no answers\nQuestion  What is red in both the liquid and solid states?  has no answers\nQuestion  Why do polar oceans support reduced amounts of life?  has no answers\nQuestion  What involves delivering a gas stream that is 9% to 93% O2?  has no answers\nQuestion  What do oxoacids evolve from?  has no answers\nQuestion  What is the essential purpose of supplementation?   has no answers\nQuestion  In case of cabin pressurization, what is available to passengers?  has no answers\nQuestion  What measures at 1.4 kPa?  has no answers\n","output_type":"stream"},{"execution_count":10,"output_type":"execute_result","data":{"text/plain":"                                                                question  \\\n0                                   In what country is Normandy located?   \n1                                     When were the Normans in Normandy?   \n2                          From which countries did the Norse originate?   \n3                                              Who was the Norse leader?   \n4       What century did the Normans first gain their separate identity?   \n...                                                                  ...   \n11868  What is the seldom used force unit equal to one thousand newtons?   \n11869                           What does not have a metric counterpart?   \n11870  What is the force exerted by standard gravity on one ton of mass?   \n11871                  What force leads to a commonly used unit of mass?   \n11872                        What force is part of the modern SI system?   \n\n                                                                                             answer  \\\n0                                                                              [(159, 165, France)]   \n1                   [(94, 117, 10th and 11th centuries), (87, 117, in the 10th and 11th centuries)]   \n2                                                         [(256, 283, Denmark, Iceland and Norway)]   \n3                                                                               [(308, 313, Rollo)]   \n4      [(671, 675, 10th), (671, 683, 10th century), (649, 683, the first half of the 10th century)]   \n...                                                                                             ...   \n11868                                                                          [(665, 671, sthène)]   \n11869                                                                        [(4, 15, pound-force)]   \n11870                                                                    [(82, 96, kilogram-force)]   \n11871                                                                  [(195, 209, kilogram-force)]   \n11872                                                                  [(383, 397, kilogram-force)]   \n\n                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    corpus  \n0      The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants ...  \n1      The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants ...  \n2      The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants ...  \n3      The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants ...  \n4      The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants ...  \n...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    ...  \n11868  The pound-force has a metric counterpart, less commonly used than the newton: the kilogram-force (kgf) (sometimes kilopond), is the force exerted by standard gravity on one kilogram of mass. The kilogram-force leads to an alternate, but rarely used unit of mass: the metric slug (sometimes mug or hyl) is that mass that accelerates at 1 m·s−2 when subjected to a force of 1 kgf. The kilogram-force is not a part of the modern SI system, and is generally deprecated; however it still sees use for ...  \n11869  The pound-force has a metric counterpart, less commonly used than the newton: the kilogram-force (kgf) (sometimes kilopond), is the force exerted by standard gravity on one kilogram of mass. The kilogram-force leads to an alternate, but rarely used unit of mass: the metric slug (sometimes mug or hyl) is that mass that accelerates at 1 m·s−2 when subjected to a force of 1 kgf. The kilogram-force is not a part of the modern SI system, and is generally deprecated; however it still sees use for ...  \n11870  The pound-force has a metric counterpart, less commonly used than the newton: the kilogram-force (kgf) (sometimes kilopond), is the force exerted by standard gravity on one kilogram of mass. The kilogram-force leads to an alternate, but rarely used unit of mass: the metric slug (sometimes mug or hyl) is that mass that accelerates at 1 m·s−2 when subjected to a force of 1 kgf. The kilogram-force is not a part of the modern SI system, and is generally deprecated; however it still sees use for ...  \n11871  The pound-force has a metric counterpart, less commonly used than the newton: the kilogram-force (kgf) (sometimes kilopond), is the force exerted by standard gravity on one kilogram of mass. The kilogram-force leads to an alternate, but rarely used unit of mass: the metric slug (sometimes mug or hyl) is that mass that accelerates at 1 m·s−2 when subjected to a force of 1 kgf. The kilogram-force is not a part of the modern SI system, and is generally deprecated; however it still sees use for ...  \n11872  The pound-force has a metric counterpart, less commonly used than the newton: the kilogram-force (kgf) (sometimes kilopond), is the force exerted by standard gravity on one kilogram of mass. The kilogram-force leads to an alternate, but rarely used unit of mass: the metric slug (sometimes mug or hyl) is that mass that accelerates at 1 m·s−2 when subjected to a force of 1 kgf. The kilogram-force is not a part of the modern SI system, and is generally deprecated; however it still sees use for ...  \n\n[11873 rows x 3 columns]","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>question</th>\n      <th>answer</th>\n      <th>corpus</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>In what country is Normandy located?</td>\n      <td>[(159, 165, France)]</td>\n      <td>The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants ...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>When were the Normans in Normandy?</td>\n      <td>[(94, 117, 10th and 11th centuries), (87, 117, in the 10th and 11th centuries)]</td>\n      <td>The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants ...</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>From which countries did the Norse originate?</td>\n      <td>[(256, 283, Denmark, Iceland and Norway)]</td>\n      <td>The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants ...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>Who was the Norse leader?</td>\n      <td>[(308, 313, Rollo)]</td>\n      <td>The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants ...</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>What century did the Normans first gain their separate identity?</td>\n      <td>[(671, 675, 10th), (671, 683, 10th century), (649, 683, the first half of the 10th century)]</td>\n      <td>The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants ...</td>\n    </tr>\n    <tr>\n      <th>...</th>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>11868</th>\n      <td>What is the seldom used force unit equal to one thousand newtons?</td>\n      <td>[(665, 671, sthène)]</td>\n      <td>The pound-force has a metric counterpart, less commonly used than the newton: the kilogram-force (kgf) (sometimes kilopond), is the force exerted by standard gravity on one kilogram of mass. The kilogram-force leads to an alternate, but rarely used unit of mass: the metric slug (sometimes mug or hyl) is that mass that accelerates at 1 m·s−2 when subjected to a force of 1 kgf. The kilogram-force is not a part of the modern SI system, and is generally deprecated; however it still sees use for ...</td>\n    </tr>\n    <tr>\n      <th>11869</th>\n      <td>What does not have a metric counterpart?</td>\n      <td>[(4, 15, pound-force)]</td>\n      <td>The pound-force has a metric counterpart, less commonly used than the newton: the kilogram-force (kgf) (sometimes kilopond), is the force exerted by standard gravity on one kilogram of mass. The kilogram-force leads to an alternate, but rarely used unit of mass: the metric slug (sometimes mug or hyl) is that mass that accelerates at 1 m·s−2 when subjected to a force of 1 kgf. The kilogram-force is not a part of the modern SI system, and is generally deprecated; however it still sees use for ...</td>\n    </tr>\n    <tr>\n      <th>11870</th>\n      <td>What is the force exerted by standard gravity on one ton of mass?</td>\n      <td>[(82, 96, kilogram-force)]</td>\n      <td>The pound-force has a metric counterpart, less commonly used than the newton: the kilogram-force (kgf) (sometimes kilopond), is the force exerted by standard gravity on one kilogram of mass. The kilogram-force leads to an alternate, but rarely used unit of mass: the metric slug (sometimes mug or hyl) is that mass that accelerates at 1 m·s−2 when subjected to a force of 1 kgf. The kilogram-force is not a part of the modern SI system, and is generally deprecated; however it still sees use for ...</td>\n    </tr>\n    <tr>\n      <th>11871</th>\n      <td>What force leads to a commonly used unit of mass?</td>\n      <td>[(195, 209, kilogram-force)]</td>\n      <td>The pound-force has a metric counterpart, less commonly used than the newton: the kilogram-force (kgf) (sometimes kilopond), is the force exerted by standard gravity on one kilogram of mass. The kilogram-force leads to an alternate, but rarely used unit of mass: the metric slug (sometimes mug or hyl) is that mass that accelerates at 1 m·s−2 when subjected to a force of 1 kgf. The kilogram-force is not a part of the modern SI system, and is generally deprecated; however it still sees use for ...</td>\n    </tr>\n    <tr>\n      <th>11872</th>\n      <td>What force is part of the modern SI system?</td>\n      <td>[(383, 397, kilogram-force)]</td>\n      <td>The pound-force has a metric counterpart, less commonly used than the newton: the kilogram-force (kgf) (sometimes kilopond), is the force exerted by standard gravity on one kilogram of mass. The kilogram-force leads to an alternate, but rarely used unit of mass: the metric slug (sometimes mug or hyl) is that mass that accelerates at 1 m·s−2 when subjected to a force of 1 kgf. The kilogram-force is not a part of the modern SI system, and is generally deprecated; however it still sees use for ...</td>\n    </tr>\n  </tbody>\n</table>\n<p>11873 rows × 3 columns</p>\n</div>"},"metadata":{}}]},{"cell_type":"markdown","source":["**Exploding datasets on multiple answers**\n","\n","Now, we will explode the datasets on the answers list field because it will help later."],"metadata":{"id":"ViYC5HkjYtEA"}},{"cell_type":"code","source":["train_dataset = train_dataset.explode('answer').reset_index()\n","train_dataset"],"metadata":{"id":"nvJXORnjY-Ji","executionInfo":{"status":"ok","timestamp":1646593743068,"user_tz":-120,"elapsed":246,"user":{"displayName":"Βασίλης Κυριακόπουλος","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjLJ843-mdA-XYMgrr2K7G5eg-3pPcT6fAhtDjZiw=s64","userId":"10189451843812858163"}},"outputId":"b9463689-b098-402e-e6b2-4700b277e394","execution":{"iopub.status.busy":"2022-03-06T19:39:33.289898Z","iopub.execute_input":"2022-03-06T19:39:33.290228Z","iopub.status.idle":"2022-03-06T19:39:33.453964Z","shell.execute_reply.started":"2022-03-06T19:39:33.290192Z","shell.execute_reply":"2022-03-06T19:39:33.453139Z"},"trusted":true},"execution_count":null,"outputs":[{"execution_count":11,"output_type":"execute_result","data":{"text/plain":"         index  \\\n0            0   \n1            1   \n2            2   \n3            3   \n4            4   \n...        ...   \n130314  130314   \n130315  130315   \n130316  130316   \n130317  130317   \n130318  130318   \n\n                                                                 question  \\\n0                                When did Beyonce start becoming popular?   \n1              What areas did Beyonce compete in when she was growing up?   \n2        When did Beyonce leave Destiny's Child and become a solo singer?   \n3                           In what city and state did Beyonce  grow up?    \n4                              In which decade did Beyonce become famous?   \n...                                                                   ...   \n130314              Physics has broadly agreed on the definition of what?   \n130315                               Who coined the term partonic matter?   \n130316                              What is another name for anti-matter?   \n130317  Matter usually does not need to be used in conjunction with what?   \n130318             What field of study has a variety of unusual contexts?   \n\n                                   answer  \\\n0           (269, 286, in the late 1990s)   \n1         (207, 226, singing and dancing)   \n2                        (526, 530, 2003)   \n3              (166, 180, Houston, Texas)   \n4                  (276, 286, late 1990s)   \n...                                   ...   \n130314                 (485, 491, matter)   \n130315                 (327, 333, Alfvén)   \n130316      (350, 367, Gk. common matter)   \n130317  (529, 550, a specifying modifier)   \n130318                  (37, 44, physics)   \n\n                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     corpus  \n0       Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debu...  \n1       Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debu...  \n2       Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debu...  \n3       Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debu...  \n4       Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debu...  \n...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ...  \n130314  The term \"matter\" is used throughout physics in a bewildering variety of contexts: for example, one refers to \"condensed matter physics\", \"elementary matter\", \"partonic\" matter, \"dark\" matter, \"anti\"-matter, \"strange\" matter, and \"nuclear\" matter. In discussions of matter and antimatter, normal matter has been referred to by Alfvén as koinomatter (Gk. common matter). It is fair to say that in physics, there is no broad consensus as to a general definition of matter, and the term \"matter\" usu...  \n130315  The term \"matter\" is used throughout physics in a bewildering variety of contexts: for example, one refers to \"condensed matter physics\", \"elementary matter\", \"partonic\" matter, \"dark\" matter, \"anti\"-matter, \"strange\" matter, and \"nuclear\" matter. In discussions of matter and antimatter, normal matter has been referred to by Alfvén as koinomatter (Gk. common matter). It is fair to say that in physics, there is no broad consensus as to a general definition of matter, and the term \"matter\" usu...  \n130316  The term \"matter\" is used throughout physics in a bewildering variety of contexts: for example, one refers to \"condensed matter physics\", \"elementary matter\", \"partonic\" matter, \"dark\" matter, \"anti\"-matter, \"strange\" matter, and \"nuclear\" matter. In discussions of matter and antimatter, normal matter has been referred to by Alfvén as koinomatter (Gk. common matter). It is fair to say that in physics, there is no broad consensus as to a general definition of matter, and the term \"matter\" usu...  \n130317  The term \"matter\" is used throughout physics in a bewildering variety of contexts: for example, one refers to \"condensed matter physics\", \"elementary matter\", \"partonic\" matter, \"dark\" matter, \"anti\"-matter, \"strange\" matter, and \"nuclear\" matter. In discussions of matter and antimatter, normal matter has been referred to by Alfvén as koinomatter (Gk. common matter). It is fair to say that in physics, there is no broad consensus as to a general definition of matter, and the term \"matter\" usu...  \n130318  The term \"matter\" is used throughout physics in a bewildering variety of contexts: for example, one refers to \"condensed matter physics\", \"elementary matter\", \"partonic\" matter, \"dark\" matter, \"anti\"-matter, \"strange\" matter, and \"nuclear\" matter. In discussions of matter and antimatter, normal matter has been referred to by Alfvén as koinomatter (Gk. common matter). It is fair to say that in physics, there is no broad consensus as to a general definition of matter, and the term \"matter\" usu...  \n\n[130319 rows x 4 columns]","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>index</th>\n      <th>question</th>\n      <th>answer</th>\n      <th>corpus</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>0</td>\n      <td>When did Beyonce start becoming popular?</td>\n      <td>(269, 286, in the late 1990s)</td>\n      <td>Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&amp;B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debu...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>1</td>\n      <td>What areas did Beyonce compete in when she was growing up?</td>\n      <td>(207, 226, singing and dancing)</td>\n      <td>Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&amp;B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debu...</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>2</td>\n      <td>When did Beyonce leave Destiny's Child and become a solo singer?</td>\n      <td>(526, 530, 2003)</td>\n      <td>Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&amp;B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debu...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>3</td>\n      <td>In what city and state did Beyonce  grow up?</td>\n      <td>(166, 180, Houston, Texas)</td>\n      <td>Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&amp;B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debu...</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>4</td>\n      <td>In which decade did Beyonce become famous?</td>\n      <td>(276, 286, late 1990s)</td>\n      <td>Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&amp;B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debu...</td>\n    </tr>\n    <tr>\n      <th>...</th>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>130314</th>\n      <td>130314</td>\n      <td>Physics has broadly agreed on the definition of what?</td>\n      <td>(485, 491, matter)</td>\n      <td>The term \"matter\" is used throughout physics in a bewildering variety of contexts: for example, one refers to \"condensed matter physics\", \"elementary matter\", \"partonic\" matter, \"dark\" matter, \"anti\"-matter, \"strange\" matter, and \"nuclear\" matter. In discussions of matter and antimatter, normal matter has been referred to by Alfvén as koinomatter (Gk. common matter). It is fair to say that in physics, there is no broad consensus as to a general definition of matter, and the term \"matter\" usu...</td>\n    </tr>\n    <tr>\n      <th>130315</th>\n      <td>130315</td>\n      <td>Who coined the term partonic matter?</td>\n      <td>(327, 333, Alfvén)</td>\n      <td>The term \"matter\" is used throughout physics in a bewildering variety of contexts: for example, one refers to \"condensed matter physics\", \"elementary matter\", \"partonic\" matter, \"dark\" matter, \"anti\"-matter, \"strange\" matter, and \"nuclear\" matter. In discussions of matter and antimatter, normal matter has been referred to by Alfvén as koinomatter (Gk. common matter). It is fair to say that in physics, there is no broad consensus as to a general definition of matter, and the term \"matter\" usu...</td>\n    </tr>\n    <tr>\n      <th>130316</th>\n      <td>130316</td>\n      <td>What is another name for anti-matter?</td>\n      <td>(350, 367, Gk. common matter)</td>\n      <td>The term \"matter\" is used throughout physics in a bewildering variety of contexts: for example, one refers to \"condensed matter physics\", \"elementary matter\", \"partonic\" matter, \"dark\" matter, \"anti\"-matter, \"strange\" matter, and \"nuclear\" matter. In discussions of matter and antimatter, normal matter has been referred to by Alfvén as koinomatter (Gk. common matter). It is fair to say that in physics, there is no broad consensus as to a general definition of matter, and the term \"matter\" usu...</td>\n    </tr>\n    <tr>\n      <th>130317</th>\n      <td>130317</td>\n      <td>Matter usually does not need to be used in conjunction with what?</td>\n      <td>(529, 550, a specifying modifier)</td>\n      <td>The term \"matter\" is used throughout physics in a bewildering variety of contexts: for example, one refers to \"condensed matter physics\", \"elementary matter\", \"partonic\" matter, \"dark\" matter, \"anti\"-matter, \"strange\" matter, and \"nuclear\" matter. In discussions of matter and antimatter, normal matter has been referred to by Alfvén as koinomatter (Gk. common matter). It is fair to say that in physics, there is no broad consensus as to a general definition of matter, and the term \"matter\" usu...</td>\n    </tr>\n    <tr>\n      <th>130318</th>\n      <td>130318</td>\n      <td>What field of study has a variety of unusual contexts?</td>\n      <td>(37, 44, physics)</td>\n      <td>The term \"matter\" is used throughout physics in a bewildering variety of contexts: for example, one refers to \"condensed matter physics\", \"elementary matter\", \"partonic\" matter, \"dark\" matter, \"anti\"-matter, \"strange\" matter, and \"nuclear\" matter. In discussions of matter and antimatter, normal matter has been referred to by Alfvén as koinomatter (Gk. common matter). It is fair to say that in physics, there is no broad consensus as to a general definition of matter, and the term \"matter\" usu...</td>\n    </tr>\n  </tbody>\n</table>\n<p>130319 rows × 4 columns</p>\n</div>"},"metadata":{}}]},{"cell_type":"code","source":["validation_dataset = validation_dataset.explode('answer').reset_index()\n","validation_dataset"],"metadata":{"id":"8XBWvjFZYkth","executionInfo":{"status":"ok","timestamp":1646593745568,"user_tz":-120,"elapsed":238,"user":{"displayName":"Βασίλης Κυριακόπουλος","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjLJ843-mdA-XYMgrr2K7G5eg-3pPcT6fAhtDjZiw=s64","userId":"10189451843812858163"}},"outputId":"8bc24a53-1da2-41b9-fd09-901ae1bdc58f","execution":{"iopub.status.busy":"2022-03-06T19:39:36.018202Z","iopub.execute_input":"2022-03-06T19:39:36.018922Z","iopub.status.idle":"2022-03-06T19:39:36.050781Z","shell.execute_reply.started":"2022-03-06T19:39:36.018886Z","shell.execute_reply":"2022-03-06T19:39:36.050081Z"},"trusted":true},"execution_count":null,"outputs":[{"execution_count":12,"output_type":"execute_result","data":{"text/plain":"       index  \\\n0          0   \n1          1   \n2          1   \n3          2   \n4          3   \n...      ...   \n16328  11868   \n16329  11869   \n16330  11870   \n16331  11871   \n16332  11872   \n\n                                                                question  \\\n0                                   In what country is Normandy located?   \n1                                     When were the Normans in Normandy?   \n2                                     When were the Normans in Normandy?   \n3                          From which countries did the Norse originate?   \n4                                              Who was the Norse leader?   \n...                                                                  ...   \n16328  What is the seldom used force unit equal to one thousand newtons?   \n16329                           What does not have a metric counterpart?   \n16330  What is the force exerted by standard gravity on one ton of mass?   \n16331                  What force leads to a commonly used unit of mass?   \n16332                        What force is part of the modern SI system?   \n\n                                          answer  \\\n0                             (159, 165, France)   \n1             (94, 117, 10th and 11th centuries)   \n2      (87, 117, in the 10th and 11th centuries)   \n3        (256, 283, Denmark, Iceland and Norway)   \n4                              (308, 313, Rollo)   \n...                                          ...   \n16328                         (665, 671, sthène)   \n16329                       (4, 15, pound-force)   \n16330                   (82, 96, kilogram-force)   \n16331                 (195, 209, kilogram-force)   \n16332                 (383, 397, kilogram-force)   \n\n                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    corpus  \n0      The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants ...  \n1      The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants ...  \n2      The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants ...  \n3      The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants ...  \n4      The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants ...  \n...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    ...  \n16328  The pound-force has a metric counterpart, less commonly used than the newton: the kilogram-force (kgf) (sometimes kilopond), is the force exerted by standard gravity on one kilogram of mass. The kilogram-force leads to an alternate, but rarely used unit of mass: the metric slug (sometimes mug or hyl) is that mass that accelerates at 1 m·s−2 when subjected to a force of 1 kgf. The kilogram-force is not a part of the modern SI system, and is generally deprecated; however it still sees use for ...  \n16329  The pound-force has a metric counterpart, less commonly used than the newton: the kilogram-force (kgf) (sometimes kilopond), is the force exerted by standard gravity on one kilogram of mass. The kilogram-force leads to an alternate, but rarely used unit of mass: the metric slug (sometimes mug or hyl) is that mass that accelerates at 1 m·s−2 when subjected to a force of 1 kgf. The kilogram-force is not a part of the modern SI system, and is generally deprecated; however it still sees use for ...  \n16330  The pound-force has a metric counterpart, less commonly used than the newton: the kilogram-force (kgf) (sometimes kilopond), is the force exerted by standard gravity on one kilogram of mass. The kilogram-force leads to an alternate, but rarely used unit of mass: the metric slug (sometimes mug or hyl) is that mass that accelerates at 1 m·s−2 when subjected to a force of 1 kgf. The kilogram-force is not a part of the modern SI system, and is generally deprecated; however it still sees use for ...  \n16331  The pound-force has a metric counterpart, less commonly used than the newton: the kilogram-force (kgf) (sometimes kilopond), is the force exerted by standard gravity on one kilogram of mass. The kilogram-force leads to an alternate, but rarely used unit of mass: the metric slug (sometimes mug or hyl) is that mass that accelerates at 1 m·s−2 when subjected to a force of 1 kgf. The kilogram-force is not a part of the modern SI system, and is generally deprecated; however it still sees use for ...  \n16332  The pound-force has a metric counterpart, less commonly used than the newton: the kilogram-force (kgf) (sometimes kilopond), is the force exerted by standard gravity on one kilogram of mass. The kilogram-force leads to an alternate, but rarely used unit of mass: the metric slug (sometimes mug or hyl) is that mass that accelerates at 1 m·s−2 when subjected to a force of 1 kgf. The kilogram-force is not a part of the modern SI system, and is generally deprecated; however it still sees use for ...  \n\n[16333 rows x 4 columns]","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>index</th>\n      <th>question</th>\n      <th>answer</th>\n      <th>corpus</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>0</td>\n      <td>In what country is Normandy located?</td>\n      <td>(159, 165, France)</td>\n      <td>The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants ...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>1</td>\n      <td>When were the Normans in Normandy?</td>\n      <td>(94, 117, 10th and 11th centuries)</td>\n      <td>The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants ...</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>1</td>\n      <td>When were the Normans in Normandy?</td>\n      <td>(87, 117, in the 10th and 11th centuries)</td>\n      <td>The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants ...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>2</td>\n      <td>From which countries did the Norse originate?</td>\n      <td>(256, 283, Denmark, Iceland and Norway)</td>\n      <td>The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants ...</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>3</td>\n      <td>Who was the Norse leader?</td>\n      <td>(308, 313, Rollo)</td>\n      <td>The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants ...</td>\n    </tr>\n    <tr>\n      <th>...</th>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>16328</th>\n      <td>11868</td>\n      <td>What is the seldom used force unit equal to one thousand newtons?</td>\n      <td>(665, 671, sthène)</td>\n      <td>The pound-force has a metric counterpart, less commonly used than the newton: the kilogram-force (kgf) (sometimes kilopond), is the force exerted by standard gravity on one kilogram of mass. The kilogram-force leads to an alternate, but rarely used unit of mass: the metric slug (sometimes mug or hyl) is that mass that accelerates at 1 m·s−2 when subjected to a force of 1 kgf. The kilogram-force is not a part of the modern SI system, and is generally deprecated; however it still sees use for ...</td>\n    </tr>\n    <tr>\n      <th>16329</th>\n      <td>11869</td>\n      <td>What does not have a metric counterpart?</td>\n      <td>(4, 15, pound-force)</td>\n      <td>The pound-force has a metric counterpart, less commonly used than the newton: the kilogram-force (kgf) (sometimes kilopond), is the force exerted by standard gravity on one kilogram of mass. The kilogram-force leads to an alternate, but rarely used unit of mass: the metric slug (sometimes mug or hyl) is that mass that accelerates at 1 m·s−2 when subjected to a force of 1 kgf. The kilogram-force is not a part of the modern SI system, and is generally deprecated; however it still sees use for ...</td>\n    </tr>\n    <tr>\n      <th>16330</th>\n      <td>11870</td>\n      <td>What is the force exerted by standard gravity on one ton of mass?</td>\n      <td>(82, 96, kilogram-force)</td>\n      <td>The pound-force has a metric counterpart, less commonly used than the newton: the kilogram-force (kgf) (sometimes kilopond), is the force exerted by standard gravity on one kilogram of mass. The kilogram-force leads to an alternate, but rarely used unit of mass: the metric slug (sometimes mug or hyl) is that mass that accelerates at 1 m·s−2 when subjected to a force of 1 kgf. The kilogram-force is not a part of the modern SI system, and is generally deprecated; however it still sees use for ...</td>\n    </tr>\n    <tr>\n      <th>16331</th>\n      <td>11871</td>\n      <td>What force leads to a commonly used unit of mass?</td>\n      <td>(195, 209, kilogram-force)</td>\n      <td>The pound-force has a metric counterpart, less commonly used than the newton: the kilogram-force (kgf) (sometimes kilopond), is the force exerted by standard gravity on one kilogram of mass. The kilogram-force leads to an alternate, but rarely used unit of mass: the metric slug (sometimes mug or hyl) is that mass that accelerates at 1 m·s−2 when subjected to a force of 1 kgf. The kilogram-force is not a part of the modern SI system, and is generally deprecated; however it still sees use for ...</td>\n    </tr>\n    <tr>\n      <th>16332</th>\n      <td>11872</td>\n      <td>What force is part of the modern SI system?</td>\n      <td>(383, 397, kilogram-force)</td>\n      <td>The pound-force has a metric counterpart, less commonly used than the newton: the kilogram-force (kgf) (sometimes kilopond), is the force exerted by standard gravity on one kilogram of mass. The kilogram-force leads to an alternate, but rarely used unit of mass: the metric slug (sometimes mug or hyl) is that mass that accelerates at 1 m·s−2 when subjected to a force of 1 kgf. The kilogram-force is not a part of the modern SI system, and is generally deprecated; however it still sees use for ...</td>\n    </tr>\n  </tbody>\n</table>\n<p>16333 rows × 4 columns</p>\n</div>"},"metadata":{}}]},{"cell_type":"markdown","source":["## Tokenization process\n","\n","After loading the datasets, before we use them for the evaluation process, we must first tokenize them to calculate the gold answer start and end token positions."],"metadata":{"id":"vyOKw4O_vTC_"}},{"cell_type":"markdown","source":["**Download BERT tokenizer**"],"metadata":{"id":"qx0bT8qWC2W2"}},{"cell_type":"code","source":["tokenizer = BertTokenizerFast.from_pretrained(BERT_MODEL_NAME)"],"metadata":{"id":"uwfHVB8fC2w7","executionInfo":{"status":"ok","timestamp":1646593748992,"user_tz":-120,"elapsed":707,"user":{"displayName":"Βασίλης Κυριακόπουλος","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjLJ843-mdA-XYMgrr2K7G5eg-3pPcT6fAhtDjZiw=s64","userId":"10189451843812858163"}},"outputId":"8349adc2-8c93-4791-9c65-8b9b20bce455","execution":{"iopub.status.busy":"2022-03-06T19:39:39.537026Z","iopub.execute_input":"2022-03-06T19:39:39.537561Z","iopub.status.idle":"2022-03-06T19:39:41.088690Z","shell.execute_reply.started":"2022-03-06T19:39:39.537524Z","shell.execute_reply":"2022-03-06T19:39:41.088029Z"},"trusted":true,"colab":{"referenced_widgets":["bc593da227e9468dbfcef9f580d36c6c","e5a9c7901b724dd0a32a896f30610635","90941cd892064e69b3f2295f3819476f","d14c6adebcab48bba4d7026c0b5eeff7"]}},"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/plain":"Downloading:   0%|          | 0.00/226k [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"bc593da227e9468dbfcef9f580d36c6c"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Downloading:   0%|          | 0.00/455k [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"e5a9c7901b724dd0a32a896f30610635"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Downloading:   0%|          | 0.00/28.0 [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"90941cd892064e69b3f2295f3819476f"}},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"Downloading:   0%|          | 0.00/570 [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"d14c6adebcab48bba4d7026c0b5eeff7"}},"metadata":{}}]},{"cell_type":"markdown","source":["**Calculate start and end tokens of the answers**"],"metadata":{"id":"EvwlQ6BOh9CS"}},{"cell_type":"code","source":["def calculate_tokenized_ans_indices(dataset: pd.DataFrame):\n","  ans_tok_start = []\n","  ans_tok_end = []\n","  ans_tok_text = []\n","  for idx, ans in enumerate(dataset['answer'].values):\n","    if not pd.isna(ans):\n","      ans_text_start = ans[0]\n","      ans_text_end = ans[1]\n","      ans_text = ans[2]\n","      encoding = tokenizer.encode_plus(text=dataset['corpus'].values[idx], text_pair=dataset['question'].values[idx], max_length=512, padding='max_length', truncation=True)\n","      ans_start = encoding.char_to_token(0, ans_text_start)\n","      ans_end = encoding.char_to_token(0, ans_text_end-1)\n","      # Handle truncated answers\n","      if ans_start is None:\n","        ans_start = ans_end = tokenizer.model_max_length\n","      elif ans_end is None:\n","        ans_end = [i for i, inp in enumerate(encoding['input_ids']) if inp == tokenizer.sep_token_id][0]\n","      # Wrong because it gives free score points on truncated answers. Fixed this on the cross evaluation script.\n","      ans_text_tok = tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(encoding['input_ids'][ans_start:ans_end+1]))\n","    else:\n","      ans_start = ans_end = tokenizer.model_max_length\n","      ans_text_tok = \"\"\n","    ans_tok_start.append(ans_start)\n","    ans_tok_end.append(ans_end)\n","    ans_tok_text.append(ans_text_tok)\n","  dataset['ans_start_tok'] = ans_tok_start\n","  dataset['ans_end_tok'] = ans_tok_end\n","  dataset['ans_tok_text'] = ans_tok_text\n","  return dataset"],"metadata":{"id":"gBuRHPaUiFp-","execution":{"iopub.status.busy":"2022-03-06T19:39:47.781654Z","iopub.execute_input":"2022-03-06T19:39:47.781921Z","iopub.status.idle":"2022-03-06T19:39:47.792151Z","shell.execute_reply.started":"2022-03-06T19:39:47.781893Z","shell.execute_reply":"2022-03-06T19:39:47.790972Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["train_dataset = calculate_tokenized_ans_indices(train_dataset)\n","train_dataset"],"metadata":{"id":"n_JjLTfglsGv","outputId":"900136de-0c37-47f8-e082-66dbfe2995cb","executionInfo":{"status":"ok","timestamp":1646593824144,"user_tz":-120,"elapsed":68894,"user":{"displayName":"Βασίλης Κυριακόπουλος","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjLJ843-mdA-XYMgrr2K7G5eg-3pPcT6fAhtDjZiw=s64","userId":"10189451843812858163"}},"execution":{"iopub.status.busy":"2022-03-06T19:39:52.762561Z","iopub.execute_input":"2022-03-06T19:39:52.763223Z","iopub.status.idle":"2022-03-06T19:41:35.797875Z","shell.execute_reply.started":"2022-03-06T19:39:52.763180Z","shell.execute_reply":"2022-03-06T19:41:35.797095Z"},"trusted":true},"execution_count":null,"outputs":[{"execution_count":15,"output_type":"execute_result","data":{"text/plain":"         index  \\\n0            0   \n1            1   \n2            2   \n3            3   \n4            4   \n...        ...   \n130314  130314   \n130315  130315   \n130316  130316   \n130317  130317   \n130318  130318   \n\n                                                                 question  \\\n0                                When did Beyonce start becoming popular?   \n1              What areas did Beyonce compete in when she was growing up?   \n2        When did Beyonce leave Destiny's Child and become a solo singer?   \n3                           In what city and state did Beyonce  grow up?    \n4                              In which decade did Beyonce become famous?   \n...                                                                   ...   \n130314              Physics has broadly agreed on the definition of what?   \n130315                               Who coined the term partonic matter?   \n130316                              What is another name for anti-matter?   \n130317  Matter usually does not need to be used in conjunction with what?   \n130318             What field of study has a variety of unusual contexts?   \n\n                                   answer  \\\n0           (269, 286, in the late 1990s)   \n1         (207, 226, singing and dancing)   \n2                        (526, 530, 2003)   \n3              (166, 180, Houston, Texas)   \n4                  (276, 286, late 1990s)   \n...                                   ...   \n130314                 (485, 491, matter)   \n130315                 (327, 333, Alfvén)   \n130316      (350, 367, Gk. common matter)   \n130317  (529, 550, a specifying modifier)   \n130318                  (37, 44, physics)   \n\n                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     corpus  \\\n0       Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debu...   \n1       Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debu...   \n2       Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debu...   \n3       Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debu...   \n4       Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debu...   \n...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     ...   \n130314  The term \"matter\" is used throughout physics in a bewildering variety of contexts: for example, one refers to \"condensed matter physics\", \"elementary matter\", \"partonic\" matter, \"dark\" matter, \"anti\"-matter, \"strange\" matter, and \"nuclear\" matter. In discussions of matter and antimatter, normal matter has been referred to by Alfvén as koinomatter (Gk. common matter). It is fair to say that in physics, there is no broad consensus as to a general definition of matter, and the term \"matter\" usu...   \n130315  The term \"matter\" is used throughout physics in a bewildering variety of contexts: for example, one refers to \"condensed matter physics\", \"elementary matter\", \"partonic\" matter, \"dark\" matter, \"anti\"-matter, \"strange\" matter, and \"nuclear\" matter. In discussions of matter and antimatter, normal matter has been referred to by Alfvén as koinomatter (Gk. common matter). It is fair to say that in physics, there is no broad consensus as to a general definition of matter, and the term \"matter\" usu...   \n130316  The term \"matter\" is used throughout physics in a bewildering variety of contexts: for example, one refers to \"condensed matter physics\", \"elementary matter\", \"partonic\" matter, \"dark\" matter, \"anti\"-matter, \"strange\" matter, and \"nuclear\" matter. In discussions of matter and antimatter, normal matter has been referred to by Alfvén as koinomatter (Gk. common matter). It is fair to say that in physics, there is no broad consensus as to a general definition of matter, and the term \"matter\" usu...   \n130317  The term \"matter\" is used throughout physics in a bewildering variety of contexts: for example, one refers to \"condensed matter physics\", \"elementary matter\", \"partonic\" matter, \"dark\" matter, \"anti\"-matter, \"strange\" matter, and \"nuclear\" matter. In discussions of matter and antimatter, normal matter has been referred to by Alfvén as koinomatter (Gk. common matter). It is fair to say that in physics, there is no broad consensus as to a general definition of matter, and the term \"matter\" usu...   \n130318  The term \"matter\" is used throughout physics in a bewildering variety of contexts: for example, one refers to \"condensed matter physics\", \"elementary matter\", \"partonic\" matter, \"dark\" matter, \"anti\"-matter, \"strange\" matter, and \"nuclear\" matter. In discussions of matter and antimatter, normal matter has been referred to by Alfvén as koinomatter (Gk. common matter). It is fair to say that in physics, there is no broad consensus as to a general definition of matter, and the term \"matter\" usu...   \n\n        ans_start_tok  ans_end_tok           ans_tok_text  \n0                  67           70      in the late 1990s  \n1                  55           57    singing and dancing  \n2                 128          128                   2003  \n3                  47           49         houston, texas  \n4                  69           70             late 1990s  \n...               ...          ...                    ...  \n130314            122          122                 matter  \n130315             81           82                 alfven  \n130316             89           93      gk. common matter  \n130317            130          134  a specifying modifier  \n130318              9            9                physics  \n\n[130319 rows x 7 columns]","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>index</th>\n      <th>question</th>\n      <th>answer</th>\n      <th>corpus</th>\n      <th>ans_start_tok</th>\n      <th>ans_end_tok</th>\n      <th>ans_tok_text</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>0</td>\n      <td>When did Beyonce start becoming popular?</td>\n      <td>(269, 286, in the late 1990s)</td>\n      <td>Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&amp;B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debu...</td>\n      <td>67</td>\n      <td>70</td>\n      <td>in the late 1990s</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>1</td>\n      <td>What areas did Beyonce compete in when she was growing up?</td>\n      <td>(207, 226, singing and dancing)</td>\n      <td>Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&amp;B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debu...</td>\n      <td>55</td>\n      <td>57</td>\n      <td>singing and dancing</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>2</td>\n      <td>When did Beyonce leave Destiny's Child and become a solo singer?</td>\n      <td>(526, 530, 2003)</td>\n      <td>Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&amp;B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debu...</td>\n      <td>128</td>\n      <td>128</td>\n      <td>2003</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>3</td>\n      <td>In what city and state did Beyonce  grow up?</td>\n      <td>(166, 180, Houston, Texas)</td>\n      <td>Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&amp;B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debu...</td>\n      <td>47</td>\n      <td>49</td>\n      <td>houston, texas</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>4</td>\n      <td>In which decade did Beyonce become famous?</td>\n      <td>(276, 286, late 1990s)</td>\n      <td>Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&amp;B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debu...</td>\n      <td>69</td>\n      <td>70</td>\n      <td>late 1990s</td>\n    </tr>\n    <tr>\n      <th>...</th>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>130314</th>\n      <td>130314</td>\n      <td>Physics has broadly agreed on the definition of what?</td>\n      <td>(485, 491, matter)</td>\n      <td>The term \"matter\" is used throughout physics in a bewildering variety of contexts: for example, one refers to \"condensed matter physics\", \"elementary matter\", \"partonic\" matter, \"dark\" matter, \"anti\"-matter, \"strange\" matter, and \"nuclear\" matter. In discussions of matter and antimatter, normal matter has been referred to by Alfvén as koinomatter (Gk. common matter). It is fair to say that in physics, there is no broad consensus as to a general definition of matter, and the term \"matter\" usu...</td>\n      <td>122</td>\n      <td>122</td>\n      <td>matter</td>\n    </tr>\n    <tr>\n      <th>130315</th>\n      <td>130315</td>\n      <td>Who coined the term partonic matter?</td>\n      <td>(327, 333, Alfvén)</td>\n      <td>The term \"matter\" is used throughout physics in a bewildering variety of contexts: for example, one refers to \"condensed matter physics\", \"elementary matter\", \"partonic\" matter, \"dark\" matter, \"anti\"-matter, \"strange\" matter, and \"nuclear\" matter. In discussions of matter and antimatter, normal matter has been referred to by Alfvén as koinomatter (Gk. common matter). It is fair to say that in physics, there is no broad consensus as to a general definition of matter, and the term \"matter\" usu...</td>\n      <td>81</td>\n      <td>82</td>\n      <td>alfven</td>\n    </tr>\n    <tr>\n      <th>130316</th>\n      <td>130316</td>\n      <td>What is another name for anti-matter?</td>\n      <td>(350, 367, Gk. common matter)</td>\n      <td>The term \"matter\" is used throughout physics in a bewildering variety of contexts: for example, one refers to \"condensed matter physics\", \"elementary matter\", \"partonic\" matter, \"dark\" matter, \"anti\"-matter, \"strange\" matter, and \"nuclear\" matter. In discussions of matter and antimatter, normal matter has been referred to by Alfvén as koinomatter (Gk. common matter). It is fair to say that in physics, there is no broad consensus as to a general definition of matter, and the term \"matter\" usu...</td>\n      <td>89</td>\n      <td>93</td>\n      <td>gk. common matter</td>\n    </tr>\n    <tr>\n      <th>130317</th>\n      <td>130317</td>\n      <td>Matter usually does not need to be used in conjunction with what?</td>\n      <td>(529, 550, a specifying modifier)</td>\n      <td>The term \"matter\" is used throughout physics in a bewildering variety of contexts: for example, one refers to \"condensed matter physics\", \"elementary matter\", \"partonic\" matter, \"dark\" matter, \"anti\"-matter, \"strange\" matter, and \"nuclear\" matter. In discussions of matter and antimatter, normal matter has been referred to by Alfvén as koinomatter (Gk. common matter). It is fair to say that in physics, there is no broad consensus as to a general definition of matter, and the term \"matter\" usu...</td>\n      <td>130</td>\n      <td>134</td>\n      <td>a specifying modifier</td>\n    </tr>\n    <tr>\n      <th>130318</th>\n      <td>130318</td>\n      <td>What field of study has a variety of unusual contexts?</td>\n      <td>(37, 44, physics)</td>\n      <td>The term \"matter\" is used throughout physics in a bewildering variety of contexts: for example, one refers to \"condensed matter physics\", \"elementary matter\", \"partonic\" matter, \"dark\" matter, \"anti\"-matter, \"strange\" matter, and \"nuclear\" matter. In discussions of matter and antimatter, normal matter has been referred to by Alfvén as koinomatter (Gk. common matter). It is fair to say that in physics, there is no broad consensus as to a general definition of matter, and the term \"matter\" usu...</td>\n      <td>9</td>\n      <td>9</td>\n      <td>physics</td>\n    </tr>\n  </tbody>\n</table>\n<p>130319 rows × 7 columns</p>\n</div>"},"metadata":{}}]},{"cell_type":"code","source":["validation_dataset = calculate_tokenized_ans_indices(validation_dataset)\n","validation_dataset"],"metadata":{"id":"yfp7_F4y7xc-","outputId":"d5ecc67c-146f-4aa3-8025-649a9f0606a4","executionInfo":{"status":"ok","timestamp":1646593852361,"user_tz":-120,"elapsed":9445,"user":{"displayName":"Βασίλης Κυριακόπουλος","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjLJ843-mdA-XYMgrr2K7G5eg-3pPcT6fAhtDjZiw=s64","userId":"10189451843812858163"}},"execution":{"iopub.status.busy":"2022-03-06T19:41:35.799871Z","iopub.execute_input":"2022-03-06T19:41:35.800433Z","iopub.status.idle":"2022-03-06T19:41:49.082777Z","shell.execute_reply.started":"2022-03-06T19:41:35.800389Z","shell.execute_reply":"2022-03-06T19:41:49.082098Z"},"trusted":true},"execution_count":null,"outputs":[{"execution_count":16,"output_type":"execute_result","data":{"text/plain":"       index  \\\n0          0   \n1          1   \n2          1   \n3          2   \n4          3   \n...      ...   \n16328  11868   \n16329  11869   \n16330  11870   \n16331  11871   \n16332  11872   \n\n                                                                question  \\\n0                                   In what country is Normandy located?   \n1                                     When were the Normans in Normandy?   \n2                                     When were the Normans in Normandy?   \n3                          From which countries did the Norse originate?   \n4                                              Who was the Norse leader?   \n...                                                                  ...   \n16328  What is the seldom used force unit equal to one thousand newtons?   \n16329                           What does not have a metric counterpart?   \n16330  What is the force exerted by standard gravity on one ton of mass?   \n16331                  What force leads to a commonly used unit of mass?   \n16332                        What force is part of the modern SI system?   \n\n                                          answer  \\\n0                             (159, 165, France)   \n1             (94, 117, 10th and 11th centuries)   \n2      (87, 117, in the 10th and 11th centuries)   \n3        (256, 283, Denmark, Iceland and Norway)   \n4                              (308, 313, Rollo)   \n...                                          ...   \n16328                         (665, 671, sthène)   \n16329                       (4, 15, pound-force)   \n16330                   (82, 96, kilogram-force)   \n16331                 (195, 209, kilogram-force)   \n16332                 (383, 397, kilogram-force)   \n\n                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    corpus  \\\n0      The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants ...   \n1      The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants ...   \n2      The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants ...   \n3      The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants ...   \n4      The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants ...   \n...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    ...   \n16328  The pound-force has a metric counterpart, less commonly used than the newton: the kilogram-force (kgf) (sometimes kilopond), is the force exerted by standard gravity on one kilogram of mass. The kilogram-force leads to an alternate, but rarely used unit of mass: the metric slug (sometimes mug or hyl) is that mass that accelerates at 1 m·s−2 when subjected to a force of 1 kgf. The kilogram-force is not a part of the modern SI system, and is generally deprecated; however it still sees use for ...   \n16329  The pound-force has a metric counterpart, less commonly used than the newton: the kilogram-force (kgf) (sometimes kilopond), is the force exerted by standard gravity on one kilogram of mass. The kilogram-force leads to an alternate, but rarely used unit of mass: the metric slug (sometimes mug or hyl) is that mass that accelerates at 1 m·s−2 when subjected to a force of 1 kgf. The kilogram-force is not a part of the modern SI system, and is generally deprecated; however it still sees use for ...   \n16330  The pound-force has a metric counterpart, less commonly used than the newton: the kilogram-force (kgf) (sometimes kilopond), is the force exerted by standard gravity on one kilogram of mass. The kilogram-force leads to an alternate, but rarely used unit of mass: the metric slug (sometimes mug or hyl) is that mass that accelerates at 1 m·s−2 when subjected to a force of 1 kgf. The kilogram-force is not a part of the modern SI system, and is generally deprecated; however it still sees use for ...   \n16331  The pound-force has a metric counterpart, less commonly used than the newton: the kilogram-force (kgf) (sometimes kilopond), is the force exerted by standard gravity on one kilogram of mass. The kilogram-force leads to an alternate, but rarely used unit of mass: the metric slug (sometimes mug or hyl) is that mass that accelerates at 1 m·s−2 when subjected to a force of 1 kgf. The kilogram-force is not a part of the modern SI system, and is generally deprecated; however it still sees use for ...   \n16332  The pound-force has a metric counterpart, less commonly used than the newton: the kilogram-force (kgf) (sometimes kilopond), is the force exerted by standard gravity on one kilogram of mass. The kilogram-force leads to an alternate, but rarely used unit of mass: the metric slug (sometimes mug or hyl) is that mass that accelerates at 1 m·s−2 when subjected to a force of 1 kgf. The kilogram-force is not a part of the modern SI system, and is generally deprecated; however it still sees use for ...   \n\n       ans_start_tok  ans_end_tok                    ans_tok_text  \n0                 41           41                          france  \n1                 28           31         10th and 11th centuries  \n2                 26           31  in the 10th and 11th centuries  \n3                 63           67     denmark, iceland and norway  \n4                 73           74                           rollo  \n...              ...          ...                             ...  \n16328            158          160                          sthene  \n16329              2            4                   pound - force  \n16330             18           21                kilogram - force  \n16331             50           53                kilogram - force  \n16332            100          103                kilogram - force  \n\n[16333 rows x 7 columns]","text/html":"<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>index</th>\n      <th>question</th>\n      <th>answer</th>\n      <th>corpus</th>\n      <th>ans_start_tok</th>\n      <th>ans_end_tok</th>\n      <th>ans_tok_text</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>0</td>\n      <td>In what country is Normandy located?</td>\n      <td>(159, 165, France)</td>\n      <td>The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants ...</td>\n      <td>41</td>\n      <td>41</td>\n      <td>france</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>1</td>\n      <td>When were the Normans in Normandy?</td>\n      <td>(94, 117, 10th and 11th centuries)</td>\n      <td>The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants ...</td>\n      <td>28</td>\n      <td>31</td>\n      <td>10th and 11th centuries</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>1</td>\n      <td>When were the Normans in Normandy?</td>\n      <td>(87, 117, in the 10th and 11th centuries)</td>\n      <td>The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants ...</td>\n      <td>26</td>\n      <td>31</td>\n      <td>in the 10th and 11th centuries</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>2</td>\n      <td>From which countries did the Norse originate?</td>\n      <td>(256, 283, Denmark, Iceland and Norway)</td>\n      <td>The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants ...</td>\n      <td>63</td>\n      <td>67</td>\n      <td>denmark, iceland and norway</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>3</td>\n      <td>Who was the Norse leader?</td>\n      <td>(308, 313, Rollo)</td>\n      <td>The Normans (Norman: Nourmands; French: Normands; Latin: Normanni) were the people who in the 10th and 11th centuries gave their name to Normandy, a region in France. They were descended from Norse (\"Norman\" comes from \"Norseman\") raiders and pirates from Denmark, Iceland and Norway who, under their leader Rollo, agreed to swear fealty to King Charles III of West Francia. Through generations of assimilation and mixing with the native Frankish and Roman-Gaulish populations, their descendants ...</td>\n      <td>73</td>\n      <td>74</td>\n      <td>rollo</td>\n    </tr>\n    <tr>\n      <th>...</th>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>16328</th>\n      <td>11868</td>\n      <td>What is the seldom used force unit equal to one thousand newtons?</td>\n      <td>(665, 671, sthène)</td>\n      <td>The pound-force has a metric counterpart, less commonly used than the newton: the kilogram-force (kgf) (sometimes kilopond), is the force exerted by standard gravity on one kilogram of mass. The kilogram-force leads to an alternate, but rarely used unit of mass: the metric slug (sometimes mug or hyl) is that mass that accelerates at 1 m·s−2 when subjected to a force of 1 kgf. The kilogram-force is not a part of the modern SI system, and is generally deprecated; however it still sees use for ...</td>\n      <td>158</td>\n      <td>160</td>\n      <td>sthene</td>\n    </tr>\n    <tr>\n      <th>16329</th>\n      <td>11869</td>\n      <td>What does not have a metric counterpart?</td>\n      <td>(4, 15, pound-force)</td>\n      <td>The pound-force has a metric counterpart, less commonly used than the newton: the kilogram-force (kgf) (sometimes kilopond), is the force exerted by standard gravity on one kilogram of mass. The kilogram-force leads to an alternate, but rarely used unit of mass: the metric slug (sometimes mug or hyl) is that mass that accelerates at 1 m·s−2 when subjected to a force of 1 kgf. The kilogram-force is not a part of the modern SI system, and is generally deprecated; however it still sees use for ...</td>\n      <td>2</td>\n      <td>4</td>\n      <td>pound - force</td>\n    </tr>\n    <tr>\n      <th>16330</th>\n      <td>11870</td>\n      <td>What is the force exerted by standard gravity on one ton of mass?</td>\n      <td>(82, 96, kilogram-force)</td>\n      <td>The pound-force has a metric counterpart, less commonly used than the newton: the kilogram-force (kgf) (sometimes kilopond), is the force exerted by standard gravity on one kilogram of mass. The kilogram-force leads to an alternate, but rarely used unit of mass: the metric slug (sometimes mug or hyl) is that mass that accelerates at 1 m·s−2 when subjected to a force of 1 kgf. The kilogram-force is not a part of the modern SI system, and is generally deprecated; however it still sees use for ...</td>\n      <td>18</td>\n      <td>21</td>\n      <td>kilogram - force</td>\n    </tr>\n    <tr>\n      <th>16331</th>\n      <td>11871</td>\n      <td>What force leads to a commonly used unit of mass?</td>\n      <td>(195, 209, kilogram-force)</td>\n      <td>The pound-force has a metric counterpart, less commonly used than the newton: the kilogram-force (kgf) (sometimes kilopond), is the force exerted by standard gravity on one kilogram of mass. The kilogram-force leads to an alternate, but rarely used unit of mass: the metric slug (sometimes mug or hyl) is that mass that accelerates at 1 m·s−2 when subjected to a force of 1 kgf. The kilogram-force is not a part of the modern SI system, and is generally deprecated; however it still sees use for ...</td>\n      <td>50</td>\n      <td>53</td>\n      <td>kilogram - force</td>\n    </tr>\n    <tr>\n      <th>16332</th>\n      <td>11872</td>\n      <td>What force is part of the modern SI system?</td>\n      <td>(383, 397, kilogram-force)</td>\n      <td>The pound-force has a metric counterpart, less commonly used than the newton: the kilogram-force (kgf) (sometimes kilopond), is the force exerted by standard gravity on one kilogram of mass. The kilogram-force leads to an alternate, but rarely used unit of mass: the metric slug (sometimes mug or hyl) is that mass that accelerates at 1 m·s−2 when subjected to a force of 1 kgf. The kilogram-force is not a part of the modern SI system, and is generally deprecated; however it still sees use for ...</td>\n      <td>100</td>\n      <td>103</td>\n      <td>kilogram - force</td>\n    </tr>\n  </tbody>\n</table>\n<p>16333 rows × 7 columns</p>\n</div>"},"metadata":{}}]},{"cell_type":"markdown","source":["**Create custom dataset class**\n","\n","Now let's create the custom dataset class which also handle the tokenization part. This way, the tokenization happens during training and validation and we don't consume ram to save all the tokenizations in prior."],"metadata":{"id":"40jHaO2LiFX4"}},{"cell_type":"code","source":["class SQuAD_Dataset(Dataset):\n","  def __init__(self, data: pd.DataFrame):\n","    self.data = data\n","\n","  def __len__(self):\n","    return len(self.data)\n","\n","  def __getitem__(self, idx):\n","    encoding = tokenizer(text=self.data['corpus'].values[idx], text_pair=self.data['question'].values[idx], max_length=512, padding='max_length', truncation=True, return_tensors='pt').to(device)\n","    return {\n","        'ans_start_tok': torch.tensor(self.data['ans_start_tok'].values[idx], dtype=torch.long, device=device),\n","        'ans_end_tok': torch.tensor(self.data['ans_end_tok'].values[idx], dtype=torch.long, device=device),\n","        'input_ids': encoding['input_ids'][0], \n","        'attention_mask': encoding['attention_mask'][0],\n","        'token_type_ids': encoding['token_type_ids'][0]\n","    }"],"metadata":{"id":"Lg8jIWz0iFmx","execution":{"iopub.status.busy":"2022-03-06T19:42:48.721360Z","iopub.execute_input":"2022-03-06T19:42:48.722139Z","iopub.status.idle":"2022-03-06T19:42:48.732060Z","shell.execute_reply.started":"2022-03-06T19:42:48.722099Z","shell.execute_reply":"2022-03-06T19:42:48.731234Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["train_squad_dataset = SQuAD_Dataset(data=train_dataset)"],"metadata":{"id":"V1Iwzgz4mgil","execution":{"iopub.status.busy":"2022-03-06T19:42:51.631842Z","iopub.execute_input":"2022-03-06T19:42:51.632681Z","iopub.status.idle":"2022-03-06T19:42:51.637704Z","shell.execute_reply.started":"2022-03-06T19:42:51.632633Z","shell.execute_reply":"2022-03-06T19:42:51.636898Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["val_squad_dataset = SQuAD_Dataset(data=validation_dataset)"],"metadata":{"id":"LC7cg4hGoosl","execution":{"iopub.status.busy":"2022-03-06T19:42:52.872102Z","iopub.execute_input":"2022-03-06T19:42:52.872640Z","iopub.status.idle":"2022-03-06T19:42:52.876395Z","shell.execute_reply.started":"2022-03-06T19:42:52.872602Z","shell.execute_reply":"2022-03-06T19:42:52.875680Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["**Create Dataloaders**"],"metadata":{"id":"EgABQ9XAp6aW"}},{"cell_type":"code","source":["BATCH_SIZE=8"],"metadata":{"id":"cl4-YeWGp_MW","execution":{"iopub.status.busy":"2022-03-06T19:42:58.167995Z","iopub.execute_input":"2022-03-06T19:42:58.168564Z","iopub.status.idle":"2022-03-06T19:42:58.172218Z","shell.execute_reply.started":"2022-03-06T19:42:58.168530Z","shell.execute_reply":"2022-03-06T19:42:58.171522Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["train_dataloader = DataLoader(train_squad_dataset, batch_size=BATCH_SIZE, shuffle=False)"],"metadata":{"id":"FpV37IAZp6p5","execution":{"iopub.status.busy":"2022-03-06T19:43:00.144395Z","iopub.execute_input":"2022-03-06T19:43:00.145281Z","iopub.status.idle":"2022-03-06T19:43:00.149884Z","shell.execute_reply.started":"2022-03-06T19:43:00.145211Z","shell.execute_reply":"2022-03-06T19:43:00.149155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["validation_dataloader = DataLoader(val_squad_dataset, batch_size=1, shuffle=False)"],"metadata":{"id":"Fit_x6PmqHIt","execution":{"iopub.status.busy":"2022-03-06T19:43:01.639835Z","iopub.execute_input":"2022-03-06T19:43:01.640090Z","iopub.status.idle":"2022-03-06T19:43:01.644296Z","shell.execute_reply.started":"2022-03-06T19:43:01.640061Z","shell.execute_reply":"2022-03-06T19:43:01.643586Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["On the validation dataloader, i gave 1 as batch size and i'll explain why beyond."],"metadata":{"id":"N-ahnbTbTajz"}},{"cell_type":"markdown","source":["**Train & Evaluation methods**\n","\n","Now, let's define the training function. It follows the same logic as the previous ones. The only major difference is the way the validation loss is calculated. In particular, for each question in the validation dataset, i take the minimum loss among all the answers because the model's goal is to approach any valid answer as good as possible, and if i consider all answer losses (and even those from answers far from the one that gives the minimum loss), this will add extra noisy error for no reason and confuse the train-validation loss plot. To calculate the validation loss this way, we must take the answers one at a time, so this is the reason i set batch size=1 on the validation dataloader."],"metadata":{"id":"OnxB1D3DqYIM"}},{"cell_type":"code","source":["def train(model: BertForQuestionAnswering, train_dataloader: DataLoader, validation_dataloader: DataLoader, learning_rate: float, epochs: int):\n","  # Define the optimizer\n","  opt = torch.optim.AdamW(model.parameters(), lr=learning_rate)\n","  # Initialize train and validation losses lists\n","  train_losses = []\n","  validation_losses = []\n","  # Train for given # of epochs\n","  for epoch in range(epochs):\n","    model.train()\n","    t_losses = []\n","    for batch in tqdm(train_dataloader):\n","      # Delete previously stored gradients\n","      opt.zero_grad()\n","      # Get loss and outputs from the model\n","      start_positions = batch['ans_start_tok'] # we can do this because we know taht squad trai dataset questions always have 1 question\n","      end_positions = batch['ans_end_tok']\n","      out = model(input_ids=batch['input_ids'], attention_mask=batch['attention_mask'], token_type_ids=batch['token_type_ids'], start_positions=start_positions, end_positions=end_positions)\n","      # Perform backpropagation starting from the loss returned in this epoch\n","      loss = out[0]\n","      loss.backward()\n","      # Update model's weights based on the gradients calculated during backprop\n","      opt.step()\n","      t_losses.append(loss.item())\n","\n","    # Calculate train loss in current epoch\n","    train_loss = np.mean(t_losses) \n","    train_losses.append(train_loss)\n","    with torch.no_grad():\n","      model.eval()\n","      v_losses = []\n","      tmp_losses = []\n","      cur_idx = -1\n","      for val_idx, val_batch in enumerate(tqdm(validation_dataloader)):\n","        start_positions = val_batch['ans_start_tok'] # we can do this because we know that squad train dataset questions always have 1 question\n","        end_positions = val_batch['ans_end_tok']\n","        out = model(input_ids=val_batch['input_ids'], attention_mask=val_batch['attention_mask'], token_type_ids=val_batch['token_type_ids'], start_positions=start_positions, end_positions=end_positions)\n","        loss = out[0]\n","        batch_idx = validation_dataset['index'].values[val_idx]\n","        if cur_idx != batch_idx:\n","          cur_idx = batch_idx\n","          if len(tmp_losses) > 0:\n","            v_losses.append(min(tmp_losses))\n","          tmp_losses = []\n","        tmp_losses.append(loss.item())\n","      if len(tmp_losses) > 0:\n","        v_losses.append(min(tmp_losses))\n","      val_loss = np.mean(v_losses) \n","      validation_losses.append(val_loss)\n","\n","    # Print current epoch status\n","    print(f\"Epoch {epoch:3}: Loss = {train_loss:.5f} Val_loss = {val_loss:.5f}\")\n","\n","  return train_losses, validation_losses"],"metadata":{"id":"7xp2O95NSt51","execution":{"iopub.status.busy":"2022-03-06T19:43:04.928679Z","iopub.execute_input":"2022-03-06T19:43:04.928944Z","iopub.status.idle":"2022-03-06T19:43:04.941926Z","shell.execute_reply.started":"2022-03-06T19:43:04.928916Z","shell.execute_reply":"2022-03-06T19:43:04.941225Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["And let's also define a function to plot the train and validation losses."],"metadata":{"id":"93LL5omyVBWM"}},{"cell_type":"code","source":["def plot_loss_vs_epochs(train_losses, val_losses):\n","  assert(len(train_losses) == len(val_losses))\n","  plt.grid()\n","  plt.plot(list(range(len(train_losses))), train_losses, \"o-\", label=\"Train\")\n","  plt.plot(list(range(len(train_losses))), val_losses, \"o-\", label=\"Validation\")\n","  plt.xlabel(\"Epochs\")\n","  plt.ylabel(\"Loss\")\n","  plt.legend()\n","  plt.show()"],"metadata":{"id":"Hq8DRSvLgevj","execution":{"iopub.status.busy":"2022-03-06T19:43:09.328289Z","iopub.execute_input":"2022-03-06T19:43:09.329116Z","iopub.status.idle":"2022-03-06T19:43:09.335461Z","shell.execute_reply.started":"2022-03-06T19:43:09.329071Z","shell.execute_reply":"2022-03-06T19:43:09.334589Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":["**Model Loading**\n","\n","Now let's load the pre-trained bert-base-uncased model."],"metadata":{"id":"4yvh_2pnSn1x"}},{"cell_type":"code","source":["reset_seed()"],"metadata":{"id":"2zGtAdQnSqiV","execution":{"iopub.status.busy":"2022-03-06T19:43:11.557970Z","iopub.execute_input":"2022-03-06T19:43:11.558240Z","iopub.status.idle":"2022-03-06T19:43:11.562908Z","shell.execute_reply.started":"2022-03-06T19:43:11.558210Z","shell.execute_reply":"2022-03-06T19:43:11.562241Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":["model = BertForQuestionAnswering.from_pretrained(BERT_MODEL_NAME).to(device)\n","model"],"metadata":{"id":"MyKrefB-SsHn","outputId":"09505609-7c10-43b2-84e5-745c119359f5","executionInfo":{"status":"ok","timestamp":1646594801054,"user_tz":-120,"elapsed":1645,"user":{"displayName":"Βασίλης Κυριακόπουλος","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjLJ843-mdA-XYMgrr2K7G5eg-3pPcT6fAhtDjZiw=s64","userId":"10189451843812858163"}},"execution":{"iopub.status.busy":"2022-03-06T19:43:13.495238Z","iopub.execute_input":"2022-03-06T19:43:13.495780Z","iopub.status.idle":"2022-03-06T19:43:27.926296Z","shell.execute_reply.started":"2022-03-06T19:43:13.495745Z","shell.execute_reply":"2022-03-06T19:43:27.925547Z"},"trusted":true,"colab":{"referenced_widgets":["0ee05047b49049b6a9d6fddf0f75ac25"]}},"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/plain":"Downloading:   0%|          | 0.00/420M [00:00<?, ?B/s]","application/vnd.jupyter.widget-view+json":{"version_major":2,"version_minor":0,"model_id":"0ee05047b49049b6a9d6fddf0f75ac25"}},"metadata":{}},{"name":"stderr","text":"Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForQuestionAnswering: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.bias']\n- This IS expected if you are initializing BertForQuestionAnswering from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n- This IS NOT expected if you are initializing BertForQuestionAnswering from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\nSome weights of BertForQuestionAnswering were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['qa_outputs.bias', 'qa_outputs.weight']\nYou should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n","output_type":"stream"},{"execution_count":26,"output_type":"execute_result","data":{"text/plain":"BertForQuestionAnswering(\n  (bert): BertModel(\n    (embeddings): BertEmbeddings(\n      (word_embeddings): Embedding(30522, 768, padding_idx=0)\n      (position_embeddings): Embedding(512, 768)\n      (token_type_embeddings): Embedding(2, 768)\n      (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n      (dropout): Dropout(p=0.1, inplace=False)\n    )\n    (encoder): BertEncoder(\n      (layer): ModuleList(\n        (0): BertLayer(\n          (attention): BertAttention(\n            (self): BertSelfAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): BertSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate): BertIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n          )\n          (output): BertOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (1): BertLayer(\n          (attention): BertAttention(\n            (self): BertSelfAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): BertSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate): BertIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n          )\n          (output): BertOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (2): BertLayer(\n          (attention): BertAttention(\n            (self): BertSelfAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): BertSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate): BertIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n          )\n          (output): BertOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (3): BertLayer(\n          (attention): BertAttention(\n            (self): BertSelfAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): BertSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate): BertIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n          )\n          (output): BertOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (4): BertLayer(\n          (attention): BertAttention(\n            (self): BertSelfAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): BertSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate): BertIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n          )\n          (output): BertOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (5): BertLayer(\n          (attention): BertAttention(\n            (self): BertSelfAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): BertSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate): BertIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n          )\n          (output): BertOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (6): BertLayer(\n          (attention): BertAttention(\n            (self): BertSelfAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): BertSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate): BertIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n          )\n          (output): BertOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (7): BertLayer(\n          (attention): BertAttention(\n            (self): BertSelfAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): BertSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate): BertIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n          )\n          (output): BertOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (8): BertLayer(\n          (attention): BertAttention(\n            (self): BertSelfAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): BertSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate): BertIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n          )\n          (output): BertOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (9): BertLayer(\n          (attention): BertAttention(\n            (self): BertSelfAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): BertSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate): BertIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n          )\n          (output): BertOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (10): BertLayer(\n          (attention): BertAttention(\n            (self): BertSelfAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): BertSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate): BertIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n          )\n          (output): BertOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n        (11): BertLayer(\n          (attention): BertAttention(\n            (self): BertSelfAttention(\n              (query): Linear(in_features=768, out_features=768, bias=True)\n              (key): Linear(in_features=768, out_features=768, bias=True)\n              (value): Linear(in_features=768, out_features=768, bias=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n            (output): BertSelfOutput(\n              (dense): Linear(in_features=768, out_features=768, bias=True)\n              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n              (dropout): Dropout(p=0.1, inplace=False)\n            )\n          )\n          (intermediate): BertIntermediate(\n            (dense): Linear(in_features=768, out_features=3072, bias=True)\n          )\n          (output): BertOutput(\n            (dense): Linear(in_features=3072, out_features=768, bias=True)\n            (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n            (dropout): Dropout(p=0.1, inplace=False)\n          )\n        )\n      )\n    )\n  )\n  (qa_outputs): Linear(in_features=768, out_features=2, bias=True)\n)"},"metadata":{}}]},{"cell_type":"markdown","source":["**Training**\n","\n","And now, time for training...\n","\n","Parameters used:\n","\n","\n","*   Batch Size = 8\n","*   Learning Rate = 1e-5\n","*   Epochs = 3\n","*   Max sequence length = 512\n","\n","\n","\n","Note that i tried different parameters on the fine tuning process. Some of which are:\n","Batch Size: [8, 16, 32, 64]\n","Learning Rate: [1e-5, 2e-5, 3e-5]\n","Max Sequence Length: [128, 180, 256, 512]"],"metadata":{"id":"DwXy6pOJVfHY"}},{"cell_type":"code","source":["train_losses, validation_losses = train(model, train_dataloader, validation_dataloader, learning_rate=1e-5, epochs=3)"],"metadata":{"id":"21VOfQ1Pg1zp","outputId":"12c2b610-2ace-4c43-b642-61a9f25f8d10","executionInfo":{"status":"ok","timestamp":1646595124195,"user_tz":-120,"elapsed":320126,"user":{"displayName":"Βασίλης Κυριακόπουλος","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14GjLJ843-mdA-XYMgrr2K7G5eg-3pPcT6fAhtDjZiw=s64","userId":"10189451843812858163"}},"execution":{"iopub.status.busy":"2022-03-06T19:43:39.169912Z","iopub.execute_input":"2022-03-06T19:43:39.170187Z","iopub.status.idle":"2022-03-07T02:18:08.881763Z","shell.execute_reply.started":"2022-03-06T19:43:39.170156Z","shell.execute_reply":"2022-03-07T02:18:08.880911Z"},"trusted":true},"execution_count":null,"outputs":[{"name":"stderr","text":"100%|██████████| 16290/16290 [2:05:14<00:00,  2.17it/s] \n100%|██████████| 16333/16333 [06:16<00:00, 43.41it/s]\n","output_type":"stream"},{"name":"stdout","text":"Epoch   0: Loss = 1.36773 Val_loss = 1.05447\n","output_type":"stream"},{"name":"stderr","text":"100%|██████████| 16290/16290 [2:05:12<00:00,  2.17it/s] \n100%|██████████| 16333/16333 [06:17<00:00, 43.22it/s]\n","output_type":"stream"},{"name":"stdout","text":"Epoch   1: Loss = 0.90817 Val_loss = 1.04992\n","output_type":"stream"},{"name":"stderr","text":"100%|██████████| 16290/16290 [2:05:10<00:00,  2.17it/s] \n100%|██████████| 16333/16333 [06:17<00:00, 43.24it/s]","output_type":"stream"},{"name":"stdout","text":"Epoch   2: Loss = 0.68231 Val_loss = 1.15268\n","output_type":"stream"},{"name":"stderr","text":"\n","output_type":"stream"}]},{"cell_type":"code","source":["plot_loss_vs_epochs(train_losses, validation_losses)"],"metadata":{"id":"TXx0EVLwmLem","execution":{"iopub.status.busy":"2022-03-07T02:18:08.883529Z","iopub.execute_input":"2022-03-07T02:18:08.883855Z","iopub.status.idle":"2022-03-07T02:18:09.146767Z","shell.execute_reply.started":"2022-03-07T02:18:08.883817Z","shell.execute_reply":"2022-03-07T02:18:09.146083Z"},"trusted":true,"outputId":"7ffabc69-6699-4bba-dbb9-6f71e6726a2d"},"execution_count":null,"outputs":[{"output_type":"display_data","data":{"text/plain":"<Figure size 432x288 with 1 Axes>","image/png":"iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAA47ElEQVR4nO3deVxVdf7H8deHHVkEJFFBRXNfQAQ3sEksp73UXLJNzDJtb37TYjXl1LTMVFPjTNpiLk0lmanTbpNJNeKGqbiklbu4ZqKgoCzf3x/3ilcEBLyHe+F+no/HfXDvWe59c7iczz3f873fI8YYlFJKeS4vVwdQSinlWloIlFLKw2khUEopD6eFQCmlPJwWAqWU8nBaCJRSysNZVghEZLqIHBCR9edYrpeIFIvIMKuyKKWUqpyVRwQzgcurWkBEvIG/Al9ZmEMppVQVLCsExpjvgN/Osdi9wEfAAatyKKWUqpqPq15YRKKBIUAq0Ku660VGRprY2NhaveaxY8cICgqq1bpWctdc4L7ZNFfNaK6aaYi5Vq1a9asx5oKK5rmsEACvAo8YY0pFpMoFRWQcMA4gKiqKl156qVYvmJ+fT3BwcK3WtZK75gL3zaa5akZz1UxDzJWamrqj0pnGGMtuQCywvpJ524Dt9ls+tuahwed6zsTERFNbixcvrvW6VnLXXMa4bzbNVTOaq2YaYi4gy1SyX3XZEYExps2p+yIyE/jUGLPAVXmUUspTWVYIRGQ2MACIFJHdwFOAL4Ax5nWrXlcppVTNWFYIjDGjarBsmlU5lFLuS0TYtm0bhYWFro5yhsaNG/Pjjz+6OsZZqpMrICCAmJgYfH19q/28rjxZrJTycEFBQYSEhBAbG8u5Oo3Upby8PEJCQlwd4yznymWM4dChQ+zevZs2bdpUulx5HjHExILVOaS88A1pXx4j5YVvWLA6x9WRlFKAt7c3TZo0casiUJ+JCE2aNKnxEVaDPyJYsDqHifPWUVBUAkBObgET560DYHBCtCujKaVAi4CT1WZ7NvgjghcXbi4rAqcUFJXw4sLNLkqklFLupcEXgj25BTWarpTyHIcOHaJHjx706NGDZs2aER0dTY8ePUhJSeHkyZNVrpuVlcV9991XR0mt1eCbhlqEBZJTwU6/aai/C9Iopc7HgtU5vLhwM3tyC2gRFshDl3U8rybeJk2asGbNGgAmTZpEcHAwf/zjH8nLy8PPz4/i4mJ8fCreTSYlJZGUlFTr13YnDf6I4KHLOhLo633W9LyCIrK2n2tMPKWUuzh1vi8ntwDD6fN9zu78kZaWxgMPPECfPn14+OGHWbFiBf369SMhIYHk5GQ2b7Y1K2dkZHD11VcDtiJy2223MWDAANq2bcvkyZOdmslqDf6I4NSnhRcXbiYnt4DosEDSkmN5f8VObpy2nFdG9OCquOYuTqmU+vMnG9i452il81fvzOVkSekZ0wqKSnh4bjazV+yscJ0uLUJ56pquNc6Sk5NDZmYm3t7eHD16lO+//x4fHx++/vprHnvsMT766KOz1tm0aROLFy8mLy+Pjh07MmHChBr15XelBl8IwFYMBidEk5GRwYABAwC4PjGGce9kcff7P7D7cCfG/a6t9l5Qyo2VLwLnmn4+Bg8ejLe3rSXhyJEjjB49mp9//hkRoaioqMJ1rrrqKvz9/fH396dp06bs37+fmJgYp2ezgkcUgopEBPnx7u19+L85a3n+i03sOnycSdd0xce7wbeWKeWWzvXJPeWFbyo83xcdFsgHd/ZzahbHoZ7/9Kc/kZqayvz589m+fXvZh8ny/P1Pn3f09vamuLjYqZms5NF7vQBfb/45KoE7f9eWd5ftZNy/V3HsRP354ynlSSo63xfo681Dl3W09HWPHDlCdLStiXnmzJmWvpareHQhAPDyEiZe2ZlnBncjY/MBRr65lANH3WvcE6WUrYn3+aHdiQ4LRLAdCTw/tLvlXwx9+OGHmThxIgkJCfXqU35NeGzTUHm39G1NdFgA97y/miFTMpkxphcdotxvrBGlPNmp831WmDRpUtn9vLy8svv9+vXjp59+Knv8l7/8BYABAwaUNRM5rguwfv16SzJaxeOPCBwN7BTFB+P6cbKklOunZpK55VdXR1JKKctpISine0xj5t+VTLPQAEZPX8G8H3a7OpJSSllKC0EFYsIbMXdCMkmtI/jDnLVMXvTzqctrKqVUg6OFoBKNA32ZdVtvhiZE8/f//sTDc7MpsqC/slJKuZqeLK6Cn48XL4+IJyaiEZMX/czeI4VMubknoQH149uCSilVHXpEcA4iwh8GdeBvw+JYtvUQI15fqiOXKqUaFC0E1TQiqSUzx/Qm53ABQ6YsYcOeI66OpJQ6T6mpqSxcuPCMaa+++ioPPvhghcsPGDCArKwsAK688kpyc3PPWmbSpEm89NJLVb7uggUL2LhxY9njJ598kq+//rqG6Z3HskIgItNF5ICIVNihVkSuE5FsEVkjIlki0t+qLM7Sv30kH07oh5cII15fSsbmA66OpJRnyZ4Dr3SDSWG2n9lzzuvpRo0aRXp6+hnT0tPTGTZs2DnX/fzzzwkLC6vV65YvBE8//TSXXnpprZ7LGaw8IpgJXF7F/EVAvDGmB3AbMM3CLE7TqVkoC+5OoXWTIMbOyqp01EOllJNlz4FP7oMjuwBj+/nJfedVDIYNG8Znn31WdhGa7du3s2fPHubOnUtSUhJdu3blqaeeqnDd2NhYfv3V9l2jZ599lg4dOtC/f/+yYaoB3nrrLXr16kV8fDzXX389x48fJzMzk48//piHHnqIHj16sGXLFtLS0pg7dy4AixYtIiEhge7du3Pbbbdx4sSJstd79tln6dmzJ927d2fTpk21/r3Ls+xksTHmOxGJrWJ+vsPDIKDe9M+MCg1gzvh+3P3eD0yct47dh4/zx9931NFLlTofXzwK+9ZVPn/3Sig5cea0ogL4zz2walbF6zTrDle8UOlTRkRE0Lt3b7744guuu+460tPTGTFiBPfeey+tW7empKSESy65hOzsbOLi4ip8jlWrVpGens6aNWsoLi6mZ8+eJCYmAjB06FDuuOMOAJ544gnefvtt7r33Xq699lquvvrqs448CgsLSUtLY9GiRXTo0IFbb72VqVOn8sADDwC2C+n88MMPTJkyhZdeeolp05zz+dml5whEZIiIbAI+w3ZUUG8E+/vw9ugkRvVuxWuLt/DAB2s4UVxy7hWVUrVTvgica3o1OTYPpaenM2rUKObPn0/Pnj1JSEhgw4YNZzTjlPf9998zZMgQGjVqRGhoKNdee23ZvPXr13PRRRfRvXt33nvvPTZs2FBlls2bN9OmTRs6dOgAwOjRo/nuu+/K5p967sTERLZv317bX/ksLu0+aoyZD8wXkd8BzwAVNpKJyDhgHEBUVBQZGRm1er38/Pxar1uZ34cbijr4MnfNHjbt2Me9CQEE+9XsyMCKXM7irtk0V824a67Q0NDT4/r0f7zKZYPe7INX3tlXIysNiebYsPQK1rBzGDeoIgMHDuSBBx7g+++/Jz8/Hz8/PyZPnkxGRgbh4eGMHz+e3Nxc8vLyKCkp4dixY+Tl5WGMIT8/n8LCQk6cOFH2e5w8ebLs8ejRo3n//ffLCsH3339PXl4eRUVFFBQUlK1z6vGxY8coKSkpm378+HGKi4vLXs/Hx4e8vLyzXrO8wsLCGv293eJ7BPZmpLYiEmmMOWuAH2PMm8CbAElJSaay8cDPxfHCNM6UmgoXrcnhoQ+zeWWdMHNMb1pGNHJ5Lmdw12yaq2bcNdfq1asJCanm4I6DJtnOCRQ5dN/2DcRr0KTqP0cFQkJCGDhwIPfeey833XQTpaWlBAUFERMTw8GDB/n6668ZNGgQISEheHt7ExQUREhICCJCcHAwv//970lLS2PSpEkUFxezcOFC7rzzTkJCQsjPz6ddu3YEBATw0UcfER0dTUhICBERERQXF5fl9vX1JTAwkJ49e7Jr1y72799Pu3bt+Oijj7jkkkvKXs/b25uQkBCCgoLK7lckICCAhISEam8DlzUNiUg7sTeqi0hPwB845Ko85+u6HtH8e2xvfs0/yZApS1i7K9fVkZRqWOJGwDWToXFLQGw/r5lsm36eRo0axdq1axk1ahTx8fHExcXRqVMnbrzxRlJSUqpct2fPnowcOZL4+HiuuOIKevXqVTbvmWeeoU+fPqSkpNCpU6ey6TfccAMvvvgiCQkJbNmypWx6QEAAM2bMYPjw4XTv3h0vLy/Gjx9/3r/fORljLLkBs4G9QBGwGxgLjAfG2+c/AmwA1gBLgf7Ved7ExERTW4sXL671utX18/48k/LCItPxic/NwvV7q7VOXeSqLXfNprlqxl1z/fDDD66OUKGjR4+6OkKFqptr48aNZ00Dskwl+1Urew2NOsf8vwJ/ter1XaVd02Dm35XC7bNWcue7q3jq6i6kpbRxdSyllKqUfrPYAheE+DN7XF8u7RzFpE828synGyktrTe9Y5VSHkYLgUUa+fnw+s2JpCXH8vb/tnHXez9QWKTdS5Uqz+gQ705Vm+2phcBC3l7CpGu78qeru7Bw4z5GvbWMQ/nn1+dZqYakpKSEQ4cOaTFwEmMMhw4dIiAgoEbruUX30YZubP82RIcFcH/6GoZOzWRGWi/aXhDs6lhKudypPvkHDx50dZQzFBYW1nhnWheqkysgIICYmJgaPa8WgjpyebfmzB4XwO2zshg6NZNptyaRFBvh6lhKuZQxhjZt3K8zRUZGRo364dcVq3Jp01Ad6tkqnPl3JRPeyI8bpy3n0+w9ro6klFJaCOpa6yZBzJuQTFx0Y+55fzWvf7tF20eVUi6lhcAFwoP8ePf2PlwV15wXvtjEOxtPUqzXQ1ZKuYgWAhcJ8PXmnzckcOfFbVm8q5hx/17FsRPFro6llPJAWghcyMtLmHhFZ27t4kfG5gOMfHMpB44WujqWUsrDaCFwAwNb+fL26F5sPXiMIVMy+Wl/1cPmKqWUM2khcBOpnZoy585+nCwp5fqpmWT+ctZo3EopZQktBG6kW3RjFtydQvPGAYyesYJ5P+x2dSSllAfQQuBmosMC+XB8Mr1iI/jDnLVMXvSzdi9VSllKC4Ebahzoy8wxvRnaM5q///cnHp6bTZF2L1VKWUSHmHBTfj5evDw8npbhjfjHop/Ze6SQKTf3JDTA19XRlFINjB4RuDER4cFBHXhxWBzLth5ixOtL2ZNbcO4VlVKqBrQQ1APDk1oyc0xvcg4XMGTKEjbsOeLqSEqpBkQLQT3Rv30kH07oh5cII15fSsbmA66OpJRqILQQ1COdmoWy4O4UWjcJYuysLGav2OnqSEqpBsCyQiAi00XkgIisr2T+TSKSLSLrRCRTROKtytKQRIUGMGd8P/q3i2TivHX87ctNej1kpdR5sfKIYCZweRXztwEXG2O6A88Ab1qYpUEJ9vfh7dFJjOrdiikZW3jggzWcKNbrISulasey7qPGmO9EJLaK+ZkOD5cBNbu2mofz8fbiuSHdaBkRyN++3My+o4W8eUsiYY38XB1NKVXPuMs5grHAF64OUd+ICHcNaMc/bujBmp25DJ2ayc5Dx10dSylVz4iVwxfYjwg+NcZ0q2KZVGAK0N8Yc6iSZcYB4wCioqIS09PTa5UnPz+f4GD3u2i8M3Jt/q2EyasL8RZ4oGcAbcO83SabFTRXzWiummmIuVJTU1cZY5IqnGmMsewGxALrq5gfB2wBOlT3ORMTE01tLV68uNbrWslZuX7en2f6/3WR6fjE52bh+r1Oec6Gvs2cTXPVjOaqmfPJBWSZSvarLmsaEpFWwDzgFmPMT67K0ZC0axrMvAkpdGwWyp3vrmLGkm2ujqSUqges7D46G1gKdBSR3SIyVkTGi8h4+yJPAk2AKSKyRkSyrMriSS4I8Sf9jr5c2jmKP3+ykac/2UiJdi9VSlXByl5Do84x/3bgdqte35MF+nnz+s2JPPPpRqYv2UZO7nFeHZlAoJ9zzhsopRoWd+k1pJzM20uYdG1Xnry6C19t3M+ot5ZxKP+Eq2MppdyQFoIG7rb+bZh6UyI/7j3K0KmZbD2Y7+pISik3o4XAA1zerRnp4/qSX1jM0KmZZG3/zdWRlFJuRAuBh0hoFc68u5KJaOTHjdOW82n2HldHUkq5CS0EHqR1kyA+mpBMfExj7nl/Na9/u0Wvh6yU0kLgacKD/Pj32D5cHdecF77YxBML1lOs10NWyr1lz4FXunFxxmB4pZvtsRPpNYs9UICvN5NvSCAmvBGvf7uFvUcK+eeoBIL89e2glNvJngOf3AdFBQjAkV22xwBxI5zyEnpE4KG8vIRHr+jEs0O6kbH5ACPfXMqBo4WujqWUKu/rSVBU7lrlRQWw6GmnvYR+BPRwN/VpTYvGgdz9/g8MmZLJjDG96BAV4upYSnmm/IOwdy3sXWP7uS8bjuZUvOyR3U57WS0EitROTZlzZz9um7mS66dm8sbNiSS3i3R1LKUaLmNsO/i9a2Fvtv3nWshz6M0X3gaa94CCw1B45OznaOy8S7hoIVAAdItuzPy7UxgzYwWjZ6zghaFxXJ+o1wpS6rwZA4e3nd7Zn7odt4+6L14Q2QHaXATN4223Zt0hoLFtvsM5gjK+gXDJk06LqIVAlYkOC+TD8clMeHcV//fhWnYfLiDOW7uXKlVtpSXw689n7vD3ZcOJo7b5Xr7QtDN0vPL0Tj+qK/gFVf6cp04IL3oac2Q30jjGVgScdKIYtBCochoH+jJzTG8enZfNK1//RP9oH1IuKsXPR/sVKHWG4pNw8MdyO/31UGz/5O4TAFHdoPvw0zv9pp3Bx7/mrxU3AuJG8G1GBgMGDHDqrwFaCFQF/Hy8eHl4PC3DG/GPRT9z28yVTLm5J6EBvq6OppRrnDwO+zfYTuLus7fp798IpUW2+X4h0DwOksac3uk3aQ/e9WMXWz9SqjonIjw4qAP5+3cwa+Mhhk9dyowxvWgRFujqaEpZq/AojXM3wNKNpz/p/7oZjP2Ll4ERth19v7tP7/TD24BX/T1q1kKgqnRRjC+pfRKY8O4qBr+2hOlpvegW3djVsZRyjmOHYF+5k7i/bSXh1PyQ5rYdfedrTu/0G8eAiCtTO50WAnVO/dtH8uGEftw2YyUj31jKv27qSWrHpq6OpVT1GQN5+84+iXtk1+llwlrZdvQ9biT7oBdxv78ZQqJcl7kOaSFQ1dKpWai9e+lKbp+VxTPXdePGPq1cHUupsxkDuTvO7qN/7IB9AYEm7aBlH+g97nR3zUYRZU/xW0aGxxQB0EKgaiAqNIA54/txz/s/8Nj8dew6fJyHft8RL6+GdZis6pHSEji0xX4Cd83pnf6pL2CJt62nTvtB9h1+HDTrBv767XlHWghUjQT7+zDt1iSe/HgDUzO2sPtwAS8Nj8PfR6+HrCxWUgQHN5dr3lkHRcds8739bX3yuw5x6K7ZFXwDXJu7HrCsEIjIdOBq4IAxplsF8zsBM4CewOPGmJesyqKcy8fbi2cHd6NleCP++uUm9h8t5M1bEglr5OfqaKqhKCqEAxscdvrZtu6bJfbrbvsG2ZpzEm4+vdO/oCN4axfn2rDyiGAm8C/gnUrm/wbcBwy2MIOyiIgwYcCFRIcH8sc5axk6NZOZab1p1aSRq6Op+uZEvu2T/T6H9vwDP4Ipsc0PaGzb0fcZZxt7p3k8RLQFLz0KdRbLCoEx5jsRia1i/gHggIhcZVUGZb1r41vQLDSAO97JYujUJUwb3YseLcNcHUu5q4LDZ5zA7b1lKWTsAexDmQRdYNvZd7jc9gWt5vEQ1rrBddd0N2LlpQrtheDTipqGHJaZBORX1TQkIuOAcQBRUVGJ6enptcqTn59PcHBwrda1krvmgupn25tfyt9XFXLkhGF8vD89o6w9/eSu20xzneZ7MpeQvC0E528hJG8rwflbCCw8UDa/0P8CcgNbURDWgfzgC8kLactJvwi32Ok3xL9jamrqKmNMUkXz6sXJYmPMm8CbAElJSaa2Y21kWDROx/ly11xQs2yDBpxg7Kws/rkmlyevvpAxKW3cIldd8shcxtjGxnfsn793LeTtPb1MRFu4MMVhdM14AoKasMkTt9d5sCpXvSgEqn6IDPYn/Y6+3J++mj9/spFdvxXw+FWd8dbupQ1HaWnFQyoX/GabL14Q2RHaXOyw0+92ekhl5Za0ECinCvTzZurNifzls41MX7KNnNzjvDoygUA/PbFX75QUw6FyQyrvzYaTebb5Xr4Q1QU6X23rn9+8h31IZe0wUN9Y2X10NjAAiBSR3cBTgC+AMeZ1EWkGZAGhQKmIPAB0McYctSqTqhveXsJT13SlZXgjnvlsI6PeWsa00UlEBtdi+F1VN4pP2HrqOO7096+HYvt1rH0CbZ/s40c6dNfsDD7aZbghsLLX0KhzzN8H6CWwGrDb+rchOjyQ+9NXM3RKJjPH9KLtBe53As7jnDxmH1L51LVxs21F4NSQyv6htk/4SWMdhlRuV2+GVFY1p39ZZanLujZj9h19uX1WFkOnZvLWrUn0io0494rKOQqPwN5sYnb9B+a9bx9S+afTQyo3amLb0Sffc3oIhno+pLKqOS0EynIJrcKZf1cKaTNXcNO05bw8PJ5r4lu4OlbDc+zXs0/iHt4GQDuAkBa2nX2X605/0g+Ndovumsq1tBCoOtGqSSPmTUhm3DuruHf2anYfLmD8xW0R3QnVnDG2rpnld/pHc04vE9batqNPuBma92DJtnxSfj/YZZGVe9NCoOpMWCM/3hnbm4fmZvPXLzex+/Bx/nxtV3y8tRmiUsbA4e1nj6N/7KB9AYHI9tA62aG7ZncIDD/jaYpyMuo6uapHtBCoOhXg680/RvYgJjyQqRlb2JNbwL9u7EmQv74VbUMq/3L2Tv/UkMpePraeOu0vs+/042wXR/fXE/Dq/HjGf1/2HFj0NBcf2Q2rY+CSJyFuhKtTeSwvL+GRyzsREx7InxasZ8QbS5me1ouoUA8aLrikCA5uqmBI5eO2+d7+tu6aXYc6DKncRYdUVpZo+IUgew58ch8UFSBguzTdx/farlbU6WrbNyHFy3YBi7L7XrYTaF7lpzkuJ3qS7Tzd1Kc1LRoHcvf7PzDktSXMGNObjs3q6QVDqvqwUVQA+zeeeeGUAxuh5KRtvl+wrTmn562nd/qRHXRIZVVnqlUIRCQIKDDGlIpIB6AT8IUxpsjSdM6w6GnbP6Kj4kJY+Ljtdl6qKhZy5rQzlpMzikqvggLYEGJfrvxzlS9SUsHzVXI7axmpoOA5LidnvWbsjp3A8mpkkypet/J8qV5efH5lIX//7zpemvoD9w/qSLfo8HNmC8rfAQc2OSx3Ptvt1N+rloW9og8bC+6CldPhxFHbJ/+yIZXD7EMqj7fv9HvYh1TW8yTKdap7RPAdcJGIhANfASuBkcBNVgVzmiO7K583eKqtP3Vpie1n2c3Y/nEdp5UtYxyml1+v1DYWS/lpZyxnznrNYwf2ExTZpIL1ymcz9scnnfK6p5cxlb5mLAZ2WPsnigUmAwjwdfXW6QW276U7VVXFrIIiemqZo3tO7+hPKS2C3cuh3aXQ6Ur7EAzxtguk65GkcjPVLQRijDkuImOBKcaYv4nIGgtzOU/jGNsntLOmt4QeN9Z9ngpszMigqRuOdAiQsXgxAy6+uJKi59zieKzwBH//ahMbcnIZ3jOaoQnNkUoK1Yb16+japXPlhbvCfM4p3GcV0TXvVbzxjIGbPqzbP5hStVDtQiAi/bAdAYy1T6sfo4hd8mTZYXsZ30DbdHVuIvZmC+ubLoKAR9qkMHHeOv4vazdLTQTPDemOn8/Zr33wQCh0G2B5pmrZ9l0lHzZ0BBVVP1T3v/sBYCIw3xizQUTaAostS+VMcSPgmsnQuCUGsR0JXDNZew25KT8fL14aHscDl7Zn7qrd3DZzJUcL3fxU1CVP2j5cONIPG6oeqdYRgTHmW+BbABHxAn41xtxnZTCnihsBcSP41k0vNqHOJCI8cGkHYsIb8ehH2QyfupQZY3rRIizw3Cu7wqkPFYuexhzZjTTWLsqqfqnWEYGIvC8iofbeQ+uBjSLykLXRlKcblhjDrNt6sye3gMGvLWF9zhFXR6pc3Ah4cD3fDlgAD67XIqDqleo2DZ26TsBg4AugDXCLVaGUOiWlXSRzJyTj4yWMfGMpizcfOPdKSqkaqW4h8BURX2yF4GP79wesu+q9Ug46Ngth/t0pxEYGcfusLN5fvtPVkZRqUKpbCN4AtmPr2PGdiLQG9Epiqs5EhQYw585+/K59JI/NX8eHm09SWqqfRZRyhmoVAmPMZGNMtDHmSmOzA0i1OJtSZwjy9+GtW5O4sU8rPttWxP0frOFEccm5V1RKVam6J4sbi8jfRSTLfnsZ29GBUnXKx9uLZwd3Y0QHXz5Zu4dbpq0g9/hJV8dSql6rbtPQdCAPGGG/HQVmWBVKqaqICFe29eOfoxJYsyuXoVMz2XnouKtjKVVvVbcQXGiMecoYs9V++zPQtqoVRGS6iBwQkfWVzBcRmSwiv4hItoj0rGl45dmuiW/Be3f04bdjJxkyZQlrduW6OpJS9VJ1C0GBiPQ/9UBEUoCCKpYHmAlcXsX8K4D29ts4YGo1syhVpldsBB9NSCbI34cb3lzKwg37XB1JqXqnuoVgPPCaiGwXke3Av4A7q1rBGPMd8FsVi1wHvGM/+bwMCBOR5tXMo1SZCy8IZt5dyXRqFsr4d1cx/X/bXB1JqXpFjKl+FzwRCQUwxhwVkQeMMa+eY/lY4FNjTLcK5n0KvGCM+Z/98SLgEWPMWYMLi8g4bEcNREVFJaanp1c7s6P8/HyCg93vsn7umgvcN1tFuU6UGN7MPsGq/SUMau3DqE5+eNXxkM/1aXu5A81VM+eTKzU1dZUxJqnCmcaYWt2AndVYJhZYX8m8T4H+Do8XAUnnes7ExERTW4sXL671ulZy11zGuG+2ynIVl5SaP3+8wbR+5FMz7p2V5viJYrfI5Wqaq2YaYi4gy1SyXz2fsYXP96NWDtDS4XGMfZpStebtJTx5TReeuqYLX23cz6i3lvFr/glXx1LKrZ1PITjfr3V+DNxq7z3UFzhijNl7ns+pFABjUtrw+s2JbNp3lKFTMtlyMN/VkZRyW1UWAhHJE5GjFdzygBbnWHc2sBToKCK7RWSsiIwXkfH2RT4HtgK/AG8Bd53/r6PUaZd1bcbsO/py7EQx10/NZOX2qvouKOW5qrwegTEmpLZPbIwZdY75Bri7ts+vVHUktApn/l0ppM1cwU3TlvPy8Hiuia/yM4xSHsf66w8q5WKtmjRi3oRkesSEce/s1UzN2HKqg4JSCi0EykOENfLjnbG9uSa+BX/9chNPLFhPcUmpq2Mp5Raqe/F6peq9AF9v/jGyBzHhgUzN2MKe3AL+dWNPgvz130B5Nj0iUB7Fy0t45PJOPDekO9/9/Csj3ljK/qOFro6llEtpIVAe6cY+rZg2Ooltvx5jyGtL2Lwvz9WRlHIZLQTKY6V2bMqcO/tRXGoYNjWTJb/86upISrmEFgLl0bpFN2b+3Sm0CAtk9PQVzF2129WRlKpzWgiUx4sOC+TDCf3o27YJf/xwLa9+/ZN2L1UeRQuBUkBogC/T03oxLDGGV7/+mYfmZnOyWLuXKs+g/eaUsvPz8eLFYXG0DG/EK1//xN4jBUy9OZHQAF9XR1PKUnpEoJQDEeH+S9vz8vB4lm/9jWFTM8nJPdfF+JSq37QQKFWB6xNjeOe23uw9UsiQ15awPueIqyMpZRktBEpVIrldJB9NSMbX24sRbyxl8aYDro6klCW0EChVhQ5RIcy/K5m2FwRx+ztZvLd8h6sjKeV0WgiUOoemoQF8MK4fv2sfyePz1/PCF5soLdXuparh0EKgVDUE+fvw1q1J3NSnFa9/u4X7P1hDYVGJq2Mp5RTafVSpavLx9uIvg7vRMqIRL3yxiX1HCnjr1iTCGvm5OppS50WPCJSqARFh/MUX8q8bE1i7+whDp2ay89BxV8dS6rzoEYFStXB1XAuiQgO4450shkxZwq3JrZmzcjc5uQVEL/uGhy7ryOCEaFfHVKpaLD0iEJHLRWSziPwiIo9WML+1iCwSkWwRyRCRGCvzKOVMvWIjmDchGYPhlf/+XPbFs5zcAibOW8eC1TkuTqhU9VhWCETEG3gNuALoAowSkS7lFnsJeMcYEwc8DTxvVR6lrND2gmD8fLzPml5QVMKLCze7IJFSNWflEUFv4BdjzFZjzEkgHbiu3DJdgG/s9xdXMF8pt7f/SMVXOMvJLeBEsfYsUu7PykIQDexyeLzbPs3RWmCo/f4QIEREmliYSSmnaxEWWOm85Oe/4eWvNrOvkmKhlDsQq8ZdF5FhwOXGmNvtj28B+hhj7nFYpgXwL6AN8B1wPdDNGJNb7rnGAeMAoqKiEtPT02uVKT8/n+Dg4FqtayV3zQXum82dcmXuKWLm+pOcdBi12s8LBrX2Yc8xw5oDJXgJJEZ5M6i1L+3CvBCROs3oTtvLkeaqmfPJlZqausoYk1TRPCt7DeUALR0ex9inlTHG7MF+RCAiwcD15YuAfbk3gTcBkpKSzIABA2oVKCMjg9quayV3zQXum82dcg0AuqzO4cWFm229hsICz+g1tPPQcd5Zup0PsnaxYl8h3aJDSUtuw9VxzQnwPfv8ghXcaXs50lw1Y1UuKwvBSqC9iLTBVgBuAG50XEBEIoHfjDGlwERguoV5lLLM4IRoBidEV/iP2qpJI564ugsPDurA/NU5zMrczh8/XMtzn//Ijb1bcXPf1jRrHOCa4Eph4TkCY0wxcA+wEPgRmGOM2SAiT4vItfbFBgCbReQnIAp41qo8SrlakL8PN/dtzVcP/o73b+9DYutwXsv4hZS/fsPd7/9A1vbf9BKZyiUs/UKZMeZz4PNy0550uD8XmGtlBqXcjYiQ3C6S5HaR7PrtOP9etoP0FTv5LHsvXVuEkpYcyzXxLeqs2UgpHWJCKRdqGdGIx67szLLHLuG5Id0pKinlobnZJL/wDS8u3MTeI3p1NGU9HWJCKTfQyM+HG/u0YlTvlizdcoiZmduZmrGF17/dyuVdmzE6OZZeseF13ttIeQYtBEq5kfLNRu8u28HsFTv5bN1eujQPJS0llmu12Ug5mTYNKeWmWkY0YqJDs1FJqeHhudn0e34Rf/tyE3tytdlIOYceESjl5s5oNtp6iFmZ23n92y288d1WLusaRVpyG202UudFC4FS9YSIkHxhJMkXnm42Sl+5i8/X7aNz81DGJMdybQ9tNlI1p01DStVDZc1GEy/h+aHdMcbw8Ee2ZqO/frmpbEhspapDjwiUqscC/bwZ1bsVN/RqybKtvzEzcxtvfLuFN77dwmVdm5GWHEvvNhGujqncnBYCpRoAEaHfhU3od2ETdh+2fUntg5W7+GL9Pjo1C6FfZBF9i0q02UhVSJuGlGpgYsIbMfGKzix99BL+en13AGasP0nf5xfxwhfabKTOpkcESjVQgX7ejOzVihFJLXlj/jesPR7Gm99t4c3vtvD7Ls1IS4mlT5sI7W2ktBAo1dCJCJ0ivBk/NJGc3IKyL6l9ucHWbJSWHMt1PaIJ9NNmI0+lTUNKeZDosEAeubwTyybamo1EhEfnraPfC4t4/osf2X34uKsjKhfQIwKlPFCA7+lmoxXbfmPW0u1M+34bb323lUFdbF9S69tWm408hRYCpTyYiNCnbRP6tG1S1myUvmInCzfsp2NUCGkpsQzWZqMGT5uGlFLA6WajpRMv4W/Xx+HlJUyct46+zy/i+c9/ZNdv2mzUUOkRgVLqDAG+3ozo1ZLhSTGs3H6YmZnbmPa/bbz1/VYu7RxFWkos/do20WajBkQLgVKqQiJC7zYR9G4TwR6H3kZfbbQ1G41OjmVwQgsa+elupL7TpiGl1Dm1CAvk4VPNRsPi8PYSHpu/jr7PLeI5bTaq97SUK6WqLcDXmxFJLRmeGEPWjsPMzNzO2//bxrTvt3JJ5yjGJMfS70JtNqpvLC0EInI58A/AG5hmjHmh3PxWwCwgzL7Mo/YL3iul3JiI0Cs2gl6xEew9UsB7y3by/oqd/HfjfjpEBTM6OZYhCdHabFRPWNY0JCLewGvAFUAXYJSIdCm32BPAHGNMAnADMMWqPEopazRvHMgfL+tI5qMDeXFYHL7eXjw+fz19n1vEs59t1GajesDKct0b+MUYsxVARNKB64CNDssYINR+vzGwx8I8SikLBfh6MzypJcMSY1hlbzaavmQ70/63jUs6RTEmJZZkbTZyS1YWgmhgl8Pj3UCfcstMAr4SkXuBIOBSC/MopeqAiJAUG0GSQ7PR7BU7+frH/bRvams2GtpTm43ciRhjrHlikWHA5caY2+2PbwH6GGPucVjmD/YML4tIP+BtoJsxprTcc40DxgFERUUlpqen1ypTfn4+wcHBtVrXSu6aC9w3m+aqGVfnOlliWLGvmP/uKGbH0VICfeB3MT70iywiNlK3V3WdT67U1NRVxpikCmcaYyy5Af2AhQ6PJwITyy2zAWjp8Hgr0LSq501MTDS1tXjx4lqvayV3zWWM+2bTXDXjLrlKS0tN1vZD5u73VpkLJ35mYh/51IyducJ8/9NBU1pa6up4Zdxle5V3PrmALFPJftXKY7OVQHsRaQPkYDsZfGO5ZXYClwAzRaQzEAActDCTUsqFRITE1hEkto5g35FCnp3zHZk7c/n6x+W0O9VslBBNkL82G9Uly3oNGWOKgXuAhcCP2HoHbRCRp0XkWvti/wfcISJrgdlAmr1yKaUauGaNA7i+vR9LHh3Iy8PjCfT15k8L1tP3+UU88+lGdhw65uqIHsPSsmts3wn4vNy0Jx3ubwRSrMyglHJvAb7eXJ8Yw9Ce0fywM5eZmduZlbmd6Uu2MbBjU9JSYunfLlJ7G1lIj7+UUm7B1mwUTmLrcPZf1Zn3lu3g/RU7ueXtFVx4QRBpybEM7RmjzUYW0LGGlFJuJyo0gD/8viNLHh3I30fEE+Tvw5/+s4G+zy3i6U82sv1XbTZyJi2tSim35e/jzdCeMQxJiGb1rlxmLtnOO0u3MyNzG6kdm5KWHMtF7bXZ6HxpIVBKuT0RoWercHq2Cufxqzrz3vKdvL98J7dOtzUbjbY3GwVrs1GtaNOQUqpeiQoN4A+DOrDk0VReGRlPsL8PT/5nA/2eW8SfP9mgzUa1oOVTKVUv+ft4MyQhhiEJMazeaRvb6N1lO5iZuZ0BHS4gLaUNF7WLxMtLm43ORQuBUqreS2gVTkKrcB6/0tZs9N7ynYyevoK2kbZmo+sTtdmoKto0pJRqMJqGBvDgoA5kPjqQV0f2IDTQl6c+tvU2mvTxBrZps1GFtEQqpRocPx8vBidEMzghmtU7DzMrczvvLbc3G3W8gLTkWH7X/gJtNrLTQqCUatBONRs9dlVn3rc3G6XNWEmbyCBG92vN9YkxhAT4ujqmS2nTkFLKIzQNCeCBSzuw5JGB/OOGHjQO9GXSJxvp9/w3TPp4A1sP5rs6osvoEYFSyqP4+XhxXY9orusRzZpduWc0G13c4QLSUmLxtLEvtRAopTxWj5Zh9BjZg4lXdmL28l28u3wHY2asJKqRMN5vG8M8pNlIm4aUUh6vaUgA91/avqzZKNhX+PMnG+n73CKe+s96tjTwZiM9IlBKKbtTzUaNc38m/MIezMrczuwVu5i1dIet2Sg5los7NLzeRnpEoJRSFYhvGcbfR/ZgyaMD+cOgDvy49yhjZq5k4MsZTP/fNo4WFrk6otNoIVBKqSpcEOLPfZe053+PDGTyqASaBPvz9Kcb6ffcIp78z3p+OVD/m420aUgpparBz8eLa+NbcG18C7J3266klr5iF+8s3cFF7SMZkxLLgA5N62WzkR4RKKVUDcXFhPH3ET3InDiQ/xvUgZ/253HbzCxSX87g7XrYbKSFQCmlaiky2J977c1G/xyVQGSwP898autt9KcF6/nlQJ6rI1aLpU1DInI58A/AG5hmjHmh3PxXgFT7w0ZAU2NMmJWZlFLK2Xy9vbgmvgXXxLdg3e4jzMzczgcrd/HvZbZmo7TkWFI7um+zkWWFQES8gdeAQcBuYKWIfGyM2XhqGWPMgw7L3wskWJVHKaXqQveYxrw8Ip6JV3YifcVO/r1sB2NnZdG6SSNu6dua4UktaRzoXl9Ss7JpqDfwizFmqzHmJJAOXFfF8qOA2RbmUUqpOhMZ7M89A23NRv+6MYELgv35y2c/0u/5RTyxYJ1bNRtZ2TQUDexyeLwb6FPRgiLSGmgDfGNhHqWUqnO+3l5cHdeCq+NasD7H1mw0J2s37y7byUXtIxndL5bUTk3xdmGzkVg1uJKIDAMuN8bcbn98C9DHGHNPBcs+AsQYY+6t5LnGAeMAoqKiEtPT02uVKT8/n+Dg4FqtayV3zQXum01z1Yzmqhmrcx09afh2VxHf7Czm8AnDBYHCJa18uSjGhyDfygvC+eRKTU1dZYxJqnCmMcaSG9APWOjweCIwsZJlVwPJ1XnexMREU1uLFy+u9bpWctdcxrhvNs1VM5qrZuoq18niEvPp2j1m2NQlpvUjn5pOT3xhHpuXbX7ad9TpuYAsU8l+1cqmoZVAexFpA+QANwA3ll9IRDoB4cBSC7MopZTb8fX24qq45lwV15z1OUeYlbmdD1ft5r3lO+nfLpLRybEM7NSUT9bu4cWFm8nJLSB62Tc8dFlHBidEOy2HZYXAGFMsIvcAC7F1H51ujNkgIk9jq0wf2xe9AUi3VyyllPJI3aIb8+LweCZe2ZnZK3by7rId3PFOFhFBvhwtKKa41LaLzMktYOK8dQBOKwaWfo/AGPM58Hm5aU+WezzJygxKKVWfRAT5cXdqO+78XVu+2rifB9LXlBWBUwqKSnhx4WanFQL9ZrFSSrkhH28vruzenKKS0grn78ktcNpraSFQSik31iIssEbTa0MLgVJKubGHLutIoK/3GdMCfb156LKOTnsNHYZaKaXc2KnzAGW9hsIC60+vIaWUUs4xOCGawQnRZGRkMGDAAKc/vzYNKaWUh9NCoJRSHk4LgVJKeTgtBEop5eG0ECillIezbBhqq4jIQWBHLVePBH51Yhxncddc4L7ZNFfNaK6aaYi5WhtjLqhoRr0rBOdDRLJMZeNxu5C75gL3zaa5akZz1Yyn5dKmIaWU8nBaCJRSysN5WiF409UBKuGuucB9s2mumtFcNeNRuTzqHIFSSqmzedoRgVJKqXIaTCEQkctFZLOI/CIij1Yw319EPrDPXy4isQ7zJtqnbxaRy+o41x9EZKOIZIvIIhFp7TCvRETW2G8fl1/X4lxpInLQ4fVvd5g3WkR+tt9G13GuVxwy/SQiuQ7zrNxe00XkgIisr2S+iMhke+5sEenpMM/K7XWuXDfZ86wTkUwRiXeYt90+fY2IZNVxrgEicsTh7/Wkw7wq3wMW53rIIdN6+3sqwj7Pku0lIi1FZLF9P7BBRO6vYBlr31+VXdW+Pt2wXRN5C9AW8APWAl3KLXMX8Lr9/g3AB/b7XezL+wNt7M/jXYe5UoFG9vsTTuWyP8534fZKA/5VwboRwFb7z3D7/fC6ylVu+XuxXQvb0u1lf+7fAT2B9ZXMvxL4AhCgL7Dc6u1VzVzJp14PuOJULvvj7UCki7bXAODT830PODtXuWWvAb6xensBzYGe9vshwE8V/D9a+v5qKEcEvYFfjDFbjTEngXTgunLLXAfMst+fC1wiImKfnm6MOWGM2Qb8Yn++OslljFlsjDluf7gMiHHSa59XripcBvzXGPObMeYw8F/gchflGgXMdtJrV8kY8x3wWxWLXAe8Y2yWAWEi0hxrt9c5cxljMu2vC3X3/qrO9qrM+bw3nZ2rTt5fxpi9xpgf7PfzgB+B8hcbsPT91VAKQTSwy+Hxbs7ekGXLGGOKgSNAk2qua2UuR2OxVf1TAkQkS0SWichgJ2WqSa7r7Yehc0WkZQ3XtTIX9ia0NsA3DpOt2l7VUVl2K7dXTZV/fxngKxFZJSLjXJCnn4isFZEvRKSrfZpbbC8RaYRth/qRw2TLt5fYmqwTgOXlZln6/tIL07gJEbkZSAIudpjc2hiTIyJtgW9EZJ0xZksdRfoEmG2MOSEid2I7mhpYR69dHTcAc40xJQ7TXLm93JqIpGIrBP0dJve3b6+mwH9FZJP9E3Nd+AHb3ytfRK4EFgDt6+i1q+MaYIkxxvHowdLtJSLB2ArPA8aYo8563upoKEcEOUBLh8cx9mkVLiMiPkBj4FA117UyFyJyKfA4cK0x5sSp6caYHPvPrUAGtk8KdZLLGHPIIcs0ILG661qZy8ENlDtst3B7VUdl2a3cXtUiInHY/obXGWMOnZrusL0OAPNxXpPoORljjhpj8u33Pwd8RSQSN9hedlW9v5y+vUTEF1sReM8YM6+CRax9fzn7xIcrbtiObLZiayo4dYKpa7ll7ubMk8Vz7Pe7cubJ4q0472RxdXIlYDs51r7c9HDA334/EvgZJ500q2au5g73hwDLzOmTU9vs+cLt9yPqKpd9uU7YTtxJXWwvh9eIpfKTn1dx5sm8FVZvr2rmaoXtvFdyuelBQIjD/Uzg8jrM1ezU3w/bDnWnfdtV6z1gVS77/MbYziME1cX2sv/e7wCvVrGMpe8vp21cV9+wnVX/CdtO9XH7tKexfcoGCAA+tP9TrADaOqz7uH29zcAVdZzra2A/sMZ++9g+PRlYZ/9HWAeMreNczwMb7K+/GOjksO5t9u34CzCmLnPZH08CXii3ntXbazawFyjC1g47FhgPjLfPF+A1e+51QFIdba9z5ZoGHHZ4f2XZp7e1b6u19r/z43Wc6x6H99cyHApVRe+BusplXyYNWwcSx/Us217YmusMkO3wd7qyLt9f+s1ipZTycA3lHIFSSqla0kKglFIeTguBUkp5OC0ESinl4bQQKKWUh9NCoJRdudFL1zhz5EsRia1sxEulXE2HmFDqtAJjTA9Xh1CqrukRgVLnYB+H/m/2sehXiEg7+/RYEflGTl9LopV9epSIzLcPqLZWRJLtT+UtIm/Zx5z/SkQC7cvfJ6evSZHuol9TeTAtBEqdFliuaWikw7wjxpjuwL+AV+3T/gnMMsbEAe8Bk+3TJwPfGmPisY19v8E+vT3wmjGmK5ALXG+f/iiQYH+e8db8akpVTr9ZrJSdiOQbY4IrmL4dGGiM2WofHGyfMaaJiPyKbUymIvv0vcaYSBE5CMQYhwEE7cML/9cY097++BHA1xjzFxH5EsjHNgLnAmMfjE2puqJHBEpVj6nkfk2ccLhfwulzdFdhG0emJ7DSPjquUnVGC4FS1TPS4edS+/1MbCPZAtwEfG+/vwjbZUcREW8RaVzZk4qIF9DSGLMYeATbyJdnHZUoZSX95KHUaYEissbh8ZfGmFNdSMNFJBvbp/pR9mn3AjNE5CHgIDDGPv1+4E0RGYvtk/8EbCNeVsQbeNdeLASYbIzJddLvo1S16DkCpc7Bfo4gyRjzq6uzKGUFbRpSSikPp0cESinl4fSIQCmlPJwWAqWU8nBaCJRSysNpIVBKKQ+nhUAppTycFgKllPJw/w+feON2tWsw2AAAAABJRU5ErkJggg==\n"},"metadata":{"needs_background":"light"}}]},{"cell_type":"markdown","source":["**Save the model**\n","\n","After training, i save the model parameters in a file. I make the evaluations in different notebooks."],"metadata":{"id":"WHGtr7vI8hAN"}},{"cell_type":"code","source":["torch.save(model.state_dict(), \"model.pt\")"],"metadata":{"id":"4NYOtGL-mMod","execution":{"iopub.status.busy":"2022-03-07T02:18:09.148030Z","iopub.execute_input":"2022-03-07T02:18:09.148857Z","iopub.status.idle":"2022-03-07T02:18:09.882482Z","shell.execute_reply.started":"2022-03-07T02:18:09.148813Z","shell.execute_reply":"2022-03-07T02:18:09.881702Z"},"trusted":true},"execution_count":null,"outputs":[]}]}