RNNbase_Predict_Generation.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PSLkT0qL3jgl"
      },
      "source": [
        "# HW4P1: Language Modelling\n",
        "\n",
        "Welcome to the final part 1 hw of this course. This is the only part 1 in which you have PyTorch training (Yay). You will be working on training language models and evaluating them on the task of prediction and generation.<br>\n",
        "Note: A major change which we have made this semester is that we have made the model which you will be coding in this HW very similar to the Speller module from HW4P2. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "cell_id": "95e48c7693e34a389da49dcb6e448e0c",
        "deepnote_cell_type": "markdown",
        "id": "EB2bOV3bzYLR"
      },
      "source": [
        "# Get modules and datasets"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "r4_-qG9rSULt",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d3474c58-8129-4a19-8cf7-a6d31e437abb"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: torchsummaryX in /usr/local/lib/python3.9/dist-packages (1.3.0)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.9/dist-packages (from torchsummaryX) (1.5.3)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.9/dist-packages (from torchsummaryX) (1.22.4)\n",
            "Requirement already satisfied: torch in /usr/local/lib/python3.9/dist-packages (from torchsummaryX) (2.0.0+cu118)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.9/dist-packages (from pandas->torchsummaryX) (2022.7.1)\n",
            "Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.9/dist-packages (from pandas->torchsummaryX) (2.8.2)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.9/dist-packages (from torch->torchsummaryX) (1.11.1)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.9/dist-packages (from torch->torchsummaryX) (3.1.2)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.9/dist-packages (from torch->torchsummaryX) (3.11.0)\n",
            "Requirement already satisfied: triton==2.0.0 in /usr/local/lib/python3.9/dist-packages (from torch->torchsummaryX) (2.0.0)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.9/dist-packages (from torch->torchsummaryX) (3.1)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.9/dist-packages (from torch->torchsummaryX) (4.5.0)\n",
            "Requirement already satisfied: lit in /usr/local/lib/python3.9/dist-packages (from triton==2.0.0->torch->torchsummaryX) (16.0.1)\n",
            "Requirement already satisfied: cmake in /usr/local/lib/python3.9/dist-packages (from triton==2.0.0->torch->torchsummaryX) (3.25.2)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.9/dist-packages (from python-dateutil>=2.8.1->pandas->torchsummaryX) (1.16.0)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.9/dist-packages (from jinja2->torch->torchsummaryX) (2.1.2)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.9/dist-packages (from sympy->torch->torchsummaryX) (1.3.0)\n"
          ]
        }
      ],
      "source": [
        "!pip install torchsummaryX"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# TODO: Import drive if you are using Colab\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RAbmezGFzbj5",
        "outputId": "b900f44b-29ae-443c-cbf1-97cccd70fb70"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Mounted at /content/drive\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "cell_id": "65e59cd2e6514d9594258167a6a0f6db",
        "deepnote_cell_type": "markdown",
        "id": "INh9p3v3zbF_"
      },
      "source": [
        "# Imports"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "cell_id": "03bf3bd639a048f098d5febc42e2baff",
        "deepnote_cell_type": "code",
        "deepnote_to_be_reexecuted": false,
        "execution_millis": 4,
        "execution_start": 1679856365820,
        "id": "QZNwme4320LW",
        "source_hash": "b7876178",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b0b857a8-30b0-4bca-992c-0ec29184e2bc"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/drive/MyDrive/CMUSpring/11.685\n"
          ]
        }
      ],
      "source": [
        "import sys\n",
        "#sys.path.append(\"HW4/\") # TODO: Add path to handout/\n",
        "%cd \"/content/drive/MyDrive/CMUSpring/11.685/\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kEDh9f44wkuP"
      },
      "outputs": [],
      "source": [
        "#!pip install -r requirements.txt\n",
        "#!pip install matplotlib\n",
        "#!pip install tqdm"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "cell_id": "b48a9e95f26c4d2e89d95b1b311cedd5",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "deepnote_cell_type": "code",
        "deepnote_to_be_reexecuted": false,
        "execution": {
          "iopub.execute_input": "2022-08-10T14:02:09.992480Z",
          "iopub.status.busy": "2022-08-10T14:02:09.987693Z",
          "iopub.status.idle": "2022-08-10T14:02:12.872562Z",
          "shell.execute_reply": "2022-08-10T14:02:12.870819Z",
          "shell.execute_reply.started": "2022-08-10T14:02:09.991351Z"
        },
        "execution_millis": 2669,
        "execution_start": 1679856365830,
        "id": "oxiZ42B4SwQ-",
        "outputId": "d25e7b23-27e1-49db-fdb6-36b9fdc0bf16",
        "source_hash": "ec149d26",
        "tags": []
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Device:  cpu\n"
          ]
        }
      ],
      "source": [
        "%matplotlib inline\n",
        "\n",
        "import torch\n",
        "from torch.autograd import Variable\n",
        "import os\n",
        "\n",
        "import time \n",
        "import numpy as np\n",
        "from matplotlib import pyplot as plt\n",
        "from tqdm.notebook import tqdm\n",
        "import torchsummaryX\n",
        "\n",
        "# Importing necessary modules from hw4\n",
        "# from hw4.tests_hw4 import test_prediction, test_generation\n",
        "from hw4.tests_hw4 import test_prediction, test_generation\n",
        "DEVICE = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
        "print(\"Device: \", DEVICE)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "cell_id": "a4ff875589ee46da8f749a7e5088a3ef",
        "deepnote_cell_type": "markdown",
        "id": "u-R794-0zc9V"
      },
      "source": [
        "# Load datasets"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HU4e_6l0Whda",
        "outputId": "c69cf119-67cf-4587-a187-a62e51898952"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Vocab length:  33280\n",
            "['!' '\"' '#' ... '～' '<sos>' '<eos>']\n"
          ]
        }
      ],
      "source": [
        "# Loading the vocabulary. Try printing and see\n",
        "VOCAB       = np.load('dataset/vocab.npy') # np.load('dataset/vocab.npy') \n",
        "\n",
        "# We have also included <sos> and <eos> in the vocabulary for you\n",
        "# However in real life, you include it explicitly if not provided\n",
        "SOS_TOKEN   = np.where(VOCAB == '<sos>')[0][0]\n",
        "EOS_TOKEN   = np.where(VOCAB == '<eos>')[0][0]\n",
        "NUM_WORDS   = len(VOCAB) - 2 # Actual number of words in vocabulary\n",
        "\n",
        "print(\"Vocab length: \", len(VOCAB))\n",
        "print(VOCAB)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "LA7SapmyXHr7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7c3e1f33-0ba5-4d13-98ab-3932d095dc1f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(3803,)\n",
            "(3805,)\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "<ipython-input-5-4b3d0bedd49f>:15: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.\n",
            "  dataset = np.array(dataset_soseos)\n"
          ]
        }
      ],
      "source": [
        "# Loding the training dataset. Refer to write up section 2 to understand the structure\n",
        "dataset     = np.load('dataset/wiki.train.npy', allow_pickle=True)\n",
        "print(dataset[0].shape)\n",
        "# The dataset does not have <sos> and <eos> because they are just regular articles. \n",
        "# TODO: Add <sos> and <eos> to every article in the dataset.\n",
        "# Before doing do, try printing the dataset to see if they are words or integers.\n",
        "dataset_soseos = []\n",
        "total_len = 0\n",
        "for d in dataset:\n",
        "    d = np.append(SOS_TOKEN, d)\n",
        "    d = np.append(d, EOS_TOKEN)\n",
        "    dataset_soseos.append(d)\n",
        "    total_len+=len(d)\n",
        "avg_len = total_len/len(dataset_soseos) # 3587\n",
        "dataset = np.array(dataset_soseos)\n",
        "print(dataset[0].shape)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "cell_id": "09f3a2efaeef49ef9f4c0b2b9a614cca",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "deepnote_cell_type": "code",
        "deepnote_to_be_reexecuted": false,
        "execution": {
          "iopub.execute_input": "2022-08-10T14:02:12.888156Z",
          "iopub.status.busy": "2022-08-10T14:02:12.884281Z",
          "iopub.status.idle": "2022-08-10T14:02:12.960590Z",
          "shell.execute_reply": "2022-08-10T14:02:12.958805Z",
          "shell.execute_reply.started": "2022-08-10T14:02:12.888058Z"
        },
        "execution_millis": 46,
        "execution_start": 1679856368507,
        "id": "x5znxQhLSwRC",
        "outputId": "b36f0d5b-550e-4bf8-c0c2-d1fefd7f5b20",
        "source_hash": "42e4c03c",
        "tags": []
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Validation shapes    :  (128, 21) (128,)\n",
            "Test shapes          :  (128, 21)\n"
          ]
        }
      ],
      "source": [
        "# Loading the fixtures for validation and test - prediction\n",
        "fixtures_pred       = np.load('fixtures/prediction.npz')        # validation\n",
        "fixtures_pred_test  = np.load('fixtures/prediction_test.npz')   # test\n",
        "\n",
        "print(\"Validation shapes    : \", fixtures_pred['inp'].shape, fixtures_pred['out'].shape)\n",
        "print(\"Test shapes          : \", fixtures_pred_test['inp'].shape)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Pes7mCr5WdAw",
        "outputId": "8aa2a2d2-78a3-49da-9efd-f36f29024c67"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Validation Gen Shapes    : (32, 21)\n",
            "Test Gen Shapes          : (128, 31)\n"
          ]
        }
      ],
      "source": [
        "# Loading the fixtures for validation and test - generation\n",
        "fixtures_gen        = np.load('fixtures/generation.npy')        # validation\n",
        "fixtures_gen_test   = np.load('fixtures/generation_test.npy')   # test\n",
        "\n",
        "print(\"Validation Gen Shapes    :\", fixtures_gen.shape)\n",
        "print(\"Test Gen Shapes          :\", fixtures_gen_test.shape)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "jO_Qt7O6rL8L",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9dfa86cb-2ecd-4e48-d0b4-ab4bbc138295"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[33278 32885 31353 22124 32747 20202 28955    76 16786 25881 31453 32846\n",
            " 32022 14686 31543 15794    79  9916 25821 31365 32846]\n",
            "[33278  8881 12851 15340  4557  7384    79  1419 30389 15659     1  4428\n",
            "     1    76 22968 31353   802 31356 27358 25821 31353 10590 11533 26805\n",
            "  9139  5843    79  7339 32747 26534 15773]\n"
          ]
        }
      ],
      "source": [
        "# Example Prediction Dev Input and Output\n",
        "# Optional TODO: You can try printing a few samples from the validation set which has both inputs and outputs\n",
        "print(fixtures_gen[0])\n",
        "print(fixtures_gen_test[0])\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "cell_id": "aec0165a3f1245dfa52a0cb80dba2578",
        "deepnote_cell_type": "markdown",
        "id": "dHjYhXAOzkrP"
      },
      "source": [
        "# Custom DataLoader"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "cell_id": "b2e63a7f6dec4a3f98588725a72a8ff2",
        "deepnote_cell_type": "code",
        "deepnote_to_be_reexecuted": false,
        "execution": {
          "iopub.execute_input": "2022-08-10T14:02:13.079390Z",
          "iopub.status.busy": "2022-08-10T14:02:13.078847Z",
          "iopub.status.idle": "2022-08-10T14:02:13.196189Z",
          "shell.execute_reply": "2022-08-10T14:02:13.192167Z",
          "shell.execute_reply.started": "2022-08-10T14:02:13.079324Z"
        },
        "execution_millis": 48,
        "execution_start": 1679856368575,
        "id": "OZNrJ8XvSwRF",
        "source_hash": "a81eaa14",
        "tags": []
      },
      "outputs": [],
      "source": [
        "class DataLoaderForLanguageModeling(torch.utils.data.DataLoader): # Inherit from torch.utils.data.DataLoader\n",
        "    \"\"\"\n",
        "        TODO: Define data loader logic here\n",
        "    \"\"\"\n",
        "    # TODO: You can probably add more parameters as well. Eg. sequence length\n",
        "    def __init__(self, dataset, batch_size, seq_len=10, shuffle= True, drop_last= False): \n",
        "        \n",
        "        # If you remember, these are the standard things which you give while defining a dataloader.\n",
        "        # Now you are just customizing your dataloader\n",
        "        self.dataset    = dataset\n",
        "        self.batch_size = batch_size\n",
        "        self.seq_len    = seq_len # any fixed length is Okay according to piazza\n",
        "        self.shuffle    = shuffle\n",
        "        self.drop_last  = drop_last\n",
        "\n",
        "    def __len__(self):\n",
        "        # What output do you get when you print len(loader)? You get the number of batches\n",
        "        # Your dataset has (579, ) articles and each article has a specified amount of words.\n",
        "        # You concatenate the dataset and then batch parts of it according to the sequence length\n",
        "        # TODO: return the number of batches\n",
        "        # If you are using variable sequence_length, the length might not be fixed \n",
        "        concat_data = np.concatenate(self.dataset)\n",
        "        conncat_batch_size =  self.batch_size*self.seq_len\n",
        "        # subtract 1 to handle the case when len(concat_data) could exactly be divided\n",
        "        num_batches = (len(concat_data)-1) // conncat_batch_size\n",
        "\n",
        "        \n",
        "        return num_batches if self.drop_last else num_batches+1\n",
        "\n",
        "    def __iter__(self):\n",
        "        # TODOs: \n",
        "        # 1. Shuffle data if shuffle is True\n",
        "        # 2. Concatenate articles and drop extra words\n",
        "        # 3. Divide the concetenated dataset into inputs and targets. How do they vary? \n",
        "        # 4. Reshape the inputs and targets into batches (think about the final shape)\n",
        "        # 5. Loop though the batches and yield the input and target according to the sequence length\n",
        "\n",
        "        if self.shuffle:\n",
        "            self.dataset = np.random.permutation(self.dataset)\n",
        "        concat_data = np.concatenate(self.dataset)\n",
        "        x = torch.LongTensor(concat_data[:-1])\n",
        "        y = torch.LongTensor(concat_data[1:])\n",
        "        conncat_batch_size = self.batch_size*self.seq_len\n",
        "        num_batches = len(x) // conncat_batch_size \n",
        " \n",
        "        batch_idx = 0\n",
        "        if self.drop_last:\n",
        "            # drop the last batch if drop_last\n",
        "            x = x[:num_batches*conncat_batch_size]\n",
        "            y = y[:num_batches*conncat_batch_size]\n",
        "            \n",
        "        # reshape to be batch_idx slice\n",
        "        x = np.reshape(x, (num_batches, self.batch_size, self.seq_len))\n",
        "        y = np.reshape(y, (num_batches, self.batch_size, self.seq_len))\n",
        "\n",
        "        while batch_idx < num_batches:\n",
        "\n",
        "            x_i = x[batch_idx]\n",
        "            y_i = y[batch_idx]\n",
        "            batch_idx += 1\n",
        "            yield x_i, y_i\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "cell_id": "773573c8374048d4bcb5a67b905ee2e0",
        "deepnote_cell_type": "code",
        "deepnote_to_be_reexecuted": false,
        "execution_millis": 3,
        "execution_start": 1679856368714,
        "id": "fBZSzmy10M9M",
        "source_hash": "27952b8c",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "a4ce5cee-a0b3-4ce0-c5af-163dc27cbb7c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.Size([32, 10]) torch.Size([32, 10])\n",
            "x:  ['<sos>', '=', 'Plum', 'cake', '=', '<eol>', 'Plum', 'cake', 'refers', 'to']\n",
            "y:  ['=', 'Plum', 'cake', '=', '<eol>', 'Plum', 'cake', 'refers', 'to', 'a']\n"
          ]
        }
      ],
      "source": [
        "# Some sanity checks\n",
        "\n",
        "dl = DataLoaderForLanguageModeling(\n",
        "    dataset     = dataset, \n",
        "    batch_size  = 32, \n",
        "    shuffle     = True, \n",
        "    drop_last   = True\n",
        "    # Input Extra parameters here if needed\n",
        ")\n",
        "\n",
        "inputs, targets = next(iter(dl))\n",
        "print(inputs.shape, targets.shape)\n",
        "\n",
        "for x, y in dl:\n",
        "    print(\"x: \", [VOCAB[i] for i in x[0, :]])\n",
        "    print(\"y: \", [VOCAB[i] for i in y[0, :]])\n",
        "    break"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "cell_id": "0e75c3c3318d481aa99230d81eb68c13",
        "deepnote_cell_type": "markdown",
        "id": "WcWU0YlnzmVM"
      },
      "source": [
        "# LanguageModel"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "class LockedDropout(torch.nn.Module):\n",
        "    def __init__(self, dropout=0.35):\n",
        "        super().__init__()\n",
        "        self.dropout = dropout\n",
        "\n",
        "    def forward(self, x):\n",
        "        if not self.training or not self.dropout:\n",
        "            return x\n",
        "        m = x.data.new(1, x.size(1), x.size(2)).bernoulli_(1 - self.dropout)\n",
        "        mask = Variable(m, requires_grad=False) / (1 - self.dropout)\n",
        "        mask = mask.expand_as(x)\n",
        "        return mask * x\n"
      ],
      "metadata": {
        "id": "6mW6YtQDhfE4"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {
        "id": "cebwoorWttWe"
      },
      "outputs": [],
      "source": [
        "# Here comes the main portion of this HW.\n",
        "# You can do this with a regular LSTM similar to HW3P2. \n",
        "# However, using LSTMCells will make this Language model very similar to the decoder in HW4P2 and we recommend you use that for writing resuable code.\n",
        "\n",
        "class LanguageModel(torch.nn.Module):\n",
        "\n",
        "    def __init__(self, vocab_size, embed_size=256, hidden_size=256): # TODO: Add more parameters if you want\n",
        "        super().__init__()\n",
        "\n",
        "        # For all the layers which you will define, please read the documentation thoroughly before implementation\n",
        "        self.token_embedding    = torch.nn.Embedding(vocab_size, embed_size) # TODO: Define a PyTorch embedding layer \n",
        "\n",
        "        self.lstm_cells         = torch.nn.Sequential(\n",
        "            torch.nn.LSTMCell(embed_size, hidden_size*2), \n",
        "            torch.nn.LSTMCell(hidden_size*2, hidden_size)\n",
        "            # TODO: Enter the parameters for the LSTMCells\n",
        "            # You can add multiple LSTMCells too if you want\n",
        "        )\n",
        "        self.hidden_size_dim = [hidden_size*2, hidden_size]\n",
        "        self.lockdropout = LockedDropout()\n",
        "\n",
        "        self.token_probability  = torch.nn.Linear(hidden_size, vocab_size) # TODO: Define the parameters\n",
        "\n",
        "        # Optional TODO: Weight Tying. You just need to make the embedding layer weights equal to the Linear layer weight. \n",
        "        self.token_embedding.weight = self.token_probability.weight      \n",
        "        # So the basic pipline is:\n",
        "        # word -> embedding -> lstm -> projection (linear) to get  probability distribution\n",
        "        # And this is happening across all time steps\n",
        "\n",
        "    def rnn_step(self, embedding, hidden_states_list):\n",
        "        for i in range(len(self.lstm_cells)):\n",
        "          hidden_states_list[i] = self.lstm_cells[i](embedding, hidden_states_list[i])\n",
        "          embedding = hidden_states_list[i][0]\n",
        "\n",
        "        return embedding, hidden_states_list\n",
        "        \"\"\"\n",
        "        new_hidden_list = []\n",
        "        batch_size = embedding.shape[0]\n",
        "        for i in range(len(self.lstm_cells)):\n",
        "            if hidden_states_list[i] is None:\n",
        "                h0 = torch.zeros(batch_size, self.hidden_size_dim[i]).requires_grad_()\n",
        "                c0 = torch.zeros(batch_size, self.hidden_size_dim[i]).requires_grad_()\n",
        "            else:\n",
        "                h0, c0 = hidden_states_list[i]\n",
        "            # TODO: Forward pass through each LSTMCell\n",
        "            hi, ci = self.lstm_cells[i](embedding, (h0, c0))\n",
        "            embedding = hi\n",
        "            new_hidden_list.append((hi, ci))\n",
        "            \n",
        "        return embedding, new_hidden_list\n",
        "        \"\"\"\n",
        "\n",
        "    def predict(self, x):\n",
        "        # Refer to Section 1.3.1 to understand this function\n",
        "        if not torch.is_tensor(x):\n",
        "            x = torch.tensor(x).long().to(DEVICE)\n",
        "\n",
        "        with torch.inference_mode():\n",
        "            # TODO: Pass the input sequence through the model \n",
        "            token_prob_dist, hidden_states_list  = self.forward(x)\n",
        "            # and return the probability distribution of the last timestep\n",
        "\n",
        "            return token_prob_dist[:, -1]\n",
        "\n",
        "    def generate(self, x, timesteps): \n",
        "        # Refer to section 1.3.2 to understand this function\n",
        "        # Important Note: We do not draw <eos> from the distribution unlike the writeup\n",
        "        if not torch.is_tensor(x):\n",
        "            x = torch.tensor(x).long().to(DEVICE)\n",
        "\n",
        "        # TODO: Pass the input sequence through the model \n",
        "        # Obtain the probability distribution and hidden_states_list of the last timestep\n",
        "        \n",
        "        token_prob_dist, hidden_states_list     = self.forward(x)\n",
        "        #next_token                              = torch.argmax(token_prob_dist, dim=2)[:,-1].unsqueeze(1)\n",
        "        next_token                              = torch.unsqueeze(torch.argmax(token_prob_dist[:, -1, :-2], axis=1), axis=1)\n",
        "        # TODO: Draw the next predicted token from the probability distribution ()  \n",
        "\n",
        "        generated_sequence  = [] \n",
        "        with torch.inference_mode():\n",
        "            for t in range(timesteps): # Loop through the timesteps\n",
        "                #   TODO: Pass the next_token and hidden_states_list through the model\n",
        "                token_prob_dist, hidden_states_list     = self.forward(next_token, hidden_states_list)\n",
        "                #   TODO: You will get 2 outputs. What is the shape of the probability distribution?\n",
        "                #   TODO: Get the most probable token for the next timestep\n",
        "                next_token                              = torch.unsqueeze(torch.argmax(token_prob_dist[:, -1], axis=1), axis=1)     \n",
        "                generated_sequence.append(torch.squeeze(next_token))\n",
        "            \n",
        "            generated_sequence = torch.stack(generated_sequence, dim= 1) # keep last timesteps generated words\n",
        "            #generated_sequence = torch.cat(generated_sequence, axis= 1)\n",
        "        return generated_sequence\n",
        "\n",
        "    # We are also having a hidden_states_list parameter because you need that in generation\n",
        "    def forward(self, x, hidden_states_list= None): # train model\n",
        "        # x (Batch, Seq_len)\n",
        "        # Note: you dont have to return the sum of log probabilities according to Pseudocode 1 in the writeup\n",
        "        # However, feel free to calculate and print it if you are curious\n",
        "\n",
        "        batch_size, timesteps   = x.shape \n",
        "\n",
        "        token_prob_distribution = [] # list which will contain probability distributions for all timesteps\n",
        "        # Initializing the hidden states\n",
        "        hidden_states_list      = [None]*len(self.lstm_cells) if hidden_states_list == None else hidden_states_list       \n",
        "\n",
        "        token_embeddings        = self.token_embedding(x) # TODO\n",
        "        token_embeddings        = self.lockdropout(token_embeddings)\n",
        "        # When you get the embeddings of the input x, remember that you get it for all time steps.\n",
        "        # Embedding is just a linear transformation so you can precompute it for all time steps.\n",
        "\n",
        "        for t in range(timesteps): # LSTMCell is for just 1 timestep. Hence you need to loop through the total timesteps\n",
        "\n",
        "            token_embedding_t           = token_embeddings[:, t] # TODO  \n",
        "            # batch, embed size\n",
        "\n",
        "            rnn_out, hidden_states_list = self.rnn_step(token_embedding_t, hidden_states_list) \n",
        "            # TODO (What should you do with the hidden_states_list?)\n",
        "            \n",
        "            token_prob_dist_t           = self.token_probability(rnn_out) # TODO\n",
        "\n",
        "            token_prob_distribution.append(token_prob_dist_t) \n",
        "\n",
        "        token_prob_distribution = torch.stack(token_prob_distribution, dim=1) # TODO: Stack along the timesteps dimension\n",
        "\n",
        "        return token_prob_distribution, hidden_states_list "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "cell_id": "8ed5a6ef54f9446fab752b79c70a0216",
        "deepnote_cell_type": "markdown",
        "id": "TlWF_bpLznup"
      },
      "source": [
        "# Trainer Class"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "cell_id": "8ea986fc372643389d1ab4c445659e9d",
        "deepnote_cell_type": "code",
        "deepnote_to_be_reexecuted": true,
        "execution": {
          "iopub.execute_input": "2022-08-10T14:02:13.440820Z",
          "iopub.status.busy": "2022-08-10T14:02:13.440281Z",
          "iopub.status.idle": "2022-08-10T14:02:13.644455Z",
          "shell.execute_reply": "2022-08-10T14:02:13.642614Z",
          "shell.execute_reply.started": "2022-08-10T14:02:13.440752Z"
        },
        "id": "kIvZOIfjSwRK",
        "source_hash": "451a140f",
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "67a6ae3f-7846-4ad9-ae24-77f1d39a8a0d"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.9/dist-packages/torch/cuda/amp/grad_scaler.py:120: UserWarning: torch.cuda.amp.GradScaler is enabled, but CUDA is not available.  Disabling.\n",
            "  warnings.warn(\"torch.cuda.amp.GradScaler is enabled, but CUDA is not available.  Disabling.\")\n"
          ]
        }
      ],
      "source": [
        "# Unlike all the P2s, we are using a Trainer class for this HW.\n",
        "# Many researchers also use classes like this for training. You may have encountered them in your project as well.\n",
        "# You dont have to complete everything in this class, you only need to complete the train function.\n",
        "# However, its good to go through the code and see what it does. \n",
        "scaler = torch.cuda.amp.GradScaler()\n",
        "\n",
        "class Trainer:\n",
        "    def __init__(self, model, loader, optimizer, criterion, scheduler, max_epochs = 1, run_id= 'exp'):\n",
        "        \"\"\"\n",
        "            Use this class to train your model\n",
        "        \"\"\"\n",
        "        # feel free to add any other parameters here\n",
        "        self.model      = model\n",
        "        self.loader     = loader\n",
        "        self.optimizer  = optimizer\n",
        "        self.criterion  = criterion\n",
        "\n",
        "        self.train_losses           = []\n",
        "        self.val_losses             = []\n",
        "        self.predictions            = []\n",
        "        self.predictions_test       = []\n",
        "        self.generated_logits       = []\n",
        "        self.generated              = []\n",
        "        self.generated_logits_test  = []\n",
        "        self.generated_test         = []\n",
        "        self.epochs                 = 0\n",
        "        self.max_epochs             = max_epochs\n",
        "        self.run_id                 = run_id\n",
        "\n",
        "\n",
        "    def calculate_loss(self, out, target):\n",
        "        # output: (B, T, Vocab_size) - probability distributions\n",
        "        # target: (B, T)\n",
        "        # Read the documentation of CrossEntropyLoss and try to understand how it takes inputs\n",
        "\n",
        "        # Tip: If your target is of shape (B, T) it means that you have B batches with T words. \n",
        "        # Tip: What is the total number of words in this batch? \n",
        "        # Tip: Crossentropy calculates the loss between a label and its probability distribution.\n",
        "\n",
        "        out     = out.view(-1, out.size(2)) # TODO\n",
        "        targets = target.flatten() # TODO\n",
        "        loss    = self.criterion(out, targets)\n",
        "\n",
        "        return loss\n",
        "\n",
        "\n",
        "    def train(self):\n",
        "\n",
        "        self.model.train() # set to training mode\n",
        "        self.model.to(DEVICE)\n",
        "        epoch_loss  = 0\n",
        "        num_batches = 0\n",
        "        batch_bar = tqdm(total=len(self.loader), dynamic_ncols=True, leave=False, position=0, desc='Train') \n",
        "        for batch_num, (inputs, targets) in enumerate(self.loader):\n",
        "            self.optimizer.zero_grad()\n",
        "            inputs, targets = inputs.to(DEVICE), targets.to(DEVICE)\n",
        "            with torch.cuda.amp.autocast(): \n",
        "                out, _ = self.model(inputs)\n",
        "                loss = self.calculate_loss(out, targets)\n",
        "\n",
        "                # TODO: Complete the loop. You should be able to complete this without any helper comments after 3 HWs\n",
        "                # Tip: Mixed precision training\n",
        "                # For loss calculation, use the calculate_loss function. You need to complete it before using.\n",
        "    \n",
        "            epoch_loss += loss.item()\n",
        "            \n",
        "            batch_bar.set_postfix(\n",
        "                loss=\"{:.04f}\".format(float(epoch_loss / (batch_num + 1))),\n",
        "                lr=\"{:.06f}\".format(float(self.optimizer.param_groups[0]['lr'])))\n",
        "            batch_bar.update()\n",
        "\n",
        "            if DEVICE == 'cpu':\n",
        "                loss.backward()\n",
        "                self.optimizer.step()\n",
        "            else:\n",
        "                scaler.scale(loss).backward() # This is a replacement for loss.backward()\n",
        "                scaler.step(self.optimizer) # This is a replacement for optimizer.step()\n",
        "                scaler.update() \n",
        "            del inputs, targets, out, _, loss\n",
        "            #torch.cuda.empty_cache()\n",
        "        batch_bar.close()\n",
        "        epoch_loss = epoch_loss / (batch_num + 1)\n",
        "        self.epochs += 1\n",
        "        print('[TRAIN] \\tEpoch [%d/%d] \\tLoss: %.4f \\tLr: %.6f'\n",
        "                      % (self.epochs, self.max_epochs, epoch_loss, self.optimizer.param_groups[0]['lr']))\n",
        "        self.train_losses.append(epoch_loss)\n",
        "\n",
        "\n",
        "    \n",
        "    def test(self): # Don't change this function\n",
        "\n",
        "        self.model.eval() # set to eval mode\n",
        "        predictions     = model.predict(fixtures_pred['inp']).detach().cpu().numpy() # get predictions\n",
        "        self.predictions.append(predictions)\n",
        "        \n",
        "        generated_logits        = model.generate(fixtures_gen, 10).detach().cpu().numpy() # generated predictions for 10 words\n",
        "        generated_logits_test   = model.generate(fixtures_gen_test, 10).detach().cpu().numpy()\n",
        "\n",
        "        nll             = test_prediction(predictions, fixtures_pred['out'])\n",
        "        generated       = test_generation(fixtures_gen, generated_logits, VOCAB)\n",
        "        generated_test  = test_generation(fixtures_gen_test, generated_logits_test, VOCAB)\n",
        "        self.val_losses.append(nll)\n",
        "        \n",
        "        self.generated.append(generated)\n",
        "        self.generated_test.append(generated_test)\n",
        "        self.generated_logits.append(generated_logits)\n",
        "        self.generated_logits_test.append(generated_logits_test)\n",
        "        \n",
        "        # generate predictions for test data\n",
        "        predictions_test = model.predict(fixtures_pred_test['inp']).detach().cpu().numpy() # get predictions\n",
        "        self.predictions_test.append(predictions_test)\n",
        "            \n",
        "        print('[VAL] \\tEpoch [%d/%d] \\tLoss: %.4f'\n",
        "                      % (self.epochs, self.max_epochs, nll))\n",
        "        return nll\n",
        "\n",
        "    \n",
        "    def save(self): # Don't change this function\n",
        "\n",
        "        model_path = os.path.join('hw4/experiments', self.run_id, 'model-{}.pkl'.format(self.epochs))\n",
        "        torch.save({'state_dict': self.model.state_dict()}, model_path)\n",
        "        np.save(os.path.join('hw4/experiments', self.run_id, 'predictions-{}.npy'.format(self.epochs)), self.predictions[-1])\n",
        "        np.save(os.path.join('hw4/experiments', self.run_id, 'predictions-test-{}.npy'.format(self.epochs)), self.predictions_test[-1])\n",
        "        np.save(os.path.join('hw4/experiments', self.run_id, 'generated_logits-{}.npy'.format(self.epochs)), self.generated_logits[-1])\n",
        "        np.save(os.path.join('hw4/experiments', self.run_id, 'generated_logits-test-{}.npy'.format(self.epochs)), self.generated_logits_test[-1])\n",
        "        \n",
        "        with open(os.path.join('hw4/experiments', self.run_id, 'generated-{}.txt'.format(self.epochs)), 'w') as fw:\n",
        "            fw.write(self.generated[-1])\n",
        "\n",
        "        with open(os.path.join('hw4/experiments', self.run_id, 'generated-{}-test.txt'.format(self.epochs)), 'w') as fw:\n",
        "            fw.write(self.generated_test[-1])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "cell_id": "db5de3ac0c6e48ca9cff0cd79bef7ae8",
        "deepnote_cell_type": "markdown",
        "id": "E6NKG0j8zsv-"
      },
      "source": [
        "# Experiment setup"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 36,
      "metadata": {
        "cell_id": "7fc44ee4771a42f996d0a00d35529fb6",
        "deepnote_cell_type": "code",
        "deepnote_to_be_reexecuted": true,
        "execution": {
          "iopub.execute_input": "2022-08-10T14:02:13.852171Z",
          "iopub.status.busy": "2022-08-10T14:02:13.850633Z",
          "iopub.status.idle": "2022-08-10T14:02:13.927227Z",
          "shell.execute_reply": "2022-08-10T14:02:13.924500Z",
          "shell.execute_reply.started": "2022-08-10T14:02:13.852093Z"
        },
        "id": "TiUrjbEjSwRQ",
        "source_hash": "f7524436",
        "tags": []
      },
      "outputs": [],
      "source": [
        "# TODO: define other hyperparameters here\n",
        "\n",
        "configs = dict(\n",
        "    batch_size  = 512,\n",
        "    num_epochs  = 20, # 10 or 20 epochs should be enough given the model is good\n",
        "\n",
        "    init_lr     = 0.002 # TODO\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 37,
      "metadata": {
        "cell_id": "4aaccf1c32fa480a9a15e8bb8bc4d9e4",
        "deepnote_cell_type": "code",
        "deepnote_to_be_reexecuted": true,
        "execution": {
          "iopub.execute_input": "2022-08-10T14:02:14.110787Z",
          "iopub.status.busy": "2022-08-10T14:02:14.109778Z",
          "iopub.status.idle": "2022-08-10T14:02:14.929087Z",
          "shell.execute_reply": "2022-08-10T14:02:14.925078Z",
          "shell.execute_reply.started": "2022-08-10T14:02:14.110707Z"
        },
        "id": "DbHH6zXTSwRa",
        "source_hash": "2acff566",
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "5f14f623-c117-4f4d-f018-c6e844d13b15"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "LanguageModel(\n",
            "  (token_embedding): Embedding(33280, 256)\n",
            "  (lstm_cells): Sequential(\n",
            "    (0): LSTMCell(256, 512)\n",
            "    (1): LSTMCell(512, 256)\n",
            "  )\n",
            "  (lockdropout): LockedDropout()\n",
            "  (token_probability): Linear(in_features=256, out_features=33280, bias=True)\n",
            ")\n",
            "============================================================================\n",
            "                          Kernel Shape    Output Shape    Params  Mult-Adds\n",
            "Layer                                                                      \n",
            "0_token_embedding         [256, 33280]  [512, 10, 256]  8.51968M   8.51968M\n",
            "1_lockdropout                        -  [512, 10, 256]         -          -\n",
            "2_lstm_cells.LSTMCell_0              -      [512, 512]  1.57696M  1.572864M\n",
            "3_lstm_cells.LSTMCell_1              -      [512, 256]   788.48k   786.432k\n",
            "4_token_probability       [256, 33280]    [512, 33280]  8.55296M   8.51968M\n",
            "5_lstm_cells.LSTMCell_0              -      [512, 512]         -  1.572864M\n",
            "6_lstm_cells.LSTMCell_1              -      [512, 256]         -   786.432k\n",
            "7_token_probability       [256, 33280]    [512, 33280]         -   8.51968M\n",
            "8_lstm_cells.LSTMCell_0              -      [512, 512]         -  1.572864M\n",
            "9_lstm_cells.LSTMCell_1              -      [512, 256]         -   786.432k\n",
            "10_token_probability      [256, 33280]    [512, 33280]         -   8.51968M\n",
            "11_lstm_cells.LSTMCell_0             -      [512, 512]         -  1.572864M\n",
            "12_lstm_cells.LSTMCell_1             -      [512, 256]         -   786.432k\n",
            "13_token_probability      [256, 33280]    [512, 33280]         -   8.51968M\n",
            "14_lstm_cells.LSTMCell_0             -      [512, 512]         -  1.572864M\n",
            "15_lstm_cells.LSTMCell_1             -      [512, 256]         -   786.432k\n",
            "16_token_probability      [256, 33280]    [512, 33280]         -   8.51968M\n",
            "17_lstm_cells.LSTMCell_0             -      [512, 512]         -  1.572864M\n",
            "18_lstm_cells.LSTMCell_1             -      [512, 256]         -   786.432k\n",
            "19_token_probability      [256, 33280]    [512, 33280]         -   8.51968M\n",
            "20_lstm_cells.LSTMCell_0             -      [512, 512]         -  1.572864M\n",
            "21_lstm_cells.LSTMCell_1             -      [512, 256]         -   786.432k\n",
            "22_token_probability      [256, 33280]    [512, 33280]         -   8.51968M\n",
            "23_lstm_cells.LSTMCell_0             -      [512, 512]         -  1.572864M\n",
            "24_lstm_cells.LSTMCell_1             -      [512, 256]         -   786.432k\n",
            "25_token_probability      [256, 33280]    [512, 33280]         -   8.51968M\n",
            "26_lstm_cells.LSTMCell_0             -      [512, 512]         -  1.572864M\n",
            "27_lstm_cells.LSTMCell_1             -      [512, 256]         -   786.432k\n",
            "28_token_probability      [256, 33280]    [512, 33280]         -   8.51968M\n",
            "29_lstm_cells.LSTMCell_0             -      [512, 512]         -  1.572864M\n",
            "30_lstm_cells.LSTMCell_1             -      [512, 256]         -   786.432k\n",
            "31_token_probability      [256, 33280]    [512, 33280]         -   8.51968M\n",
            "----------------------------------------------------------------------------\n",
            "                          Totals\n",
            "Total params           19.43808M\n",
            "Trainable params       19.43808M\n",
            "Non-trainable params         0.0\n",
            "Mult-Adds             117.30944M\n",
            "============================================================================\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.9/dist-packages/torchsummaryX/torchsummaryX.py:101: FutureWarning: The default value of numeric_only in DataFrame.sum is deprecated. In a future version, it will default to False. In addition, specifying 'numeric_only=None' is deprecated. Select only valid columns or specify the value of numeric_only to silence this warning.\n",
            "  df_sum = df.sum()\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                          Kernel Shape    Output Shape     Params  Mult-Adds\n",
              "Layer                                                                       \n",
              "0_token_embedding         [256, 33280]  [512, 10, 256]  8519680.0  8519680.0\n",
              "1_lockdropout                        -  [512, 10, 256]        NaN        NaN\n",
              "2_lstm_cells.LSTMCell_0              -      [512, 512]  1576960.0  1572864.0\n",
              "3_lstm_cells.LSTMCell_1              -      [512, 256]   788480.0   786432.0\n",
              "4_token_probability       [256, 33280]    [512, 33280]  8552960.0  8519680.0\n",
              "5_lstm_cells.LSTMCell_0              -      [512, 512]        NaN  1572864.0\n",
              "6_lstm_cells.LSTMCell_1              -      [512, 256]        NaN   786432.0\n",
              "7_token_probability       [256, 33280]    [512, 33280]        NaN  8519680.0\n",
              "8_lstm_cells.LSTMCell_0              -      [512, 512]        NaN  1572864.0\n",
              "9_lstm_cells.LSTMCell_1              -      [512, 256]        NaN   786432.0\n",
              "10_token_probability      [256, 33280]    [512, 33280]        NaN  8519680.0\n",
              "11_lstm_cells.LSTMCell_0             -      [512, 512]        NaN  1572864.0\n",
              "12_lstm_cells.LSTMCell_1             -      [512, 256]        NaN   786432.0\n",
              "13_token_probability      [256, 33280]    [512, 33280]        NaN  8519680.0\n",
              "14_lstm_cells.LSTMCell_0             -      [512, 512]        NaN  1572864.0\n",
              "15_lstm_cells.LSTMCell_1             -      [512, 256]        NaN   786432.0\n",
              "16_token_probability      [256, 33280]    [512, 33280]        NaN  8519680.0\n",
              "17_lstm_cells.LSTMCell_0             -      [512, 512]        NaN  1572864.0\n",
              "18_lstm_cells.LSTMCell_1             -      [512, 256]        NaN   786432.0\n",
              "19_token_probability      [256, 33280]    [512, 33280]        NaN  8519680.0\n",
              "20_lstm_cells.LSTMCell_0             -      [512, 512]        NaN  1572864.0\n",
              "21_lstm_cells.LSTMCell_1             -      [512, 256]        NaN   786432.0\n",
              "22_token_probability      [256, 33280]    [512, 33280]        NaN  8519680.0\n",
              "23_lstm_cells.LSTMCell_0             -      [512, 512]        NaN  1572864.0\n",
              "24_lstm_cells.LSTMCell_1             -      [512, 256]        NaN   786432.0\n",
              "25_token_probability      [256, 33280]    [512, 33280]        NaN  8519680.0\n",
              "26_lstm_cells.LSTMCell_0             -      [512, 512]        NaN  1572864.0\n",
              "27_lstm_cells.LSTMCell_1             -      [512, 256]        NaN   786432.0\n",
              "28_token_probability      [256, 33280]    [512, 33280]        NaN  8519680.0\n",
              "29_lstm_cells.LSTMCell_0             -      [512, 512]        NaN  1572864.0\n",
              "30_lstm_cells.LSTMCell_1             -      [512, 256]        NaN   786432.0\n",
              "31_token_probability      [256, 33280]    [512, 33280]        NaN  8519680.0"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-4dacafdb-3143-471a-961a-7549bf506793\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Kernel Shape</th>\n",
              "      <th>Output Shape</th>\n",
              "      <th>Params</th>\n",
              "      <th>Mult-Adds</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Layer</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0_token_embedding</th>\n",
              "      <td>[256, 33280]</td>\n",
              "      <td>[512, 10, 256]</td>\n",
              "      <td>8519680.0</td>\n",
              "      <td>8519680.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1_lockdropout</th>\n",
              "      <td>-</td>\n",
              "      <td>[512, 10, 256]</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2_lstm_cells.LSTMCell_0</th>\n",
              "      <td>-</td>\n",
              "      <td>[512, 512]</td>\n",
              "      <td>1576960.0</td>\n",
              "      <td>1572864.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3_lstm_cells.LSTMCell_1</th>\n",
              "      <td>-</td>\n",
              "      <td>[512, 256]</td>\n",
              "      <td>788480.0</td>\n",
              "      <td>786432.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4_token_probability</th>\n",
              "      <td>[256, 33280]</td>\n",
              "      <td>[512, 33280]</td>\n",
              "      <td>8552960.0</td>\n",
              "      <td>8519680.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5_lstm_cells.LSTMCell_0</th>\n",
              "      <td>-</td>\n",
              "      <td>[512, 512]</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1572864.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6_lstm_cells.LSTMCell_1</th>\n",
              "      <td>-</td>\n",
              "      <td>[512, 256]</td>\n",
              "      <td>NaN</td>\n",
              "      <td>786432.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7_token_probability</th>\n",
              "      <td>[256, 33280]</td>\n",
              "      <td>[512, 33280]</td>\n",
              "      <td>NaN</td>\n",
              "      <td>8519680.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8_lstm_cells.LSTMCell_0</th>\n",
              "      <td>-</td>\n",
              "      <td>[512, 512]</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1572864.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9_lstm_cells.LSTMCell_1</th>\n",
              "      <td>-</td>\n",
              "      <td>[512, 256]</td>\n",
              "      <td>NaN</td>\n",
              "      <td>786432.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10_token_probability</th>\n",
              "      <td>[256, 33280]</td>\n",
              "      <td>[512, 33280]</td>\n",
              "      <td>NaN</td>\n",
              "      <td>8519680.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11_lstm_cells.LSTMCell_0</th>\n",
              "      <td>-</td>\n",
              "      <td>[512, 512]</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1572864.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12_lstm_cells.LSTMCell_1</th>\n",
              "      <td>-</td>\n",
              "      <td>[512, 256]</td>\n",
              "      <td>NaN</td>\n",
              "      <td>786432.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13_token_probability</th>\n",
              "      <td>[256, 33280]</td>\n",
              "      <td>[512, 33280]</td>\n",
              "      <td>NaN</td>\n",
              "      <td>8519680.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14_lstm_cells.LSTMCell_0</th>\n",
              "      <td>-</td>\n",
              "      <td>[512, 512]</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1572864.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15_lstm_cells.LSTMCell_1</th>\n",
              "      <td>-</td>\n",
              "      <td>[512, 256]</td>\n",
              "      <td>NaN</td>\n",
              "      <td>786432.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16_token_probability</th>\n",
              "      <td>[256, 33280]</td>\n",
              "      <td>[512, 33280]</td>\n",
              "      <td>NaN</td>\n",
              "      <td>8519680.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17_lstm_cells.LSTMCell_0</th>\n",
              "      <td>-</td>\n",
              "      <td>[512, 512]</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1572864.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18_lstm_cells.LSTMCell_1</th>\n",
              "      <td>-</td>\n",
              "      <td>[512, 256]</td>\n",
              "      <td>NaN</td>\n",
              "      <td>786432.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19_token_probability</th>\n",
              "      <td>[256, 33280]</td>\n",
              "      <td>[512, 33280]</td>\n",
              "      <td>NaN</td>\n",
              "      <td>8519680.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20_lstm_cells.LSTMCell_0</th>\n",
              "      <td>-</td>\n",
              "      <td>[512, 512]</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1572864.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21_lstm_cells.LSTMCell_1</th>\n",
              "      <td>-</td>\n",
              "      <td>[512, 256]</td>\n",
              "      <td>NaN</td>\n",
              "      <td>786432.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22_token_probability</th>\n",
              "      <td>[256, 33280]</td>\n",
              "      <td>[512, 33280]</td>\n",
              "      <td>NaN</td>\n",
              "      <td>8519680.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23_lstm_cells.LSTMCell_0</th>\n",
              "      <td>-</td>\n",
              "      <td>[512, 512]</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1572864.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24_lstm_cells.LSTMCell_1</th>\n",
              "      <td>-</td>\n",
              "      <td>[512, 256]</td>\n",
              "      <td>NaN</td>\n",
              "      <td>786432.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25_token_probability</th>\n",
              "      <td>[256, 33280]</td>\n",
              "      <td>[512, 33280]</td>\n",
              "      <td>NaN</td>\n",
              "      <td>8519680.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26_lstm_cells.LSTMCell_0</th>\n",
              "      <td>-</td>\n",
              "      <td>[512, 512]</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1572864.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>27_lstm_cells.LSTMCell_1</th>\n",
              "      <td>-</td>\n",
              "      <td>[512, 256]</td>\n",
              "      <td>NaN</td>\n",
              "      <td>786432.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28_token_probability</th>\n",
              "      <td>[256, 33280]</td>\n",
              "      <td>[512, 33280]</td>\n",
              "      <td>NaN</td>\n",
              "      <td>8519680.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29_lstm_cells.LSTMCell_0</th>\n",
              "      <td>-</td>\n",
              "      <td>[512, 512]</td>\n",
              "      <td>NaN</td>\n",
              "      <td>1572864.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>30_lstm_cells.LSTMCell_1</th>\n",
              "      <td>-</td>\n",
              "      <td>[512, 256]</td>\n",
              "      <td>NaN</td>\n",
              "      <td>786432.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>31_token_probability</th>\n",
              "      <td>[256, 33280]</td>\n",
              "      <td>[512, 33280]</td>\n",
              "      <td>NaN</td>\n",
              "      <td>8519680.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-4dacafdb-3143-471a-961a-7549bf506793')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-4dacafdb-3143-471a-961a-7549bf506793 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-4dacafdb-3143-471a-961a-7549bf506793');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 37
        }
      ],
      "source": [
        "model       = LanguageModel(NUM_WORDS+2, 256, 256) # TODO: Define the model\n",
        "\n",
        "loader      = DataLoaderForLanguageModeling(\n",
        "    dataset     = dataset, \n",
        "    batch_size  = configs['batch_size'], \n",
        "    shuffle     = True, \n",
        "    drop_last   = True\n",
        "    # Input Extra parameters here if needed\n",
        ")\n",
        "inputs, targets = next(iter(loader))\n",
        "\n",
        "criterion   = torch.nn.CrossEntropyLoss() \n",
        "\n",
        "optimizer   = torch.optim.AdamW(model.parameters(), lr=configs['init_lr'], weight_decay=5e-5) # TODO: Define the optimizer. Adam/AdamW usually works good for this HW\n",
        "\n",
        "# Optional TODO: Use a scheduler if you want\n",
        "scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min', factor=0.5, patience=1, threshold=0.01, verbose=True)\n",
        "\n",
        "print(model)\n",
        "torchsummaryX.summary(model, x = inputs.to(DEVICE))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 38,
      "metadata": {
        "cell_id": "aaff53cf948e44b7b9bd49cbcad0ac58",
        "deepnote_cell_type": "code",
        "deepnote_to_be_reexecuted": true,
        "execution": {
          "iopub.execute_input": "2022-08-10T14:02:13.931258Z",
          "iopub.status.busy": "2022-08-10T14:02:13.930204Z",
          "iopub.status.idle": "2022-08-10T14:02:14.107883Z",
          "shell.execute_reply": "2022-08-10T14:02:14.105987Z",
          "shell.execute_reply.started": "2022-08-10T14:02:13.931185Z"
        },
        "id": "2HCVG5YISwRW",
        "source_hash": "c9f4594a",
        "tags": [],
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "dd1eddd6-b2b9-4983-de3b-30a626d47a6c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving models, predictions, and generated words to ./hw4/experiments/1681491014\n"
          ]
        }
      ],
      "source": [
        "# Dont change this cell\n",
        "\n",
        "run_id = str(int(time.time()))\n",
        "if not os.path.exists('./hw4/experiments'):\n",
        "    os.mkdir('./hw4/experiments')\n",
        "os.mkdir('./hw4/experiments/%s' % run_id)\n",
        "print(\"Saving models, predictions, and generated words to ./hw4/experiments/%s\" % run_id)\n",
        "\n",
        "# The object of the Trainer class takes in everything\n",
        "trainer = Trainer(\n",
        "    model       = model, \n",
        "    loader      = loader, \n",
        "\n",
        "    optimizer   = optimizer,\n",
        "    criterion   = criterion, \n",
        "    \n",
        "    max_epochs  = configs['num_epochs'], \n",
        "    run_id      = run_id, \n",
        "    scheduler   = scheduler\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 39,
      "metadata": {
        "id": "V0dy4CpXJgN2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000,
          "referenced_widgets": [
            "93ee11dbe97549169a13fd54af2d8ee5",
            "6e1ee11b061a4efda575aa5ffbdfdba8",
            "e019d72995ae4bc88bc0f6c9ced39110",
            "8ab5ef90439049c7b445c15c8351897f",
            "2b2403d043c444138af2273277bb0d15",
            "760c0bf783a3425584ecb499fa300c6e",
            "b1da510235cc407e987378ac3043e46e",
            "08f71349eecc4cf9b7e44231c7e6db27",
            "c7c830ce84e94d5195c8a4761f0be418",
            "733ae01dbd8d4c59b499a05479753336",
            "6b038ccf31ca46edad0699f98c1b0cb5",
            "b39c0e91e24c4100a8f79d3ce430384d",
            "4c1e4a8ac5314f3ba61339cc22bcf05f",
            "e97b519d9fbb46e0af06bbb9c1af21a3",
            "326eb55f4a5b4403b322a211c3ed3625",
            "623d5bec94ed44d8b374c0d937e6e8bf",
            "855022913d3f46c394a313dc3bbbffb1",
            "87dfe488a08c409ebf252d9e495e3d3f",
            "8eefef7fe76f4984b07c16882be741ae",
            "2361037cf8394e5d9a8debd564123f11",
            "4023f67468474e2688c2c966e6097f9c",
            "02eeb409d66c4619901b8ee4d100fa41",
            "38a9691c40d6407ab44288c23ff43834",
            "9abe806a1b9d4df7aed868e04d6b8705",
            "45e19ea7f04e4adc97261227d4f6ac3e",
            "65d247ebba67448d956f4affa491d3fa",
            "8ca83d4b103b413fb2c1c4481e8de366",
            "4a42be20ea574b2aadfc2c454e8b66cc",
            "48832de761264093a303e87e99790a2d",
            "c1bdbeee2f204978bce00ca0d6b8de8b",
            "f435a7abb0aa4e3a9cf7860ddc941a06",
            "6635f96c58d04e36bf7f23432720e066",
            "3326e933cd7447479a1142170d5410c7",
            "7a4565f23db34c4e8fcaa0676ffa723a",
            "4bf87b575c1847159090a137dff72066",
            "d8a70235a5734c389cf0eefaeedb442d",
            "d937f515da9d402eae8f0bd44c72125b",
            "c3045f0beed24347b7d8fd9f914b8a18",
            "c5beb7ae4f6347ba861767cdea27a52f",
            "dc903ee667c344ce9d6f27166569ad02",
            "059539cda3b9443d80e7c2f153de983c",
            "32ec551d9d254d44ad282baa738a64b3",
            "c386a23d25f94581834bca85d7bdc8d1",
            "b3a09c2cbc4d4724a6bb9d5bb7ae63c0",
            "bb07bd41fff0481b9c170c8cc0fb4d1b",
            "e27deb279b5146ca980c5a725fa76541",
            "931802f4fbe64de3b9f57ba857e97abd",
            "39f8e59519f2429c806658408eb6ecdc",
            "121938fecaed4e02a9bab874195a2bbe",
            "51dd96681212465e81829e2ae13a3010",
            "af10b57e94b64d16993c4e156b1d0e00",
            "a673851a3c544d738f6bd75baa1d5824",
            "33d6d69541df445192cd4820dbc37743",
            "2b5b8506f1a84ba68b2b67ee05b3907b",
            "4559c9f8031844e9b615533f6e0451ac",
            "28e2ea7ddc784911818e1aabd2e23dd3",
            "ae90f7ec8e9049ea8b9d207acaf8a19a",
            "d632b0eac9884acdbd102654055400e7",
            "c986d44f82724c3a913088c1bae4480c",
            "5b73d8943e954a2cad3304e939af33c7",
            "a0f3c56684e646e79b40c4992305fb91",
            "9f7274943cbb4110ba5231b6a016066f",
            "9749e109aecb42ce856b972ad9cbeb88",
            "8e636da2803d4b948f0b0c23afbe69b0",
            "2db0b30dfba748bc99666cdc19252caf",
            "fead4e0fe003409d82475b68deb86cf2",
            "48f462d7a4a749d28363c5991de793f5",
            "e4a7ba6ef3004501aa63ec5d19e9b75c",
            "798997169cf8436bba6c17ab613efdca",
            "d7e1da29d44d4b24b08db7961244f131",
            "9848631abd4143139a86b6b182a73c84",
            "d7c1d8434d9547efb4201857d632209f",
            "49deff179c10429f9931099aa36c66f0",
            "3e5e22b057fd400d806ae9c7ac7379aa",
            "df1cff61af98433a87edc71930dc2b77",
            "721ea819258c4869a1b245787e488391",
            "627b7e59dd9b41da832b11c514d725ce",
            "1bc3421283cd488cbfb7c9898de3c9d9",
            "d58d21a8bff14863a157c3792872b020",
            "32c3cff66e434df98273286546238567",
            "a3f8aee9bb3c44ae99917b5493b3fa2d",
            "00303b9fdd5242888a4f0ce7d32ef0be",
            "56399144f20a4f179fd992acd727033c",
            "0eb29c2af9564bea86cc9cb381170f5e",
            "f95e0391f058449982dff354199a7445",
            "997a2cef0f05460ca29b2c805f00d3a1",
            "a3e64cc0751043f3b3794fe6e424584c",
            "aaf8e6aeccd241c28e796deb839136c0",
            "99dfc32cc70f45c08fd45afd269b16ca",
            "d8d44fcdf2ff46e9826ce3dde43066d8",
            "4dcb21d623fa4fcda2d8e6e3b4e6acac",
            "b6f2d6465ff2428a88ee78bac1db34f3",
            "9b6c87ad1bd34d68b28ab8ffd9d13f35",
            "c1d5058d103d4376ba48f394632ffe25",
            "d99f679168d54ff899a001a980b68f64",
            "731f486356f0489f82c9d402c9291812",
            "ed81a8feef46408fb6d31620720d200b",
            "b54ea61c38684b29999db3dcb88e265f",
            "15894500fddf4dd7b7aa5adf404de539",
            "270e999344564b279594c506106d95d5",
            "e9d99aad68694b7d92c5cee850a48b96",
            "3c9e259fcd544040886f5567388ecdee",
            "db3e7cb5fcf54f688d6ca4505e420b95",
            "26e14138e615440ca413843afd11a1d5",
            "fdfbeaf191ca48009a2d3b030481d27d",
            "e860218667d14311b5d849cfe1be7761",
            "bb17114fe6734d708c0aef1732130658",
            "7b91ecf6c5c348879965a47ce2a9f37b",
            "bb2c9a401f414d059722576203875b9c",
            "1324940779854f1891135c3a0731380c",
            "fd9bbdaf6c35409196d1178d87e40264",
            "9561207d2a3a441ba6c6360b0c2aad30",
            "2fec87e5a8d04af8919a065834105aa8",
            "08c93e89970f4e81a647e8d6ed68e550",
            "445ed794c95e40c3b31a13f5c2533e7e",
            "4cfa58f903c04988bf3ca887afee8943",
            "6d67253376fb4b3ab8d8d6e4c0cdf99f",
            "e4ba91850f3d426eb4434d7ac6ea0f6b",
            "2e4a82150f4949e9a967c56058e7da8b",
            "6d9be590fd7a4073882a62ad1cbbe36a",
            "6c348830b7004f9db744787650c9311b"
          ]
        },
        "outputId": "be635729-44d8-438b-bd3f-069c518ed326"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "93ee11dbe97549169a13fd54af2d8ee5",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Train:   0%|          | 0/405 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "metadata": {
            "tags": null
          },
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.9/dist-packages/torch/amp/autocast_mode.py:204: UserWarning: User provided device_type of 'cuda', but CUDA is not available. Disabling\n",
            "  warnings.warn('User provided device_type of \\'cuda\\', but CUDA is not available. Disabling')\n"
          ]
        },
        {
          "metadata": {
            "tags": null
          },
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[TRAIN] \tEpoch [1/20] \tLoss: 7.4022 \tLr: 0.002000\n",
            "[VAL] \tEpoch [1/20] \tLoss: 6.3581\n",
            "Saving model, predictions and generated output for epoch 1 with NLL: 6.358064\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "b39c0e91e24c4100a8f79d3ce430384d",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Train:   0%|          | 0/405 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[TRAIN] \tEpoch [2/20] \tLoss: 6.5995 \tLr: 0.002000\n",
            "[VAL] \tEpoch [2/20] \tLoss: 5.7188\n",
            "Saving model, predictions and generated output for epoch 2 with NLL: 5.7187853\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Train:   0%|          | 0/405 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "38a9691c40d6407ab44288c23ff43834"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[TRAIN] \tEpoch [3/20] \tLoss: 6.0736 \tLr: 0.002000\n",
            "[VAL] \tEpoch [3/20] \tLoss: 5.1887\n",
            "Saving model, predictions and generated output for epoch 3 with NLL: 5.1886635\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Train:   0%|          | 0/405 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "7a4565f23db34c4e8fcaa0676ffa723a"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[TRAIN] \tEpoch [4/20] \tLoss: 5.7556 \tLr: 0.002000\n",
            "[VAL] \tEpoch [4/20] \tLoss: 4.9859\n",
            "Saving model, predictions and generated output for epoch 4 with NLL: 4.9859242\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Train:   0%|          | 0/405 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "bb07bd41fff0481b9c170c8cc0fb4d1b"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[TRAIN] \tEpoch [5/20] \tLoss: 5.5400 \tLr: 0.002000\n",
            "[VAL] \tEpoch [5/20] \tLoss: 4.8189\n",
            "Saving model, predictions and generated output for epoch 5 with NLL: 4.8189363\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Train:   0%|          | 0/405 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "28e2ea7ddc784911818e1aabd2e23dd3"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[TRAIN] \tEpoch [6/20] \tLoss: 5.3743 \tLr: 0.002000\n",
            "[VAL] \tEpoch [6/20] \tLoss: 4.6723\n",
            "Saving model, predictions and generated output for epoch 6 with NLL: 4.672324\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Train:   0%|          | 0/405 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "48f462d7a4a749d28363c5991de793f5"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[TRAIN] \tEpoch [7/20] \tLoss: 5.2412 \tLr: 0.002000\n",
            "[VAL] \tEpoch [7/20] \tLoss: 4.6216\n",
            "Saving model, predictions and generated output for epoch 7 with NLL: 4.621583\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Train:   0%|          | 0/405 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "1bc3421283cd488cbfb7c9898de3c9d9"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[TRAIN] \tEpoch [8/20] \tLoss: 5.1257 \tLr: 0.002000\n",
            "[VAL] \tEpoch [8/20] \tLoss: 4.5542\n",
            "Saving model, predictions and generated output for epoch 8 with NLL: 4.5542355\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Train:   0%|          | 0/405 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "99dfc32cc70f45c08fd45afd269b16ca"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[TRAIN] \tEpoch [9/20] \tLoss: 5.0268 \tLr: 0.002000\n",
            "[VAL] \tEpoch [9/20] \tLoss: 4.5685\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Train:   0%|          | 0/405 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "270e999344564b279594c506106d95d5"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[TRAIN] \tEpoch [10/20] \tLoss: 4.9395 \tLr: 0.002000\n",
            "[VAL] \tEpoch [10/20] \tLoss: 4.4737\n",
            "Saving model, predictions and generated output for epoch 10 with NLL: 4.473728\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Train:   0%|          | 0/405 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "fd9bbdaf6c35409196d1178d87e40264"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<timed exec>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-24-157954709ed1>\u001b[0m in \u001b[0;36mtrain\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     72\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mDEVICE\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m'cpu'\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     73\u001b[0m                 \u001b[0mloss\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 74\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     75\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     76\u001b[0m                 \u001b[0mscaler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mscale\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mloss\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;31m# This is a replacement for loss.backward()\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.9/dist-packages/torch/optim/optimizer.py\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    278\u001b[0m                                                f\"but got {result}.\")\n\u001b[1;32m    279\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 280\u001b[0;31m                 \u001b[0mout\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    281\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_optimizer_step_code\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    282\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.9/dist-packages/torch/optim/optimizer.py\u001b[0m in \u001b[0;36m_use_grad\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m     31\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     32\u001b[0m             \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mset_grad_enabled\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdefaults\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'differentiable'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 33\u001b[0;31m             \u001b[0mret\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     34\u001b[0m         \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     35\u001b[0m             \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mset_grad_enabled\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprev_grad\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.9/dist-packages/torch/optim/adamw.py\u001b[0m in \u001b[0;36mstep\u001b[0;34m(self, closure)\u001b[0m\n\u001b[1;32m    169\u001b[0m             )\n\u001b[1;32m    170\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 171\u001b[0;31m             adamw(\n\u001b[0m\u001b[1;32m    172\u001b[0m                 \u001b[0mparams_with_grad\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    173\u001b[0m                 \u001b[0mgrads\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.9/dist-packages/torch/optim/adamw.py\u001b[0m in \u001b[0;36madamw\u001b[0;34m(params, grads, exp_avgs, exp_avg_sqs, max_exp_avg_sqs, state_steps, foreach, capturable, differentiable, fused, grad_scale, found_inf, amsgrad, beta1, beta2, lr, weight_decay, eps, maximize)\u001b[0m\n\u001b[1;32m    319\u001b[0m         \u001b[0mfunc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_single_tensor_adamw\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    320\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 321\u001b[0;31m     func(\n\u001b[0m\u001b[1;32m    322\u001b[0m         \u001b[0mparams\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    323\u001b[0m         \u001b[0mgrads\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.9/dist-packages/torch/optim/adamw.py\u001b[0m in \u001b[0;36m_single_tensor_adamw\u001b[0;34m(params, grads, exp_avgs, exp_avg_sqs, max_exp_avg_sqs, state_steps, grad_scale, found_inf, amsgrad, beta1, beta2, lr, weight_decay, eps, maximize, capturable, differentiable)\u001b[0m\n\u001b[1;32m    438\u001b[0m                 \u001b[0mdenom\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mmax_exp_avg_sqs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msqrt\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0mbias_correction2_sqrt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0meps\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    439\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 440\u001b[0;31m                 \u001b[0mdenom\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mexp_avg_sq\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msqrt\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0mbias_correction2_sqrt\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madd_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0meps\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    441\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    442\u001b[0m             \u001b[0mparam\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0maddcdiv_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mexp_avg\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdenom\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0mstep_size\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ],
      "source": [
        "%%time\n",
        "# Run the experiments loop. \n",
        "# Each epoch wont take more than 2-3min. If its taking more time, it might be due to (but not limited to) the following:\n",
        "#   * You might be overlapping batches \n",
        "#       Eg. Input: \"I had biryani for lunch today\" and sequence length = 3,\n",
        "#           --> \"I had biryani\", \"for lunch today\" are ideal examples for inputs\n",
        "#           --> \"I had biryani\", \"had biryani for\", \"biryani for lunch\", ... is just redundant info :')\n",
        "#   * Your length calculation in the dataloader might be wrong\n",
        "# If you haven't had biryani, try it :D \n",
        "\n",
        "best_nll = 1e30 \n",
        "for epoch in range(configs['num_epochs']):\n",
        "    trainer.train()\n",
        "    nll = trainer.test()\n",
        "    if nll < best_nll:\n",
        "        best_nll = nll\n",
        "        print(\"Saving model, predictions and generated output for epoch \"+str(epoch+1)+\" with NLL: \"+ str(best_nll))\n",
        "        trainer.save()    "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 40,
      "metadata": {
        "id": "i_gYqXq9Jgo1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 449
        },
        "outputId": "08d4ea94-840b-4765-8dba-a0a9c39ab0b9"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGwCAYAAABVdURTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABjU0lEQVR4nO3deVxU9f7H8dcMOwiIyCqIC4i4r5VimmmpmTe9puW11BbbNJeyW/66tt60xcpWTW9pZWlZaq6ZmJaplRvmviu4gBuCoKwzvz/GRkmdsIAzDO/n43Eed+acMzMfxOu8O+f7/X5MVqvVioiIiIiLMBtdgIiIiEhpUrgRERERl6JwIyIiIi5F4UZERERcisKNiIiIuBSFGxEREXEpCjciIiLiUtyNLqC8WSwWjhw5gr+/PyaTyehyREREpASsVitnzpwhMjISs9nxtZlKF26OHDlCdHS00WWIiIjIX5CamkpUVJTDcypduPH39wdsfzgBAQEGVyMiIiIlkZWVRXR0tP173BFDw02tWrU4ePDgJfsfeeQR3nvvvUv2T5s2jXvuuafYPi8vL3Jzc0v8mb/figoICFC4ERERqWBKMqTE0HCzdu1aioqK7M+3bNnCTTfdRJ8+fa74moCAAHbu3Gl/rnEzIiIicjFDw01ISEix5y+//DJ169alQ4cOV3yNyWQiPDy8rEsTERGRCspppoLn5+czffp07r33XodXY7Kzs4mJiSE6OprbbruNrVu3OnzfvLw8srKyim0iIiLiupxmQPHcuXM5ffo0gwYNuuI58fHxfPTRRzRp0oTMzEzGjx9P27Zt2bp16xVHTo8bN47nn3++jKoWEXENRUVFFBQUGF2GVHKenp5/Os27JExWq9VaCvX8bV26dMHT05P58+eX+DUFBQUkJCTQr18/Xnzxxcuek5eXR15env3576OtMzMzNaBYRCo9q9VKWloap0+fNroUEcxmM7Vr18bT0/OSY1lZWQQGBpbo+9sprtwcPHiQpKQkZs+efVWv8/DwoHnz5uzZs+eK53h5eeHl5fV3SxQRcUm/B5vQ0FB8fX01SUMM8/siu0ePHqVmzZp/6++iU4SbqVOnEhoaSvfu3a/qdUVFRWzevJlbbrmljCoTEXFdRUVF9mATHBxsdDkihISEcOTIEQoLC/Hw8PjL72P4gGKLxcLUqVMZOHAg7u7Fs9aAAQMYPXq0/fkLL7zAd999x759+9iwYQN33XUXBw8e5P777y/vskVEKrzfx9j4+voaXImIze+3oy5eJuavMPzKTVJSEikpKdx7772XHEtJSSk2sCgjI4PBgweTlpZGUFAQLVu2ZPXq1TRo0KA8SxYRcSm6FSXOorT+LjrNgOLycjUDkkREXFlubi779++ndu3aeHt7G12OiMO/k1fz/W34bSkRERGR0qRwIyIilV6tWrWYMGFCic9fsWIFJpOpzKfQT5s2japVq5bpZ7gihZtSdCjjLDvTzhhdhoiIyzKZTA6355577i+979q1a3nggQdKfH7btm05evQogYGBf+nzpGwZPqDYVSzefJThM5NpWCOA2Q+31QA9EZEycPToUfvjL774gmeeeaZYM+UqVarYH1utVoqKii6ZiXs5f+x1+Gc8PT3V59CJ6cpNKWkZE4Sb2cTGlNN8uyXN6HJERK6a1WrlbH6hIVtJ57aEh4fbt8DAQHsz5fDwcHbs2IG/vz+LFy+mZcuWeHl58dNPP7F3715uu+02wsLCqFKlCq1btyYpKanY+/7xtpTJZOJ///sfvXr1wtfXl7i4OObNm2c//sfbUr/fPlqyZAkJCQlUqVKFrl27FgtjhYWFDBs2jKpVqxIcHMyTTz7JwIED6dmz51X9niZOnEjdunXx9PQkPj6eTz/9tNjv8LnnnqNmzZp4eXkRGRnJsGHD7Mfff/994uLi8Pb2JiwsjNtvv91+zGKxMG7cOGrXro2Pjw9Nmzblq6++sh/PyMigf//+hISE4OPjQ1xcHFOnTr2q2suLrtyUktAAbwZfX5u3v9/Dq0t20rlBGB5uyo4iUnGcKyiiwTNLDPnsbS90wdezdL6SnnrqKcaPH0+dOnUICgoiNTWVW265hZdeegkvLy8++eQTevTowc6dO6lZs+YV3+f555/n1Vdf5bXXXuOdd96hf//+HDx4kGrVql32/LNnzzJ+/Hg+/fRTzGYzd911F6NGjeKzzz4D4JVXXuGzzz5j6tSpJCQk8NZbbzF37lw6duxY4p9tzpw5DB8+nAkTJtC5c2cWLFjAPffcQ1RUFB07duTrr7/mzTffZObMmTRs2JC0tDQ2bdoEwLp16xg2bBiffvopbdu25dSpU6xcudL+3uPGjWP69OlMmjSJuLg4fvzxR+666y5CQkLo0KEDY8aMYdu2bSxevJjq1auzZ88ezp07V+Lay5PCTSl6oENdPvslhf0ncpj5awp3t6lldEkiIpXOCy+8wE033WR/Xq1aNZo2bWp//uKLLzJnzhzmzZvH0KFDr/g+gwYNol+/fgCMHTuWt99+m19//ZWuXbte9vyCggImTZpE3bp1ARg6dCgvvPCC/fg777zD6NGj6dWrFwDvvvsuixYtuqqfbfz48QwaNIhHHnkEgMcee4yff/6Z8ePH07FjR1JSUggPD6dz5854eHhQs2ZNrrnmGsC2dpyfnx+33nor/v7+xMTE0Lx5c8DWh3Hs2LEkJSXRpk0bAOrUqcNPP/3EBx98QIcOHUhJSaF58+a0atUKsF3tclYKN6Woipc7wzvH8cw3W3lr2W56tYiiipf+iEWkYvDxcGPbC10M++zS8vuX7++ys7N57rnnWLhwIUePHqWwsJBz586RkpLi8H2aNGlif+zn50dAQADHjh274vm+vr72YAMQERFhPz8zM5P09HR70ABwc3OjZcuWWCyWEv9s27dvv2Tgc2JiIm+99RYAffr0YcKECdSpU4euXbtyyy230KNHD9zd3bnpppuIiYmxH+vatav9ttuePXs4e/ZssVAIkJ+fbw9ADz/8ML1792bDhg3cfPPN9OzZk7Zt25a49vKk+yalrN81NakV7MuJ7Hym/LjP6HJERErMZDLh6+luyFaakzD8/PyKPR81ahRz5sxh7NixrFy5kuTkZBo3bkx+fr7D9/ljbyOTyeQwiFzu/PJeJzc6OpqdO3fy/vvv4+PjwyOPPEL79u0pKCjA39+fDRs2MGPGDCIiInjmmWdo2rQpp0+fJjs7G4CFCxeSnJxs37Zt22Yfd9OtWzcOHjzIyJEjOXLkCJ06dWLUqFHl+vOVlMJNKfNwM/NEl/oATFm5j2Nncg2uSESkclu1ahWDBg2iV69eNG7cmPDwcA4cOFCuNQQGBhIWFsbatWvt+4qKitiwYcNVvU9CQgKrVq0qtm/VqlXF2hD5+PjQo0cP3n77bVasWMGaNWvYvHkzAO7u7nTu3JlXX32V3377jQMHDvD999/ToEEDvLy8SElJITY2ttgWHR1tf++QkBAGDhzI9OnTmTBhApMnT/4rfxxlTvdMysAtjcNpGl2VTamneStpNy/1amx0SSIilVZcXByzZ8+mR48emEwmxowZc1W3gkrLo48+yrhx44iNjaV+/fq88847ZGRkXNVVqyeeeIK+ffvSvHlzOnfuzPz585k9e7Z99te0adMoKiri2muvxdfXl+nTp+Pj40NMTAwLFixg3759tG/fnqCgIBYtWoTFYiE+Ph5/f39GjRrFyJEjsVgstGvXjszMTFatWkVAQAADBw7kmWeeoWXLljRs2JC8vDwWLFhAQkJCWf1x/S26clMGTCYTo7vZrt7MXJvK3uPZBlckIlJ5vfHGGwQFBdG2bVt69OhBly5daNGiRbnX8eSTT9KvXz8GDBhAmzZtqFKlCl26dLmqvl49e/bkrbfeYvz48TRs2JAPPviAqVOncsMNNwBQtWpVpkyZQmJiIk2aNCEpKYn58+cTHBxM1apVmT17NjfeeCMJCQlMmjSJGTNm0LBhQ8A20HrMmDGMGzeOhIQEunbtysKFC6lduzZgW9tn9OjRNGnShPbt2+Pm5sbMmTNL/c+pNKhxZhm6b9palu04RteG4Uy6u2WZfpaIyNVS40xjWSwWEhIS6Nu3Ly+++KLR5TgFNc6sAJ7sVh+zCb7dmsb6g6eMLkdERAx08OBBpkyZwq5du9i8eTMPP/ww+/fv51//+pfRpbkchZsyVC/Mnz4tbQOxxi3aUe6j5kVExHmYzWamTZtG69atSUxMZPPmzSQlJTntuJWKTAOKy9jIm+rxzabDrDuYwdJt6dzcUL1IREQqo+jo6EtmOknZ0JWbMhYe6M29ibbBWK98u4PCovIfoS8iIlKZKNyUg4duqEuQrwd7j+fw5bpDRpcjIiLi0hRuykGAtweP3hgHwJtJuzibX2hwRSIiIq5L4aac9L+uJtHVfDh+Jo8PV+43uhwRERGXpXBTTrzc3Rh1czwAk37Yy4nsPIMrEhERcU0KN+WoR5NIGtcIJCe/iHeW7Ta6HBGRSuuGG25gxIgR9ue1atViwoQJDl9jMpmYO3fu3/7s0nofR5577jmaNWtWpp/hzBRuypHZfKEtw2e/pHDgRI7BFYmIVCw9evSga9eulz22cuVKTCYTv/3221W/79q1a3nggQf+bnnFXClgHD16lG7dupXqZ0lxCjflrG1sdTrUC6HQYuW173YaXY6ISIVy3333sXTpUg4dunTm6dSpU2nVqhVNmjS56vcNCQnB19e3NEr8U+Hh4Xh5eZXLZ1VWCjcGeKpbfUwmWPjbUZJTTxtdjohIhXHrrbcSEhLCtGnTiu3Pzs5m1qxZ3HfffZw8eZJ+/fpRo0YNfH19ady4MTNmzHD4vn+8LbV7927at2+Pt7c3DRo0YOnSpZe85sknn6RevXr4+vpSp04dxowZQ0FBAWDrzv3888+zadMmTCYTJpPJXvMfb0tt3ryZG2+8ER8fH4KDg3nggQfIzr7QcHnQoEH07NmT8ePHExERQXBwMEOGDLF/VklYLBZeeOEFoqKi8PLyolmzZnz77bf24/n5+QwdOpSIiAi8vb2JiYlh3LhxAFitVp577jlq1qyJl5cXkZGRDBs2zP7avLw8Ro0aRY0aNfDz8+Paa69lxYoV9uMHDx6kR48eBAUF4efnR8OGDVm0aFGJa/8rtEKxARIiAujVvAazNxxm3KLtzHzguqtqeS8iUiasVig4a8xne/hCCf4ddHd3Z8CAAUybNo2nn37a/m/nrFmzKCoqol+/fmRnZ9OyZUuefPJJAgICWLhwIXfffTd169blmmuu+dPPsFgs/POf/yQsLIxffvmFzMzMYuNzfufv78+0adOIjIxk8+bNDB48GH9/f/79739zxx13sGXLFr799luSkpIACAwMvOQ9cnJy6NKlC23atGHt2rUcO3aM+++/n6FDhxYLcMuXLyciIoLly5ezZ88e7rjjDpo1a8bgwYP/9OcBeOutt3j99df54IMPaN68OR999BH/+Mc/2Lp1K3Fxcbz99tvMmzePL7/8kpo1a5KamkpqaioAX3/9NW+++SYzZ86kYcOGpKWlsWnTJvt7Dx06lG3btjFz5kwiIyOZM2cOXbt2ZfPmzcTFxTFkyBDy8/P58ccf8fPzY9u2bVSpUqVEdf9VCjcGefzmeBb8dpRf9p9i+c5j3Fg/zOiSRKSyKzgLYyON+ez/OwKefiU69d577+W1117jhx9+4IYbbgBst6R69+5NYGAggYGBjBo1yn7+o48+ypIlS/jyyy9LFG6SkpLYsWMHS5YsITLS9ucxduzYS8bJ/Oc//7E/rlWrFqNGjWLmzJn8+9//xsfHhypVquDu7k54+JXb7nz++efk5ubyySef4Odn+/nfffddevTowSuvvEJYmO27ISgoiHfffRc3Nzfq169P9+7dWbZsWYnDzfjx43nyySe58847AXjllVdYvnw5EyZM4L333iMlJYW4uDjatWuHyWQiJibG/tqUlBTCw8Pp3LkzHh4e1KxZ0/7nmJKSwtSpU0lJSbH/WY0aNYpvv/2WqVOnMnbsWFJSUujduzeNGzcGoE6dOiWq+e/QbSmD1Kjqwz1tawHw8uIdFFnUVFNEpCTq169P27Zt+eijjwDYs2cPK1eu5L777gOgqKiIF198kcaNG1OtWjWqVKnCkiVLSElJKdH7b9++nejoaPuXNUCbNm0uOe+LL74gMTGR8PBwqlSpwn/+858Sf8bFn9W0aVN7sAFITEzEYrGwc+eFcZkNGzbEzc3N/jwiIoJjx46V6DOysrI4cuQIiYmJxfYnJiayfft2wHbrKzk5mfj4eIYNG8Z3331nP69Pnz6cO3eOOnXqMHjwYObMmUNhoW0x2s2bN1NUVES9evWoUqWKffvhhx/Yu3cvAMOGDeO///0viYmJPPvss39pwPfV0pUbAz1yQywz16ayKz2br9cfom/raKNLEpHKzMPXdgXFqM++Cvfddx+PPvoo7733HlOnTqVu3bp06NABgNdee4233nqLCRMm0LhxY/z8/BgxYgT5+fmlVu6aNWvo378/zz//PF26dCEwMJCZM2fy+uuvl9pnXMzDw6PYc5PJhMVSer0KW7Rowf79+1m8eDFJSUn07duXzp0789VXXxEdHc3OnTtJSkpi6dKlPPLII/YrZ9nZ2bi5ubF+/fpi4Quw33q6//776dKlCwsXLuS7775j3LhxvP766zz66KOlVv8f6cqNgQJ9PRjaMRaAN5bu4lx+kcEViUilZjLZbg0ZsV3luMO+fftiNpv5/PPP+eSTT7j33nvt429WrVrFbbfdxl133UXTpk2pU6cOu3btKvF7JyQkkJqaytGjR+37fv7552LnrF69mpiYGJ5++mlatWpFXFwcBw8eLHaOp6cnRUWO/11PSEhg06ZN5ORcWBpk1apVmM1m4uPjS1yzIwEBAURGRl7SkXzVqlU0aNCg2Hl33HEHU6ZM4YsvvuDrr7/m1KlTAPj4+NCjRw/efvttVqxYwZo1a9i8eTPNmzenqKiIY8eOERsbW2y7+HZcdHQ0Dz30ELNnz+bxxx9nypQppfKzXYnCjcHubhNDjao+pGXlMnW12jKIiJRElSpVuOOOOxg9ejRHjx5l0KBB9mNxcXEsXbqU1atXs337dh588EHS09NL/N6dO3emXr16DBw4kE2bNrFy5UqefvrpYufExcWRkpLCzJkz2bt3L2+//TZz5swpdk6tWrXYv38/ycnJnDhxgry8S1em79+/P97e3gwcOJAtW7awfPlyHn30Ue6++277eJvS8MQTT/DKK6/wxRdfsHPnTp566imSk5MZPnw4AG+88QYzZsxgx44d7Nq1i1mzZhEeHk7VqlWZNm0aH374IVu2bGHfvn1Mnz4dHx8fYmJiqFevHv3792fAgAHMnj2b/fv38+uvvzJu3DgWLlwIwIgRI1iyZAn79+9nw4YNLF++nISEhFL72S5H4cZg3h5uPH5zPQAmLt/LqZzSu2wqIuLK7rvvPjIyMujSpUux8TH/+c9/aNGiBV26dOGGG24gPDycnj17lvh9zWYzc+bM4dy5c1xzzTXcf//9vPTSS8XO+cc//sHIkSMZOnQozZo1Y/Xq1YwZM6bYOb1796Zr16507NiRkJCQy05H9/X1ZcmSJZw6dYrWrVtz++2306lTJ959992r+8P4E8OGDeOxxx7j8ccfp3Hjxnz77bfMmzePuDhbU2d/f39effVVWrVqRevWrTlw4ACLFi3CbDZTtWpVpkyZQmJiIk2aNCEpKYn58+cTHBwM2AZzDxgwgMcff5z4+Hh69uzJ2rVrqVmzJmAbAzVkyBASEhLo2rUr9erV4/333y/Vn++PTFartVKNZM3KyiIwMJDMzEwCAgKMLgcAi8VK93d+YvvRLO5NrM0zPRr8+YtERP6m3Nxc9u/fT+3atfH29ja6HBGHfyev5vtbV26cwMVtGT79+QCppwxaZ0JERMQFKNw4ifb1QmgXW52CIivj1ZZBRETkL1O4cSJPnb96803yEbYczjS4GhERkYpJ4caJNKoRyG3NbIPixi3eTiUbDiUiIlIqFG6czKib4/F0M7Nqz0l+3H3C6HJEpBLQf0iJsyitv4sKN04mupovd7ex9fR4efEOLGrLICJl5PdVb8+e1SQGcQ6/ryL9x9WOr5baLzihoR1j+XJdKtuPZjE3+TD/bBFldEki4oLc3NyoWrWqvUeRr6+vfZVfkfJmsVg4fvw4vr6+uLv/vXiicOOEgvw8eeSGWF75dgevf7eLWxpH4O3x91KsiMjl/L5EfkmbMIqUJbPZTM2aNf92yFa4cVL3JNbikzUHOHz6HJ+sOcAD7esaXZKIuCCTyURERAShoaEUFBQYXY5Ucp6enpjNf3/EjMKNk/L2cGPkTfX491e/8e73e+jbKpqqvp5GlyUiLsrNze1vj3MQcRYaUOzEereIIj7Mn6zcQt5fsdfockRERCoEhRsn5mY22Rf2m7badotKREREHFO4cXI3xIdwXZ1q5BdaeF1tGURERP6UoeGmVq1amEymS7YhQ4Zc8TWzZs2ifv36eHt707hxYxYtWlSOFZc/k8nE6G4JAMzZeJhtR7IMrkhERMS5GRpu1q5dy9GjR+3b0qVLAejTp89lz1+9ejX9+vXjvvvuY+PGjfTs2ZOePXuyZcuW8iy73DWNrkr3JhFYrfDytzuMLkdERMSpmaxOtO72iBEjWLBgAbt3777sHPc77riDnJwcFixYYN933XXX0axZMyZNmlSiz8jKyiIwMJDMzEwCAgJKrfaydvBkDp3f+IGCIiuf3X8tibHVjS5JRESk3FzN97fTjLnJz89n+vTp3HvvvVdcvGfNmjV07ty52L4uXbqwZs2aK75vXl4eWVlZxbaKKCbYj/7X2toyjFu8XW0ZRERErsBpws3cuXM5ffo0gwYNuuI5aWlphIWFFdsXFhZGWlraFV8zbtw4AgMD7Vt0dHRplVzuHr0xlipe7mw5nMX8344YXY6IiIhTcppw8+GHH9KtWzciIyNL9X1Hjx5NZmamfUtNTS3V9y9PwVW8eLB9HQBeW7KTvMIigysSERFxPk4Rbg4ePEhSUhL333+/w/PCw8NJT08vti89Pd3eG+VyvLy8CAgIKLZVZPddX5tQfy8OZZxj+s8pRpcjIiLidJwi3EydOpXQ0FC6d+/u8Lw2bdqwbNmyYvuWLl1KmzZtyrI8p+Lr6c7Im+oB8O73u8nKVS8YERGRixkebiwWC1OnTmXgwIGXtDgfMGAAo0ePtj8fPnw43377La+//jo7duzgueeeY926dQwdOrS8yzZUn5ZR1A3xI+NsAZPUlkFERKQYw8NNUlISKSkp3HvvvZccS0lJ4ejRo/bnbdu25fPPP2fy5Mk0bdqUr776irlz59KoUaPyLNlw7m5mnjq/sN+HP+3naKbaMoiIiPzOqda5KQ8VdZ2bP7JarfT9YA1rD2TQt1UUr97e1OiSREREykyFXOdGro7JZLJfvflq/SF2pZ8xuCIRERHnoHBTgbWMCaJrw3AsVnhlsdoyiIiIgMJNhffvrvG4mU0s23GMn/edNLocERERwyncVHB1QqrQ7xrbqsvjFu+gkg2hEhERuYTCjQsY3qkevp5ubEo9zaLNV25FISIiUhko3LiAEH8vBl9va8vw6pId5BdaDK5IRETEOAo3LmJw+zpUr+LFwZNnmfGr2jKIiEjlpXDjIqp4uTO8cxwAby/bzRm1ZRARkUpK4caF3Nk6mjrV/TiZk8+UH/cZXY6IiIghFG5ciIebmSe6xAMwZeV+jmXlGlyRiIhI+VO4cTFdG4XTvGZVzhUU8WbSbqPLERERKXcKNy7GZDLxf7fY2jJ8uS6VPceyDa5IRESkfCncuKDWtarROSGMIouVV79VWwYREalcFG5c1FPd4jGb4Ltt6aw7cMrockRERMqNwo2Lig31547WtrYMYxdtV1sGERGpNBRuXNiIzvXw9jCzIeU0S7amG12OiIhIuVC4cWFhAd7c3+58W4Zvd1BQpLYMIiLi+hRuXNyDHepQzc+TfSdy+GJtqtHliIiIlDmFGxfn7+3BsBtjAZiQtJucvEKDKxIRESlbCjeVwL+ujSEm2JcT2Xn8b+V+o8sREREpUwo3lYCnu5lRN9vaMkz+cS/Hz+QZXJGIiEjZUbipJLo3jqBpVCA5+UW8vUxtGURExHUp3FQSZrOJp7rZ2jLM+DWF/SdyDK5IRESkbCjcVCJt6gbTMT6EQouV15aoLYOIiLgmhZtK5slu9TGZYNHmNDakZBhdjoiISKlTuKlk6ocHcHuLKABeXrRDbRlERMTlKNxUQo/dXA8vdzO/HjjFsu3HjC5HRESkVCncVEIRgT7ck1gbgFe+3UGh2jKIiIgLUbippB6+oS5VfT3YfSybr9YfMrocERGRUqNwU0kF+ngwtKOtLcObSbs4l19kcEUiIiKlQ+GmEru7TQxRQT6kZ+Xx0Sq1ZRAREdegcFOJebm72dsyTFyxl5PZassgIiIVn8JNJfePppE0jAwgO6+Qd77fY3Q5IiIif5vCTSVnNpsYfb4tw2e/HCTl5FmDKxIREfl7FG6EdnHVuT6uOgVFVl77bqfR5YiIiPwtCjcCwFPn2zLM33SE3w6dNrocERGRv0zhRgBoGBlIr2Y1ABintgwiIlKBKdyI3WM318PTzcyafSdZseu40eWIiIj8JQo3YhcV5MvAtjEAvLRwO2fzCw2uSERE5Oop3EgxQzrGUr2KJ3uOZfPYF5uwWHR7SkREKhaFGymmqq8nH9zdEk83M99uTePNpF1GlyQiInJVFG7kEi1jqjH2n40BeOf7PXyTfNjgikREREpO4UYu6/aWUTzYvg4A//7qN5JTTxtbkIiISAkp3MgV/btrfTrVDyWv0MIDn6wjLTPX6JJERET+lMKNXJGb2cSEO5tRL6wKx87kMfiTdZzLLzK6LBEREYcUbsQhf28P/jegNUG+Hmw+nMmorzZpgT8REXFqCjfyp2oG+zLprpZ4uJlY+NtR3l6m7uEiIuK8DA83hw8f5q677iI4OBgfHx8aN27MunXrrnj+ihUrMJlMl2xpaWnlWHXlc22dYP7bsxEAbybtYuFvRw2uSERE5PLcjfzwjIwMEhMT6dixI4sXLyYkJITdu3cTFBT0p6/duXMnAQEB9uehoaFlWaoAd7Suyc60bD5atZ/HZyUTE+xLoxqBRpclIiJSjKHh5pVXXiE6OpqpU6fa99WuXbtErw0NDaVq1ap/el5eXh55eXn251lZWVddp1zwf7fUZ+/xbH7YdZz7P17HvKGJhAZ4G12WiIiInaG3pebNm0erVq3o06cPoaGhNG/enClTppTotc2aNSMiIoKbbrqJVatWXfG8cePGERgYaN+io6NLq/xKyd3NzDv/ak7dED/SsnIZ/Ol6cgs0g0pERJyHoeFm3759TJw4kbi4OJYsWcLDDz/MsGHD+Pjjj6/4moiICCZNmsTXX3/N119/TXR0NDfccAMbNmy47PmjR48mMzPTvqWmppbVj1NpBHh78OHA1gT6eLAp9TRPfv2bZlCJiIjTMFkN/Fby9PSkVatWrF692r5v2LBhrF27ljVr1pT4fTp06EDNmjX59NNP//TcrKwsAgMDyczMLDZmR67e6j0nGPDRrxRarDzRJZ4hHWONLklERFzU1Xx/G3rlJiIiggYNGhTbl5CQQEpKylW9zzXXXMOePZqeXN7axlbnuX80BOC1JTtZslUz1kRExHiGhpvExER27txZbN+uXbuIiYm5qvdJTk4mIiKiNEuTErrruhgGtLH9vkZ+kcy2IxqwLSIixjI03IwcOZKff/6ZsWPHsmfPHj7//HMmT57MkCFD7OeMHj2aAQMG2J9PmDCBb775hj179rBlyxZGjBjB999/X+w1Ur6eubUB7WKrcza/iMGfrOP4mbw/f5GIiEgZMTTctG7dmjlz5jBjxgwaNWrEiy++yIQJE+jfv7/9nKNHjxa7TZWfn8/jjz9O48aN6dChA5s2bSIpKYlOnToZ8SMIthlU7/2rBbWr+3H49Dkemr6evELNoBIREWMYOqDYCBpQXHb2Hs+m13uryMotpHeLKMb3aYLJZDK6LBERcQEVZkCxuJa6IVV4r38L3Mwmvt5wiMk/7jO6JBERqYQUbqRUXR8XwpjuCQC8/O0Olm1PN7giERGpbBRupNQNbFuLf11bE6sVhs3YyM60M0aXJCIilYjCjZQ6k8nE8/9oyHV1qpGTX8R9H6/lZLZmUImISPlQuJEy4eFmZmL/lsQE+3Io4xwPf7aB/EKL0WWJiEgloHAjZSbIz5MPB7bC38udX/efYszcLepBJSIiZU7hRspUbKg/b/+rOWYTfLEulY9WHTC6JBERcXEKN6Vp2zzYv9LoKpxOx/hQ/u8W2wyqlxZuY8XOYwZXJCIirkzhprRsnA5f3g1zH4bcTKOrcTr3tatN31ZRWKzw6Ocb2XNMM6hERKRsKNyUlgY9IagWZKbCt/9ndDVOx2Qy8d+ejbmmVjXO5BVy38fryMjJN7osERFxQQo3pcWrCvScCJggeTrsXGx0RU7H093MxLtaEBXkw8GTZ3nksw0UFGkGlYiIlC6Fm9IU0xbaDrU9njcMck4aW48TCq7ixf8GtsLP0401+07y3LytmkElIiKlSuGmtHX8D4TUh5xjsHAk6Iv7EvXDA3jrzuaYTPDZLyl8+vNBo0sSEREXonBT2jy8odckMLvDtm9g81dGV+SUOjcI48mu9QF4fv42Vu4+bnBFIiLiKhRuykJkc2j/hO3xosch64ix9TipB9vX4Z8talBksTLksw3sO55tdEkiIuICFG7KyvWPQ0Qz27TweY/q9tRlmEwmxvZqTIuaVcnKLeT+j9eRebbA6LJERKSCU7gpK24e0OsDcPOCPUmwfprRFTklbw83Pri7FZGB3uw7kcPQGRso1AwqERH5GxRuylJofej0jO3xkqfh1H5j63FSIf5eTBnYCh8PN1buPsF/F243uiQREanAFG7K2nWPQEwiFOTA3EfAUmR0RU6pYWQgb97RDIBpqw/w2S+aQSUiIn+Nwk1ZM5vhtvfAww9SVsPP7xtdkdPq2iicUTfXA+DZb7ayeu8JgysSEZGKSOGmPFSrDV1esj1e9iIc022XKxnSMZbbmkVSaLHyyGcbOHAix+iSRESkglG4KS8tB0HsTVCUB3MehCLNCrock8nEK72b0DS6KqfPFnD/J+vIytWflYiIlJzCTXkxmeAf74B3VTi6CVa+bnRFTsvbw40pd7ckPMCbPceyGTZjI0UWTaUXEZGSUbgpTwER0P18qPnxNTiy0dh6nFhogDdTBrTC28PMip3HGbdIt/JERKRkFG7KW6Pe0KAnWAphzkNQkGt0RU6rcVQgr/dpBsD/ftrPF2tTjC1IREQqBIWb8mYyQfc3wC8Uju+A7180uiKn1r1JBCM6xwHwn7lb+HX/KYMrEhERZ6dwYwS/YPjH27bHa96DA6uMrcfJDbsxju6NIygosvLQ9PWknjprdEkiIuLEFG6MEt8Nmt0FWGHuw5B3xuiKnJbZbGJ8n6Y0qhHAqZx87v94Hdl5hUaXJSIiTkrhxkhdx0FgNJw+CN/9x+hqnJqPpxtTBrQi1N+LnelnGK4ZVCIicgUKN0byDoCe51csXj8NdicZWo6ziwj0YfKAVni5m1m24xivLtlhdEkiIuKEFG6MVrs9XPuQ7fG8oXAuw9h6nFyz6Kq8ensTAD74YR9frz9kcEUiIuJsFG6cQadnITgWzhyFRU8YXY3Tu61ZDYZ2jAVg9OzNrD+oGVQiInKBwo0z8PSFXh+AyQybZ8HWuUZX5PQeu6keXRqGkV9k4cFP13P49DmjSxIRESehcOMsolpBu8dsjxeMhDPpxtbj5MxmE2/0bUZCRAAnsm0zqHI0g0pERFC4cS4dnoSwxnDuFMwfDlbNBnLEz8ud/w1sRfUqnmw/msVjXyZj0QwqEZFKT+HGmbh7Qq9JYPaAXYsh+TOjK3J6Nar68MHdrfB0M7NkazpvLN1ldEkiImIwhRtnE94IOv6f7fHip+C0+in9mZYxQbzcuzEA7y7fwzfJhw2uSEREjKRw44wSh0PUNZB/Br4ZAhaL0RU5vX+2iOKhDnUBeOKr30hOPW1sQSIiYhiFG2dkdrPdnvLwhf0/wtopRldUITzRJZ7OCaHkF1oY/Mk6jmZqBpWISGWkcOOsguvCTS/YHi99Fk7sMbaeCsDNbGLCnc2JD/Pn+Jk8Bn+yjnP5RUaXJSIi5Uzhxpm1ug/q3ACF52DOg1Ckqc5/psr5GVTV/DzZcjiLEV9sJLdAAUdEpDJRuHFmZjPc9h54BcDhdbBqgtEVVQjR1XyZdFdLPNxMLNmazj/e/Yld6eq6LiJSWSjcOLvAKOj2iu3xipchbbOx9VQQ19SuxtRB11C9ihe70rPp8c5PfPbLQaxaO0hExOUp3FQETftBfHewFMDsB6Ewz+iKKoR2cdVZPPx62tcLIa/QwtNztvDIZxvIPFtgdGkiIlKGFG4qApMJerwFvsFwbKvtCo6USIi/F9MGtebpWxLwcDOxeEsa3d76kbUH1GxTRMRVKdxUFFVC4NYJtserJkDqr0ZWU6GYzSYGt6/D1w+3pVawL0cyc7njgzW8lbSbIrVrEBFxOYaHm8OHD3PXXXcRHByMj48PjRs3Zt26dQ5fs2LFClq0aIGXlxexsbFMmzatfIo1WoN/QJM7wGqBOQ9Bfo7RFVUoTaKqsmDY9fRqXgOLFd5M2sW/pvys9XBERFyMoeEmIyODxMREPDw8WLx4Mdu2beP1118nKCjoiq/Zv38/3bt3p2PHjiQnJzNixAjuv/9+lixZUo6VG6jbK+AfCaf2QtJzRldT4VTxcufNO5rxRt+m+Hm68cv+U3R7ayXfbU0zujQRESklJmspTh/Zt28fDz30EN99912Jzn/qqadYtWoVK1euLPFnPPnkkyxcuJAtW7bY9915552cPn2ab7/99k9fn5WVRWBgIJmZmQQEBJT4c53KniSY3tv2eMA3trVw5KrtP5HDsBkb2Xw4E4ABbWL4v1sS8PZwM7gyERH5o6v5/i7VKzdnzpxh2bJlJT5/3rx5tGrVij59+hAaGkrz5s2ZMsVxq4E1a9bQuXPnYvu6dOnCmjVrLnt+Xl4eWVlZxbYKL7YztLrX9njuEMjNNLaeCqp2dT++frgtg6+vDcAnaw7S871V7NaaOCIiFZqht6X27dvHxIkTiYuLY8mSJTz88MMMGzaMjz/++IqvSUtLIywsrNi+sLAwsrKyOHfu0rET48aNIzAw0L5FR0eX+s9hiJtehKBakHXI1j1c/hJPdzNPd2/A1HtaE+znyY60M/R49ydm/JqiNXFERCooQ8ONxWKhRYsWjB07lubNm/PAAw8wePBgJk2aVGqfMXr0aDIzM+1bampqqb23obyqQM9JgAk2fQ47FhpdUYXWMT6UxSOu5/q46uQWWBg9ezNDP99I5jmtiSMiUtEYGm4iIiJo0KBBsX0JCQmkpKRc8TXh4eGkp6cX25eenk5AQAA+Pj6XnO/l5UVAQECxzWXEtIG2j9oezx8OOSeMraeCC/X35uN7ruGpbvVxN5tYuPkot7y1kvUHtSaOiEhF4n41Jzdv3hyTyXTF42fPnr2qD09MTGTnzp3F9u3atYuYmJgrvqZNmzYsWrSo2L6lS5fSpk2bq/psl9Hxadi9FI5vhwUjoe8ntkX/5C8xm0081KEu19UJZtiMjaScOkvfD35mZOc4Hr4hFjez/mxFRJzdVYWbnj17luqHjxw5krZt2zJ27Fj69u3Lr7/+yuTJk5k8ebL9nNGjR3P48GE++eQTAB566CHeffdd/v3vf3Pvvffy/fff8+WXX7JwYSW9LePhDb0mwf86wfZ5sHkWNOlrdFUVXrPoqiwc1o6n52xh3qYjjP9uF6v2nOTNO5oRHuhtdHkiIuJAqU4F/ysWLFjA6NGj2b17N7Vr1+axxx5j8ODB9uODBg3iwIEDrFixwr5vxYoVjBw5km3bthEVFcWYMWMYNGhQiT7PJaaCX86KV2DFWPAOhEd+hoBIoytyCVarla83HOaZb7ZwNr+IIF8PxvdpSqeEsD9/sYiIlJqr+f4u1XDz22+/0apVK/Lz80vrLUudy4abogL48CY4shHqdoK7vtbtqVK093g2w2ZsZOsR21ICg9rW4qlu9bUmjohIOTFsnRur1UphYWFpvqWUlJsH9PoA3Lxg7zJY95HRFbmUuiFVmP1IW+5NtK2JM231AXq9v5o9x7INrkxERP6o1GdLORpwLGUsJB46P2t7/N0YOLXP2HpcjJe7G8/0aMBHg1pRzc+T7Uez6PHOT3y5NlVr4oiIOBHDG2dKKbv2YYhpBwU5MPcRsBQZXZHLubF+GIuHX0/busGcKyji31//xqMzNpKVqzVxREScwVWFmz+2MfjjduaMlq03nNkMPd8DzyqQsgbWvGd0RS4pLMCbT++7ln93jcfNbGLBb7Y1cTakZBhdmohIpXdVA4rNZrPD205WqxWTyURRkfNeLXDZAcV/tP5jmD8M3DzhwR8hNMHoilzWhpQMhs3YyKGMc7iZTTx+cz0eal8Xs9bEEREpNWU2W2rFihUlGlPToUOHkr5luas04cZqhc/7wu7vIKIp3L/MNuhYykRWbgH/N3szC347CkBibDBv9m1GaIDWxBERKQ2GTQWvCCpNuAHIOgrvXwe5p6HDk9Dx/4yuyKVZrVZmrTvEs/O2cq6giGp+nrzepykd64caXZqISIVXZlPBzWYzbm5uDjd396ta9FjKUkAEdH/d9vjH8XB4vbH1uDiTyUTf1tHMf7QdCREBnMrJ555pa3lxwTbyCp33Vq2IiKu5qis333zzzRWPrVmzhrfffhuLxUJubm6pFFcWKtWVm9/NGgRb50D1eHjwB/C4tMGolK7cgiJeXryDaasPANAwMoB3+jWnTkgVYwsTEamgyvW21M6dO3nqqaeYP38+/fv354UXXnDY+NJolTLcnD1luz2VnQ5thkKXl4yuqNJI2pbOE19tIuNsAb6ebjz/j4bc3jJK60GJiFylclmh+MiRIwwePJjGjRtTWFhIcnIyH3/8sVMHm0rLtxr84x3b4zXvwYGfjK2nEuncIIzFw9tzXZ1qnM0v4omvfmPEF8mc0Zo4IiJl5qrDTWZmJk8++SSxsbFs3bqVZcuWMX/+fBo1alQW9UlpqdcFmt8NWGHuw5CnNYnKS3igN5/dfx2jbq6Hm9nEN8lH6P72TySnnja6NBERl3RV4ebVV1+lTp06LFiwgBkzZrB69Wquv/76sqpNSluXsRBYE06nwJKnja6mUnEzmxh6YxxfPngdNar6kHLqLLdPXM2kH/ZisVSqCYsiImXuqhfx8/HxoXPnzri5Xbkb8uzZs0uluLJQKcfcXGz/j/BxD9vjf82CejcbW08llHmugNGzf2PR5jQAro+rzut9mxLqrzVxRESupMwGFA8aNKhEAyGnTp1a0rcsd5U+3AAsfgp+mQhVwuGRNbYxOVKurFYrM9em8vz8reQWWKhexZPxfZpyQ7zWxBERuRwt4ueAwg1QcA4mXQ8nd0Oj2+H2D42uqNLanX6GR2dsZEeabQzU4Otr80SX+ni6q6etiMjFymW2lFRgHj7Q6wMwucGWr2xr4Igh4sL8mTskkQFtbLMMp6zcT++Jq9l/IsfgykREKi6Fm8oqqiVc/5jt8YLH4Ey6sfVUYt4ebrxwWyMm392Sqr4ebD6cya1vr2T2hkNGlyYiUiEp3FRm7f8N4U3g3ClbB/HKdYfS6dzcMJzFw6/nmtrVyMkv4rEvNzHyi2Sy8wqNLk1EpEJRuKnM3D2h1yRw84Rd38LG6UZXVOlFBPowY/B1PHZTPcwmmLPxMLe+vZLfDp02ujQRkQpD4aayC2t4oVv4t6Mh46Cx9QhuZhPDOsXxxYNtiAz05sDJs/R8bxXDZ25kzzEtvigi8mcUbgTaDoPoayH/DHwzBCwWoysSoHWtaiwe3p4eTSOxWOGb5CPc9OaPDP18AzvTFHJERK5EU8HF5uRemNQOCs5C11fguoeMrkgusuVwJu98v5slWy8M/O7WKJxhneJIiNDfYxFxfVrnxgGFGwd+nQKLRoG7Nzz0E1SPM7oi+YNtR7J4d/lu++rGADc3CGNYpzga1Qg0sDIRkbKlcOOAwo0DVit82gv2LYfI5nDPt+ChlgDOaGfaGd75fjcLNx+1T3LrnBDKsE5xNImqamhtIiJlQeHGAYWbP5F5GCa2hdzT0LQf9JwIJWi5IcbYc+wM73y/h/mbjvB7/82O8SEM6xRH85pBxhYnIlKKFG4cULgpgb3LYXpvsBbBTS9C4jCjK5I/sfd4Nu8t38PcjYftIad9vRCGd4qlZYx6h4lIxadw44DCTQn98gEs/jdggn99qe7hFcSBEzm8t3wPszcepuh8ykmMDWZ4p3pcU1shR0QqLoUbBxRuSshqhfnDYcPH4BUA9ydBSLzRVUkJpZw8y/sr9vDV+kMUng8519WpxvBO9WhTN9jg6kRErp7CjQMKN1ehMB8+7QkHV0FQbRj8Pfjqv/4rkkMZZ3l/xV5mrUuloMj2f/VraldjeKc42tYNxqTxVCJSQSjcOKBwc5VyTsCUjnA6BWp3gLu+BjcPo6uSq3Tk9DkmrtjLF2tTyS+yLdLYMiaI4Z3iuD6uukKOiDg9hRsHFG7+grQt8OHNUJAD1zwIt7xqdEXyF6Vl5jLph718/msK+YW2kNMsuirDO8VxQ3yIQo6IOC2FGwcUbv6i7Qvgi/62xz3egpaDDC1H/p5jWblM+mEfn/1ykLzzIadJVCDDboyjU0KoQo6IOB2FGwcUbv6GH16D5f8FszsMmAe1Eo2uSP6mY2dy+d/K/Xy65iDnCooAaBgZwLBOcdzcIEwhR0SchsKNAwo3f4PVCl/dC1tng28wDF4OQTFGVyWl4ER2Hv9buZ9P1hzgbL4t5CREBDDsxli6NAzHbFbIERFjKdw4oHDzN+Wfhand4GgyhDaE+5aAl7/RVUkpOZWTz4c/7ePj1QfJzisEID7Mn0c7xdKtUQRuCjkiYhCFGwcUbkpB5mHbDKrsdIjvDndMB7PZ6KqkFJ0+m89HP+1n6qoDnDkfcmJDq/DojbHc2iRSIUdEyp3CjQMKN6UkdS1M6w5FeXD9KOg0xuiKpAxknitg6qr9fPTTfrJybSGnTnU/ht4Yyz+aRuLuplArIuVD4cYBhZtStGkmzHnQ9rj3h9D4dmPrkTKTlVvAJ6sP8L+f9nP6bAEAtYJ9GdIxll7NayjkiEiZU7hxQOGmlH03Bla/De7ecO+3ENnc6IqkDGXnFfLJmgNM+XEfGedDTs1qvgzpWJd/tojCQyFHRMqIwo0DCjelzFIEM+6E3d+BfyQ8sBz8w42uSspYTl4h038+yOQf93EyJx+AGlV9GNIxlttbRuHprpAjIqVL4cYBhZsykJsJ/7sJTuyEGq1g0ELw8Da6KikHZ/ML+fyXFCb9sI8T2XkARAZ683DHWPq2isLL3c3gCkXEVSjcOKBwU0ZO7oUpN0LuaWhyJ/SaBFoArtLILSg6H3L2cuyMLeSEB3jz8A11uaN1NN4eCjki8vco3DigcFOG9q2AT/8J1iK46QVIHG50RVLOcguK+GJtKhNX7CUtKxeAUH8vHupQl39dW1MhR0T+MoUbBxRuytgvk2HxE4AJ/vUF1OtidEVigLzCIr5cd4iJy/dwJNMWcqpX8WLw9bXp0yqaan6eBlcoIhWNwo0DCjdlzGqFBSNg/TTw9If7kyC0vtFViUHyCy18tf4Q7y3fw+HT5wDwcDPROSGMPq2iaB8XomnkIlIiCjcOKNyUg8J8+LQnHFwFQbVh8PfgW83oqsRABUUW5mw4zKc/H2Tz4Uz7/lB/L3q1qEGfltHEhlYxsEIRcXZX8/1t6H8yPffcc5hMpmJb/fpX/q/8adOmXXK+t7dm5Tgdd0/o+wlUrQkZ+2HWICgqMLoqMZCHm5m+raOZ/2g7Fg+/nnsTa1PNz5NjZ/L44Id9dH7jB3q9v4oZv6ZwJld/V0Tk73E3uoCGDRuSlJRkf+7u7rikgIAAdu7caX9u0owc5+RXHe6cAR/eDPt/gCX/B7e8ZnRV4gQSIgJ4pkcDnupWn+93HOOr9aks33mcjSmn2Zhymufnb6Vbowj6tIziujrB6kguIlfN8HDj7u5OeHjJF30zmUxXdX5eXh55eXn251lZWVdVn/wN4Y3gn5Phi/7w62QIbQCt7jG6KnESnu5mujYKp2ujcI6dyWXuxsN8ue4Qe45lM2fjYeZsPExUkA+9W0Rxe8sooqv5Gl2yiFQQho/k2717N5GRkdSpU4f+/fuTkpLi8Pzs7GxiYmKIjo7mtttuY+vWrQ7PHzduHIGBgfYtOjq6NMuXP5NwK9z4H9vjRaPgwE/G1iNOKdTfmwfa12XpyPbMeaQt/7q2Jv5e7hzKOMdby3Zz/avL+deUn5mz8RDn8ouMLldEnJyhA4oXL15MdnY28fHxHD16lOeff57Dhw+zZcsW/P39Lzl/zZo17N69myZNmpCZmcn48eP58ccf2bp1K1FRUZf9jMtduYmOjtaA4vJktcLX98GWr8Gnmq1FQ1Ato6sSJ5dbUMSSrWl8uS6V1XtP8vu/VP5e7tzaNILbW0bTomZV3ZoWqSQq7Gyp06dPExMTwxtvvMF99933p+cXFBSQkJBAv379ePHFF0v0GZotZZD8szC1GxxNtt2euu878Lo0wIpczqGMs3y9/jBfbUgl9dQ5+/66IX70aRXNP5vXIDRAkwtEXFmFmS31R1WrVqVevXrs2bOnROd7eHjQvHnzEp8vBvL0hX4zoEoYHNsGsx8Ei8XoqqSCiAryZXjnOH4Y1ZEZg6/jny1q4O1hZu/xHF5evIM2L3/PvdPWsnjzUfIL9fdKpLJzqnCTnZ3N3r17iYiIKNH5RUVFbN68ucTni8ECIuHOz8HNC3YuhOUvGV2RVDBms4k2dYN5o28z1j7dmZf/2ZiWMUEUWax8v+MYD3+2gWvHJvH8/K1sO6LJAyKVlaG3pUaNGkWPHj2IiYnhyJEjPPvssyQnJ7Nt2zZCQkIYMGAANWrUYNy4cQC88MILXHfddcTGxnL69Glee+015s6dy/r162nQoEGJPlO3pZzAppkw50Hb494fQuPbja1HKry9x7P5av0hZm84RHrWhTF2DSMD6NMyitua1SBILR9EKrSr+f42dCr4oUOH6NevHydPniQkJIR27drx888/ExISAkBKSgpm84WLSxkZGQwePJi0tDSCgoJo2bIlq1evLnGwESfR9E5I3wqr34ZvhkC1OlCjhdFVSQVWN6QKT3atz+M31WPl7hPMWp9K0rZjbD2SxdYj2xi7aAedG4TSp1U07eNCcNPaOSIuzakGFJcHXblxEpYimNEPdi8B/0jbDCr/kq9fJPJnMnLy+Sb5MLPWH2LrRbeowgK8+GeLKPq0jKJOiFo+iFQUFXa2VHlQuHEiuVnwv85wYifUaAmDFoGHZrxI6dt2JItZ61OZu/EwGWcvtHdoGRNEn5ZRdG8Sgb+3h4EVisifUbhxQOHGyZzcC1NuhNzT0OQO6PUBaN0SKSP5hRaWbU9n1vpDrNh5DMv5f/18PNzo1jicPi2jubZ2NbV8EHFCCjcOKNw4oX0/wKe9wFoEnZ+HdiOMrkgqgWNZuczeeJhZ61LZezzHvj+6mg+3t4imd8saRAWp5YOIs1C4cUDhxkn9OsXWngET9JsJ8V2NrkgqCavVysbU08xal8r8TUfJzisEbBcQ29YNpm+raLo0DMfbw83gSkUqN4UbBxRunJTVCgtGwvqp4OkP9ydBaH2jq5JK5lx+Ed9uPcqsdYdYvfekfb+/tzs9mkbSp2UUzaLV8kHECAo3DijcOLHCfNvtqYM/QVBtGPw9+FYzuiqppFJPneWr9Yf4av0hDp++0PIhLrQKvVtG0a1RODHBfgZWKFK5KNw4oHDj5HJOwpQb4HQK1G4Pd80GN81iEeNYLFZ+3neSWesPsXjLUXILLrR3iAutQucGYXROCKNZdFWtnyNShhRuHFC4qQDSt8KHN0N+NrQeDN3HG12RCABZuQUs/O0oC347wi/7TlFoufDPZ/UqntxYP5TOCWG0i6uOr6eha6SKuByFGwcUbiqIHQthZn/ACre+Ca3uNboikWIyzxXww67jJG1LZ/nOY5zJLbQf83I30y62Op0bhNGpfqg6louUAoUbBxRuKpAfx8P3L4LZHe6eC7WvN7oikcsqKLKwdv8plm5PZ+m2dA5lnCt2vGl0VW5KCKVzgzDiw/w1IFnkL1C4cUDhpgKxWuHr+2HLV+BTzTbAuFpto6sScchqtbIrPZuk80EnOfV0seNRQT50TgjjpgZhXFO7Gh5u5su/kYgUo3DjgMJNBVNwDqZ2gyMbIbQB3PcdePkbXZVIiR3LyuX7HcdI2p7Oyt0nyCu8MCDZ39udG+JD6ZwQyg3xoQT6aPC8yJUo3DigcFMBZR2ByTdAdjrE3wJ3fAZm/deuVDzn8ov4ac8Jkrals2xHOiey8+3H3M0mrqldjc4JttlXNYO1OrLIxRRuHFC4qaAOrYOpt0BRHlz/OHR6xuiKRP4Wi8VK8qHTJG1LJ2l7OrvSs4sdjw/zp3MD2+yrplFV1e9KKj2FGwcUbiqwTV/AnAdsj3t/CI1vN7YekVJ08GQOSduPkbQtnV8PnKKo2DRzLzon2IJOYmx1fDzVCkIqH4UbBxRuKrilz8Cqt8DdG+5ZBDVaGl2RSKnLPFvAil3HWLotnR92HudM3oVp5t4eZtrFhnBTg1BurB9GiL+XgZWKlB+FGwcUbio4SxHM6Ae7l4B/BAxeDgERRlclUmbyCy38uv+UffbVxa0gTCZoFl3VPvsqLrSKppmLy1K4cUDhxgXkZsGHN8HxHbYrN4MWgoeP0VWJlDmr1cqOtDP2cTqbDmUWO16zmq9tQHKDUFrX0jRzcS0KNw4o3LiIU/tgyo1wLgMa94V/Trb9Z6xIJZKelcuy7bZp5j/tOUH+RdPMA7zd6Xi+HUSH+BACvDXNXCo2hRsHFG5cyL4fbF3ErUXQ+TloN9LoikQMcza/kJW7bdPMv99xjJM5xaeZX1cnmM4JoXRKCCO6mqaZS8WjcOOAwo2L+XUKLBoFmKDfTIjvanRFIoYrslhJTs1g6TbbVZ09x4pPM68f7s9NDcLolBBG4xqB6mYuFYLCjQMKNy7GaoUFI2H9VPD0h/uXQmiC0VWJOJX9J3JYdn5A8toDp7holjlVfT1oUyeYxNjqtIutTkywrwYli1NSuHFA4cYFFRXAJz3h4E8QVMs2g8q3mtFViTiljJx8Vuw6RtK2Y/yw6zjZF00zB6hR1YfEWFvYaVu3uqaai9NQuHFA4cZF5ZyEKR3h9EGodT3cPQfcNIBSxJHCIgubDmWyes8Jftpzgg0pGRQUFf9KqB/ub7+qc03tavh5uRtUrVR2CjcOKNy4sPRttini+dnQ+n7o/rrRFYlUKGfzC1l7IINVe07w0+4TbDuaVey4u9lE85pV7WGnaXRVTTeXcqNw44DCjYvbsQhm/guwQvc3oPV9RlckUmGdzM5jzb6TtrCz5wSpp84VO+7n6ca158frJMYGEx/mr/E6UmYUbhxQuKkEVr4Oy14AszvcPRdqX290RSIuIeXkWVbttQWd1XtOkHG2oNjx6lW87ON1EmOrU6OqFteU0qNw44DCTSVgtcLswbB5Frj7QIu7oc1QCIoxujIRl2GxWNmelnX+qs5Jft1/ktwCS7Fzalf3IzE2mHax1WlTpzqBvhoHJ3+dwo0DCjeVRME5+PwO2P+D7bnJDRr9ExKHQ3hjY2sTcUF5hUVsTDltv4X126HMYp3NTSZoXCPQdlWnbnVa1QrC20PdzaXkFG4cULipRKxWW7j5aQLsW35hf91O0G6EbVaVxgeIlIms3AJ+2XeKVXtOsGrPCXb/YSFBT3czrWsF2cNOIy0mKH9C4cYBhZtK6kgyrH4bts4B6/lL55EtbCGn/q1g1n9BipSl9Kxc+1WdVXtOkJ6VV+x4gLc7betWt4/ZqV3dT4OTpRiFGwcUbiq5U/thzbuwcToU5tr2VasLicOgyZ3g4W1sfSKVgNVqZe/xHHvY+XnfSc7kFl9MMDLQm7bnp5y3jQ0m1F//36zsFG4cULgRALKPw68f2HpT5Z627asSBtc+ZJs+7h1oaHkilUlhkYXNhzPP38I6yfqDGeQXFR+cHB/mT9vzg5OvrRNMFS0mWOko3DigcCPF5GXDho9hzXuQddi2z9MfWt0D1z0CARHG1idSCZ3LL2LtgVOs2mu7hbX1SBYXf1O5m000i65qv7LTLLoqnu5aTNDVKdw4oHAjl1VUAJu/glVvwfHttn1untDkDtsMq+pxxtYnUomdyslnzd6T9rBz8OTZYsd9Pd1oFl2VljFBtIgJokV0kKaduyCFGwcUbsQhiwV2fwerJkDKmvM7TVC/O7QbCVGtjKxORIDUU2dZvde2vs7qPSc4mZN/yTlxoVXsYadlTBB1NEC5wlO4cUDhRkos5RdbyNm56MK+mHa2KzlxN2kauYgTsFis7D6WzfqDGaw/mMGGlAz2n8i55LwgXw9a1LwQdppGVcXHU7MkKxKFGwcUbuSqHd8Jq96G374Ay/nl5kMb2kJOo3+q+7iIkzmRncfGlNO2sHMwg02HTpNXWHyAsrvZRIPIAFrUtIWdljFBRKpdhFNTuHFA4Ub+sszD8PP7sH6arfM4QGC0rbVDi7vB08/Q8kTk8vILLWw7mmUPO+sOnrpknR2AiEBv25Wd84GnQWSAup47EYUbBxRu5G87lwFrP4RfJkHOcds+n2pwzQO2zS/Y2PpExCGr1cqRzFx72Fl/MINtR7OKtYsA8PYw0yTKNlC55flbWtX8PA2qWhRuHFC4kVJTcA6SP4fV70DGfts+NeoUqZDO5heyKTWTDSkZ9vE7mecKLjmvTnU/+7idljFBxIZUway2EeVC4cYBhRspdZYi2D7P1sPqaLJtnxp1ilRoFouVfSdy7Fd21qdksOcP/bEA/L3di43baRpdVQsMlhGFGwcUbqTMXKlRZ2xnSBwBtdpphpVIBXb6bL59oPL6gxkkp57mXEFRsXPMJqgfHmAPOy1jgogK8tE09FKgcOOAwo2UiyPJtgUBt81Vo04RF1VYZGFH2hl72Fl/MIPDp89dcl6Iv5d9kHKLmCAa1QjAy13/BlwthRsHFG6kXF2uUWdwLLR9FJr2A3cvY+sTkVKVlplbbNzO1iOZFBQV/5r1dDPTOCqQFjUvrKqsxqB/TuHGAYUbMcSVGnVe9zC0uleNOkVcVG5BEZsPZ15YZPBgxmVXVI6u5kPLmrYxO41rBNIgMgBfT43duViFCTfPPfcczz//fLF98fHx7Nix44qvmTVrFmPGjOHAgQPExcXxyiuvcMstt5T4MxVuxFBq1ClSqVmtVg6ePGsfpLzhYAY708/wx29iswnqhlShcY1AGkcFKvBQwcLNV199RVJSkn2fu7s71atXv+z5q1evpn379owbN45bb72Vzz//nFdeeYUNGzbQqFGjEn2mwo04hcJ82PK1GnWKCFm5BSSnnGZDSgabD2Wy+XAmx85cusjgxYGnUY1AmkRVrsBTocLN3LlzSU5OLtH5d9xxBzk5OSxYsMC+77rrrqNZs2ZMmjSpRO+hcCNORY06ReQyjmXlsvmwLego8Nhczfe34T/97t27iYyMxNvbmzZt2jBu3Dhq1qx52XPXrFnDY489Vmxfly5dmDt37hXfPy8vj7y8C38hsrKySqVukVJhNkN8V9t2caPOHQtsW0w72wyr2M6aRi5SiYQGeNMpwJtOCWH2fRcHni2HM/ntkC3w7D6Wze5j2czeaLvVXZkCz5UY+pNee+21TJs2jfj4eI4ePcrzzz/P9ddfz5YtW/D397/k/LS0NMLCwortCwsLIy0t7YqfMW7cuEvG9Yg4pZrXQs0ZcGyHbdXj376Agz/ZtpAEaDHAdttK7R1EKqWSBJ7NhzNJz/rzwNM4KpAGEQH4ueiCg041W+r06dPExMTwxhtvcN99911y3NPTk48//ph+/frZ973//vs8//zzpKenX/Y9L3flJjo6WrelxPldrlGn2QPiu0Hzu6HujeDmmv8wichfd6XA80cVLfBUqNtSF6tatSr16tVjz549lz0eHh5+SYhJT08nPDz8iu/p5eWFl5fWEpEKKLAGdHkJ2j8BW76yrZVzZKOt1cP2eeAfYVsrp/ldEFzX6GpFxEnoCo+TXbnJzs6mZs2aPPfccwwbNuyS43fccQdnz55l/vz59n1t27alSZMmGlAslUPaFkj+DDbNhHOnLuyv2dYWchrcBl5VjKtPRCqMkl7hMZkg1gkCT4WZLTVq1Ch69OhBTEwMR44c4dlnnyU5OZlt27YREhLCgAEDqFGjBuPGjQNsU8E7dOjAyy+/TPfu3Zk5cyZjx47VVHCpfArzYddi29WcPUkXWjx4+EGjXrbbVtHXahCyiFyVY2dy7YOVnS3wVJhwc+edd/Ljjz9y8uRJQkJCaNeuHS+99BJ169ousd9www3UqlWLadOm2V8za9Ys/vOf/9gX8Xv11Ve1iJ9UbllHbFdyNk6HU3sv7A+OtV3NadoP/K9861ZExJHfA8/mQ1lsPnzaYeCpG1KFJjUCaRETxF3XxZRqHRUm3BhB4UZcltUKKT/bQs7WOVCQY9tvcoO4m2xBJ64LuHsaW6eIVHh/FniaRgXyzdB2pfqZCjcOKNxIpZB3BrbOtQWd1J8v7PetbptO3vwuCGtgWHki4nouDjzBVTx15aY8KdxIpXNity3kbJoJ2RetCRXZwhZyGvUGn6qGlSciUhIKNw4o3EilVVQIe5fBxk9h52KwFNr2u3tDwj9sQafW9bZVk0VEnIzCjQMKNyJA9nHY/CVs+PRC406AqjWh2V3QrJ/tsYiIk1C4cUDhRuQiVisc2WC7bbX5K8j7vfeaCep0sE0pr98dPHwMLVNEROHGAYUbkSvIP2tr1rlxOuz/4cJ+70Bo3Md22yqimdbOERFDKNw4oHAjUgIZByB5hm015MzUC/vDGtlCTuO+auApIuVK4cYBhRuRq2Cx2K7ibJwO2+dD0fl1LNTAU0TKmcKNAwo3In/RuQzbuJzkz2wNPH+nBp4iUg4UbhxQuBEpBWrgKSLlTOHGAYUbkVJ0pQaenlWgYS9b0FEDTxEpBQo3DijciJSRrCOwacb5Bp77LuxXA08RKQUKNw4o3IiUMasVUtacb+A5t3gDz9jOtnYP8d3AW///E5GSU7hxQOFGpBxdqYGnm5ct6DTsBfFdwcvfsBJFpGJQuHFA4UbEICd222ZbbZ0NJ3Zd2O/mBXE32YJOva4aiCwil6Vw44DCjYjBrFY4ts12RWfrbDi558Ixd+/iQcfTz7AyRcS5KNw4oHAj4kSsVkjfClvn2ILOxQOR3X2g3s22oBN3s4KOSCWncOOAwo2Ik7JaIW3zhaCTceDCMQ9fqNfFFnRibwJPX8PKFBFjKNw4oHAjUgFYrXB00/mgMwdOH7xwzMPPNgi5YS/boGR1LBepFBRuHFC4EalgrFZbu4etc2zjdDJTLhzzrGKbVt6wF9TtBB7ehpUpImVL4cYBhRuRCsxqhcMbbLetts6FrEMXjnn6Q/1bzgedG8Hdy7AyRaT0Kdw4oHAj4iIsFji83nZFZ9tcyDp84ZhXANTvbgs6dTqCu6dhZYpI6VC4cUDhRsQFWSxwaO2FoHPm6IVjXoGQcKst6NTuoKAjUkEp3DigcCPi4iwWSP3lfND5BrLTLhzzrgr1zwedOh3AzcOwMkXk6ijcOKBwI1KJWIog5ecLQSfn2IVjPkEXgk7t9go6Ik5O4cYBhRuRSspSBAdX24LO9nmQc/zCMZ9qkNDDFnRqXQ9u7sbVKSKXpXDjgMKNiNiCzqrzV3TmwdkTF475BkPCP2xBJyZRQUfESSjcOKBwIyLFFBXCwZ8uBJ1zpy4c8wu5KOi0BbObcXWKVHIKNw4o3IjIFRUVwIGV529dzYdzGReO+YVCg9tsQafmdQo6IuVM4cYBhRsRKZGiAtj/w/mgswByT1845lMNqtWBwKjzWzQE1rjw2DcYTCbDShdxRQo3DijciMhVK8wvHnTyMh2f7+4NAReFncCo4uEnoIaaf4pcJYUbBxRuRORvKcyD9K2QeejClnXR4+z0kr2Pb/D5ABR90RWgi55XCdOtL5GLXM33t6YBiIhcDXcvqNHCtl1OYR5kHbly+Mk8BPnZcPakbUv77fLvY3YH/8iLgs8fwk9gFHgHlt3PKVKBKdyIiJQmdy+oVtu2XY7VCrmZF4WdVFtfrGJh6AhYCm0d0C/ugv5Hnv6Ow49/pNpNSKWkcCMiUp5MJvCpatvCG13+HEsRnEm7QvhJhczDtinr+Wfg+HbbdvkPs93eKhaALt40+Flck8KNiIizMbudvwpTA7j28ufk59hCzpXCT+YhKMqz9dbKToPD6y7/Pt6BEBwH1etB9Vjb/wbH2WaD6aqPVFAKNyIiFZGnH4TUs22XY7VCzgnH4Sc7zXaL7PC6S8OPyQ2CYs6HnfOhp/r5EKSrPeLkFG5ERFyRyQRVQmzblQY/F5yDk3vh5G448fu2C07usQ16PrXPtv2Rd9WLwk7chSs/1WqrAak4BYUbEZHKysPHNu7nj2N/rFbbmJ8Tu/4QfHbbrvzknoZDv9q2i5ncbAEnOO5C8Pn9NpdfcLn9WCJa50ZEREou/yyc2nsh7Jw8f7XnxB4oyLny63yqXXqlp3ocBNXS1R4pEa1zIyIiZcPTF8Ib27aLWa22KeyXu8WVmWqb3ZX6i227mNkdgmpfPvj4Viu/n0tcisKNiIj8fSbThRledW4ofiw/xza25/ewc3HwKThrC0Qnd8POP7ynb/Dlb3EF1QK3Mvz6shRBYa5tQcaCcxceF547/7+5UJBbyvtzbf3MgmMhqtX5rTWE1NdK1X+BbkuJiIgxLBY4c+Tyt7iyDl35dWYP29ie32dyVatj21+Y+4dQknfRvqvYbykon5+/JDyrQGRzW9CJagU1WoF/mNFVGUK9pRxQuBERqQDycy66yrP7osHNe2xXPsqL2cPWCNXdyzYA293r/HPvy+y/6LjHRcdLuh+TrW/ZobXnp+dvsM1a+6PAmsWv7oQ3Of9616Zw44DCjYhIBWax2NbtuXhsz+mDtplapRY0Lnps5C0hSxEc3wGH1p0PPOvh2HbgD1/bZg/bGKjfw05UK9s4Jhdbi0jhxgGFGxERqbBys+DIxgth59BayDl+6Xm+wbZbWL9f4YlsYWv5UYFVyHDz8ssvM3r0aIYPH86ECRMue860adO45557iu3z8vIiNze3xJ+jcCMiIi7DaoXTKcXDztFNUJR/6bnV4y+EnRqtILRB2Q7MLmUVbir42rVr+eCDD2jSpMmfnhsQEMDOnReG1Jtc7LKbiIhIiZlMtjYZQTHQ+HbbvsI8SNtyYezOobWQcQBO7LRtyZ/ZzvPwPT9Y+XzYiWoNARGG/SilyfBwk52dTf/+/ZkyZQr//e9///R8k8lEeHh4OVQmIiJSAbl7QVRL2/a7nBO2sTu/h53DGyAvCw6usm2/C6hxYexOjVYQ0dS2tlEFY3i4GTJkCN27d6dz584lCjfZ2dnExMRgsVho0aIFY8eOpWHDhlc8Py8vj7y8PPvzrKysUqlbRESkwvCrDvFdbRvYBmaf2HUh7BxaD8e22gZrbzsM276xnWd2h7CGF8JOVGsIruv0g5UNDTczZ85kw4YNrF27tkTnx8fH89FHH9GkSRMyMzMZP348bdu2ZevWrURFRV32NePGjeP5558vzbJFREQqNrMZQuvbtuZ32fblZdsGKx9ed2GGVna6bQzP0U2w9n+287yrFr+VVaOF060mbdiA4tTUVFq1asXSpUvtY21uuOEGmjVrdsUBxX9UUFBAQkIC/fr148UXX7zsOZe7chMdHa0BxSIiIo5YrZB56KKwsw6OJtsWOvyj4NjzQael7X/DGpZ6z7AKMVtq7ty59OrVCze3C2sIFBUVYTKZMJvN5OXlFTt2JX369MHd3Z0ZM2aU6HM1W0pEROQvKiqA9C0Xws6htbZGqn9UrQ4M21iqH10hZkt16tSJzZs3F9t3zz33UL9+fZ588skSBZuioiI2b97MLbfcUlZlioiIyO/cPGwzrCKbwzWDbfvOnrowDf33QcthjQwt07Bw4+/vT6NGxX94Pz8/goOD7fsHDBhAjRo1GDduHAAvvPAC1113HbGxsZw+fZrXXnuNgwcPcv/995d7/SIiIoJtvE3cTbYNbIOV888YWpLhs6UcSUlJwWw2259nZGQwePBg0tLSCAoKomXLlqxevZoGDRoYWKWIiIjYmc3gHWhoCU6zQnF50ZgbERGRiudqvr/NDo+KiIiIVDAKNyIiIuJSFG5ERETEpSjciIiIiEtRuBERERGXonAjIiIiLkXhRkRERFyKwo2IiIi4FIUbERERcSkKNyIiIuJSFG5ERETEpSjciIiIiEtx6q7gZeH3PqFZWVkGVyIiIiIl9fv3dkn6fVe6cHPmzBkAoqOjDa5ERERErtaZM2cIDAx0eI7JWpII5EIsFgtHjhzB398fk8lkdDlOKSsri+joaFJTU/+0rbyUPf0+nIt+H85HvxPnUla/D6vVypkzZ4iMjMRsdjyqptJduTGbzURFRRldRoUQEBCgfyiciH4fzkW/D+ej34lzKYvfx59dsfmdBhSLiIiIS1G4EREREZeicCOX8PLy4tlnn8XLy8voUgT9PpyNfh/OR78T5+IMv49KN6BYREREXJuu3IiIiIhLUbgRERERl6JwIyIiIi5F4UZERERcisKN2I0bN47WrVvj7+9PaGgoPXv2ZOfOnUaXJcDLL7+MyWRixIgRRpdSqR0+fJi77rqL4OBgfHx8aNy4MevWrTO6rEqpqKiIMWPGULt2bXx8fKhbty4vvvhiifoOyd/3448/0qNHDyIjIzGZTMydO7fYcavVyjPPPENERAQ+Pj507tyZ3bt3l1t9Cjdi98MPPzBkyBB+/vlnli5dSkFBATfffDM5OTlGl1aprV27lg8++IAmTZoYXUqllpGRQWJiIh4eHixevJht27bx+uuvExQUZHRpldIrr7zCxIkTeffdd9m+fTuvvPIKr776Ku+8847RpVUKOTk5NG3alPfee++yx1999VXefvttJk2axC+//IKfnx9dunQhNze3XOrTVHC5ouPHjxMaGsoPP/xA+/btjS6nUsrOzqZFixa8//77/Pe//6VZs2ZMmDDB6LIqpaeeeopVq1axcuVKo0sR4NZbbyUsLIwPP/zQvq937974+Pgwffp0AyurfEwmE3PmzKFnz56A7apNZGQkjz/+OKNGjQIgMzOTsLAwpk2bxp133lnmNenKjVxRZmYmANWqVTO4kspryJAhdO/enc6dOxtdSqU3b948WrVqRZ8+fQgNDaV58+ZMmTLF6LIqrbZt27Js2TJ27doFwKZNm/jpp5/o1q2bwZXJ/v37SUtLK/bvVmBgINdeey1r1qwplxoqXeNMKRmLxcKIESNITEykUaNGRpdTKc2cOZMNGzawdu1ao0sRYN++fUycOJHHHnuM//u//2Pt2rUMGzYMT09PBg4caHR5lc5TTz1FVlYW9evXx83NjaKiIl566SX69+9vdGmVXlpaGgBhYWHF9oeFhdmPlTWFG7msIUOGsGXLFn766SejS6mUUlNTGT58OEuXLsXb29vocgRb4G/VqhVjx44FoHnz5mzZsoVJkyYp3Bjgyy+/5LPPPuPzzz+nYcOGJCcnM2LECCIjI/X7EN2WkksNHTqUBQsWsHz5cqKioowup1Jav349x44do0WLFri7u+Pu7s4PP/zA22+/jbu7O0VFRUaXWOlERETQoEGDYvsSEhJISUkxqKLK7YknnuCpp57izjvvpHHjxtx9992MHDmScePGGV1apRceHg5Aenp6sf3p6en2Y2VN4UbsrFYrQ4cOZc6cOXz//ffUrl3b6JIqrU6dOrF582aSk5PtW6tWrejfvz/Jycm4ubkZXWKlk5iYeMnSCLt27SImJsagiiq3s2fPYjYX/wpzc3PDYrEYVJH8rnbt2oSHh7Ns2TL7vqysLH755RfatGlTLjXotpTYDRkyhM8//5xvvvkGf39/+73RwMBAfHx8DK6ucvH3979krJOfnx/BwcEaA2WQkSNH0rZtW8aOHUvfvn359ddfmTx5MpMnTza6tEqpR48evPTSS9SsWZOGDRuyceNG3njjDe69916jS6sUsrOz2bNnj/35/v37SU5Oplq1atSsWZMRI0bw3//+l7i4OGrXrs2YMWOIjIy0z6gqc1aR84DLblOnTjW6NLFarR06dLAOHz7c6DIqtfnz51sbNWpk9fLystavX986efJko0uqtLKysqzDhw+31qxZ0+rt7W2tU6eO9emnn7bm5eUZXVqlsHz58st+XwwcONBqtVqtFovFOmbMGGtYWJjVy8vL2qlTJ+vOnTvLrT6tcyMiIiIuRWNuRERExKUo3IiIiIhLUbgRERERl6JwIyIiIi5F4UZERERcisKNiIiIuBSFGxEREXEpCjciIiLiUhRuRKRSMplMzJ071+gyRKQMKNyISLkbNGgQJpPpkq1r165GlyYiLkCNM0XEEF27dmXq1KnF9nl5eRlUjYi4El25ERFDeHl5ER4eXmwLCgoCbLeMJk6cSLdu3fDx8aFOnTp89dVXxV6/efNmbrzxRnx8fAgODuaBBx4gOzu72DkfffQRDRs2xMvLi4iICIYOHVrs+IkTJ+jVqxe+vr7ExcUxb948+7GMjAz69+9PSEgIPj4+xMXFXRLGRMQ5KdyIiFMaM2YMvXv3ZtOmTfTv358777yT7du3A5CTk0OXLl0ICgpi7dq1zJo1i6SkpGLhZeLEiQwZMoQHHniAzZs3M2/ePGJjY4t9xvPPP0/fvn357bffuOWWW+jfvz+nTp2yf/62bdtYvHgx27dvZ+LEiVSvXr38/gBE5K8rt/7jIiLnDRw40Orm5mb18/Mrtr300ktWq9VqBawPPfRQsddce+211ocffthqtVqtkydPtgYFBVmzs7PtxxcuXGg1m83WtLQ0q9VqtUZGRlqffvrpK9YAWP/zn//Yn2dnZ1sB6+LFi61Wq9Xao0cP6z333FM6P7CIlCuNuRERQ3Ts2JGJEycW21etWjX74zZt2hQ71qZNG5KTkwHYvn07TZs2xc/Pz348MTERi8XCzp07MZlMHDlyhE6dOjmsoUmTJvbHfn5+BAQEcOzYMQAefvhhevfuzYYNG7j55pvp2bMnbdu2/Us/q4iUL4UbETGEn5/fJbeJSouPj0+JzvPw8Cj23GQyYbFYAOjWrRsHDx5k0aJFLF26lE6dOjFkyBDGjx9f6vWKSOnSmBsRcUo///zzJc8TEhIASEhIYNOmTeTk5NiPr1q1CrPZTHx8PP7+/tSqVYtly5b9rRpCQkIYOHAg06dPZ8KECUyePPlvvZ+IlA9duRERQ+Tl5ZGWllZsn7u7u33Q7qxZs2jVqhXt2rXjs88+49dff+XDDz8EoH///jz77LMMHDiQ5557juPHj/Poo49y9913ExYWBsBzzz3HQw89RGhoKN26dePMmTOsWrWKRx99tET1PfPMM7Rs2ZKGDRuSl5fHggUL7OFKRJybwo2IGOLbb78lIiKi2L74+Hh27NgB2GYyzZw5k0ceeYSIiAhmzJhBgwYNAPD19WXJkiUMHz6c1q1b4+vrS+/evXnjjTfs7zVw4EByc3N58803GTVqFNWrV+f2228vcX2enp6MHj2aAwcO4OPjw/XXX8/MmTNL4ScXkbJmslqtVqOLEBG5mMlkYs6cOfTs2dPoUkSkAtKYGxEREXEpCjciIiLiUjTmRkScju6Wi8jfoSs3IiIi4lIUbkRERMSlKNyIiIiIS1G4EREREZeicCMiIiIuReFGREREXIrCjYiIiLgUhRsRERFxKf8P5k5xU/w7tFYAAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "plt.figure()\n",
        "plt.plot(range(1, trainer.epochs + 1), trainer.train_losses, label='Training losses')\n",
        "plt.plot(range(1, trainer.epochs + 1), trainer.val_losses[0:10], label='Validation losses')\n",
        "plt.xlabel('Epochs')\n",
        "plt.ylabel('NLL')\n",
        "plt.legend()\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BBqjqy-EyU27"
      },
      "source": [
        "# Create handin"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 49,
      "metadata": {
        "id": "hWRyPvWmgLQs",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "011b8af4-19c9-46c3-8ac4-a337888bd755"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/bin/bash: -c: line 0: syntax error near unexpected token `10'\n",
            "/bin/bash: -c: line 0: `make runid=<'1681491014'> epoch=<10>'\n"
          ]
        }
      ],
      "source": [
        "# TODO: Generate the handin to submit to autolab\n",
        "!make runid=<'1681491014'> epoch=<10>"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "machine_shape": "hm",
      "provenance": []
    },
    "deepnote": {},
    "deepnote_execution_queue": [],
    "deepnote_notebook_id": "989a3c3836794109ac641230122845a3",
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.0"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "93ee11dbe97549169a13fd54af2d8ee5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_6e1ee11b061a4efda575aa5ffbdfdba8",
              "IPY_MODEL_e019d72995ae4bc88bc0f6c9ced39110",
              "IPY_MODEL_8ab5ef90439049c7b445c15c8351897f"
            ],
            "layout": "IPY_MODEL_2b2403d043c444138af2273277bb0d15"
          }
        },
        "6e1ee11b061a4efda575aa5ffbdfdba8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_760c0bf783a3425584ecb499fa300c6e",
            "placeholder": "​",
            "style": "IPY_MODEL_b1da510235cc407e987378ac3043e46e",
            "value": "Train: 100%"
          }
        },
        "e019d72995ae4bc88bc0f6c9ced39110": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_08f71349eecc4cf9b7e44231c7e6db27",
            "max": 405,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_c7c830ce84e94d5195c8a4761f0be418",
            "value": 405
          }
        },
        "8ab5ef90439049c7b445c15c8351897f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_733ae01dbd8d4c59b499a05479753336",
            "placeholder": "​",
            "style": "IPY_MODEL_6b038ccf31ca46edad0699f98c1b0cb5",
            "value": " 405/405 [1:05:42&lt;00:00,  5.92s/it, loss=7.4022, lr=0.002000]"
          }
        },
        "2b2403d043c444138af2273277bb0d15": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": "inline-flex",
            "flex": null,
            "flex_flow": "row wrap",
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": "100%"
          }
        },
        "760c0bf783a3425584ecb499fa300c6e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b1da510235cc407e987378ac3043e46e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "08f71349eecc4cf9b7e44231c7e6db27": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": "2",
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c7c830ce84e94d5195c8a4761f0be418": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "733ae01dbd8d4c59b499a05479753336": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6b038ccf31ca46edad0699f98c1b0cb5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "b39c0e91e24c4100a8f79d3ce430384d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_4c1e4a8ac5314f3ba61339cc22bcf05f",
              "IPY_MODEL_e97b519d9fbb46e0af06bbb9c1af21a3",
              "IPY_MODEL_326eb55f4a5b4403b322a211c3ed3625"
            ],
            "layout": "IPY_MODEL_623d5bec94ed44d8b374c0d937e6e8bf"
          }
        },
        "4c1e4a8ac5314f3ba61339cc22bcf05f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_855022913d3f46c394a313dc3bbbffb1",
            "placeholder": "​",
            "style": "IPY_MODEL_87dfe488a08c409ebf252d9e495e3d3f",
            "value": "Train: 100%"
          }
        },
        "e97b519d9fbb46e0af06bbb9c1af21a3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8eefef7fe76f4984b07c16882be741ae",
            "max": 405,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_2361037cf8394e5d9a8debd564123f11",
            "value": 405
          }
        },
        "326eb55f4a5b4403b322a211c3ed3625": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_4023f67468474e2688c2c966e6097f9c",
            "placeholder": "​",
            "style": "IPY_MODEL_02eeb409d66c4619901b8ee4d100fa41",
            "value": " 405/405 [38:14&lt;00:00,  4.62s/it, loss=6.5995, lr=0.002000]"
          }
        },
        "623d5bec94ed44d8b374c0d937e6e8bf": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": "inline-flex",
            "flex": null,
            "flex_flow": "row wrap",
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": "100%"
          }
        },
        "855022913d3f46c394a313dc3bbbffb1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "87dfe488a08c409ebf252d9e495e3d3f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "8eefef7fe76f4984b07c16882be741ae": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": "2",
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2361037cf8394e5d9a8debd564123f11": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "4023f67468474e2688c2c966e6097f9c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "02eeb409d66c4619901b8ee4d100fa41": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "38a9691c40d6407ab44288c23ff43834": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_9abe806a1b9d4df7aed868e04d6b8705",
              "IPY_MODEL_45e19ea7f04e4adc97261227d4f6ac3e",
              "IPY_MODEL_65d247ebba67448d956f4affa491d3fa"
            ],
            "layout": "IPY_MODEL_8ca83d4b103b413fb2c1c4481e8de366"
          }
        },
        "9abe806a1b9d4df7aed868e04d6b8705": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_4a42be20ea574b2aadfc2c454e8b66cc",
            "placeholder": "​",
            "style": "IPY_MODEL_48832de761264093a303e87e99790a2d",
            "value": "Train: 100%"
          }
        },
        "45e19ea7f04e4adc97261227d4f6ac3e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c1bdbeee2f204978bce00ca0d6b8de8b",
            "max": 405,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_f435a7abb0aa4e3a9cf7860ddc941a06",
            "value": 405
          }
        },
        "65d247ebba67448d956f4affa491d3fa": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6635f96c58d04e36bf7f23432720e066",
            "placeholder": "​",
            "style": "IPY_MODEL_3326e933cd7447479a1142170d5410c7",
            "value": " 405/405 [30:20&lt;00:00,  4.39s/it, loss=6.0736, lr=0.002000]"
          }
        },
        "8ca83d4b103b413fb2c1c4481e8de366": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": "inline-flex",
            "flex": null,
            "flex_flow": "row wrap",
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": "100%"
          }
        },
        "4a42be20ea574b2aadfc2c454e8b66cc": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "48832de761264093a303e87e99790a2d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c1bdbeee2f204978bce00ca0d6b8de8b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": "2",
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f435a7abb0aa4e3a9cf7860ddc941a06": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "6635f96c58d04e36bf7f23432720e066": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "3326e933cd7447479a1142170d5410c7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "7a4565f23db34c4e8fcaa0676ffa723a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_4bf87b575c1847159090a137dff72066",
              "IPY_MODEL_d8a70235a5734c389cf0eefaeedb442d",
              "IPY_MODEL_d937f515da9d402eae8f0bd44c72125b"
            ],
            "layout": "IPY_MODEL_c3045f0beed24347b7d8fd9f914b8a18"
          }
        },
        "4bf87b575c1847159090a137dff72066": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c5beb7ae4f6347ba861767cdea27a52f",
            "placeholder": "​",
            "style": "IPY_MODEL_dc903ee667c344ce9d6f27166569ad02",
            "value": "Train: 100%"
          }
        },
        "d8a70235a5734c389cf0eefaeedb442d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_059539cda3b9443d80e7c2f153de983c",
            "max": 405,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_32ec551d9d254d44ad282baa738a64b3",
            "value": 405
          }
        },
        "d937f515da9d402eae8f0bd44c72125b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c386a23d25f94581834bca85d7bdc8d1",
            "placeholder": "​",
            "style": "IPY_MODEL_b3a09c2cbc4d4724a6bb9d5bb7ae63c0",
            "value": " 405/405 [30:28&lt;00:00,  4.64s/it, loss=5.7556, lr=0.002000]"
          }
        },
        "c3045f0beed24347b7d8fd9f914b8a18": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": "inline-flex",
            "flex": null,
            "flex_flow": "row wrap",
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": "100%"
          }
        },
        "c5beb7ae4f6347ba861767cdea27a52f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "dc903ee667c344ce9d6f27166569ad02": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "059539cda3b9443d80e7c2f153de983c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": "2",
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "32ec551d9d254d44ad282baa738a64b3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "c386a23d25f94581834bca85d7bdc8d1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b3a09c2cbc4d4724a6bb9d5bb7ae63c0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "bb07bd41fff0481b9c170c8cc0fb4d1b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_e27deb279b5146ca980c5a725fa76541",
              "IPY_MODEL_931802f4fbe64de3b9f57ba857e97abd",
              "IPY_MODEL_39f8e59519f2429c806658408eb6ecdc"
            ],
            "layout": "IPY_MODEL_121938fecaed4e02a9bab874195a2bbe"
          }
        },
        "e27deb279b5146ca980c5a725fa76541": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_51dd96681212465e81829e2ae13a3010",
            "placeholder": "​",
            "style": "IPY_MODEL_af10b57e94b64d16993c4e156b1d0e00",
            "value": "Train: 100%"
          }
        },
        "931802f4fbe64de3b9f57ba857e97abd": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a673851a3c544d738f6bd75baa1d5824",
            "max": 405,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_33d6d69541df445192cd4820dbc37743",
            "value": 405
          }
        },
        "39f8e59519f2429c806658408eb6ecdc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2b5b8506f1a84ba68b2b67ee05b3907b",
            "placeholder": "​",
            "style": "IPY_MODEL_4559c9f8031844e9b615533f6e0451ac",
            "value": " 405/405 [32:11&lt;00:00,  8.73s/it, loss=5.5400, lr=0.002000]"
          }
        },
        "121938fecaed4e02a9bab874195a2bbe": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": "inline-flex",
            "flex": null,
            "flex_flow": "row wrap",
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": "100%"
          }
        },
        "51dd96681212465e81829e2ae13a3010": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "af10b57e94b64d16993c4e156b1d0e00": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "a673851a3c544d738f6bd75baa1d5824": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": "2",
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "33d6d69541df445192cd4820dbc37743": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "2b5b8506f1a84ba68b2b67ee05b3907b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4559c9f8031844e9b615533f6e0451ac": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "28e2ea7ddc784911818e1aabd2e23dd3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_ae90f7ec8e9049ea8b9d207acaf8a19a",
              "IPY_MODEL_d632b0eac9884acdbd102654055400e7",
              "IPY_MODEL_c986d44f82724c3a913088c1bae4480c"
            ],
            "layout": "IPY_MODEL_5b73d8943e954a2cad3304e939af33c7"
          }
        },
        "ae90f7ec8e9049ea8b9d207acaf8a19a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a0f3c56684e646e79b40c4992305fb91",
            "placeholder": "​",
            "style": "IPY_MODEL_9f7274943cbb4110ba5231b6a016066f",
            "value": "Train: 100%"
          }
        },
        "d632b0eac9884acdbd102654055400e7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_9749e109aecb42ce856b972ad9cbeb88",
            "max": 405,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_8e636da2803d4b948f0b0c23afbe69b0",
            "value": 405
          }
        },
        "c986d44f82724c3a913088c1bae4480c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_2db0b30dfba748bc99666cdc19252caf",
            "placeholder": "​",
            "style": "IPY_MODEL_fead4e0fe003409d82475b68deb86cf2",
            "value": " 405/405 [32:02&lt;00:00,  4.92s/it, loss=5.3743, lr=0.002000]"
          }
        },
        "5b73d8943e954a2cad3304e939af33c7": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": "inline-flex",
            "flex": null,
            "flex_flow": "row wrap",
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": "100%"
          }
        },
        "a0f3c56684e646e79b40c4992305fb91": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9f7274943cbb4110ba5231b6a016066f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "9749e109aecb42ce856b972ad9cbeb88": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": "2",
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8e636da2803d4b948f0b0c23afbe69b0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "2db0b30dfba748bc99666cdc19252caf": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "fead4e0fe003409d82475b68deb86cf2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "48f462d7a4a749d28363c5991de793f5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_e4a7ba6ef3004501aa63ec5d19e9b75c",
              "IPY_MODEL_798997169cf8436bba6c17ab613efdca",
              "IPY_MODEL_d7e1da29d44d4b24b08db7961244f131"
            ],
            "layout": "IPY_MODEL_9848631abd4143139a86b6b182a73c84"
          }
        },
        "e4a7ba6ef3004501aa63ec5d19e9b75c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d7c1d8434d9547efb4201857d632209f",
            "placeholder": "​",
            "style": "IPY_MODEL_49deff179c10429f9931099aa36c66f0",
            "value": "Train: 100%"
          }
        },
        "798997169cf8436bba6c17ab613efdca": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_3e5e22b057fd400d806ae9c7ac7379aa",
            "max": 405,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_df1cff61af98433a87edc71930dc2b77",
            "value": 405
          }
        },
        "d7e1da29d44d4b24b08db7961244f131": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_721ea819258c4869a1b245787e488391",
            "placeholder": "​",
            "style": "IPY_MODEL_627b7e59dd9b41da832b11c514d725ce",
            "value": " 405/405 [31:55&lt;00:00,  4.74s/it, loss=5.2412, lr=0.002000]"
          }
        },
        "9848631abd4143139a86b6b182a73c84": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": "inline-flex",
            "flex": null,
            "flex_flow": "row wrap",
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": "100%"
          }
        },
        "d7c1d8434d9547efb4201857d632209f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "49deff179c10429f9931099aa36c66f0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "3e5e22b057fd400d806ae9c7ac7379aa": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": "2",
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "df1cff61af98433a87edc71930dc2b77": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "721ea819258c4869a1b245787e488391": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "627b7e59dd9b41da832b11c514d725ce": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "1bc3421283cd488cbfb7c9898de3c9d9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_d58d21a8bff14863a157c3792872b020",
              "IPY_MODEL_32c3cff66e434df98273286546238567",
              "IPY_MODEL_a3f8aee9bb3c44ae99917b5493b3fa2d"
            ],
            "layout": "IPY_MODEL_00303b9fdd5242888a4f0ce7d32ef0be"
          }
        },
        "d58d21a8bff14863a157c3792872b020": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_56399144f20a4f179fd992acd727033c",
            "placeholder": "​",
            "style": "IPY_MODEL_0eb29c2af9564bea86cc9cb381170f5e",
            "value": "Train: 100%"
          }
        },
        "32c3cff66e434df98273286546238567": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f95e0391f058449982dff354199a7445",
            "max": 405,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_997a2cef0f05460ca29b2c805f00d3a1",
            "value": 405
          }
        },
        "a3f8aee9bb3c44ae99917b5493b3fa2d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a3e64cc0751043f3b3794fe6e424584c",
            "placeholder": "​",
            "style": "IPY_MODEL_aaf8e6aeccd241c28e796deb839136c0",
            "value": " 405/405 [31:39&lt;00:00,  4.72s/it, loss=5.1257, lr=0.002000]"
          }
        },
        "00303b9fdd5242888a4f0ce7d32ef0be": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": "inline-flex",
            "flex": null,
            "flex_flow": "row wrap",
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": "100%"
          }
        },
        "56399144f20a4f179fd992acd727033c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0eb29c2af9564bea86cc9cb381170f5e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "f95e0391f058449982dff354199a7445": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": "2",
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "997a2cef0f05460ca29b2c805f00d3a1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "a3e64cc0751043f3b3794fe6e424584c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "aaf8e6aeccd241c28e796deb839136c0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "99dfc32cc70f45c08fd45afd269b16ca": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_d8d44fcdf2ff46e9826ce3dde43066d8",
              "IPY_MODEL_4dcb21d623fa4fcda2d8e6e3b4e6acac",
              "IPY_MODEL_b6f2d6465ff2428a88ee78bac1db34f3"
            ],
            "layout": "IPY_MODEL_9b6c87ad1bd34d68b28ab8ffd9d13f35"
          }
        },
        "d8d44fcdf2ff46e9826ce3dde43066d8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c1d5058d103d4376ba48f394632ffe25",
            "placeholder": "​",
            "style": "IPY_MODEL_d99f679168d54ff899a001a980b68f64",
            "value": "Train: 100%"
          }
        },
        "4dcb21d623fa4fcda2d8e6e3b4e6acac": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_731f486356f0489f82c9d402c9291812",
            "max": 405,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_ed81a8feef46408fb6d31620720d200b",
            "value": 405
          }
        },
        "b6f2d6465ff2428a88ee78bac1db34f3": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b54ea61c38684b29999db3dcb88e265f",
            "placeholder": "​",
            "style": "IPY_MODEL_15894500fddf4dd7b7aa5adf404de539",
            "value": " 405/405 [31:59&lt;00:00,  4.71s/it, loss=5.0268, lr=0.002000]"
          }
        },
        "9b6c87ad1bd34d68b28ab8ffd9d13f35": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": "inline-flex",
            "flex": null,
            "flex_flow": "row wrap",
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": "100%"
          }
        },
        "c1d5058d103d4376ba48f394632ffe25": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d99f679168d54ff899a001a980b68f64": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "731f486356f0489f82c9d402c9291812": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": "2",
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ed81a8feef46408fb6d31620720d200b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "b54ea61c38684b29999db3dcb88e265f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "15894500fddf4dd7b7aa5adf404de539": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "270e999344564b279594c506106d95d5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_e9d99aad68694b7d92c5cee850a48b96",
              "IPY_MODEL_3c9e259fcd544040886f5567388ecdee",
              "IPY_MODEL_db3e7cb5fcf54f688d6ca4505e420b95"
            ],
            "layout": "IPY_MODEL_26e14138e615440ca413843afd11a1d5"
          }
        },
        "e9d99aad68694b7d92c5cee850a48b96": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_fdfbeaf191ca48009a2d3b030481d27d",
            "placeholder": "​",
            "style": "IPY_MODEL_e860218667d14311b5d849cfe1be7761",
            "value": "Train: 100%"
          }
        },
        "3c9e259fcd544040886f5567388ecdee": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_bb17114fe6734d708c0aef1732130658",
            "max": 405,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_7b91ecf6c5c348879965a47ce2a9f37b",
            "value": 405
          }
        },
        "db3e7cb5fcf54f688d6ca4505e420b95": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_bb2c9a401f414d059722576203875b9c",
            "placeholder": "​",
            "style": "IPY_MODEL_1324940779854f1891135c3a0731380c",
            "value": " 405/405 [31:20&lt;00:00,  4.68s/it, loss=4.9395, lr=0.002000]"
          }
        },
        "26e14138e615440ca413843afd11a1d5": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": "inline-flex",
            "flex": null,
            "flex_flow": "row wrap",
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": "hidden",
            "width": "100%"
          }
        },
        "fdfbeaf191ca48009a2d3b030481d27d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e860218667d14311b5d849cfe1be7761": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "bb17114fe6734d708c0aef1732130658": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": "2",
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7b91ecf6c5c348879965a47ce2a9f37b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "bb2c9a401f414d059722576203875b9c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1324940779854f1891135c3a0731380c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "fd9bbdaf6c35409196d1178d87e40264": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_9561207d2a3a441ba6c6360b0c2aad30",
              "IPY_MODEL_2fec87e5a8d04af8919a065834105aa8",
              "IPY_MODEL_08c93e89970f4e81a647e8d6ed68e550"
            ],
            "layout": "IPY_MODEL_445ed794c95e40c3b31a13f5c2533e7e"
          }
        },
        "9561207d2a3a441ba6c6360b0c2aad30": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_4cfa58f903c04988bf3ca887afee8943",
            "placeholder": "​",
            "style": "IPY_MODEL_6d67253376fb4b3ab8d8d6e4c0cdf99f",
            "value": "Train:  90%"
          }
        },
        "2fec87e5a8d04af8919a065834105aa8": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e4ba91850f3d426eb4434d7ac6ea0f6b",
            "max": 405,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_2e4a82150f4949e9a967c56058e7da8b",
            "value": 364
          }
        },
        "08c93e89970f4e81a647e8d6ed68e550": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_6d9be590fd7a4073882a62ad1cbbe36a",
            "placeholder": "​",
            "style": "IPY_MODEL_6c348830b7004f9db744787650c9311b",
            "value": " 364/405 [28:02&lt;03:05,  4.52s/it, loss=4.8556, lr=0.002000]"
          }
        },
        "445ed794c95e40c3b31a13f5c2533e7e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": "inline-flex",
            "flex": null,
            "flex_flow": "row wrap",
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": "100%"
          }
        },
        "4cfa58f903c04988bf3ca887afee8943": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6d67253376fb4b3ab8d8d6e4c0cdf99f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e4ba91850f3d426eb4434d7ac6ea0f6b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": "2",
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2e4a82150f4949e9a967c56058e7da8b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "6d9be590fd7a4073882a62ad1cbbe36a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6c348830b7004f9db744787650c9311b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}