Neural_Machine_Translation_tf_2L_4H.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Neural_Machine_Translation_tf_2L_4H.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "gpuClass": "standard",
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "#### Mounting Google Drive"
      ],
      "metadata": {
        "id": "0dqNGaFLKKAJ"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bVenoImvJZs9"
      },
      "outputs": [],
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "cd \"/content/drive/MyDrive/IASNLP\""
      ],
      "metadata": {
        "id": "iNpRDc1-KMj8"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Importing Necessary Libraries"
      ],
      "metadata": {
        "id": "OlMuBUdIApMX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install sentencepiece\n",
        "!pip install sacrebleu\n",
        "!pip install tensorflow-gpu # ---> For GPU"
      ],
      "metadata": {
        "id": "Iwi6TDKsB0St"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "from tensorflow.keras import layers\n",
        "from tensorflow.keras import Model\n",
        "from tensorflow.keras import Sequential\n",
        "from tensorflow.keras.layers import Dense, Lambda, Layer, Embedding, LayerNormalization\n",
        "import tensorflow_probability as tfp\n",
        "import sentencepiece as spm\n",
        "from sacrebleu.metrics import BLEU, CHRF, TER\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "from collections import Counter"
      ],
      "metadata": {
        "id": "DXSYYy2b-jMM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Loading Data"
      ],
      "metadata": {
        "id": "ToLMSNAHBnvJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We load the `train_data`, `train_dev_data`, `test_val_data` and `test_data` as well as the byte-pair encoder tokenizer for English and Bengali i.e. `sp_en_bpe` and `s_ben_bpe`."
      ],
      "metadata": {
        "id": "zXIf-FL7IG7f"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "train_data, train_dev_data = pd.read_csv(\"train_data.csv\")[['src', 'tgt']], pd.read_csv(\"train_dev.csv\")[['src', 'tgt']] \n",
        "test_val_data, test_data = pd.read_csv(\"test_val.csv\")[['src', 'tgt']], pd.read_csv(\"test_data.csv\")[['src', 'tgt']]\n",
        "sp_en_bpe, sp_ben_bpe = spm.SentencePieceProcessor(), spm.SentencePieceProcessor()\n",
        "sp_en_bpe.load('eng_bpe.model'); sp_ben_bpe.load('ben_bpe.model');"
      ],
      "metadata": {
        "id": "IXsCg4mBBeoo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Data Preprocessing"
      ],
      "metadata": {
        "id": "auf4TlPFhNtn"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We decide on some formats of the data and batches. We take the maximum tokens we can have per tokenized English and Bengali sentence to be `MAX_TOKENS = 60`. We use the vocabulary size of `32000`(i.e. `ENCODER_VOCAB = DECODER_VOCAB = 32000`). A batch size of `256` was choosen for our Mini-Batch Gradient Descent which is used to train our Model."
      ],
      "metadata": {
        "id": "3z7BUnn_G3WL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "MAX_TOKENS = 60\n",
        "VOCAB_SIZE = 32000\n",
        "BATCH_SIZE = 256\n",
        "BUFFER_SIZE = BATCH_SIZE*4"
      ],
      "metadata": {
        "id": "7jTuQSqOAybO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The following two functions help in tokenizing an detokenizing the data based on our sentencepiece byte-pair encoding model."
      ],
      "metadata": {
        "id": "NlEW7GndbCfw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def tokenize(sentence, sp_model, add_bos = True, add_eos = True):\n",
        "    bos = [sp_model.bos_id()] if add_bos else []\n",
        "    eos = [sp_model.eos_id()] if add_eos else []\n",
        "    inputs = bos + sp_model.encode_as_ids(sentence) + eos\n",
        "    return np.reshape(np.array(inputs), [1, -1])\n",
        "def detokenize(tokenized, sp_model, is_bos = True, is_eos = True):\n",
        "    integers = np.squeeze(tokenized).tolist()\n",
        "    if is_eos:\n",
        "        return sp_model.DecodeIdsWithCheck(integers[int(is_bos):integers.index(sp_model.eos_id())])\n",
        "    else:\n",
        "        if sp_model.pad_id() in tokenized:\n",
        "            return sp_model.DecodeIdsWithCheck(integers[int(is_bos):integers.index(sp_model.pad_id())])\n",
        "        else:\n",
        "            return sp_model.DecodeIdsWithCheck(integers[int(is_bos):])"
      ],
      "metadata": {
        "id": "VC-6hav2BrJJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We have tokenized the data."
      ],
      "metadata": {
        "id": "5-76mwBnbuzp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Tokenization\n",
        "src_train_data_enc = [np.squeeze(tokenize(train_data['src'].iloc[i], sp_en_bpe)) for i in range(train_data.shape[0])]\n",
        "tgt_train_data_enc = [np.squeeze(tokenize(train_data['tgt'].iloc[i], sp_ben_bpe)) for i in range(train_data.shape[0])]\n",
        "src_train_dev_data_enc = [np.squeeze(tokenize(train_dev_data['src'].iloc[i], sp_en_bpe)) for i in range(train_dev_data.shape[0])]\n",
        "tgt_train_dev_data_enc = [np.squeeze(tokenize(train_dev_data['tgt'].iloc[i], sp_ben_bpe)) for i in range(train_dev_data.shape[0])]\n",
        "src_test_val_data_enc = [np.squeeze(tokenize(test_val_data['src'].iloc[i], sp_en_bpe)) for i in range(test_val_data.shape[0])]\n",
        "tgt_test_val_data_enc = [np.squeeze(tokenize(test_val_data['tgt'].iloc[i], sp_ben_bpe)) for i in range(test_val_data.shape[0])]"
      ],
      "metadata": {
        "id": "nT0QpepuDoFF"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We have padded the data with post-truncation and post padding upto MAX_TOKENS number of tokens."
      ],
      "metadata": {
        "id": "wdyEjbm5hWXg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Padding\n",
        "# The reason why we add MAX_TOKENS+1 after-pad length for Bengali is that when we form the tf.Dataset later, we will shift the target sentence once and will truncate the last token once\n",
        "# Hence, it will get adjusted\n",
        "train_src = tf.keras.preprocessing.sequence.pad_sequences(src_train_data_enc, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "train_tgt = tf.keras.preprocessing.sequence.pad_sequences(tgt_train_data_enc, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "train_dev_src = tf.keras.preprocessing.sequence.pad_sequences(src_train_dev_data_enc, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "train_dev_tgt = tf.keras.preprocessing.sequence.pad_sequences(tgt_train_dev_data_enc, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "test_val_src = tf.keras.preprocessing.sequence.pad_sequences(src_test_val_data_enc, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "test_val_tgt = tf.keras.preprocessing.sequence.pad_sequences(tgt_test_val_data_enc, maxlen = MAX_TOKENS, padding='post', truncating='post')"
      ],
      "metadata": {
        "id": "NT63vC8PcK-2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Below, we discard all those pairs where the tokenized source or target sentences are larger than MAX_TOKENS number of tokens."
      ],
      "metadata": {
        "id": "FvxtnfKniMRA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Removing all pairs whose either the target sentence or source setence exceeds MAX_TOKENS number of tokens\n",
        "same_idx = list(set([i for i in range(train_src.shape[0]) if train_src[i][-1] == 1 or train_src[i][-1] == 0]).intersection(set([i for i in range(train_tgt.shape[0]) if train_tgt[i][-1] == 1 or train_tgt[i][-1] == 0])))\n",
        "train_src = train_src[same_idx]\n",
        "train_tgt = train_tgt[same_idx]\n",
        "same_idx = list(set([i for i in range(train_dev_src.shape[0]) if train_dev_src[i][-1] == 1 or train_dev_src[i][-1] == 0]).intersection(set([i for i in range(train_dev_tgt.shape[0]) if train_dev_tgt[i][-1] == 1 or train_dev_tgt[i][-1] == 0])))\n",
        "train_dev_src = train_dev_src[same_idx]\n",
        "train_dev_tgt = train_dev_tgt[same_idx]\n",
        "same_idx = list(set([i for i in range(test_val_src.shape[0]) if test_val_src[i][-1] == 1 or test_val_src[i][-1] == 0]).intersection(set([i for i in range(test_val_tgt.shape[0]) if test_val_tgt[i][-1] == 1 or test_val_tgt[i][-1] == 0])))\n",
        "test_val_src = test_val_src[same_idx]\n",
        "test_val_tgt = test_val_tgt[same_idx]"
      ],
      "metadata": {
        "id": "L6-cV6zpcO5e"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "train_labels = np.zeros(train_tgt.shape)\n",
        "train_labels[:,0:train_tgt.shape[1] -1] = train_tgt[:,1:]\n",
        "train_dev_labels = np.zeros(train_dev_tgt.shape)\n",
        "train_dev_labels[:,0:train_dev_tgt.shape[1] -1] = train_dev_tgt[:,1:]\n",
        "test_val_labels = np.zeros(test_val_tgt.shape)\n",
        "test_val_labels[:,0:test_val_tgt.shape[1] -1] = test_val_tgt[:,1:]"
      ],
      "metadata": {
        "id": "JGMvPvIxOx-0"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"*** Preprocessed Data ***\")\n",
        "print(\"Train Data Shape(Source): \", train_src.shape)\n",
        "print(\"Train Data Shape(Target): \", train_tgt.shape)\n",
        "print(\"Train Labels Shape(Target): \", train_labels.shape)\n",
        "print(\"Train Dev Data Shape(Source): \", train_dev_src.shape)\n",
        "print(\"Train Dev Data Shape(Target): \", train_dev_tgt.shape)\n",
        "print(\"Train Dev Labels Shape(Target): \", train_dev_labels.shape)\n",
        "print(\"Test Val Data Shape(Source): \", test_val_src.shape)\n",
        "print(\"Test Val Data Shape(Target): \", test_val_tgt.shape)\n",
        "print(\"Test Labels Shape(Target): \", test_val_labels.shape)"
      ],
      "metadata": {
        "id": "rO5HD33XOGxf",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e0e36c6d-8020-4fa2-c049-0cf5a28ba556"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "*** Preprocessed Data ***\n",
            "Train Data Shape(Source):  (154836, 60)\n",
            "Train Data Shape(Target):  (154836, 60)\n",
            "Train Labels Shape(Target):  (154836, 60)\n",
            "Train Dev Data Shape(Source):  (3649, 60)\n",
            "Train Dev Data Shape(Target):  (3649, 60)\n",
            "Train Dev Labels Shape(Target):  (3649, 60)\n",
            "Test Val Data Shape(Source):  (1194, 60)\n",
            "Test Val Data Shape(Target):  (1194, 60)\n",
            "Test Labels Shape(Target):  (1194, 60)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Below we prepare the data in the form ready to feed to the Model."
      ],
      "metadata": {
        "id": "xtsMxbAYzePz"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "train_data_prep = tf.data.Dataset.from_tensor_slices((train_src, train_tgt, train_labels))\n",
        "train_dev_data_prep = tf.data.Dataset.from_tensor_slices((train_dev_src, train_dev_tgt, train_dev_labels))\n",
        "test_val_data_prep = tf.data.Dataset.from_tensor_slices((test_val_src, test_val_tgt, test_val_labels))"
      ],
      "metadata": {
        "id": "_OkR5tbtyBQj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Below, we can see how our data looks"
      ],
      "metadata": {
        "id": "jSGBKDhc2sc-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for encoder_inputs, decoder_inputs, labels in train_data_prep.take(1):\n",
        "    print(f'encoder_inputs.shape: {encoder_inputs.shape}')\n",
        "    print(f'decoder_inputs.shape: {decoder_inputs.shape}')\n",
        "    print(f\"labels.shape: {labels.shape}\")"
      ],
      "metadata": {
        "id": "Ao1TX6GHzWn7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "06478bad-5198-4dd2-9318-25d6200bcd03"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "encoder_inputs.shape: (60,)\n",
            "decoder_inputs.shape: (60,)\n",
            "labels.shape: (60,)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Model"
      ],
      "metadata": {
        "id": "zjXk6pt-28kL"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We will use the Transformer Architechture with the following hyperparameters."
      ],
      "metadata": {
        "id": "a9UFHuFO3dHe"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Transformer parameters\n",
        "d_model = 256 # 512 in the original paper\n",
        "d_k = 32 # 64 in the original paper\n",
        "d_v = 32 # 64 in the original paper\n",
        "n_heads = 4 # 8 in the original paper\n",
        "n_encoder_layers = 2 # 6 in the original paper\n",
        "n_decoder_layers = 2 # 6 in the original paper\n",
        "\n",
        "max_token_length = MAX_TOKENS # 512 in the original paper"
      ],
      "metadata": {
        "id": "v9ULcpgbJ9WP"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class SingleHeadAttention(Layer):\n",
        "  def __init__(self, input_shape=(3, -1, d_model), dropout=.0, masked=None):\n",
        "    super(SingleHeadAttention, self).__init__()\n",
        "    self.q = Dense(d_k, input_shape=(-1, d_model), kernel_initializer='glorot_uniform', \n",
        "                   bias_initializer='glorot_uniform')\n",
        "    self.normalize_q = Lambda(lambda x: x / np.sqrt(d_k))\n",
        "    self.k = Dense(d_k, input_shape=(-1, d_model), kernel_initializer='glorot_uniform', \n",
        "                   bias_initializer='glorot_uniform')\n",
        "    self.v = Dense(d_v, input_shape=(-1, d_model), kernel_initializer='glorot_uniform', \n",
        "                   bias_initializer='glorot_uniform')\n",
        "    self.dropout = dropout\n",
        "    self.masked = masked\n",
        "  \n",
        "  # Inputs: [query, key, value]\n",
        "  def call(self, inputs, training=None):\n",
        "    assert len(inputs) == 3\n",
        "    # We use a lambda layer to divide vector q by sqrt(d_k) according to the equation\n",
        "    q = self.normalize_q(self.q(inputs[0]))\n",
        "    k = self.k(inputs[1])\n",
        "    # The dimensionality of q is (batch_size, query_length, d_k) and that of k is (batch_size, key_length, d_k)\n",
        "    # So we will do a matrix multication by batch after transposing last 2 dimensions of k\n",
        "    # tf.shape(attn_weights) = (batch_size, query_length, key_length)\n",
        "    attn_weights = tf.matmul(q, tf.transpose(k, perm=[0,2,1]))\n",
        "    if self.masked: # Prevent future attentions in decoding self-attention\n",
        "      # Create a matrix where the strict upper triangle (not including main diagonal) is filled with -inf and 0 elsewhere\n",
        "      length = tf.shape(attn_weights)[-1]\n",
        "      #attn_mask = np.triu(tf.fill((length, length), -np.inf), k=1) # We need to use tensorflow functions instead of numpy\n",
        "      attn_mask = tf.fill((length, length), -np.inf)\n",
        "      attn_mask = tf.linalg.band_part(attn_mask, 0, -1) # Get upper triangle\n",
        "      attn_mask = tf.linalg.set_diag(attn_mask, tf.zeros((length))) # Set diagonal to zeros to avoid operations with infinity\n",
        "      # This matrix is added to the attention weights so all future attention will have -inf logits (0 after softmax)\n",
        "      attn_weights += attn_mask\n",
        "    # Softmax along the last dimension\n",
        "    attn_weights = tf.nn.softmax(attn_weights, axis=-1)\n",
        "    if training: # Attention dropout included in the original paper. This is possibly to encourage multihead diversity.\n",
        "      attn_weights = tf.nn.dropout(attn_weights, rate=self.dropout)\n",
        "    v = self.v(inputs[2])\n",
        "    return tf.matmul(attn_weights, v)\n",
        "\n",
        "class MultiHeadAttention_my(Layer):\n",
        "  def __init__(self, dropout=.0, masked=None):\n",
        "    super(MultiHeadAttention_my, self).__init__()\n",
        "    self.attn_heads = list()\n",
        "    for i in range(n_heads): \n",
        "      self.attn_heads.append(SingleHeadAttention(dropout=dropout, masked=masked))\n",
        "    self.linear = Dense(d_model, input_shape=(-1, n_heads * d_v), kernel_initializer='glorot_uniform', \n",
        "                   bias_initializer='glorot_uniform')\n",
        "    \n",
        "  def call(self, x, training=None):\n",
        "    attentions = [self.attn_heads[i](x, training=training) for i in range(n_heads)]\n",
        "    concatenated_attentions = tf.concat(attentions, axis=-1)\n",
        "    return self.linear(concatenated_attentions)\n",
        "\n",
        "class TransformerEncoder(Layer):\n",
        "  def __init__(self, dropout=.1, attention_dropout=.0, **kwargs):\n",
        "    super(TransformerEncoder, self).__init__(**kwargs)\n",
        "    self.dropout_rate = dropout\n",
        "    self.attention_dropout_rate = attention_dropout\n",
        "  def build(self, input_shape):\n",
        "    self.multihead_attention = MultiHeadAttention_my(dropout=self.attention_dropout_rate)\n",
        "    self.dropout1 = tf.keras.layers.Dropout(self.dropout_rate)\n",
        "    self.layer_normalization1 = LayerNormalization(input_shape=input_shape, epsilon=1e-6)\n",
        "    \n",
        "    self.linear1 = Dense(input_shape[-1] * 4, input_shape=input_shape, activation='relu',\n",
        "                        kernel_initializer='glorot_uniform', bias_initializer='glorot_uniform')\n",
        "    self.linear2 = Dense(input_shape[-1], input_shape=self.linear1.compute_output_shape(input_shape),\n",
        "                        kernel_initializer='glorot_uniform', bias_initializer='glorot_uniform')\n",
        "    self.dropout2 = tf.keras.layers.Dropout(self.dropout_rate)\n",
        "    self.layer_normalization2 = LayerNormalization(input_shape=input_shape, epsilon=1e-6)\n",
        "    super(TransformerEncoder, self).build(input_shape)\n",
        "  def call(self, x, training=None):\n",
        "    sublayer1 = self.multihead_attention((x, x, x), training=training)\n",
        "    sublayer1 = self.dropout1(sublayer1, training=training)\n",
        "    layernorm1 = self.layer_normalization1(x + sublayer1)\n",
        "    \n",
        "    sublayer2 = self.linear2(self.linear1(layernorm1))\n",
        "    sublayer1 = self.dropout2(sublayer2, training=training)\n",
        "    layernorm2 = self.layer_normalization2(layernorm1 + sublayer2)\n",
        "    return layernorm2\n",
        "  def compute_output_shape(self, input_shape):\n",
        "    return input_shape\n",
        "\n",
        "class TransformerDecoder(Layer):\n",
        "  def __init__(self, dropout=.0, attention_dropout=.0, **kwargs):\n",
        "    super(TransformerDecoder, self).__init__(**kwargs)\n",
        "    self.dropout_rate = dropout\n",
        "    self.attention_dropout_rate = attention_dropout\n",
        "  def build(self, input_shape):\n",
        "    self.multihead_self_attention = MultiHeadAttention_my(dropout=self.attention_dropout_rate, masked=True)\n",
        "    self.dropout1 = tf.keras.layers.Dropout(self.dropout_rate)\n",
        "    self.layer_normalization1 = LayerNormalization(input_shape=input_shape, epsilon=1e-6)\n",
        "    \n",
        "    self.multihead_encoder_attention = MultiHeadAttention_my(dropout=self.attention_dropout_rate)\n",
        "    self.dropout2 = tf.keras.layers.Dropout(self.dropout_rate)\n",
        "    self.layer_normalization2 = LayerNormalization(input_shape=input_shape, epsilon=1e-6)\n",
        "    \n",
        "    self.linear1 = Dense(input_shape[-1] * 4, input_shape=input_shape, activation='relu',\n",
        "                        kernel_initializer='glorot_uniform', bias_initializer='glorot_uniform')\n",
        "    self.linear2 = Dense(input_shape[-1], input_shape=self.linear1.compute_output_shape(input_shape),\n",
        "                        kernel_initializer='glorot_uniform', bias_initializer='glorot_uniform')\n",
        "    self.dropout3 = tf.keras.layers.Dropout(self.dropout_rate)\n",
        "    self.layer_normalization3 = LayerNormalization(input_shape=input_shape, epsilon=1e-6)\n",
        "    super(TransformerDecoder, self).build(input_shape)\n",
        "  def call(self, x, hidden, training=None):\n",
        "    sublayer1 = self.multihead_self_attention((x, x, x))\n",
        "    sublayer1 = self.dropout1(sublayer1, training=training)\n",
        "    layernorm1 = self.layer_normalization1(x + sublayer1)\n",
        "    \n",
        "    sublayer2 = self.multihead_encoder_attention((x, hidden, hidden))\n",
        "    sublayer2 = self.dropout2(sublayer2, training=training)\n",
        "    layernorm2 = self.layer_normalization2(layernorm1 + sublayer2)\n",
        "    \n",
        "    sublayer3 = self.linear2(self.linear1(layernorm1))\n",
        "    sublayer3 = self.dropout3(sublayer3, training=training)\n",
        "    layernorm3 = self.layer_normalization2(layernorm2 + sublayer3)\n",
        "    return layernorm3\n",
        "  def compute_output_shape(self, input_shape):\n",
        "    return input_shape\n",
        "\n",
        "class SinusoidalPositionalEncoding(Layer): # This is a TensorFlow implementation of https://github.com/graykode/nlp-tutorial/blob/master/5-1.Transformer/Transformer_Torch.ipynb\n",
        "  def __init__(self):\n",
        "    super(SinusoidalPositionalEncoding, self).__init__()\n",
        "    self.sinusoidal_encoding = np.array([self.get_positional_angle(pos) for pos in range(max_token_length)], dtype=np.float32)\n",
        "    self.sinusoidal_encoding[:, 0::2] = np.sin(self.sinusoidal_encoding[:, 0::2])\n",
        "    self.sinusoidal_encoding[:, 1::2] = np.cos(self.sinusoidal_encoding[:, 1::2])\n",
        "    self.sinusoidal_encoding = tf.cast(self.sinusoidal_encoding, dtype=tf.float32) # Casting the array to Tensor for slicing\n",
        "  def call(self, x):\n",
        "    return x + self.sinusoidal_encoding[:tf.shape(x)[1]]\n",
        "  def compute_output_shape(self, input_shape):\n",
        "    return input_shape\n",
        "  def get_angle(self, pos, dim):\n",
        "    return pos / np.power(10000, 2 * (dim // 2) / d_model)\n",
        "  def get_positional_angle(self, pos):\n",
        "    return [self.get_angle(pos, dim) for dim in range(d_model)]\n",
        "\n",
        "class Transformer(Model):\n",
        "  def __init__(self, dropout=.1, attention_dropout=.0, **kwargs):\n",
        "    super(Transformer, self).__init__(**kwargs)\n",
        "    self.encoding_embedding = Embedding(VOCAB_SIZE, d_model)\n",
        "    self.decoding_embedding = Embedding(VOCAB_SIZE, d_model)\n",
        "    self.pos_encoding = SinusoidalPositionalEncoding()\n",
        "    self.encoder = [TransformerEncoder(dropout=dropout, attention_dropout=attention_dropout) for i in range(n_encoder_layers)]\n",
        "    self.decoder = [TransformerDecoder(dropout=dropout, attention_dropout=attention_dropout) for i in range(n_decoder_layers)]\n",
        "    self.decoder_final = Dense(VOCAB_SIZE, input_shape=(None, d_model))\n",
        "  def call(self, inputs, training=None): # Source_sentence and decoder_input\n",
        "    source_sentence, decoder_input = inputs\n",
        "    embedded_source = self.encoding_embedding(source_sentence)\n",
        "    encoder_output = self.pos_encoding(embedded_source)\n",
        "    for encoder_unit in self.encoder:\n",
        "      encoder_output = encoder_unit(encoder_output, training=training)\n",
        "    \n",
        "    embedded_target = self.decoding_embedding(decoder_input)\n",
        "    decoder_output = self.pos_encoding(embedded_target)\n",
        "    for decoder_unit in self.decoder:\n",
        "      decoder_output = decoder_unit(decoder_output, encoder_output, training=training)\n",
        "    if training:\n",
        "      decoder_output = self.decoder_final(decoder_output)\n",
        "      decoder_output = tf.nn.softmax(decoder_output, axis=-1)\n",
        "    else:\n",
        "      decoder_output = self.decoder_final(decoder_output[:, -1:, :])\n",
        "      decoder_output = tf.nn.softmax(decoder_output, axis=-1)\n",
        "    return decoder_output"
      ],
      "metadata": {
        "id": "KvsIOzPnzt4F"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "transformer = Transformer(dropout=0.3) # Instantiating a new transformer model\n",
        "train = [tf.cast(train_src, dtype=tf.float32), tf.cast(train_tgt, dtype=tf.float32)] # Cast the tuples to tensors\n",
        "validation = [tf.cast(train_dev_src, dtype=tf.float32), tf.cast(train_dev_tgt, dtype=tf.float32)]"
      ],
      "metadata": {
        "id": "eRsGRYTR6XQW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "transformer.build([train_src.shape, train_tgt.shape])\n",
        "transformer.summary()"
      ],
      "metadata": {
        "id": "IHXUpv-0eh-b",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e6c9d44a-5fb3-4a35-d0f1-5c7bd84b61ae"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"transformer\"\n",
            "_________________________________________________________________\n",
            " Layer (type)                Output Shape              Param #   \n",
            "=================================================================\n",
            " embedding (Embedding)       multiple                  8192000   \n",
            "                                                                 \n",
            " embedding_1 (Embedding)     multiple                  8192000   \n",
            "                                                                 \n",
            " sinusoidal_positional_encod  multiple                 0         \n",
            " ing (SinusoidalPositionalEn                                     \n",
            " coding)                                                         \n",
            "                                                                 \n",
            " transformer_encoder (Transf  multiple                 658304    \n",
            " ormerEncoder)                                                   \n",
            "                                                                 \n",
            " transformer_encoder_1 (Tran  multiple                 658304    \n",
            " sformerEncoder)                                                 \n",
            "                                                                 \n",
            " transformer_decoder (Transf  multiple                 790016    \n",
            " ormerDecoder)                                                   \n",
            "                                                                 \n",
            " transformer_decoder_1 (Tran  multiple                 790016    \n",
            " sformerDecoder)                                                 \n",
            "                                                                 \n",
            " dense (Dense)               multiple                  8224000   \n",
            "                                                                 \n",
            "=================================================================\n",
            "Total params: 27,504,640\n",
            "Trainable params: 27,504,640\n",
            "Non-trainable params: 0\n",
            "_________________________________________________________________\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Training"
      ],
      "metadata": {
        "id": "0JkChx5Ndp1s"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "transformer.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])"
      ],
      "metadata": {
        "id": "InXkWf-LawJi"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "EPOCHS = 10\n",
        "checkpoint_filepath = '/Models/checkpoint'\n",
        "model_checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(\n",
        "    filepath=checkpoint_filepath,\n",
        "    save_weights_only=True,\n",
        "    monitor='val_acc',\n",
        "    mode='max',\n",
        "    save_best_only=True)"
      ],
      "metadata": {
        "id": "LMttctIwaNxG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "with tf.device('/device:GPU:0'):\n",
        "    history = transformer.fit(train, tf.cast(train_labels, dtype=tf.float32), batch_size=256, epochs=EPOCHS, callbacks=[model_checkpoint_callback])"
      ],
      "metadata": {
        "id": "bzdsT8kmBEMo",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7f5669fd-73b4-437e-f6ad-b3bcec356fc8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/10\n",
            "605/605 [==============================] - ETA: 0s - loss: 1.8724 - accuracy: 0.7966WARNING:tensorflow:Can save best model only with val_acc available, skipping.\n",
            "605/605 [==============================] - 368s 583ms/step - loss: 1.8724 - accuracy: 0.7966\n",
            "Epoch 2/10\n",
            "605/605 [==============================] - ETA: 0s - loss: 1.5746 - accuracy: 0.8042WARNING:tensorflow:Can save best model only with val_acc available, skipping.\n",
            "605/605 [==============================] - 370s 611ms/step - loss: 1.5746 - accuracy: 0.8042\n",
            "Epoch 3/10\n",
            "605/605 [==============================] - ETA: 0s - loss: 1.4610 - accuracy: 0.8108WARNING:tensorflow:Can save best model only with val_acc available, skipping.\n",
            "605/605 [==============================] - 370s 611ms/step - loss: 1.4610 - accuracy: 0.8108\n",
            "Epoch 4/10\n",
            "605/605 [==============================] - ETA: 0s - loss: 1.3757 - accuracy: 0.8151WARNING:tensorflow:Can save best model only with val_acc available, skipping.\n",
            "605/605 [==============================] - 370s 611ms/step - loss: 1.3757 - accuracy: 0.8151\n",
            "Epoch 5/10\n",
            "605/605 [==============================] - ETA: 0s - loss: 1.2967 - accuracy: 0.8191WARNING:tensorflow:Can save best model only with val_acc available, skipping.\n",
            "605/605 [==============================] - 370s 612ms/step - loss: 1.2967 - accuracy: 0.8191\n",
            "Epoch 6/10\n",
            "605/605 [==============================] - ETA: 0s - loss: 1.2259 - accuracy: 0.8226WARNING:tensorflow:Can save best model only with val_acc available, skipping.\n",
            "605/605 [==============================] - 370s 612ms/step - loss: 1.2259 - accuracy: 0.8226\n",
            "Epoch 7/10\n",
            "605/605 [==============================] - ETA: 0s - loss: 1.1670 - accuracy: 0.8258WARNING:tensorflow:Can save best model only with val_acc available, skipping.\n",
            "605/605 [==============================] - 370s 612ms/step - loss: 1.1670 - accuracy: 0.8258\n",
            "Epoch 8/10\n",
            "605/605 [==============================] - ETA: 0s - loss: 1.1166 - accuracy: 0.8288WARNING:tensorflow:Can save best model only with val_acc available, skipping.\n",
            "605/605 [==============================] - 372s 614ms/step - loss: 1.1166 - accuracy: 0.8288\n",
            "Epoch 9/10\n",
            "605/605 [==============================] - ETA: 0s - loss: 1.0731 - accuracy: 0.8314WARNING:tensorflow:Can save best model only with val_acc available, skipping.\n",
            "605/605 [==============================] - 371s 614ms/step - loss: 1.0731 - accuracy: 0.8314\n",
            "Epoch 10/10\n",
            "605/605 [==============================] - ETA: 0s - loss: 1.0333 - accuracy: 0.8340WARNING:tensorflow:Can save best model only with val_acc available, skipping.\n",
            "605/605 [==============================] - 371s 614ms/step - loss: 1.0333 - accuracy: 0.8340\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "transformer.save_weights(\"model_weights\")"
      ],
      "metadata": {
        "id": "IRD-txOmMk0h"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "plt.plot(history.history['accuracy'])\n",
        "plt.title('model accuracy')\n",
        "plt.ylabel('accuracy')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['train'], loc='upper left')\n",
        "plt.figure(figsize=(10,10))\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "JbSCoj0cft6O",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 312
        },
        "outputId": "5cc13756-8911-432d-cdeb-599283821dde"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEWCAYAAABxMXBSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXhU5dnH8e+PEEhAdlCBEAgIsskiYatYtS5Fxa1aBQWlotS6VH1tX7FVS237ttq6tm6giCKIK0qtG6i4VLaw76ssCVsAwx4gyf3+MYd2iEEmmMnJcn+uKxczZ5t7BphfnvOc8zwyM5xzzrlYVQm7AOecc+WLB4dzzrli8eBwzjlXLB4czjnnisWDwznnXLF4cDjnnCsWDw7nvoOk0ZL+GOO2aySdE++anAubB4dzzrli8eBwrhKQVDXsGlzF4cHhyr3gFNGvJc2XtEfS85JOkPS+pF2SJkuqF7X9xZIWScqRNEVSu6h1XSXNDvZ7FUgq9Fr9JM0N9v1KUqcYa7xQ0hxJOyWtlzS80Po+wfFygvWDg+XJkh6WtFbSDklfBsvOlJRZxOdwTvB4uKQ3JL0saScwWFIPSVOD19go6R+SqkXt30HSJEnbJW2W9BtJJ0raK6lB1HanSsqWlBjLe3cVjweHqyguB84F2gAXAe8DvwEaEfl3/ksASW2AV4A7gnXvAf+UVC34En0bGAPUB14Pjkuwb1dgFPBzoAHwLDBRUvUY6tsDXAvUBS4EfiHp0uC4zYN6/x7U1AWYG+z3N6Ab8IOgpv8FCmL8TC4B3ghecyyQD9wJNAR6A2cDNwc11AImAx8ATYCTgI/NbBMwBbgy6riDgPFmdjDGOlwF48HhKoq/m9lmM8sCvgCmm9kcM8sFJgBdg+2uAv5lZpOCL76/AclEvph7AYnAY2Z20MzeAGZGvcZQ4Fkzm25m+Wb2IrA/2O87mdkUM1tgZgVmNp9IeJ0RrL4amGxmrwSvu83M5kqqAlwP3G5mWcFrfmVm+2P8TKaa2dvBa+4zs1lmNs3M8sxsDZHgO1RDP2CTmT1sZrlmtsvMpgfrXgQGAkhKAAYQCVdXSXlwuIpic9TjfUU8Py543ARYe2iFmRUA64GmwbosO3zkz7VRj5sDdwWnenIk5QDNgv2+k6Sekj4NTvHsAG4i8ps/wTFWFbFbQyKnyopaF4v1hWpoI+ldSZuC01f/F0MNAO8A7SWlEWnV7TCzGcdYk6sAPDhcZbOBSAAAIElEvjSzgI1A02DZIalRj9cDfzKzulE/NczslRhedxwwEWhmZnWAZ4BDr7MeaFXEPluB3COs2wPUiHofCUROc0UrPPT108BSoLWZ1SZyKi+6hpZFFR602l4j0uoYhLc2Kj0PDlfZvAZcKOnsoHP3LiKnm74CpgJ5wC8lJUr6CdAjat+RwE1B60GSagad3rVieN1awHYzy5XUg8jpqUPGAudIulJSVUkNJHUJWkOjgEckNZGUIKl30KeyHEgKXj8RuBc4Wl9LLWAnsFtSW+AXUeveBRpLukNSdUm1JPWMWv8SMBi4GA+OSs+Dw1UqZraMyG/OfyfyG/1FwEVmdsDMDgA/IfIFuZ1If8hbUftmADcC/wC+AVYG28biZuABSbuA+4kE2KHjrgMuIBJi24l0jHcOVv8KWECkr2U78CBQxcx2BMd8jkhraQ9w2FVWRfgVkcDaRSQEX42qYReR01AXAZuAFcBZUev/TaRTfraZRZ++c5WQfCIn51wsJH0CjDOz58KuxYXLg8M5d1SSugOTiPTR7Aq7HhcuP1XlnPtOkl4kco/HHR4aDrzF4Zxzrpi8xeGcc65YKsXAZw0bNrQWLVqEXYZzzpUrs2bN2mpmhe8PqhzB0aJFCzIyMsIuwznnyhVJRV56HddTVZL6SlomaaWkYUWsTw2GYZijyMimFwTLewQjkM6VNE/SZVH7rJG0IFjnaeCcc6Usbi2OYAiEJ4ncVJQJzJQ00cwWR212L/CamT0tqT2RkUpbAAuBdDPLk9QYmCfpn2aWF+x3lpltjVftzjnnjiyeLY4ewEozWx3ckTueyDDP0QyoHTyuQ2QcIcxsb1RIJPHtMXecc86FJJ59HE05fHTOTKBnoW2GAx9Jug2oCfxnvuZgnJxRRAakGxQVJBbsY0SGuB5R1ItLGkpkGGxSU1O/tf7gwYNkZmaSm5tb/HdWjiQlJZGSkkJios+545wrGWF3jg8ARpvZw5J6A2MkdQzmD5gOdFBkdrYXJb0fjNLZx8yyJB0PTJK01Mw+L3zgIFBGAKSnp3+rxZKZmUmtWrVo0aIFhw+GWnGYGdu2bSMzM5O0tLSwy3HOVRDxPFWVRWS46kNSgmXRhhAM9mZmU4mclmoYvYGZLQF2Ax2D51nBn1uITNATPXppzHJzc2nQoEGFDQ0ASTRo0KDCt6qcc6UrnsExE2gtKS2YkrM/kfkIoq0jMn0lQcsiCcgO9qkaLG8OtAXWBMNY1wqW1wTOI9KRfkwqcmgcUhneo3OudMUtOII+iVuBD4ElRK6eWiTpAUkXB5vdBdwoaR6RqTQHB7Ov9SFyJdVcIq2Km4OrqE4Avgy2n0FkCtAP4vUenHOuvJqfmcPv/7mIg/mxTlEfu7j2cZjZe0QusY1edn/U48XAaUXsN4YiJosxs9X8d56Cci0nJ4dx48Zx8803F2u/Cy64gHHjxlG3bt04VeacK6/27M9j4rwNjJu+jgVZO0hOTODyU1Po2LROib5O2J3jlVZOTg5PPfXUt4IjLy+PqlWP/Nfy3nvvHXGdc65yWrJxJ2Onr+XtORvYvT+Pk0+oxQOXdODSrk2pnVTyV1R6cIRk2LBhrFq1ii5dupCYmEhSUhL16tVj6dKlLF++nEsvvZT169eTm5vL7bffztChQ4H/Dp+ye/duzj//fPr06cNXX31F06ZNeeedd0hOTg75nTnnSkPuwXzenb+RcdPXMntdDtWqVqHfKY25plcqp6bWi2v/pgcH8Pt/LmLxhp0lesz2TWrzu4s6HHH9X/7yFxYuXMjcuXOZMmUKF154IQsXLvzPZbOjRo2ifv367Nu3j+7du3P55ZfToEGDw46xYsUKXnnlFUaOHMmVV17Jm2++ycCBA0v0fTjnypaVW3Yzbvo63pydyY59B2nZsCb3XtiOy09NoV7NaqVSgwdHGdGjR4/D7rV44oknmDBhAgDr169nxYoV3wqOtLQ0unTpAkC3bt1Ys2ZNqdXrnCs9+/Py+XDRZsZOW8v0r7eTmCB+3OFErunZnF4t65f61ZMeHPCdLYPSUrNmzf88njJlCpMnT2bq1KnUqFGDM888s8h7MapXr/6fxwkJCezbt69UanXOlY612/YwbsY6Xs/IZPueAzSrn8zdfdvy0/QUGh5X/egHiBMPjpDUqlWLXbuKnoVzx44d1KtXjxo1arB06VKmTZtWytU558JyML+Aj5dsZuz0dXyxYisJVcQ57Y7n6p7NOf2khlSpEv69WR4cIWnQoAGnnXYaHTt2JDk5mRNOOOE/6/r27cszzzxDu3btOPnkk+nVq1eIlTrnSkNWzj7Gz1jH+Jnryd61n8Z1krjznDZc1b0ZJ9ZJCru8w1SKOcfT09Ot8EROS5YsoV27diFVVLoq03t1rjzJLzCmLNvC2OnrmLJsCwac2aYR1/RszpknN6JqQrize0uaZWbphZd7i8M550rZ5p25vDpzPeNnrGPDjlwa1arOzWeeRP8ezUipVyPs8o7Kg8M550pBQYHx5cqtjJ2+lslLtpBfYPQ5qSH39WvPOe1PIDHk1kVxVOrgMLMKPwhgZTgV6VxZtnX3fl7PyOSVGetYt30v9WokckOfNAb0SKVFw5pHP0AZVGmDIykpiW3btlXoodUPzceRlFS2Otacq+jMjGmrtzNuxjo+WLiRg/lGj7T63HVeG/p2PJHqVRPCLvF7qbTBkZKSQmZmJtnZ2WGXEleHZgB0zsXfztyDvDkrk5enrWVV9h5qJ1VlYK/mXN0jldYn1Aq7vBJTaYMjMTHRZ8VzzpWIpZt28tLUtbw9J4u9B/Lp3Kwuf72iE/06NSG5WvluXRSl0gaHc859HwfyCvhw0SbGTF3LjDXbqV61Chd1bsK1vZvTKaViT3vgweGcc8WwaUcu42as45UZ68jetZ9m9ZO55/y2XJnerNQGGQybB4dzzh3Foc7uMdPW8OGizRSYcUabRlzbuzlntDmehDIwDEhpimtwSOoLPA4kAM+Z2V8KrU8FXgTqBtsMM7P3JPUARhzaDBhuZhNiOaZzzpWU3fvzmDA7kzHT1rJ8827qJCcypE8a1/RMpXmD8nkpbUmIW3BISgCeBM4FMoGZkiYG08Ueci+RucifltSeyDSzLYCFQLqZ5UlqTGT+8X8CFsMxnXPue1mxeRdjpq3lrdlZ7N6fxylN6/DQFZ24uHMTkhIrXmd3ccWzxdEDWBnME46k8cAlQPSXvAG1g8d1gA0AZrY3apukYLtYj+mcc8WWl1/ApMWbeWnqWqau3ka1hCr069SYQb2b06VZ3Qp7v9exiGdwNAXWRz3PBHoW2mY48JGk24CawDmHVkjqCYwCmgODgtZHLMd0zrmYbdmVy/gZ6xk3fR2bdubStG4y/9v3ZK5Kb0aDEOe8KMvC7hwfAIw2s4cl9QbGSOpoZgVmNh3oIKkd8KKk94tzYElDgaEAqampJV64c678MjMy1n7DS1PX8v6CjeQVGKe3bsgfLu3Ij9pWvs7u4opncGQBzaKepwTLog0B+gKY2VRJSUBDYMuhDcxsiaTdQMcYj3lovxEEHezp6ek+YJNzjj3783hn7gZemrqGpZt2USupKtf2bsHAXqm0bHRc2OWVG/EMjplAa0lpRL7c+wNXF9pmHXA2MDpoWSQB2cE+64PTU82BtsAaICeGYzrn3GFWZ+9mzLS1vJGRya79ebRrXJs//+QULunShBrVwj7xUv7E7RMLvvRvBT4kcunsKDNbJOkBIMPMJgJ3ASMl3UmkA3ywmZmkPsAwSQeBAuBmM9sKUNQx4/UenHPlV15+AZ8s3cKYaWv5YsVWEhPE+R0bc23v5nRrXs87u7+HSjsDoHOuYtq2ez/jZ0Y6u7Ny9tG4ThJX90ilf49UGtXyzu7i8BkAnXMV2pKNOxn5+Wrenb+RA/kFnHZSg8gkSe2OD30K1orGg8M5V26ZGdO/3s4zn61iyrJsalZLYECPZgzq3YKTjvfO7njx4HDOlTsFBcZHizfzzGermLs+hwY1q/Gr89owqFcL6tRIDLu8Cs+DwzlXbuzPy+ftOVk8+/lqVmfvIbV+Df5waUd+2i3FhwIpRR4czrkyb1fuQcZOX8eoL79my679dGhSm78P6Mr5HU/0/osQeHA458qsLTtzGfXvNYydtpZd+/M47aQGPHxlZ/qc1NAvpw2RB4dzrsz5euseRny+ijdnZZFXUMD5HRtz0xmtOCWlTtilOTw4nHNlyLz1OTzz2So+WLSJxIQq/DQ9hRtPb0mLhpV37ouyyIPDORcqM+PzFVt5Zsoqpq7eRu2kqtx8ZisG/yDNb9grozw4nHOhyMsv4F8LNvLsZ6tZvHEnJ9ZO4rcXtGNAz1SOq+5fTWWZ/+0450rVvgP5vD5rPSO/WM367fto1agmD13RiUu7NKVaVb9Cqjzw4HDOlYpv9hxgzLS1jP5qDdv3HODU1Lrcd2F7zml3AlV8/otyxYPDORdXWTn7eO6L1bw6cz17D+Tzo7bHc9MZrejewkeoLa88OJxzcbFs0y6e/WwVE+dtAODizk0YekZL2p5YO+TK3PflweGcKzFmxsw13/DMZ6v4ZOkWkhMTGNS7OTec3pKmdZPDLs+VEA8O59z3VlBgTF4SGXRw9roc6tesxp3ntOHa3s2pV7Na2OW5EubB4Zw7ZgUFxvsLN/H4x8tZvnk3KfWS+f3FHbgyvRnJ1XzQwYrKg8M5V2yRYc038djkFSzdtItWjWry2FVd6NepsQ86WAnENTgk9QUeJzI/+HNm9pdC61OBF4G6wTbDzOw9SecCfwGqAQeAX5vZJ8E+U4DGwL7gMOeZ2ZZ4vg/nXISZMWnxZh6dvIIlG3fSsmFNHu/fhX6dmpDgl9RWGnELDkkJwJPAuUAmMFPSRDNbHLXZvcBrZva0pPbAe0ALYCtwkZltkNQR+BBoGrXfNWbmk4g7V0rMjE+WbuHRyctZmLWT5g1q8MiVnbm4cxNvYVRC8Wxx9ABWmtlqAEnjgUuA6OAw4NC1eXWADQBmNidqm0VAsqTqZrY/jvU65woxM6Ysz+axScuZl7mDZvWT+esVnbisa1MPjEosnsHRFFgf9TwT6Flom+HAR5JuA2oC5xRxnMuB2YVC4wVJ+cCbwB/NzArvJGkoMBQgNTX1WN+Dc5WSmfHFiq08Onk5c9bl0LRuMg9efgo/OTWFRA+MSi/szvEBwGgze1hSb2CMpI5mVgAgqQPwIHBe1D7XmFmWpFpEgmMQ8FLhA5vZCGAEQHp6+reCxTn3bWbGV6u28eik5WSs/YYmdZL4v8tO4YpuKT6OlPuPeAZHFtAs6nlKsCzaEKAvgJlNlZQENAS2SEoBJgDXmtmqQzuYWVbw5y5J44icEvtWcDjnimfa6m08Mmk5M77ezom1k/jDpR25Mj2F6lX9slp3uHgGx0ygtaQ0IoHRH7i60DbrgLOB0ZLaAUlAtqS6wL+IXGX170MbS6oK1DWzrZISgX7A5Di+B+cqvBlfb+fRScuZunobx9eqzvCL2tO/RypJiR4YrmhxCw4zy5N0K5ErohKAUWa2SNIDQIaZTQTuAkZKupNIR/lgM7Ngv5OA+yXdHxzyPGAP8GEQGglEQmNkvN6DcxXZrLXbeXTSCr5cuZWGx1Xnvn7tuaanB4Y7OhXRr1zhpKenW0aGX73rHMCcdd/w6OQVfL48mwY1q3HTGa0Y2Ku53+ntvkXSLDNLL7w87M5x51wpmZ+Zw6OTlvPpsmzq1Uhk2PltubZ3c2pU868BVzz+L8a5Cm5h1g4em7ycyUu2UCc5kV//+GSu+0ELn57VHTP/l+NcBbVk404em7ycDxdtpnZSVe46tw2DT2tBraTEsEtz5ZwHh3MVzLJNu3j84+W8t2ATtapX5Y5zWvOz09Kok+yB4UqGB4dzFcTKLbt4bPIK/rVgIzWrVeWXPzqJIX1aUqeGB4YrWR4czpVzq7J388THK5g4bwPJiQncfGYrbujT0idQcnHjweFcObUhZx+PTFrOW7MzqV41gZ//sBVDf9iS+h4YLs48OJwrZ3bsO8jTU1bxwr+/xgyuPy2Nm85sRcPjqoddmqskPDicKyf25+UzZupa/vHpSnbsO8hlXZryP+e1IaVejbBLc5WMB4dzZVxBgTFx3gb+9tEyMr/Zx+mtGzLs/LZ0aFIn7NJcJeXB4VwZ9uWKrfz5/SUs2rCT9o1rM2bIKZzeulHYZblKzoPDuTJo8Yad/OWDpXy+PJumdZN57KouXNy5CVV8Xm9XBnhwOFeGZOXs4+GPljFhTha1kxK598J2DOzV3EesdWWKB4dzZcCOvQd5cspKRn+1BoChP2zJzWec5DfvuTLJg8O5EOUezOelqWv4xycr2bU/j590TeF/zmtD07rJYZfm3BF5cDgXgoIC4+25WTz80XKycvZxRptGDDu/Le0a1w67NOeOyoPDuVL2+fJs/vz+UpZs3EnHprV56IpOnHZSw7DLci5mVeJ5cEl9JS2TtFLSsCLWp0r6VNIcSfMlXRAsP1fSLEkLgj9/FLVPt2D5SklPSPLLTFy5sDBrBwOfm861o2awK/cgj/fvwsRb+nhouHInbi0OSQnAk8C5QCYwU9JEM1sctdm9wGtm9rSk9sB7QAtgK3CRmW2Q1JHIvOVNg32eBm4Epgfb9wXej9f7cO77Wr99Lw9/tIy3526gbo1E7uvXnoG9Uqle1a+UcuVTPE9V9QBWmtlqAEnjgUuA6OAw4NBJ3TrABgAzmxO1zSIgWVJ1oD5Q28ymBcd8CbgUDw5XBn2z5wBPfrqSl6auRYJfnNmKm85o5fNiuHIvnsHRFFgf9TwT6Flom+HAR5JuA2oC5xRxnMuB2Wa2X1LT4DjRx2xaxD5IGgoMBUhNTT2W+p07JrkH8xn91Rqe/HQlu/fnccWpkSulGtfxK6VcxRB25/gAYLSZPSypNzBGUkczKwCQ1AF4EDivuAc2sxHACID09HQrwZqdK1J+gfHW7EwembScjTtyOevkRtx9flvanuhXSrmKJZ7BkQU0i3qeEiyLNoRIHwVmNlVSEtAQ2CIpBZgAXGtmq6KOmXKUYzpXqsyMKcuzefD9pSzdtItOKXV45Mou9G7VIOzSnIuLeAbHTKC1pDQiX+79gasLbbMOOBsYLakdkARkS6oL/AsYZmb/PrSxmW2UtFNSLyKd49cCf4/je3DuOy3I3MGf31/CV6u2kVq/Bn8f0JULT2nsY0q5Ci1uwWFmeZJuJXJFVAIwyswWSXoAyDCzicBdwEhJdxLpKB9sZhbsdxJwv6T7g0OeZ2ZbgJuB0UAykU5x7xh3pS4rZx8Pvr+UifM2UL9mNX53UXuu6dmcalXjeoW7c2WCzI5++l/SW8DzwPuH+h/Kk/T0dMvIyAi7DFcBHMwvYNSXX/PY5BUYxpA+afz8jFbUTvIrpVzFI2mWmaUXXh5ri+Mp4GfAE5JeB14ws2UlWaBzZd2stdv57YSFLN20i3PaHc/wizv47HuuUoopOMxsMjBZUh0iV0JNlrQeGAm8bGYH41ijc6HK2XuABz9Yyisz1tO4ThLPDurGee1PwActcJVVzH0ckhoAA4FBwBxgLNAHuA44Mx7FORcmM2PCnCz+9K8l5Ow7yI2np3HHOW2oWT3sq9idC1dM/wMkTQBOBsYQGQpkY7DqVUneeeAqnJVbdnPv2wuYtno7XVPrMubSU2jfxO/HcA5ib3E8YWafFrWiqI4T58qr3IP5PPnpSp75bBXJiQn86bKODOie6pfXOhcl1uBoL2mOmeUASKoHDDCzp+JXmnOl67Pl2dz39kLWbd/LZV2b8psL2tGoVvWwy3KuzIk1OG40sycPPTGzbyTdSORqK+fKtS07c3ng3cW8O38jLRvWZNwNPfmBD3Xu3BHFGhwJkmTBTR/BkOnV4leWc/GXX2C8PG0tf/twGfvzC/ifc9vw8zNa+nDnzh1FrMHxAZGO8GeD5z8PljlXLi3I3MFvJixgQdYOTm/dkD9c0pEWDWuGXZZz5UKswXE3kbD4RfB8EvBcXCpyLo525R7k4Y+W89LUNdSvWZ0nBnTlok6N/Z4M54oh1hsAC4jMvPd0fMtxLj7MjPcWbOL3/1xE9u79DOrVnLvOO9knVXLuGMR6H0dr4M9AeyIj2AJgZi3jVJdzJWbdtr3c985CPlueTYcmtRl5bTqdm9UNuyznyq1YT1W9APwOeBQ4i8i4VT4MqCvTDuQVMPKL1Tzx8QoSE6pwf7/2XNu7OVUT/J+uc99HrMGRbGYfB1dWrQWGS5oF3H+0HZ0Lw7TV27j37YWs3LKbC045kfv7deDEOklH39E5d1SxBsd+SVWAFcFcGVnAcfEry7ljs233fv7vvaW8OTuTlHrJvDC4O2e1PT7sspyrUGINjtuBGsAvgT8QOV11XbyKcq64CgqM1zLW8+f3l7L3QB63nNWKW89qTXI1vyfDuZJ21OAIbva7ysx+Bewm0r/hXJmxbNMufjthARlrv6FHi/r86bKOtD6hVthlOVdhHTU4zCxfUp/SKMa54th7II/HP17B8198Ta2kqvz1ik5c0S3F78lwLs5iPVU1R9JE4HVgz6GFZvbWd+0kqS/wOJE5x58zs78UWp8KvAjUDbYZZmbvBXN/vAF0B0ab2a1R+0wBGgP7gkWH5iJ3lcjHSzZz/zuLyMrZx1XpzRh2flvq1fRRcJwrDbEGRxKwDfhR1DIDjhgcwSmuJ4FzgUxgpqSJZrY4arN7gdfM7GlJ7YH3gBZALnAf0DH4KewaM/N5QCqhzTtzuf+dhXy4aDNtTjiO12/qTfcW9cMuy7lKJdY7x4+lX6MHsNLMVgNIGg9cAkQHhwGHZsepA2wIXm8P8KWkk47hdV0FZGa8NTuL3/9zEQfyC7i7b1uG9EmjWlW/J8O50hbrneMvEPmSP4yZXf8duzUF1kc9zwR6FtpmOPCRpNuAmsA5sdQDvCApH3gT+OOhUXsL1TwUGAqQmpoa42FdWbR5Zy73vLWAT5ZuoXuLejx0RWfSfEBC50IT66mqd6MeJwGXEbQOvqcBRPowHpbUGxgjqWMwNtaRXGNmWZJqEQmOQcBLhTcysxHACID09PRvBYsr+8yMN2dn8UDQyri/X3sG/6CFz8bnXMhiPVX1ZvRzSa8AXx5ltyygWdTzlGBZtCFA3+A1pkpKAhoCR+zsNrOs4M9dksYROSX2reBw5dumHbn8ZoK3Mpwri2JtcRTWGjja7bgzgdaS0ogERn/g6kLbrAPOBkZLakekNZN9pANKqgrUNbOtkhKBfsDkY3sLriw61Mr4/T8XcdBbGc6VSbH2cezi8D6OTUTm6DgiM8sLhif5kMiltqPMbJGkB4AMM5sI3AWMlHRncPzBUbMMriHScV5N0qXAecBa4MMgNBKIhMbIWN+sK9uiWxk9WtTnoSs6+eRKzpVBKqJfucJJT0+3jAy/eresMjPemJXJA+8u5mBwxdR1vb2V4VzYJM0ys/TCy2NtcVwGfGJmO4LndYEzzeztki3TVTabduRyz1vz+XRZtrcynCsnYu3j+J2ZTTj0xMxyJP0O8OBwx6RwK+N3F7X3VoZz5USswVHUXVbH2rHuKrnDWhlp9Xnocm9lOFeexPrlnyHpESJDiADcAsyKT0muojIzXp+VyR/eXUxevjH8ovZc660M58qdWIPjNiJjR71K5OqnSUTCw7mYFG5l/PWKTjRv4K0M58qjWG8A3AMMi3MtrgLyVoZzFU+sV1VNAn5qZjnB83rAeDP7cTyLc+Xbxh37uOetBUzxVoZzFUqsp6oaHgoNADP7RpJP5OyK5K0M5yq2WIOjQFKqma0DkNSCIkbLdW7jjn0Me3MBny3Ppmda5L4Mb2U4V7HEGhy/JTI/xmeAgNMJhix3DjJOvEQAABMYSURBVIJWRkbQyigwfn9xBwb1au6tDOcqoFg7xz+QlE4kLOYQufFv33fv5SqLDTmRvgxvZThXOcTaOX4DcDuRodHnAr2AqRw+layrZMyM1zLW88d3l5BXYDxwSQcG9vRWhnMVXaynqm4HugPTzOwsSW2B/4tfWa6sK9zK+OsVnUltUCPsspxzpSDW4Mg1s1xJSKpuZkslnRzXylyZ5K0M51yswZEZjIj7NjBJ0jdE5sZwlciGnH0Me2sBny/PplfL+jx0ubcynKuMYu0cvyx4OFzSp0Ad4IO4VeXKnHnrcxj0/HRvZTjnij/CrZl9Fo9CXNm1aMMOrh01gzo1Enl5SE+/Ysq5Sq6o4dJLjKS+kpZJWinpW2NdSUqV9KmkOZLmS7ogWN4gWL5b0j8K7dNN0oLgmE9I8l9742j55l0Men4GNaolMO6GXh4azrn4BYekBCLDsJ8PtAcGSGpfaLN7gdfMrCvQH3gqWJ5LZDTeXxVx6KeBG4HWwU/fkq/eAazO3s01z00noYoYd2MvmtX3/gznXHxbHD2AlWa22swOAOOBSwptY0Dt4HEdYANERuM1sy+JBMh/SGoM1DazaRaZLP0l4NI4vodKa/32vVzz3HQKCoxxN/QkzSdacs4F4hkcTYH1Uc8zg2XRhgMDJWUC7xGZ9+Nox8w8yjEBkDRUUoakjOzs7OLUXeltyNnHgJHT2HsgnzFDetL6hFphl+ScK0Pi2scRgwHAaDNLAS4AxkgqkZrMbISZpZtZeqNGjUrikJXClp25XD1yGjv2HmTMkB60b1L76Ds55yqVeAZHFtAs6nlKsCzaEOA1ADObCiQBDY9yzJSjHNMdo62793P1c9PZsms/o6/vQaeUumGX5Jwrg+IZHDOB1pLSJFUj0vk9sdA264CzASS1IxIcRzyvZGYbgZ2SegVXU10LvBOP4iubnL0HGPjcdDK/2cuowd3p1rxe2CU558qoYt/HESszy5N0K/AhkACMMrNFkh4AMsxsInAXMFLSnUQ6ygcHnd5IWkOk47yapEuB88xsMXAzMBpIBt4Pftz3sDP3IIOen8HqrXt4/rp0erVsEHZJzrkyTMH3dIWWnp5uGRkZYZdRJu3en8e1z09nQdYOnhnYjbPbnRB2Sc65MkLSLDNLL7w8bi0OV/btO5DPkNEzmZe5gyev7uqh4ZyLSdhXVbmQ5B7MZ+iYDGas2c4jV3amb8fGYZfknCsnPDgqoQN5Bdw8djZfrNjKQ5d34pIuRd4K45xzRfLgqGTy8gv45Stz+GTpFv50WUd+mt7s6Ds551wUD45KJL/A+J/X5vHBok3c36891/RsHnZJzrlyyIOjkigoMO5+cz4T523g7r5tub5PWtglOefKKQ+OSsDMuO+dhbwxK5M7zmnNL85sFXZJzrlyzIOjgjMzHnh3MWOnr+OmM1px+9mtwy7JOVfOeXBUYGbGQx8u44V/r+Fnp7Xg7r4n4/NeOee+Lw+OCuyJj1fy9JRVXNMzlfv7tffQcM6VCA+OCurpKat4dPJyruiWwh8u6eih4ZwrMR4cFdCoL7/mwQ+WclHnJjx4eSeqVPHQcM6VHA+OCmbs9LU88O5i+nY4kUeu7EyCh4ZzroR5cFQgb8zK5LcTFvKjtsfzxICuJCb4X69zruT5N0sFMXHeBv73jXmc3rohT11zKtWq+l+tcy4+/NulAvhg4SbufHUu6S3qM2JQOkmJCWGX5JyrwDw4yrlPl27htldm0zmlDqMGdye5moeGcy6+4hockvpKWiZppaRhRaxPlfSppDmS5ku6IGrdPcF+yyT9OGr5GkkLJM2VVKmn9ftyxVZ+/vIs2p5Ym9HX9+C46j4vl3Mu/uL2TSMpAXgSOBfIBGZKmhjMG37IvcBrZva0pPbAe0CL4HF/oAPQBJgsqY2Z5Qf7nWVmW+NVe3kwffU2bnhpJi0b1uSl63tQOykx7JKcc5VEPFscPYCVZrbazA4A44FLCm1jQO3gcR1gQ/D4EmC8me03s6+BlcHxHDBr7TdcP3omKfVq8PINPalXs1rYJTnnKpF4BkdTYH3U88xgWbThwEBJmURaG7fFsK8BH0maJWloSRdd1i3I3MHgF2bQqFZ1xt3Qk4bHVQ+7JOdcJRN25/gAYLSZpQAXAGMkHa2mPmZ2KnA+cIukHxa1kaShkjIkZWRnZ5ds1SFZsnEng0ZNp05yIuNu7MXxtZPCLsk5VwnFMziygOh5SVOCZdGGAK8BmNlUIAlo+F37mtmhP7cAEzjCKSwzG2Fm6WaW3qhRo+/9ZsK2cssuBj43neTEBF65sRdN6iaHXZJzrpKKZ3DMBFpLSpNUjUhn98RC26wDzgaQ1I5IcGQH2/WXVF1SGtAamCGppqRawfY1gfOAhXF8D2XCmq17uHrkdKpUEWNv6Emz+jXCLsk5V4nF7aoqM8uTdCvwIZAAjDKzRZIeADLMbCJwFzBS0p1E+i4Gm5kBiyS9BiwG8oBbzCxf0gnAhGCk16rAODP7IF7voSzI3rWfa56bTl6BMX5oL1o2Oi7skpxzlZwi39MVW3p6umVklL9bPnIP5nP1yGks3riT13/+A05JqRN2Sc65SkTSLDNLL7zc7xgro8yMu9+cz+x1OTwz8FQPDedcmRH2VVXuCP7+yUrembuBX//4ZPp2bBx2Oc459x8eHGXQu/M38Mik5fzk1KbcfGarsMtxzrnDeHCUMfPW53DXa/NIb16PP//kFJ/y1TlX5nhwlCEbd+zjxpcyaFSrOs8O6kb1qj7SrXOu7PHgKCP27M9jyOgM9h7IZ9Tg7jTwoUScc2WUB0cZUFBg3PnqXJZu2sk/ru5KmxNqhV2Sc84dkQdHGfDQh8v4aPFm7uvXnjNPPj7scpxz7jt5cITs9Yz1PPPZKq7pmcrgH7QIuxznnDsqD44Qzfh6O7+ZsIDTTmrA8Is7+BVUzrlywYMjJOu27eXnYzJoVr8GT13djcQE/6twzpUP/m0Vgp25B7n+xZkYMOq67tSp4dO+OufKDw+OUpaXX8AtY2ezZusenr6mGy0a1gy7JOecKxYf5LCU/eHdxXyxYisPXn4KvVs1CLsc55wrNm9xlKIxU9fw4tS13Hh6Gld1Tw27HOecOyYeHKXkixXZDP/nYs5pdzzDzm8XdjnOOXfMPDhKwcotu7h57GxaH38cj/fvSkIVv+zWOVd+eXDE2fY9B7h+dAbVqybw/ODu1Kzu3UrOufItrsEhqa+kZZJWShpWxPpUSZ9KmiNpvqQLotbdE+y3TNKPYz1mWXIgr4CbXp7Fpp25jLi2G03rJoddknPOfW9xCw5JCcCTwPlAe2CApPaFNrsXeM3MugL9gaeCfdsHzzsAfYGnJCXEeMwywcz47YQFzPh6O3+9ohOnptYLuyTnnCsR8Wxx9ABWmtlqMzsAjAcuKbSNAbWDx3WADcHjS4DxZrbfzL4GVgbHi+WYZcKIz1fz+qxMbj+7NZd0aRp2Oc45V2LiGRxNgfVRzzODZdGGAwMlZQLvAbcdZd9YjgmApKGSMiRlZGdnH+t7OCYfLdrEXz5YSr9OjbnjnNal+trOORdvYXeODwBGm1kKcAEwRlKJ1GRmI8ws3czSGzVqVBKHjMmiDTu4ffxcOqXU5W8/7ewDFzrnKpx4XuKTBTSLep4SLIs2hEgfBmY2VVIS0PAo+x7tmKHZsjOXG17MoG6NREYO6kZSok/96pyreOLZ4pgJtJaUJqkakc7uiYW2WQecDSCpHZAEZAfb9ZdUXVIa0BqYEeMxQ5F7MJ8bx8xix76DPHddOsfXTgq7JOeci4u4tTjMLE/SrcCHQAIwyswWSXoAyDCzicBdwEhJdxLpKB9sZgYskvQasBjIA24xs3yAoo4Zr/cQKzPjV6/PY35mDs8O7EaHJnXCLsk55+JGke/pii09Pd0yMjLidvxHJy3n8Y9XcM/5bfn5Ga3i9jrOOVeaJM0ys/TCy8PuHC/33pmbxeMfr+Cn3VIY+sOWYZfjnHNx58HxPcxe9w2/fmM+PdLq86fLTvErqJxzlYIHxzHKytnH0JdmcWLtJJ4Z2I1qVf2jdM5VDj7i3jHYvT+PIaNnsj8vn/FDe1K/ZrWwS3LOuVLjwVFM+QXGHePnsGLLbkb/rDsnHV8r7JKcc65U+fmVYnrwg6VMXrKF4Re15/TWpXdHunPOlRUeHMXw6sx1jPh8Ndf1bs6g3i3CLsc550LhwRGjqau28dsJC/lhm0bc169MjuTunHOlwoMjBmu27uEXY2eR1rAm/7i6K1UT/GNzzlVe/g14FDv2HuT6F2dSReL567pTOykx7JKccy5UHhzf4WB+AbeMm8367Xt5ZmA3UhvUCLsk55wLnV+OewRmxvCJi/hy5Vb+9tPO9EirH3ZJzjlXJniL4zu0bHQct5zViiu6pYRdinPOlRne4jgCSQzpkxZ2Gc45V+Z4i8M551yxeHA455wrFg8O55xzxRLX4JDUV9IySSslDSti/aOS5gY/yyXlRK17UNLC4OeqqOWjJX0dtV+XeL4H55xzh4tb57ikBOBJ4FwgE5gpaaKZLT60jZndGbX9bUDX4PGFwKlAF6A6MEXS+2a2M9j812b2Rrxqd845d2TxbHH0AFaa2WozOwCMBy75ju0HAK8Ej9sDn5tZnpntAeYDfeNYq3POuRjFMziaAuujnmcGy75FUnMgDfgkWDQP6CuphqSGwFlAs6hd/iRpfnCqq/oRjjlUUoakjOzs7O/7XpxzzgXKSud4f+ANM8sHMLOPgPeAr4i0QqYC+cG29wBtge5AfeDuog5oZiPMLN3M0hs18nkznHOupMTzBsAsDm8lpATLitIfuCV6gZn9CfgTgKRxwPJg+cZgk/2SXgB+dbRCZs2atVXS2mJV/18Nga3HuG9F5J/Hf/lncTj/PA5XET6P5kUtjGdwzARaS0ojEhj9gasLbySpLVCPSKvi0LIEoK6ZbZPUCegEfBSsa2xmGyUJuBRYeLRCzOyYmxySMsws/Vj3r2j88/gv/ywO55/H4Sry5xG34DCzPEm3Ah8CCcAoM1sk6QEgw8wmBpv2B8abmUXtngh8EckGdgIDzSwvWDdWUiNAwFzgpni9B+ecc98W17GqzOw9In0V0cvuL/R8eBH75RK5sqqoY/6oBEt0zjlXTGWlc7wsGxF2AWWMfx7/5Z/F4fzzOFyF/Tx0+Bki55xz7rt5i8M551yxeHA455wrFg+O73C0QRorC0nNJH0qabGkRZJuD7umskBSgqQ5kt4Nu5awSaor6Q1JSyUtkdQ77JrCIunO4P/JQkmvSEoKu6aS5sFxBFGDNJ5P5AqvAZKKvNKrEsgD7jKz9kAv4JZK/FlEux1YEnYRZcTjwAdm1hboTCX9XCQ1BX4JpJtZRyK3IvQPt6qS58FxZMUdpLHCMrONZjY7eLyLyJdCkeOOVRaSUoALgefCriVskuoAPwSeBzCzA2aW8917VWhVgWRJVYEawIaQ6ylxHhxHFvMgjZWJpBZEhr+fHm4loXsM+F+gIOxCyoA0IBt4ITh195ykmmEXFQYzywL+BqwDNgI7grH3KhQPDhczSccBbwJ3RM2NUulI6gdsMbNZYddSRlQlMn/O02bWFdgDVMo+QUn1iJyZSAOaADUlDQy3qpLnwXFkxRmkscKTlEgkNMaa2Vth1xOy04CLJa0hcgrzR5JeDrekUGUCmWZ2qBX6BpEgqYzOAb42s2wzOwi8Bfwg5JpKnAfHkf1nkEZJ1Yh0cE08yj4VUjCg5PPAEjN7JOx6wmZm95hZipm1IPLv4hMzq3C/VcbKzDYB6yWdHCw6G1j8HbtUZOuAXsFcQiLyWVS4CwXiOlZVeXakQRpDLisspwGDgAWS5gbLfhOMReYcwG1EBiCtBqwGfhZyPaEws+mS3gBmE7kacQ4VcOgRH3LEOedcsfipKuecc8XiweGcc65YPDicc84ViweHc865YvHgcM45VyweHM6VcZLO9BF4XVniweGcc65YPDicKyGSBkqaIWmupGeD+Tp2S3o0mJ/hY0mNgm27SJomab6kCcEYR0g6SdJkSfMkzZbUKjj8cVHzXYwN7kp2LhQeHM6VAEntgKuA08ysC5APXAPUBDLMrAPwGfC7YJeXgLvNrBOwIGr5WOBJM+tMZIyjjcHyrsAdROaGaUnkbn7nQuFDjjhXMs4GugEzg8ZAMrCFyLDrrwbbvAy8FcxfUdfMPguWvwi8LqkW0NTMJgCYWS5AcLwZZpYZPJ8LtAC+jP/bcu7bPDicKxkCXjSzew5bKN1XaLtjHeNnf9TjfPz/rguRn6pyrmR8DFwh6XgASfUlNSfyf+yKYJurgS/NbAfwjaTTg+WDgM+C2RUzJV0aHKO6pBql+i6ci4H/1uJcCTCzxZLuBT6SVAU4CNxCZFKjHsG6LUT6QQCuA54JgiF6NNlBwLOSHgiO8dNSfBvOxcRHx3UujiTtNrPjwq7DuZLkp6qcc84Vi7c4nHPOFYu3OJxzzhWLB4dzzrli8eBwzjlXLB4czjnnisWDwznnXLH8P2yZxcDdFxcyAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 720x720 with 0 Axes>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "plt.plot(history.history['loss'])\n",
        "plt.title('model loss')\n",
        "plt.ylabel('loss')\n",
        "plt.xlabel('epoch')\n",
        "plt.legend(['train'], loc='upper left')\n",
        "plt.figure(figsize=(10,10))\n",
        "plt.show()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 312
        },
        "id": "L7V3ZfZwMX59",
        "outputId": "3b1e7ed4-a447-4d59-b29e-fd78e79355f1"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXRV5b3/8fc3MxmAkIQhYUggzChBQEYBwQEHxNaxgr21dRa1tvbW3nv78/aOXdeOKiqoaBWK9joUtd5akVkIMgpIBBIgEKYMkBAgc57fHzlS0ABJyGEn53xea7EMZ++zzydnGT7Zz7PPs805h4iIBK8QrwOIiIi3VAQiIkFORSAiEuRUBCIiQU5FICIS5FQEIiJBTkUg0kBm9qqZ/UcD991tZlec73FELgQVgYhIkFMRiIgEORWBBBTfkMxPzGyTmR03s5fNrJOZ/Z+ZlZrZQjOLP2X/G8zsCzMrNrMlZtb/lG1DzGy973lvAlFfe63rzWyj77krzeziJma+x8yyzeywmb1nZsm+x83Mfmtm+WZ21Mw2m9kg37ZrzWyrL9s+M3u8SW+YCCoCCUw3AVcCfYApwP8B/wQkUff//CMAZtYHmA/80LftQ+B9M4swswjgz8DrQAfgf33HxffcIcAc4D4gAZgFvGdmkY0JamYTgf8GbgW6ALnAG77NVwHjfN9HO98+Rb5tLwP3OefigEHAosa8rsipVAQSiJ5xzh1yzu0DlgOrnXMbnHPlwLvAEN9+twF/cc597JyrAn4FtAFGAyOBcOB3zrkq59xbwJpTXuNeYJZzbrVzrsY59wegwve8xpgGzHHOrXfOVQA/A0aZWSpQBcQB/QBzzmU55w74nlcFDDCzts65I8659Y18XZGTVAQSiA6d8nVZPX+P9X2dTN1v4AA452qBvUCKb9s+d/qqjLmnfN0D+LFvWKjYzIqBbr7nNcbXMxyj7rf+FOfcIuBZYCaQb2azzaytb9ebgGuBXDNbamajGvm6IiepCCSY7afuH3Sgbkyeun/M9wEHgBTfY1/pfsrXe4H/dM61P+VPtHNu/nlmiKFuqGkfgHPuaefcUGAAdUNEP/E9vsY5NxXoSN0Q1p8a+boiJ6kIJJj9CbjOzCaZWTjwY+qGd1YCq4Bq4BEzCzezbwOXnvLcF4H7zWyEb1I3xsyuM7O4RmaYD9xlZhm++YX/om4oa7eZDfcdPxw4DpQDtb45jGlm1s43pHUUqD2P90GCnIpAgpZzbhswHXgGKKRuYnmKc67SOVcJfBv4HnCYuvmEd0557lrgHuqGbo4A2b59G5thIfBz4G3qzkJ6Abf7NrelrnCOUDd8VAQ85dt2J7DbzI4C91M31yDSJKYb04iIBDedEYiIBDkVgYhIkFMRiIgEORWBiEiQC/M6QGMlJia61NRUr2OIiLQq69atK3TOJdW3rdUVQWpqKmvXrvU6hohIq2JmuWfapqEhEZEgpyIQEQlyKgIRkSDX6uYI6lNVVUVeXh7l5eVeR/G7qKgounbtSnh4uNdRRCRABEQR5OXlERcXR2pqKqcvFhlYnHMUFRWRl5dHWlqa13FEJEAExNBQeXk5CQkJAV0CAGZGQkJCUJz5iMiFExBFAAR8CXwlWL5PEblwAqYIzqWiqob9xWXUarVVEZHTBE8RVNdSeKyC4hNVzX7s4uJinnvuuUY/79prr6W4uLjZ84iINEbQFEFcVBhtwkMpKK2gue/BcKYiqK6uPuvzPvzwQ9q3b9+sWUREGitoisDM6BgXSUV1DSVlzXtW8MQTT5CTk0NGRgbDhw/nsssu44YbbmDAgAEA3HjjjQwdOpSBAwcye/bsk89LTU2lsLCQ3bt3079/f+655x4GDhzIVVddRVlZWbNmFBE5k4C4fPRUv3j/C7buP3rG7WWVNWDQJjy0wccckNyWJ6cMPOP2X/7yl2zZsoWNGzeyZMkSrrvuOrZs2XLyEs85c+bQoUMHysrKGD58ODfddBMJCQmnHWPHjh3Mnz+fF198kVtvvZW3336b6dOnNzijiEhTBc0ZwVfCw0KorXXU1Ppv0vjSSy897Tr/p59+msGDBzNy5Ej27t3Ljh07vvGctLQ0MjIyABg6dCi7d+/2Wz4RkVMF3BnB2X5zh7oPZW07VEpYSAi9kmL8cjlmTEzMya+XLFnCwoULWbVqFdHR0UyYMKHezwFERkae/Do0NFRDQyJywQTdGYGZkRQbyYnKao5VnH0yt6Hi4uIoLS2td1tJSQnx8fFER0fz5ZdfkpmZ2SyvKSLSXALujKAh4mMiyC+tIL+0grio81+zJyEhgTFjxjBo0CDatGlDp06dTm6bPHkyL7zwAv3796dv376MHDnyvF9PRKQ5WXNfSulvw4YNc1+/MU1WVhb9+/dv1HEKSis4UFJGr6RYYiJbVx825fsVkeBmZuucc8Pq2xZ0Q0Nf6RATQVhICPmlFV5HERHxVNAWQWiIkRgbQWl5FScqm2euQESkNQqYImjKEFdCbAShIUZBKzoraG1DeSLS8gVEEURFRVFUVNTofyRDQ0JIiImkpKyK8qoaP6VrPl/djyAqKsrrKCISQFrXLOkZdO3alby8PAoKChr93NpaR8HRckoPhtIhJsIP6ZrXV3coExFpLn4rAjObA1wP5DvnBtWzvR0wF+juy/Er59wrTXmt8PDw87pj14IPs3hp+U4WPz6BHgkx536CiEgA8efQ0KvA5LNsfwjY6pwbDEwAfm1mnvxKfvfYNMJCQ3h+SY4XLy8i4im/FYFzbhlw+Gy7AHFWt8ZDrG9fTy7f6dg2ituGdePt9XnsL9bSDiISXLycLH4W6A/sBzYDjzrnauvb0czuNbO1Zra2KfMADXHf+J44B7OX7fTL8UVEWiovi+BqYCOQDGQAz5pZ2/p2dM7Nds4Nc84NS0pK8kuYrvHRfGtICvM/29OqLicVETlfXhbBXcA7rk42sAvo52EeHpjQi6qaWl5escvLGCIiF5SXRbAHmARgZp2AvoCn4zI9k2K57uJkXl+1m+ITlV5GERG5YPxWBGY2H1gF9DWzPDP7gZndb2b3+3b5d2C0mW0GPgF+6pwr9Feehnro8l4cr6zh1ZW7vY4iInJB+O1zBM6575xj+37gKn+9flP169yWKwd04pVPd3P3ZT2JbWUrk4qINFZALDHR3GZcnk5JWRVzM3O9jiIi4ncqgnoM7taey3on8tLyna1iDSIRkfOhIjiDGZenU3iskjc+2+N1FBERv1IRnMGIngkMT41n1rKdVFbX+zk3EZGAoCI4ixkTe3OgpJx3N+R5HUVExG9UBGcxrnciF6W047klOVTX6KxARAKTiuAszIwZE9PJLTrBXzYf8DqOiIhfqAjO4cr+nejTKZZnF2VTW6vbRIpI4FERnENIiPHQ5ensyD/G37Ye8jqOiEizUxE0wHUXdSE1IZqZi7N183gRCTgqggYICw3hgQm92LyvhKXb/XM/BBERr6gIGuhbQ7qS3C6KZxfprEBEAouKoIEiwkK4b3wv1uYeYfWus92BU0SkdVERNMJtw7uRGBvJzMXZXkcREWk2KoJGiAoP5Z7L0li+o5CNe4u9jiMi0ixUBI00bWQP2rUJ59lFOisQkcCgImik2Mgwvj8mjYVZh8g6cNTrOCIi501F0ATfG51KbGSY5gpEJCCoCJqgXXQ4d47qwV82H2BnwTGv44iInBcVQRP9YGwakWEhPL8kx+soIiLnRUXQRImxkdw+vDvvbthH3pETXscREWkyFcF5uG98T8xg1tKdXkcREWkyFcF56NKuDTcP7cqba/eSf7Tc6zgiIk2iIjhPD4xPp6bW8eJynRWISOukIjhP3ROiuWFwMvNW7+Hw8Uqv44iINJqKoBk8OKEXJypreOXTXV5HERFpNBVBM+jdKY5rBnXm1ZW7OVpe5XUcEZFGURE0k4cuT6e0vJrXV+V6HUVEpFFUBM1kUEo7Lu+bxMsrdnGistrrOCIiDaYiaEYzJqZz+Hgl8z/b63UUEZEGUxE0o6E9OjCyZwdmL8uhorrG6zgiIg2iImhmD0/szaGjFby1Ls/rKCIiDaIiaGajeyWQ0a09zy/Joaqm1us4IiLnpCJoZmbGwxPTyTtSxnsb93sdR0TknFQEfjCxX0f6d2nLc0uyqa11XscRETkrFYEfmBkPXd6LnILj/PWLg17HERE5KxWBn1wzqAs9k2J4ZlE2zumsQERaLhWBn4SGGA9OSCfrwFEWb8v3Oo6IyBn5rQjMbI6Z5ZvZlrPsM8HMNprZF2a21F9ZvDI1I5mu8W10ViAiLZo/zwheBSafaaOZtQeeA25wzg0EbvFjFk+Eh4Zw//hebNhTzKqcIq/jiIjUy29F4JxbBhw+yy53AO845/b49g/I8ZObh3alY1wkzy7O9jqKiEi9vJwj6APEm9kSM1tnZt89045mdq+ZrTWztQUFBRcw4vmLCg/l3nE9WZlTxLrcI17HERH5Bi+LIAwYClwHXA383Mz61Lejc262c26Yc25YUlLShczYLO4Y0Z0OMRHM1FmBiLRAXhZBHvCRc+64c64QWAYM9jCP30RHhPGDsWks+jKfLftKvI4jInIaL4tgATDWzMLMLBoYAWR5mMev7hzVg7ioMJ5borMCEWlZwvx1YDObD0wAEs0sD3gSCAdwzr3gnMsys78Cm4Ba4CXn3BkvNW3t2kaF8w+jUpm5JJvs/FLSO8Z5HUlEBABrbde3Dxs2zK1du9brGE1y+HglY365iMmDOvPb2zK8jiMiQcTM1jnnhtW3TZ8svoA6xETw3VE9eHfDPv79g61Ua5lqEWkB/DY0JPV7/Oq+VFTX8vKKXXyxv4Rn77iExNhIr2OJSBDTGcEFFh4awr/eMJDf3jaYDXuKmfLMCjbuLfY6logEMRWBR741pCtvPzCa0BDj1hdW8eaaPV5HEpEgpSLw0KCUdrw/Yywjenbgp29v5mfvbNZN70XkglMReCw+JoJX77qUByf0Yv5ne7htViYHSsq8jiUiQURF0AKEhhj/OLkfL0y/hB2HSpnyzAoyd2q1UhG5MFQELcjkQV1YMGMMbduEM+2l1cxZsUv3MRARv1MRtDDpHeNY8NAYJvbryL99sJXH3txIWaXmDUTEf1QELVBcVDizpg/l8av6sODz/Xz7+ZXsKTrhdSwRCVAqghYqJMSYMbE3r3xvOPuLy5jy7AqW6N7HIuIHKoIWbkLfjrw/Yyxd2kVx16trmLlY9z8WkealImgFuidE886Do7lhcDJPfbSN++euo7S8yutYIhIgVAStRHREGL+7LYOfXz+AhVn53DjzU7Lzj3kdS0QCgIqgFTEzfjA2jXl3j6CkrIobZ37KX7cc9DqWiLRyKoJWaGTPBN5/eCy9OsZy/9x1PPXRl9TUat5ARJpGRdBKdWnXhj/dN5LvXNqNmYtzuOvVNRSfqPQ6loi0QiqCViwyLJT//vbF/Pe3LyIzp4gpz67gi/0lXscSkVZGRRAAvnNpd968byRV1Y6bnl/Jnzfs8zqSiLQiKoIAMaR7PO8/PJaLu7bnh29u5Bfvf0GVboUpIg2gIgggSXGRzLt7BN8fk8Yrn+5m2kurKSit8DqWiLRwKoIAEx4awv+bMoDf357Bprxirn9mOev3HPE6loi0YCqCADU1I4V3HhhDRFgIt81axR9X61aYIlK/BhWBmT1qZm2tzstmtt7MrvJ3ODk/A5Lb8v6MsYzqlcg/vbuZJ97eRHmVlrQWkdM19Izg+865o8BVQDxwJ/BLv6WSZtM+OoJXvjecGZen88aavdw2axX7i3UrTBH5u4YWgfn+ey3wunPui1MekxYuNMR4/Oq+zLpzKDkFx5nyzApW5ehWmCJSp6FFsM7M/kZdEXxkZnGArk1sZa4e2Jk/PzSG9tHhTH95NTMXZ+sSUxFpcBH8AHgCGO6cOwGEA3f5LZX4TXrHWBbMGMvVAzvx1EfbuOb3y/k0u9DrWCLioYYWwShgm3Ou2MymA/8CaC2DVio2MoyZd1zCS98dRkV1DdNeWs1D89Zr7kAkSDW0CJ4HTpjZYODHQA7wmt9Sid+ZGVcM6MTHj43nR1f2YWHWISb9eikzF2dTUa0ri0SCSUOLoNrV3R9xKvCsc24mEOe/WHKhRIWH8sik3iz80XjG90niqY+2cfVvl7H4S90fWSRYNLQISs3sZ9RdNvoXMwuhbp5AAkS3DtG8cOdQXvv+pYSEGHe9uoa7/7CGPUUnvI4mIn7W0CK4Daig7vMEB4GuwFN+SyWeGdcnib8+Oo4nrunHypwirvjtUn7z8XbKKjVcJBKorG7EpwE7mnUChvv++plzzpOxg2HDhrm1a9d68dJB52BJOf/1YRbvfb6flPZt+Pn1A7h6YCfM9BESkdbGzNY554bVt62hS0zcCnwG3ALcCqw2s5ubL6K0RJ3bRfH0d4bwxr0jiY0M4/656/junM/IKTjmdTQRaUYNOiMws8+BK786CzCzJGChc26wn/N9g84IvFFdU8vrmbn85m/bKa+u4ftj03hkYm9iIsO8jiYiDXDeZwRAyNeGgooa8VwJAGGhIdw1Jo1Fj0/gxowUZi3dycRfL+G9z/fT0OFFEWmZGvqP+V/N7CMz+56ZfQ/4C/Ch/2JJS5UUF8lTtwzmnQdHkxQXySPzN3D77Ey+PHjU62gi0kQNKgLn3E+A2cDFvj+znXM/PdtzzGyOmeWb2ZZz7DfczKo159C6XNI9ngUPjeW/vnUR2w6Vct3TK/jF+19QUlbldTQRaaQGXzXU6AObjQOOAa855wadYZ9Q4GOgHJjjnHvrXMfVHEHLc+R4Jb/+eBvzVu8hISaCn07ux02XdCUkRFcXibQUTZ4jMLNSMztaz59SMzvrWIBzbhlw+BzZHgbeBvQx1lYsPiaC/7jxIt6fMZbuHaL5yVubuOmFlWzO03JUIq3BWYvAORfnnGtbz58451zb83lhM0sBvkXdOkYSAAaltOOt+0fzq1sGs/fwCW6YuYJ/fnczR45Xeh1NRM7Cyyt/fgf81Dl3zgXxzexeM1trZmsLCgouQDRpqpAQ4+ahXVn0+ATuGp3GG2v2cvmvlzBvdS41tbq6SKQl8tscAYCZpQIf1DdHYGa7+PtdzhKBE8C9zrk/n+2YmiNoXbYdLOX/LdjC6l2HGZTSll/cMIihPeK9jiUSdJrjcwTNzjmX5pxLdc6lAm8BD56rBKT16ds5jjfuHckz3xlCYWklNz2/ksf/93MKSiu8jiYiPn77WKiZzQcmAIlmlgc8iW/FUufcC/56XWl5zIwpg5OZ2K8jzyzK5uUVO/loy0Eeu7IP3x3Vg7BQfTZRxEt+HRryBw0NtX45Bcf41/e+YPmOQvp2iuPJKQMYnZ7odSyRgNYih4YkePVKiuW171/KrDuHcryymjteWs1dr3zG9kOlXkcTCUoqAvGEmXH1wM4s/NF4fnZNP9bmHmHy75bxxNubyD9a7nU8kaCioSFpEY4cr+SZRdm8nrmbsJAQ7hnXk/vG9dTqpiLN5GxDQyoCaVFyi47zPx9t4y+bDpAYG8ljV/bmtmHdNKEscp40RyCtRo+EGGbecQnvPjiatMRo/vndLUz+/XIWbj2k5a5F/ERFIC3SkO7x/Om+Ucy6cyi1tY67X1vL7bMz+XxvsdfRRAKOikBarK8mlD96bBz/PnUg2fnHmDrzUx6ev4G9h094HU8kYGiOQFqN0vIqZi3dyUsrdlJbC98d1YMZE9NpHx3hdTSRFk+TxRJQDpSU8Zu/beet9Xm0jQpnxuXpfHd0DyLDQr2OJtJiabJYAkqXdm146pbBfPjIZQzu1p7//DCLSb9eyoKN+6jVCqcijaYikFarf5e2vPb9S3n9B5cSFxXOo29s5MbnPiVzZ5HX0URaFRWBtHqX9U7ig4fH8qtbBlNQWsHtszO5+w9ryM7XkhUiDaE5Agko5VU1vLxiF88vyaGsqobbhnfjh1f0pmNclNfRRDylyWIJOkXHKnhmUTZzM3OJCAvhvnG9uGdcGtERWrJCgpOKQILWrsLj/M9fv+T/thwkKS6SH13Zh1uGdtWSFRJ0dNWQBK20xBienz6Utx8YRbf4Nvzsnc1c+/RyFn+ZryUrRHxUBBIUhvbowNsPjOa5aZdQUV3LXa+uYdpLq9myr8TraCKeUxFI0DAzrr2oCx8/Np4npwwg68BRrn9mBY+9uZG8I1qyQoKX5ggkaB0tr+L5JTnMWbELB0wb0Z37x/eiU1tdYSSBR5PFImexr7iM3328nXc27CM0xLh9eDfuH9+L5PZtvI4m0mxUBCINsKfoBM8tyeatdXmYwS3DuvHghF50jY/2OprIeVMRiDRC3pETPL8khz+t3YtzcPPQrjw4IZ3uCSoEab1UBCJNsL+4jBeW5vDGZ3upcY5vDUnhocvTSUuM8TqaSKOpCETOw8GScmYty+GPq/dQVVPLjRkpPDQxnV5JsV5HE2kwFYFIM8gvLefFZTt5PTOXiupaplyczMMT0+ndKc7raCLnpCIQaUaFxyp4afkuXlu1m7KqGq4d1IWHJ6XTr3Nbr6OJnJGKQMQPDh+v5OUVO/nDylyOVVQzeWBnHp6UzsDkdl5HE/kGFYGIHxWfqGTOp7t55dNdlJZXc0X/TjwyKZ2Lu7b3OprISSoCkQugpKyKP6zczcsrdlFSVsXlfZN4ZFJvhnSP9zqaiIpA5EIqLa/itVW5vLh8J8UnqhjXJ4lHJ6UztEcHr6NJEFMRiHjgWEU1czNzeXHZToqOVzImPYFHJvZmRM8Er6NJEFIRiHjoRGU1f1y9hxeW7qTwWAUj0jrw6BW9GdUzATPzOp4ECRWBSAtQXlXjK4Qc8ksrGJ4azyOTejM2PVGFIH6nIhBpQcqravjT2r08vySHAyXlDOnenkcm9WZCnyQVgviNikCkBaqoruGtdXk8tziHfcVlDO7ajocn9mZiv46EhKgQpHmpCERasMrqWt7dkMezi7PZe7iM1IRo7hjRnVuGdiM+JsLreBIgVAQirUBVTS0fbj7A3Mxc1uw+QkRYCNdf1IVpI3twSff2GjaS86IiEGllvjx4lHmZe3h3wz6OVVTTv0tbpo/szo0ZKcREhnkdT1ohT4rAzOYA1wP5zrlB9WyfBvwUMKAUeMA59/m5jqsikGByrKKaBRv3MTdzD1kHjhIbGca3hqQwfWQP+nbWqqfScF4VwTjgGPDaGYpgNJDlnDtiZtcA/+qcG3Gu46oIJBg551i/p5h5mbl8sPkAldW1DE+NZ/rIHkwe1JnIsFCvI0oL59nQkJmlAh/UVwRf2y8e2OKcSznXMVUEEuwOH6/krXV7mbd6D7lFJ0iIieCWYd2YNqI73TrodppSv9ZQBI8D/Zxzd5/rmCoCkTq1tY7l2YXMzczlk6xDOGBCnySmj+zBhL4dCdUlqHKKFl0EZnY58Bww1jlXdIZ97gXuBejevfvQ3Nzc5g8r0ortLy7jjc/28MaaveSXVpDSvg13jOjOrcO6kRQX6XU8aQFabBGY2cXAu8A1zrntDTmmzghEzqyqppaPtx5ibmYuK3OKCA81rh7YmekjezAirYMuQQ1iZysCz65DM7PuwDvAnQ0tARE5u/DQEK69qAvXXtSFnIJjzMvcw1vr9vLBpgP07hjLtBHd+fbQrrSNCvc6qrQg/rxqaD4wAUgEDgFPAuEAzrkXzOwl4Cbgq3Ge6jO11al0RiDSOGWVNby/aT/zMnP5PK+ENuGhTM1IZvrIHgxK0W01g4U+UCYiAGzOK2FuZi4LPt9HeVUtg7u1Z/qI7kwZnExUuC5BDWQqAhE5TUlZFe+sz2NuZi45Bcdp1yacm4d2ZdqI7vRMivU6nviBikBE6uWcI3PnYeauzuWjLQeprnWMSU9g2ogeTOzXUWcJAaRFThaLiPfMjFG9EhjVK4H80nL+tGYvf1y9hwfnrScuKoxrBnVmakYKI3sm6HMJAUxnBCJymppax4rsQhZs3MdHWw5yvLKGjnGRXH9xMlMzkrm4aztdhtoKaWhIRJqkvKqGT7Lyee/zfSz+soDKmlpSE6K5ISOFqRnJ9NJ8QquhIhCR81ZSVsVHWw6y4PN9rMwpwjkYlNKWqYNTmDI4mc7toryOKGehIhCRZnXoaDkfbDrAexv38XleCWYwIq0DUzNSuGZQZ9pH685qLY2KQET8Zlfhcd7buJ8FG/exs/A44aHG+D4dmZqRzBX9O9EmQlcetQQqAhHxO+ccW/YdZcHGfby/aT+HjlYQExHKVQM7c0NGMmPTEwkPDfE6ZtBSEYjIBVVT61i9q4j3P9/PXzYd4Gh5NR1iIrjuoi5MzUjmku7xhOhy1AtKRSAinqmormHZ9rrLURdmHaK8qpaU9m2YmpHM1IwU3XLzAlERiEiLcKyimo+3HmTBxv0s31FITa2jX+c4bshIZsrFybrDmh+pCESkxSk6VsGHmw+wYON+1uYeAWBYj3imZiRz7UVdSIjVDXWak4pARFq0vYdP8P6m/SzYsJ9th0oJDTHGpicyZXAyE/t1pEOMLkc9XyoCEWk1vjx4lAUb9/Pexv3sKy4jxGBoj3gm9e/EFf070SspRktcNIGKQERana8uR/046xCfZB3ii/1HAUhNiOaK/p2Y1L8Tw1PjCdMlqQ2iIhCRVm9/cRmffJnPwq2HWJVTRGVNLW2jwri8X0cm9e/E+D5JtGujW3CeiYpARALKsYpqVuwoYGFWPou+zOfw8UrCQoxL0zpwhW8IqXuCrkA6lYpARAJWTa1j494jfLw1n0+yDrEj/xgAfTrFnpxXyOjWPujvp6AiEJGgkVt0nIVZdaWwetdhamodCTERTPQNIV3WO5GYyOC7J5eKQESCUklZFUu3F7Bw6yGWbMvnaHk1EWEhjO6V4Jtw7kiXdm28jnlBqAhEJOhV1dSyZvdhPsnKZ2HWIXKLTgB191SY1K8TVw7oxMDktgF7aaqKQETkFM45svOPnRxCWrfnCM5B57ZRTOrfkSv6d2JUrwSiwgNnCW0VgYjIWRQdq2DxtrohpGU7CjhRWUOb8FAu653IFf07MaFfEh3jWvcd2FQEIiINVF5VQ+bOopNDSAdKygHo36Ut4/okMr53EkNT44kMa11nCyoCEXoNbGgAAAcFSURBVJEmcM6x9cBRlmwrYNn2AtblHqG61hEdEcrIngmM653IuD5JpCW2/GUvVAQiIs3gWEU1q3KKWLa9gGU7Ck5OOHeNb8O4PkmM653E6PQE2ka1vE84qwhERPwgt+i4rxQKWZldyPHKGkJDjEu6t2dc7yTG9UliUEq7FvFhNhWBiIifVdXUsj73CMt2FLBseyGb95UAEB8dztjeSSeHkTq19WbSWUUgInKBFR2rYEV2IUu3F7B8RyEFpRUA9Oscx7g+SVzWO5HhqR0u2CWqKgIREQ8558g6UOo7Wyhg7e4jVNbUEhUewoi0BMb1SWJ8n0R6JcX6bdJZRSAi0oKcqKwmc2cRy7YXsmx7ATsLjwOQ3C6qbtK5TxJjeiXSLrr5Jp1VBCIiLdjewydYvqOuFD7NLqS0opoQg4xu7U8Ww+Cu57eCqopARKSVqK6pZePeYpZtL2DpjkI25RXjHLRrE87DE9O5+7KeTTru2Yog+NZiFRFpwcJCQxiW2oFhqR340VV9OXK8khXZdWcL/rriSEUgItKCxcdEMGVwMlMGJ/vtNXTXZxGRIKciEBEJcioCEZEg57ciMLM5ZpZvZlvOsN3M7GkzyzazTWZ2ib+yiIjImfnzjOBVYPJZtl8D9Pb9uRd43o9ZRETkDPxWBM65ZcDhs+wyFXjN1ckE2ptZF3/lERGR+nk5R5AC7D3l73m+x77BzO41s7VmtragoOCChBMRCRatYrLYOTfbOTfMOTcsKSnJ6zgiIgHFyw+U7QO6nfL3rr7HzmrdunWFZpbbxNdMBAqb+NxApPfjdHo//k7vxekC4f3ocaYNXhbBe8AMM3sDGAGUOOcOnOtJzrkmnxKY2dozrbURjPR+nE7vx9/pvThdoL8ffisCM5sPTAASzSwPeBIIB3DOvQB8CFwLZAMngLv8lUVERM7Mb0XgnPvOObY74CF/vb6IiDRMq5gsbkazvQ7Qwuj9OJ3ej7/Te3G6gH4/Wt39CEREpHkF2xmBiIh8jYpARCTIBU0RmNlkM9vmW+TuCa/zeMnMupnZYjPbamZfmNmjXmfympmFmtkGM/vA6yxeM7P2ZvaWmX1pZllmNsrrTF4xs8d8PyNbzGy+mfnnFmEeC4oiMLNQYCZ1C90NAL5jZgO8TeWpauDHzrkBwEjgoSB/PwAeBbK8DtFC/B74q3OuHzCYIH1fzCwFeAQY5pwbBIQCt3ubyj+CogiAS4Fs59xO51wl8AZ1i94FJefcAefcet/XpdT9oNe7zlMwMLOuwHXAS15n8ZqZtQPGAS8DOOcqnXPF3qbyVBjQxszCgGhgv8d5/CJYiqDBC9wFGzNLBYYAq71N4qnfAf8I1HodpAVIAwqAV3xDZS+ZWYzXobzgnNsH/ArYAxygbvWDv3mbyj+CpQikHmYWC7wN/NA5d9TrPF4ws+uBfOfcOq+ztBBhwCXA8865IcBxICjn1MwsnrqRgzQgGYgxs+nepvKPYCmCJi1wF8jMLJy6EpjnnHvH6zweGgPcYGa7qRsynGhmc72N5Kk8IM8599UZ4lvUFUMwugLY5ZwrcM5VAe8Aoz3O5BfBUgRrgN5mlmZmEdRN+LzncSbPmJlRNwac5Zz7jdd5vOSc+5lzrqtzLpW6/y8WOecC8re+hnDOHQT2mllf30OTgK0eRvLSHmCkmUX7fmYmEaAT516uPnrBOOeqzWwG8BF1M/9znHNfeBzLS2OAO4HNZrbR99g/Oec+9DCTtBwPA/N8vzTtJEgXhHTOrTazt4D11F1pt4EAXWpCS0yIiAS5YBkaEhGRM1ARiIgEORWBiEiQUxGIiAQ5FYGISJBTEYhcQGY2QSucSkujIhARCXIqApF6mNl0M/vMzDaa2Szf/QqOmdlvfevTf2JmSb59M8ws08w2mdm7vjVqMLN0M1toZp+b2Xoz6+U7fOwp6/3P831qVcQzKgKRrzGz/sBtwBjnXAZQA0wDYoC1zrmBwFLgSd9TXgN+6py7GNh8yuPzgJnOucHUrVFzwPf4EOCH1N0boyd1n/QW8UxQLDEh0kiTgKHAGt8v622AfOqWqX7Tt89c4B3f+v3tnXNLfY//AfhfM4sDUpxz7wI458oBfMf7zDmX5/v7RiAVWOH/b0ukfioCkW8y4A/OuZ+d9qDZz7+2X1PXZ6k45esa9HMoHtPQkMg3fQLcbGYdAcysg5n1oO7n5WbfPncAK5xzJcARM7vM9/idwFLfnd/yzOxG3zEizSz6gn4XIg2k30REvsY5t9XM/gX4m5mFAFXAQ9TdpOVS37Z86uYRAP4BeMH3D/2pq3XeCcwys3/zHeOWC/htiDSYVh8VaSAzO+aci/U6h0hz09CQiEiQ0xmBiEiQ0xmBiEiQUxGIiAQ5FYGISJBTEYiIBDkVgYhIkPv/szyDqobyBXMAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 720x720 with 0 Axes>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "dot_img_file = \"d_mod-\"+str(d_model)+\"-d_k-\"+str(d_k)+\"-d_v-\"+str(d_v)+\"-n_heads-\"+str(n_heads)+\"-enc_lays-\"+str(n_encoder_layers)+\"-dec_lays-\"+str(n_decoder_layers)+'-transformer.png'\n",
        "tf.keras.utils.plot_model(transformer, to_file=dot_img_file, show_shapes=True)"
      ],
      "metadata": {
        "id": "RSK1eufbeQPL",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 77
        },
        "outputId": "56b8b7ff-36d9-4bb0-88d6-5748f8ee7e2d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<IPython.core.display.Image object>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAHgAAAA8CAIAAAAiz+n/AAAABmJLR0QA/wD/AP+gvaeTAAAGHElEQVR4nO2cbUhTXxzHf7O5u3ud886aW+Y0NoSi9cKlYLEg6I3vwifwRRC9KfNFg/WwyB4k6GGUFZS9CHoRVrocIZQwoiIKlFWUtMxrDzBTls0eNvdwc07P/8XlP5ZXy7Y8/vN/Pu/Ouef8zvd+uDscLpdJEEJAmH8yFjrA/wUiGhNENCaIaExIkxs9PT1nz55dqCiLDKvVun79+kTzhyd6aGjI6XRij7QIcTqdQ0NDyT1S8aCOjg5ceRYtEolkWg/ZozFBRGOCiMYEEY0JIhoTRDQmiGhMENGYIKIxQURjgojGBBGNCSIaE0Q0Jv4C0ePj4xaLRavVMgzjcrkWOk6KzPA++r9Gc3Ozy+XiOO7mzZvhcHih46RIKqJ5nt+8eXN3d/cfTzMjnZ2dpaWlLMvu2LEDz4rzQSpbx5UrV/x+/x+PMhvDw8OZmZnYlpsvUBIOh2NajxiLxSKTyYS5BoPBbrfTNK1QKD59+mS1WvPz8zmOe/To0erVq5VKJUVRRqPR5XIhhFpaWhiGoWm6s7OzoqIiOzt7xYoVN27cSFR++PBhWVkZTdPZ2dlGozEYDN69e9dgMCSiZmVlIYSmpqaam5tXrVolk8lYlt2yZUt/f79QQRymvr6eYRiJRGIymfLy8qRSKcMwJSUlZrO5oKCAoqicnJx9+/YlMsTj8cOHD+t0Orlcvnbt2vb29hnLchz3c0sA4HA4fuj5XdEIoerqaoPBkGg2NjYCgMViuXDhQlVVVX9/f0dHR1NT09evX798+VJeXr506dLkkffv3w8Gg36/f+PGjVlZWbFYDCEUDoeVSqXdbud5fmRkpKqqanR0VJil0Wi2bduWWO7IkSMymay1tTUQCLx8+dJkMi1btmxkZGS2MEePHgUAt9sdiUQ+f/5cUVEBAF1dXaOjo5FIZPfu3QDQ29srTN+7dy9FUU6n89u3bwcPHszIyHj69OmMZRdMNM/zMw4+ceIEAPj9fvHIlpYWAHj37h1C6NWrVwBw584dcYVk0dFoVKFQ1NXVJa4+efIEAI4dOzZbGEF0KBQSmlevXgUAj8eTPF14cnmeZxgmUTwajVIU1dDQ8Mt7FCMWPe/HO2F7nZycFF8StqCJiQkA0Ov1eXl5W7dubWpq8nq9s1Xr6+sLh8OlpaWJnrKyMplM5na755hHWDQejyfHEzIMDAxEo1Gj0Shcomlaq9VyHDfHyj9nXkR3dXVt2rRJrVZTFLV///65TKFp+sGDB2az+fjx43q9vq6ujud58bBAIAAACoUiuZNl2VAolH7sSCQCAIcOHZL8y+DgYDQaTb8yzIfoDx8+VFZWarVat9sdDAbtdvscJ65Zs+b27ds+n89mszkcjjNnzojHsCwLANO0BgKBgoKC9JOr1WoAOHfuXPJPvqenJ/3KMB+iPR7PxMREQ0ODXq+Xy+XiT0lmxOfzvX79GgDUavXJkydNJpPQnIbRaFQoFM+ePUv0uN3uWCy2bt269JMLh43e3t70S4lJRXRubq7P5/N6vaFQSNjdkiksLASAe/fuff/+/e3bt3PcPX0+X319PcdxsVjsxYsXg4OD5eXl4mFyuXzPnj23bt26du3a2NiYx+PZtWvX8uXLd+7cmcKNiItv3769ra3t0qVLY2Njk5OTw8PDHz9+TL8ywO+foxFCz58/LyoqomnabDZbrVaapgFAp9O1trYKA2w2W25uLsuytbW1Fy9eBACDwXDgwAGGYQCguLj4/fv3ly9fViqVAFBUVPTmzRuv17thwwaVSrVkyZL8/PzGxsZ4PO71ektKSgBAKpWaTCan04kQmpqaOn36dHFxcWZmpkqlqqysHBgYENYVDrzJYc6fPy8sunLlysePH586dSonJwcANBrN9evX29vbNRoNAKhUqra2NoTQ+Pi4zWYrLCyUSqVqtbq6urqvr09cNoVTRyqiCb9ELPoveHu3OCCiMUFEY4KIxgQRjQkiGhNENCaIaEwQ0ZggojFBRGOCiMYEEY0JIhoTRDQmiGhMENGYmOEjx9raWvw5Fj0/PNE6na6mpmahoiwmampqdDpdco8Ekb9jwwLZozFBRGOCiMYEEY2JfwAA6fCQC71p+gAAAABJRU5ErkJggg==\n"
          },
          "metadata": {},
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# transformer = keras.models.load_model('./Model-2/')"
      ],
      "metadata": {
        "id": "Oly3kYeU3FnV"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Testing"
      ],
      "metadata": {
        "id": "KJ_Gx7RVeNla"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We use greedy selection."
      ],
      "metadata": {
        "id": "9Qzw8P_iemwG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def greedy_translate(model, source_sentence, target_sentence_start=sp_ben_bpe.bos_id()):\n",
        "  # Tokenizing and padding\n",
        "  source_seq = tokenize(source_sentence, sp_en_bpe)\n",
        "  source_seq = tf.keras.preprocessing.sequence.pad_sequences(source_seq, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "  predict_seq = [[target_sentence_start]]\n",
        "  \n",
        "  predict_sentence = [target_sentence_start] # Deep copy here to prevent updates on target_sentence_start\n",
        "  while predict_sentence[-1] != sp_ben_bpe.eos_id() and len(predict_sentence) < MAX_TOKENS:\n",
        "    predict_output = model([np.array(source_seq), np.array(predict_seq)], training=None)\n",
        "    predict_label = tf.argmax(predict_output, axis=-1) # Pick the label with highest softmax score\n",
        "    predict_seq = tf.concat([predict_seq, predict_label], axis=-1) # Updating the prediction sequence\n",
        "    predict_sentence.append(predict_label[0][0].numpy())\n",
        "  \n",
        "  if len(predict_sentence) == MAX_TOKENS:\n",
        "      return detokenize(predict_sentence, sp_ben_bpe, True, False)\n",
        "  return detokenize(predict_sentence, sp_ben_bpe)"
      ],
      "metadata": {
        "id": "zfg1AaKDnPQM"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Playing aroung with the model to see where it makes mistake, as of now, because of low training data, I feel the model is not well exposed to various terms and words."
      ],
      "metadata": {
        "id": "4r3bC_jieuaH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"I love you.\"\n",
        "print(\"English Sentence: \", sentence)\n",
        "print(\"Bengali Sentence: \", greedy_translate(transformer, sentence))"
      ],
      "metadata": {
        "id": "KtWeJl1JoapJ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "85c885d3-fa79-4679-dd20-13631f963720"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English Sentence:  I love you.\n",
            "Bengali Sentence:  আমি তোমাকে ভালবাসি।\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"Hello!\"\n",
        "print(\"English Sentence: \", sentence)\n",
        "print(\"Bengali Sentence: \", greedy_translate(transformer, sentence))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "21O_azGuSs02",
        "outputId": "0b1ffc70-36af-4928-bad1-74345ef4dcbb"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English Sentence:  Hello!\n",
            "Bengali Sentence:  হ্যালো!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"Let's start!\"\n",
        "print(\"English Sentence: \", sentence)\n",
        "print(\"Bengali Sentence: \", greedy_translate(transformer, sentence))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lk9bKTwRS321",
        "outputId": "616c2e18-339b-42b1-f7e6-7ee0edee3239"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English Sentence:  Let's start!\n",
            "Bengali Sentence:  আসুন শুরু যাক!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"Life is like a walking shadow.\"\n",
        "print(\"English Sentence: \", sentence)\n",
        "print(\"Bengali Sentence: \", greedy_translate(transformer, sentence))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Z9y4UFWqTE-_",
        "outputId": "3b69fd58-bc8a-41ca-95de-4b2bfb68c817"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English Sentence:  Life is like a walking shadow.\n",
            "Bengali Sentence:  জীবন খুবই খারাপ।\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"I need more data!\"\n",
        "print(\"English Sentence: \", sentence)\n",
        "print(\"Bengali Sentence: \", greedy_translate(transformer, sentence))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9j7JpbhHTPCz",
        "outputId": "13844136-e374-427b-b73d-565d4530e7dc"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English Sentence:  I need more data!\n",
            "Bengali Sentence:  আমি কি আরও বেশি তথ্য পেয়েছি!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"I need more data,\"\n",
        "print(\"English Sentence: \", sentence)\n",
        "print(\"Bengali Sentence: \", greedy_translate(transformer, sentence))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7qEGaIhvTUiS",
        "outputId": "d45fb0ea-f95d-44c7-bea2-924f299ce319"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English Sentence:  I need more data,\n",
            "Bengali Sentence:  আমি আরও বিস্তারিত বিস্তারিত বিস্তারিত বিস্তারিত বলব\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"I need more data.\"\n",
        "print(\"English Sentence: \", sentence)\n",
        "print(\"Bengali Sentence: \", greedy_translate(transformer, sentence))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "x7bPQf9fTY2o",
        "outputId": "e97d400b-e2f6-481d-eb67-c8eb189e5ace"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English Sentence:  I need more data.\n",
            "Bengali Sentence:  আমি আরও বিস্তারিত তথ্য পেয়েছি।\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Machine Translation Evaluation Metric"
      ],
      "metadata": {
        "id": "M_XY969fe95l"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Being a small model for obvious computational resource constraint. We see that the MT Evaluation metrics are quite poor."
      ],
      "metadata": {
        "id": "v0OV3TosfXN9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "sentences = list(test_val_data['src'][:250])\n",
        "refs = [list(test_val_data['tgt'][:250])]\n",
        "sys = [greedy_translate(transformer, sentence) for sentence in sentences]"
      ],
      "metadata": {
        "id": "9XNckV-wBgiN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "bleu = BLEU()\n",
        "bleu.corpus_score(sys, refs)"
      ],
      "metadata": {
        "id": "otRhPJC1VKyj",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c079e794-1aa8-4ea8-ea97-40f3c1e49dbd"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "BLEU = 0.64 12.9/1.8/0.4/0.0 (BP = 0.967 ratio = 0.968 hyp_len = 3468 ref_len = 3583)"
            ]
          },
          "metadata": {},
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "chrf = CHRF()\n",
        "chrf.corpus_score(sys, refs)"
      ],
      "metadata": {
        "id": "mlhToSrvYNqd",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6fd63ac5-7a60-4bd8-c675-5d9a5620af2a"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "chrF2 = 19.99"
            ]
          },
          "metadata": {},
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "ter = TER()\n",
        "ter.corpus_score(sys, refs)"
      ],
      "metadata": {
        "id": "DtIadLMJYe1w",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3de354ad-b890-42c7-aea1-ad7a1544b955"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TER = 104.46"
            ]
          },
          "metadata": {},
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We look at the model's output for unseen test data which is prepared from `Indic-WAT`."
      ],
      "metadata": {
        "id": "2qAQbxoGfjun"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "val_perf = pd.DataFrame({\"src\":sentences, \"sys\":sys, \"refs\":refs[0]})\n",
        "val_perf"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "_tCWU_NBVOwh",
        "outputId": "d07ed052-343a-45be-f113-0e43a80fc76e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                   src  \\\n",
              "0    Through her work, she spread the message of th...   \n",
              "1    Interacting with beneficiaries and store owner...   \n",
              "2                                       These include:   \n",
              "3    It is no surprise that today Japan is India’s ...   \n",
              "4           Record growth in last two and a half years   \n",
              "..                                                 ...   \n",
              "245  •India will emerge as a global leader in trans...   \n",
              "246        BHIM stands for Bharat Interface for Money.   \n",
              "247          Heartiest congratulations, Beti Shivangi.   \n",
              "248                But, we are now awaiting the rains.   \n",
              "249  As reported by the Swiss Bank Authorities, the...   \n",
              "\n",
              "                                                   sys  \\\n",
              "0    তিনি বলেন, ‘এই ধরনের উদ্যোগ ও পরিবেশ ও অক্ষমতা...   \n",
              "1    কেন্দ্রীয় মন্ত্রী শ্রী নরেন্দ্র মোদী ও নিকোবর...   \n",
              "2                         এই ধরনের বৈশিষ্ট্য নিম্নরূপ:   \n",
              "3    এটি বিশ্বের বৃহত্তম বিশ্বের বৃহত্তম অর্থনীতির ...   \n",
              "4    গত বছর ধরে দুই পক্ষের মধ্যে শীর্ষে শীর্ষে শীর্...   \n",
              "..                                                 ...   \n",
              "245  @আরএম-এর দশকের মধ্যে ভারত-প্রশান্ত মহাসাগরীয় ...   \n",
              "246                   ভারত-বাংলাদেশ কার্ড নিয়ে এসেছে।   \n",
              "247                       হৃদয়-দাওয়া, হেমন্ত-পাউন্ড।   \n",
              "248                   কিন্তু, আমরা বৃষ্টি পর্যন্ত আছি।   \n",
              "249  সম্প্রতি ভারতীয় বিমান সংস্থা জানিয়েছে, ‘করোন...   \n",
              "\n",
              "                                                  refs  \n",
              "0    নিজের কর্মেরউদাহরণ স্থাপন করে তিনি মানুষকে সেব...  \n",
              "1    সারা দেশের ৫ হাজারেরও বেশি স্থান থেকে দোকান মা...  \n",
              "2                         এই প্রকল্পগুলির মধ্যে রয়েছে-  \n",
              "3    জাপান যে বর্তমানে ভারতের চতুর্থ বৃহত্তম প্রত্য...  \n",
              "4                    গত আড়াই বছরেরেকর্ড পরিমাণ অগ্রগতি  \n",
              "..                                                 ...  \n",
              "245  · সর্বাধুনিক প্রযুক্তি এবং দক্ষ শ্রমিকের ব্যবহ...  \n",
              "246                  BHIM মানে ভারত ইন্টারফেস ফর মানি।  \n",
              "247               কন্যা শিবাঙ্গীকে অনেক অনেক অভিনন্দন!  \n",
              "248                           বর্ষার প্রতীক্ষা চলছেই ।  \n",
              "249  স্যুইস ব্যাঙ্কবলেছে যে ভারতীয়দের দ্বারা জমা ধন...  \n",
              "\n",
              "[250 rows x 3 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-cb7712e6-fe3d-4235-911b-c2d28bfab6fc\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>src</th>\n",
              "      <th>sys</th>\n",
              "      <th>refs</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Through her work, she spread the message of th...</td>\n",
              "      <td>তিনি বলেন, ‘এই ধরনের উদ্যোগ ও পরিবেশ ও অক্ষমতা...</td>\n",
              "      <td>নিজের কর্মেরউদাহরণ স্থাপন করে তিনি মানুষকে সেব...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Interacting with beneficiaries and store owner...</td>\n",
              "      <td>কেন্দ্রীয় মন্ত্রী শ্রী নরেন্দ্র মোদী ও নিকোবর...</td>\n",
              "      <td>সারা দেশের ৫ হাজারেরও বেশি স্থান থেকে দোকান মা...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>These include:</td>\n",
              "      <td>এই ধরনের বৈশিষ্ট্য নিম্নরূপ:</td>\n",
              "      <td>এই প্রকল্পগুলির মধ্যে রয়েছে-</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>It is no surprise that today Japan is India’s ...</td>\n",
              "      <td>এটি বিশ্বের বৃহত্তম বিশ্বের বৃহত্তম অর্থনীতির ...</td>\n",
              "      <td>জাপান যে বর্তমানে ভারতের চতুর্থ বৃহত্তম প্রত্য...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Record growth in last two and a half years</td>\n",
              "      <td>গত বছর ধরে দুই পক্ষের মধ্যে শীর্ষে শীর্ষে শীর্...</td>\n",
              "      <td>গত আড়াই বছরেরেকর্ড পরিমাণ অগ্রগতি</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>245</th>\n",
              "      <td>•India will emerge as a global leader in trans...</td>\n",
              "      <td>@আরএম-এর দশকের মধ্যে ভারত-প্রশান্ত মহাসাগরীয় ...</td>\n",
              "      <td>· সর্বাধুনিক প্রযুক্তি এবং দক্ষ শ্রমিকের ব্যবহ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>246</th>\n",
              "      <td>BHIM stands for Bharat Interface for Money.</td>\n",
              "      <td>ভারত-বাংলাদেশ কার্ড নিয়ে এসেছে।</td>\n",
              "      <td>BHIM মানে ভারত ইন্টারফেস ফর মানি।</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>247</th>\n",
              "      <td>Heartiest congratulations, Beti Shivangi.</td>\n",
              "      <td>হৃদয়-দাওয়া, হেমন্ত-পাউন্ড।</td>\n",
              "      <td>কন্যা শিবাঙ্গীকে অনেক অনেক অভিনন্দন!</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>248</th>\n",
              "      <td>But, we are now awaiting the rains.</td>\n",
              "      <td>কিন্তু, আমরা বৃষ্টি পর্যন্ত আছি।</td>\n",
              "      <td>বর্ষার প্রতীক্ষা চলছেই ।</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>249</th>\n",
              "      <td>As reported by the Swiss Bank Authorities, the...</td>\n",
              "      <td>সম্প্রতি ভারতীয় বিমান সংস্থা জানিয়েছে, ‘করোন...</td>\n",
              "      <td>স্যুইস ব্যাঙ্কবলেছে যে ভারতীয়দের দ্বারা জমা ধন...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>250 rows × 3 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-cb7712e6-fe3d-4235-911b-c2d28bfab6fc')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-cb7712e6-fe3d-4235-911b-c2d28bfab6fc button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-cb7712e6-fe3d-4235-911b-c2d28bfab6fc');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 62
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "val_perf.to_csv(\"val_data_perf.csv\")"
      ],
      "metadata": {
        "id": "tq3Inx4xVznw"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We below see how the model predicts on the held out data from the same dataset. The model performs slightly better."
      ],
      "metadata": {
        "id": "7bLAQ-62gHVA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "sentences = list(train_dev_data['src'][:250])\n",
        "refs = [list(train_dev_data['tgt'][:250])]\n",
        "sys = [greedy_translate(transformer, sentence) for sentence in sentences]"
      ],
      "metadata": {
        "id": "Gt0e4NdmV7aZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "bleu = BLEU()\n",
        "bleu.corpus_score(sys, refs)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UB9GVDhpWhWO",
        "outputId": "38c4b319-c88a-4282-f404-6a8bd4397fff"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "BLEU = 0.80 15.2/2.5/0.4/0.0 (BP = 1.000 ratio = 1.064 hyp_len = 2561 ref_len = 2406)"
            ]
          },
          "metadata": {},
          "execution_count": 67
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "chrf = CHRF()\n",
        "chrf.corpus_score(sys, refs)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vCMAl0uLWiGl",
        "outputId": "afe6f8e1-b8e7-41e6-924a-f0895c6a078b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "chrF2 = 20.25"
            ]
          },
          "metadata": {},
          "execution_count": 68
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "ter = TER()\n",
        "ter.corpus_score(sys, refs)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eKDOldlLWkk-",
        "outputId": "783b0009-e230-4019-d19e-d9b3a039b6fc"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "TER = 107.92"
            ]
          },
          "metadata": {},
          "execution_count": 69
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "train_dev_perf = pd.DataFrame({\"src\":sentences, \"sys\":sys, \"refs\":refs[0]})\n",
        "train_dev_perf"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "fbRC21OiYUlg",
        "outputId": "f4d177e2-af07-41b7-dd07-d4d2bc690f31"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                   src  \\\n",
              "0    We beg our Protestant and Jewish friends to pu...   \n",
              "1    Sa'd advised Muhammad: \"Don't be hard on him. ...   \n",
              "2                         Photo by 'Save Gaza Project'   \n",
              "3    So, therefore, we need to test batteries under...   \n",
              "4      This party is also contesting in the elections.   \n",
              "..                                                 ...   \n",
              "245            All of them went in favour of Congress.   \n",
              "246  But even if India is not as safe as Europe, it...   \n",
              "247                            What does a human have?   \n",
              "248                           Have you seen it lately?   \n",
              "249                                I groaned a little.   \n",
              "\n",
              "                                                   sys  \\\n",
              "0    আমরা যখন আমাদের খ্রিস্টানদেরকে সমর্থন করি, তখন...   \n",
              "1    ড.পূ.পূ.পূ: \"আমরা তোমাদের কাছে তাঁর কাছে কথা ব...   \n",
              "2    ছবি ফ্লিকার ব্যবহারকারী বলেন, ‘আমি ‘আমি এই প্র...   \n",
              "3    সুতরাং, আমাদের লক্ষ্য ছিল যে, আমাদের জন্য আমাদ...   \n",
              "4            এই নির্বাচনে রাজনৈতিক দলের নির্বাচন চলছে।   \n",
              "..                                                 ...   \n",
              "245        কংগ্রেস থেকে তাঁদের মধ্যে ক্ষোভ প্রকাশ করে।   \n",
              "246  কিন্তু ভারত যদি বাংলাদেশের বাইরে থাকে, তাহলে ব...   \n",
              "247                                     মানুষের কি কি?   \n",
              "248                                   আপনি কি দেখেছেন?   \n",
              "249                                আমি একটু খারাপ আছি।   \n",
              "\n",
              "                                                  refs  \n",
              "0    কোন কোন ক্ষেত্রে কর্তৃপক্ষ এবং ধর্মীয় নেতারা ...  \n",
              "1    সা'দ মুহাম্মাদকে বলেন: \"তার প্রতি কঠোর হবেন না...  \n",
              "2                          ছবি \"সেভ গাজা প্রজেক্টের\"।'  \n",
              "3    অতএব, আমআদের কিছুটা মান অবস্থাগুলির অধীনে ব্যা...  \n",
              "4       নির্বাচনে এই দলের মধ্যেই প্রতিদ্বন্দ্বিতা হবে।  \n",
              "..                                                 ...  \n",
              "245                      এ সবই কংগ্রেসের পক্ষে গিয়েছে।  \n",
              "246  ভারত ইউরোপের মত নিরাপদ নয় ঠিকই, তবে বাংলাদেশের...  \n",
              "247                              কোনও পুরুষেরই কি আছে?  \n",
              "248                          ইতি পূর্বে দেখা হয়েছে কি?  \n",
              "249                                   একটু আদরও করলাম।  \n",
              "\n",
              "[250 rows x 3 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-ce9ac3e5-47b2-4cff-915b-a9d866d493ed\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>src</th>\n",
              "      <th>sys</th>\n",
              "      <th>refs</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>We beg our Protestant and Jewish friends to pu...</td>\n",
              "      <td>আমরা যখন আমাদের খ্রিস্টানদেরকে সমর্থন করি, তখন...</td>\n",
              "      <td>কোন কোন ক্ষেত্রে কর্তৃপক্ষ এবং ধর্মীয় নেতারা ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Sa'd advised Muhammad: \"Don't be hard on him. ...</td>\n",
              "      <td>ড.পূ.পূ.পূ: \"আমরা তোমাদের কাছে তাঁর কাছে কথা ব...</td>\n",
              "      <td>সা'দ মুহাম্মাদকে বলেন: \"তার প্রতি কঠোর হবেন না...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Photo by 'Save Gaza Project'</td>\n",
              "      <td>ছবি ফ্লিকার ব্যবহারকারী বলেন, ‘আমি ‘আমি এই প্র...</td>\n",
              "      <td>ছবি \"সেভ গাজা প্রজেক্টের\"।'</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>So, therefore, we need to test batteries under...</td>\n",
              "      <td>সুতরাং, আমাদের লক্ষ্য ছিল যে, আমাদের জন্য আমাদ...</td>\n",
              "      <td>অতএব, আমআদের কিছুটা মান অবস্থাগুলির অধীনে ব্যা...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>This party is also contesting in the elections.</td>\n",
              "      <td>এই নির্বাচনে রাজনৈতিক দলের নির্বাচন চলছে।</td>\n",
              "      <td>নির্বাচনে এই দলের মধ্যেই প্রতিদ্বন্দ্বিতা হবে।</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>245</th>\n",
              "      <td>All of them went in favour of Congress.</td>\n",
              "      <td>কংগ্রেস থেকে তাঁদের মধ্যে ক্ষোভ প্রকাশ করে।</td>\n",
              "      <td>এ সবই কংগ্রেসের পক্ষে গিয়েছে।</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>246</th>\n",
              "      <td>But even if India is not as safe as Europe, it...</td>\n",
              "      <td>কিন্তু ভারত যদি বাংলাদেশের বাইরে থাকে, তাহলে ব...</td>\n",
              "      <td>ভারত ইউরোপের মত নিরাপদ নয় ঠিকই, তবে বাংলাদেশের...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>247</th>\n",
              "      <td>What does a human have?</td>\n",
              "      <td>মানুষের কি কি?</td>\n",
              "      <td>কোনও পুরুষেরই কি আছে?</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>248</th>\n",
              "      <td>Have you seen it lately?</td>\n",
              "      <td>আপনি কি দেখেছেন?</td>\n",
              "      <td>ইতি পূর্বে দেখা হয়েছে কি?</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>249</th>\n",
              "      <td>I groaned a little.</td>\n",
              "      <td>আমি একটু খারাপ আছি।</td>\n",
              "      <td>একটু আদরও করলাম।</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>250 rows × 3 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-ce9ac3e5-47b2-4cff-915b-a9d866d493ed')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-ce9ac3e5-47b2-4cff-915b-a9d866d493ed button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-ce9ac3e5-47b2-4cff-915b-a9d866d493ed');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 70
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "val_perf.to_csv(\"train_dev_data_perf.csv\")"
      ],
      "metadata": {
        "id": "g20LLW2JYllI"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "We below sample the words using the Transformer predictions for probability values. But here the difference is we \"sample\" the value and not just choose the max value. This gives us interesting variations of a sentence's translation. It is based on the idea of minimizing Bayes's Risk."
      ],
      "metadata": {
        "id": "qSx-hRQpgTPN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def sampling_translate(model, source_sentence, temperature=0.6, target_sentence_start=sp_ben_bpe.bos_id()):\n",
        "  # Tokenizing and padding\n",
        "  source_seq = tokenize(source_sentence, sp_en_bpe)\n",
        "  source_seq = tf.keras.preprocessing.sequence.pad_sequences(source_seq, maxlen = MAX_TOKENS, padding='post', truncating='post')\n",
        "  predict_seq = [[target_sentence_start]]\n",
        "  logprob = 0\n",
        "  \n",
        "  predict_sentence = [target_sentence_start] # Deep copy here to prevent updates on target_sentence_start\n",
        "  while predict_sentence[-1] != sp_ben_bpe.eos_id() and len(predict_sentence) < MAX_TOKENS:\n",
        "    predict_output = model([np.array(source_seq), np.array(predict_seq)], training=None)\n",
        "    probs = predict_output[-1].numpy()[0]\n",
        "    log_probs = np.log(probs)\n",
        "    u = np.random.uniform(low=1e-6, high=1.0 - 1e-6, size=log_probs.shape)\n",
        "    g = -np.log(-np.log(u))\n",
        "    sample_idx = np.argmax(log_probs + g * temperature, axis=-1)\n",
        "    sample_prob = predict_output[-1].numpy()[0][sample_idx]\n",
        "    logprob += np.log(sample_prob)\n",
        "    predict_label = np.arange(predict_output.shape[-1])[sample_idx] # Sample the label softmax score\n",
        "    predict_seq = tf.concat([predict_seq, tf.cast([[predict_label]], dtype=tf.int32)], axis=-1) # Updating the prediction sequence\n",
        "    predict_sentence.append(predict_label)\n",
        "  \n",
        "  if len(predict_sentence) == MAX_TOKENS:\n",
        "      return detokenize(predict_sentence, sp_ben_bpe, True, False), logprob\n",
        "  return detokenize(predict_sentence, sp_ben_bpe), logprob"
      ],
      "metadata": {
        "id": "2dmVo_eQhE8v"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"I love you.\"\n",
        "print(\"English Sentence: \", sentence)\n",
        "print(\"Bengali Sentence: \", sampling_translate(transformer, sentence))"
      ],
      "metadata": {
        "id": "CB5iHHlxh1cC",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "fa190bc1-079f-474d-c931-24dfc61d51a5"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English Sentence:  I love you.\n",
            "Bengali Sentence:  ('আমি ভালো ভালোবাসি।', -5.938353013189044)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"Hi.\"\n",
        "print(\"English Sentence: \", sentence)\n",
        "print(\"Bengali Sentence: \", sampling_translate(transformer, sentence))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IOtqI4xzSg0c",
        "outputId": "43dde7bf-7b51-4e96-f06c-ad21bfacbf1b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English Sentence:  Hi.\n",
            "Bengali Sentence:  ('এক্কেবারে,', -11.502934694290161)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We below generaete multiple outcomes of a sentence's translation with a ROUGHE SIMILARITY score. It shows interesting results of which sentences the model thinks are closer to each other and to the translation."
      ],
      "metadata": {
        "id": "ACmQjLiHgpYk"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def generate_samples(sentence, n_samples, model=None, temperature=0.6):\n",
        "    samples, log_probs = [], []\n",
        "    for _ in range(n_samples):\n",
        "        sample, logp = sampling_translate(model, sentence, temperature)\n",
        "        samples.append(sample)\n",
        "        log_probs.append(logp)\n",
        "    return samples, log_probs"
      ],
      "metadata": {
        "id": "xIxJ9HqTtxHY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def rouge1_similarity(system, reference):\n",
        "    sys_counter = Counter(system)\n",
        "    ref_counter = Counter(reference)\n",
        "    overlap = 0\n",
        "    for token in sys_counter:\n",
        "        token_count_sys = sys_counter[token]\n",
        "        token_count_ref = ref_counter[token]\n",
        "        overlap += min(token_count_ref, token_count_sys)\n",
        "    precision = overlap / sum(sys_counter.values())\n",
        "    recall = overlap / sum(ref_counter.values())\n",
        "    if precision + recall != 0:\n",
        "        rouge1_score = 2 * ((precision * recall)/(precision + recall))\n",
        "    else:\n",
        "        rouge1_score = 0     \n",
        "    return rouge1_score"
      ],
      "metadata": {
        "id": "uhIlVoVwt7YC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def weighted_avg_overlap(samples, log_probs):\n",
        "    scores = {}\n",
        "    for index_candidate, candidate in enumerate(samples):    \n",
        "        overlap, weight_sum = 0.0, 0.0\n",
        "        for index_sample, (sample, logp) in enumerate(zip(samples, log_probs)):           \n",
        "            if index_candidate == index_sample:\n",
        "                continue\n",
        "            sample_p = float(np.exp(logp))\n",
        "            weight_sum += sample_p\n",
        "            sample_overlap = rouge1_similarity(candidate, sample)\n",
        "            overlap += sample_p * sample_overlap\n",
        "        score = overlap / weight_sum\n",
        "        scores[index_candidate] = score\n",
        "    return scores"
      ],
      "metadata": {
        "id": "7KxiaRENt-C_"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def mbr_decode(sentence, n_samples, model=None, temperature=0.6):\n",
        "    samples, log_probs = generate_samples(sentence, n_samples, model, temperature)\n",
        "    scores = weighted_avg_overlap(samples, log_probs)\n",
        "    max_score_key = max(scores, key=scores.get)\n",
        "    return Counter({sample:score for sample, score in zip(samples, scores.values())})"
      ],
      "metadata": {
        "id": "ny5EYLFWt_5u"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Playing around with the model a bit."
      ],
      "metadata": {
        "id": "wux6ZjUlg9Il"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"I love you.\"\n",
        "translated_sentence = mbr_decode(sentence, 10, transformer)\n",
        "print(\"English: \", sentence)\n",
        "print(\"Bengali: \", translated_sentence)"
      ],
      "metadata": {
        "id": "lzrxKRMAaOgr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3202eb57-0177-4384-9ceb-f679c2acc346"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English:  I love you.\n",
            "Bengali:  Counter({'আমি তোমাকে ভালবাসি।': 0.9714389708969126, 'আমি তোমাকে ভালোবাসি।': 0.9566426896653991, 'তোমাকে ভালোবাসি।': 0.8540486552294865, 'আমি ভালবাসি।': 0.7617118809931761, 'আমি তোমার ভালবাসা উপভোগ করি।': 0.7612547697912802, 'তুমি ভালো করেছো.': 0.6337957026536614, 'আমি তো খুশি।': 0.5696605045023561, 'আমি আপনার বন্ধু।': 0.45223510334778483})\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"Modiji is India's Prime Minister.\"\n",
        "translated_sentence = mbr_decode(sentence, 10, transformer)\n",
        "print(\"English: \", sentence)\n",
        "print(\"Bengali: \", translated_sentence)"
      ],
      "metadata": {
        "id": "3ACvLvfouZz0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1aa1485c-651b-4066-e0fd-04f4c3bddb45"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English:  Modiji is India's Prime Minister.\n",
            "Bengali:  Counter({'কিন্তু নরেন্দ্র মোদী সরকারের।': 0.8481085839527179, 'কিন্তু নরেন্দ্র মোদী সরকার।': 0.8195907344954799, 'কিন্তু নরেন্দ্র মোদীর বায়োপ্রি।': 0.7855588704289592, 'নরেন্দ্র মোদি সরকারের।': 0.7792983727877886, 'নরেন্দ্র মোদী সরকারের ভারত।': 0.7783406115892316, 'নরেন্দ্র মোদী নরেন্দ্র মোদি।': 0.7538688664938117, 'মোদীর কথা নরেন্দ্র মোদী।': 0.7447190169731827, 'কিন্তু নরেন্দ্র মোদীর এই বৈঠকে নরেন্দ্র মোদী।': 0.7115931266867341, 'কিন্তু নরেন্দ্র মোদীর প্রধানমন্ত্রী নরেন্দ্র মোদীর।': 0.6610375561473062, 'নরেন্দ্র মোদী বলেন, ‘\\u200c\\u200c\\u200c': 0.6327239463255885})\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"Thank You!\"\n",
        "translated_sentence = mbr_decode(sentence, 10, transformer)\n",
        "print(\"English: \", sentence)\n",
        "print(\"Bengali: \", translated_sentence)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YVU_W-dmQmXN",
        "outputId": "1cdaecb9-d90f-4164-e98c-a4dfdce8ed35"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English:  Thank You!\n",
            "Bengali:  Counter({'ধন্যবাদ!': 0.994706436015923, 'ধন্যবাদ তোমার!': 0.7265599904143797, 'ধন্যবাদ ধন্যবাদ!': 0.6666666666666669})\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"India is a shy country with second largest population.\"\n",
        "translated_sentence = mbr_decode(sentence, 10, transformer)\n",
        "print(\"English: \", sentence)\n",
        "print(\"Bengali: \", translated_sentence)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-_yaEmqSTmOu",
        "outputId": "3861264d-8208-4664-b924-f83feada91d7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English:  India is a shy country with second largest population.\n",
            "Bengali:  Counter({'বিশ্বে বিশ্বের অনেক সংখ্যক দেশের জনসংখ্যা রয়েছে।': 0.7946034939254609, 'বিশ্বের জনসংখ্যার মধ্যে সবচেয়ে বেশি সংখ্যক দেশ।': 0.7707803780121407, 'বিশ্বে বিশ্বের সবচেয়ে বেশি সংখ্যক দেশ জুড়ে।': 0.7632283651059119, 'বিশ্বের জনসংখ্যার মধ্যে অনেক বড় ব্যক্তি শহর।': 0.758930892179458, 'বিশ্বের সবচেয়ে বেশি সংখ্যক নাগরিক।': 0.7567314161796094, 'বিশ্বের সবচেয়ে বড় সংখ্যক দেশ জুড়ে রয়েছে ভারত।': 0.7288540805660473, 'বিশ্বের সবচেয়ে বড় দেশ ভারত জুড়ে অন্যতম দেশ।': 0.6965185583529492, 'দেশের সবচেয়ে বড় শহর সারা বিশ্বের সবচেয়ে বেশি।': 0.6729627220996024, 'বিশ্বের জনসংখ্যার মধ্যে রয়েছে ভারতের অন্যতম শহর।': 0.652937147520711, 'বিশ্বের জনসংখ্যার মধ্যে ভারত-মুসলিম দেশ জুড়ে রয়েছে ভারত।': 0.6417598337890371})\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model1 = Transformer(dropout=0.3)\n",
        "model1.load_weights(\"model_weights\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Pvp8V3WNRNeb",
        "outputId": "9e5c7879-27c0-4eb1-aceb-7a2235537adc"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.training.tracking.util.CheckpointLoadStatus at 0x7f0667e11750>"
            ]
          },
          "metadata": {},
          "execution_count": 72
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "sentence = \"I love you.\"\n",
        "translated_sentence = mbr_decode(sentence, 10, model1)\n",
        "print(\"English: \", sentence)\n",
        "print(\"Bengali: \", translated_sentence)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4JEyEFKZZU_i",
        "outputId": "deaaabe3-ee2d-4174-bfd9-0f48b43e654f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "English:  I love you.\n",
            "Bengali:  Counter({'আমি তোমাকে ভালোবাসি।': 0.9810885925136041, 'আমি তোমাকে ভালবাসি।': 0.9738396374710919, 'আমি তোমায় ভালবাসি।': 0.879383436230428, 'আমি সবাই ভালোবাসি।': 0.7971107263952748, 'আমার প্রিয় ভালো লাগছে।': 0.6125011949707815, 'আমি আমার প্রিয় বন্ধু।': 0.48543670036371755})\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "muuHMHqGZXnS"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}