Encoder_Generator_with_rationale.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Encoder/Generator with rationale.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "machine_shape": "hm",
      "authorship_tag": "ABX9TyPPT9A38Kige//nYcWPps0c",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mbmackenzie/peace-speech-project/blob/master/Encoder_Generator_with_rationale.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JTVQ0f0xhA6O"
      },
      "source": [
        "# About Notebook\r\n",
        "\r\n",
        "This notebook builds a model mentioned from the [paper](https://arxiv.org/abs/1606.04155). One can directly infer to the github repo mentioned in the paper for the code. However, I have used the following [repo](https://github.com/yala/text_nn) for the model. Most of the codes are from the repo, and I have transformed the functions in order to fit our data. Instead of putting the raw textfile for the input, I have changed it to accept the *pd.DataFrame* Format. One can manipulate the AbstractDataset class to change it according to one's data format.\r\n",
        "\r\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sAWMjGvihBoi"
      },
      "source": [
        "from google.colab import drive\r\n",
        "import sys\r\n",
        "drive.mount('/content/drive')\r\n",
        "\r\n",
        "#Reference\r\n",
        "!git clone https://github.com/yala/text_nn.git\r\n",
        "\r\n",
        "sys.path.append('/content/text_nn')\r\n",
        "sys.path.append('/content/text_nn/scripts')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FKMMAPjajL0z"
      },
      "source": [
        "#Libraries\r\n",
        "import os\r\n",
        "import pandas as pd\r\n",
        "import numpy as np\r\n",
        "from sklearn.model_selection import train_test_split\r\n",
        "\r\n",
        "from importlib import reload\r\n",
        "import sys\r\n",
        "from imp import reload\r\n",
        "import warnings\r\n",
        "\r\n",
        "\r\n",
        "warnings.filterwarnings('ignore')\r\n",
        "if sys.version[0] == '2':\r\n",
        "    reload(sys)\r\n",
        "    sys.setdefaultencoding(\"utf-8\")"
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jN6iKAxPjOCa",
        "outputId": "f506d89a-a559-439f-8d87-7418fe295879"
      },
      "source": [
        "import tensorflow as tf\r\n",
        "import torch\r\n",
        "\r\n",
        "# Get the GPU device name.\r\n",
        "device_name = tf.test.gpu_device_name()\r\n",
        "\r\n",
        "# The device name should look like the following:\r\n",
        "if device_name == '/device:GPU:0':\r\n",
        "    print('Found GPU at: {}'.format(device_name))\r\n",
        "else:\r\n",
        "    raise SystemError('GPU device not found')\r\n",
        "\r\n",
        "#To confirm that we are using GPU for the training later\r\n",
        "\r\n",
        "# If there's a GPU available...\r\n",
        "if torch.cuda.is_available():    \r\n",
        "\r\n",
        "    # Tell PyTorch to use the GPU.    \r\n",
        "    device = torch.device(\"cuda\")\r\n",
        "\r\n",
        "    print('There are %d GPU(s) available.' % torch.cuda.device_count())\r\n",
        "\r\n",
        "    print('We will use the GPU:', torch.cuda.get_device_name(0))\r\n",
        "\r\n",
        "# If not...\r\n",
        "else:\r\n",
        "    print('No GPU available, using the CPU instead.')\r\n",
        "    device = torch.device(\"cpu\")"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Found GPU at: /device:GPU:0\n",
            "There are 1 GPU(s) available.\n",
            "We will use the GPU: Tesla V100-SXM2-16GB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sXnAg6cIh_58"
      },
      "source": [
        "## 1. Import Data\r\n",
        "\r\n",
        "The dataset is n-gram processed articles with lemmatization and stop word removal. One can refer to each preprocess on the repo."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PIp0Y_s-jPms"
      },
      "source": [
        "peaceful_countries = ['GB','AU','CA','SG','NZ','IE']\r\n",
        "non_peaceful_countries = ['PK','BD','NG','KE','ZA','TZ']\r\n",
        "directory = '/content/drive/My Drive/capstone_data/data/5_domestic_filter_Ngram_stopwords_lemmatize'\r\n",
        "\r\n",
        "#data = pd.DataFrame([])\r\n",
        "\r\n",
        "data = []\r\n",
        "\r\n",
        "for entry in os.scandir(directory):\r\n",
        "  if (\"domestic_Ngram_stopword_lematize.csv\" in entry.path):\r\n",
        "    country = entry.name.split(\"_domestic_Ngram_stopword_lematize.csv\")[0]\r\n",
        "    # print(country)\r\n",
        "    if (country in peaceful_countries):\r\n",
        "      country_csv_path = entry.path\r\n",
        "      df = pd.read_csv(country_csv_path,index_col=[0])\r\n",
        "      df.rename(columns={'article_text_Ngram_stopword_lemmatize':'Processed_Reviews'}, inplace=True)\r\n",
        "      df['peaceful'] = 1\r\n",
        "      df = df[['Processed_Reviews','peaceful']]\r\n",
        "      data.append(df)\r\n",
        "      # print(data)\r\n",
        "      # print(df)\r\n",
        "    elif (country in non_peaceful_countries):\r\n",
        "      country_csv_path = entry.path\r\n",
        "      df = pd.read_csv(country_csv_path,index_col=[0])\r\n",
        "      df.rename(columns={'article_text_Ngram_stopword_lemmatize':'Processed_Reviews'}, inplace=True)\r\n",
        "      df['peaceful'] = 0\r\n",
        "      df = df[['Processed_Reviews','peaceful']]\r\n",
        "      data.append(df)\r\n",
        "      # print(df)\r\n",
        "      # print(data)\r\n",
        "    else:\r\n",
        "      continue"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "U-avEovujQyb",
        "outputId": "36387268-994e-44cc-ea7b-d733c3f2d5d5"
      },
      "source": [
        "df_full = pd.concat(data, axis=0, ignore_index=True)\r\n",
        "df_full.dropna(inplace=True)\r\n",
        "df_full.reset_index(drop=True, inplace=True)\r\n",
        "df_full.info()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 552329 entries, 0 to 552328\n",
            "Data columns (total 2 columns):\n",
            " #   Column             Non-Null Count   Dtype \n",
            "---  ------             --------------   ----- \n",
            " 0   Processed_Reviews  552329 non-null  object\n",
            " 1   peaceful           552329 non-null  int64 \n",
            "dtypes: int64(1), object(1)\n",
            "memory usage: 8.4+ MB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 417
        },
        "id": "4PChW-ccjVhR",
        "outputId": "ed33005f-7089-4ab1-9c51-7687c6359c80"
      },
      "source": [
        "#Sampling with seed\r\n",
        "\r\n",
        "import random\r\n",
        "\r\n",
        "random.seed(42)\r\n",
        "\r\n",
        "peace_index = random.sample(list(df_full[df_full['peaceful'] == 0].index), 100000)\r\n",
        "nonpeaceful_index = random.sample(list(df_full[df_full['peaceful'] == 1].index), 100000)\r\n",
        "index = peace_index + nonpeaceful_index\r\n",
        "\r\n",
        "sample_df = df_full.iloc[index, :]\r\n",
        "sample_df"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Processed_Reviews</th>\n",
              "      <th>peaceful</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>480560</th>\n",
              "      <td>Maverick Citizen Reflection Langa heal one sin...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>49380</th>\n",
              "      <td>Well country 's young TV station officially la...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26752</th>\n",
              "      <td>President Uhuru Kenyatta outline Kenya 's clim...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>507332</th>\n",
              "      <td>Road traffic collective consciousness In book ...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>92293</th>\n",
              "      <td>Alter Ego brings luxury lifestyle Nigeria Priv...</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>276321</th>\n",
              "      <td>Surfers pounce Prowlers roar back life Six luc...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>163731</th>\n",
              "      <td>Christchurch base Stoney Range Wines joint ven...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>288728</th>\n",
              "      <td>In song Red Rose Caf Fureys sing place poet sa...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>206317</th>\n",
              "      <td>Dr. Aw Knowing give terminal illness fight may...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>314624</th>\n",
              "      <td>Glennie School prays teenage burn victim Tara ...</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>200000 rows × 2 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                                        Processed_Reviews  peaceful\n",
              "480560  Maverick Citizen Reflection Langa heal one sin...         0\n",
              "49380   Well country 's young TV station officially la...         0\n",
              "26752   President Uhuru Kenyatta outline Kenya 's clim...         0\n",
              "507332  Road traffic collective consciousness In book ...         0\n",
              "92293   Alter Ego brings luxury lifestyle Nigeria Priv...         0\n",
              "...                                                   ...       ...\n",
              "276321  Surfers pounce Prowlers roar back life Six luc...         1\n",
              "163731  Christchurch base Stoney Range Wines joint ven...         1\n",
              "288728  In song Red Rose Caf Fureys sing place poet sa...         1\n",
              "206317  Dr. Aw Knowing give terminal illness fight may...         1\n",
              "314624  Glennie School prays teenage burn victim Tara ...         1\n",
              "\n",
              "[200000 rows x 2 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5nxGe_B-PDaW"
      },
      "source": [
        "sample_df.index = np.arange(0, len(sample_df))\r\n",
        "\r\n",
        "def remove_digit(s):\r\n",
        "  return ''.join([i for i in s if not i.isdigit()])\r\n",
        "\r\n",
        "for i in range(len(sample_df)):\r\n",
        "  sample_df['Processed_Reviews'][i] = remove_digit(sample_df['Processed_Reviews'][i])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QqJ18mhZiO6b"
      },
      "source": [
        "## 2. Further Prerpocess\r\n",
        "\r\n",
        "Since the result from the above dataset was not satisfactory enough, we have removed the NORP, PERSON, GPE annotated words from the articles, and digits also."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_Z_fWEQXev0G"
      },
      "source": [
        "!python -m spacy download en_core_web_md\r\n",
        "\r\n",
        "import spacy\r\n",
        "from spacy import displacy\r\n",
        "from collections import Counter\r\n",
        "import en_core_web_md\r\n",
        "nlp = en_core_web_md.load()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iYrXEIUj7tdz"
      },
      "source": [
        "def apo_s_remove(s):\r\n",
        "  s = re.sub(r'\\'s', '', s)\r\n",
        "  return s\r\n",
        "\r\n",
        "for i in range(len(sample_df)):\r\n",
        "  sample_df['Processed_Reviews'][i] = apo_s_remove(sample_df['Processed_Reviews'][i])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "epp1Ri3ziv1z"
      },
      "source": [
        "def exclude_entity(s):\r\n",
        "  doc = nlp(s)\r\n",
        "  entity_list = [(X.text, X.label_) for X in doc.ents if (X.label_ == 'GPE') or X.label_ == 'PERSON' or (X.label_ == 'NORP')]\r\n",
        "  entity_list = set(entity_list)\r\n",
        "  for entity in entity_list:\r\n",
        "    s = re.sub(entity[0], '', s)\r\n",
        "  return s"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kildcw_KjFoW",
        "outputId": "3fb09037-0292-4c8f-b1f7-74c6174623ba"
      },
      "source": [
        "%time\r\n",
        "import re\r\n",
        "for i in range(len(sample_df)):\r\n",
        "  s = sample_df['Processed_Reviews'][i]\r\n",
        "  if len(s) > 1000000:\r\n",
        "    s = s[:1000000-1]\r\n",
        "  doc = nlp(s)\r\n",
        "  entity_list = [(X.text, X.label_) for X in doc.ents if (X.label_ == 'GPE') or X.label_ == 'PERSON' or (X.label_ == 'NORP')]\r\n",
        "  entity_list = set(entity_list)\r\n",
        "  for entity in entity_list:\r\n",
        "    s = s.replace(entity[0], '')\r\n",
        "  sample_df['Processed_Reviews'][i] = s"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "CPU times: user 3 µs, sys: 0 ns, total: 3 µs\n",
            "Wall time: 5.96 µs\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3ewWQdmJ6d8D"
      },
      "source": [
        "sample_df.to_csv('/content/drive/MyDrive/2020 Capstone/processed_sample_df.csv')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P_pGrKlNjWoE"
      },
      "source": [
        "train_sample, test_dev_sample = train_test_split(sample_df.index, test_size = 0.3, random_state = 42)\r\n",
        "dev_sample, test_sample = train_test_split(test_dev_sample, test_size = 0.5, random_state = 42)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UMVqV-wzjYCn"
      },
      "source": [
        "sample_df['set'] = ''\r\n",
        "sample_df.loc[train_sample, 'set'] = 'train'\r\n",
        "sample_df.loc[dev_sample, 'set'] ='dev'\r\n",
        "sample_df.loc[test_sample,'set'] = 'test'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t-hJLDesw3Rf"
      },
      "source": [
        "## 3. Embedding\r\n",
        "\r\n",
        "In later section, we have to use word embedding for the representation of the word. We have created word index linked to the 300 dimensional word vectors from the glove.6B.300d. One needs to download the file from the following [link](https://nlp.stanford.edu/projects/glove/)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eXfIPPNxjZkr"
      },
      "source": [
        "\r\n",
        "# Load the Embeddings\r\n",
        "import numpy as np\r\n",
        "\r\n",
        "# Set the path where you have downloaded the embeddings\r\n",
        "emb_path = \"/content/drive/MyDrive/glove_word_vec/glove.6B.300d.txt\"\r\n",
        "\r\n",
        "# Set the embedding size\r\n",
        "emb_dims = 300\r\n",
        "\r\n",
        "\r\n",
        "def load_embeddings(emb_path, emb_dims):\r\n",
        "    '''\r\n",
        "    Load the embeddings from a text file\r\n",
        "    \r\n",
        "        :param emb_path: Path of the text file\r\n",
        "        :param emb_dims: Embedding dimensions\r\n",
        "        \r\n",
        "        :return emb_tensor: tensor containing all word embeedings\r\n",
        "        :return word_to_indx: dictionary with word:index\r\n",
        "    '''\r\n",
        "\r\n",
        "    # Load the file\r\n",
        "    lines = open(emb_path).readlines()\r\n",
        "    \r\n",
        "    # Creating the list and adding the PADDING embedding\r\n",
        "    emb_tensor = [np.zeros(emb_dims)]\r\n",
        "    word_to_indx = {'PADDING_WORD':0}\r\n",
        "    \r\n",
        "    # For each line, save the embedding and the word:index\r\n",
        "    for indx, l in enumerate(lines):\r\n",
        "        word, emb = l.split()[0], l.split()[1:]\r\n",
        "        \r\n",
        "        if not len(emb) == emb_dims:\r\n",
        "            continue\r\n",
        "        \r\n",
        "        # Update the embedding list and the word:index dictionary\r\n",
        "        emb_tensor.append([float(x) for x in emb])\r\n",
        "        word_to_indx[word] = indx+1\r\n",
        "    \r\n",
        "    # Turning the list into a numpy object\r\n",
        "    emb_tensor = np.array(emb_tensor, dtype=np.float32)\r\n",
        "    return emb_tensor, word_to_indx"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-9Dey3ArjbQn",
        "outputId": "f153d53f-23a3-47d5-fdbe-f4e2affb26ea"
      },
      "source": [
        "# Calling load_embeddings and printing the size of the returned objects\r\n",
        "emb_tensor, word_to_indx = load_embeddings(emb_path, emb_dims)\r\n",
        "\r\n",
        "print('Words: {}\\nVectors (+ zero-padding): {}'.format(len(word_to_indx.keys()), emb_tensor.shape))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Words: 400001\n",
            "Vectors (+ zero-padding): (400001, 300)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ogMaYRzyi7rH"
      },
      "source": [
        "## 4. Convert dataset\r\n",
        "\r\n",
        "We need to convert our dataset that is suitable for the models later. One can change the AbstractDataset Class for one's own dataset. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iainbULNjcih"
      },
      "source": [
        "from abc import ABCMeta, abstractmethod, abstractproperty\r\n",
        "import torch.utils.data as data\r\n",
        "import torch\r\n",
        "\r\n",
        "import re\r\n",
        "import random\r\n",
        "import tqdm\r\n",
        "\r\n",
        "\r\n",
        "# Classes in the dataset\r\n",
        "classes = {1:'peaceful',\r\n",
        "           0:'non-peaceful'}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "msLL0Xjolt8d"
      },
      "source": [
        "def preprocess(data):\r\n",
        "        '''\r\n",
        "        Return a list of (text, label and label_name)\r\n",
        "\r\n",
        "            :param data: 20 newsgroup dataset as imported by SK-Learn\r\n",
        "            \r\n",
        "            :return processed_data: list of text, label and label_name\r\n",
        "        '''\r\n",
        "        processed_data = []\r\n",
        "        for indx, sample in enumerate(data['data']):\r\n",
        "            text, label = sample, data['target'][indx]\r\n",
        "            label_name = data['target_names'][label]\r\n",
        "            text = re.sub('\\W+', ' ', text).lower().strip()\r\n",
        "            processed_data.append((text, label, label_name))\r\n",
        "        return processed_data"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xpX_0wrElu1T"
      },
      "source": [
        "# Load the Dataset\r\n",
        "\r\n",
        "from sklearn.datasets import fetch_20newsgroups\r\n",
        "from abc import ABCMeta, abstractmethod, abstractproperty\r\n",
        "import torch.utils.data as data\r\n",
        "import torch\r\n",
        "\r\n",
        "import re\r\n",
        "import random\r\n",
        "import tqdm\r\n",
        "\r\n",
        "\r\n",
        "# Classes in the dataset\r\n",
        "classes = {0:'peaceful',\r\n",
        "           1:'non-peaceful'}\r\n",
        "\r\n",
        "class AbstractDataset(data.Dataset):\r\n",
        "    '''\r\n",
        "    Abstract class that adds general method to the Newsgroup dataset\r\n",
        "    '''\r\n",
        "    \r\n",
        "    __metaclass__ = ABCMeta\r\n",
        "\r\n",
        "    def __len__(self):\r\n",
        "        return len(self.dataset)\r\n",
        "\r\n",
        "    def __getitem__(self, index):\r\n",
        "        sample = self.dataset[index]\r\n",
        "        return sample\r\n",
        "\r\n",
        "\r\n",
        "class Article(AbstractDataset):\r\n",
        "    '''\r\n",
        "    Newsgroup dataset loader\r\n",
        "    '''\r\n",
        "    \r\n",
        "    def __init__(self,dataframe, set_type, classes, word_to_indx, max_length=80):\r\n",
        "        '''\r\n",
        "        Load the dataset from SK-Learn\r\n",
        "\r\n",
        "            :param set_type: string containing either 'train', 'dev' or 'test'\r\n",
        "            :param classes: list of strings containing the classes\r\n",
        "            :param word_to_indx: dictionary of word:index\r\n",
        "            :param max_length: integer with max word to consider\r\n",
        "            :return: nothing\r\n",
        "        '''\r\n",
        "\r\n",
        "        # Deterministic randomization\r\n",
        "        random.seed(0)\r\n",
        "        \r\n",
        "        n_classes = len(classes)\r\n",
        "        class_balance = {}\r\n",
        "        self.dataset = []\r\n",
        "\r\n",
        "        # If train or dev...\r\n",
        "        if set_type == 'train':\r\n",
        "          df = dataframe[dataframe['set'] == 'train']\r\n",
        "        elif set_type == 'dev':\r\n",
        "          df = dataframe[dataframe['set'] == 'dev']\r\n",
        "        else:\r\n",
        "          df = dataframe[dataframe['set'] == 'test']\r\n",
        "\r\n",
        "        df.index = np.arange(0,len(df))\r\n",
        "        data = [(df['Processed_Reviews'][i], df['peaceful'][i], classes[df['peaceful'][i]]) for i in range(len(df))]\r\n",
        "       \r\n",
        "        # For every unprocessed_sample in the created set, process it\r\n",
        "        for indx, unprocessed_sample in tqdm.tqdm(enumerate(data)):\r\n",
        "            sample = self.process_line(unprocessed_sample, word_to_indx, max_length)\r\n",
        "            \r\n",
        "            # If the sample is not empty, save it and add its y to the class_balance dictionary\r\n",
        "            if sample['text'] != '':\r\n",
        "                if not sample['y'] in class_balance:\r\n",
        "                    class_balance[sample['y']] = 0\r\n",
        "                class_balance[sample['y']] += 1\r\n",
        "                self.dataset.append(sample)\r\n",
        "\r\n",
        "            \r\n",
        "   \r\n",
        "\r\n",
        "    \r\n",
        "    def get_indices_tensor(self, text_arr, word_to_indx, max_length):\r\n",
        "        '''\r\n",
        "        Return a tensor of max_length with the word indices\r\n",
        "        \r\n",
        "            :param text_arr: text array\r\n",
        "            :param word_to_indx: dictionary word:index\r\n",
        "            :param max_length: maximum length of returned tensors\r\n",
        "            \r\n",
        "            :return x: tensor containing the indices\r\n",
        "        '''\r\n",
        "        \r\n",
        "        pad_indx = 0\r\n",
        "        text_indx = [word_to_indx[x] if x in word_to_indx else pad_indx for x in text_arr][:max_length]\r\n",
        "        \r\n",
        "        # Padding\r\n",
        "        if len(text_indx) < max_length:\r\n",
        "            text_indx.extend([pad_indx for _ in range(max_length - len(text_indx))])\r\n",
        "\r\n",
        "        x =  torch.LongTensor([text_indx])\r\n",
        "\r\n",
        "        return x\r\n",
        "\r\n",
        "\r\n",
        "    def process_line(self, row, word_to_indx, max_length, case_insensitive=True):\r\n",
        "        '''\r\n",
        "        Return every line as a dictionary with text, x, y, y_name\r\n",
        "\r\n",
        "            :param row: document (or comment)\r\n",
        "            :param word_to_indx: dictionary of word:index\r\n",
        "            :param max_length: integer with max word to consider\r\n",
        "            \r\n",
        "            :return sample: dictionary of text, x, y, y_name\r\n",
        "        '''\r\n",
        "        \r\n",
        "        text, label, label_name = row\r\n",
        "        \r\n",
        "        if case_insensitive:\r\n",
        "            text = \" \".join(text.split()[:max_length]).lower()\r\n",
        "        else:\r\n",
        "            text = \" \".join(text.split()[:max_length])\r\n",
        "            \r\n",
        "        x =  self.get_indices_tensor(text.split(), word_to_indx, max_length)\r\n",
        "        \r\n",
        "        sample = {'text':text,'x':x, 'y':label, 'y_name': label_name}\r\n",
        "        return sample"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Uos2Q4mVlxQW",
        "outputId": "17b66c3d-afcc-4ca6-ef98-71c0b64a9831"
      },
      "source": [
        "# Loading the dataset\r\n",
        "train_data = Article(sample_df,'train', classes, word_to_indx,  max_length=512)\r\n",
        "dev_data = Article(sample_df,'dev', classes, word_to_indx, max_length=512)\r\n",
        "test_data = Article(sample_df,'test', classes, word_to_indx,  max_length=512)\r\n",
        "\r\n",
        "# Printing 3 datapoints\r\n",
        "for datapoint in train_data[:1]:\r\n",
        "    print(datapoint)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "140000it [00:19, 7134.65it/s]\n",
            "30000it [00:04, 6904.69it/s]\n",
            "30000it [00:04, 7275.42it/s]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "{'text': 'well country young tv station officially launch yesterday glorify reception excitement yet wear the move rms game changer especially come time nothing fresh go tv front standard group hr news channel ktn news while viewer rms doubt celebrate party go quite butt hurt either ignore move utilize opportunity victim change indirect competitor so five main loser game nation media standard group mediamax basically however smart unruffled big medium house may try appear know almost everyone know lose big time with digital migration guy view content big negotiation ability advertiser company broadcasting they already trail citizen tv term viewership get another news channel struggle make unique qtv quite ratchet tv station compete like gbs rather local content offer citizen so long rely butter form citizen tv rms equity bank best target million low market segment with inooro tv vernacular station come rms quite stranglehold viewership country the exist vernacular station with trumpet chest beating folk rms would think first vernacular tv station country you would even think pioneer new trend on contrary vernacular station njata tv kass tv baite tv several others reason may heard poor promotional effort maybe comfortable now advent muscle rms party when every digital tv carrier get late thing happen industry left stand wrong reason given startimes premium content lose content target demand common guy pretty much disaster just would anyone get decoder dj afro company for sometime guy source much entertainment many street low class home the narration do mix kikuyu rivet action screen now watch plan program soap thriller realize lose quite chunk target audience movies i hardly fan i would rather room professionally do program local language i clue rather engross movie get serious competitor albeit indirectly i really hope barber local electronics mechanic co. inform make necessary change', 'x': tensor([[   144,    124,    462,    817,    516,   2392,   1784,   2858,  47459,\n",
            "           3703,   9031,    554,   3434,      1,    484,  34451,    187,  39448,\n",
            "            859,    327,     80,    937,   1904,    243,    817,    670,   1225,\n",
            "            130,  18828,    173,   1630, 140740,    173,    111,  12515,  34451,\n",
            "           2566,   4687,    166,    243,   1690,  11011,   2089,    901,   7150,\n",
            "            484,  16674,   1540,   3123,    512,  10432,   7687,    101,    175,\n",
            "            445,  10459,    187,    514,    494,   1225,    130, 311820,   5204,\n",
            "            213,   4435,  67414,    366,   3186,    167,    108,    842,   1651,\n",
            "            347,    592,   1403,    347,   1896,    366,     80,     18,   2151,\n",
            "           7292,   1857,   1140,   2769,    366,   8636,   1658,  25414,    129,\n",
            "           3281,     40,    412,   3517,   3942,    817,    571,  25899,    170,\n",
            "            171,    173,   1630,   2652,    160,   3007, 224896,   1690,  32527,\n",
            "            817,    516,   2798,    118, 102242,    872,    251,   2769,    902,\n",
            "           3942,    101,    174,   5657,   6459,    684,   3942,    817,  34451,\n",
            "           3360,    232,    255,   1489,     94,    654,    212,   6119,     18,\n",
            "              0,    817,  21557,    516,    327,  34451,   1690,  30070,  25899,\n",
            "            124,      1,   3605,  21557,    516,     18,  15745,   5779,   2840,\n",
            "           5008,  34451,     55,    270,     59,  21557,    817,    516,    124,\n",
            "             82,     55,    152,    270,   6415,     51,   3288,     14,   6606,\n",
            "          21557,    516,      0,    817,  51139,    817,      0,    817,    202,\n",
            "            424,   1248,    108,   1436,    993,   9176,    969,   1882,   4032,\n",
            "            115,  13203,   5637,  34451,    166,     62,    360,   2151,    817,\n",
            "           3484,    170,    289,    874,   1928,    460,    219,   1347,   1798,\n",
            "           1248,    455,      0,   6067,   2769,   1896,   2769,   1489,    957,\n",
            "            862,   1857,   1923,    182,   2396,    121,     55,   1545,    170,\n",
            "          49713,   7621,  24847,    129,     11,   7323,   1857,   1174,    182,\n",
            "           2046,    110,    492,    654,    906,    164,      1,  22532,     89,\n",
            "           3635,  37336,  65341,    609,   2492,    115,   1717,    395,    372,\n",
            "           7319,   8966,   4224,   1896,   1690,  13402,   1489,   2053,   2460,\n",
            "             42,   4593,   3268,     42,     55,    872,    928,  12147,     89,\n",
            "            372,    251,   1033,     42,  13339,    872,      0,   1006,    170,\n",
            "           1035,   7687,   9720,  12067,     42,    589,    825,  11044,    251,\n",
            "           3720,  14278,   1169,   8988,    160,   1505,    512,      0,      0,\n",
            "              0,      0,      0,      0,      0,      0,      0,      0,      0,\n",
            "              0,      0,      0,      0,      0,      0,      0,      0,      0,\n",
            "              0,      0,      0,      0,      0,      0,      0,      0,      0,\n",
            "              0,      0,      0,      0,      0,      0,      0,      0,      0,\n",
            "              0,      0,      0,      0,      0,      0,      0,      0,      0,\n",
            "              0,      0,      0,      0,      0,      0,      0,      0,      0,\n",
            "              0,      0,      0,      0,      0,      0,      0,      0,      0,\n",
            "              0,      0,      0,      0,      0,      0,      0,      0,      0,\n",
            "              0,      0,      0,      0,      0,      0,      0,      0,      0,\n",
            "              0,      0,      0,      0,      0,      0,      0,      0,      0,\n",
            "              0,      0,      0,      0,      0,      0,      0,      0,      0,\n",
            "              0,      0,      0,      0,      0,      0,      0,      0,      0,\n",
            "              0,      0,      0,      0,      0,      0,      0,      0,      0,\n",
            "              0,      0,      0,      0,      0,      0,      0,      0,      0,\n",
            "              0,      0,      0,      0,      0,      0,      0,      0,      0,\n",
            "              0,      0,      0,      0,      0,      0,      0,      0,      0,\n",
            "              0,      0,      0,      0,      0,      0,      0,      0,      0,\n",
            "              0,      0,      0,      0,      0,      0,      0,      0,      0,\n",
            "              0,      0,      0,      0,      0,      0,      0,      0,      0,\n",
            "              0,      0,      0,      0,      0,      0,      0,      0,      0,\n",
            "              0,      0,      0,      0,      0,      0,      0,      0,      0,\n",
            "              0,      0,      0,      0,      0,      0,      0,      0,      0,\n",
            "              0,      0,      0,      0,      0,      0,      0,      0,      0,\n",
            "              0,      0,      0,      0,      0,      0,      0,      0]]), 'y': 0, 'y_name': 'peaceful'}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LSPO-XDPl1rY"
      },
      "source": [
        "## 5. Model Parameters\r\n",
        "\r\n",
        "The following model parameters is for the models (encoder, decoder, tagger). One can change the following into argparser for the python file."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mO_Mb7Iz1CTq"
      },
      "source": [
        "args = {'train':True, 'test':True, 'cuda':True, 'class_balance':False, 'num_gpus':1, 'debug_mode':False,\r\n",
        "        'model_form':'cnn', 'dropout': 0.1,\r\n",
        "        'init_lr':0.001, 'epochs':2, 'batch_size':512, 'patience':10,\r\n",
        "        'save_dir':'snapshot', 'model_path':'/content/model.pt', 'results_path':' /content', 'model':'TextCNN',\r\n",
        "        'hidden_dims':100, 'num_layers':1, 'dropout':0.1, 'weight_decay':1e-3,\r\n",
        "        'filter_num':100, 'filters':[3, 4, 5], 'num_class':2, 'emb_dims':300,\r\n",
        "        'tuning_metric':'loss', 'num_workers':4, 'objective':'cross_entropy',\r\n",
        "        'use_as_tagger':False, 'get_rationales': True,\r\n",
        "        'selection_lambda': 0.01,'continuity_lambda':0.01, 'snapshot':None,\r\n",
        "        'gumbel_temprature':1, 'gumbel_decay': 1e-5\r\n",
        "        }"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FEs64Nnjl_bB"
      },
      "source": [
        "## 6. Encoder\r\n",
        "\r\n",
        "The following part is to implement the Encoder mentioned from the paper. In this notebook, we have used CNN from Yoon Kim, but one can implement different methods. The purpose of encoder is to classify the article with a set of Bernoulli variable masks."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rcZ-rOIrl4bZ"
      },
      "source": [
        "import pdb\r\n",
        "import torch\r\n",
        "import torch.nn as nn\r\n",
        "import torch.autograd as autograd\r\n",
        "import torch.nn.functional as F\r\n",
        "\r\n",
        "\r\n",
        "# Encoder\r\n",
        "class Encoder(nn.Module):\r\n",
        "    '''\r\n",
        "    Load the embeddings and encode them\r\n",
        "    '''\r\n",
        "\r\n",
        "    def __init__(self, embeddings, args):\r\n",
        "        '''\r\n",
        "        Load embeddings and call the TextCNN model\r\n",
        "        \r\n",
        "            :param embeddings: tensor with word embeddings\r\n",
        "            :param model: default is 'TextCNN'\r\n",
        "            \r\n",
        "            :return: nothing\r\n",
        "        '''\r\n",
        "        super(Encoder, self).__init__()\r\n",
        "        \r\n",
        "        # Saving the parameters\r\n",
        "        self.model = args['model']\r\n",
        "        self.num_class = args['num_class']\r\n",
        "        self.hidden_dims = args['hidden_dims']\r\n",
        "        self.num_layers = args['num_layers']\r\n",
        "        self.filters = args['filters']\r\n",
        "        self.filter_num = args['filter_num']\r\n",
        "        self.cuda = args['cuda']\r\n",
        "        self.dropout = args['dropout']\r\n",
        "        \r\n",
        "        # Loading the word embeddings in the Neural Network\r\n",
        "        vocab_size, hidden_dim = embeddings.shape\r\n",
        "        self.emb_dims = hidden_dim\r\n",
        "        self.emb_layer = nn.Embedding(vocab_size, hidden_dim)\r\n",
        "        self.emb_layer.weight.data = torch.from_numpy(embeddings)\r\n",
        "        self.emb_layer.weight.requires_grad = True\r\n",
        "        self.emb_fc = nn.Linear(hidden_dim, hidden_dim)\r\n",
        "        self.emb_bn = nn.BatchNorm1d(hidden_dim)\r\n",
        "        \r\n",
        "        # Calling the model, followed by a fully connected hidden layer\r\n",
        "        if self.model == 'TextCNN':\r\n",
        "            self.cnn = TextCNN(args, max_pool_over_time=True)\r\n",
        "            # The hidden fully connected layer size is given by the number of filters\r\n",
        "            # times the filter size, by the number of hidden dimensions\r\n",
        "            self.fc = nn.Linear(len(self.filters) * self.filter_num, hidden_dim)\r\n",
        "        else:\r\n",
        "            raise NotImplementedError(\"Model {} not yet supported for encoder!\".format(model))\r\n",
        "\r\n",
        "        # Dropout and final layer\r\n",
        "        self.dropout = nn.Dropout(self.dropout)\r\n",
        "        self.hidden = nn.Linear(hidden_dim, self.num_class)\r\n",
        "        \r\n",
        "        \r\n",
        "    def forward(self, x_indx, mask = None):\r\n",
        "        '''\r\n",
        "        Forward step\r\n",
        "        \r\n",
        "            :param x_indx: batch of word indices\r\n",
        "            \r\n",
        "            :return logit: predictions\r\n",
        "            :return: hidden layer\r\n",
        "        '''\r\n",
        "        \r\n",
        "        x = self.emb_layer(x_indx.squeeze(1))\r\n",
        "        if self.cuda:\r\n",
        "            x = x.to(device)\r\n",
        "        if not mask is None:\r\n",
        "            x = x * mask.unsqueeze(-1)\r\n",
        "\r\n",
        "        # Non linear projection with dropout\r\n",
        "        x = F.relu(self.emb_fc(x))\r\n",
        "        x = self.dropout(x)\r\n",
        "        # TextNN, fully connected and non linearity\r\n",
        "        if self.model == 'TextCNN':\r\n",
        "            x = torch.transpose(x, 1, 2) # Transpose x dimensions into (Batch, Emb, Length)\r\n",
        "            hidden = self.cnn(x)\r\n",
        "            hidden = F.relu(self.fc(hidden))\r\n",
        "        else:\r\n",
        "            raise Exception(\"Model {} not yet supported for encoder!\".format(self.model))\r\n",
        "\r\n",
        "        # Dropout and final layer\r\n",
        "        hidden = self.dropout(hidden)\r\n",
        "        logit = self.hidden(hidden)\r\n",
        "        return logit, hidden\r\n",
        "\r\n",
        "\r\n",
        "# Model\r\n",
        "class TextCNN(nn.Module):\r\n",
        "    '''\r\n",
        "    CNN for Text Classification\r\n",
        "    '''\r\n",
        "\r\n",
        "    def __init__(self, args, max_pool_over_time=False):\r\n",
        "        '''\r\n",
        "        Convolutional Neural Network\r\n",
        "        \r\n",
        "            :param num_layers: number of layers\r\n",
        "            :param filters: filters shape\r\n",
        "            :param filter_num: number of filters\r\n",
        "            :param emb_dims: embedding dimensions\r\n",
        "            :param max_pool_over_time: boolean\r\n",
        "            \r\n",
        "            :return: nothing\r\n",
        "        '''\r\n",
        "        super(TextCNN, self).__init__()\r\n",
        "\r\n",
        "        # Saving the parameters\r\n",
        "        self.num_layers = args['num_layers']\r\n",
        "        self.filters = args['filters']\r\n",
        "        self.filter_num = args['filter_num']\r\n",
        "        self.emb_dims = args['emb_dims']\r\n",
        "        self.cuda = args['cuda']\r\n",
        "        self.max_pool = max_pool_over_time\r\n",
        "        \r\n",
        "        self.layers = []\r\n",
        "        \r\n",
        "        # For every layer...\r\n",
        "        for l in range(self.num_layers):\r\n",
        "            convs = []\r\n",
        "            \r\n",
        "            # For every filter...\r\n",
        "            for f in self.filters:\r\n",
        "                # Defining the sizes\r\n",
        "                in_channels =  self.emb_dims if l == 0 else self.filter_num * len(self.filters)\r\n",
        "                kernel_size = f\r\n",
        "                \r\n",
        "                # Adding the convolutions in the list\r\n",
        "                conv = nn.Conv1d(in_channels=in_channels, out_channels=self.filter_num, kernel_size=kernel_size)\r\n",
        "                self.add_module('layer_' + str(l) + '_conv_' + str(f), conv)\r\n",
        "                convs.append(conv)\r\n",
        "                \r\n",
        "            self.layers.append(convs)\r\n",
        "\r\n",
        "\r\n",
        "    def _conv(self, x):\r\n",
        "        '''\r\n",
        "        Left padding and returning the activation\r\n",
        "        \r\n",
        "            :param x: input tensor (batch, emb, length)\r\n",
        "            :return layer_activ: activation\r\n",
        "        '''\r\n",
        "        \r\n",
        "        layer_activ = x\r\n",
        "        \r\n",
        "        for layer in self.layers:\r\n",
        "            next_activ = []\r\n",
        "            \r\n",
        "            for conv in layer:\r\n",
        "                # Setting the padding dimensions: it is like adding\r\n",
        "                # kernel_size - 1 empty embeddings\r\n",
        "                left_pad = conv.kernel_size[0] - 1\r\n",
        "                pad_tensor_size = [d for d in layer_activ.size()]\r\n",
        "                pad_tensor_size[2] = left_pad\r\n",
        "                left_pad_tensor = autograd.Variable(torch.zeros(pad_tensor_size))\r\n",
        "                \r\n",
        "                if self.cuda:\r\n",
        "                    left_pad_tensor = left_pad_tensor.to(device)\r\n",
        "                    \r\n",
        "                # Concatenating the padding to the tensor\r\n",
        "                padded_activ = torch.cat((left_pad_tensor, layer_activ), dim=2)\r\n",
        "                \r\n",
        "                # onvolution activation\r\n",
        "                next_activ.append(conv(padded_activ))\r\n",
        "\r\n",
        "            # Concatenating accross channels\r\n",
        "            layer_activ = F.relu(torch.cat(next_activ, 1))\r\n",
        "            #pdb.set_trace()\r\n",
        "        return layer_activ\r\n",
        "\r\n",
        "\r\n",
        "    def _pool(self, relu):\r\n",
        "        '''\r\n",
        "        Max Pool Over Time\r\n",
        "        '''\r\n",
        "        \r\n",
        "        pool = F.max_pool1d(relu, relu.size(2)).squeeze(-1)\r\n",
        "        return pool\r\n",
        "\r\n",
        "\r\n",
        "    def forward(self, x):\r\n",
        "        '''\r\n",
        "        Forward steps over the x\r\n",
        "        \r\n",
        "            :param x: input (batch, emb, length)\r\n",
        "\r\n",
        "            :return activ: activation\r\n",
        "        '''\r\n",
        "        \r\n",
        "        activ = self._conv(x)\r\n",
        "        \r\n",
        "        # Pooling over time?\r\n",
        "        if self.max_pool:\r\n",
        "            activ = self._pool(activ)\r\n",
        "            \r\n",
        "        return activ"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0ZdTJj5umHWd"
      },
      "source": [
        "## 7. Generator\r\n",
        "\r\n",
        "This part is to generate the rationale mentioned from the paper. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BPpf9UrJmAnf"
      },
      "source": [
        "import torch\r\n",
        "import torch.nn as nn\r\n",
        "import torch.autograd as autograd\r\n",
        "import torch.nn.functional as F\r\n",
        "import rationale_net.models.cnn as cnn\r\n",
        "import rationale_net.utils.learn as learn\r\n",
        "import pdb\r\n",
        "\r\n",
        "'''\r\n",
        "    The generator selects a rationale z from a document x that should be sufficient\r\n",
        "    for the encoder to make it's prediction.\r\n",
        "    Several froms of Generator are supported. Namely CNN with arbitary number of layers, and @taolei's FastKNN\r\n",
        "'''\r\n",
        "\r\n",
        "\r\n",
        "\r\n",
        "class Generator(nn.Module):\r\n",
        "\r\n",
        "    def __init__(self, embeddings, args):\r\n",
        "        super(Generator, self).__init__()\r\n",
        "        vocab_size, hidden_dim = embeddings.shape\r\n",
        "        self.embedding_layer = nn.Embedding( vocab_size, hidden_dim)\r\n",
        "        self.embedding_layer.weight.data = torch.from_numpy( embeddings )\r\n",
        "        self.embedding_layer.weight.requires_grad = False\r\n",
        "\r\n",
        "        self.model = args['model']\r\n",
        "        self.filters = args['filters']\r\n",
        "        self.filter_num = args['filter_num']\r\n",
        "        self.dropout = args['dropout']\r\n",
        "        self.cuda = args['cuda']\r\n",
        "        self.gumbel_temprature = args['gumbel_temprature']\r\n",
        "        self.gumbel_decay = args['gumbel_decay']\r\n",
        "\r\n",
        "        if self.model == 'TextCNN':\r\n",
        "            self.cnn = TextCNN(args, max_pool_over_time=False)\r\n",
        "        else:\r\n",
        "            raise NotImplementedError(\"Model {} not yet supported for encoder!\".format(model))\r\n",
        "\r\n",
        "        self.z_dim = 2\r\n",
        "\r\n",
        "        self.hidden = nn.Linear((len(self.filters)* self.filter_num), self.z_dim)\r\n",
        "        self.dropout = nn.Dropout(self.dropout)\r\n",
        "\r\n",
        "\r\n",
        "\r\n",
        "    def  __z_forward(self, activ):\r\n",
        "        '''\r\n",
        "            Returns prob of each token being selected\r\n",
        "        '''\r\n",
        "        activ = activ.transpose(1,2)\r\n",
        "        logits = self.hidden(activ)\r\n",
        "        probs = gumbel_softmax(logits, self.gumbel_temprature, self.cuda)\r\n",
        "        z = probs[:,:,1]\r\n",
        "        return z\r\n",
        "\r\n",
        "\r\n",
        "    def forward(self, x_indx):\r\n",
        "        '''\r\n",
        "            Given input x_indx of dim (batch, length), return z (batch, length) such that z\r\n",
        "            can act as element-wise mask on x\r\n",
        "        '''\r\n",
        "        if self.model== 'TextCNN':\r\n",
        "            x = self.embedding_layer(x_indx.squeeze(1))\r\n",
        "            if self.cuda:\r\n",
        "                x = x.to(device)\r\n",
        "            \r\n",
        "            x = torch.transpose(x, 1, 2) # Switch X to (Batch, Embed, Length)\r\n",
        "            activ = self.cnn(x)\r\n",
        "            if self.cuda:\r\n",
        "              activ = activ.to(device)\r\n",
        "           \r\n",
        "        else:\r\n",
        "            raise NotImplementedError(\"Model form {} not yet supported for generator!\".format(self.model))\r\n",
        "\r\n",
        "        z = self.__z_forward(F.relu(activ))\r\n",
        "        if self.cuda:\r\n",
        "          z= z.to(device)\r\n",
        "        mask = self.sample(z)\r\n",
        "        if self.cuda:\r\n",
        "          mask = mask.to(device)\r\n",
        "        return mask, z\r\n",
        "\r\n",
        "\r\n",
        "    def sample(self, z):\r\n",
        "        '''\r\n",
        "            Get mask from probablites at each token. Use gumbel\r\n",
        "            softmax at train time, hard mask at test time\r\n",
        "        '''\r\n",
        "        mask = z\r\n",
        "        if self.training:\r\n",
        "            mask = z\r\n",
        "        else:\r\n",
        "            ## pointwise set <.5 to 0 >=.5 to 1\r\n",
        "            mask = get_hard_mask(z)\r\n",
        "        return mask\r\n",
        "\r\n",
        "\r\n",
        "    def loss(self, mask, x_indx):\r\n",
        "        '''\r\n",
        "            Compute the generator specific costs, i.e selection cost, continuity cost, and global vocab cost\r\n",
        "        '''\r\n",
        "        selection_cost = torch.mean( torch.sum(mask, dim=1) )\r\n",
        "        l_padded_mask =  torch.cat( [mask[:,0].unsqueeze(1), mask] , dim=1)\r\n",
        "        r_padded_mask =  torch.cat( [mask, mask[:,-1].unsqueeze(1)] , dim=1)\r\n",
        "        continuity_cost = torch.mean( torch.sum( torch.abs( l_padded_mask - r_padded_mask ) , dim=1) )\r\n",
        "        return selection_cost, continuity_cost"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UL9HKrm3j-eo"
      },
      "source": [
        "## 8. Empty module"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ubHf7pugmIT2"
      },
      "source": [
        "import torch\r\n",
        "import torch.nn as nn\r\n",
        "import pdb\r\n",
        "\r\n",
        "class Empty(torch.nn.Module):\r\n",
        "    def __init__(self):\r\n",
        "        super(Empty, self).__init__()\r\n",
        "\r\n",
        "    def forward(self, x):\r\n",
        "        return x"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "23wYs4dxo4qj"
      },
      "source": [
        "## 9. Tagger\r\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hF7K-Rcqo5mN"
      },
      "source": [
        "import torch\r\n",
        "import torch.nn as nn\r\n",
        "import torch.autograd as autograd\r\n",
        "import torch.nn.functional as F\r\n",
        "import rationale_net.models.cnn as cnn\r\n",
        "import pdb\r\n",
        "\r\n",
        "'''\r\n",
        "    Implements a CNN with arbitary number of layers for tagging (predicts 0/1 for each token in text if token matches label), no max pool over time.\r\n",
        "'''\r\n",
        "\r\n",
        "\r\n",
        "\r\n",
        "class Tagger(nn.Module):\r\n",
        "\r\n",
        "    def __init__(self, embeddings, args):\r\n",
        "        super(Tagger, self).__init__()\r\n",
        "        vocab_size, hidden_dim = embeddings.shape\r\n",
        "        self.embedding_layer = nn.Embedding(vocab_size, hidden_dim)\r\n",
        "        self.embedding_layer.weight.data = torch.from_numpy(embeddings)\r\n",
        "        self.embedding_layer.weight.requires_grad = False\r\n",
        "        \r\n",
        "        self.model = args['model']\r\n",
        "        self.filters = args['filters']\r\n",
        "        self.filter_num = args['filter_num']\r\n",
        "        self.num_tags = args['num_class']\r\n",
        "        self.dropout = args['dropout']\r\n",
        "        self.cuda = args['cuda']\r\n",
        "\r\n",
        "        if args.model == 'TextCNN':\r\n",
        "            self.cnn = TextCNN(args, max_pool_over_time=False)\r\n",
        "\r\n",
        "        self.hidden = nn.Linear((len(self.filters)*self.filter_num), self.num_tags)\r\n",
        "        self.dropout = nn.Dropout(self.dropout)\r\n",
        "\r\n",
        "    \r\n",
        "    def forward(self, x_indx, mask):\r\n",
        "        '''Given input x_indx of dim (batch_size, 1, max_length), return z (batch, length) such that z\r\n",
        "        can act as element-wise mask on x'''\r\n",
        "        if self.model == 'TextCNN':\r\n",
        "            ## embedding layer takes in dim (batch_size, max_length), outputs x of dim (batch_size, max_length, hidden_dim)\r\n",
        "            x = self.embedding_layer(x_indx.squeeze(1))\r\n",
        "        \r\n",
        "            if self.cuda:\r\n",
        "                x = x.to(device)\r\n",
        "            ## switch x to dim (batch_size, hidden_dim, max_length)\r\n",
        "            x = torch.transpose(x, 1, 2)\r\n",
        "            ## activ of dim (batch_size, len(filters)*filter_num, max_length)\r\n",
        "            activ = self.cnn(x)\r\n",
        "        else:\r\n",
        "            raise NotImplementedError(\"Model form {} not yet supported for generator!\".format(model))\r\n",
        "\r\n",
        "        ## hidden layer takes activ transposed to dim (batch_size, max_length, len(filters)*filter_num) and outputs logit of dim (batch_size, max_length, num_tags)\r\n",
        "        logit = self.hidden(torch.transpose(activ, 1, 2))\r\n",
        "        return logit, self.hidden"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dQHjZOnLugfg"
      },
      "source": [
        "## 10. Get Model\r\n",
        "\r\n",
        "get encoder, generator or tagger and empty generator"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I5qnNVGAsVas"
      },
      "source": [
        "def get_model(args, embeddings, train_data):\r\n",
        "\r\n",
        "    if args['snapshot'] is None:\r\n",
        "        if args['use_as_tagger'] == True:\r\n",
        "            gen = empty.Empty()\r\n",
        "            model = Tagger(embeddings, args)\r\n",
        "        else:\r\n",
        "            gen   = Generator(embeddings, args)\r\n",
        "            model = Encoder(embeddings, args)\r\n",
        "    else :\r\n",
        "        print('\\nLoading model from [%s]...' % args['snapshot'])\r\n",
        "        try:\r\n",
        "            gen_path = learn.get_gen_path(args['snapshot'])\r\n",
        "            if os.path.exists(gen_path):\r\n",
        "                gen   = torch.load(gen_path)\r\n",
        "            model = torch.load(args['snapshot'])\r\n",
        "        except :\r\n",
        "            print(\"Sorry, This snapshot doesn't exist.\"); exit()\r\n",
        "\r\n",
        "    if args['num_gpus'] > 1:\r\n",
        "        model = nn.DataParallel(model,\r\n",
        "                                    device_ids=range(args['num_gpus']))\r\n",
        "\r\n",
        "        if not gen is None:\r\n",
        "            gen = nn.DataParallel(gen,\r\n",
        "                                    device_ids=range(args['num_gpus']))\r\n",
        "    return gen, model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vk03OHE5uq7p"
      },
      "source": [
        "## 11. Utils.learn\r\n",
        "\r\n",
        "utility functions for the models and training. The following functions are changed into our dataformat."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "92aGbSizutCg"
      },
      "source": [
        "# Train the model\r\n",
        "import sklearn.metrics\r\n",
        "import sys, os\r\n",
        "\r\n",
        "def get_hard_mask(z, return_ind=False):\r\n",
        "    '''\r\n",
        "        -z: torch Tensor where each element probablity of element\r\n",
        "        being selected\r\n",
        "        -args: experiment level config\r\n",
        "        returns: A torch variable that is binary mask of z >= .5\r\n",
        "    '''\r\n",
        "    max_z, ind = torch.max(z, dim=-1)\r\n",
        "    if return_ind:\r\n",
        "        del z\r\n",
        "        return ind\r\n",
        "    masked = torch.ge(z, max_z.unsqueeze(-1)).float()\r\n",
        "    del z\r\n",
        "    return masked\r\n",
        "\r\n",
        "def get_gen_path(model_path):\r\n",
        "    '''\r\n",
        "        -model_path: path of encoder model\r\n",
        "        returns: path of generator\r\n",
        "    '''\r\n",
        "    return '{}.gen'.format(model_path)\r\n",
        "\r\n",
        "def one_hot(label, num_class):\r\n",
        "    vec = torch.zeros( (1, num_class) )\r\n",
        "    vec[0][label] = 1\r\n",
        "    return vec\r\n",
        "\r\n",
        "    \r\n",
        "def gumbel_softmax(input, temperature, cuda):\r\n",
        "    noise = torch.rand(input.size())\r\n",
        "    noise.add_(1e-9).log_().neg_()\r\n",
        "    noise.add_(1e-9).log_().neg_()\r\n",
        "    noise = autograd.Variable(noise)\r\n",
        "    if cuda:\r\n",
        "        noise = noise.to(device)\r\n",
        "    x = (input + noise) / temperature\r\n",
        "    x = F.softmax(x.view(-1,  x.size()[-1]), dim=-1)\r\n",
        "    return x.view_as(input)\r\n",
        "\r\n",
        "\r\n",
        "def get_optimizer(models, args):\r\n",
        "    '''\r\n",
        "    Save the parameters of every model in models and pass them to\r\n",
        "    Adam optimizer.\r\n",
        "    \r\n",
        "        :param models: list of models (such as TextCNN, etc.)\r\n",
        "        :param args: arguments\r\n",
        "        \r\n",
        "        :return: torch optimizer over models\r\n",
        "    '''\r\n",
        "    params = []\r\n",
        "    for model in models:\r\n",
        "        params.extend([param for param in model.parameters() if param.requires_grad])\r\n",
        "    return torch.optim.Adam(params, lr=args['lr'],  weight_decay=args['weight_decay'])\r\n",
        "\r\n",
        "\r\n",
        "def init_metrics_dictionary(modes):\r\n",
        "    '''\r\n",
        "    Create dictionary with empty array for each metric in each mode\r\n",
        "    \r\n",
        "        :param modes: list with either train, dev or test\r\n",
        "        \r\n",
        "        :return epoch_stats: statistics for a given epoch\r\n",
        "    '''\r\n",
        "    epoch_stats = {}\r\n",
        "    metrics = ['loss', 'obj_loss', 'k_selection_loss', 'k_continuity_loss',\r\n",
        "               'accuracy', 'precision', 'recall', 'f1', 'confusion_matrix', 'mse']\r\n",
        "    for metric in metrics:\r\n",
        "        for mode in modes:\r\n",
        "            key = \"{}_{}\".format(mode, metric)\r\n",
        "            epoch_stats[key] = []\r\n",
        "    return epoch_stats\r\n",
        "\r\n",
        "\r\n",
        "def get_train_loader(train_data, args):\r\n",
        "    '''\r\n",
        "    Iterative train loader with sampler and replacer if class_balance\r\n",
        "    is true, normal otherwise.\r\n",
        "    \r\n",
        "        :param train_data: training data\r\n",
        "        :param args: arguments\r\n",
        "        \r\n",
        "        :return train_loader: iterable training set\r\n",
        "    '''\r\n",
        "    \r\n",
        "    if args['class_balance']:\r\n",
        "        # If the class_balance is true: sample and replace\r\n",
        "        sampler = data.sampler.WeightedRandomSampler(\r\n",
        "                weights=train_data.weights,\r\n",
        "                num_samples=len(train_data),\r\n",
        "                replacement=True)\r\n",
        "        train_loader = data.DataLoader(\r\n",
        "                train_data,\r\n",
        "                num_workers=args['num_workers'],\r\n",
        "                sampler=sampler,\r\n",
        "                batch_size=args['batch_size'])\r\n",
        "    else:\r\n",
        "        # If the class_balance is false, do not sample\r\n",
        "        train_loader = data.DataLoader(\r\n",
        "            train_data,\r\n",
        "            batch_size=args['batch_size'],\r\n",
        "            shuffle=True,\r\n",
        "            num_workers=args['num_workers'],\r\n",
        "            drop_last=False)\r\n",
        "    return train_loader\r\n",
        "\r\n",
        "\r\n",
        "def get_dev_loader(dev_data, args):\r\n",
        "    '''\r\n",
        "    Iterative dev loader\r\n",
        "    \r\n",
        "        :param dev_data: dev set\r\n",
        "        :param args: arguments\r\n",
        "        \r\n",
        "        :return dev_loader: iterative dev set\r\n",
        "    '''\r\n",
        "    \r\n",
        "    dev_loader = data.DataLoader(\r\n",
        "        dev_data,\r\n",
        "        batch_size=args['batch_size'],\r\n",
        "        shuffle=False,\r\n",
        "        num_workers=args['num_workers'],\r\n",
        "        drop_last=False)\r\n",
        "    return dev_loader\r\n",
        "\r\n",
        "\r\n",
        "def get_x_indx(batch, eval_model):\r\n",
        "    '''\r\n",
        "    Given a batch, return all the x\r\n",
        "    \r\n",
        "        :param batch: batch of dictionaries\r\n",
        "        :param eval_model: true or false, for volatile\r\n",
        "        \r\n",
        "        :return x_indx: tensor of batch*x\r\n",
        "    '''\r\n",
        "    \r\n",
        "    x_indx = autograd.Variable(batch['x'], volatile=eval_model)\r\n",
        "    return x_indx\r\n",
        "\r\n",
        "\r\n",
        "def get_loss(logit, y, args):\r\n",
        "    '''\r\n",
        "    Return the cross entropy or mse loss\r\n",
        "    \r\n",
        "        :param logit: predictions\r\n",
        "        :param y: gold standard\r\n",
        "        :param args: arguments\r\n",
        "        \r\n",
        "        :return loss: loss\r\n",
        "    '''\r\n",
        "    \r\n",
        "    if args['objective'] == 'cross_entropy':\r\n",
        "        loss_fn = nn.CrossEntropyLoss()\r\n",
        "        loss = F.cross_entropy(logit, y)\r\n",
        "    elif args['objective'] == 'mse':\r\n",
        "        loss = F.mse_loss(logit, y.float())\r\n",
        "    else:\r\n",
        "        raise Exception(\"Objective {} not supported!\".format(args['objective']))\r\n",
        "    return loss\r\n",
        "\r\n",
        "\r\n",
        "def tensor_to_numpy(tensor):\r\n",
        "    '''\r\n",
        "    Return a numpy matrix from a tensor\r\n",
        "\r\n",
        "        :param tensor: tensor\r\n",
        "        \r\n",
        "        :return numpy_matrix: numpy matrix\r\n",
        "    '''\r\n",
        "    return tensor.data[0]\r\n",
        "\r\n",
        "\r\n",
        "def get_metrics(preds, golds, args):\r\n",
        "    '''\r\n",
        "    Return the metrics given predictions and golds\r\n",
        "    \r\n",
        "        :param preds: list of predictions\r\n",
        "        :param golds: list of golds\r\n",
        "        :param args: arguments\r\n",
        "        \r\n",
        "        :return metrics: metrics dictionary\r\n",
        "    '''\r\n",
        "    metrics = {}\r\n",
        "\r\n",
        "    if args['objective']  in ['cross_entropy', 'margin']:\r\n",
        "        metrics['accuracy'] = sklearn.metrics.accuracy_score(y_true=golds, y_pred=preds)\r\n",
        "        metrics['confusion_matrix'] = sklearn.metrics.confusion_matrix(y_true=golds,y_pred=preds)\r\n",
        "        metrics['precision'] = sklearn.metrics.precision_score(y_true=golds, y_pred=preds, average=\"weighted\")\r\n",
        "        metrics['recall'] = sklearn.metrics.recall_score(y_true=golds,y_pred=preds, average=\"weighted\")\r\n",
        "        metrics['f1'] = sklearn.metrics.f1_score(y_true=golds,y_pred=preds, average=\"weighted\")\r\n",
        "        metrics['mse'] = \"NA\"\r\n",
        "    elif args['objective'] == 'mse':\r\n",
        "        metrics['mse'] = sklearn.metrics.mean_squared_error(y_true=golds, y_pred=preds)\r\n",
        "        metrics['confusion_matrix'] = \"NA\"\r\n",
        "        metrics['accuracy'] = \"NA\"\r\n",
        "        metrics['precision'] = \"NA\"\r\n",
        "        metrics['recall'] = \"NA\"\r\n",
        "        metrics['f1'] = 'NA'\r\n",
        "    return metrics\r\n",
        "\r\n",
        "\r\n",
        "def collate_epoch_stat(stat_dict, epoch_details, mode, args):\r\n",
        "    '''\r\n",
        "    Update stat_dict with details from epoch_details and create\r\n",
        "    log statement\r\n",
        "\r\n",
        "        :param stat_dict: a dictionary of statistics lists to update\r\n",
        "        :param epoch_details: list of statistics for a given epoch\r\n",
        "        :param mode: train, dev or test\r\n",
        "        :param args: model run configuration\r\n",
        "\r\n",
        "        :return stat_dict: updated stat_dict with epoch details\r\n",
        "        :return log_statement: log statement sumarizing new epoch\r\n",
        "\r\n",
        "    '''\r\n",
        "    log_statement_details = ''\r\n",
        "    for metric in epoch_details:\r\n",
        "        loss = epoch_details[metric]\r\n",
        "        stat_dict['{}_{}'.format(mode, metric)].append(loss)\r\n",
        "\r\n",
        "        log_statement_details += ' -{}: {}'.format(metric, loss)\r\n",
        "\r\n",
        "    log_statement = '\\n {} - {}\\n--'.format(args['objective'], log_statement_details )\r\n",
        "\r\n",
        "    return stat_dict, log_statement"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HjwmACh8uees"
      },
      "source": [
        "## 12. Train model\r\n",
        "\r\n",
        "Function to train model and extracts rationales, statistics"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Swsvadd2t1ss"
      },
      "source": [
        "import os\r\n",
        "import sys\r\n",
        "import torch\r\n",
        "import torch.autograd as autograd\r\n",
        "import torch.nn.functional as F\r\n",
        "import rationale_net.utils.generic as generic\r\n",
        "import rationale_net.utils.metrics as metrics\r\n",
        "import tqdm\r\n",
        "import numpy as np\r\n",
        "import pdb\r\n",
        "import sklearn.metrics\r\n",
        "import rationale_net.utils.learn as learn\r\n",
        "\r\n",
        "\r\n",
        "\r\n",
        "def train_model(train_data, dev_data, model, gen, args):\r\n",
        "    '''\r\n",
        "    Train model and tune on dev set. If model doesn't improve dev performance within args.patience\r\n",
        "    epochs, then halve the learning rate, restore the model to best and continue training.\r\n",
        "    At the end of training, the function will restore the model to best dev version.\r\n",
        "    returns epoch_stats: a dictionary of epoch level metrics for train and test\r\n",
        "    returns model : best model from this call to train\r\n",
        "    '''\r\n",
        "\r\n",
        "    snapshot = '{}'.format(os.path.join(args['save_dir'], args['model_path']))\r\n",
        "\r\n",
        "    if args['cuda']:\r\n",
        "        model = model.to(device)\r\n",
        "        gen = gen.to(device)\r\n",
        "\r\n",
        "    args['lr'] = args['init_lr']\r\n",
        "    optimizer = get_optimizer([model, gen], args)\r\n",
        "\r\n",
        "    num_epoch_sans_improvement = 0\r\n",
        "    epoch_stats = init_metrics_dictionary(modes=['train', 'dev'])\r\n",
        "    step = 0\r\n",
        "    tuning_key = \"dev_{}\".format(args['tuning_metric'])\r\n",
        "    best_epoch_func = min if tuning_key == 'loss' else max\r\n",
        "\r\n",
        "    train_loader = get_train_loader(train_data, args)\r\n",
        "    dev_loader = get_dev_loader(dev_data, args)\r\n",
        "\r\n",
        "    rationale_list = []\r\n",
        "    gold_list = []\r\n",
        "    y_list = []\r\n",
        "    text_list = []\r\n",
        "\r\n",
        "    for epoch in range(1, args['epochs'] + 1):\r\n",
        "\r\n",
        "        print(\"-------------\\nEpoch {}:\\n\".format(epoch))\r\n",
        "        for mode, dataset, loader in [('Train', train_data, train_loader), \r\n",
        "                                      ('Dev', dev_data, dev_loader)]:\r\n",
        "\r\n",
        "            train_model = mode == 'Train'\r\n",
        "            print('{}'.format(mode))\r\n",
        "            key_prefix = mode.lower()\r\n",
        "            epoch_details, step, losses, preds, golds, rationales, text, y = run_epoch(\r\n",
        "                data_loader=loader,\r\n",
        "                train_model=train_model,\r\n",
        "                model=model,\r\n",
        "                gen=gen,\r\n",
        "                optimizer=optimizer,\r\n",
        "                step=step,\r\n",
        "                args=args)\r\n",
        "\r\n",
        "\r\n",
        "            rationale_list.append(rationales)\r\n",
        "            gold_list.append(golds)\r\n",
        "            y_list.append(y)\r\n",
        "            text_list.append(text)\r\n",
        "            epoch_stats, log_statement = collate_epoch_stat(epoch_stats, epoch_details, key_prefix, args)\r\n",
        "\r\n",
        "            # Log  performance\r\n",
        "            print(log_statement)\r\n",
        "\r\n",
        "\r\n",
        "        # Save model if beats best dev\r\n",
        "        best_func = min if args['tuning_metric'] == 'loss' else max\r\n",
        "        if best_func(epoch_stats[tuning_key]) == epoch_stats[tuning_key][-1]:\r\n",
        "            num_epoch_sans_improvement = 0\r\n",
        "            if not os.path.isdir(args['save_dir']):\r\n",
        "                os.makedirs(args['save_dir'])\r\n",
        "            # Subtract one because epoch is 1-indexed and arr is 0-indexed\r\n",
        "            epoch_stats['best_epoch'] = epoch - 1\r\n",
        "            torch.save(model, snapshot)\r\n",
        "            torch.save(gen, get_gen_path(args['model_path']))\r\n",
        "        else:\r\n",
        "            num_epoch_sans_improvement += 1\r\n",
        "\r\n",
        "        if not train_model:\r\n",
        "            print('---- Best Dev {} is {:.4f} at epoch {}'.format(\r\n",
        "                args['tuning_metric'],\r\n",
        "                epoch_stats[tuning_key][epoch_stats['best_epoch']],\r\n",
        "                epoch_stats['best_epoch'] + 1))\r\n",
        "\r\n",
        "        if num_epoch_sans_improvement >= args['patience']:\r\n",
        "            print(\"Reducing learning rate\")\r\n",
        "            num_epoch_sans_improvement = 0\r\n",
        "            model.cpu()\r\n",
        "            gen.cpu()\r\n",
        "            model = torch.load(snapshot)\r\n",
        "            gen = torch.loadget_gen_path(snapshot)\r\n",
        "\r\n",
        "            if args['cuda']:\r\n",
        "                model = model.to(device)\r\n",
        "                gen   = gen.to(device)\r\n",
        "            args['lr'] *= .5\r\n",
        "            optimizer = get_optimizer([model, gen], args)\r\n",
        "\r\n",
        "    # Restore model to best dev performance\r\n",
        "    if os.path.exists(args['model_path']):\r\n",
        "        model.cpu()\r\n",
        "        model = torch.load(args['model_path'])\r\n",
        "        gen.cpu()\r\n",
        "        gen = torch.load(get_gen_path(args['model_path']))\r\n",
        "\r\n",
        "    return epoch_stats, model, gen, rationale_list, gold_list, y_list, text_list"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u_e3_fmYwuCV"
      },
      "source": [
        "## 13. Run Epoch\r\n",
        "\r\n",
        "Function used in train model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SHCbxqoawwey"
      },
      "source": [
        "def run_epoch(data_loader, train_model, model, gen, optimizer, step, args):\r\n",
        "    '''\r\n",
        "    Train model for one pass of train data, and return loss, acccuracy\r\n",
        "    \r\n",
        "        :param data_loader: iterable dataset\r\n",
        "        :param train_model: true if training, false otherwise\r\n",
        "        :param model: text classifier, such as TextCNN\r\n",
        "        :param optimizer: Adam\r\n",
        "        :param args: arguments\r\n",
        "        \r\n",
        "        :return epoch_stat:\r\n",
        "        :return step: number of steps\r\n",
        "        :return losses: list of losses\r\n",
        "        :return preds: list of predictions\r\n",
        "        :return golds: list of gold standards\r\n",
        "    '''\r\n",
        "    \r\n",
        "    eval_model = not train_model\r\n",
        "    data_iter = data_loader.__iter__()\r\n",
        "\r\n",
        "    losses = []\r\n",
        "    obj_losses = []\r\n",
        "    k_selection_losses = []\r\n",
        "    k_continuity_losses = []\r\n",
        "\r\n",
        "    preds = []\r\n",
        "    golds = []\r\n",
        "    losses = []\r\n",
        "    texts = []\r\n",
        "    rationales = []\r\n",
        "    y_list = []\r\n",
        "    text_list = []\r\n",
        "\r\n",
        "    if train_model:\r\n",
        "        model.train()\r\n",
        "        gen.train()\r\n",
        "    else:\r\n",
        "        model.eval()\r\n",
        "        gen.eval()\r\n",
        "\r\n",
        "    num_batches_per_epoch = len(data_iter)\r\n",
        "    if train_model:\r\n",
        "        num_batches_per_epoch = min(len(data_iter), 10000)\r\n",
        "\r\n",
        "    for _ in tqdm.tqdm(range(num_batches_per_epoch)):\r\n",
        "        # Get the batch\r\n",
        "        batch = data_iter.next()\r\n",
        "        \r\n",
        "        if train_model:\r\n",
        "            step += 1\r\n",
        "            #if step % 100 == 0:\r\n",
        "            #    args['gumbel_temprature'] = max(np.exp((step+1) * -1 * args['gumbel_decay']), .05)\r\n",
        "\r\n",
        "        # Load X and Y\r\n",
        "        x_indx = get_x_indx(batch, eval_model)\r\n",
        "        text = batch['text']\r\n",
        "        y = autograd.Variable(batch['y'], volatile=eval_model)\r\n",
        "\r\n",
        "        text_list.append(text)\r\n",
        "        y_list.append(y)\r\n",
        "\r\n",
        "        if args['cuda']:\r\n",
        "            x_indx, y = x_indx.to(device), y.to(device)\r\n",
        "\r\n",
        "        if train_model:\r\n",
        "            optimizer.zero_grad()\r\n",
        "        \r\n",
        "        if args['get_rationales']:\r\n",
        "            mask, z = gen(x_indx)\r\n",
        "        else:\r\n",
        "          mask = None\r\n",
        "\r\n",
        "        logit, _ = model(x_indx, mask = mask)\r\n",
        "        \r\n",
        "        # Calculate the loss\r\n",
        "        loss = get_loss(logit, y, args)\r\n",
        "        obj_loss = loss\r\n",
        "        \r\n",
        "        if args['get_rationales']:\r\n",
        "            selection_cost, continuity_cost = gen.loss(mask, x_indx)\r\n",
        "\r\n",
        "            loss += args['selection_lambda'] * selection_cost\r\n",
        "            loss += args['continuity_lambda'] * continuity_cost\r\n",
        "\r\n",
        "\r\n",
        "        # Backward step\r\n",
        "        if train_model:\r\n",
        "            loss.backward()\r\n",
        "            optimizer.step()\r\n",
        "        \r\n",
        "        if args['get_rationales']:\r\n",
        "            k_selection_losses.append( selection_cost.item())\r\n",
        "            k_continuity_losses.append(continuity_cost.item())\r\n",
        "\r\n",
        "        # Saving loss\r\n",
        "        obj_losses.append(obj_loss.item()) #obj_losses.append(obj_loss)\r\n",
        "        losses.append(loss.item())         #losses.append(loss.item) \r\n",
        "        \r\n",
        "        # Softmax, preds, text and gold\r\n",
        "        batch_softmax = F.softmax(logit, dim=-1).cpu()\r\n",
        "        preds.extend(torch.max(batch_softmax, 1)[1].view(y.size()).data.numpy())\r\n",
        "        \r\n",
        "        texts.extend(text)\r\n",
        "        rationales.extend(learn.get_rationales(mask, text))\r\n",
        "\r\n",
        "\r\n",
        "        if args['use_as_tagger']:\r\n",
        "            golds.extend(batch['y'].view(-1).numpy())\r\n",
        "        else:\r\n",
        "            golds.extend(batch['y'].numpy())\r\n",
        "\r\n",
        "        \r\n",
        "\r\n",
        "    # Get metrics\r\n",
        "    epoch_metrics = get_metrics(preds, golds, args)\r\n",
        "    epoch_stat = {'loss' : np.mean(losses), 'obj_loss': np.mean(obj_losses)}\r\n",
        "\r\n",
        "    for metric_k in epoch_metrics.keys():\r\n",
        "        epoch_stat[metric_k] = epoch_metrics[metric_k]\r\n",
        "    \r\n",
        "    if args['get_rationales']:\r\n",
        "        epoch_stat['k_selection_loss'] = np.mean(k_selection_losses)\r\n",
        "        epoch_stat['k_continuity_loss'] = np.mean(k_continuity_losses)\r\n",
        "\r\n",
        "    return epoch_stat, step, losses, preds, golds, rationales, text_list, y_list"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-lF6ETnurMsJ"
      },
      "source": [
        "## 14. Training the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XvxE_abwrNFM",
        "outputId": "5ed0889d-b3be-456e-fcfd-1e2867ecd0b1"
      },
      "source": [
        "# Creating the encoder and TextCNN, and printing an output from a random input\r\n",
        "\r\n",
        "encoder = Encoder(emb_tensor, args)                    \r\n",
        "#encoder.to(device)\r\n",
        "print(\"Output logits for the first (randomly sorted) element of the dataset:\\n\\n\")\r\n",
        "\r\n",
        "print(encoder(train_data[0]['x']))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Output logits for the first (randomly sorted) element of the dataset:\n",
            "\n",
            "\n",
            "(tensor([[-0.0080, -0.0564]], grad_fn=<AddmmBackward>), tensor([[0.0000, 0.0000, 0.1002, 0.0000, 0.0000, 0.0000, 0.1807, 0.0000, 0.0000,\n",
            "         0.0000, 0.0436, 0.0164, 0.0000, 0.0220, 0.0000, 0.0000, 0.0000, 0.0646,\n",
            "         0.0000, 0.0000, 0.1417, 0.0866, 0.0929, 0.0000, 0.0368, 0.1307, 0.0000,\n",
            "         0.0000, 0.0561, 0.0620, 0.1641, 0.0000, 0.0000, 0.2966, 0.0000, 0.0000,\n",
            "         0.0000, 0.0000, 0.2636, 0.0000, 0.0000, 0.0585, 0.0149, 0.0000, 0.0000,\n",
            "         0.0326, 0.0895, 0.0000, 0.0000, 0.2108, 0.0000, 0.0487, 0.0000, 0.0841,\n",
            "         0.0000, 0.3120, 0.0000, 0.1544, 0.1043, 0.0000, 0.0952, 0.0000, 0.0000,\n",
            "         0.0366, 0.0000, 0.0053, 0.0000, 0.1681, 0.0964, 0.0000, 0.0000, 0.0403,\n",
            "         0.0000, 0.0175, 0.0000, 0.0000, 0.0000, 0.2582, 0.0000, 0.0209, 0.0000,\n",
            "         0.0000, 0.1520, 0.0769, 0.0000, 0.0000, 0.0000, 0.0887, 0.0000, 0.0000,\n",
            "         0.0000, 0.0000, 0.2660, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0316,\n",
            "         0.0611, 0.0000, 0.0821, 0.0859, 0.1734, 0.0000, 0.1215, 0.0000, 0.0000,\n",
            "         0.0000, 0.0000, 0.0391, 0.0000, 0.0108, 0.0126, 0.0621, 0.0000, 0.0310,\n",
            "         0.0000, 0.0000, 0.2170, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0277,\n",
            "         0.0268, 0.0000, 0.3881, 0.0585, 0.0000, 0.0000, 0.0948, 0.0000, 0.0000,\n",
            "         0.3251, 0.0771, 0.0000, 0.0865, 0.0718, 0.0000, 0.0032, 0.0189, 0.1298,\n",
            "         0.0000, 0.0000, 0.0141, 0.0000, 0.0676, 0.0090, 0.0000, 0.0000, 0.0000,\n",
            "         0.1408, 0.0556, 0.1734, 0.0406, 0.1553, 0.0000, 0.0000, 0.0000, 0.0000,\n",
            "         0.0000, 0.1190, 0.0890, 0.4063, 0.1205, 0.0000, 0.2322, 0.0679, 0.0269,\n",
            "         0.0000, 0.0000, 0.0000, 0.0000, 0.1799, 0.1177, 0.0000, 0.0696, 0.0000,\n",
            "         0.0000, 0.0000, 0.3078, 0.0000, 0.1663, 0.1540, 0.1751, 0.0000, 0.0000,\n",
            "         0.3008, 0.0000, 0.0000, 0.0000, 0.0000, 0.1408, 0.0349, 0.0042, 0.1680,\n",
            "         0.0521, 0.1012, 0.2040, 0.0417, 0.0237, 0.0000, 0.0000, 0.0126, 0.0000,\n",
            "         0.0000, 0.0000, 0.0000, 0.0000, 0.2417, 0.2618, 0.0000, 0.0000, 0.0854,\n",
            "         0.0000, 0.0000, 0.0493, 0.0000, 0.0000, 0.0382, 0.0000, 0.0000, 0.0000,\n",
            "         0.0000, 0.0000, 0.0000, 0.0306, 0.0000, 0.1118, 0.0000, 0.0000, 0.1772,\n",
            "         0.0000, 0.0700, 0.2529, 0.0000, 0.3616, 0.0985, 0.0000, 0.0000, 0.2184,\n",
            "         0.1935, 0.0000, 0.0000, 0.1778, 0.0000, 0.0414, 0.0000, 0.0000, 0.0268,\n",
            "         0.0000, 0.0000, 0.0423, 0.0629, 0.0000, 0.0000, 0.0848, 0.0030, 0.1075,\n",
            "         0.2451, 0.0000, 0.1039, 0.0430, 0.1705, 0.0000, 0.0463, 0.0000, 0.0000,\n",
            "         0.2622, 0.0329, 0.0403, 0.2237, 0.0000, 0.0000, 0.0000, 0.0499, 0.0000,\n",
            "         0.0000, 0.0000, 0.0000, 0.0488, 0.0000, 0.0859, 0.0000, 0.0000, 0.0000,\n",
            "         0.0000, 0.0000, 0.1294, 0.0000, 0.0000, 0.0196, 0.0000, 0.0868, 0.0000,\n",
            "         0.0000, 0.0000, 0.0000]], grad_fn=<MulBackward0>))\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cGktuBK5tBts"
      },
      "source": [
        "gen, enc = get_model(args, emb_tensor, train_data)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5vSUziUlrEIw",
        "outputId": "142ce9cd-c632-400a-b1aa-542a7c7cb1cc"
      },
      "source": [
        "epoch_stats, model, gen, rationale_list, gold_list, y_list, text_list = train_model(train_data, dev_data, enc, gen, args)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "-------------\n",
            "Epoch 1:\n",
            "\n",
            "Train\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 274/274 [28:33<00:00,  6.25s/it]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            " cross_entropy -  -loss: 1.0681992891061045 -obj_loss: 1.0681992891061045 -accuracy: 0.7982214285714285 -confusion_matrix: [[49831 20318]\n",
            " [ 7931 61920]] -precision: 0.8079327126025113 -recall: 0.7982214285714285 -f1: 0.7966682340159457 -mse: NA -k_selection_loss: 34.716494389694105 -k_continuity_loss: 31.95054263268074\n",
            "--\n",
            "Dev\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 59/59 [05:52<00:00,  5.98s/it]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            " cross_entropy -  -loss: 0.4079273651211949 -obj_loss: 0.4079273651211949 -accuracy: 0.8262 -confusion_matrix: [[10803  4097]\n",
            " [ 1117 13983]] -precision: 0.8394009955752212 -recall: 0.8262 -f1: 0.8243492063492064 -mse: NA -k_selection_loss: 1.0 -k_continuity_loss: 1.9846276347920047\n",
            "--\n",
            "---- Best Dev loss is 0.4079 at epoch 1\n",
            "-------------\n",
            "Epoch 2:\n",
            "\n",
            "Train\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 274/274 [29:18<00:00,  6.42s/it]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            " cross_entropy -  -loss: 0.33957835020375077 -obj_loss: 0.33957835020375077 -accuracy: 0.8700642857142857 -confusion_matrix: [[56062 14087]\n",
            " [ 4104 65747]] -precision: 0.8777827390839731 -recall: 0.8700642857142857 -f1: 0.8694202402229159 -mse: NA -k_selection_loss: 1.7312692229765174 -k_continuity_loss: 3.264578269345917\n",
            "--\n",
            "Dev\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "100%|██████████| 59/59 [06:10<00:00,  6.29s/it]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "\n",
            " cross_entropy -  -loss: 0.3985718892792524 -obj_loss: 0.3985718892792524 -accuracy: 0.8438666666666667 -confusion_matrix: [[11358  3542]\n",
            " [ 1142 13958]] -precision: 0.8527498666666666 -recall: 0.8438666666666667 -f1: 0.8427759497260907 -mse: NA -k_selection_loss: 1.0 -k_continuity_loss: 1.9820768994800115\n",
            "--\n",
            "---- Best Dev loss is 0.3986 at epoch 2\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ABTafHCokpf5"
      },
      "source": [
        "## 15. Retrieve the lexicons from the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "K5ycxiROC4HG",
        "outputId": "b94a1160-4a62-4af8-9e89-2c79b2145c9f"
      },
      "source": [
        "#Using the rationale set from last epochs' training and validation set.\r\n",
        "final_epoch_train_rationale = rationale_list[-2]\r\n",
        "final_epoch_dev_rationale = rationale_list[-1]\r\n",
        "\r\n",
        "final_epoch_train_y = []\r\n",
        "final_epoch_dev_y = []\r\n",
        "\r\n",
        "for i in range(len(y_list[-2])):\r\n",
        "  final_epoch_train_y += list(y_list[-2][i].cpu().numpy())\r\n",
        "\r\n",
        "for i in range(len(y_list[-1])):\r\n",
        "  final_epoch_dev_y += list(y_list[-1][i].cpu().numpy())\r\n",
        "\r\n",
        "#Check whether they match or not\r\n",
        "print(len(final_epoch_train_rationale), len(final_epoch_train_y))\r\n",
        "print(len(final_epoch_dev_rationale), len(final_epoch_dev_y))\r\n",
        "\r\n",
        "final_epoch_train_text = []\r\n",
        "final_epoch_dev_text = []\r\n",
        "\r\n",
        "for i in range(len(text_list[-2])):\r\n",
        "  final_epoch_train_text += text_list[-2][i]\r\n",
        "\r\n",
        "for i in range(len(text_list[-1])):\r\n",
        "  final_epoch_dev_text += text_list[-1][i]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "140000 140000\n",
            "30000 30000\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zQGtVWp8H2T7"
      },
      "source": [
        "#Extract the vocabulary from _ _ _ _ word _ _ _ form\r\n",
        "\r\n",
        "import re\r\n",
        "def extract_vocab(rationale):\r\n",
        "  ret_set = []\r\n",
        "  for article in rationale:\r\n",
        "    vocab_list = re.findall(r'[a-zA-Z0-9]+', article)\r\n",
        "    ret_set.append(vocab_list)\r\n",
        "  return ret_set\r\n",
        "\r\n",
        "\r\n",
        "val_vocab = extract_vocab(final_epoch_dev_rationale)\r\n",
        "train_vocab = extract_vocab(final_epoch_train_rationale)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_ffrmv-MCvWq",
        "outputId": "acc7fc7c-a295-4efe-d735-7e2adecb3c0a"
      },
      "source": [
        "final_epoch_train =pd.DataFrame(final_epoch_train_rationale)\r\n",
        "final_epoch_train['label'] = final_epoch_train_y\r\n",
        "final_epoch_train['text'] = final_epoch_train_text\r\n",
        "final_epoch_train['vocab'] = train_vocab\r\n",
        "\r\n",
        "final_epoch_dev =pd.DataFrame(final_epoch_dev_rationale)\r\n",
        "final_epoch_dev['label'] = final_epoch_dev_y\r\n",
        "final_epoch_dev['text']  = final_epoch_dev_text\r\n",
        "final_epoch_dev['vocab'] = val_vocab\r\n",
        "final_epoch_dev"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>0</th>\n",
              "      <th>label</th>\n",
              "      <th>text</th>\n",
              "      <th>vocab</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ...</td>\n",
              "      <td>0</td>\n",
              "      <td>brings luxury lifestyle private atelier privat...</td>\n",
              "      <td>[abuja]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ...</td>\n",
              "      <td>0</td>\n",
              "      <td>at last a frustrate officer revealed magical p...</td>\n",
              "      <td>[]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ...</td>\n",
              "      <td>0</td>\n",
              "      <td>minister foreign affairs tuesday say important...</td>\n",
              "      <td>[kashmir]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ...</td>\n",
              "      <td>0</td>\n",
              "      <td>again flood sacks hundreds calabar a heavy dow...</td>\n",
              "      <td>[calabar]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>jonty _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ...</td>\n",
              "      <td>0</td>\n",
              "      <td>jonty work great accepted work specialist cons...</td>\n",
              "      <td>[jonty]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29995</th>\n",
              "      <td>_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ...</td>\n",
              "      <td>1</td>\n",
              "      <td>representatives woman soccer team lock intense...</td>\n",
              "      <td>[footballers]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29996</th>\n",
              "      <td>_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ jalan _ ...</td>\n",
              "      <td>1</td>\n",
              "      <td>singapore the football association fas conduct...</td>\n",
              "      <td>[jalan]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29997</th>\n",
              "      <td>_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ...</td>\n",
              "      <td>1</td>\n",
              "      <td>a man accuse repeatedly rap girl take explicit...</td>\n",
              "      <td>[]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29998</th>\n",
              "      <td>_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ...</td>\n",
              "      <td>1</td>\n",
              "      <td>but decision make government would proceed sec...</td>\n",
              "      <td>[stirling]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29999</th>\n",
              "      <td>_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ...</td>\n",
              "      <td>1</td>\n",
              "      <td>in song red rose caf fureys sing place poet sa...</td>\n",
              "      <td>[pub]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>30000 rows × 4 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                       0  ...          vocab\n",
              "0      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ...  ...        [abuja]\n",
              "1      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ...  ...             []\n",
              "2      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ...  ...      [kashmir]\n",
              "3      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ...  ...      [calabar]\n",
              "4      jonty _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ...  ...        [jonty]\n",
              "...                                                  ...  ...            ...\n",
              "29995  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ...  ...  [footballers]\n",
              "29996  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ jalan _ ...  ...        [jalan]\n",
              "29997  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ...  ...             []\n",
              "29998  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ...  ...     [stirling]\n",
              "29999  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ...  ...          [pub]\n",
              "\n",
              "[30000 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 99
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BYmh655QH_G9"
      },
      "source": [
        "train_peace = final_epoch_train[final_epoch_train['label'] == 1]\r\n",
        "train_nonpeace = final_epoch_train[final_epoch_train['label'] == 0]\r\n",
        "val_peace = final_epoch_dev[final_epoch_dev['label'] == 1]\r\n",
        "val_nonpeace = final_epoch_dev[final_epoch_dev['label'] == 0]"
      ],
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1Ya22TgpIhhR"
      },
      "source": [
        "peace_vocab_list = []\r\n",
        "nonpeace_vocab_list = []\r\n",
        "for vocabs in train_peace['vocab'].values:\r\n",
        "  peace_vocab_list += vocabs\r\n",
        "\r\n",
        "for vocabs in val_peace['vocab'].values:\r\n",
        "  peace_vocab_list += vocabs\r\n",
        "\r\n",
        "for vocabs in train_nonpeace['vocab'].values:\r\n",
        "  nonpeace_vocab_list+= vocabs\r\n",
        "\r\n",
        "for vocabs in val_nonpeace['vocab'].values:\r\n",
        "  nonpeace_vocab_list += vocabs\r\n"
      ],
      "execution_count": 42,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jlWABHJWFvYN",
        "outputId": "401e0755-3bae-463d-947f-6178ba8c6813"
      },
      "source": [
        "print( len(set(peace_vocab_list)), len(set(nonpeace_vocab_list)))"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "9653 8694\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2-uWK_XZMGLO",
        "outputId": "cc0aa79c-9581-435e-bd51-33b748359359"
      },
      "source": [
        "peace_vocab_set = set(peace_vocab_list) - set(nonpeace_vocab_list)\r\n",
        "nonpeace_vocab_set = set(nonpeace_vocab_list) -set(peace_vocab_list)\r\n",
        "print(len(set(peace_vocab_set)), len(set(nonpeace_vocab_set)))"
      ],
      "execution_count": 74,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "7080 6121\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t4bozY2wENJg"
      },
      "source": [
        "df_exp_peace_vocab = pd.DataFrame(set(peace_vocab_set))\r\n",
        "df_exp_nonpeace_vocab = pd.DataFrame(set(nonpeace_vocab_set))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tEv9CMBReUmJ",
        "outputId": "0c3bd116-c19f-408d-d48f-3d25b613d1d6"
      },
      "source": [
        "peace_vocab = list(peace_vocab_set)\r\n",
        "nonpeace_vocab = list(nonpeace_vocab_set)\r\n",
        "print(len(peace_vocab), len(nonpeace_vocab))"
      ],
      "execution_count": 83,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "7080 6121\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "clrqT0gpywav",
        "outputId": "0a7248bf-f634-4a92-e689-6ae6aa01f34c"
      },
      "source": [
        "print(len(set(peace_vocab) - set(nonpeace_vocab)),len(set(nonpeace_vocab) - set(nonpeace_vocab)))"
      ],
      "execution_count": 84,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "7080 6121\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pPPkJsd1r4wY"
      },
      "source": [
        "#Count the frequency of words for the ranking\r\n",
        "\r\n",
        "peace_dict = {peace_vocab[i]: 0 for i in range(len(peace_vocab))}\r\n",
        "nonpeace_dict = {nonpeace_vocab[i]: 0 for i in range(len(nonpeace_vocab))}\r\n",
        "\r\n",
        "for i in range(len(final_epoch_dev)):\r\n",
        "  vocab_list = final_epoch_dev['vocab'][i]\r\n",
        "  if vocab_list == []:\r\n",
        "    continue\r\n",
        "  if final_epoch_train['label'][i] == 0:\r\n",
        "    for vocab_ in vocab_list:\r\n",
        "      try:\r\n",
        "        peace_dict[vocab_] += 1\r\n",
        "      except:\r\n",
        "        pass\r\n",
        "  else:\r\n",
        "    for vocab_ in vocab_list:\r\n",
        "      try:\r\n",
        "        nonpeace_dict[vocab_] += 1\r\n",
        "      except:\r\n",
        "        pass\r\n",
        "\r\n",
        "for i in range(len(final_epoch_train)):\r\n",
        "  vocab_list = final_epoch_train['vocab'][i]\r\n",
        "  if vocab_list == []:\r\n",
        "    continue\r\n",
        "  if final_epoch_train['label'][i] == 0:\r\n",
        "    for vocab_ in vocab_list:\r\n",
        "      try:\r\n",
        "        peace_dict[vocab_] += 1\r\n",
        "      except:\r\n",
        "        pass\r\n",
        "  else:\r\n",
        "    for vocab_ in vocab_list:\r\n",
        "      try:\r\n",
        "        nonpeace_dict[vocab_] += 1\r\n",
        "      except:\r\n",
        "        pass\r\n"
      ],
      "execution_count": 93,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "A6M3hSSk0Mmd",
        "outputId": "d9cf45df-433b-4d27-c460-9a55330f55ab"
      },
      "source": [
        "dict(sorted(peace_dict.items(), key=lambda item: item[1], reverse = True))"
      ],
      "execution_count": 95,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'rogers': 24,\n",
              " 'rcmp': 22,\n",
              " 'rte': 21,\n",
              " 'meath': 15,\n",
              " 'news': 15,\n",
              " 'limerick': 14,\n",
              " 'gardai': 14,\n",
              " 'taranaki': 12,\n",
              " 'postmedia': 12,\n",
              " 'iwi': 11,\n",
              " 'hutt': 10,\n",
              " 'southland': 9,\n",
              " 'tallaght': 9,\n",
              " 'defenceman': 9,\n",
              " 'wicklow': 9,\n",
              " 'rotorua': 9,\n",
              " 'croke': 8,\n",
              " 'afl': 7,\n",
              " 'invercargill': 7,\n",
              " 'cashel': 7,\n",
              " 'westmeath': 7,\n",
              " 'tipperary': 7,\n",
              " 'carlow': 7,\n",
              " 'clair': 7,\n",
              " 'autocar': 7,\n",
              " 'rnz': 6,\n",
              " 'ocbc': 6,\n",
              " 'bathurst': 6,\n",
              " 'asb': 6,\n",
              " 'palmerston': 6,\n",
              " 'optus': 6,\n",
              " 'oireachtas': 6,\n",
              " 'nzx': 6,\n",
              " 'qantas': 6,\n",
              " 'aotearoa': 5,\n",
              " 'quebec': 5,\n",
              " 'gillard': 5,\n",
              " 'wairarapa': 5,\n",
              " 'ont': 5,\n",
              " 'timaru': 5,\n",
              " 'whanganui': 5,\n",
              " 'ballina': 5,\n",
              " 'hdb': 5,\n",
              " 'truro': 5,\n",
              " 'mediacorp': 5,\n",
              " 'sbs': 5,\n",
              " 'newsasia': 5,\n",
              " 'raptors': 5,\n",
              " 'leitrim': 5,\n",
              " 'northland': 5,\n",
              " 'scottish': 5,\n",
              " 'navan': 5,\n",
              " 'sligo': 5,\n",
              " 'ardern': 4,\n",
              " 'taoiseach': 4,\n",
              " 'wanganui': 4,\n",
              " 'brunton': 4,\n",
              " 'newstalk': 4,\n",
              " 'unsw': 4,\n",
              " 'crowe': 4,\n",
              " 'niall': 4,\n",
              " 'stoppers': 4,\n",
              " 'wollongong': 4,\n",
              " 'taupo': 4,\n",
              " 'guelph': 4,\n",
              " 'ute': 4,\n",
              " 'dail': 4,\n",
              " 'nenagh': 4,\n",
              " 'mick': 4,\n",
              " 'papakura': 4,\n",
              " 'justin': 4,\n",
              " 'simcoe': 4,\n",
              " 'punggol': 4,\n",
              " 'roscommon': 4,\n",
              " 'etobicoke': 4,\n",
              " 'donegal': 4,\n",
              " 'fremantle': 4,\n",
              " 'aest': 4,\n",
              " 'penrith': 4,\n",
              " 'gaelic': 4,\n",
              " 'portlaoise': 4,\n",
              " 'flinders': 4,\n",
              " 'liam': 4,\n",
              " 'dup': 4,\n",
              " 'inuit': 4,\n",
              " 'tampines': 4,\n",
              " 'snp': 4,\n",
              " 'ave': 3,\n",
              " 'holden': 3,\n",
              " 'sussex': 3,\n",
              " 'ponsonby': 3,\n",
              " 'belconnen': 3,\n",
              " 'kris': 3,\n",
              " 'lismore': 3,\n",
              " 'longford': 3,\n",
              " 'kyle': 3,\n",
              " 'conor': 3,\n",
              " 'award': 3,\n",
              " 'telstra': 3,\n",
              " 'yarra': 3,\n",
              " 'russell': 3,\n",
              " 'accc': 3,\n",
              " 'infocomm': 3,\n",
              " 'kotaku': 3,\n",
              " 'hervey': 3,\n",
              " 'takapuna': 3,\n",
              " 'wexford': 3,\n",
              " 'harp': 3,\n",
              " 'raglan': 3,\n",
              " 'fitzroy': 3,\n",
              " 'randwick': 3,\n",
              " 'gympie': 3,\n",
              " 'okanagan': 3,\n",
              " 'shane': 3,\n",
              " 'laoghaire': 3,\n",
              " 'sudbury': 3,\n",
              " 'dorset': 3,\n",
              " 'cheltenham': 3,\n",
              " 'derry': 3,\n",
              " 'hauraki': 3,\n",
              " 'ngv': 3,\n",
              " 'sharemarket': 3,\n",
              " 'rockhampton': 3,\n",
              " 'marlborough': 3,\n",
              " 'nzdf': 3,\n",
              " 'kmh': 3,\n",
              " 'newmarket': 3,\n",
              " 'burlington': 3,\n",
              " 'ordinaries': 3,\n",
              " 'y': 3,\n",
              " 'mcg': 3,\n",
              " 'woodbine': 3,\n",
              " 'coromandel': 3,\n",
              " 'hse': 3,\n",
              " 'jalan': 3,\n",
              " 'maryborough': 3,\n",
              " 'kavanagh': 3,\n",
              " 'wight': 3,\n",
              " 'fullerton': 3,\n",
              " 'nanaimo': 3,\n",
              " 'tauranga': 3,\n",
              " 'lamb': 3,\n",
              " 'cornwall': 3,\n",
              " 'ryerson': 3,\n",
              " 'council': 3,\n",
              " 'kilmore': 3,\n",
              " 'publican': 3,\n",
              " 'canucks': 3,\n",
              " 'yonge': 3,\n",
              " 'hurling': 3,\n",
              " 'manawatu': 2,\n",
              " 'anau': 2,\n",
              " 'sgx': 2,\n",
              " 'muir': 2,\n",
              " 'carey': 2,\n",
              " 'nme': 2,\n",
              " 'hq': 2,\n",
              " 'cllr': 2,\n",
              " 'garmin': 2,\n",
              " 'oamaru': 2,\n",
              " 'cavan': 2,\n",
              " 'waitangi': 2,\n",
              " 'toowoomba': 2,\n",
              " 'ridley': 2,\n",
              " 'hemsworth': 2,\n",
              " 'nzru': 2,\n",
              " 'byron': 2,\n",
              " 'techradar': 2,\n",
              " 'daryl': 2,\n",
              " 'pence': 2,\n",
              " 'bushfires': 2,\n",
              " 'fingal': 2,\n",
              " 'tierra': 2,\n",
              " 'garner': 2,\n",
              " 'dcu': 2,\n",
              " 'dixon': 2,\n",
              " 'kinsale': 2,\n",
              " 'multicultural': 2,\n",
              " 'mcmanus': 2,\n",
              " 'choir': 2,\n",
              " 'brant': 2,\n",
              " 'metservice': 2,\n",
              " 'crimestoppers': 2,\n",
              " 'bayfield': 2,\n",
              " 'masterton': 2,\n",
              " 'grazier': 2,\n",
              " 'ladner': 2,\n",
              " 'brookvale': 2,\n",
              " 'townland': 2,\n",
              " 'marae': 2,\n",
              " 'cooma': 2,\n",
              " 'antrim': 2,\n",
              " 'charlottetown': 2,\n",
              " 'flp': 2,\n",
              " 'wanaka': 2,\n",
              " 'rutherford': 2,\n",
              " 'payoh': 2,\n",
              " 'craigs': 2,\n",
              " 'temasek': 2,\n",
              " 'waitemata': 2,\n",
              " 'blazers': 2,\n",
              " 'matamata': 2,\n",
              " 'bmo': 2,\n",
              " 'pershore': 2,\n",
              " 'tayside': 2,\n",
              " 'showrunner': 2,\n",
              " 'kalgoorlie': 2,\n",
              " 'gippsland': 2,\n",
              " 'salford': 2,\n",
              " 'mcgregor': 2,\n",
              " 'lannister': 2,\n",
              " 'rosehill': 2,\n",
              " 'bendigo': 2,\n",
              " 'eastside': 2,\n",
              " 'doug': 2,\n",
              " 'stratford': 2,\n",
              " 'lakefield': 2,\n",
              " 'skydrive': 2,\n",
              " 'colm': 2,\n",
              " 'beg': 2,\n",
              " 'roar': 2,\n",
              " 'seremban': 2,\n",
              " 'plymouth': 2,\n",
              " 'birkenhead': 2,\n",
              " 'penguin': 2,\n",
              " 'cycleways': 2,\n",
              " 'newfoundland': 2,\n",
              " 'desjardins': 2,\n",
              " 'thiruvananthapuram': 2,\n",
              " 'ashurst': 2,\n",
              " 'parramatta': 2,\n",
              " 'bnz': 2,\n",
              " 'crumlin': 2,\n",
              " 'merrion': 2,\n",
              " 'racially': 2,\n",
              " 'winger': 2,\n",
              " 'foxtel': 2,\n",
              " 'townsville': 2,\n",
              " 'yukon': 2,\n",
              " 'otahuhu': 2,\n",
              " 'campervan': 2,\n",
              " 'cardigan': 2,\n",
              " 'victorian': 2,\n",
              " 'tamaki': 2,\n",
              " 'jim': 2,\n",
              " 'fitzpatrick': 2,\n",
              " 'adare': 2,\n",
              " 'leichhardt': 2,\n",
              " 'pacers': 2,\n",
              " 'canaccord': 2,\n",
              " 'rabbitohs': 2,\n",
              " 'connor': 2,\n",
              " 'aff': 2,\n",
              " 'tesco': 2,\n",
              " 'markham': 2,\n",
              " 'ennis': 2,\n",
              " 'kaipara': 2,\n",
              " 'monaro': 2,\n",
              " 'windward': 2,\n",
              " 'aberdeen': 2,\n",
              " 'rhp': 2,\n",
              " 'fonterra': 2,\n",
              " 'clarke': 2,\n",
              " 'app': 2,\n",
              " 'cochrane': 2,\n",
              " 'bai': 2,\n",
              " 'ulster': 2,\n",
              " 'lethbridge': 2,\n",
              " 'kilkenny': 2,\n",
              " 'plunket': 2,\n",
              " 'temuka': 2,\n",
              " 'mullumbimby': 2,\n",
              " 'killiney': 2,\n",
              " 'bagot': 2,\n",
              " 'franklin': 2,\n",
              " 'connolly': 2,\n",
              " 'clontarf': 2,\n",
              " 'kaitaia': 2,\n",
              " 'mclaughlin': 2,\n",
              " 'ngati': 2,\n",
              " 'played': 2,\n",
              " 'sdlp': 2,\n",
              " 'saugeen': 2,\n",
              " 'kelleher': 2,\n",
              " 'wests': 2,\n",
              " 'llanelli': 2,\n",
              " 'nav': 2,\n",
              " 'burnaby': 2,\n",
              " 'yr': 2,\n",
              " 'thurles': 2,\n",
              " 'woden': 2,\n",
              " 'blah': 2,\n",
              " 'portobello': 2,\n",
              " 'oakville': 2,\n",
              " 'netminder': 2,\n",
              " 'walsh': 2,\n",
              " 'goaltender': 2,\n",
              " 'humber': 2,\n",
              " 'woollahra': 2,\n",
              " 'satya': 2,\n",
              " 'toss': 2,\n",
              " 'hmas': 2,\n",
              " 'sabres': 2,\n",
              " 'buderim': 2,\n",
              " 'sidney': 2,\n",
              " 'offaly': 2,\n",
              " 'kamloops': 2,\n",
              " 'geelong': 2,\n",
              " 'redmond': 2,\n",
              " 'jenny': 2,\n",
              " 'bha': 2,\n",
              " 'whitgift': 2,\n",
              " 'ofsaa': 2,\n",
              " 'hogan': 2,\n",
              " 'irfu': 2,\n",
              " 'huntly': 2,\n",
              " 'landcare': 2,\n",
              " 'enda': 2,\n",
              " 'uic': 2,\n",
              " 'childcare': 2,\n",
              " 'nambour': 2,\n",
              " 'bedok': 1,\n",
              " 'haw': 1,\n",
              " 'meadowvale': 1,\n",
              " 'viscount': 1,\n",
              " 'terranova': 1,\n",
              " 'fonds': 1,\n",
              " 'jame': 1,\n",
              " 'cullen': 1,\n",
              " 'clark': 1,\n",
              " 'cahir': 1,\n",
              " 'nauru': 1,\n",
              " 'helensburgh': 1,\n",
              " 'bicentennial': 1,\n",
              " 'coolum': 1,\n",
              " 'catharines': 1,\n",
              " 'w': 1,\n",
              " 'mlse': 1,\n",
              " 'ricciardo': 1,\n",
              " 'xxl': 1,\n",
              " 'gonski': 1,\n",
              " 'unauthorised': 1,\n",
              " 'busselton': 1,\n",
              " 'terenure': 1,\n",
              " 'gainsborough': 1,\n",
              " 'hayes': 1,\n",
              " 'policing': 1,\n",
              " 'unceded': 1,\n",
              " 'sullivan': 1,\n",
              " 'rideau': 1,\n",
              " 'quimper': 1,\n",
              " 'izm': 1,\n",
              " 'umpired': 1,\n",
              " 'mailthanks': 1,\n",
              " 'ao': 1,\n",
              " 'dunkley': 1,\n",
              " 'dolan': 1,\n",
              " 'mic': 1,\n",
              " 'openers': 1,\n",
              " 'aylesford': 1,\n",
              " 'waitaki': 1,\n",
              " 'iwk': 1,\n",
              " 'cheerios': 1,\n",
              " 'lumsden': 1,\n",
              " 'northwestern': 1,\n",
              " 'tamaulipas': 1,\n",
              " 'browns': 1,\n",
              " 'wetaskiwin': 1,\n",
              " 'elkhart': 1,\n",
              " 'msps': 1,\n",
              " 'watson': 1,\n",
              " 'wagga': 1,\n",
              " 'katy': 1,\n",
              " 'corel': 1,\n",
              " 'creedy': 1,\n",
              " 'mclaren': 1,\n",
              " 'rimu': 1,\n",
              " 'lw': 1,\n",
              " 'lyttelton': 1,\n",
              " 'dundalk': 1,\n",
              " 'siders': 1,\n",
              " 'webseries': 1,\n",
              " 'bulldogs': 1,\n",
              " 'reel': 1,\n",
              " 'gatwick': 1,\n",
              " 'taman': 1,\n",
              " 'mattawa': 1,\n",
              " 'finlay': 1,\n",
              " 'veers': 1,\n",
              " 'butchart': 1,\n",
              " 'gsc': 1,\n",
              " 'antarctica': 1,\n",
              " 'canungra': 1,\n",
              " 'travis': 1,\n",
              " 'az': 1,\n",
              " 'drogheda': 1,\n",
              " 'jason': 1,\n",
              " 'brady': 1,\n",
              " 'cfi': 1,\n",
              " 'kms': 1,\n",
              " 'dame': 1,\n",
              " 'tamworth': 1,\n",
              " 'crozier': 1,\n",
              " 'singapore': 1,\n",
              " 'am': 1,\n",
              " 'inuvialuit': 1,\n",
              " 'moose': 1,\n",
              " 'hortons': 1,\n",
              " 'hazelton': 1,\n",
              " 'rory': 1,\n",
              " 'gord': 1,\n",
              " 'ingleburn': 1,\n",
              " 'stockwell': 1,\n",
              " 'moraine': 1,\n",
              " 'knifepoint': 1,\n",
              " 'unmapped': 1,\n",
              " 'willis': 1,\n",
              " 'www': 1,\n",
              " 'parkdale': 1,\n",
              " 'shelford': 1,\n",
              " 'carr': 1,\n",
              " 'mughal': 1,\n",
              " 'harbourfront': 1,\n",
              " 'bletchley': 1,\n",
              " 'heeney': 1,\n",
              " 'wangaratta': 1,\n",
              " 'whincup': 1,\n",
              " 'psychopathy': 1,\n",
              " 'wwd': 1,\n",
              " 'wanneroo': 1,\n",
              " 'welling': 1,\n",
              " 'pacification': 1,\n",
              " 'hahndorf': 1,\n",
              " 'gatineau': 1,\n",
              " 'flack': 1,\n",
              " 'piquet': 1,\n",
              " 'undock': 1,\n",
              " 'caas': 1,\n",
              " 'macarthur': 1,\n",
              " 'gosford': 1,\n",
              " 'mississauga': 1,\n",
              " 'strandings': 1,\n",
              " 'purvis': 1,\n",
              " 'dunne': 1,\n",
              " 'nui': 1,\n",
              " 'aljunied': 1,\n",
              " 'belvoir': 1,\n",
              " 'dunure': 1,\n",
              " 'merivale': 1,\n",
              " 'blanchardstown': 1,\n",
              " 'chignecto': 1,\n",
              " 'delmar': 1,\n",
              " 'canyon': 1,\n",
              " 'werris': 1,\n",
              " 'oates': 1,\n",
              " 'sarah': 1,\n",
              " 'gerrymander': 1,\n",
              " 'honiara': 1,\n",
              " 'rmit': 1,\n",
              " 'clew': 1,\n",
              " 'couchiching': 1,\n",
              " 'ru': 1,\n",
              " 'keat': 1,\n",
              " 'patrice': 1,\n",
              " 'giver': 1,\n",
              " 'thee': 1,\n",
              " 'illawarra': 1,\n",
              " 'marnie': 1,\n",
              " 'wharncliffe': 1,\n",
              " 'ismet': 1,\n",
              " 'townsend': 1,\n",
              " 'carillon': 1,\n",
              " 'tatenda': 1,\n",
              " 'brewster': 1,\n",
              " 'siddle': 1,\n",
              " 'secondaire': 1,\n",
              " 'magilligan': 1,\n",
              " 'loe': 1,\n",
              " 'resupply': 1,\n",
              " 'stratfor': 1,\n",
              " 'doig': 1,\n",
              " 'nypd': 1,\n",
              " 'fyfe': 1,\n",
              " 'montfort': 1,\n",
              " 'herries': 1,\n",
              " 'icg': 1,\n",
              " 'mcgann': 1,\n",
              " 'tsa': 1,\n",
              " 'shri': 1,\n",
              " 'kinsmen': 1,\n",
              " 'steve': 1,\n",
              " 'gumla': 1,\n",
              " 'newbridge': 1,\n",
              " 'hornby': 1,\n",
              " 'whittle': 1,\n",
              " 'yaron': 1,\n",
              " 'wilkie': 1,\n",
              " 'hulu': 1,\n",
              " 'tralee': 1,\n",
              " 'bain': 1,\n",
              " 'ballinteer': 1,\n",
              " 'kanawa': 1,\n",
              " 'wd': 1,\n",
              " 'millennium': 1,\n",
              " 'kenaston': 1,\n",
              " 'brydon': 1,\n",
              " 'eftpos': 1,\n",
              " 'cquniversity': 1,\n",
              " 'muskoka': 1,\n",
              " 'rbi': 1,\n",
              " 'kg': 1,\n",
              " 'robb': 1,\n",
              " 'leafgreen': 1,\n",
              " 'oral': 1,\n",
              " 'moreland': 1,\n",
              " 'zrc': 1,\n",
              " 'kawana': 1,\n",
              " 'saskatoon': 1,\n",
              " 'crockett': 1,\n",
              " 'cbsa': 1,\n",
              " 'cullingworth': 1,\n",
              " 'nea': 1,\n",
              " 'ov': 1,\n",
              " 'ddb': 1,\n",
              " 'twyford': 1,\n",
              " 'tahuna': 1,\n",
              " 'toorak': 1,\n",
              " 'ashgrove': 1,\n",
              " 'kootenay': 1,\n",
              " 'festiva': 1,\n",
              " 'molesworth': 1,\n",
              " 'benetti': 1,\n",
              " 'kawartha': 1,\n",
              " 'killorglin': 1,\n",
              " 'darlinghurst': 1,\n",
              " 'berkshire': 1,\n",
              " 'nipissing': 1,\n",
              " 'coniston': 1,\n",
              " 'debbie': 1,\n",
              " 'kiwifruit': 1,\n",
              " 'overlea': 1,\n",
              " 'tywyn': 1,\n",
              " 'vaucluse': 1,\n",
              " 'mccormick': 1,\n",
              " 'midget': 1,\n",
              " 'salvos': 1,\n",
              " 'bundaberg': 1,\n",
              " 'manpower': 1,\n",
              " 'sji': 1,\n",
              " 'gunz': 1,\n",
              " 'plains': 1,\n",
              " 'castanet': 1,\n",
              " 'wynyard': 1,\n",
              " 'kowalski': 1,\n",
              " 'altizer': 1,\n",
              " 'landholder': 1,\n",
              " 'fastnet': 1,\n",
              " 'mackin': 1,\n",
              " 'attentio': 1,\n",
              " 'wetherspoons': 1,\n",
              " 'ltv': 1,\n",
              " 'taronga': 1,\n",
              " 'webb': 1,\n",
              " 'jake': 1,\n",
              " 'breathalyzer': 1,\n",
              " 'nuyorican': 1,\n",
              " 'snipers': 1,\n",
              " 'farrer': 1,\n",
              " 'vanstone': 1,\n",
              " 'ncai': 1,\n",
              " 'mcwilliams': 1,\n",
              " 'anonymously': 1,\n",
              " 'mcloughlin': 1,\n",
              " 'lacasse': 1,\n",
              " 'brookfield': 1,\n",
              " 'cranford': 1,\n",
              " 'gothenburg': 1,\n",
              " 'finglas': 1,\n",
              " 'dundas': 1,\n",
              " 'oilers': 1,\n",
              " 'rmbs': 1,\n",
              " 'eramosa': 1,\n",
              " 'kivu': 1,\n",
              " 'middlemarch': 1,\n",
              " 'oakley': 1,\n",
              " 'qld': 1,\n",
              " 'sugar': 1,\n",
              " 'sector': 1,\n",
              " 'bayne': 1,\n",
              " 'qu': 1,\n",
              " 'salutes': 1,\n",
              " 'falle': 1,\n",
              " 'welland': 1,\n",
              " 'nunavut': 1,\n",
              " 'emirates': 1,\n",
              " 'taproom': 1,\n",
              " 'tui': 1,\n",
              " 'maggie': 1,\n",
              " 'porirua': 1,\n",
              " 'gamepro': 1,\n",
              " 'tokoroa': 1,\n",
              " 'bowmanville': 1,\n",
              " 'cfl': 1,\n",
              " 'firstgroup': 1,\n",
              " 'brompton': 1,\n",
              " 'jarque': 1,\n",
              " 'ponca': 1,\n",
              " 'baggot': 1,\n",
              " 'roebuck': 1,\n",
              " 'northants': 1,\n",
              " 'hougang': 1,\n",
              " 'gardiner': 1,\n",
              " 'ravenhill': 1,\n",
              " 'kaikoura': 1,\n",
              " 'wodonga': 1,\n",
              " 'corden': 1,\n",
              " 'srt': 1,\n",
              " 'rajeshwari': 1,\n",
              " 'bevis': 1,\n",
              " 'anglesea': 1,\n",
              " 'glogovac': 1,\n",
              " 'mum': 1,\n",
              " 'glenlyon': 1,\n",
              " 'mount': 1,\n",
              " 'centrelink': 1,\n",
              " 'carisbrook': 1,\n",
              " 'laois': 1,\n",
              " 'rds': 1,\n",
              " 'danforth': 1,\n",
              " 'puraskaram': 1,\n",
              " 'barker': 1,\n",
              " 'matthew': 1,\n",
              " 'boga': 1,\n",
              " 'bresler': 1,\n",
              " 'np': 1,\n",
              " 'ako': 1,\n",
              " 'skelly': 1,\n",
              " 'dyke': 1,\n",
              " 'hs': 1,\n",
              " 'hants': 1,\n",
              " 'clutha': 1,\n",
              " 'chek': 1,\n",
              " 'mclennan': 1,\n",
              " 'ladd': 1,\n",
              " 'tiriti': 1,\n",
              " 'breakers': 1,\n",
              " 'gamesmaster': 1,\n",
              " 'brexit': 1,\n",
              " 'gsmarena': 1,\n",
              " 'quip': 1,\n",
              " 'skadar': 1,\n",
              " 'fc': 1,\n",
              " 'petone': 1,\n",
              " 'dhx': 1,\n",
              " 'northumberland': 1,\n",
              " 'cris': 1,\n",
              " 'brockville': 1,\n",
              " 'nrma': 1,\n",
              " 'remeber': 1,\n",
              " 'francois': 1,\n",
              " 'bunnings': 1,\n",
              " 'braden': 1,\n",
              " 'mckeown': 1,\n",
              " 'farian': 1,\n",
              " 'amphion': 1,\n",
              " 'blair': 1,\n",
              " 'ambleside': 1,\n",
              " 'maling': 1,\n",
              " 'ua': 1,\n",
              " 'madras': 1,\n",
              " 'ilkeston': 1,\n",
              " 'org': 1,\n",
              " 'curragh': 1,\n",
              " 'sobeys': 1,\n",
              " 'mokbel': 1,\n",
              " 'arran': 1,\n",
              " 'jervis': 1,\n",
              " 'ints': 1,\n",
              " 'rafter': 1,\n",
              " 'garth': 1,\n",
              " 'comox': 1,\n",
              " 'maison': 1,\n",
              " 'canning': 1,\n",
              " 'eastwick': 1,\n",
              " 'williamstown': 1,\n",
              " 'submission': 1,\n",
              " 'tiernan': 1,\n",
              " 'holdens': 1,\n",
              " 'annually': 1,\n",
              " 'dunmore': 1,\n",
              " 'cookstown': 1,\n",
              " 'knowle': 1,\n",
              " 'timmins': 1,\n",
              " 'ft': 1,\n",
              " 'paradis': 1,\n",
              " 'redcar': 1,\n",
              " 'evanston': 1,\n",
              " 'wigram': 1,\n",
              " 'panjang': 1,\n",
              " 'shankill': 1,\n",
              " 'cida': 1,\n",
              " 'mios': 1,\n",
              " 'valtteri': 1,\n",
              " 'bertuzzi': 1,\n",
              " 'horan': 1,\n",
              " 'carpentaria': 1,\n",
              " 'rexdale': 1,\n",
              " 'citylink': 1,\n",
              " 'reilly': 1,\n",
              " 'skittles': 1,\n",
              " 'upendra': 1,\n",
              " 'glencoe': 1,\n",
              " 'castlemaine': 1,\n",
              " 'sidewall': 1,\n",
              " 'parihaka': 1,\n",
              " 'suresh': 1,\n",
              " 'cutler': 1,\n",
              " 'iga': 1,\n",
              " 'travelled': 1,\n",
              " 'christel': 1,\n",
              " 'kercher': 1,\n",
              " 'novikova': 1,\n",
              " 'yester': 1,\n",
              " 'telford': 1,\n",
              " 'ultimo': 1,\n",
              " 'kitimat': 1,\n",
              " 'sauv': 1,\n",
              " 'ainsley': 1,\n",
              " 'northeast': 1,\n",
              " 'smi': 1,\n",
              " 'venturas': 1,\n",
              " 'honouring': 1,\n",
              " 'luber': 1,\n",
              " 'onslow': 1,\n",
              " 'gosling': 1,\n",
              " 'wabo': 1,\n",
              " 'charlo': 1,\n",
              " 'wurm': 1,\n",
              " 'payette': 1,\n",
              " 'pats': 1,\n",
              " 'armadale': 1,\n",
              " 'territorials': 1,\n",
              " 'blemish': 1,\n",
              " 'madeley': 1,\n",
              " 'fleming': 1,\n",
              " 'luv': 1,\n",
              " 'devonian': 1,\n",
              " 'hawai': 1,\n",
              " 'pur': 1,\n",
              " 'ashfield': 1,\n",
              " 'tsawwassen': 1,\n",
              " 'katoomba': 1,\n",
              " 'oteil': 1,\n",
              " 'stemmle': 1,\n",
              " 'sheens': 1,\n",
              " 'pakuranga': 1,\n",
              " 'nwt': 1,\n",
              " 'eamon': 1,\n",
              " 'smrt': 1,\n",
              " 'atherton': 1,\n",
              " 'lezard': 1,\n",
              " 'ymc': 1,\n",
              " 'umno': 1,\n",
              " 'winterfell': 1,\n",
              " 'rsn': 1,\n",
              " 'sveriges': 1,\n",
              " 'labrador': 1,\n",
              " 'prendergast': 1,\n",
              " 'psu': 1,\n",
              " 'kitikmeot': 1,\n",
              " 'starcraft': 1,\n",
              " 'machale': 1,\n",
              " 'alouettes': 1,\n",
              " 'jersey': 1,\n",
              " 'grandfathering': 1,\n",
              " 'kestrel': 1,\n",
              " 'burnett': 1,\n",
              " 'nus': 1,\n",
              " 'natixis': 1,\n",
              " 'kemptville': 1,\n",
              " 'mosman': 1,\n",
              " 'teatro': 1,\n",
              " 'bunyip': 1,\n",
              " 'burley': 1,\n",
              " 'willow': 1,\n",
              " 'to': 1,\n",
              " 'athabasca': 1,\n",
              " 'cupe': 1,\n",
              " 'ics': 1,\n",
              " 'joystiq': 1,\n",
              " 'rts': 1,\n",
              " 'grainger': 1,\n",
              " 'colborne': 1,\n",
              " 'kentville': 1,\n",
              " 'hawker': 1,\n",
              " 'pliskova': 1,\n",
              " 'contributed': 1,\n",
              " 'mccrae': 1,\n",
              " 'erindale': 1,\n",
              " 'shield': 1,\n",
              " 'yindi': 1,\n",
              " 'puke': 1,\n",
              " 'mullally': 1,\n",
              " 'soundsystem': 1,\n",
              " 'sali': 1,\n",
              " 'rail': 1,\n",
              " 'tories': 1,\n",
              " 'crushers': 1,\n",
              " 'macduff': 1,\n",
              " 'airdrie': 1,\n",
              " 'yorkdale': 1,\n",
              " 'yap': 1,\n",
              " 'williston': 1,\n",
              " 'cowra': 1,\n",
              " 'nesbitt': 1,\n",
              " 'cheers': 1,\n",
              " 'metrotown': 1,\n",
              " 'cahill': 1,\n",
              " 'gcs': 1,\n",
              " 'uktv': 1,\n",
              " 'winch': 1,\n",
              " 'baltinglass': 1,\n",
              " 'douro': 1,\n",
              " 'mulga': 1,\n",
              " 'huntingdon': 1,\n",
              " 'histoire': 1,\n",
              " 'gigawatts': 1,\n",
              " 'northbridge': 1,\n",
              " 'mazie': 1,\n",
              " 'prorogation': 1,\n",
              " 'mcgrath': 1,\n",
              " 'drury': 1,\n",
              " 'triumphed': 1,\n",
              " 'rathore': 1,\n",
              " 'tq': 1,\n",
              " 'renewables': 1,\n",
              " 'matildas': 1,\n",
              " 'ducati': 1,\n",
              " 'fermoy': 1,\n",
              " 'pearcey': 1,\n",
              " 'rawkus': 1,\n",
              " 'grady': 1,\n",
              " 'mccarthy': 1,\n",
              " 'sione': 1,\n",
              " 'cheer': 1,\n",
              " 'vidarbha': 1,\n",
              " 'ipcc': 1,\n",
              " 'afield': 1,\n",
              " 'collaroy': 1,\n",
              " 'costello': 1,\n",
              " 'moresby': 1,\n",
              " 'liberalize': 1,\n",
              " 'mallya': 1,\n",
              " 'utilise': 1,\n",
              " 'leaside': 1,\n",
              " 'borroloola': 1,\n",
              " 'deakin': 1,\n",
              " 'werribee': 1,\n",
              " 'taniwha': 1,\n",
              " 'stirling': 1,\n",
              " 'seneca': 1,\n",
              " 'qut': 1,\n",
              " 'brera': 1,\n",
              " 'aird': 1,\n",
              " 'whangarei': 1,\n",
              " 'jacobs': 1,\n",
              " 'lillooet': 1,\n",
              " 'grafton': 1,\n",
              " 'lambert': 1,\n",
              " 'bridgend': 1,\n",
              " 'brantford': 1,\n",
              " 'shaker': 1,\n",
              " 'bylaw': 1,\n",
              " 'confident': 1,\n",
              " 'frontline': 1,\n",
              " 'newsday': 1,\n",
              " 'dgp': 1,\n",
              " 'lake': 1,\n",
              " 'walkley': 1,\n",
              " 'eaves': 1,\n",
              " 'danvers': 1,\n",
              " 'macdonagh': 1,\n",
              " 'spake': 1,\n",
              " 'maclaren': 1,\n",
              " 'nitv': 1,\n",
              " 'postmaster': 1,\n",
              " 'lia': 1,\n",
              " 'dvla': 1,\n",
              " 'cowichan': 1,\n",
              " 'snowy': 1,\n",
              " 'kospi': 1,\n",
              " 'waimate': 1,\n",
              " 'whitchurch': 1,\n",
              " 'heartily': 1,\n",
              " 'sweetheart': 1,\n",
              " 'broncos': 1,\n",
              " 'thomond': 1,\n",
              " 'hms': 1,\n",
              " 'sheppard': 1,\n",
              " 'bray': 1,\n",
              " 'sheree': 1,\n",
              " 'interbrand': 1,\n",
              " 'jordanstown': 1,\n",
              " 'mayo': 1,\n",
              " 'hickock': 1,\n",
              " 'maunganui': 1,\n",
              " 'aidan': 1,\n",
              " 'pictou': 1,\n",
              " 'elephant': 1,\n",
              " 'aborigines': 1,\n",
              " 'icas': 1,\n",
              " 'hines': 1,\n",
              " 'cabernet': 1,\n",
              " 'scarface': 1,\n",
              " 'technica': 1,\n",
              " 'renegades': 1,\n",
              " 'hokianga': 1,\n",
              " 'from': 1,\n",
              " 'rta': 1,\n",
              " 'amherst': 1,\n",
              " 'newsreader': 1,\n",
              " 'southwest': 1,\n",
              " 'malthouse': 1,\n",
              " 'celbridge': 1,\n",
              " 'duncan': 1,\n",
              " 'wash': 1,\n",
              " 'greenway': 1,\n",
              " 'shortland': 1,\n",
              " 'crosbie': 1,\n",
              " 'chargers': 1,\n",
              " 'brunswick': 1,\n",
              " 'fibres': 1,\n",
              " 'carmichael': 1,\n",
              " 'bimini': 1,\n",
              " 'blackthorn': 1,\n",
              " 'transpo': 1,\n",
              " 'vianney': 1,\n",
              " 'orioles': 1,\n",
              " 'mountie': 1,\n",
              " 'darach': 1,\n",
              " 'kempsey': 1,\n",
              " 'ksi': 1,\n",
              " 'spokeswoman': 1,\n",
              " 'enes': 1,\n",
              " 'snigger': 1,\n",
              " 'villiers': 1,\n",
              " 'honohan': 1,\n",
              " 'oregon': 1,\n",
              " 'xe': 1,\n",
              " 'eviscerate': 1,\n",
              " 'hng': 1,\n",
              " 'ceallaigh': 1,\n",
              " 'obstetricians': 1,\n",
              " 'antibiotics': 1,\n",
              " 'whitehall': 1,\n",
              " 'alcock': 1,\n",
              " 'cockatoo': 1,\n",
              " 'exeter': 1,\n",
              " 'creswell': 1,\n",
              " 'geothermal': 1,\n",
              " 'bc': 1,\n",
              " 'coldstream': 1,\n",
              " 'ansell': 1,\n",
              " 'grace': 1,\n",
              " 'rosedale': 1,\n",
              " 'morphett': 1,\n",
              " 'carrum': 1,\n",
              " 'thornhill': 1,\n",
              " 'defi': 1,\n",
              " 'keppel': 1,\n",
              " 'engine': 1,\n",
              " 'corduff': 1,\n",
              " 'outram': 1,\n",
              " 'collie': 1,\n",
              " 'paperbark': 1,\n",
              " 'fentanyl': 1,\n",
              " 'halawa': 1,\n",
              " 'frost': 1,\n",
              " 'sits': 1,\n",
              " 'elan': 1,\n",
              " 'ebury': 1,\n",
              " 'pl': 1,\n",
              " 'karjala': 1,\n",
              " 'tararua': 1,\n",
              " 'wetherspoon': 1,\n",
              " 'waimakariri': 1,\n",
              " 'norseman': 1,\n",
              " 'chrysotile': 1,\n",
              " 'csiro': 1,\n",
              " 'meteorology': 1,\n",
              " 'forfar': 1,\n",
              " 'plumbago': 1,\n",
              " 'fredericton': 1,\n",
              " 'dunnes': 1,\n",
              " 'beenleigh': 1,\n",
              " 'dalrymple': 1,\n",
              " 'nikki': 1,\n",
              " 'callum': 1,\n",
              " 'hapu': 1,\n",
              " 'harlequins': 1,\n",
              " ...}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 95
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KNyfWHTEzME8",
        "outputId": "b46646cc-664e-4475-e111-e4e1ca6d6717"
      },
      "source": [
        "dict(sorted(nonpeace_dict.items(), key=lambda item: item[1], reverse = True))"
      ],
      "execution_count": 96,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'absa': 39,\n",
              " 'enugu': 36,\n",
              " 'naira': 35,\n",
              " 'mombasa': 25,\n",
              " 'tehreek': 18,\n",
              " 'awami': 18,\n",
              " 'sbp': 17,\n",
              " 'odm': 15,\n",
              " 'vodacom': 14,\n",
              " 'nakuru': 14,\n",
              " 'inec': 14,\n",
              " 'muhammadu': 14,\n",
              " 'mtn': 13,\n",
              " 'multan': 12,\n",
              " 'moi': 12,\n",
              " 'anambra': 12,\n",
              " 'upazila': 12,\n",
              " 'shilling': 11,\n",
              " 'thisday': 11,\n",
              " 'stellenbosch': 11,\n",
              " 'nse': 11,\n",
              " 'kenyatta': 11,\n",
              " 'limpopo': 11,\n",
              " 'nigerian': 10,\n",
              " 'iqbal': 10,\n",
              " 'incite': 10,\n",
              " 'iebc': 9,\n",
              " 'ispr': 9,\n",
              " 'bangla': 9,\n",
              " 'idps': 9,\n",
              " 'wits': 9,\n",
              " 'retd': 9,\n",
              " 'kaduna': 9,\n",
              " 'authenticates': 8,\n",
              " 'ekiti': 8,\n",
              " 'abia': 8,\n",
              " 'kse': 8,\n",
              " 'faisalabad': 8,\n",
              " 'kisumu': 8,\n",
              " 'jinnah': 8,\n",
              " 'sialkot': 8,\n",
              " 'machakos': 8,\n",
              " 'sukkur': 8,\n",
              " 'balochistan': 8,\n",
              " 'bangabandhu': 8,\n",
              " 'benazir': 8,\n",
              " 'awolowo': 8,\n",
              " 'kiambu': 8,\n",
              " 'tehsil': 7,\n",
              " 'kra': 7,\n",
              " 'igp': 7,\n",
              " 'sargodha': 7,\n",
              " 'sh': 7,\n",
              " 'eldoret': 7,\n",
              " 'saddar': 7,\n",
              " 'katsina': 7,\n",
              " 'haq': 6,\n",
              " 'raila': 6,\n",
              " 'kzn': 6,\n",
              " 'kcb': 6,\n",
              " 'mirpur': 6,\n",
              " 'wapda': 6,\n",
              " 'syed': 6,\n",
              " 'obasanjo': 6,\n",
              " 'bhutto': 6,\n",
              " 'quetta': 6,\n",
              " 'hossain': 6,\n",
              " 'saps': 6,\n",
              " 'ondo': 6,\n",
              " 'pokot': 5,\n",
              " 'mahlangu': 5,\n",
              " 'eskom': 5,\n",
              " 'adamawa': 5,\n",
              " 'qasim': 5,\n",
              " 'orangi': 5,\n",
              " 'rustenburg': 5,\n",
              " 'alhaji': 5,\n",
              " 'sapa': 5,\n",
              " 'warri': 5,\n",
              " 'nyanza': 5,\n",
              " 'gilani': 5,\n",
              " 'uitenhage': 5,\n",
              " 'abeokuta': 5,\n",
              " 'qaumi': 5,\n",
              " 'pietermaritzburg': 5,\n",
              " 'goodluck': 5,\n",
              " 'educationist': 5,\n",
              " 'rajshahi': 5,\n",
              " 'ruto': 4,\n",
              " 'abubakar': 4,\n",
              " 'bauchi': 4,\n",
              " 'hydel': 4,\n",
              " 'secp': 4,\n",
              " 'uba': 4,\n",
              " 'ahmadu': 4,\n",
              " 'allama': 4,\n",
              " 'lago': 4,\n",
              " 'insaf': 4,\n",
              " 'ogun': 4,\n",
              " 'islami': 4,\n",
              " 'efcc': 4,\n",
              " 'paarl': 4,\n",
              " 'frsc': 4,\n",
              " 'bahawalpur': 4,\n",
              " 'tanesco': 4,\n",
              " 'sokoto': 4,\n",
              " 'garissa': 4,\n",
              " 'ikeja': 4,\n",
              " 'tanganyika': 4,\n",
              " 'ebonyi': 4,\n",
              " 'naivasha': 4,\n",
              " 'muttahida': 4,\n",
              " 'charsadda': 4,\n",
              " 'kasarani': 4,\n",
              " 'bayelsa': 4,\n",
              " 'owerri': 4,\n",
              " 'cele': 3,\n",
              " 'chaudhry': 3,\n",
              " 'qazi': 3,\n",
              " 'motlanthe': 3,\n",
              " 'ikoyi': 3,\n",
              " 'cbk': 3,\n",
              " 'ksh': 3,\n",
              " 'muhimbili': 3,\n",
              " 'mirpurkhas': 3,\n",
              " 'awan': 3,\n",
              " 'khulna': 3,\n",
              " 'transnet': 3,\n",
              " 'leopards': 3,\n",
              " 'tambo': 3,\n",
              " 'dinajpur': 3,\n",
              " 'ikorodu': 3,\n",
              " 'ifp': 3,\n",
              " 'mwanza': 3,\n",
              " 'dfid': 3,\n",
              " 'gilgit': 3,\n",
              " 'pulwama': 3,\n",
              " 'onitsha': 3,\n",
              " 'lament': 3,\n",
              " 'nullah': 3,\n",
              " 'nyeri': 3,\n",
              " 'murtala': 3,\n",
              " 'thika': 3,\n",
              " 'mujib': 3,\n",
              " 'akure': 3,\n",
              " 'tshwane': 3,\n",
              " 'sahib': 3,\n",
              " 'gulshan': 3,\n",
              " 'kakamega': 3,\n",
              " 'potchefstroom': 3,\n",
              " 'junaid': 3,\n",
              " 'stakeholders': 3,\n",
              " 'ilorin': 3,\n",
              " 'chhatra': 3,\n",
              " 'uct': 3,\n",
              " 'ngugi': 3,\n",
              " 'kitui': 3,\n",
              " 'kebbi': 3,\n",
              " 'witwatersrand': 3,\n",
              " 'kisii': 3,\n",
              " 'kikuyu': 3,\n",
              " 'ptv': 3,\n",
              " 'qureshi': 3,\n",
              " 'borno': 3,\n",
              " 'jang': 3,\n",
              " 'khattak': 3,\n",
              " 'nasarawa': 3,\n",
              " 'urdu': 3,\n",
              " 'younis': 2,\n",
              " 'afrikaans': 2,\n",
              " 'isiolo': 2,\n",
              " 'sahiwal': 2,\n",
              " 'eleme': 2,\n",
              " 'rtd': 2,\n",
              " 'mymensingh': 2,\n",
              " 'gwadar': 2,\n",
              " 'hackspaces': 2,\n",
              " 'anf': 2,\n",
              " 'comsats': 2,\n",
              " 'airtel': 2,\n",
              " 'nandi': 2,\n",
              " 'umaru': 2,\n",
              " 'dhanmondi': 2,\n",
              " 'rahim': 2,\n",
              " 'bharatiya': 2,\n",
              " 'upliftment': 2,\n",
              " 'kogi': 2,\n",
              " 'twin': 2,\n",
              " 'westlands': 2,\n",
              " 'netcare': 2,\n",
              " 'gatundu': 2,\n",
              " 'bazar': 2,\n",
              " 'shahbaz': 2,\n",
              " 'afzal': 2,\n",
              " 'sundowns': 2,\n",
              " 'khawar': 2,\n",
              " 'lodwar': 2,\n",
              " 'revolutionise': 2,\n",
              " 'indira': 2,\n",
              " 'pia': 2,\n",
              " 'mandera': 2,\n",
              " 'kwara': 2,\n",
              " 'athi': 2,\n",
              " 'ary': 2,\n",
              " 'ur': 2,\n",
              " 'kwale': 2,\n",
              " 'dlamini': 2,\n",
              " 'gujranwala': 2,\n",
              " 'shakeel': 2,\n",
              " 'zwane': 2,\n",
              " 'begum': 2,\n",
              " 'siddique': 2,\n",
              " 'fkf': 2,\n",
              " 'icc': 2,\n",
              " 'counterpart': 2,\n",
              " 'zafar': 2,\n",
              " 'jhelum': 2,\n",
              " 'edhi': 2,\n",
              " 'btrc': 2,\n",
              " 'sugarcane': 2,\n",
              " 'src': 2,\n",
              " 'pervaiz': 2,\n",
              " 'combin': 2,\n",
              " 'rands': 2,\n",
              " 'motorable': 2,\n",
              " 'khuda': 2,\n",
              " 'kandy': 2,\n",
              " 'abiola': 2,\n",
              " 'murree': 2,\n",
              " 'mehran': 2,\n",
              " 'balewa': 2,\n",
              " 'habib': 2,\n",
              " 'mthatha': 2,\n",
              " 'taka': 2,\n",
              " 'nfd': 2,\n",
              " 'poverty': 2,\n",
              " 'postcolonial': 2,\n",
              " 'kna': 2,\n",
              " 'meghna': 2,\n",
              " 'waqar': 2,\n",
              " 'alamgir': 2,\n",
              " 'bilawal': 2,\n",
              " 'kaa': 2,\n",
              " 'kq': 2,\n",
              " 'gulberg': 2,\n",
              " 'musoma': 2,\n",
              " 'mizengo': 2,\n",
              " 'malakand': 2,\n",
              " 'mwangi': 2,\n",
              " 'hazara': 2,\n",
              " 'maina': 2,\n",
              " 'nuhu': 2,\n",
              " 'suparco': 2,\n",
              " 'technikon': 2,\n",
              " 'cantt': 2,\n",
              " 'rohingya': 2,\n",
              " 'maritzburg': 2,\n",
              " 'chowk': 2,\n",
              " 'klerk': 2,\n",
              " 'cti': 2,\n",
              " 'magistrate': 2,\n",
              " 'sonargaon': 2,\n",
              " 'kasur': 2,\n",
              " 'boksburg': 2,\n",
              " 'tehrik': 2,\n",
              " 'dpo': 2,\n",
              " 'waterkloof': 2,\n",
              " 'osun': 2,\n",
              " 'naqvi': 2,\n",
              " 'visagie': 2,\n",
              " 'artiste': 2,\n",
              " 'sensitization': 2,\n",
              " 'themba': 2,\n",
              " 'lekki': 2,\n",
              " 'langata': 2,\n",
              " 'gurdaspur': 2,\n",
              " 'maternal': 2,\n",
              " 'marakwet': 2,\n",
              " 'wfp': 2,\n",
              " 'umuahia': 2,\n",
              " 'mdantsane': 2,\n",
              " 'krcs': 2,\n",
              " 'ojo': 2,\n",
              " 'yakubu': 2,\n",
              " 'turkana': 2,\n",
              " 'shaukat': 2,\n",
              " 'korangi': 2,\n",
              " 'midrand': 2,\n",
              " 'kplc': 2,\n",
              " 'kanu': 2,\n",
              " 'ndlea': 2,\n",
              " 'ijaws': 2,\n",
              " 'pinetown': 2,\n",
              " 'ancyl': 2,\n",
              " 'nowshera': 2,\n",
              " 'bukhari': 2,\n",
              " 'sahaba': 1,\n",
              " 'bindra': 1,\n",
              " 'pukke': 1,\n",
              " 'doonside': 1,\n",
              " 'inam': 1,\n",
              " 'ehsan': 1,\n",
              " 'amjad': 1,\n",
              " 'granton': 1,\n",
              " 'mungai': 1,\n",
              " 'kajiado': 1,\n",
              " 'aitchison': 1,\n",
              " 'pervez': 1,\n",
              " 'atas': 1,\n",
              " 'bafokeng': 1,\n",
              " 'madaraka': 1,\n",
              " 'jirga': 1,\n",
              " 'temeke': 1,\n",
              " 'ps': 1,\n",
              " 'nazareth': 1,\n",
              " 'coetzer': 1,\n",
              " 'piped': 1,\n",
              " 'dagga': 1,\n",
              " 'tinubu': 1,\n",
              " 'ghulam': 1,\n",
              " 'maldivian': 1,\n",
              " 'alhassan': 1,\n",
              " 'ryneveld': 1,\n",
              " 'teargas': 1,\n",
              " 'whilst': 1,\n",
              " 'emohua': 1,\n",
              " 'schoeman': 1,\n",
              " 'kloof': 1,\n",
              " 'atn': 1,\n",
              " 'taiba': 1,\n",
              " 'idc': 1,\n",
              " 'liliesleaf': 1,\n",
              " 'lambast': 1,\n",
              " 'cct': 1,\n",
              " 'fse': 1,\n",
              " 'evesham': 1,\n",
              " 'nais': 1,\n",
              " 'siliguri': 1,\n",
              " 'aur': 1,\n",
              " 'slap': 1,\n",
              " 'lsk': 1,\n",
              " 'fakhruddin': 1,\n",
              " 'trance': 1,\n",
              " 'yousif': 1,\n",
              " 'mudassar': 1,\n",
              " 'lazaro': 1,\n",
              " 'fazal': 1,\n",
              " 'caretaker': 1,\n",
              " 'unctad': 1,\n",
              " 'stanbic': 1,\n",
              " 'complicit': 1,\n",
              " 'fbcci': 1,\n",
              " 'khandaker': 1,\n",
              " 'lestrade': 1,\n",
              " 'overlie': 1,\n",
              " 'jaji': 1,\n",
              " 'dk': 1,\n",
              " 'murtaza': 1,\n",
              " 'lalmonirhat': 1,\n",
              " 'kemboi': 1,\n",
              " 'hitmaker': 1,\n",
              " 'ifpi': 1,\n",
              " 'chieftain': 1,\n",
              " 'sona': 1,\n",
              " 'serap': 1,\n",
              " 'jamia': 1,\n",
              " 'kagwe': 1,\n",
              " 'bgmea': 1,\n",
              " 'elahi': 1,\n",
              " 'roodepoort': 1,\n",
              " 'wpa': 1,\n",
              " 'nedlac': 1,\n",
              " 'udm': 1,\n",
              " 'baf': 1,\n",
              " 'teboho': 1,\n",
              " 'mdgs': 1,\n",
              " 'ndlovu': 1,\n",
              " 'mujahid': 1,\n",
              " 'sandf': 1,\n",
              " 'borde': 1,\n",
              " 'aliyu': 1,\n",
              " 'consulate': 1,\n",
              " 'icta': 1,\n",
              " 'moni': 1,\n",
              " 'migori': 1,\n",
              " 'oba': 1,\n",
              " 'asaduzzaman': 1,\n",
              " 'ishola': 1,\n",
              " 'kalonzo': 1,\n",
              " 'kakar': 1,\n",
              " 'malir': 1,\n",
              " 'martell': 1,\n",
              " 'afric': 1,\n",
              " 'congres': 1,\n",
              " 'policeman': 1,\n",
              " 'link': 1,\n",
              " 'khanewal': 1,\n",
              " 'ladysmith': 1,\n",
              " 'umar': 1,\n",
              " 'mbhazima': 1,\n",
              " 'bunge': 1,\n",
              " 'eran': 1,\n",
              " 'kini': 1,\n",
              " 'sipho': 1,\n",
              " 'allahabad': 1,\n",
              " 'sandton': 1,\n",
              " 'deputize': 1,\n",
              " 'kganyago': 1,\n",
              " 'kagiso': 1,\n",
              " 'malawi': 1,\n",
              " 'btw': 1,\n",
              " 'islam': 1,\n",
              " 'eurasia': 1,\n",
              " 'bukola': 1,\n",
              " 'ibb': 1,\n",
              " 'rebecca': 1,\n",
              " 'makhado': 1,\n",
              " 'ekurhuleni': 1,\n",
              " 'zabi': 1,\n",
              " 'segun': 1,\n",
              " 'tahir': 1,\n",
              " 'amref': 1,\n",
              " 'chowdhury': 1,\n",
              " 'kraal': 1,\n",
              " 'torched': 1,\n",
              " 'waziri': 1,\n",
              " 'intones': 1,\n",
              " 'arv': 1,\n",
              " 'lieutenant': 1,\n",
              " 'metrorail': 1,\n",
              " 'hajiya': 1,\n",
              " 'sangeet': 1,\n",
              " 'rowlett': 1,\n",
              " 'salva': 1,\n",
              " 'fons': 1,\n",
              " 'scarp': 1,\n",
              " 'alfa': 1,\n",
              " 'lastly': 1,\n",
              " 'maseko': 1,\n",
              " 'uniondale': 1,\n",
              " 'maithripala': 1,\n",
              " 'rab': 1,\n",
              " 'enyimba': 1,\n",
              " 'bouillon': 1,\n",
              " 'hazrat': 1,\n",
              " 'khakwani': 1,\n",
              " 'rafi': 1,\n",
              " 'dodoma': 1,\n",
              " 'musyoka': 1,\n",
              " 'dco': 1,\n",
              " 'charonne': 1,\n",
              " 'fani': 1,\n",
              " 'badar': 1,\n",
              " 'nir': 1,\n",
              " 'daystar': 1,\n",
              " 'fak': 1,\n",
              " 'imagina': 1,\n",
              " 'omu': 1,\n",
              " 'dhc': 1,\n",
              " 'mokaba': 1,\n",
              " 'tcl': 1,\n",
              " 'hoppa': 1,\n",
              " 'galt': 1,\n",
              " 'radebe': 1,\n",
              " 'geingob': 1,\n",
              " 'durbar': 1,\n",
              " 'ferozepur': 1,\n",
              " 'bayo': 1,\n",
              " 'tanvir': 1,\n",
              " 'aurat': 1,\n",
              " 'holidaymaker': 1,\n",
              " 'samuelsson': 1,\n",
              " 'matete': 1,\n",
              " 'afrika': 1,\n",
              " 'hadith': 1,\n",
              " 'nazir': 1,\n",
              " 'tlp': 1,\n",
              " 'alao': 1,\n",
              " 'kisutu': 1,\n",
              " 'bahn': 1,\n",
              " 'woolen': 1,\n",
              " 'absar': 1,\n",
              " 'addis': 1,\n",
              " 'popularly': 1,\n",
              " 'kotri': 1,\n",
              " 'irvington': 1,\n",
              " 'nyasa': 1,\n",
              " 'dev': 1,\n",
              " 'autocorrect': 1,\n",
              " 'bahauddin': 1,\n",
              " 'nas': 1,\n",
              " 'kanal': 1,\n",
              " 'igda': 1,\n",
              " 'yobe': 1,\n",
              " 'ncp': 1,\n",
              " 'swahili': 1,\n",
              " 'fulltime': 1,\n",
              " 'reh': 1,\n",
              " 'khawaja': 1,\n",
              " 'devolved': 1,\n",
              " 'kalan': 1,\n",
              " 'mixtapes': 1,\n",
              " 'scrabble': 1,\n",
              " 'awka': 1,\n",
              " 'tarrant': 1,\n",
              " 'srk': 1,\n",
              " 'chipolopolo': 1,\n",
              " 'zaria': 1,\n",
              " 'tusker': 1,\n",
              " 'etisalat': 1,\n",
              " 'mopani': 1,\n",
              " 'yalom': 1,\n",
              " 'nanda': 1,\n",
              " 'muli': 1,\n",
              " 'mobo': 1,\n",
              " 'fourteenth': 1,\n",
              " 'ube': 1,\n",
              " 'jakes': 1,\n",
              " 'shanty': 1,\n",
              " 'baltistan': 1,\n",
              " 'subaltern': 1,\n",
              " 'angwenyi': 1,\n",
              " 'akello': 1,\n",
              " 'abit': 1,\n",
              " 'nayeem': 1,\n",
              " 'hamis': 1,\n",
              " 'ele': 1,\n",
              " 'zaman': 1,\n",
              " 'jamaat': 1,\n",
              " 'psych': 1,\n",
              " 'csir': 1,\n",
              " 'rahman': 1,\n",
              " 'flyover': 1,\n",
              " 'godwin': 1,\n",
              " 'steyn': 1,\n",
              " 'ksa': 1,\n",
              " 'ferozabad': 1,\n",
              " 'parco': 1,\n",
              " 'dokubo': 1,\n",
              " 'affirms': 1,\n",
              " 'citizenry': 1,\n",
              " 'whatsapp': 1,\n",
              " 'marwat': 1,\n",
              " 'papel': 1,\n",
              " 'assent': 1,\n",
              " 'shepstone': 1,\n",
              " 'jra': 1,\n",
              " 'gen': 1,\n",
              " 'spinal': 1,\n",
              " 'nwfp': 1,\n",
              " 'mozammel': 1,\n",
              " 'isf': 1,\n",
              " 'caab': 1,\n",
              " 'upto': 1,\n",
              " 'marthinus': 1,\n",
              " 'karamat': 1,\n",
              " 'nemc': 1,\n",
              " 'commissioners': 1,\n",
              " 'seashore': 1,\n",
              " 'vb': 1,\n",
              " 'wey': 1,\n",
              " 'ugali': 1,\n",
              " 'swab': 1,\n",
              " 'taluka': 1,\n",
              " 'sithole': 1,\n",
              " 'kabul': 1,\n",
              " 'administrated': 1,\n",
              " 'azu': 1,\n",
              " 'morshed': 1,\n",
              " 'setswana': 1,\n",
              " 'datin': 1,\n",
              " 'abdul': 1,\n",
              " 'slavery': 1,\n",
              " 'nyali': 1,\n",
              " 'dcb': 1,\n",
              " 'fta': 1,\n",
              " 'commiserate': 1,\n",
              " 'bashir': 1,\n",
              " 'urgently': 1,\n",
              " 'irin': 1,\n",
              " 'imtiaz': 1,\n",
              " 'unep': 1,\n",
              " 'durbanville': 1,\n",
              " 'ulema': 1,\n",
              " 'praia': 1,\n",
              " 'arinze': 1,\n",
              " 'bench': 1,\n",
              " 'atiku': 1,\n",
              " 'artistes': 1,\n",
              " 'commend': 1,\n",
              " 'unced': 1,\n",
              " 'yemi': 1,\n",
              " 'lasbela': 1,\n",
              " 'modi': 1,\n",
              " 'vorpal': 1,\n",
              " 'undp': 1,\n",
              " 'tbn': 1,\n",
              " 'sartaj': 1,\n",
              " 'redress': 1,\n",
              " 'nwc': 1,\n",
              " 'convocation': 1,\n",
              " 'cck': 1,\n",
              " 'payola': 1,\n",
              " 'anas': 1,\n",
              " 'rungwe': 1,\n",
              " 'singida': 1,\n",
              " 'wajir': 1,\n",
              " 'mylapore': 1,\n",
              " 'chadema': 1,\n",
              " 'mengal': 1,\n",
              " 'kronor': 1,\n",
              " 'sexwale': 1,\n",
              " 'attahiru': 1,\n",
              " 'kombo': 1,\n",
              " 'hilal': 1,\n",
              " 'koech': 1,\n",
              " 'clementine': 1,\n",
              " 'azlan': 1,\n",
              " 'saifullah': 1,\n",
              " 'waziristan': 1,\n",
              " 'baringo': 1,\n",
              " 'nnewi': 1,\n",
              " 'gujjar': 1,\n",
              " 'yaseen': 1,\n",
              " 'chadha': 1,\n",
              " 'fifa': 1,\n",
              " 'kitale': 1,\n",
              " 'pali': 1,\n",
              " 'syleena': 1,\n",
              " 'felicitate': 1,\n",
              " 'emerald': 1,\n",
              " 'netrakona': 1,\n",
              " 'tashkent': 1,\n",
              " 'passbook': 1,\n",
              " 'parveen': 1,\n",
              " 'bazaar': 1,\n",
              " 'tta': 1,\n",
              " 'audiobooks': 1,\n",
              " 'sarkin': 1,\n",
              " 'augustine': 1,\n",
              " 'nyachae': 1,\n",
              " 'nhrc': 1,\n",
              " 'alo': 1,\n",
              " 'nazrul': 1,\n",
              " 'lums': 1,\n",
              " 'kurigram': 1,\n",
              " 'zonal': 1,\n",
              " 'beitbridge': 1,\n",
              " 'inzamam': 1,\n",
              " 'bhengu': 1,\n",
              " 'ijaw': 1,\n",
              " 'azikiwe': 1,\n",
              " 'inda': 1,\n",
              " 'gordimer': 1,\n",
              " 'koigi': 1,\n",
              " 'sansui': 1,\n",
              " 'pemra': 1,\n",
              " 'jing': 1,\n",
              " 'songea': 1,\n",
              " 'krugersdorp': 1,\n",
              " 'nsri': 1,\n",
              " 'amokachi': 1,\n",
              " 'montecasino': 1,\n",
              " 'agence': 1,\n",
              " 'zm': 1,\n",
              " 'csj': 1,\n",
              " 'wanga': 1,\n",
              " 'pfn': 1,\n",
              " 'hunza': 1,\n",
              " 'arakan': 1,\n",
              " 'ashulia': 1,\n",
              " 'jni': 1,\n",
              " 'blf': 1,\n",
              " 'issa': 1,\n",
              " 'ayubi': 1,\n",
              " 'tangail': 1,\n",
              " 'fb': 1,\n",
              " 'ouma': 1,\n",
              " 'makame': 1,\n",
              " 'inhambane': 1,\n",
              " 'sparc': 1,\n",
              " 'bukoba': 1,\n",
              " 'nazimabad': 1,\n",
              " 'afridi': 1,\n",
              " 'gujar': 1,\n",
              " 'tim': 1,\n",
              " 'nitel': 1,\n",
              " 'magubane': 1,\n",
              " 'rankuwa': 1,\n",
              " 'sipah': 1,\n",
              " 'kharal': 1,\n",
              " 'makarfi': 1,\n",
              " 'marriot': 1,\n",
              " 'wagah': 1,\n",
              " 'kyalami': 1,\n",
              " 'homewood': 1,\n",
              " 'storyteller': 1,\n",
              " 'arewa': 1,\n",
              " 'amani': 1,\n",
              " 'mehar': 1,\n",
              " 'rizwan': 1,\n",
              " 'loadshedding': 1,\n",
              " 'zamfara': 1,\n",
              " 'mpofu': 1,\n",
              " 'cnl': 1,\n",
              " 'parvez': 1,\n",
              " 'jahangirnagar': 1,\n",
              " 'arrestees': 1,\n",
              " 'kada': 1,\n",
              " 'umoja': 1,\n",
              " 'wazir': 1,\n",
              " 'kriegler': 1,\n",
              " 'iringa': 1,\n",
              " 'germiston': 1,\n",
              " 'censuses': 1,\n",
              " 'randburg': 1,\n",
              " 'lawan': 1,\n",
              " 'jnu': 1,\n",
              " 'petroleum': 1,\n",
              " 'diss': 1,\n",
              " 'shamsuddin': 1,\n",
              " 'dastagir': 1,\n",
              " 'umhlanga': 1,\n",
              " 'voor': 1,\n",
              " 'upington': 1,\n",
              " 'hogsback': 1,\n",
              " 'culvert': 1,\n",
              " 'petrobangla': 1,\n",
              " 'allister': 1,\n",
              " 'yolanda': 1,\n",
              " 'leprae': 1,\n",
              " 'sunrise': 1,\n",
              " 'asuu': 1,\n",
              " 'unfreeze': 1,\n",
              " 'dele': 1,\n",
              " 'wacs': 1,\n",
              " 'rawson': 1,\n",
              " 'ngubane': 1,\n",
              " 'shariah': 1,\n",
              " 'pompidou': 1,\n",
              " 'mbalula': 1,\n",
              " 'senior': 1,\n",
              " 'makurdi': 1,\n",
              " 'kiraitu': 1,\n",
              " 'kinondoni': 1,\n",
              " 'lyari': 1,\n",
              " 'nollywood': 1,\n",
              " 'mcr': 1,\n",
              " 'cyclone': 1,\n",
              " 'ijaz': 1,\n",
              " 'bba': 1,\n",
              " 'moshi': 1,\n",
              " 'nns': 1,\n",
              " 'askaris': 1,\n",
              " 'kigali': 1,\n",
              " 'mapinduzi': 1,\n",
              " 'bayard': 1,\n",
              " 'ismo': 1,\n",
              " 'rotimi': 1,\n",
              " 'nupeng': 1,\n",
              " 'tb': 1,\n",
              " 'kohat': 1,\n",
              " 'ospreys': 1,\n",
              " 'mohajir': 1,\n",
              " 'ahsan': 1,\n",
              " 'lira': 1,\n",
              " 'ngilu': 1,\n",
              " 'patuakhali': 1,\n",
              " 'lashkars': 1,\n",
              " 'ahmad': 1,\n",
              " 'standford': 1,\n",
              " 'jacana': 1,\n",
              " 'manzor': 1,\n",
              " 'oando': 1,\n",
              " 'pld': 1,\n",
              " 'unconfirmed': 1,\n",
              " 'rajanpur': 1,\n",
              " 'huq': 1,\n",
              " 'ogba': 1,\n",
              " 'inanda': 1,\n",
              " 'sagar': 1,\n",
              " 'bownes': 1,\n",
              " 'meru': 1,\n",
              " 'sab': 1,\n",
              " 'madrasa': 1,\n",
              " 'gweru': 1,\n",
              " 'engro': 1,\n",
              " 'dipu': 1,\n",
              " 'reprocess': 1,\n",
              " 'tigo': 1,\n",
              " 'asghar': 1,\n",
              " 'istan': 1,\n",
              " 'nelspruit': 1,\n",
              " 'rhodes': 1,\n",
              " 'govindsamy': 1,\n",
              " 'randt': 1,\n",
              " 'zesa': 1,\n",
              " 'burnt': 1,\n",
              " 'sohail': 1,\n",
              " 'orally': 1,\n",
              " 'za': 1,\n",
              " 'ecomog': 1,\n",
              " 'kirsten': 1,\n",
              " 'kadhi': 1,\n",
              " 'bahawalnagar': 1,\n",
              " 'teesta': 1,\n",
              " 'irungu': 1,\n",
              " 'powe': 1,\n",
              " 'pentecostal': 1,\n",
              " 'sensibly': 1,\n",
              " 'noakhali': 1,\n",
              " 'kericho': 1,\n",
              " 'bil': 1,\n",
              " 'bandarban': 1,\n",
              " 'ndola': 1,\n",
              " 'nabi': 1,\n",
              " 'sospeter': 1,\n",
              " 'shehu': 1,\n",
              " 'majhi': 1,\n",
              " 'nile': 1,\n",
              " 'ezza': 1,\n",
              " 'villalobos': 1,\n",
              " 'ead': 1,\n",
              " 'mahmood': 1,\n",
              " 'adua': 1,\n",
              " 'ndpvf': 1,\n",
              " 'florence': 1,\n",
              " 'pankaj': 1,\n",
              " 'forum': 1,\n",
              " 'hahnemann': 1,\n",
              " 'nbp': 1,\n",
              " 'badagry': 1,\n",
              " 'tijani': 1,\n",
              " 'maulana': 1,\n",
              " 'ngao': 1,\n",
              " 'nnsc': 1,\n",
              " 'suman': 1,\n",
              " 'nnamdi': 1,\n",
              " 'arshad': 1,\n",
              " 'uemoa': 1,\n",
              " 'thar': 1,\n",
              " 'oshiomhole': 1,\n",
              " 'wir': 1,\n",
              " 'hrcp': 1,\n",
              " 'ershad': 1,\n",
              " 'goree': 1,\n",
              " 'unequivocally': 1,\n",
              " 'chiniot': 1,\n",
              " 'lhc': 1,\n",
              " 'pff': 1,\n",
              " 'pirzada': 1,\n",
              " 'chishti': 1,\n",
              " 'lwazi': 1,\n",
              " 'siddharth': 1,\n",
              " 'stanger': 1,\n",
              " 'jilt': 1,\n",
              " 'gulab': 1,\n",
              " 'barguna': 1,\n",
              " 'nsukka': 1,\n",
              " 'saaf': 1,\n",
              " 'kj': 1,\n",
              " 'realy': 1,\n",
              " 'safm': 1,\n",
              " 'parastatals': 1,\n",
              " 'mumias': 1,\n",
              " 'brotherhood': 1,\n",
              " 'esan': 1,\n",
              " 'pso': 1,\n",
              " 'bamba': 1,\n",
              " 'laikipia': 1,\n",
              " 'cadd': 1,\n",
              " 'nadra': 1,\n",
              " 'kamra': 1,\n",
              " 'cuf': 1,\n",
              " 'faridpur': 1,\n",
              " 'deon': 1,\n",
              " 'xtra': 1,\n",
              " 'krugerrand': 1,\n",
              " 'grear': 1,\n",
              " 'ppb': 1,\n",
              " 'pnu': 1,\n",
              " 'wyk': 1,\n",
              " 'nishat': 1,\n",
              " 'ghazal': 1,\n",
              " 'gusau': 1,\n",
              " 'umair': 1,\n",
              " 'mu': 1,\n",
              " 'knysna': 1,\n",
              " 'wahab': 1,\n",
              " 'njeri': 1,\n",
              " 'shaka': 1,\n",
              " 'chaman': 1,\n",
              " 'zolani': 1,\n",
              " 'ms': 1,\n",
              " 'jamshed': 1,\n",
              " 'osogbo': 1,\n",
              " 'divisie': 1,\n",
              " 'chitral': 1,\n",
              " 'webpage': 1,\n",
              " 'kobo': 1,\n",
              " 'multiverse': 1,\n",
              " 'shadrack': 1,\n",
              " 'bikaner': 1,\n",
              " 'cid': 1,\n",
              " 'deji': 1,\n",
              " 'nwachukwu': 1,\n",
              " 'masih': 1,\n",
              " 'canoeist': 1,\n",
              " 'muhith': 1,\n",
              " 'abacha': 1,\n",
              " 'thr': 1,\n",
              " 'inu': 1,\n",
              " 'falsely': 1,\n",
              " 'safa': 1,\n",
              " 'amanzimtoti': 1,\n",
              " 'nyahururu': 1,\n",
              " 'haque': 1,\n",
              " 'chamkani': 1,\n",
              " 'boa': 1,\n",
              " 'belafonte': 1,\n",
              " 'mabuya': 1,\n",
              " 'zulfiqar': 1,\n",
              " 'jigawa': 1,\n",
              " 'conciliation': 1,\n",
              " 'trader': 1,\n",
              " 'jatiya': 1,\n",
              " 'ndma': 1,\n",
              " 'gidi': 1,\n",
              " 'cliff': 1,\n",
              " 'polokwane': 1,\n",
              " 'supple': 1,\n",
              " 'azhar': 1,\n",
              " 'cassava': 1,\n",
              " 'sani': 1,\n",
              " 'province': 1,\n",
              " 'menon': 1,\n",
              " 'gsr': 1,\n",
              " 'orator': 1,\n",
              " 'omondi': 1,\n",
              " 'jawahar': 1,\n",
              " 'boks': 1,\n",
              " 'philemon': 1,\n",
              " 'eurocentric': 1,\n",
              " 'conservancies': 1,\n",
              " 'redeployed': 1,\n",
              " 'ferreira': 1,\n",
              " 'verulam': 1,\n",
              " 'ngozi': 1,\n",
              " 'chawinda': 1,\n",
              " 'dogana': 1,\n",
              " 'bhati': 1,\n",
              " 'attache': 1,\n",
              " 'hurter': 1,\n",
              " 'multichoice': 1,\n",
              " 'kigoma': 1,\n",
              " 'otieno': 1,\n",
              " 'ancestry': 1,\n",
              " 'voortrekker': 1,\n",
              " 'heals': 1,\n",
              " 'mbeki': 1,\n",
              " 'ptf': 1,\n",
              " 'wana': 1,\n",
              " 'teletalk': 1,\n",
              " 'kemi': 1,\n",
              " 'gazipur': 1,\n",
              " 'maasai': 1,\n",
              " 'pki': 1,\n",
              " 'bantu': 1,\n",
              " 'aarhus': 1,\n",
              " 'amina': 1,\n",
              " 'randall': 1,\n",
              " 'rmg': 1,\n",
              " 'manj': 1,\n",
              " 'takht': 1,\n",
              " 'diffa': 1,\n",
              " 'asim': 1,\n",
              " 'lauding': 1,\n",
              " 'bpe': 1,\n",
              " 'mansoor': 1,\n",
              " 'hoti': 1,\n",
              " 'cjp': 1,\n",
              " 'izu': 1,\n",
              " 'muigai': 1,\n",
              " 'kot': 1,\n",
              " 'sheikhupura': 1,\n",
              " 'margalla': 1,\n",
              " 'comair': 1,\n",
              " 'lalu': 1,\n",
              " 'indus': 1,\n",
              " 'kosgei': 1,\n",
              " 'benue': 1,\n",
              " 'parliamentarian': 1,\n",
              " 'khewra': 1,\n",
              " 'karu': 1,\n",
              " 'cl': 1,\n",
              " 'riaz': 1,\n",
              " 'nprc': 1,\n",
              " 'woodfull': 1,\n",
              " 'saidpur': 1,\n",
              " 'vusi': 1,\n",
              " ...}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 96
        }
      ]
    }
  ]
}