transformer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stance Classification of Tweets using Transfer Learning\n",
    "This notebook shows how *transfer learning*, an extension of deep learning, can be used for predicting Tweet stance toward a particular topic."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Motivation\n",
    "\n",
    "The traditional approach to applying deep learning methods in NLP have involved feeding a model large amounts of labeled training data, and fitting the model's parameters to this data. In practice, natural language data is highly variable and can come in a variety of forms (tweets, blog posts, reviews etc.), and hence, a model trained for a particular language task does not generalize well to new data from another distribution. In addition, many natural language applications do not come with an abundance of labeled examples, and human annotation can get very expensive as the datasets get larger.\n",
    "\n",
    "This offers good motivation to explore the notion of [transfer learning](http://ruder.io/transfer-learning/index.html#whatistransferlearning) - a machine-learning technique that has the ability to transfer knowledge to novel scenarios not encountered during training. While transfer learning has been ubiquitous throughout computer vision applications since the advent of huge datasets such as ImageNet, it is only since 2017-18 that significant progress has been made for transfer learning in NLP applications. There have been a string of interesting papers in 2018 that discuss the power of language models in natural language understanding and how they can be used to provide pre-trained representations of a language's syntax, which can be far more useful when training a neural network for previously unseen tasks.\n",
    "\n",
    "Twitter data is a very interesting use case for transfer learning, mainly because the typical language syntax seen in Tweets is quite different from that which is used to train language models. For these reasons, the **2016 SemEval Stance Detection task** is chosen for studying the effectiveness of our transfer learning approach. The dataset, experiments and the evaluation criteria used are explained in below sections. \n",
    "\n",
    "The aim of this notebook is to highlight the development of a model that can help answer the following questions:\n",
    "- How does our approach generalize to Twitter-specific language syntax?\n",
    "- Are we able to achieve reasonable results (comparable to the winning team of SemEval 2016 Task 6) with *limited amounts of training data* and *limited computing resources*?\n",
    "- How much fine-tuning effort is required to achieve reasonable results?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Approach\n",
    "In this notebook, we approach the stance detection problem using a [PyTorch port](https://github.com/huggingface/pytorch-openai-transformer-lm) of the **OpenAI transformer** [[paper]](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf) as per Radford et. al, 2018."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 Transformer Model Architecture\n",
    "The OpenAI transformer is an adaptation of the well-known transformer from Google Brain's 2017 paper. \n",
    "\n",
    "![title](assets/transformer_arch.png)\n",
    "\n",
    "Source: [Attention is all you need, 2017](https://arxiv.org/pdf/1706.03762.pdf)\n",
    "\n",
    "While the original version from Google Brain used an identical encoder-decoder 6-layer stack, the OpenAI transformer uses similar dual 6-layer encoder-decoder stacks. Each layer has two sub-layers, consisting of a multi-head self-attention mechanism, and a fully connected (position-wise) feed-forward network. A full description of the transformer architecture used for transfer learning is given in the paper. \n",
    "\n",
    "Classification is performed in the following stages:\n",
    "1. **Unsupervised pretaining**: The OpenAI transformer is given an unsupervised corpus of tokens from the Google Books corpus (thousands of books) and the pretrained weights are made publicly available for further fine-tuning.\n",
    "2. **Supervised fine-tuning**: We can adapt the parameters to the supervised target task. The inputs are passed through the pre-trained model to obtain the final transformer block's activation $h_{l}^{m}$, which is then fed into an added linear output layer with parameters $W-y$ to predict $y$:\n",
    "\n",
    "$P(y | x^1, ..., x^m) = softmax(h_{l}^{m}W_y)$\n",
    "\n",
    "The first step (unsupervised pretaining) is *very* expensive, and was done by OpenAI on a large GPU cluster, and need not be repeated in our case - we can directly use the pretrained weights and fine-tune them before training the classifier. \n",
    "\n",
    "To perform out-of-domain target tasks such as text classification, the transformer includes language modeling as an additional objective to the fine-tuning, which helps generalized learning [[OpenAI transformer, 2018]](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf). This auxiliary language modeling objective is specified with a weighting parameter $\\lambda$ as shown below.\n",
    "\n",
    "$L_3(C) = L_2(C) + \\lambda \\cdot L_1(C)$\n",
    "\n",
    "where $L1$, $L2$ and $L3$ are the likelihoods for the language modeling objective, task-specific objective and combined objective respectively.\n",
    "\n",
    "## 2.2 Task-specific input transformations\n",
    "OpenAI designed their transformer to generalize to a range of natural language tasks. In order to do this, they allow the definition of custom task-specific \"heads\" as per the below schematic. The task-specific head acts on top of the base transformer language model, and is defined in the ```DoubleHeadModel``` class in ```model_pytorch.py```. \n",
    "\n",
    "![title](assets/openai_taskheads.png)\n",
    "\n",
    "Source: [Improving language understanding by generative pre-training, 2018](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf)\n",
    "\n",
    "\n",
    "The [PyTorch port](https://github.com/huggingface/pytorch-openai-transformer-lm) of the OpenAI transformer that is used was originally tested on a multiple choice classification dataset (ROCStories). For this Tweet stance detection task, we utilize the above image to write a classification transform, such that we pad every text (representing each Tweet, in our case) with a start symbol and tokenize them for input to the encoder layer. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Data\n",
    "\n",
    "We perform stance detection of Tweets on five distinct topics, as per [SemEval 2016: Task 6](http://alt.qcri.org/semeval2016/task6/). Due to time constraints, we only look at Task A: \"Supervised Framework\" in this notebook. The train and test data (including the gold) are in the the ```data/``` directory provided along with this repository.\n",
    "\n",
    "The five topics for which we classify the stance are given below. \n",
    "\n",
    "| Topic        \n",
    "|:------------: | \n",
    "| Atheism     |\n",
    "| Climate Change is a Real Concern  | \n",
    "| Feminist Movement | \n",
    "| Hillary Clinton  |\n",
    "| Legalization of Abortion  | \n",
    "\n",
    "A more detailed breakdown of the tweets for this shared task is provided in [this link](http://www.saifmohammad.com/WebPages/StanceDataset.htm). \n",
    "\n",
    "## Size of dataset\n",
    "The total number of Tweets (in the training set) available for this task is roughly 2700, which amounts to roughly 500-600 Tweets per topic. Thus, this can be considered a small dataset. \n",
    "\n",
    "![title](assets/stance_balance.png)\n",
    "\n",
    "Upon inspecting the training data, it is clear that there is quite a large variance in terms of the number of Tweets in favor vs. those against a topic. There is quite a large variance *within* classes as well as the overall data as well. \n",
    "\n",
    "An interactive visualization of the complete dataset (along with a topic-specific breakdown) is provided by the organizers of the competition [in this link](http://www.saifmohammad.com/WebPages/StanceDataset.htm).\n",
    "\n",
    "## Pretrained language models\n",
    "\n",
    "The OpenAI transformer language model pretrained weights can be downloaded directly from the [OpenAI GitHub repository](https://github.com/openai/finetune-transformer-lm). These are then stored in a directory called ```model/```."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Code\n",
    "\n",
    "The code used for classifying Tweet stance is [this PyTorch port of the OpenAI transformer](https://github.com/huggingface/pytorch-openai-transformer-lm). The following modifications are made to the original version of the transformer that was written to perform multiple-choice classification. \n",
    "\n",
    "1.  ```datasets.py```: A custom dataloader was written that processes the Tweets and splits the output into training, validation and test data\n",
    "\n",
    "2.  ```train_stance.py```: A custom classification input transform is written as per the image in section 2.3 (above), to feed in the Tweets to the transformer for classification.\n",
    "\n",
    "3.  ```parse_output.py```: The predicted stance for each Tweet is written out in a format that can be read in by the evaluation script, for scoring the model. \n",
    "\n",
    "The below code shows the dataloader functionality in ```datasets.py```."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import csv\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from pathlib import Path\n",
    "from sklearn.utils import shuffle\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cleaning code for the input dataloader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def _stance(path, topic=None):\n",
    "    def clean_ascii(text):\n",
    "        # function to remove non-ASCII chars from data\n",
    "        return ''.join(i for i in text if ord(i) < 128)\n",
    "    orig = pd.read_csv(path, delimiter='\\t', header=0, encoding = \"latin-1\")\n",
    "    orig['Tweet'] = orig['Tweet'].apply(clean_ascii)\n",
    "    df = orig\n",
    "    # Get only those tweets that pertain to a single topic in the training data\n",
    "    if topic is not None:\n",
    "        df = df.loc[df['Target'] == topic]\n",
    "    X = df.Tweet.values\n",
    "    stances = [\"AGAINST\", \"FAVOR\", \"NONE\", \"UNKNOWN\"]\n",
    "    class_nums = {s: i for i, s in enumerate(stances)}\n",
    "    Y = np.array([class_nums[s] for s in df.Stance])\n",
    "    return X, Y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split raw Tweet data into training, validation and test sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def stance(data_dir, topic=None):\n",
    "    path = Path(data_dir)\n",
    "    trainfile = 'semeval2016-task6-trainingdata.txt'\n",
    "    testfile = 'SemEval2016-Task6-subtaskA-testdata.txt'\n",
    "\n",
    "    X, Y = _stance(path/trainfile, topic=topic)\n",
    "    teX, _ = _stance(path/testfile, topic=topic)\n",
    "    tr_text, va_text, tr_sent, va_sent = train_test_split(X, Y, test_size=0.2, random_state=seed)\n",
    "    trX = []\n",
    "    trY = []\n",
    "    for t, s in zip(tr_text, tr_sent):\n",
    "        trX.append(t)\n",
    "        trY.append(s)\n",
    "\n",
    "    vaX = []\n",
    "    vaY = []\n",
    "    for t, s in zip(va_text, va_sent):\n",
    "        vaX.append(t)\n",
    "        vaY.append(s)\n",
    "    trY = np.asarray(trY, dtype=np.int32)\n",
    "    vaY = np.asarray(vaY, dtype=np.int32)\n",
    "    return (trX, trY), (vaX, vaY), (teX, )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example output from dataloader\n",
    "The dataloader methods output the train/validation/test data in the form of a list of tuples, as shown below. These will be used to feed the input transforms and perform stance classification in ```train_stance.py```. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define random seed for train/val split\n",
    "seed = 3535999445\n",
    "data_dir = \"./data\"\n",
    "# Dataloader output\n",
    "(trX, trY), (vaX, vaY), teX = stance(data_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can print each training set's Tweet along with its numericalized stance (0: AGAINST, 1: FAVOR, 2: NONE). Note that we do not perform any special cleaning of the data to remove '@' mentions or hashtags - all information is retained and encoded so that the language model can be fine-tuned using as much input information as possible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".@msimire says children, women, elderly people especially at risk to health impacts of #ClimateReporting2015 #SemST 1\n",
      "Plenty of stem cells without baby smashing, by the way. #SemST 0\n",
      "Is there a breeze i can catch Lakefront or will I die of a heat stroke there as well? #heatstroke #SemST 1\n",
      "Road to #Paris2015 \"#ADP2015 co-chairs' new tool will be presented on July 24th\" @manupulgarvidal at @UN_PGA event on #SemST 2\n",
      "Are the same people who are red faced and frothing over abortion also against the death penalty? Just wondering... #deathpenalty #SemST 1\n",
      "DID YOU KNOW: The 2nd Amendment is in place 'cause politicians ignore the #CONSTITUTION. #PJNET #SOT #tcot #bcot #ccot #2ndAmendment #SemST 0\n",
      "I Appreciate almighty God for waking me up diz beautiful day + giving me brilliant ideas to grow my #Hustle #SemST 0\n",
      "Speaking from the heart is rarely wise especially if you have a retarded heart like most feminists.   #GamerGate #SemST 0\n",
      "@Shy_Buffy welcome little sister! Love you! #SemST 2\n",
      "@GregAbbott_TX which god? Yours? not mine. oh wait i don't have one. #LoveWins #SemST 1\n"
     ]
    }
   ],
   "source": [
    "for i, t in enumerate(trX[:10]):\n",
    "    print(t, trY[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The test data does not come with a stance (that's what we need to predict)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['He who exalts himself shall      be humbled; and he who humbles himself shall be exalted.Matt 23:12.     #SemST'\n",
      " 'RT @prayerbullets: I remove Nehushtan -previous moves of God that have become idols, from the high places -2 Kings 18:4 #SemST'\n",
      " '@Brainman365 @heidtjj @BenjaminLives I have sought the truth of my soul and found it strong enough to stand on its own merits. #SemST'\n",
      " '#God is utterly powerless without Human intervention... #SemST'\n",
      " '@David_Cameron   Miracles of #Multiculturalism   Miracles of shady 786  #Taqiya #Tawriya #Jaziya #Kafirs #Dhimmi #Jihad #Allah #SemST']\n"
     ]
    }
   ],
   "source": [
    "# Test set\n",
    "print(teX[0][:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Experimental Setup\n",
    "\n",
    "For stance detection, we use a semi-supervised approach where we reuse weights from a pretrained language model, and perform multi-class classification for the training data over the three classes ('FAVOR', 'AGAINST' and 'NONE').\n",
    "\n",
    "### Reference result\n",
    "To inform our methods and to have a benchmark to compare our results against, we looked at the winning paper for this shared task, from team *MITRE*, who [published their methodology and approach](https://arxiv.org/pdf/1606.03784.pdf). \n",
    "\n",
    "### Evaluation\n",
    "The metric used to score the stance classification is **F-score**. The SemEval event organizers provided an [evaluation script](http://alt.qcri.org/semeval2016/task6/index.php?id=data-and-tools) that calculates the macro-average of F-score (FAVOR) and F-score (AGAINST) for task A. This compares our model's predicted stance for each Tweet against the gold reference.\n",
    "\n",
    "We use the *perl* script provided by the organizers to generate our F- score. The evaluation script is in ```data/eval/``` and has the following usage:\n",
    "    \n",
    "    cd data/eval\n",
    "    perl eval.pl -u\n",
    "\n",
    "    ---------------------------\n",
    "    Usage:\n",
    "    perl eval.pl goldFile guessFile\n",
    "\n",
    "    goldFile:  file containing gold standards;\n",
    "    guessFile: file containing your prediction.\n",
    "    \n",
    "### Stance Prediction\n",
    "The predicted output stances on the test dataset is written out according to the format expected by the evaluation *perl* script, and the F-scores are published as per this evaluation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.1 Transfer Learning Using the Transformer\n",
    "We use the pretrained weights for the model provided in [OpenAI's GitHub repository](https://github.com/openai/finetune-transformer-lm). The parameter files are placed inside the ```model/``` directory so that they can be called into ```train_stance.py```.  As described in section 4, various parts of the code were modified to suit the Tweet stance detection task.\n",
    "\n",
    "To study the performance of the transformer model, we first study the hyperparameter selection.  The main focus is on adjusting the dropout of the various layers, and the LM coefficient $\\lambda$, which controls the tradeoff between the language modeling head and the task head as per the following formula from the OpenAI paper:\n",
    "\n",
    "$$L_3(C) = L_2(C) + \\lambda \\cdot L_1(C)$$\n",
    "\n",
    "where $C$ is a labelled input dataset, $L_2$ is the likelihood function for the classification head, and $L_1$ is the likehilood function for the language modeling head.  Three trials are run for most configurations - the only times single trials are run are in cases where the first trial showed a large enough change in results to be fairly certain that the hyperparameter change was causing a significant difference."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The transformer uses [spaCy](https://spacy.io/) for tokenization. The file ```text_utils.py``` contains a wrapper for a byte-pair encoded tokenizer.  In order to use spaCy's English tokenizer, first make sure that it was installed properly.\n",
    "\n",
    "    python3 -m spacy download en "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To start running the training, type the following into the terminal\n",
    "\n",
    "    cd transformer-openai\n",
    "    python3 train_stance.py --dataset stance --desc stance --submit --data_dir ../data --submission_dir default\n",
    "\n",
    "In order to evaluate the results,, we need to get the predictions into the format required by the eval script. This is done using the script `parse_output.py`.  It takes the path of the test data file, the path of the predictions created by `train_stance.py`, the path to write the results to, and an optional topic to filter by as arguments.  Example usage is below:\n",
    "\n",
    "    python3 parse_output.py ../data/SemEval2016-Task6-subtaskA-testdata.txt default/stance.tsv ../results/predicted_default.txt\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can check the F-score using the evaluation script provided with the dataset.  For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "============\n",
      "Results\t\t\t\t \n",
      "============\n",
      "FAVOR     precision: 0.5899 recall: 0.6908 f-score: 0.6364\n",
      "AGAINST   precision: 0.7990 recall: 0.7007 f-score: 0.7466\n",
      "------------\n",
      "Macro F: 0.6915\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!perl data/eval/eval.pl data/eval/gold.txt results/transformer_predicted_2.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This result appears to be marginally higher than the one reported by *MITRE* in their winning submission!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experiments\n",
    "Below are the training commands used for several experiments searching for optimal hyperparameter settings.  For cases with three trials, the training was run with a `--seed` argument, using values 42, 43, and 44 for the each separate trial.  For brevity, only the command used to train a single trial is shown beloww for each configuration."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Default\n",
    "    python3 train_stance.py --dataset stance --desc stance --submit --data_dir ../data --submission_dir default"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dropout\n",
    "The training accuracies tend to be extremely high compared to the validation and test accuracies, most likely due to the small size of the dataset relative to the number of parameters in the model.  Typically, this can be mitigated by varying dropout.  Since we are not modifying the unsupervised pretrained weights (we are reusing OpenAI's publicly shared parameter files), we only modify the dropout of the classification layer in our model.\n",
    "\n",
    "The below four commands show the input arguments used for the first trial of each value of classification dropout tested."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    python3 train_stance.py --dataset stance --desc stance --submit --data_dir ../data --submission_dir --clf_pdrop 0.05 clf_drop\n",
    "    python3 train_stance.py --dataset stance --desc stance --submit --data_dir ../data --submission_dir --clf_pdrop 0.2 clf_less_drop\n",
    "    python3 train_stance.py --dataset stance --desc stance --submit --data_dir ../data --submission_dir --clf_pdrop 0.3 clf_3drop\n",
    "    python3 train_stance.py --dataset stance --desc stance --submit --data_dir ../data --submission_dir --clf_pdrop 0.5 clf_5drop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results of this are output to the `results` folder, called `predicted_lessdrop`, `predicted_drop`, `predicted_3drop`, and `predicted_5drop` respectively.  The case with 0.05 dropout included only one trial since it didn't seem very promising."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we can increase the dropout in all layers, this time only trying with 0.3 and 0.5 since 0.2 did not seem to affect things much in the previous experiment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    python3 train_stance.py --dataset stance --desc stance --submit --data_dir ../data --clf_pdrop 0.3 --embd_pdrop 0.3 --resid_pdrop 0.3 --submission_dir all_3drop\n",
    "    python3 train_stance.py --dataset stance --desc stance --submit --data_dir ../data --submission_dir --clf_pdrop 0.5 --embd_pdrop 0.5 --resid_pdrop 0.5 --submission_dir all_5drop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These results are called `predicted_super3drop` and `predicted_super5drop` respectively."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LM Coefficient\n",
    "Next, we set the LM coefficient to zero so that the weights are updated purely based on the performance of the classification head rather than the language model head. This was something suggested in the [OpenAI paper](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf) in cases where the amount of training data is small. Hence, it makes logical sense to try this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    python3 train_stance.py --dataset stance --desc stance --submit --data_dir ../data --submission_dir no_lm --lm_coeff 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results\n",
    "Case | Trial 1 F-score | Trial 2 F-score | Trial 3 F-score\n",
    "--- | --- | ---- | ---\n",
    "default | 0.6790 | 0.6915 | 0.6649\n",
    "less_drop | 0.6432 | \n",
    "drop | 0.6625 | 0.6734 | 0.6799\n",
    "3drop | 0.6758 | 0.6725 | 0.6701\n",
    "5drop | 0.6690 | 0.6826 | \n",
    "super3drop | 0.6610 | 0.6808 | 0.6767\n",
    "super5drop | 0.3876 |\n",
    "no_lm | 0.6293 |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Overall, modifying the dropout seemed to make very little difference unless it was lowered, or raised to an extreme amount on all layers. The single best F-score obtained was when the default values in ```train_stance.py``` were applied for all hyperparameters.  The training accuracy *reduced* with higher dropout, especially in the case of 0.3 dropout on all layers, but this did nothing to improve the F-score.\n",
    "\n",
    "Despite the paper's suggestion that excluding (or reducing the weight of) the language modeling objective could help for smaller datasets, in this Tweet stance detection task, excluding the language modeling objective seemed to make things noticeably worse.  This is likely due to how different the training set distribution (of Tweets with their quirky syntax) is from the Google Books corpus that the language model was pretrained on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 8. Conclusions\n",
    "This notebook showed a training and classification pipeline for a PyTorch-port of the OpenAI transformer for evaluating stance of Tweets towards a particular topic. With some basic data cleaning operations and adding some custom code for task-specific goals, the transfer learning approach using the OpenAI transformer seems to provide good classification accuracy F-scores (compared with the winning results from the SemEval 2016 winning team *MITRE*). \n",
    "\n",
    "The main benefit of using the OpenAI transformer is that it appears to generalize well to a completely different distribution (i.e., Tweets), with very minimal modifications to the original framework and minimal hyperparameter fine-tuning. In addition to the relative ease of adapting the transformer code, the model's is also quite inexpensive. The fine-tuning of the language model and the classification layer happens simultaneously, and we achieve a good F-score (of 0.69) comparable to that of the winning team *MITRE* (0.67), in **just 2-3 epochs of training!** This shows that transformers are a promising tool to perform generalizable transfer learning for a wide range of classification tasks, even when the input distribution is very different from the pretrained language model's distribution.\n",
    "\n",
    "One limitation of the transformer is that it is prone to overfitting when we have a very small input dataset (of ~500 Tweets). This is noticeable when we try to perform classification on each topic *individually*, where we only have around 500 Tweets per topic. In that case, the F-score drops as does the validation accuracy (whereas training accuracy remains very high) indicating significant overfitting. This makes sense considering that the transformer has 786 dimensions on the task-heads, so we would logically have to train it on a dataset several times this size. \n",
    "\n",
    "We can likely further improve the classification accuracy across all topics by feeding in a larger dataset, of several thousand Tweets to the transformer model. \n",
    "\n",
    "In general, Tweets are sufficiently different from typical language data used to generate pre-trained language models, and  hence are an interesting usecase for analyzing the effectiveness of transfer learning techniques. It will be interesting to see how transfer learning techniques for such NLP tasks evolve with time."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}