Experiments21_norec_bert.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Monolingual Norec experiments with bert\n",
    "Targeted sentiment analysis with simpletransformers for sequence tagging. Highly recommend to do this in a dedicated Python environment. You need PyTorch to interact with Cuda, and Simpletransformers to interact with pytorch, and you need the right Python version to support this chain. I suggest you begin with having a Cuda version that is listed in the pytorch installation guide, and take it from there. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cuda: True\n"
     ]
    }
   ],
   "source": [
    "from simpletransformers.ner import NERModel, NERArgs\n",
    "import torch\n",
    "import pandas as pd\n",
    "import pickle as pk\n",
    "import os\n",
    "import random\n",
    "import json\n",
    "import time, datetime\n",
    "random.seed(64)\n",
    "# import nltk\n",
    "import re\n",
    "# from nltk.tokenize.simple import SpaceTokenizer\n",
    "from helpers import *\n",
    "\n",
    "print(f\"Cuda: {torch.cuda.is_available()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare the data.\n",
    "I collect the experiments to run as a list of tuples. Each experiment consists of training set and testing set, and a list of tags. This list of tags is used only for checking that the files have the tags we expect. Highly recommend to do this checking, that we have the expected tags in the datasets. Token and tags need to be spaceseparated"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tags checked OK\n"
     ]
    }
   ],
   "source": [
    "datasets = [(\"data/spaceseparated/norec_fine/norec_fine_train.conll\", 'data/spaceseparated/norec_fine/norec_fine_dev.conll', ['I-targ-negative', 'I-targ-positive',\n",
    " 'B-targ-negative', 'B-targ-positive', 'O'])]\n",
    "'''\n",
    "datasets = [\n",
    " ( \"data/spaceseparated/norec_fine_nopolarity/nopol_norec_fine_train.conll\"\n",
    " , \"data/spaceseparated/norec_fine_nopolarity/nopol_norec_fine_dev.conll\",\n",
    "  ['I-targ', 'B-targ',  'O'] )] # To not repeat the first which was successful\n",
    "'''\n",
    "for train_path, dev_path, tags in datasets:\n",
    "    for path in [dev_path,train_path]:\n",
    "        with open (path) as rf:\n",
    "            text = rf.read()\n",
    "            # print(path)\n",
    "            # print(tags)\n",
    "            assert tagsset(text, separator = \" \") == set(tags), tagsset(text, separator = \" \")\n",
    "print(\"Tags checked OK\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Perform the fine-tuning\n",
    "Note that this script does not automatically  create the folders needed to save the model and to record the output. I recommend you run the following cell with 1 epoch to see that this works, befor setting it back to 8 or whatever you consider to be adequate. 3 should be enough, but I got a litte better result with 8 so I kept that.\n",
    "\n",
    "Simpletransformers has included the code for wandb but I have not tried to connect and use that what is supposed to be a great reporting and logging tool.\n",
    "\n",
    "Note that if you run many epochs and save the models, you will need a lot of space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": [
     "outputPrepend"
    ]
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at NbAiLab/nb-bert-base were not used when initializing BertForTokenClassification: ['cls.predictions.transform.LayerNorm.bias', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias']\n",
      "- This IS expected if you are initializing BertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing BertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "Some weights of BertForTokenClassification were not initialized from the model checkpoint at NbAiLab/nb-bert-base and are newly initialized: ['classifier.bias', 'classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n",
      "100%|██████████| 18/18 [00:05<00:00,  3.34it/s]\n",
      "Epoch 1 of 3:   0%|          | 0/3 [00:00<?, ?it/s]/home/egil/miniconda3/envs/st_gpu/lib/python3.9/site-packages/torch/optim/lr_scheduler.py:129: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate\n",
      "  warnings.warn(\"Detected call of `lr_scheduler.step()` before `optimizer.step()`. \"\n",
      "Epochs 0/3. Running Loss:    0.3853: 100%|██████████| 720/720 [04:12<00:00,  2.85it/s]\n",
      "Epochs 1/3. Running Loss:    0.3734: 100%|██████████| 720/720 [04:11<00:00,  2.86it/s]\n",
      "Epochs 2/3. Running Loss:    0.0032: 100%|██████████| 720/720 [04:11<00:00,  2.87it/s]\n",
      "Epoch 3 of 3: 100%|██████████| 3/3 [14:13<00:00, 284.61s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NbAiLab/nb-bert-base Done training\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:00<00:00,  4.33it/s]\n",
      "Running Evaluation: 100%|██████████| 192/192 [00:11<00:00, 17.14it/s]\n"
     ]
    }
   ],
   "source": [
    "# Run bert multilingual with the data from previous cell\n",
    "\n",
    "family = \"bert\"\n",
    "# transformersmodel = 'bert-base-multilingual-cased' \n",
    "# transformersmodel = 'ltgoslo/norbert'\n",
    "transformersmodel = 'NbAiLab/nb-bert-base'\n",
    "transformermodel_txt= transformersmodel.replace(\"/\", \"_\")\n",
    "results = []\n",
    "\n",
    "for train_path, dev_path, tags in datasets:\n",
    "    model_args = NERArgs() # New args loading fall 2020\n",
    "    model_args.train_batch_size = 12\n",
    "    model_args.num_train_epochs = 3\n",
    "    model_args.weight_decay = 0.001\n",
    "    model_args.overwrite_output_dir = True\n",
    "    model_args.silent = False\n",
    "    model_args.save_steps = 200000\n",
    "\n",
    "    model = NERModel(family,transformersmodel , labels = tags,args=model_args)\n",
    "\n",
    "    out_d = \"outputs/simpletransformers/\"+transformermodel_txt+\"_\"+train_path.split(\"/\")[-2]\n",
    "    running = os.path.join(out_d, \"running\") # Logging individual results\n",
    "    if not os.path.exists(os.path.dirname(running)):\n",
    "        os.makedirs(os.path.dirname(running))\n",
    "\n",
    "\n",
    "    model.train_model(train_path, output_dir= out_d)\n",
    "    print(transformersmodel, \"Done training\")\n",
    "\n",
    "    result, model_outputs, predictions = model.eval_model(dev_path)\n",
    "\n",
    "    #Record settings and results\n",
    "    result[\"train\"] = train_path\n",
    "    result[\"dev_test\"] = dev_path\n",
    "    result[\"training_epochs\"] = model_args.num_train_epochs\n",
    "    result[\"transformer_model\"] = transformersmodel\n",
    "    json_path = os.path.join(running,\"result_\"+datetime.datetime.now().strftime(\"%Y%m%d%H%M\")+\".json\")\n",
    "    if not os.path.exists(os.path.dirname(json_path)):\n",
    "        os.makedirs(os.path.dirname(json_path))\n",
    "    with open(json_path, \"w\") as wf:\n",
    "        json.dump(result, wf)\n",
    "    results.append(result)\n",
    "    with open(os.path.join(running, \"norec_fine_mono_predictions\" + datetime.datetime.now().strftime(\"%Y%m%d%H%M\")+\".json\" ), \"w\") as wf:\n",
    "        json.dump(predictions, wf)\n",
    "\n",
    "json_path = \"summaries/results_\"+datetime.datetime.now().strftime(\"%Y%m%d%H%M\")+\".json\"\n",
    "if not os.path.exists(os.path.dirname(json_path)):\n",
    "    os.makedirs(os.path.dirname(json_path))\n",
    "with open(json_path, \"w\") as wf:\n",
    "    json.dump(results, wf)\n",
    "df = pd.DataFrame.from_dict(results)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>eval_loss</th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "      <th>f1_score</th>\n",
       "      <th>train</th>\n",
       "      <th>dev_test</th>\n",
       "      <th>training_epochs</th>\n",
       "      <th>transformer_model</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.176104</td>\n",
       "      <td>0.500561</td>\n",
       "      <td>0.508552</td>\n",
       "      <td>0.504525</td>\n",
       "      <td>data/spaceseparated/norec_fine/norec_fine_trai...</td>\n",
       "      <td>data/spaceseparated/norec_fine/norec_fine_dev....</td>\n",
       "      <td>3</td>\n",
       "      <td>NbAiLab/nb-bert-base</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   eval_loss  precision    recall  f1_score  \\\n",
       "0   0.176104   0.500561  0.508552  0.504525   \n",
       "\n",
       "                                               train  \\\n",
       "0  data/spaceseparated/norec_fine/norec_fine_trai...   \n",
       "\n",
       "                                            dev_test  training_epochs  \\\n",
       "0  data/spaceseparated/norec_fine/norec_fine_dev....                3   \n",
       "\n",
       "      transformer_model  \n",
       "0  NbAiLab/nb-bert-base  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Run all above\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv(f\"summaries/{transformermodel_txt}.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inference\n",
    "First on the existing model, then an example of how to load a saved model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1/1 [00:00<00:00, 12.86it/s]\n",
      "Running Prediction: 100%|██████████| 1/1 [00:00<00:00, 38.66it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'Mannen': 'O'}, {'på': 'O'}, {'scenen': 'O'}, {'synger': 'O'}, {'stygt': 'O'}]\n",
      "[{'Damen': 'B-targ-negative'}, {'på': 'O'}, {'scenen': 'O'}, {'synger': 'O'}, {'stygt': 'O'}]\n",
      "[{'Disse': 'B-targ-positive'}, {'bilene': 'I-targ-positive'}, {'har': 'O'}, {'et': 'O'}, {'fantastisk': 'O'}, {'veigrep': 'O'}]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "predictions, raw_outputs = model.predict ([\"Mannen på scenen synger stygt\", \"Damen på scenen synger stygt\" , \"Disse bilene har et fantastisk veigrep\"])\n",
    "for sentence in predictions:\n",
    "    print(sentence)\n",
    "    # print(json.dumps(sentence, indent=3, ensure_ascii=False))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inference on saved model\n",
    "# out_d: 'outputs/simpletransformers/bert-base-multilingual-cased_norec_fine'\n",
    "if False:\n",
    "    last_epoch= sorted([f for f in os.listdir(out_d) if \"-\" in f], key = lambda x: int(x.split(\"-\")[-1]) )[-1]\n",
    "    print(last_epoch)\n",
    "    model2 = NERModel(family, os.path.join(out_d, last_epoch))\n",
    "    predictions, raw_outputs = model2.predict ([\"Mannen på scenen synger stygt\", \"Damen på scenen synger stygt\" , \"Disse bilene har et fantastisk veigrep\"])\n",
    "    for sentence in predictions:\n",
    "        print(sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "57cdd8f699a25da049745842ce18d98f4e8fe35b457f95f9184a84f9f44a193c"
  },
  "kernelspec": {
   "display_name": "Python 3.7.9 64-bit ('simplet': conda)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "orig_nbformat": 2
 },
 "nbformat": 4,
 "nbformat_minor": 2
}