Named-Entity-Recognition (NER)-spaCy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Named Entity Recognition (NER) with Spacy\n",
    "\n",
    "The purpose of this notebook is to demonstrate the entire process of name-entity recognition([**NER**](https://nlp.stanford.edu/ner/)) from start to the end with [**Spacy**](https://spacy.io/). \n",
    "This notebook also explore **pattern matching** as an alternative to **NER** when there is a known small set of fixed values. \n",
    "\n",
    "This will be a complete end-to-end demonstration of the entire process, including both labelling and model training.\n",
    "\n",
    "In this notebook, we train a model to detect entities related to **oil/petrol** from this [public dataset](https://www.kaggle.com/mitusha/email-dataset) which contains a list of emails related to the oil industry. This is an over simplification because we want more generic entities, but it shows how pattern matching is a better alternative than NER in this case. To summarise, we will extract oil-related elements from email messages.\n",
    "\n",
    "Below are the process perform in this notebook:\n",
    "- Read the emails data set which has an email per line.\n",
    "- Label the emails with the OIL entity using **[Doccano](http://doccano.herokuapp.com/)** labeling tool. This is a manual process.\n",
    "- Save the labels in a text file as **JSONL**.\n",
    "- Use **Spacy** Neural Network model to train a new statistical model. \n",
    "- Save the model.\n",
    "- Create a **Spacy** NLP pipeline and use the new model to detect oil entities never seen before.\n",
    "- Use pattern matching instead of a deep learning model to compare both method.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Label the Data\n",
    "\n",
    "First Step: Level the data using open source platform **Doccano**.\n",
    "\n",
    "Follow **[Doccano](https://doccano.github.io/doccano/tutorial/)** instructions to install and open Doccano.\n",
    "\n",
    "If you use **Linux/Mac**, I recommend using the docker image:\n",
    "- `docker pull doccano/doccano`\n",
    "- `docker container create --name doccano -e \"ADMIN_USERNAME=admin\" -e \"ADMIN_EMAIL=admin@example.com\" -e \"ADMIN_PASSWORD=password\" -p 8000:8000 doccano/doccano`\n",
    "- `docker container start doccano`\n",
    "    \n",
    "For **Windows**, just use **pip**: \n",
    "- `pip install doccano`\n",
    "- `doccano`\n",
    "\n",
    "Go to http://127.0.0.1:8000/.\n",
    "\n",
    "Next, label the data using Doccano. Find entities which talk about oil, petrol, petroleum, etc and label them with the tag **OIL**. \n",
    "\n",
    "Export the result as **JSONL(Text label)** format."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Training\n",
    "\n",
    "First, let's read the JSONL file using format:\n",
    "\n",
    "`{\"id\": 15, \"text\": \"....\", \"meta\": {}, \"annotation_approver\": null, \"labels\": [[226, 234, \"OIL\"]]}\n",
    "`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "labeled_data = []\n",
    "with open(r\"emails_labeled.jsonl\", \"r\") as read_file:\n",
    "    for line in read_file:\n",
    "        data = json.loads(line)\n",
    "        labeled_data.append(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Convert the format to spacy format\n",
    "\n",
    "Next, let's convert the Deccano format to Spacy format.\n",
    "\n",
    "We will also remove extra columns and rename labels to entities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "TRAINING_DATA = []\n",
    "for entry in labeled_data:\n",
    "    entities = []\n",
    "    for e in entry['labels']:\n",
    "        entities.append((e[0], e[1],e[2]))\n",
    "    spacy_entry = (entry['text'], {\"entities\": entities})\n",
    "    TRAINING_DATA.append(spacy_entry)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train the model!\n",
    "\n",
    "Use Deep Learning (NN) with a 0.3 dropout rate to avoid overfitting.\n",
    "\n",
    "The idea is to use a Neural Network with numerous layers and a large number of neurons. We present them text that has already been classified, so the answer is already known. We'll run a lot of iterations, and on each one, we'll calculate the error using a Loss Function, which will modify the weight of the neurons, causing them to fire. The weight of the network will be modified over time in order to eliminate improper learning patterns and solve the problem.\n",
    "\n",
    "To avoid overfitting, which means the model \"memorises\" the training data and does not perform well with new data, we remove specific neurons at random on each iteration. This makes it easier for the model to generalise."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember to install Spacy first:\n",
    "```\n",
    "pip install -U pip setuptools wheel\n",
    "pip install -U spacy\n",
    "python -m spacy download en_core_web_sm\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'ner': 1915.9444246831372}\n",
      "{'ner': 107.82956493094102}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/risehill/opt/anaconda3/lib/python3.8/site-packages/thinc/neural/_classes/layernorm.py:74: RuntimeWarning: invalid value encountered in reciprocal\n",
      "  d_xhat = N * dy - sum_dy - dist * var ** (-1.0) * sum_dy_dist\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'ner': 114.18649069852805}\n",
      "{'ner': 30.69304090604681}\n",
      "{'ner': 2.103905114710004}\n",
      "{'ner': 3.85198211228822}\n",
      "{'ner': 1.9485266517025366}\n",
      "{'ner': 0.004584697754935514}\n",
      "{'ner': 0.0006568707259644221}\n",
      "{'ner': 0.25235692383820013}\n",
      "{'ner': 1.399154501044839}\n",
      "{'ner': 0.005002993828410104}\n",
      "{'ner': 0.0007396650207534202}\n",
      "{'ner': 3.399979473944603e-07}\n",
      "{'ner': 8.389657480840494e-07}\n",
      "{'ner': 1.4286800035740475e-09}\n",
      "{'ner': 4.3158335529742466e-07}\n",
      "{'ner': 1.0808071583551658e-06}\n",
      "{'ner': 3.095908195536521e-08}\n",
      "{'ner': 2.361087045811699e-06}\n",
      "{'ner': 7.632661516871893e-05}\n",
      "{'ner': 9.230668441407683e-10}\n",
      "{'ner': 2.430684610377712e-09}\n",
      "{'ner': 2.0937108988823943e-06}\n",
      "{'ner': 3.121169423770219e-08}\n",
      "{'ner': 0.0005652823139269539}\n",
      "{'ner': 0.003151026469285797}\n",
      "{'ner': 8.318149025712086e-08}\n",
      "{'ner': 1.707827043826918e-05}\n",
      "{'ner': 0.0013131838653112446}\n",
      "{'ner': 0.0024066308325335723}\n",
      "{'ner': 3.7774335260993454e-05}\n",
      "{'ner': 0.4828146020814521}\n",
      "{'ner': 0.004459282312749806}\n",
      "{'ner': 4.888743675530538e-07}\n",
      "{'ner': 1.8908408503194669e-06}\n",
      "{'ner': 3.453178471484433e-07}\n",
      "{'ner': 2.7894427019818005e-07}\n",
      "{'ner': 5.044446233919177e-07}\n",
      "{'ner': 3.048672749029026e-08}\n"
     ]
    }
   ],
   "source": [
    "# Import models\n",
    "import spacy\n",
    "import random\n",
    "import json\n",
    "\n",
    "nlp = spacy.blank(\"en\")\n",
    "ner = nlp.create_pipe(\"ner\")\n",
    "nlp.add_pipe(ner)\n",
    "ner.add_label(\"OIL\")\n",
    "\n",
    "# Start the training\n",
    "nlp.begin_training()\n",
    "\n",
    "# Loop for 40 iterations\n",
    "for itn in range(40):\n",
    "    # Shuffle the training data\n",
    "    random.shuffle(TRAINING_DATA)\n",
    "    losses = {}\n",
    "\n",
    "    # Batch the examples and iterate over them\n",
    "    for batch in spacy.util.minibatch(TRAINING_DATA, size=2):\n",
    "        texts = [text for text, entities in batch]\n",
    "        annotations = [entities for text, entities in batch]\n",
    "\n",
    "        # Update the model\n",
    "        nlp.update(texts, annotations, losses=losses, drop=0.3)\n",
    "    print(losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should see the error decreasing as iterations go by, note that some times it may increase due to the dropout setting.\n",
    "\n",
    "#### Save the model to disk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp.to_disk(\"oil.model\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test the model\n",
    "\n",
    "Let's test the model.  For this we use displacy which will display the entities in the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from spacy import displacy\n",
    "example = \"service postings marathon petroleum co said it reduced the contract price it will pay for all grades of service oil one dlr a barrel effective today the decrease brings marathon s posted price for both west texas intermediate and west texas sour to dlrs a bbl the south louisiana sweet grade of service was reduced to dlrs a bbl the company last changed its service postings on jan reuter\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">service postings marathon \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    petroleum\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">OIL</span>\n",
       "</mark>\n",
       " co said it reduced the contract price it will pay for all grades of service \n",
       "<mark class=\"entity\" style=\"background: #ddd; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    oil\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; text-transform: uppercase; vertical-align: middle; margin-left: 0.5rem\">OIL</span>\n",
       "</mark>\n",
       " one dlr a barrel effective today the decrease brings marathon s posted price for both west texas intermediate and west texas sour to dlrs a bbl the south louisiana sweet grade of service was reduced to dlrs a bbl the company last changed its service postings on jan reuter</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "doc = nlp(example)\n",
    "displacy.render(doc, style='ent')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "\n",
    "This project shown how to label data with **Doccano** and create a custom model with **spaCy**. Happy explore! You now can customize your own model with spaCy. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Phrase Matching\n",
    "\n",
    "The second approach is to use pattern matching to look for certain keywords and patterns in the text. \n",
    "\n",
    "**Spacy** provides matchers which can be easily used to look for specific substrings, digits, etc. We can also set rules based on the part-of-speech tags."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Matches: ['petroleum', 'oil']\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "# Import the Matcher\n",
    "from spacy.matcher import Matcher\n",
    "\n",
    "nlp = spacy.load(name='en_core_web_sm')\n",
    "doc = nlp(example)\n",
    "\n",
    "# Initialize the Matcher with the shared vocabulary\n",
    "matcher = Matcher(nlp.vocab)\n",
    "\n",
    "# Add the pattern to the matcher\n",
    "matcher.add(\"OIL_PATTERN\", None, [{\"LOWER\": \"oil\"}], [{\"LOWER\": \"petroleum\"}])\n",
    "\n",
    "# Use the matcher on the doc\n",
    "matches = matcher(doc)\n",
    "print(\"Matches:\", [doc[start:end].text for match_id, start, end in matches])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see how we also found the right tag, but in this case typos or similar words are not detected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "We have seen to different approaches used for entity recognition. \n",
    "\n",
    "**Pattern Matching** can be used in the following use cases:\n",
    "- Low cardinality attributes\n",
    "- Common patterns such dates, quantities, numbers, etc.\n",
    "- Patterns occuring in certain parts of the speech\n",
    "- When typos are not expected\n",
    "- Structured data\n",
    "\n",
    "\n",
    "\n",
    "**Statistical Models** are great to learn complex patterns in the data and can \"guess\" and categorize data never seem before. Use cases:\n",
    "- High Cardinality attributes\n",
    "- You need to deal with typos (fuzzy matches)\n",
    "- You need to categorize new, never seen data.\n",
    "- Unstructured data\n",
    "\n",
    "These models are much more powerful since they can make decisions on things that were never trained on. It can detect new entities without any code change.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}