lab3

2024/lab3/Lab3 - KG from unstructured data.ipynb

@@ -610,6 +610,29 @@
     "# displacy.render(next(doc.sents), style='dep', jupyter=True)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 🧫 Train spacy to recognize diseases\n",
+    "\n",
+    "Spacy is not able to recognize diseases. So we will train it for this purpose. \n",
+    "\n",
+    "We will use an existing [dataset where diseases have been annotated](https://raw.githubusercontent.com/MaastrichtU-IDS/prodigy-drug-indication-annotation/master/relation/dailymed_disease3.jsonl) in english sentences, containing ~500 annotations for diseases, sample:\n",
+    "\n",
+    "```json\n",
+    "{\n",
+    "    \"text\":\"    Iritis, iridocyclitis.\",\n",
+    "    \"spans\": [\n",
+    "        {\"start\":4,\"end\":10,\"token_start\":1,\"token_end\":1,\"label\":\"DISEASE\"},\n",
+    "        {\"start\":12,\"end\":25,\"token_start\":3,\"token_end\":3,\"label\":\"DISEASE\"}\n",
+    "    ]\n",
+    "}\n",
+    "```\n",
+    "\n",
+    "First generate the training data:\n"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},