ontology-analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Welcome to the Ontology Tutorial hand-on session. \n",
    "\n",
    "Here, we will demonstrate how to use ontologies through the OWLAPI, classify ontologies, reason over them, and compute semantic similarity. In the second part, we will also use machine learning with ontologies.\n",
    "\n",
    "If you have not done so, please download our data package from http://aber-owl.net/aber-owl/diseasephenotypes/ontology/ontology-tutorial.tar.gz\n",
    "\n",
    "To run a piece of code, just go to the box, and either press Shift+Enter, or click on the \"run cell\" button in the menu bar.\n",
    "\n",
    "__Please run the first cell before the tutorial!__\n",
    "This will download all the Java libraries we need.\n",
    "\n",
    "Go ahead now and run the first cell. It may take a while to download all the libraries, so you may want to get a coffee while it's running."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!groovy\n",
    "// @NotebookService nb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "// Download dependencies using Groovy's Grape system\n",
    "import groovy.grape.Grape\n",
    "Grape.grab(group:'org.semanticweb.elk', module:'elk-owlapi', version:'0.4.3')\n",
    "Grape.grab(group:'net.sourceforge.owlapi', module:'owlapi-api', version:'4.2.5')\n",
    "Grape.grab(group:'net.sourceforge.owlapi', module:'owlapi-apibinding', version:'4.2.5')\n",
    "Grape.grab(group:'net.sourceforge.owlapi', module:'owlapi-impl', version:'4.2.5')\n",
    "Grape.grab(group:'com.github.sharispe', module:'slib-sml', version:'0.9.1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now show some basic operations on ontologies. We need to import lots of classes from the OWLAPI, create an OWL ontology manager, load the ontology, and start exploring a bit.\n",
    "\n",
    "The code below may need to be modified a bit. In the line `ont = manager.loadOntologyFromOntologyDocument(new IRI(\"file:merged-phenomenet.owl\"))`, please change the path to where you have extracted the content of our data package. For example, if you extracted the content of our data package to `/tmp/`, replace `file:merged-phenomenet.owl` with `file:/tmp/merged-phenomenet.owl`.\n",
    "\n",
    "__If you have memory problems:__ The ontology is rather large and requires around 10GB of memory to load (and later to classify). If you have problems loading this ontology, download `phenomenet-inferred.owl` from our Github site and replace `file:merged-phenomenet.owl` with `file:phenomenet-inferred.owl` in the code below (you just need to comment out the line which loads the ontology and uncomment the alternative)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "224748"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "// opens ontology file and prints the number of classes\n",
    "\n",
    "// import what we need\n",
    "import java.net.*\n",
    "import org.semanticweb.owlapi.model.parameters.*\n",
    "import org.semanticweb.elk.owlapi.ElkReasonerFactory;\n",
    "import org.semanticweb.elk.owlapi.ElkReasonerConfiguration\n",
    "import org.semanticweb.elk.reasoner.config.*\n",
    "import org.semanticweb.owlapi.apibinding.OWLManager;\n",
    "import org.semanticweb.owlapi.reasoner.*\n",
    "import org.semanticweb.owlapi.reasoner.structural.StructuralReasoner\n",
    "import org.semanticweb.owlapi.vocab.OWLRDFVocabulary;\n",
    "import org.semanticweb.owlapi.model.*;\n",
    "import org.semanticweb.owlapi.io.*;\n",
    "import org.semanticweb.owlapi.owllink.*;\n",
    "import org.semanticweb.owlapi.util.*;\n",
    "import org.semanticweb.owlapi.search.*;\n",
    "import org.semanticweb.owlapi.manchestersyntax.renderer.*;\n",
    "import org.semanticweb.owlapi.reasoner.structural.*\n",
    " \n",
    "// Let's load an ontology and output the number of classes\n",
    "// Create the ontology manager\n",
    "manager = OWLManager.createOWLOntologyManager()\n",
    "// create data factory (to create axioms, classes, etc.)\n",
    "fac = manager.getOWLDataFactory()\n",
    "// Load the latest version of the PhenomeNET Ontology; for more information about this ontology, see http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1005500\n",
    "\n",
    "ont = manager.loadOntologyFromOntologyDocument(new IRI(\"file:merged-phenomenet.owl\"))\n",
    "\n",
    "// Use the next line instead if you have memory problems loading the ontology above (comment out the line above and uncomment this line):\n",
    "//ont = manager.loadOntologyFromOntologyDocument(new IRI(\"file:phenomenet-inferred.owl\"))\n",
    "\n",
    "ont.getClassesInSignature(true).size()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reasoning with ontologies\n",
    "\n",
    "Reasoning means we use inference rules that use the axioms within the ontology to extract new knowledge (which may or may not be explicitly stated). The most basic operation an OWL reasoner does is to perform _classification_, i.e., it identifies for each class its sub- and super-classes. OWL reasoners can also be used to query ontologies, and find, for example, sub-classes of named classes or complex class descriptions.\n",
    "\n",
    "Reasoning in OWL may be very complex. For example, automated reasoning in OWL 2 is 2NEXPTIME-complete, and will therefore not scale well on arbitrary ontologies. Fortunately, many optimized reasoners exist, so the theoretical limitations are not always a barrier in practise. Nevertheless, OWL 2 comes with several sub-langes (OWL 2 EL, QL, and RL) which guarantee more efficient automated reasoning. In particular the OWL 2 EL language profile is widely used in biological and biomedical ontologies.\n",
    "\n",
    "In the first box below, we will just create a new OWL reasoner object (using the OWL 2 EL reasoner [ELK](https://github.com/liveontologies/elk-reasoner)). In the second box, we will use this reasoner to find all the subclasses of the \"Mode of inheritance\" class in the Human Phenotype Ontology, and output the classes and their labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "org&#8203;.semanticweb&#8203;.elk&#8203;.owlapi&#8203;.ElkReasoner&#8203;@620e3f62"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "// import what we need\n",
    "import java.net.*\n",
    "import org.semanticweb.owlapi.model.parameters.*\n",
    "import org.semanticweb.elk.owlapi.ElkReasonerFactory;\n",
    "import org.semanticweb.elk.owlapi.ElkReasonerConfiguration\n",
    "import org.semanticweb.elk.reasoner.config.*\n",
    "import org.semanticweb.owlapi.apibinding.OWLManager;\n",
    "import org.semanticweb.owlapi.reasoner.*\n",
    "import org.semanticweb.owlapi.reasoner.structural.StructuralReasoner\n",
    "import org.semanticweb.owlapi.vocab.OWLRDFVocabulary;\n",
    "import org.semanticweb.owlapi.model.*;\n",
    "import org.semanticweb.owlapi.io.*;\n",
    "import org.semanticweb.owlapi.owllink.*;\n",
    "import org.semanticweb.owlapi.util.*;\n",
    "import org.semanticweb.owlapi.search.*;\n",
    "import org.semanticweb.owlapi.manchestersyntax.renderer.*;\n",
    "import org.semanticweb.owlapi.reasoner.structural.*\n",
    "\n",
    "ConsoleProgressMonitor progressMonitor = new ConsoleProgressMonitor()\n",
    "OWLReasonerConfiguration config = new SimpleConfiguration(progressMonitor)\n",
    "ElkReasonerFactory f1 = new ElkReasonerFactory()\n",
    "reasoner = f1.createReasoner(ont,config)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading ...\n",
      "    1%\n",
      "    2%\n",
      "    3%\n",
      "    4%\n",
      "    5%\n",
      "    6%\n",
      "    7%\n",
      "    8%\n",
      "    9%\n",
      "    10%\n",
      "    11%\n",
      "    12%\n",
      "    13%\n",
      "    14%\n",
      "    15%\n",
      "    16%\n",
      "    17%\n",
      "    18%\n",
      "    19%\n",
      "    20%\n",
      "    21%\n",
      "    22%\n",
      "    23%\n",
      "    24%\n",
      "    25%\n",
      "    26%\n",
      "    27%\n",
      "    28%\n",
      "    29%\n",
      "    30%\n",
      "    31%\n",
      "    32%\n",
      "    33%\n",
      "    34%\n",
      "    35%\n",
      "    36%\n",
      "    37%\n",
      "    38%\n",
      "    39%\n",
      "    40%\n",
      "    41%\n",
      "    42%\n",
      "    43%\n",
      "    44%\n",
      "    45%\n",
      "    46%\n",
      "    47%\n",
      "    48%\n",
      "    49%\n",
      "    50%\n",
      "    51%\n",
      "    52%\n",
      "    53%\n",
      "    54%\n",
      "    55%\n",
      "    56%\n",
      "    57%\n",
      "    58%\n",
      "    59%\n",
      "    60%\n",
      "    61%\n",
      "    62%\n",
      "    63%\n",
      "    64%\n",
      "    65%\n",
      "    66%\n",
      "    67%\n",
      "    68%\n",
      "    69%\n",
      "    70%\n",
      "    71%\n",
      "    72%\n",
      "    73%\n",
      "    74%\n",
      "    75%\n",
      "    76%\n",
      "    77%\n",
      "    78%\n",
      "    79%\n",
      "    80%\n",
      "    81%\n",
      "    82%\n",
      "    83%\n",
      "    84%\n",
      "    85%\n",
      "    86%\n",
      "    87%\n",
      "    88%\n",
      "    89%\n",
      "    90%\n",
      "    91%\n",
      "    92%\n",
      "    93%\n",
      "    94%\n",
      "    95%\n",
      "    96%\n",
      "    97%\n",
      "    98%\n",
      "    99%\n",
      "    ... finished\n",
      "Gonosomal inheritance (http://purl.obolibrary.org/obo/HP_0010985)\n",
      "Digenic inheritance (http://purl.obolibrary.org/obo/HP_0010984)\n",
      "Oligogenic inheritance (http://purl.obolibrary.org/obo/HP_0010983)\n",
      "Polygenic inheritance (http://purl.obolibrary.org/obo/HP_0010982)\n",
      "X-linked inheritance (http://purl.obolibrary.org/obo/HP_0001417)\n",
      "X-linked recessive inheritance (http://purl.obolibrary.org/obo/HP_0001419)\n",
      "Autosomal recessive inheritance (http://purl.obolibrary.org/obo/HP_0000007)\n",
      "Autosomal dominant (http://purl.obolibrary.org/obo/HP_0000006)\n",
      "Autosomal dominant inheritance (http://purl.obolibrary.org/obo/HP_0000006)\n",
      "Male-limited autosomal dominant (http://purl.obolibrary.org/obo/HP_0001475)\n",
      "Autosomal dominant contiguous gene syndrome (http://purl.obolibrary.org/obo/HP_0001452)\n",
      "Sex-limited autosomal dominant (http://purl.obolibrary.org/obo/HP_0001470)\n",
      "Familial predisposition (http://purl.obolibrary.org/obo/HP_0001472)\n",
      "Y-linked inheritance (http://purl.obolibrary.org/obo/HP_0001450)\n",
      "Autosomal dominant inheritance with paternal imprinting (http://purl.obolibrary.org/obo/HP_0012274)\n",
      "Autosomal dominant inheritance with maternal imprinting (http://purl.obolibrary.org/obo/HP_0012275)\n",
      "Somatic mutation (http://purl.obolibrary.org/obo/HP_0001428)\n",
      "Mitochondrial inheritance (http://purl.obolibrary.org/obo/HP_0001427)\n",
      "X-linked dominant inheritance (http://purl.obolibrary.org/obo/HP_0001423)\n",
      "Multifactorial inheritance (http://purl.obolibrary.org/obo/HP_0001426)\n",
      "Multifactorial Inheritance (http://purl.obolibrary.org/obo/HP_0001426)\n",
      "Genetic anticipation with paternal anticipation bias (http://purl.obolibrary.org/obo/HP_0003744)\n",
      "Heterogeneous (http://purl.obolibrary.org/obo/HP_0001425)\n",
      "Sporadic (http://purl.obolibrary.org/obo/HP_0003745)\n",
      "Somatic mosaicism (http://purl.obolibrary.org/obo/HP_0001442)\n",
      "Genetic anticipation (http://purl.obolibrary.org/obo/HP_0003743)\n",
      "Anticipation, Genetic (http://purl.obolibrary.org/obo/HP_0003743)\n",
      "Autosomal dominant somatic cell mutation (http://purl.obolibrary.org/obo/HP_0001444)\n",
      "Contiguous gene syndrome (http://purl.obolibrary.org/obo/HP_0001466)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "[&lt;http://purl&#8203;.obolibrary&#8203;.org/obo/HP&#8203;_0010985&gt;, &lt;http://purl&#8203;.obolibrary&#8203;.org/obo/HP&#8203;_0010984&gt;, &lt;http://purl&#8203;.obolibrary&#8203;.org/obo/HP&#8203;_0010983&gt;, &lt;http://purl&#8203;.obolibrary&#8203;.org/obo/HP&#8203;_0010982&gt;, &lt;http://purl&#8203;.obolibrary&#8203;.org/obo/HP&#8203;_0001417&gt;, &lt;http://purl&#8203;.obolibrary&#8203;.org/obo/HP&#8203;_0001419&gt;, &lt;http://purl&#8203;.obolibrary&#8203;.org/obo/HP&#8203;_0000007&gt;, &lt;http://purl&#8203;.obolibrary&#8203;.org/obo/HP&#8203;_0000006&gt;, &lt;http://purl&#8203;.obolibrary&#8203;.org/obo/HP&#8203;_0001475&gt;, &lt;http://purl&#8203;.obolibrary&#8203;.org/obo/HP&#8203;_0001452&gt;, &lt;http://purl&#8203;.obolibrary&#8203;.org/obo/HP&#8203;_0001470&gt;, &lt;http://purl&#8203;.obolibrary&#8203;.org/obo/HP&#8203;_0001472&gt;, &lt;http://purl&#8203;.obolibrary&#8203;.org/obo/HP&#8203;_0001450&gt;, &lt;http://purl&#8203;.obolibrary&#8203;.org/obo/HP&#8203;_0012274&gt;, &lt;http://purl&#8203;.obolibrary&#8203;.org/obo/HP&#8203;_0012275&gt;, &lt;http://purl&#8203;.obolibrary&#8203;.org/obo/HP&#8203;_0001428&gt;, &lt;http://purl&#8203;.obolibrary&#8203;.org/obo/HP&#8203;_0001427&gt;, &lt;http://purl&#8203;.obolibrary&#8203;.org/obo/HP&#8203;_0001423&gt;, &lt;http://purl&#8203;.obolibrary&#8203;.org/obo/HP&#8203;_0001426&gt;, &lt;http://purl&#8203;.obolibrary&#8203;.org/obo/HP&#8203;_0003744&gt;, &lt;http://purl&#8203;.obolibrary&#8203;.org/obo/HP&#8203;_0001425&gt;, &lt;http://purl&#8203;.obolibrary&#8203;.org/obo/HP&#8203;_0003745&gt;, &lt;http://purl&#8203;.obolibrary&#8203;.org/obo/HP&#8203;_0001442&gt;, owl:Nothing, &lt;http://purl&#8203;.obolibrary&#8203;.org/obo/HP&#8203;_0003743&gt;, &lt;http://purl&#8203;.obolibrary&#8203;.org/obo/HP&#8203;_0001444&gt;, &lt;http://purl&#8203;.obolibrary&#8203;.org/obo/HP&#8203;_0001466&gt;]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "// import what we need\n",
    "import java.net.*\n",
    "import org.semanticweb.owlapi.model.parameters.*\n",
    "import org.semanticweb.elk.owlapi.ElkReasonerFactory;\n",
    "import org.semanticweb.elk.owlapi.ElkReasonerConfiguration\n",
    "import org.semanticweb.elk.reasoner.config.*\n",
    "import org.semanticweb.owlapi.apibinding.OWLManager;\n",
    "import org.semanticweb.owlapi.reasoner.*\n",
    "import org.semanticweb.owlapi.reasoner.structural.StructuralReasoner\n",
    "import org.semanticweb.owlapi.vocab.OWLRDFVocabulary;\n",
    "import org.semanticweb.owlapi.model.*;\n",
    "import org.semanticweb.owlapi.io.*;\n",
    "import org.semanticweb.owlapi.owllink.*;\n",
    "import org.semanticweb.owlapi.util.*;\n",
    "import org.semanticweb.owlapi.search.*;\n",
    "import org.semanticweb.owlapi.manchestersyntax.renderer.*;\n",
    "import org.semanticweb.owlapi.reasoner.structural.*\n",
    "\n",
    "// get all subclasses of \"Mode of inheritance\", including all descendant classes (direct set to \"false\")\n",
    "reasoner.getSubClasses(fac.getOWLClass(IRI.create(\"http://purl.obolibrary.org/obo/HP_0000005\")), false).getFlattened().each { cl ->\n",
    "  def ciri = cl.getIRI()\n",
    "  // now we get the label (rdfs:label) for each of the classes and print to stdout\n",
    "  EntitySearcher.getAnnotations(cl, ont, fac.getRDFSLabel()).each { a ->\n",
    "    OWLAnnotationValue val = a.getValue()\n",
    "    if(val instanceof OWLLiteral) {\n",
    "      def label = ((OWLLiteral)val).getLiteral()\n",
    "      println \"$label ($ciri)\"\n",
    "    }\n",
    "  }\n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can play with this code a bit to find subclasses as well as superclasses for \"Mode of inheritance\" and other classes. A class in OWL does not have to have a URI (these are the named classes) but can also be a complex class description; for example, you can query for subclasses of phenotypes affecting some part of the \"heart\" by querying describing the class using quantifiers, relations, conjunction, disjunction, or negation, and retrieve classes that satisfy this description.\n",
    "\n",
    "\n",
    "Next, we will classify the ontology (from our data package) using the ELK reasoner, and store an inferred version of the ontology in the user home directory. The inferred version will be classified, i.e., all implied sub- and super-class relations are inferred and a transitive reduction is performed so that the resulting taxonomy does not contain redundant sub- or super-class relations.\n",
    "\n",
    "If you do not want to store the file in `$HOME/phenomenet-inferred.owl`, change the code below accordingly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "// import what we need\n",
    "import java.net.*\n",
    "import org.semanticweb.owlapi.model.parameters.*\n",
    "import org.semanticweb.elk.owlapi.ElkReasonerFactory;\n",
    "import org.semanticweb.elk.owlapi.ElkReasonerConfiguration\n",
    "import org.semanticweb.elk.reasoner.config.*\n",
    "import org.semanticweb.owlapi.apibinding.OWLManager;\n",
    "import org.semanticweb.owlapi.reasoner.*\n",
    "import org.semanticweb.owlapi.reasoner.structural.StructuralReasoner\n",
    "import org.semanticweb.owlapi.vocab.OWLRDFVocabulary;\n",
    "import org.semanticweb.owlapi.model.*;\n",
    "import org.semanticweb.owlapi.io.*;\n",
    "import org.semanticweb.owlapi.owllink.*;\n",
    "import org.semanticweb.owlapi.util.*;\n",
    "import org.semanticweb.owlapi.search.*;\n",
    "import org.semanticweb.owlapi.manchestersyntax.renderer.*;\n",
    "import org.semanticweb.owlapi.reasoner.structural.*\n",
    "    \n",
    "// write the inferred version of the ontology to a file (phenomenet-inferred.owl in the user homedir)\n",
    "def homeDir = System.getProperty(\"user.home\")\n",
    "File outfile = new File (homeDir, \"phenomenet-inferred.owl\")\n",
    "def outont = manager.createOntology(new IRI(\"http://aber-owl.net/ismb-tutorial/phenomenet-inferred.owl\"))\n",
    "InferredOntologyGenerator generator = new InferredOntologyGenerator(reasoner, [new InferredSubClassAxiomGenerator()])\n",
    "generator.fillOntology(fac, outont)\n",
    "\n",
    "manager.saveOntology(outont, IRI.create(\"file:\"+outfile.getAbsolutePath()))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Semantic similarity\n",
    "\n",
    "We use the [Semantic Measures Library](http://semantic-measures-library.org) to compute semantic similarity over the inferred ontology. We first generate some basic objects we need to define semantic similarity, and populate the semantic graph (used by the Semantic Measures Library) with the inferred taxonomy generated above.\n",
    "\n",
    "Note: while it is possible to compute semantic similarity over the non-inferred version of the ontology, most similarity measures crucially rely on a graph structure that resembles (or approximates) the semantic content of the ontology. Using a non-inferred version will hide this content. We therefore _always_ recommend to apply semantic similarity measures on the fully inferred version of an ontology.\n",
    "\n",
    "First we set up some basic data structures, mainly the graph over which the similarity will be computed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "http://aber-owl&#8203;.net/ismb-tutorial/\n",
       "Vertices\n",
       "\tTotal   : 0  \n",
       "Edges \t  : 0\n",
       "\n"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import org.openrdf.model.vocabulary.*\n",
    "import slib.sglib.io.loader.*\n",
    "import slib.sml.sm.core.metrics.ic.utils.*\n",
    "import slib.sml.sm.core.utils.*\n",
    "import slib.sglib.io.loader.bio.obo.*\n",
    "import org.openrdf.model.URI\n",
    "import slib.graph.algo.extraction.rvf.instances.*\n",
    "import slib.sglib.algo.graph.utils.*\n",
    "import slib.utils.impl.Timer\n",
    "import slib.graph.algo.extraction.utils.*\n",
    "import slib.graph.model.graph.*\n",
    "import slib.graph.model.repo.*\n",
    "import slib.graph.model.impl.graph.memory.*\n",
    "import slib.sml.sm.core.engine.*\n",
    "import slib.graph.io.conf.*\n",
    "import slib.graph.model.impl.graph.elements.*\n",
    "import slib.graph.algo.extraction.rvf.instances.impl.*\n",
    "import slib.graph.model.impl.repo.*\n",
    "import slib.graph.io.util.*\n",
    "import slib.graph.io.loader.*\n",
    "    \n",
    "factory = URIFactoryMemory.getSingleton()\n",
    "URI graph_uri = factory.getURI(\"http://aber-owl.net/ismb-tutorial/\")\n",
    "graph = new GraphMemory(graph_uri)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we populate the graph with the inferred ontology. We use the RDF_XML format which we generated before. There are some warning messages appearing here which can be safely ignored (because the INFO logger seems to write to stderr by default)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.RDFLoader - -------------------------------------\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.RDFLoader -  RDF Loader\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.RDFLoader - -------------------------------------\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.RDFLoader - Populate graph http://aber-owl.net/ismb-tutorial/ from /home/leechuck/phenomenet-inferred.owl\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.RDFLoader - Parser loaded for: RDF/XML (mimeTypes=application/rdf+xml, application/xml; ext=rdf, rdfs, owl, xml)\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.RDFLoader - Parsing RDF file...\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.SlibRdfHandler - Start Process\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.SlibRdfHandler - 100000 statements already loaded\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.SlibRdfHandler - Number of vertices: 36716\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.SlibRdfHandler - Number of edges   : 100000\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.SlibRdfHandler - 200000 statements already loaded\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.SlibRdfHandler - Number of vertices: 79744\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.SlibRdfHandler - Number of edges   : 200000\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.SlibRdfHandler - 300000 statements already loaded\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.SlibRdfHandler - Number of vertices: 115106\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.SlibRdfHandler - Number of edges   : 300000\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.SlibRdfHandler - 400000 statements already loaded\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.SlibRdfHandler - Number of vertices: 157633\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.SlibRdfHandler - Number of edges   : 400000\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.SlibRdfHandler - 500000 statements already loaded\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.SlibRdfHandler - Number of vertices: 189817\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.SlibRdfHandler - Number of edges   : 500000\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.SlibRdfHandler - 600000 statements already loaded\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.SlibRdfHandler - Number of vertices: 220936\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.SlibRdfHandler - Number of edges   : 600000\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.SlibRdfHandler - Ending Process 612744 statements loaded \n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.SlibRdfHandler - vertices: 224751\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.SlibRdfHandler - edges   : 612744\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.SlibRdfHandler - Skipped (statement involving non URI ressources) : 0\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.RDFLoader - Graph http://aber-owl.net/ismb-tutorial/ populated by RDF data \n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.io.loader.rdf.RDFLoader - -------------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "http://aber-owl&#8203;.net/ismb-tutorial/\n",
       "Vertices\n",
       "\tTotal   : 224751  {e&#8203;.g&#8203;. http://purl&#8203;.obolibrary&#8203;.org/obo/HP&#8203;_0011770}\n",
       "Edges \t  : 612744\n",
       "\n"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import org.openrdf.model.vocabulary.*\n",
    "import slib.sglib.io.loader.*\n",
    "import slib.sml.sm.core.metrics.ic.utils.*\n",
    "import slib.sml.sm.core.utils.*\n",
    "import slib.sglib.io.loader.bio.obo.*\n",
    "import org.openrdf.model.URI\n",
    "import slib.graph.algo.extraction.rvf.instances.*\n",
    "import slib.sglib.algo.graph.utils.*\n",
    "import slib.utils.impl.Timer\n",
    "import slib.graph.algo.extraction.utils.*\n",
    "import slib.graph.model.graph.*\n",
    "import slib.graph.model.repo.*\n",
    "import slib.graph.model.impl.graph.memory.*\n",
    "import slib.sml.sm.core.engine.*\n",
    "import slib.graph.io.conf.*\n",
    "import slib.graph.model.impl.graph.elements.*\n",
    "import slib.graph.algo.extraction.rvf.instances.impl.*\n",
    "import slib.graph.model.impl.repo.*\n",
    "import slib.graph.io.util.*\n",
    "import slib.graph.io.loader.*\n",
    "    \n",
    "def homeDir = System.getProperty(\"user.home\")\n",
    "File ontfile = new File (homeDir, \"phenomenet-inferred.owl\")\n",
    "GDataConf graphconf = new GDataConf(GFormat.RDF_XML, ontfile.getCanonicalPath())\n",
    "GraphLoaderGeneric.populate(graphconf, graph)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, ontologies (or the taxonomies underlying them) have more than one root (below `owl:Thing`). For example, the Gene Ontology has three roots, _Biological process_, _Molecular function_, and _Cellular Component_. To make classes comparable when they do not share a root (necessary for example in path-based similarity measures), we generate a new artificial root and make this the new root of our semantic graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.algo.extraction.utils.GraphActionExecutor - -------------------------------------\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.algo.extraction.utils.GraphActionExecutor - Rerooting\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.algo.extraction.utils.GraphActionExecutor - -------------------------------------\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.algo.extraction.utils.GraphActionExecutor - Fetching root node, uri: http://aber-owl.net/ismb-tutorial/virtualRoot\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.algo.extraction.utils.RooterDAG - Rooting taxonomic Graph using http://aber-owl.net/ismb-tutorial/virtualRoot\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.algo.validator.dag.ValidatorDAG - Number of roots 3\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.algo.validator.dag.ValidatorDAG - Starting process from 168068 vertices\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.algo.validator.dag.ValidatorDAG - isDag : true\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.algo.extraction.utils.RooterDAG - Number of roots detected: 3\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.algo.extraction.utils.RooterDAG - Rooting performed using http://aber-owl.net/ismb-tutorial/virtualRoot as root 3 edges created\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.algo.validator.dag.ValidatorDAG - Number of roots 1\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.algo.validator.dag.ValidatorDAG - isDag : true\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.algo.extraction.utils.GraphActionExecutor - Rerooting performed\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.algo.extraction.utils.GraphActionExecutor - -------------------------------------\n"
     ]
    }
   ],
   "source": [
    "import org.openrdf.model.vocabulary.*\n",
    "import slib.sglib.io.loader.*\n",
    "import slib.sml.sm.core.metrics.ic.utils.*\n",
    "import slib.sml.sm.core.utils.*\n",
    "import slib.sglib.io.loader.bio.obo.*\n",
    "import org.openrdf.model.URI\n",
    "import slib.graph.algo.extraction.rvf.instances.*\n",
    "import slib.sglib.algo.graph.utils.*\n",
    "import slib.utils.impl.Timer\n",
    "import slib.graph.algo.extraction.utils.*\n",
    "import slib.graph.model.graph.*\n",
    "import slib.graph.model.repo.*\n",
    "import slib.graph.model.impl.graph.memory.*\n",
    "import slib.sml.sm.core.engine.*\n",
    "import slib.graph.io.conf.*\n",
    "import slib.graph.model.impl.graph.elements.*\n",
    "import slib.graph.algo.extraction.rvf.instances.impl.*\n",
    "import slib.graph.model.impl.repo.*\n",
    "import slib.graph.io.util.*\n",
    "import slib.graph.io.loader.*\n",
    "\n",
    "\n",
    "URI virtualRoot = factory.getURI(\"http://aber-owl.net/ismb-tutorial/virtualRoot\")\n",
    "graph.addV(virtualRoot)\n",
    "GAction rooting = new GAction(GActionType.REROOTING)\n",
    "rooting.addParameter(\"root_uri\", virtualRoot.stringValue())\n",
    "GraphActionExecutor.applyAction(factory, rooting, graph)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we configure the semantic similarity measure we want to use.\n",
    "\n",
    "Because we do not have instances, we use only the ontology structure to determine how specific a class is (`IC_Conf_Topo` below for use of a specificity measure based only on the topology of the semantic graph). We use Resnik's similarity measure (\"similarity of two classes is the specificity value of their most specific common ancestor\")."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - ================================================================\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - Loading Semantic Measures Engine for graph http://aber-owl.net/ismb-tutorial/\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - ================================================================\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - Graph Info: \n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - http://aber-owl.net/ismb-tutorial/\n",
      "Vertices\n",
      "\tTotal   : 224752  {e.g. http://purl.obolibrary.org/obo/HP_0011770}\n",
      "Edges \t  : 612747\n",
      "\n",
      "\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - ---------------------------------------------------------------\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - Pre-processing\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - ---------------------------------------------------------------\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - Computing classes...\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - Computing instances...\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - Classes  : 224751\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - Instances: 1\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - ---------------------------------------------------------------\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - Inferences \n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - ---------------------------------------------------------------\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - Inferring ancestors...\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.algo.extraction.rvf.AncestorEngine - Checking Treatment coherency\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.algo.extraction.rvf.AncestorEngine - Incoherencies : 0\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - Inferring descendants...\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.algo.extraction.rvf.DescendantEngine - Checking Treatment coherency\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.algo.extraction.rvf.DescendantEngine - Incoherencies : 0\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - Inferring Conceptual Leaves...\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - ---------------------------------------------------------------\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - Engine initialized\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - ================================================================\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "slib&#8203;.sml&#8203;.sm&#8203;.core&#8203;.engine&#8203;.SM&#8203;_Engine&#8203;@649d4c92"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import org.openrdf.model.vocabulary.*\n",
    "import slib.sglib.io.loader.*\n",
    "import slib.sml.sm.core.metrics.ic.utils.*\n",
    "import slib.sml.sm.core.utils.*\n",
    "import slib.sglib.io.loader.bio.obo.*\n",
    "import org.openrdf.model.URI\n",
    "import slib.graph.algo.extraction.rvf.instances.*\n",
    "import slib.sglib.algo.graph.utils.*\n",
    "import slib.utils.impl.Timer\n",
    "import slib.graph.algo.extraction.utils.*\n",
    "import slib.graph.model.graph.*\n",
    "import slib.graph.model.repo.*\n",
    "import slib.graph.model.impl.graph.memory.*\n",
    "import slib.sml.sm.core.engine.*\n",
    "import slib.graph.io.conf.*\n",
    "import slib.graph.model.impl.graph.elements.*\n",
    "import slib.graph.algo.extraction.rvf.instances.impl.*\n",
    "import slib.graph.model.impl.repo.*\n",
    "import slib.graph.io.util.*\n",
    "import slib.graph.io.loader.*\n",
    "\n",
    "// configure the semantic similarity measure: use Resnik's (extrinsic) information content measure, and Resnik's similarity measure\n",
    "icConf = new IC_Conf_Topo(\"Sanchez\", SMConstants.FLAG_ICI_SANCHEZ_2011)\n",
    "smConfPairwise = new SMconf(\"Resnik\", SMConstants.FLAG_SIM_PAIRWISE_DAG_NODE_RESNIK_1995 )\n",
    "smConfPairwise.setICconf(icConf)\n",
    "\n",
    "// initialize the engine\n",
    "engine = new SM_Engine(graph)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can compare some classes and determine how similar they are. Feel free to change the code to some new classes and see how the similarity changes.\n",
    "\n",
    "You can use classes from the Human Phenotype Ontology (HPO) and the Mammalian Phenotype Ontology (MPO) for similarity computation between phenotypes. Since we are using the PhenomeNET ontology (see [our paper](http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1005500)), classes in both ontologies are actually comparable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - ---------------------------------------------------------------\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - computing IC Sanchez\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - ---------------------------------------------------------------\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - Class name slib.sml.sm.core.metrics.ic.topo.ICi_sanchez_2011\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - Computing Nb Reachable Leaves : start\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.algo.extraction.rvf.DescendantEngine - Retrieving all reachable leaves\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.algo.extraction.rvf.DescendantEngine - Propagation of leave counts start from 168068 leaves on 224752 concepts\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - Computing Nb Reachable Leaves : end\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - Checking null or infinite in the ICs computed\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - ic Sanchez computed\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - ---------------------------------------------------------------\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Similarity between 'Ear pain' and 'mid frequency hearing loss': 8.617973362046424\n"
     ]
    }
   ],
   "source": [
    "// similarity between two classes from HPO\n",
    "cl1 = factory.getURI(\"http://purl.obolibrary.org/obo/HP_0030766\") // Ear pain\n",
    "cl2 = factory.getURI(\"http://purl.obolibrary.org/obo/HP_0012781\") // mid frequency hearing loss\n",
    "println \"Similarity between 'Ear pain' and 'mid frequency hearing loss': \"+engine.compare(smConfPairwise, cl1, cl2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Similarity between 'Ear pain' and 'tetralogy of fallot': 5.755799731300896\n"
     ]
    }
   ],
   "source": [
    "// similarity between two classes from HPO\n",
    "cl2 = factory.getURI(\"http://purl.obolibrary.org/obo/HP_0001636\") // tetralogy of fallot\n",
    "println \"Similarity between 'Ear pain' and 'tetralogy of fallot': \"+engine.compare(smConfPairwise, cl1, cl2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Similarity between 'abnormal cardiac muscle relaxation' (MP) and 'tetralogy of fallot' (HP): 6.185691113293361\n"
     ]
    }
   ],
   "source": [
    "// similarity between two classes, one from HPO and one from MP\n",
    "cl1 = factory.getURI(\"http://purl.obolibrary.org/obo/MP_0004084\") // abnormal cardiac muscle relaxation\n",
    "cl2 = factory.getURI(\"http://purl.obolibrary.org/obo/HP_0001636\") // tetralogy of fallot\n",
    "println \"Similarity between 'abnormal cardiac muscle relaxation' (MP) and 'tetralogy of fallot' (HP): \"+engine.compare(smConfPairwise, cl1, cl2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Similarity between 'ventricular septal defect' (MP) and 'tetralogy of fallot' (HP): 11.744447815899305\n"
     ]
    }
   ],
   "source": [
    "// similarity between two classes, one from HPO and one from MP\n",
    "cl1 = factory.getURI(\"http://purl.obolibrary.org/obo/MP_0010402\") // ventricular septal defect\n",
    "println \"Similarity between 'ventricular septal defect' (MP) and 'tetralogy of fallot' (HP): \"+engine.compare(smConfPairwise, cl1, cl2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far, we have only looked at comparisons between two individual classes. This may be nice if we just want to understand how the classes in an ontology are related, but usually we want to compare _things that are annotated with classes_ from an ontology.\n",
    "\n",
    "We now download mouse phenotypes resulting from single gene knockouts (note: replace `http://www.informatics.jax.org/downloads/reports/MGI_GenePheno.rpt` with a `file:...` URL in case of slow Internet connection; you can find the file `MGI_GenePheno.rpt` in our data package), and add each gene as an instance of its phenotype classes (using an `rdf:type` edge). \n",
    "\n",
    "Important: this is __not__ the right way to do this in an ontology but is an artifact of the Semantic Measures Library. When building an ontology, a _gene_ is not an _instance_ of a phenotype, but will be related to the phenotype through other kinds of relations. In the Semantic Measures Library, we will treat them as instances, however."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import org.openrdf.model.vocabulary.*\n",
    "import slib.sglib.io.loader.*\n",
    "import slib.sml.sm.core.metrics.ic.utils.*\n",
    "import slib.sml.sm.core.utils.*\n",
    "import slib.sglib.io.loader.bio.obo.*\n",
    "import org.openrdf.model.URI\n",
    "import slib.graph.algo.extraction.rvf.instances.*\n",
    "import slib.sglib.algo.graph.utils.*\n",
    "import slib.utils.impl.Timer\n",
    "import slib.graph.algo.extraction.utils.*\n",
    "import slib.graph.model.graph.*\n",
    "import slib.graph.model.repo.*\n",
    "import slib.graph.model.impl.graph.memory.*\n",
    "import slib.sml.sm.core.engine.*\n",
    "import slib.graph.io.conf.*\n",
    "import slib.graph.model.impl.graph.elements.*\n",
    "import slib.graph.algo.extraction.rvf.instances.impl.*\n",
    "import slib.graph.model.impl.repo.*\n",
    "import slib.graph.io.util.*\n",
    "import slib.graph.io.loader.*\n",
    "\n",
    "// now we download mouse phenotype annotations and add them to our graph as instances (using rdf:type)\n",
    "new URL (\"file:misc/MGI_GenePheno.rpt\").getText().splitEachLine(\"\\t\") { line ->\n",
    "  def geneid = line[6]\n",
    "  def idUri = factory.getURI(\"http://phenomebrowser.net/ismb-tutorial/gene/\"+geneid)\n",
    "  def pheno = line[4].replaceAll(\":\",\"_\")\n",
    "  def phenoUri = factory.getURI(\"http://purl.obolibrary.org/obo/\"+pheno)\n",
    "  Edge e = new Edge(idUri, RDF.TYPE, phenoUri)\n",
    "  graph.addE(e)\n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the presence of instances, we can change our class specificity measure. We now use Resnik's specificity measure, which is just the information content of a class (i.e., `-log(p(c))` where `p(c)` is the probability to have an instance annotated with the class. This is a specificity measure that relies on instances, and is therefore a _corpus-based_ measure (using `IC_Conf_Corpus` below).\n",
    "\n",
    "We also configure a group-wise measure now, using the Best-Matching Average strategy. This will allow us to compare sets of classes to sets of classes.\n",
    "\n",
    "Finally, the `InstanceAccessor` will allow us to identify all instances of our classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - ================================================================\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - Loading Semantic Measures Engine for graph http://aber-owl.net/ismb-tutorial/\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - ================================================================\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - Graph Info: \n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - http://aber-owl.net/ismb-tutorial/\n",
      "Vertices\n",
      "\tTotal   : 237689  {e.g. http://purl.obolibrary.org/obo/HP_0011770}\n",
      "Edges \t  : 779607\n",
      "\n",
      "\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - ---------------------------------------------------------------\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - Pre-processing\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - ---------------------------------------------------------------\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - Computing classes...\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - Computing instances...\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - Classes  : 225475\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - Instances: 12938\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - ---------------------------------------------------------------\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - Inferences \n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - ---------------------------------------------------------------\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - Inferring ancestors...\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.algo.extraction.rvf.AncestorEngine - Checking Treatment coherency\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.algo.extraction.rvf.AncestorEngine - Incoherencies : 0\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - Inferring descendants...\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.algo.extraction.rvf.DescendantEngine - Checking Treatment coherency\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.graph.algo.extraction.rvf.DescendantEngine - Incoherencies : 0\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - Inferring Conceptual Leaves...\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - ---------------------------------------------------------------\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - Engine initialized\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - ================================================================\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "slib&#8203;.graph&#8203;.algo&#8203;.extraction&#8203;.rvf&#8203;.instances&#8203;.impl&#8203;.InstanceAccessor&#8203;_RDF&#8203;_TYPE&#8203;@7e97ac3d"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import org.openrdf.model.vocabulary.*\n",
    "import slib.sglib.io.loader.*\n",
    "import slib.sml.sm.core.metrics.ic.utils.*\n",
    "import slib.sml.sm.core.utils.*\n",
    "import slib.sglib.io.loader.bio.obo.*\n",
    "import org.openrdf.model.URI\n",
    "import slib.graph.algo.extraction.rvf.instances.*\n",
    "import slib.sglib.algo.graph.utils.*\n",
    "import slib.utils.impl.Timer\n",
    "import slib.graph.algo.extraction.utils.*\n",
    "import slib.graph.model.graph.*\n",
    "import slib.graph.model.repo.*\n",
    "import slib.graph.model.impl.graph.memory.*\n",
    "import slib.sml.sm.core.engine.*\n",
    "import slib.graph.io.conf.*\n",
    "import slib.graph.model.impl.graph.elements.*\n",
    "import slib.graph.algo.extraction.rvf.instances.impl.*\n",
    "import slib.graph.model.impl.repo.*\n",
    "import slib.graph.io.util.*\n",
    "import slib.graph.io.loader.*\n",
    "\n",
    "\n",
    "// now that we have instances/annotations, we switch to Resnik's information content measure\n",
    "icConf = new IC_Conf_Corpus(\"ResnikIC\", SMConstants.FLAG_IC_ANNOT_RESNIK_1995_NORMALIZED)\n",
    "smConfPairwise.setICconf(icConf)\n",
    "// using Best-Matching Average to merge annotations when comparing sets of classes\n",
    "smConfGroupwise = new SMconf(\"BMA\", SMConstants.FLAG_SIM_GROUPWISE_BMA)\n",
    "// and re-initialize the graph\n",
    "engine = new SM_Engine(graph)\n",
    "\n",
    "// we need this to find our instances again\n",
    "ia = new InstanceAccessor_RDF_TYPE(graph)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now compute again similarity between two classes. Note how the similarity values change. Feel free to experiment with this and compare different classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - ---------------------------------------------------------------\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - computing IC ResnikIC\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - ---------------------------------------------------------------\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - Class name slib.sml.sm.core.metrics.ic.annot.IC_annot_resnik_1995_Normalized\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - Checking null or infinite in the ICs computed\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - ic ResnikIC computed\n",
      "[SciJava-24880340-Thread-org.scijava.jupyter.kernel.evaluator.ScijavaEvaluator] INFO slib.sml.sm.core.engine.SM_Engine - ---------------------------------------------------------------\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Similarity (Resnik) between 'abnormal cardiac muscle relaxation' (MP) and 'tetralogy of fallot' (HP): 0.30687152162569886\n"
     ]
    }
   ],
   "source": [
    "// similarity between two classes, one from HPO and one from MP\n",
    "cl1 = factory.getURI(\"http://purl.obolibrary.org/obo/MP_0004084\") // abnormal cardiac muscle relaxation\n",
    "cl2 = factory.getURI(\"http://purl.obolibrary.org/obo/HP_0001636\") // tetralogy of fallot\n",
    "println \"Similarity (Resnik) between 'abnormal cardiac muscle relaxation' (MP) and 'tetralogy of fallot' (HP): \"+engine.compare(smConfPairwise, cl1, cl2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Similarity (Resnik) between 'ventricular septal defect' (MP) and 'tetralogy of fallot' (HP): 0.5605218915985221\n"
     ]
    }
   ],
   "source": [
    "// similarity between two classes, one from HPO and one from MP\n",
    "cl1 = factory.getURI(\"http://purl.obolibrary.org/obo/MP_0010402\") // ventricular septal defect\n",
    "println \"Similarity (Resnik) between 'ventricular septal defect' (MP) and 'tetralogy of fallot' (HP): \"+engine.compare(smConfPairwise, cl1, cl2)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the presence of mouse genes as instances in our semantic graph, we can do some more sophisticated analyses. Here, we define a patient through a set of phenotypes (i.e., clinical signs and symptoms). The patient has a rare, heritable form of macular dystrophy without known etiology (see https://www.omim.org/entry/153840). We hypothesize that there may be a mouse knockout that resembles this disease, and possibly the human ortholog of the mouse gene may also be involved in this form of macular dystrophy.\n",
    "\n",
    "We compare our patient to _all_ instances in our graph (i.e., all mouse knockouts) using our measure of phenotypic similarity. We then sort the resulting list by their similarity score in descending order, and output the top 10 predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Highest ranking genes for patient 1 (macular dystrophy): \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>table.converted {color: #333; font-family: Helvetica, Arial, sans-serif; border-collapse: collapse; border-spacing: 0;}table.converted td, table.converted th {border: 1px solid #C9C7C7;}table.converted th, table.converted td.rowLabel {background: #626262; color: #FFFFFF; font-weight: bold; text-align: left;}table.converted td {text-align: left;}table.converted tr:nth-child(even) {background: #F3F3F3;}table.converted tr:nth-child(odd) {background: #FFFFFF;}table.converted tbody tr:hover {background: #BDF4B5;}table.converted td.rowLabel, table.converted th.rowLabel {display: none;}</style><table class=\"converted\"><tbody><tr><td>{gene=http://phenomebrowser&#8203;.net/ismb-tutorial/gene/MGI:5792188, sim=0&#8203;.5555893918721817}</td></tr><tr><td>{gene=http://phenomebrowser&#8203;.net/ismb-tutorial/gene/MGI:1097692, sim=0&#8203;.5423467880015543}</td></tr><tr><td>{gene=http://phenomebrowser&#8203;.net/ismb-tutorial/gene/MGI:1915084, sim=0&#8203;.5276590352710075}</td></tr><tr><td>{gene=http://phenomebrowser&#8203;.net/ismb-tutorial/gene/MGI:2384871, sim=0&#8203;.5198831701567547}</td></tr><tr><td>{gene=http://phenomebrowser&#8203;.net/ismb-tutorial/gene/MGI:97525, sim=0&#8203;.5064965880467487}</td></tr><tr><td>{gene=http://phenomebrowser&#8203;.net/ismb-tutorial/gene/MGI:2685267, sim=0&#8203;.5015339678444678}</td></tr><tr><td>{gene=http://phenomebrowser&#8203;.net/ismb-tutorial/gene/MGI:95793, sim=0&#8203;.49775295336856695}</td></tr><tr><td>{gene=http://phenomebrowser&#8203;.net/ismb-tutorial/gene/MGI:1277953, sim=0&#8203;.4960533580227945}</td></tr><tr><td>{gene=http://phenomebrowser&#8203;.net/ismb-tutorial/gene/MGI:2448607, sim=0&#8203;.49545547645498755}</td></tr><tr><td>{gene=http://phenomebrowser&#8203;.net/ismb-tutorial/gene/MGI:1341818, sim=0&#8203;.4940524682750143}</td></tr><tr><td>{gene=http://phenomebrowser&#8203;.net/ismb-tutorial/gene/MGI:1339969, sim=0&#8203;.489986747260717}</td></tr></tbody></table>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "// that's our patient, with a single phenotype of an orphan disease, \"MACULAR DYSTROPHY, VITELLIFORM, 1; VMD1\" (https://www.omim.org/entry/153840)\n",
    "Set patient = [ factory.getURI(\"http://purl.obolibrary.org/obo/HP_0007754\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0007663\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0001123\"), \n",
    "  factory.getURI(\"http://purl.obolibrary.org/obo/HP_0000505\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0007677\") ]\n",
    "// we store the similarity values in the results set\n",
    "def results = new LinkedHashSet()\n",
    "engine.getInstances().each { gene ->\n",
    "  def phenoSet = ia.getDirectClass(gene) // gets all the annotations of each instance (gene)\n",
    "  Expando exp = new Expando()\n",
    "  try { // this part might fail if the version of MP we use to compute similarity and the MP used for annotation is different; we ignore all errors for now\n",
    "    exp.sim = engine.compare(smConfGroupwise, smConfPairwise, phenoSet, patient)\n",
    "    exp.gene = gene\n",
    "    results.add(exp)\n",
    "  } catch (Exception E) {}\n",
    "}\n",
    "// sorting results by similarity (highest first) and output the top 10 predictions\n",
    "println \"Highest ranking genes for patient 1 (macular dystrophy): \"\n",
    "results.sort { it.sim }.reverse()[0..10]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We test this with another patient, this time with Wiskott-Aldrich Syndrome (https://www.omim.org/entry/301000). \n",
    "Wiskott-Aldrich Syndrome is a rare genetically-based disease in which the WAS gene is known to be causally involved. In the mouse, there is also a FOXP3 model that is used as a disease model for Wiskott-Aldrich syndrome (http://www.informatics.jax.org/allele/MGI:1857034)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Highest ranking genes for patient 2 (Wiskott Aldrich Syndrom): \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>table.converted {color: #333; font-family: Helvetica, Arial, sans-serif; border-collapse: collapse; border-spacing: 0;}table.converted td, table.converted th {border: 1px solid #C9C7C7;}table.converted th, table.converted td.rowLabel {background: #626262; color: #FFFFFF; font-weight: bold; text-align: left;}table.converted td {text-align: left;}table.converted tr:nth-child(even) {background: #F3F3F3;}table.converted tr:nth-child(odd) {background: #FFFFFF;}table.converted tbody tr:hover {background: #BDF4B5;}table.converted td.rowLabel, table.converted th.rowLabel {display: none;}</style><table class=\"converted\"><tbody><tr><td>{gene=http://phenomebrowser&#8203;.net/ismb-tutorial/gene/MGI:102849, sim=0&#8203;.5188118012539226}</td></tr><tr><td>{gene=http://phenomebrowser&#8203;.net/ismb-tutorial/gene/MGI:96548, sim=0&#8203;.5107686733622316}</td></tr><tr><td>{gene=http://phenomebrowser&#8203;.net/ismb-tutorial/gene/MGI:98941, sim=0&#8203;.5092662894962212}</td></tr><tr><td>{gene=http://phenomebrowser&#8203;.net/ismb-tutorial/gene/MGI:1349436, sim=0&#8203;.5001870224320165}</td></tr><tr><td>{gene=http://phenomebrowser&#8203;.net/ismb-tutorial/gene/MGI:1918576, sim=0&#8203;.48993342722986777}</td></tr><tr><td>{gene=http://phenomebrowser&#8203;.net/ismb-tutorial/gene/MGI:87967, sim=0&#8203;.48933405215860903}</td></tr><tr><td>{gene=http://phenomebrowser&#8203;.net/ismb-tutorial/gene/MGI:5792227, sim=0&#8203;.4891438237476968}</td></tr><tr><td>{gene=http://phenomebrowser&#8203;.net/ismb-tutorial/gene/MGI:1891436, sim=0&#8203;.48810218219880075}</td></tr><tr><td>{gene=http://phenomebrowser&#8203;.net/ismb-tutorial/gene/MGI:2180699, sim=0&#8203;.4876913998812737}</td></tr><tr><td>{gene=http://phenomebrowser&#8203;.net/ismb-tutorial/gene/MGI:96549, sim=0&#8203;.4847793602161192}</td></tr><tr><td>{gene=http://phenomebrowser&#8203;.net/ismb-tutorial/gene/MGI:1919073, sim=0&#8203;.4841476580703824}</td></tr></tbody></table>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "// another patient with Wiskott-Aldrich Syndrome\n",
    "Set patient = [ factory.getURI(\"http://purl.obolibrary.org/obo/HP_0000112\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0000225\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0000246\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0000388\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0000421\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0000964\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0000967\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0000979\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0001287\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0001419\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0001873\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0001878\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0001888\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0001891\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0001983\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0002037\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0002090\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0002248\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0002249\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0002783\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0002788\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0002848\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0002850\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0002963\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0002971\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0003010\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0003212\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0003261\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0005310\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0005537\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0011944\"), factory.getURI(\"http://purl.obolibrary.org/obo/HP_0040184\") ]\n",
    "def results = new LinkedHashSet()\n",
    "engine.getInstances().each { gene ->\n",
    "  def phenoSet = ia.getDirectClass(gene) // gets all the annotations of each instance (gene)\n",
    "  Expando exp = new Expando()\n",
    "  try { // this might fail if the version of MP we use to compute similarity and the MP used for annotation is different; we ignore this for now\n",
    "    exp.sim = engine.compare(smConfGroupwise, smConfPairwise, phenoSet, patient)\n",
    "    exp.gene = gene\n",
    "    results.add(exp)\n",
    "  } catch (Exception E) { }\n",
    "}\n",
    "// sorting results by similarity (highest first) and output the top 10 predictions\n",
    "println \"Highest ranking genes for patient 2 (Wiskott Aldrich Syndrom): \"\n",
    "results.sort { it.sim }.reverse()[0..10] // includes MGI:1891436, FOXP3\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ontology embeddings\n",
    "\n",
    "Given an ontology $O = (C, R, I; ax; \\vdash)$ with classes $C$, relations $R$, instances $I$, axioms $ax$, and an inference relation $\\vdash$, an ontology embedding is a function $f: C \\cup R \\cup I \\mapsto \\mathbf{R}^n$ that maps the entities in the ontology into an $n$-dimension vector space (subject to certain constraints).\n",
    "\n",
    "Here, we use the OPA2Vec tool (https://github.com/bio-ontology-research-group/opa2vec/) to generate ontology embeddings for classes in PhenomeNET, mouse genes (and their phenotypes), and OMIM diseases (and their phenotypes). We get the data from http://www.informatics.jax.org/downloads/reports/MGI_GenePheno.rpt and http://compbio.charite.de/jenkins/job/hpo.annotations/lastStableBuild/artifact/misc/phenotype_annotation.tab and then prepare it for OPA2Vec. We will generate the association file for OPA2Vec first.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PrintWriter fout = new PrintWriter(new BufferedWriter(new FileWriter(\"misc/phenotype-associations.txt\")))\n",
    "new File(\"misc/MGI_GenePheno.rpt\").splitEachLine(\"\\t\") { line ->\n",
    "    def id = line[-2]\n",
    "    def p = \"<http://purl.obolibrary.org/\"+line[-4].replaceAll(\":\",\"_\")+\">\"\n",
    "    fout.println(id+\"\\t\"+p)\n",
    "}\n",
    "\n",
    "new File(\"misc/phenotype_annotation.tab\").splitEachLine(\"\\t\") { line ->\n",
    "    def id = line[0]+\":\"+line[1]\n",
    "    def p = \"<http://purl.obolibrary.org/obo/\"+line[4].replaceAll(\":\", \"_\")+\">\"\n",
    "    fout.println(id+\"\\t\"+p)\n",
    "}\n",
    "fout.flush()\n",
    "fout.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step is to generate the embedding function and embedding vectors. We are not going to do this here, but will use the OPA2Vec command line tool. You can do this during this tutorial, but it may take quite some time to finish, so we will just continue with the output of OPA2Vec in the next steps.\n",
    "To run OPA2Vec with PhenomeNET and the association file we just generated, use this (adjust the pathnames to your own locations):\n",
    "`python2 opa2vec/runOPA2Vec.py -ontology merged-phenomenet.owl -associations misc/phenotype-associations.txt -outfile misc/phenome-vec.txt -embedsize 200 -pretrained opa2vec/PMC_model/PMC_model.txt -reasoner elk`\n",
    "\n",
    "Do not run this now as it can take quite some time (hours!) to complete."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"python2 opa2vec/runOPA2Vec.py -ontology merged-phenomenet.owl -associations misc/phenotype-associations.txt -outfile misc/phenome-vec.txt -embedsize 200 -pretrained opa2vec/PMC_model/PMC_model.txt -reasoner elk\".execute()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result of this command will be one file, `misc/phenome-vec.txt`, which contains the vectors (of size 200) for each class, each gene, and each disease. We provide a pre-computed file that we also gzipped to save some space. It's not very nicely formatted for what we want to do here, and it contains vectors for every class in the ontology which we won't use, so we will do some reformatting and skip the things we don't need, and the remaining vectors will end up in a map (strategically called `map` below).\n",
    "\n",
    "In the end, we output the embedding vector for the Tetralogy of Fallot (OMIM:187500)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import java.util.zip.*\n",
    "\n",
    "map = [:]\n",
    "InputStream fileStream = new FileInputStream(\"misc/phenome-vec-small.txt.gz\")\n",
    "InputStream gzipStream = new GZIPInputStream(fileStream)\n",
    "Reader decoder = new InputStreamReader(gzipStream)\n",
    "BufferedReader fin = new BufferedReader(decoder)\n",
    "def current = null\n",
    "fin.eachLine { line ->\n",
    "    if (line.indexOf(\"[\")>-1) { // new vector starts here\n",
    "      if (line.indexOf(\"MGI\")>-1 || line.indexOf(\"OMIM\")>-1) { // gene or disease; otherwise ignore\n",
    "          toks = line.replaceAll(\"]\",\"\").replaceAll(\"\\\\[\",\"\")trim().split(\" +\")\n",
    "          current = toks[0]\n",
    "          map[current] = []\n",
    "          toks[1..-1].each { map[current] << new Double(it) }\n",
    "      } else {\n",
    "          current = null\n",
    "      }\n",
    "    } else {\n",
    "        if (current != null) {\n",
    "          toks = line.replaceAll(\"]\",\"\").replaceAll(\"\\\\[\",\"\")trim().split(\" +\")\n",
    "          toks[0..-1].each { map[current] << new Double(it) }\n",
    "        }\n",
    "    }\n",
    "}\n",
    "map[\"OMIM:187400\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we define a cosine similarity function between the vectors and we can compute similarity between different diseases, or between diseases and mouse genes. We compute the similarity between Tetralogy of Fallot and Bardet Biedl Syndrome, and we output the vectors to have a look at how this actually works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9747012245942094\n",
      "-0.00294696214\t0.0549273975\n",
      "-0.0270647556\t-0.0720392689\n",
      "-0.193653509\t-0.298497558\n",
      "-0.172999829\t-0.275265545\n",
      "0.136029065\t0.233238444\n",
      "0.158432424\t0.241885334\n",
      "-0.00991798099\t-0.0146203339\n",
      "-0.154472724\t-0.324556679\n",
      "-0.0320384465\t-0.0280262958\n",
      "0.0853883326\t0.124234304\n",
      "0.0456528366\t0.0413244814\n",
      "-0.15578106\t-0.268387109\n",
      "-0.00323162531\t0.0315771326\n",
      "1.70144522E-5\t0.0130543225\n",
      "-0.0501444452\t-0.0796046406\n",
      "-0.0648688078\t-0.0757064149\n",
      "-0.0512228906\t-0.0745768771\n",
      "0.141643733\t0.242940038\n",
      "0.0525128879\t0.0860690773\n",
      "-0.0704722032\t-0.0871746317\n",
      "-0.0535691492\t-0.05803442\n",
      "0.00218985695\t-0.0318232514\n",
      "0.250548661\t0.37484923\n",
      "-0.0466899127\t-0.067502059\n",
      "0.329944789\t0.55085814\n",
      "0.0275957007\t0.0554764867\n",
      "-0.169040829\t-0.262651503\n",
      "-0.0645452961\t-0.0391483009\n",
      "0.148918718\t0.253781289\n",
      "-0.0903182104\t-0.160860658\n",
      "0.0235525183\t0.0817191079\n",
      "-0.058755137\t-0.0525999404\n",
      "0.0733507127\t0.142178997\n",
      "-0.190450475\t-0.319490165\n",
      "0.149012908\t0.309142649\n",
      "-0.053800419\t-0.152330339\n",
      "-0.11991629\t-0.231362179\n",
      "-0.0236793999\t-0.0421257317\n",
      "-0.0644110739\t-0.176130757\n",
      "-0.175423905\t-0.296501815\n",
      "0.0870194212\t0.112548642\n",
      "-0.0198529381\t0.0177505221\n",
      "-0.00785583258\t-0.0217087697\n",
      "0.0596724264\t0.0692394599\n",
      "0.0187031198\t0.00921780523\n",
      "0.0407795869\t-0.0563046001\n",
      "0.0121144699\t0.0694144815\n",
      "0.0838241205\t0.0174688436\n",
      "-7.63655349E-4\t0.00369976298\n",
      "-0.0705021769\t-0.20314306\n",
      "-0.0250161197\t0.0245561861\n",
      "-0.111160055\t-0.144777685\n",
      "-0.0423884876\t-0.0214377176\n",
      "-0.100119926\t-0.230527535\n",
      "-0.057130944\t-0.0166756306\n",
      "0.151380628\t0.229280233\n",
      "-0.182914689\t-0.245934665\n",
      "0.0258484352\t0.0728285909\n",
      "-0.0596921518\t-0.0748158917\n",
      "-0.196359918\t-0.34243378\n",
      "-0.180380493\t-0.272901475\n",
      "0.129456177\t0.133688971\n",
      "0.0191724412\t0.0241816733\n",
      "-0.0655994639\t-0.116012521\n",
      "0.00540358061\t0.0413952842\n",
      "0.0111114588\t0.0322075784\n",
      "-0.0304644164\t1.13968395E-4\n",
      "-0.0112223364\t-0.0326593183\n",
      "0.1232614\t0.175022587\n",
      "0.0543632284\t0.0701312944\n",
      "-0.0966260061\t-0.122351445\n",
      "-0.0938073695\t-0.17191942\n",
      "-0.117150828\t-0.263945609\n",
      "0.0731397867\t0.12115559\n",
      "-0.187257931\t-0.263961613\n",
      "0.0749238655\t0.104069419\n",
      "-0.0533305071\t-0.144153833\n",
      "-0.104053594\t-0.178470328\n",
      "0.0847862735\t0.117194846\n",
      "0.0264097303\t0.00772372074\n",
      "-0.0518108867\t-0.0357336514\n",
      "0.0323290825\t0.170545653\n",
      "0.068571724\t0.137461871\n",
      "0.105762556\t0.172676697\n",
      "0.172039822\t0.248715639\n",
      "-0.184841245\t-0.344886363\n",
      "0.0189784952\t0.0887511969\n",
      "0.123669013\t0.256442249\n",
      "-0.00428741379\t-4.82192932E-4\n",
      "0.0932959244\t0.145652249\n",
      "0.0011977799\t3.4713489E-4\n",
      "-0.166294485\t-0.271901995\n",
      "-0.0454978831\t-0.0323156565\n",
      "0.156218365\t0.217347085\n",
      "0.136966482\t0.261145771\n",
      "0.067119889\t0.15969646\n",
      "0.0482001863\t0.0813623816\n",
      "-0.0858587101\t-0.115265705\n",
      "-0.00605459418\t0.0436750129\n",
      "0.00614591455\t-0.0112466961\n",
      "0.148381397\t0.291687697\n",
      "0.0638150722\t0.0610562935\n",
      "-0.0746859759\t-0.0829486772\n",
      "-0.210017517\t-0.301038593\n",
      "0.0335925557\t0.108989529\n",
      "-0.0810692683\t-0.130844057\n",
      "0.0641314834\t0.0718617141\n",
      "-0.0384603292\t-0.0437784754\n",
      "0.0555930287\t0.0307312757\n",
      "0.175006419\t0.279604048\n",
      "-0.437570244\t-0.697687387\n",
      "-0.248815119\t-0.459299237\n",
      "0.221900478\t0.42637527\n",
      "-0.203253105\t-0.332058787\n",
      "-0.0914158747\t-0.221268177\n",
      "0.0259441566\t0.0490576625\n",
      "-0.201014578\t-0.342361063\n",
      "0.051482819\t0.0960320011\n",
      "-0.0811219811\t-0.118013673\n",
      "-0.0401276052\t-0.0809862241\n",
      "-0.0958162323\t-0.131775171\n",
      "0.0843711346\t0.13786158\n",
      "0.240389213\t0.415950745\n",
      "0.119668096\t0.228142634\n",
      "-0.128149569\t-0.256502926\n",
      "-0.179918513\t-0.280255258\n",
      "0.100719616\t0.260480702\n",
      "0.00687271124\t0.0282014646\n",
      "-0.0272114649\t-0.0211002175\n",
      "-0.123488925\t-0.242871314\n",
      "0.0942061543\t0.0649862215\n",
      "-0.0506157055\t-0.0999082103\n",
      "-0.165949523\t-0.261886358\n",
      "0.178447396\t0.253581375\n",
      "-0.149077043\t-0.181089804\n",
      "0.154175594\t0.229517415\n",
      "0.0944308564\t0.119291723\n",
      "-0.0155965118\t0.0471149385\n",
      "-0.174309418\t-0.328567803\n",
      "0.00323456083\t0.0406578407\n",
      "0.139639154\t0.202003315\n",
      "0.00590363936\t0.0290677287\n",
      "0.0696611553\t0.118098594\n",
      "-0.006036181\t0.0435953252\n",
      "-0.0832858011\t-0.247949064\n",
      "0.0214415733\t-0.013205437\n",
      "-0.114393599\t-0.256210744\n",
      "0.209769145\t0.327165335\n",
      "-0.00158455386\t-0.0155732268\n",
      "-0.0332095362\t-5.147435E-4\n",
      "-0.0909017697\t-0.0827923715\n",
      "-0.152812198\t-0.191873282\n",
      "-0.0303479601\t-0.158581629\n",
      "-0.0436451137\t-0.0580645241\n",
      "0.00528519647\t-0.0133940242\n",
      "0.0664318204\t0.106083758\n",
      "-0.081088081\t-0.137623832\n",
      "-0.108130455\t-0.205714509\n",
      "0.0126960911\t0.0635210946\n",
      "-0.0894517303\t-0.128767163\n",
      "-0.0845101476\t-0.166305155\n",
      "0.118511088\t0.185910732\n",
      "0.0753914416\t0.131913885\n",
      "-0.117568724\t-0.179751083\n",
      "-0.0716669783\t-0.104054302\n",
      "-0.0577574447\t-0.0986273065\n",
      "0.0913375169\t0.128892645\n",
      "-0.0101482924\t0.00210908172\n",
      "0.0651449412\t0.100882441\n",
      "0.0450026765\t0.0877014995\n",
      "0.00115139212\t-0.0277231187\n",
      "0.186025649\t0.336733818\n",
      "-0.212952599\t-0.406745523\n",
      "0.0643999651\t0.0812549889\n",
      "0.152973026\t0.260070652\n",
      "-0.00168371468\t-0.00740026357\n",
      "-0.0549611524\t-0.00877903681\n",
      "0.026675567\t0.0561313741\n",
      "0.0102874804\t-0.0077008279\n",
      "-0.0991948768\t-0.138408303\n",
      "-0.13411276\t-0.205024317\n",
      "0.0230387226\t0.0415109433\n",
      "-0.00102237251\t-0.00140166632\n",
      "0.0589674301\t0.157667831\n",
      "0.0144354673\t0.0699318051\n",
      "-0.0155805219\t-0.0133075844\n",
      "-0.0539553128\t-0.0454844125\n",
      "-0.034273643\t-0.0967051163\n",
      "-0.0289904084\t-0.072263144\n",
      "-0.0704820231\t-0.157297626\n",
      "0.132256791\t0.241591558\n",
      "0.0266469065\t0.00991372671\n",
      "-4.87076759E-5\t-0.0381360911\n",
      "0.0824281499\t0.183738574\n",
      "-0.112137161\t-0.173639148\n",
      "-0.0306796264\t-0.080107078\n",
      "0.0961207896\t0.18307443\n",
      "-0.0217470378\t0.0304641966\n",
      "0.0847615376\t0.106685184\n",
      "0.0253298022\t0.0351955518\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>table.converted {color: #333; font-family: Helvetica, Arial, sans-serif; border-collapse: collapse; border-spacing: 0;}table.converted td, table.converted th {border: 1px solid #C9C7C7;}table.converted th, table.converted td.rowLabel {background: #626262; color: #FFFFFF; font-weight: bold; text-align: left;}table.converted td {text-align: left;}table.converted tr:nth-child(even) {background: #F3F3F3;}table.converted tr:nth-child(odd) {background: #FFFFFF;}table.converted tbody tr:hover {background: #BDF4B5;}table.converted td.rowLabel, table.converted th.rowLabel {display: none;}</style><table class=\"converted\"><tbody><tr><td>-0&#8203;.00294696214</td></tr><tr><td>-0&#8203;.0270647556</td></tr><tr><td>-0&#8203;.193653509</td></tr><tr><td>-0&#8203;.172999829</td></tr><tr><td>0&#8203;.136029065</td></tr><tr><td>0&#8203;.158432424</td></tr><tr><td>-0&#8203;.00991798099</td></tr><tr><td>-0&#8203;.154472724</td></tr><tr><td>-0&#8203;.0320384465</td></tr><tr><td>0&#8203;.0853883326</td></tr><tr><td>0&#8203;.0456528366</td></tr><tr><td>-0&#8203;.15578106</td></tr><tr><td>-0&#8203;.00323162531</td></tr><tr><td>1&#8203;.70144522E-5</td></tr><tr><td>-0&#8203;.0501444452</td></tr><tr><td>-0&#8203;.0648688078</td></tr><tr><td>-0&#8203;.0512228906</td></tr><tr><td>0&#8203;.141643733</td></tr><tr><td>0&#8203;.0525128879</td></tr><tr><td>-0&#8203;.0704722032</td></tr><tr><td>-0&#8203;.0535691492</td></tr><tr><td>0&#8203;.00218985695</td></tr><tr><td>0&#8203;.250548661</td></tr><tr><td>-0&#8203;.0466899127</td></tr><tr><td>0&#8203;.329944789</td></tr><tr><td>0&#8203;.0275957007</td></tr><tr><td>-0&#8203;.169040829</td></tr><tr><td>-0&#8203;.0645452961</td></tr><tr><td>0&#8203;.148918718</td></tr><tr><td>-0&#8203;.0903182104</td></tr><tr><td>0&#8203;.0235525183</td></tr><tr><td>-0&#8203;.058755137</td></tr><tr><td>0&#8203;.0733507127</td></tr><tr><td>-0&#8203;.190450475</td></tr><tr><td>0&#8203;.149012908</td></tr><tr><td>-0&#8203;.053800419</td></tr><tr><td>-0&#8203;.11991629</td></tr><tr><td>-0&#8203;.0236793999</td></tr><tr><td>-0&#8203;.0644110739</td></tr><tr><td>-0&#8203;.175423905</td></tr><tr><td>0&#8203;.0870194212</td></tr><tr><td>-0&#8203;.0198529381</td></tr><tr><td>-0&#8203;.00785583258</td></tr><tr><td>0&#8203;.0596724264</td></tr><tr><td>0&#8203;.0187031198</td></tr><tr><td>0&#8203;.0407795869</td></tr><tr><td>0&#8203;.0121144699</td></tr><tr><td>0&#8203;.0838241205</td></tr><tr><td>-7&#8203;.63655349E-4</td></tr><tr><td>-0&#8203;.0705021769</td></tr><tr><td>-0&#8203;.0250161197</td></tr><tr><td>-0&#8203;.111160055</td></tr><tr><td>-0&#8203;.0423884876</td></tr><tr><td>-0&#8203;.100119926</td></tr><tr><td>-0&#8203;.057130944</td></tr><tr><td>0&#8203;.151380628</td></tr><tr><td>-0&#8203;.182914689</td></tr><tr><td>0&#8203;.0258484352</td></tr><tr><td>-0&#8203;.0596921518</td></tr><tr><td>-0&#8203;.196359918</td></tr><tr><td>-0&#8203;.180380493</td></tr><tr><td>0&#8203;.129456177</td></tr><tr><td>0&#8203;.0191724412</td></tr><tr><td>-0&#8203;.0655994639</td></tr><tr><td>0&#8203;.00540358061</td></tr><tr><td>0&#8203;.0111114588</td></tr><tr><td>-0&#8203;.0304644164</td></tr><tr><td>-0&#8203;.0112223364</td></tr><tr><td>0&#8203;.1232614</td></tr><tr><td>0&#8203;.0543632284</td></tr><tr><td>-0&#8203;.0966260061</td></tr><tr><td>-0&#8203;.0938073695</td></tr><tr><td>-0&#8203;.117150828</td></tr><tr><td>0&#8203;.0731397867</td></tr><tr><td>-0&#8203;.187257931</td></tr><tr><td>0&#8203;.0749238655</td></tr><tr><td>-0&#8203;.0533305071</td></tr><tr><td>-0&#8203;.104053594</td></tr><tr><td>0&#8203;.0847862735</td></tr><tr><td>0&#8203;.0264097303</td></tr><tr><td>-0&#8203;.0518108867</td></tr><tr><td>0&#8203;.0323290825</td></tr><tr><td>0&#8203;.068571724</td></tr><tr><td>0&#8203;.105762556</td></tr><tr><td>0&#8203;.172039822</td></tr><tr><td>-0&#8203;.184841245</td></tr><tr><td>0&#8203;.0189784952</td></tr><tr><td>0&#8203;.123669013</td></tr><tr><td>-0&#8203;.00428741379</td></tr><tr><td>0&#8203;.0932959244</td></tr><tr><td>0&#8203;.0011977799</td></tr><tr><td>-0&#8203;.166294485</td></tr><tr><td>-0&#8203;.0454978831</td></tr><tr><td>0&#8203;.156218365</td></tr><tr><td>0&#8203;.136966482</td></tr><tr><td>0&#8203;.067119889</td></tr><tr><td>0&#8203;.0482001863</td></tr><tr><td>-0&#8203;.0858587101</td></tr><tr><td>-0&#8203;.00605459418</td></tr><tr><td>0&#8203;.00614591455</td></tr><tr><td>0&#8203;.148381397</td></tr><tr><td>0&#8203;.0638150722</td></tr><tr><td>-0&#8203;.0746859759</td></tr><tr><td>-0&#8203;.210017517</td></tr><tr><td>0&#8203;.0335925557</td></tr><tr><td>-0&#8203;.0810692683</td></tr><tr><td>0&#8203;.0641314834</td></tr><tr><td>-0&#8203;.0384603292</td></tr><tr><td>0&#8203;.0555930287</td></tr><tr><td>0&#8203;.175006419</td></tr><tr><td>-0&#8203;.437570244</td></tr><tr><td>-0&#8203;.248815119</td></tr><tr><td>0&#8203;.221900478</td></tr><tr><td>-0&#8203;.203253105</td></tr><tr><td>-0&#8203;.0914158747</td></tr><tr><td>0&#8203;.0259441566</td></tr><tr><td>-0&#8203;.201014578</td></tr><tr><td>0&#8203;.051482819</td></tr><tr><td>-0&#8203;.0811219811</td></tr><tr><td>-0&#8203;.0401276052</td></tr><tr><td>-0&#8203;.0958162323</td></tr><tr><td>0&#8203;.0843711346</td></tr><tr><td>0&#8203;.240389213</td></tr><tr><td>0&#8203;.119668096</td></tr><tr><td>-0&#8203;.128149569</td></tr><tr><td>-0&#8203;.179918513</td></tr><tr><td>0&#8203;.100719616</td></tr><tr><td>0&#8203;.00687271124</td></tr><tr><td>-0&#8203;.0272114649</td></tr><tr><td>-0&#8203;.123488925</td></tr><tr><td>0&#8203;.0942061543</td></tr><tr><td>-0&#8203;.0506157055</td></tr><tr><td>-0&#8203;.165949523</td></tr><tr><td>0&#8203;.178447396</td></tr><tr><td>-0&#8203;.149077043</td></tr><tr><td>0&#8203;.154175594</td></tr><tr><td>0&#8203;.0944308564</td></tr><tr><td>-0&#8203;.0155965118</td></tr><tr><td>-0&#8203;.174309418</td></tr><tr><td>0&#8203;.00323456083</td></tr><tr><td>0&#8203;.139639154</td></tr><tr><td>0&#8203;.00590363936</td></tr><tr><td>0&#8203;.0696611553</td></tr><tr><td>-0&#8203;.006036181</td></tr><tr><td>-0&#8203;.0832858011</td></tr><tr><td>0&#8203;.0214415733</td></tr><tr><td>-0&#8203;.114393599</td></tr><tr><td>0&#8203;.209769145</td></tr><tr><td>-0&#8203;.00158455386</td></tr><tr><td>-0&#8203;.0332095362</td></tr><tr><td>-0&#8203;.0909017697</td></tr><tr><td>-0&#8203;.152812198</td></tr><tr><td>-0&#8203;.0303479601</td></tr><tr><td>-0&#8203;.0436451137</td></tr><tr><td>0&#8203;.00528519647</td></tr><tr><td>0&#8203;.0664318204</td></tr><tr><td>-0&#8203;.081088081</td></tr><tr><td>-0&#8203;.108130455</td></tr><tr><td>0&#8203;.0126960911</td></tr><tr><td>-0&#8203;.0894517303</td></tr><tr><td>-0&#8203;.0845101476</td></tr><tr><td>0&#8203;.118511088</td></tr><tr><td>0&#8203;.0753914416</td></tr><tr><td>-0&#8203;.117568724</td></tr><tr><td>-0&#8203;.0716669783</td></tr><tr><td>-0&#8203;.0577574447</td></tr><tr><td>0&#8203;.0913375169</td></tr><tr><td>-0&#8203;.0101482924</td></tr><tr><td>0&#8203;.0651449412</td></tr><tr><td>0&#8203;.0450026765</td></tr><tr><td>0&#8203;.00115139212</td></tr><tr><td>0&#8203;.186025649</td></tr><tr><td>-0&#8203;.212952599</td></tr><tr><td>0&#8203;.0643999651</td></tr><tr><td>0&#8203;.152973026</td></tr><tr><td>-0&#8203;.00168371468</td></tr><tr><td>-0&#8203;.0549611524</td></tr><tr><td>0&#8203;.026675567</td></tr><tr><td>0&#8203;.0102874804</td></tr><tr><td>-0&#8203;.0991948768</td></tr><tr><td>-0&#8203;.13411276</td></tr><tr><td>0&#8203;.0230387226</td></tr><tr><td>-0&#8203;.00102237251</td></tr><tr><td>0&#8203;.0589674301</td></tr><tr><td>0&#8203;.0144354673</td></tr><tr><td>-0&#8203;.0155805219</td></tr><tr><td>-0&#8203;.0539553128</td></tr><tr><td>-0&#8203;.034273643</td></tr><tr><td>-0&#8203;.0289904084</td></tr><tr><td>-0&#8203;.0704820231</td></tr><tr><td>0&#8203;.132256791</td></tr><tr><td>0&#8203;.0266469065</td></tr><tr><td>-4&#8203;.87076759E-5</td></tr><tr><td>0&#8203;.0824281499</td></tr><tr><td>-0&#8203;.112137161</td></tr><tr><td>-0&#8203;.0306796264</td></tr><tr><td>0&#8203;.0961207896</td></tr><tr><td>-0&#8203;.0217470378</td></tr><tr><td>0&#8203;.0847615376</td></tr><tr><td>0&#8203;.0253298022</td></tr></tbody></table>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Double cosineSimilarity(def a, def b) {\n",
    "    double dotProduct = 0.0\n",
    "    double normA = 0.0\n",
    "    double normB = 0.0\n",
    "    for (int i = 0; i < a.size(); i++) {\n",
    "        dotProduct += a[i] * b[i]\n",
    "        normA += Math.pow(a[i], 2);\n",
    "        normB += Math.pow(b[i], 2);\n",
    "    }   \n",
    "    return dotProduct / (Math.sqrt(normA) * Math.sqrt(normB))\n",
    "}\n",
    "//println cosineSimilarity(map[\"OMIM:187500\"], map[\"MGI:99604\"])\n",
    "println cosineSimilarity(map[\"OMIM:187500\"], map[\"OMIM:209900\"])\n",
    "map[\"OMIM:187500\"].eachWithIndex { k, i -> println \"$k\\t\"+map[\"OMIM:209900\"][i]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to do the same we did with semantic similarity measures now and find the closest matching gene for Tetralogy of Fallot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def maxval = 0\n",
    "def max = \"\"\n",
    "map.keySet().each {k ->\n",
    "    if (k.indexOf(\"MGI\")>-1 && map[k]) { // mouse genes only\n",
    "        if (map[\"OMIM:187500\"].size()!=map[k].size()) {\n",
    "            println \"$k: \"+map[k].size()+\" \"+map[\"OMIM:187500\"].size()\n",
    "        }\n",
    "        def val = cosineSimilarity(map[\"OMIM:187500\"], map[k])\n",
    "        if (val > maxval) {\n",
    "            maxval = val\n",
    "            max = k\n",
    "        }\n",
    "    }\n",
    "}\n",
    "println \"$max $maxval\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Naturally, cosine similarity is only a single function and there are many more. We can use correlation between the vectors, Euclidean distance, and so on. We can also visualize the similarity between these vectors using a "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PrintWriter fout = new PrintWriter(new BufferedWriter(new FileWriter(\"misc/phenome-tsne.txt\")))\n",
    "Set s = new LinkedHashSet()\n",
    "def count = 0\n",
    "map.each { k, v -> \n",
    "    if ((k.indexOf(\"MGI\")>-1) && !(k in s) && (count < 500)) {\n",
    "        s.add(k)\n",
    "        count += 1\n",
    "        v.each { fout.print(it+\"\\t\") }\n",
    "        fout.println(\"\")\n",
    "    }\n",
    "}\n",
    "fout.flush()\n",
    "fout.close()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "SciJava",
   "language": "groovy",
   "name": "scijava"
  },
  "language_info": {
   "codemirror_mode": "groovy",
   "file_extension": "",
   "mimetype": "",
   "name": "scijava",
   "nbconverter_exporter": "",
   "pygments_lexer": "groovy",
   "version": "1.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}