Labeling_Functions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img align=\"left\" src=\"imgs/logo.jpg\" width=\"50px\" style=\"margin-right:10px\">\n",
    "# Snorkel Workshop: Extracting Spouse Relations <br> from the News\n",
    "## Part 2: Writing  Labeling Functions\n",
    "\n",
    "In Snorkel, our primary interface through which we provide training signal to the end extraction model we are training is by writing **labeling functions (LFs)** (as opposed to hand-labeling massive training sets).  We'll go through some examples for our spouse extraction task below.\n",
    "\n",
    "A labeling function isn't anything special. It's just a Python function that accepts a `Candidate` as the input argument and returns `1` if it says the `Candidate` should be marked as true, `-1` if it says the `Candidate` should be marked as false, and `0` if it doesn't know how to vote and abstains. In practice, many labeling functions are unipolar: it labels only `1`s and `0`s, or it labels only `-1`s and `0`s.\n",
    "\n",
    "Recall that our goal is to ultimately train a high-performance classification model that predicts which of our `Candidate`s are true mentions of spouse relations.  It turns out that we can do this by writing potentially low-quality labeling functions!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "\n",
    "import re\n",
    "import sys\n",
    "import numpy as np\n",
    "\n",
    "# Connect to the database backend and initalize a Snorkel session\n",
    "from lib.init import *\n",
    "from lib.scoring import *\n",
    "from lib.lf_factories import *\n",
    "\n",
    "from snorkel.lf_helpers import test_LF\n",
    "from snorkel.annotations import load_gold_labels\n",
    "from snorkel.lf_helpers import (\n",
    "    get_left_tokens, get_right_tokens, get_between_tokens,\n",
    "    get_text_between, get_tagged_text,\n",
    ")\n",
    "\n",
    "# initialize our candidate type definition\n",
    "Spouse = candidate_subclass('Spouse', ['person1', 'person2'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# I. Background\n",
    "\n",
    "## A. Preprocessing the Database\n",
    "\n",
    "In a real application, there is a lot of data preparation, parsing, and database loading that needs to be completed before we dive into writing labeling functions. Here we've pre-generated a database instance for you. All _candidates_ and _gold labels_ (i.e., human-generated labels) are queried from this database for use in the the tutorial. \n",
    "\n",
    "See our preprocessing tutorial <a href=\"Workshop_5_Advanced_Preprocessing.ipynb\">Workshop 5 Advanced Preprocessing</a> for more details on how this database is built.\n",
    "\n",
    "## B. Using a _Development Set_ of Human-labeled Data\n",
    "\n",
    "In our setting, we will use the phrase _development set_ to refer to a set of examples (here, a subset of our training set) which we label by hand and use to help us develop and refine labeling functions.  Unlike the _test set_, which we do not look at and use for final evaluation, we can inspect the development set while writing labeling functions. This is a list of `{-1,1}` labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "L_gold_dev = load_gold_labels(session, annotator_name='gold', split=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## C. Data Exploration\n",
    "\n",
    "How do we come up with good keywords and patterns to encode as labeling functions? One way is to manually explore our training data. Here we load a subset of our training candidates into a `SentenceNgramViewer` object to examine candidates in their parent context. Our goal is to build an intuition for patterns and keywords that are predictive of a candidate's true label. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "require.undef('viewer');\n",
       "\n",
       "// NOTE: all elements should be selected using this.$el.find to avoid collisions with other Viewers\n",
       "\n",
       "define('viewer', [\"@jupyter-widgets/base\"], function(widgets) {\n",
       "    var ViewerView = widgets.DOMWidgetView.extend({\n",
       "        render: function() {\n",
       "            this.cids   = this.model.get('cids');\n",
       "            this.nPages = this.cids.length;\n",
       "            this.pid  = 0;\n",
       "            this.cxid = 0;\n",
       "            this.cid  = 0;\n",
       "\n",
       "            // Insert the html payload\n",
       "            this.$el.append(this.model.get('html'));\n",
       "\n",
       "            // Initialize all labels from previous sessions\n",
       "            this.labels = this.deserializeDict(this.model.get('_labels_serialized'));\n",
       "            for (var i=0; i < this.nPages; i++) {\n",
       "                this.pid = i;\n",
       "                for (var j=0; j < this.cids[i].length; j++) {\n",
       "                    this.cxid = j;\n",
       "                    for (var k=0; k < this.cids[i][j].length; k++) {\n",
       "                        this.cid = k;\n",
       "                        if (this.cids[i][j][k] in this.labels) {\n",
       "                            this.markCurrentCandidate(false);\n",
       "                        }\n",
       "                    }\n",
       "                }\n",
       "            }\n",
       "            this.pid  = 0;\n",
       "            this.cxid = 0;\n",
       "            this.cid  = 0;\n",
       "\n",
       "            // Enable button functionality for navigation\n",
       "            var that = this;\n",
       "            this.$el.find(\"#next-cand\").click(function() {\n",
       "                that.switchCandidate(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-cand\").click(function() {\n",
       "                that.switchCandidate(-1);\n",
       "            });\n",
       "            this.$el.find(\"#next-context\").click(function() {\n",
       "                that.switchContext(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-context\").click(function() {\n",
       "                that.switchContext(-1);\n",
       "            });\n",
       "            this.$el.find(\"#next-page\").click(function() {\n",
       "                that.switchPage(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-page\").click(function() {\n",
       "                that.switchPage(-1);\n",
       "            });\n",
       "            this.$el.find(\"#label-true\").click(function() {\n",
       "                that.labelCandidate(true, true);\n",
       "            });\n",
       "            this.$el.find(\"#label-false\").click(function() {\n",
       "                that.labelCandidate(false, true);\n",
       "            });\n",
       "\n",
       "            // Arrow key functionality\n",
       "            this.$el.keydown(function(e) {\n",
       "                switch(e.which) {\n",
       "                    case 74: // j\n",
       "                    that.switchCandidate(-1);\n",
       "                    break;\n",
       "\n",
       "                    case 73: // i\n",
       "                    that.switchPage(-1);\n",
       "                    break;\n",
       "\n",
       "                    case 76: // l\n",
       "                    that.switchCandidate(1);\n",
       "                    break;\n",
       "\n",
       "                    case 75: // k\n",
       "                    that.switchPage(1);\n",
       "                    break;\n",
       "\n",
       "                    case 84: // t\n",
       "                    that.labelCandidate(true, true);\n",
       "                    break;\n",
       "\n",
       "                    case 70: // f\n",
       "                    that.labelCandidate(false, true);\n",
       "                    break;\n",
       "                }\n",
       "            });\n",
       "\n",
       "            // Show the first page and highlight the first candidate\n",
       "            this.$el.find(\"#viewer-page-0\").show();\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Get candidate selector for currently selected candidate, escaping id properly\n",
       "        getCandidate: function() {\n",
       "            return this.$el.find(\".\"+this.cids[this.pid][this.cxid][this.cid]);\n",
       "        },  \n",
       "\n",
       "        // Color the candidate correctly according to registered label, as well as set highlighting\n",
       "        markCurrentCandidate: function(highlight) {\n",
       "            var cid  = this.cids[this.pid][this.cxid][this.cid];\n",
       "            var tags = this.$el.find(\".\"+cid);\n",
       "\n",
       "            // Clear color classes\n",
       "            tags.removeClass(\"candidate-h\");\n",
       "            tags.removeClass(\"true-candidate\");\n",
       "            tags.removeClass(\"true-candidate-h\");\n",
       "            tags.removeClass(\"false-candidate\");\n",
       "            tags.removeClass(\"false-candidate-h\");\n",
       "            tags.removeClass(\"highlighted\");\n",
       "\n",
       "            if (highlight) {\n",
       "                if (cid in this.labels) {\n",
       "                    tags.addClass(String(this.labels[cid]) + \"-candidate-h\");\n",
       "                } else {\n",
       "                    tags.addClass(\"candidate-h\");\n",
       "                }\n",
       "            \n",
       "            // If un-highlighting, leave with first non-null coloring\n",
       "            } else {\n",
       "                var that = this;\n",
       "                tags.each(function() {\n",
       "                    var cids = $(this).attr('class').split(/\\s+/).map(function(item) {\n",
       "                        return parseInt(item);\n",
       "                    });\n",
       "                    cids.sort();\n",
       "                    for (var i in cids) {\n",
       "                        if (cids[i] in that.labels) {\n",
       "                            var label = that.labels[cids[i]];\n",
       "                            $(this).addClass(String(label) + \"-candidate\");\n",
       "                            $(this).removeClass(String(!label) + \"-candidate\");\n",
       "                            break;\n",
       "                        }\n",
       "                    }\n",
       "                });\n",
       "            }\n",
       "\n",
       "            // Extra highlighting css\n",
       "            if (highlight) {\n",
       "                tags.addClass(\"highlighted\");\n",
       "            }\n",
       "\n",
       "            // Classes for showing direction of relation\n",
       "            if (highlight) {\n",
       "                this.$el.find(\".\"+cid+\"-0\").addClass(\"left-candidate\");\n",
       "                this.$el.find(\".\"+cid+\"-1\").addClass(\"right-candidate\");\n",
       "            } else {\n",
       "                this.$el.find(\".\"+cid+\"-0\").removeClass(\"left-candidate\");\n",
       "                this.$el.find(\".\"+cid+\"-1\").removeClass(\"right-candidate\");\n",
       "            }\n",
       "        },\n",
       "\n",
       "        // Cycle through candidates and highlight, by increment inc\n",
       "        switchCandidate: function(inc) {\n",
       "            var N = this.cids[this.pid].length\n",
       "            var M = this.cids[this.pid][this.cxid].length;\n",
       "            if (N == 0 || M == 0) { return false; }\n",
       "\n",
       "            // Clear highlighting from previous candidate\n",
       "            if (inc != 0) {\n",
       "                this.markCurrentCandidate(false);\n",
       "\n",
       "                // Increment the cid counter\n",
       "\n",
       "                // Move to next context\n",
       "                if (this.cid + inc >= M) {\n",
       "                    while (this.cid + inc >= M) {\n",
       "                        \n",
       "                        // At last context on page, halt\n",
       "                        if (this.cxid == N - 1) {\n",
       "                            this.cid = M - 1;\n",
       "                            inc = 0;\n",
       "                            break;\n",
       "                        \n",
       "                        // Increment to next context\n",
       "                        } else {\n",
       "                            inc -= M - this.cid;\n",
       "                            this.cxid += 1;\n",
       "                            M = this.cids[this.pid][this.cxid].length;\n",
       "                            this.cid = 0;\n",
       "                        }\n",
       "                    }\n",
       "\n",
       "                // Move to previous context\n",
       "                } else if (this.cid + inc < 0) {\n",
       "                    while (this.cid + inc < 0) {\n",
       "                        \n",
       "                        // At first context on page, halt\n",
       "                        if (this.cxid == 0) {\n",
       "                            this.cid = 0;\n",
       "                            inc = 0;\n",
       "                            break;\n",
       "                        \n",
       "                        // Increment to previous context\n",
       "                        } else {\n",
       "                            inc += this.cid + 1;\n",
       "                            this.cxid -= 1;\n",
       "                            M = this.cids[this.pid][this.cxid].length;\n",
       "                            this.cid = M - 1;\n",
       "                        }\n",
       "                    }\n",
       "                }\n",
       "\n",
       "                // Move within current context\n",
       "                this.cid += inc;\n",
       "            }\n",
       "            this.markCurrentCandidate(true);\n",
       "\n",
       "            // Push this new cid to the model\n",
       "            this.model.set('_selected_cid', this.cids[this.pid][this.cxid][this.cid]);\n",
       "            this.touch();\n",
       "        },\n",
       "\n",
       "        // Switch through contexts\n",
       "        switchContext: function(inc) {\n",
       "            this.markCurrentCandidate(false);\n",
       "\n",
       "            // Iterate context on this page\n",
       "            var M = this.cids[this.pid].length;\n",
       "            if (this.cxid + inc < 0) {\n",
       "                this.cxid = 0;\n",
       "            } else if (this.cxid + inc >= M) {\n",
       "                this.cxid = M - 1;\n",
       "            } else {\n",
       "                this.cxid += inc;\n",
       "            }\n",
       "\n",
       "            // Reset cid and set to first candidate\n",
       "            this.cid = 0;\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Switch through pages\n",
       "        switchPage: function(inc) {\n",
       "            this.markCurrentCandidate(false);\n",
       "            this.$el.find(\".viewer-page\").hide();\n",
       "            if (this.pid + inc < 0) {\n",
       "                this.pid = 0;\n",
       "            } else if (this.pid + inc > this.nPages - 1) {\n",
       "                this.pid = this.nPages - 1;\n",
       "            } else {\n",
       "                this.pid += inc;\n",
       "            }\n",
       "            this.$el.find(\"#viewer-page-\"+this.pid).show();\n",
       "\n",
       "            // Show pagination\n",
       "            this.$el.find(\"#page\").html(this.pid);\n",
       "\n",
       "            // Reset cid and set to first candidate\n",
       "            this.cid = 0;\n",
       "            this.cxid = 0;\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Label currently-selected candidate\n",
       "        labelCandidate: function(label, highlighted) {\n",
       "            var c    = this.getCandidate();\n",
       "            var cid  = this.cids[this.pid][this.cxid][this.cid];\n",
       "            var cl   = String(label) + \"-candidate\";\n",
       "            var clh  = String(label) + \"-candidate-h\";\n",
       "            var cln  = String(!label) + \"-candidate\";\n",
       "            var clnh = String(!label) + \"-candidate-h\";\n",
       "\n",
       "            // Toggle label highlighting\n",
       "            if (c.hasClass(cl) || c.hasClass(clh)) {\n",
       "                c.removeClass(cl);\n",
       "                c.removeClass(clh);\n",
       "                if (highlighted) {\n",
       "                    c.addClass(\"candidate-h\");\n",
       "                }\n",
       "                this.labels[cid] = null;\n",
       "                this.send({event: 'delete_label', cid: cid});\n",
       "            } else {\n",
       "                c.removeClass(cln);\n",
       "                c.removeClass(clnh);\n",
       "                if (highlighted) {\n",
       "                    c.addClass(clh);\n",
       "                } else {\n",
       "                    c.addClass(cl);\n",
       "                }\n",
       "                this.labels[cid] = label;\n",
       "                this.send({event: 'set_label', cid: cid, value: label});\n",
       "            }\n",
       "\n",
       "            // Set the label and pass back to the model\n",
       "            this.model.set('_labels_serialized', this.serializeDict(this.labels));\n",
       "            this.touch();\n",
       "        },\n",
       "\n",
       "        // Serialization of hash maps, because traitlets Dict doesn't seem to work...\n",
       "        serializeDict: function(d) {\n",
       "            var s = [];\n",
       "            for (var key in d) {\n",
       "                s.push(key+\"~~\"+d[key]);\n",
       "            }\n",
       "            return s.join();\n",
       "        },\n",
       "\n",
       "        // Deserialization of hash maps\n",
       "        deserializeDict: function(s) {\n",
       "            var d = {};\n",
       "            var entries = s.split(/,/);\n",
       "            var kv;\n",
       "            for (var i in entries) {\n",
       "                kv = entries[i].split(/~~/);\n",
       "                if (kv[1] == \"true\") {\n",
       "                    d[kv[0]] = true;\n",
       "                } else if (kv[1] == \"false\") {\n",
       "                    d[kv[0]] = false;\n",
       "                }\n",
       "            }\n",
       "            return d;\n",
       "        },\n",
       "    });\n",
       "\n",
       "    return {\n",
       "        ViewerView: ViewerView\n",
       "    };\n",
       "});\n"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "600b6627f0db41329c944f24fbf6ecb1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "SentenceNgramViewer(cids=[[[25]], [[204]], [[268]], [[308, 309, 310, 378, 379, 452]], [[289, 290, 346]], [[143…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from snorkel.viewer import SentenceNgramViewer\n",
    "\n",
    "# load our list of training & development candidates\n",
    "train_cands = session.query(Candidate).filter(Candidate.split == 0).all()\n",
    "dev_cands   = session.query(Candidate).filter(Candidate.split == 1).all()\n",
    "\n",
    "SentenceNgramViewer(train_cands[0:500], session, n_per_page=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## D. Labeling Function Metrics\n",
    "\n",
    "### 1. Coverage\n",
    "One simple metric we can compute quickly is our _coverage_, the number of candidates labeled by our LF, on our training set (or any other set).\n",
    "\n",
    "### 2. Precision / Recall / F1\n",
    "If we have gold labeled data, we can also compute standard precision, recall, and F1 metrics for the output of a single labeling function. These metrics are computed over 4 _error buckets_: _True Positives_ (tp), _False Positives_ (fp), _True Negatives_ (tn), and _False Negatives_ (fn).\n",
    "\n",
    "\\begin{equation*}\n",
    "precision = \\frac{tp}{(tp + fp)}\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "recall = \\frac{tp}{(tp + fn)}\n",
    "\\end{equation*}\n",
    "\n",
    "\\begin{equation*}\n",
    "F1 = 2 \\cdot \\frac{ (precision \\cdot recall)}{(precision + recall)}\n",
    "\\end{equation*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# II. Labeling Functions\n",
    "\n",
    "## A. Pattern Matching Labeling Functions\n",
    "\n",
    "One powerful form of labeling function design is defining sets of keywords or regular expressions that, as a human labeler, you know are correlated with the true label. In the terminology of [Bayesian inference](https://en.wikipedia.org/wiki/Statistical_inference#Bayesian_inference), this can be thought of as defining a [_prior_](https://en.wikipedia.org/wiki/Prior_probability) over your word features. \n",
    "\n",
    "For example, we could define a dictionary of terms that occur between person names in a candidate. One simple dictionary of terms indicating a true relation could be:\n",
    "    \n",
    "    marriage = {'husband', 'wife'}\n",
    " \n",
    "We can then write a labeling function that checks for a match with these terms in the text that occurs between person names.\n",
    "\n",
    "    def LF_marriage_terms_between(c):\n",
    "        return 1 if len(marriage.intersection(get_between_tokens(c))) > 0 else 0\n",
    "        \n",
    "The idea is that we can easily create dictionaries that encode themes or categories descibing all kinds of relationships between 2 people and then use these objects to _weakly supervise_ our classification task.\n",
    "\n",
    "    other_relationship = {'boyfriend', 'girlfriend'}\n",
    "    \n",
    "**IMPORTANT** Good labeling functions manage a trade-off between high coverage and high precision. When constructing your dictionaries, think about building larger, noiser sets of terms instead of relying on 1 or 2 keywords. Sometimes a single word can be very predictive (e.g., `ex-wife`) but it's almost always better to define something more general, such as a regular expression pattern capturing _any_ string with the `ex-` prefix. \n",
    "    \n",
    "\n",
    "### 1. Labeling Function Factories\n",
    "The above is a reasonable way to write labeling functions. However, this type of design pattern is so common that we rely on another abstraction to help us build LFs more quickly: _labeling function factories_. Factories accept simple inputs, like dictionaries or a set of regular expressions, and automatically builds labeling functions for you.\n",
    "\n",
    "The `MatchTerms` and `MatchRegex` factories require a few parameter definitions to setup:\n",
    "    \n",
    "    name:    a string that describes the category of terms/regular expressons\n",
    "    label:   patterns correlate with a True or False label (1 or -1) \n",
    "    search:  search a specific part of the sentence ('left'|'right'|'between'|'sentence')\n",
    "    window:  the length of tokens to match against for ('left'|'right') search spaces \n",
    "\n",
    "### 2. Term Matching Factory\n",
    "We illustrate below how you can use the `MatchTerms` factory to create and test an LF on training candidates. When examining candidates in the `SentenceNgramViewer`, notice that husband or wife always occurs between person names. That is the supervision signal encoded by this LF!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Coverage: 7.75% (1724/22254)\n"
     ]
    },
    {
     "data": {
      "application/javascript": [
       "require.undef('viewer');\n",
       "\n",
       "// NOTE: all elements should be selected using this.$el.find to avoid collisions with other Viewers\n",
       "\n",
       "define('viewer', [\"@jupyter-widgets/base\"], function(widgets) {\n",
       "    var ViewerView = widgets.DOMWidgetView.extend({\n",
       "        render: function() {\n",
       "            this.cids   = this.model.get('cids');\n",
       "            this.nPages = this.cids.length;\n",
       "            this.pid  = 0;\n",
       "            this.cxid = 0;\n",
       "            this.cid  = 0;\n",
       "\n",
       "            // Insert the html payload\n",
       "            this.$el.append(this.model.get('html'));\n",
       "\n",
       "            // Initialize all labels from previous sessions\n",
       "            this.labels = this.deserializeDict(this.model.get('_labels_serialized'));\n",
       "            for (var i=0; i < this.nPages; i++) {\n",
       "                this.pid = i;\n",
       "                for (var j=0; j < this.cids[i].length; j++) {\n",
       "                    this.cxid = j;\n",
       "                    for (var k=0; k < this.cids[i][j].length; k++) {\n",
       "                        this.cid = k;\n",
       "                        if (this.cids[i][j][k] in this.labels) {\n",
       "                            this.markCurrentCandidate(false);\n",
       "                        }\n",
       "                    }\n",
       "                }\n",
       "            }\n",
       "            this.pid  = 0;\n",
       "            this.cxid = 0;\n",
       "            this.cid  = 0;\n",
       "\n",
       "            // Enable button functionality for navigation\n",
       "            var that = this;\n",
       "            this.$el.find(\"#next-cand\").click(function() {\n",
       "                that.switchCandidate(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-cand\").click(function() {\n",
       "                that.switchCandidate(-1);\n",
       "            });\n",
       "            this.$el.find(\"#next-context\").click(function() {\n",
       "                that.switchContext(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-context\").click(function() {\n",
       "                that.switchContext(-1);\n",
       "            });\n",
       "            this.$el.find(\"#next-page\").click(function() {\n",
       "                that.switchPage(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-page\").click(function() {\n",
       "                that.switchPage(-1);\n",
       "            });\n",
       "            this.$el.find(\"#label-true\").click(function() {\n",
       "                that.labelCandidate(true, true);\n",
       "            });\n",
       "            this.$el.find(\"#label-false\").click(function() {\n",
       "                that.labelCandidate(false, true);\n",
       "            });\n",
       "\n",
       "            // Arrow key functionality\n",
       "            this.$el.keydown(function(e) {\n",
       "                switch(e.which) {\n",
       "                    case 74: // j\n",
       "                    that.switchCandidate(-1);\n",
       "                    break;\n",
       "\n",
       "                    case 73: // i\n",
       "                    that.switchPage(-1);\n",
       "                    break;\n",
       "\n",
       "                    case 76: // l\n",
       "                    that.switchCandidate(1);\n",
       "                    break;\n",
       "\n",
       "                    case 75: // k\n",
       "                    that.switchPage(1);\n",
       "                    break;\n",
       "\n",
       "                    case 84: // t\n",
       "                    that.labelCandidate(true, true);\n",
       "                    break;\n",
       "\n",
       "                    case 70: // f\n",
       "                    that.labelCandidate(false, true);\n",
       "                    break;\n",
       "                }\n",
       "            });\n",
       "\n",
       "            // Show the first page and highlight the first candidate\n",
       "            this.$el.find(\"#viewer-page-0\").show();\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Get candidate selector for currently selected candidate, escaping id properly\n",
       "        getCandidate: function() {\n",
       "            return this.$el.find(\".\"+this.cids[this.pid][this.cxid][this.cid]);\n",
       "        },  \n",
       "\n",
       "        // Color the candidate correctly according to registered label, as well as set highlighting\n",
       "        markCurrentCandidate: function(highlight) {\n",
       "            var cid  = this.cids[this.pid][this.cxid][this.cid];\n",
       "            var tags = this.$el.find(\".\"+cid);\n",
       "\n",
       "            // Clear color classes\n",
       "            tags.removeClass(\"candidate-h\");\n",
       "            tags.removeClass(\"true-candidate\");\n",
       "            tags.removeClass(\"true-candidate-h\");\n",
       "            tags.removeClass(\"false-candidate\");\n",
       "            tags.removeClass(\"false-candidate-h\");\n",
       "            tags.removeClass(\"highlighted\");\n",
       "\n",
       "            if (highlight) {\n",
       "                if (cid in this.labels) {\n",
       "                    tags.addClass(String(this.labels[cid]) + \"-candidate-h\");\n",
       "                } else {\n",
       "                    tags.addClass(\"candidate-h\");\n",
       "                }\n",
       "            \n",
       "            // If un-highlighting, leave with first non-null coloring\n",
       "            } else {\n",
       "                var that = this;\n",
       "                tags.each(function() {\n",
       "                    var cids = $(this).attr('class').split(/\\s+/).map(function(item) {\n",
       "                        return parseInt(item);\n",
       "                    });\n",
       "                    cids.sort();\n",
       "                    for (var i in cids) {\n",
       "                        if (cids[i] in that.labels) {\n",
       "                            var label = that.labels[cids[i]];\n",
       "                            $(this).addClass(String(label) + \"-candidate\");\n",
       "                            $(this).removeClass(String(!label) + \"-candidate\");\n",
       "                            break;\n",
       "                        }\n",
       "                    }\n",
       "                });\n",
       "            }\n",
       "\n",
       "            // Extra highlighting css\n",
       "            if (highlight) {\n",
       "                tags.addClass(\"highlighted\");\n",
       "            }\n",
       "\n",
       "            // Classes for showing direction of relation\n",
       "            if (highlight) {\n",
       "                this.$el.find(\".\"+cid+\"-0\").addClass(\"left-candidate\");\n",
       "                this.$el.find(\".\"+cid+\"-1\").addClass(\"right-candidate\");\n",
       "            } else {\n",
       "                this.$el.find(\".\"+cid+\"-0\").removeClass(\"left-candidate\");\n",
       "                this.$el.find(\".\"+cid+\"-1\").removeClass(\"right-candidate\");\n",
       "            }\n",
       "        },\n",
       "\n",
       "        // Cycle through candidates and highlight, by increment inc\n",
       "        switchCandidate: function(inc) {\n",
       "            var N = this.cids[this.pid].length\n",
       "            var M = this.cids[this.pid][this.cxid].length;\n",
       "            if (N == 0 || M == 0) { return false; }\n",
       "\n",
       "            // Clear highlighting from previous candidate\n",
       "            if (inc != 0) {\n",
       "                this.markCurrentCandidate(false);\n",
       "\n",
       "                // Increment the cid counter\n",
       "\n",
       "                // Move to next context\n",
       "                if (this.cid + inc >= M) {\n",
       "                    while (this.cid + inc >= M) {\n",
       "                        \n",
       "                        // At last context on page, halt\n",
       "                        if (this.cxid == N - 1) {\n",
       "                            this.cid = M - 1;\n",
       "                            inc = 0;\n",
       "                            break;\n",
       "                        \n",
       "                        // Increment to next context\n",
       "                        } else {\n",
       "                            inc -= M - this.cid;\n",
       "                            this.cxid += 1;\n",
       "                            M = this.cids[this.pid][this.cxid].length;\n",
       "                            this.cid = 0;\n",
       "                        }\n",
       "                    }\n",
       "\n",
       "                // Move to previous context\n",
       "                } else if (this.cid + inc < 0) {\n",
       "                    while (this.cid + inc < 0) {\n",
       "                        \n",
       "                        // At first context on page, halt\n",
       "                        if (this.cxid == 0) {\n",
       "                            this.cid = 0;\n",
       "                            inc = 0;\n",
       "                            break;\n",
       "                        \n",
       "                        // Increment to previous context\n",
       "                        } else {\n",
       "                            inc += this.cid + 1;\n",
       "                            this.cxid -= 1;\n",
       "                            M = this.cids[this.pid][this.cxid].length;\n",
       "                            this.cid = M - 1;\n",
       "                        }\n",
       "                    }\n",
       "                }\n",
       "\n",
       "                // Move within current context\n",
       "                this.cid += inc;\n",
       "            }\n",
       "            this.markCurrentCandidate(true);\n",
       "\n",
       "            // Push this new cid to the model\n",
       "            this.model.set('_selected_cid', this.cids[this.pid][this.cxid][this.cid]);\n",
       "            this.touch();\n",
       "        },\n",
       "\n",
       "        // Switch through contexts\n",
       "        switchContext: function(inc) {\n",
       "            this.markCurrentCandidate(false);\n",
       "\n",
       "            // Iterate context on this page\n",
       "            var M = this.cids[this.pid].length;\n",
       "            if (this.cxid + inc < 0) {\n",
       "                this.cxid = 0;\n",
       "            } else if (this.cxid + inc >= M) {\n",
       "                this.cxid = M - 1;\n",
       "            } else {\n",
       "                this.cxid += inc;\n",
       "            }\n",
       "\n",
       "            // Reset cid and set to first candidate\n",
       "            this.cid = 0;\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Switch through pages\n",
       "        switchPage: function(inc) {\n",
       "            this.markCurrentCandidate(false);\n",
       "            this.$el.find(\".viewer-page\").hide();\n",
       "            if (this.pid + inc < 0) {\n",
       "                this.pid = 0;\n",
       "            } else if (this.pid + inc > this.nPages - 1) {\n",
       "                this.pid = this.nPages - 1;\n",
       "            } else {\n",
       "                this.pid += inc;\n",
       "            }\n",
       "            this.$el.find(\"#viewer-page-\"+this.pid).show();\n",
       "\n",
       "            // Show pagination\n",
       "            this.$el.find(\"#page\").html(this.pid);\n",
       "\n",
       "            // Reset cid and set to first candidate\n",
       "            this.cid = 0;\n",
       "            this.cxid = 0;\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Label currently-selected candidate\n",
       "        labelCandidate: function(label, highlighted) {\n",
       "            var c    = this.getCandidate();\n",
       "            var cid  = this.cids[this.pid][this.cxid][this.cid];\n",
       "            var cl   = String(label) + \"-candidate\";\n",
       "            var clh  = String(label) + \"-candidate-h\";\n",
       "            var cln  = String(!label) + \"-candidate\";\n",
       "            var clnh = String(!label) + \"-candidate-h\";\n",
       "\n",
       "            // Toggle label highlighting\n",
       "            if (c.hasClass(cl) || c.hasClass(clh)) {\n",
       "                c.removeClass(cl);\n",
       "                c.removeClass(clh);\n",
       "                if (highlighted) {\n",
       "                    c.addClass(\"candidate-h\");\n",
       "                }\n",
       "                this.labels[cid] = null;\n",
       "                this.send({event: 'delete_label', cid: cid});\n",
       "            } else {\n",
       "                c.removeClass(cln);\n",
       "                c.removeClass(clnh);\n",
       "                if (highlighted) {\n",
       "                    c.addClass(clh);\n",
       "                } else {\n",
       "                    c.addClass(cl);\n",
       "                }\n",
       "                this.labels[cid] = label;\n",
       "                this.send({event: 'set_label', cid: cid, value: label});\n",
       "            }\n",
       "\n",
       "            // Set the label and pass back to the model\n",
       "            this.model.set('_labels_serialized', this.serializeDict(this.labels));\n",
       "            this.touch();\n",
       "        },\n",
       "\n",
       "        // Serialization of hash maps, because traitlets Dict doesn't seem to work...\n",
       "        serializeDict: function(d) {\n",
       "            var s = [];\n",
       "            for (var key in d) {\n",
       "                s.push(key+\"~~\"+d[key]);\n",
       "            }\n",
       "            return s.join();\n",
       "        },\n",
       "\n",
       "        // Deserialization of hash maps\n",
       "        deserializeDict: function(s) {\n",
       "            var d = {};\n",
       "            var entries = s.split(/,/);\n",
       "            var kv;\n",
       "            for (var i in entries) {\n",
       "                kv = entries[i].split(/~~/);\n",
       "                if (kv[1] == \"true\") {\n",
       "                    d[kv[0]] = true;\n",
       "                } else if (kv[1] == \"false\") {\n",
       "                    d[kv[0]] = false;\n",
       "                }\n",
       "            }\n",
       "            return d;\n",
       "        },\n",
       "    });\n",
       "\n",
       "    return {\n",
       "        ViewerView: ViewerView\n",
       "    };\n",
       "});\n"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7eae34a12fcf4541ae35af9ca87ac14a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "SentenceNgramViewer(cids=[[[586, 1197]], [[270]], [[1021]], [[1022, 1023]], [[633]], [[288, 289, 290]], [[1009…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "marriage  = {'husband', 'wife'}\n",
    "\n",
    "# we'll initialize our LFG and test its coverage on training candidates\n",
    "LF_marriage = MatchTerms(name='marriage', terms=marriage, label=1, search='between').lf()\n",
    "\n",
    "# what candidates are covered by this LF?\n",
    "labeled = coverage(session, LF_marriage, split=0)\n",
    "\n",
    "# now let's view what this LF labeled\n",
    "SentenceNgramViewer(labeled, session, n_per_page=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Viewing Error Buckets\n",
    "If we have gold labeled data, we can evaluate formal metrics. It's useful to view specific errors for a given LF input in the `SentenceNgramViewer`.\n",
    "\n",
    "Below, we'll compute our empirical scores using human-labeled development set data and then look at any false positive matches by our `LF_marriage` LF. We can see below from our scores that this LF isn't very accurate -- only 36% precision!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========================================\n",
      "LF Score\n",
      "========================================\n",
      "Pos. class accuracy: 1.0\n",
      "Neg. class accuracy: 0.0\n",
      "Precision            0.355\n",
      "Recall               1.0\n",
      "F1                   0.524\n",
      "----------------------------------------\n",
      "TP: 60 | FP: 109 | TN: 0 | FN: 0\n",
      "========================================\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/javascript": [
       "require.undef('viewer');\n",
       "\n",
       "// NOTE: all elements should be selected using this.$el.find to avoid collisions with other Viewers\n",
       "\n",
       "define('viewer', [\"@jupyter-widgets/base\"], function(widgets) {\n",
       "    var ViewerView = widgets.DOMWidgetView.extend({\n",
       "        render: function() {\n",
       "            this.cids   = this.model.get('cids');\n",
       "            this.nPages = this.cids.length;\n",
       "            this.pid  = 0;\n",
       "            this.cxid = 0;\n",
       "            this.cid  = 0;\n",
       "\n",
       "            // Insert the html payload\n",
       "            this.$el.append(this.model.get('html'));\n",
       "\n",
       "            // Initialize all labels from previous sessions\n",
       "            this.labels = this.deserializeDict(this.model.get('_labels_serialized'));\n",
       "            for (var i=0; i < this.nPages; i++) {\n",
       "                this.pid = i;\n",
       "                for (var j=0; j < this.cids[i].length; j++) {\n",
       "                    this.cxid = j;\n",
       "                    for (var k=0; k < this.cids[i][j].length; k++) {\n",
       "                        this.cid = k;\n",
       "                        if (this.cids[i][j][k] in this.labels) {\n",
       "                            this.markCurrentCandidate(false);\n",
       "                        }\n",
       "                    }\n",
       "                }\n",
       "            }\n",
       "            this.pid  = 0;\n",
       "            this.cxid = 0;\n",
       "            this.cid  = 0;\n",
       "\n",
       "            // Enable button functionality for navigation\n",
       "            var that = this;\n",
       "            this.$el.find(\"#next-cand\").click(function() {\n",
       "                that.switchCandidate(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-cand\").click(function() {\n",
       "                that.switchCandidate(-1);\n",
       "            });\n",
       "            this.$el.find(\"#next-context\").click(function() {\n",
       "                that.switchContext(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-context\").click(function() {\n",
       "                that.switchContext(-1);\n",
       "            });\n",
       "            this.$el.find(\"#next-page\").click(function() {\n",
       "                that.switchPage(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-page\").click(function() {\n",
       "                that.switchPage(-1);\n",
       "            });\n",
       "            this.$el.find(\"#label-true\").click(function() {\n",
       "                that.labelCandidate(true, true);\n",
       "            });\n",
       "            this.$el.find(\"#label-false\").click(function() {\n",
       "                that.labelCandidate(false, true);\n",
       "            });\n",
       "\n",
       "            // Arrow key functionality\n",
       "            this.$el.keydown(function(e) {\n",
       "                switch(e.which) {\n",
       "                    case 74: // j\n",
       "                    that.switchCandidate(-1);\n",
       "                    break;\n",
       "\n",
       "                    case 73: // i\n",
       "                    that.switchPage(-1);\n",
       "                    break;\n",
       "\n",
       "                    case 76: // l\n",
       "                    that.switchCandidate(1);\n",
       "                    break;\n",
       "\n",
       "                    case 75: // k\n",
       "                    that.switchPage(1);\n",
       "                    break;\n",
       "\n",
       "                    case 84: // t\n",
       "                    that.labelCandidate(true, true);\n",
       "                    break;\n",
       "\n",
       "                    case 70: // f\n",
       "                    that.labelCandidate(false, true);\n",
       "                    break;\n",
       "                }\n",
       "            });\n",
       "\n",
       "            // Show the first page and highlight the first candidate\n",
       "            this.$el.find(\"#viewer-page-0\").show();\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Get candidate selector for currently selected candidate, escaping id properly\n",
       "        getCandidate: function() {\n",
       "            return this.$el.find(\".\"+this.cids[this.pid][this.cxid][this.cid]);\n",
       "        },  \n",
       "\n",
       "        // Color the candidate correctly according to registered label, as well as set highlighting\n",
       "        markCurrentCandidate: function(highlight) {\n",
       "            var cid  = this.cids[this.pid][this.cxid][this.cid];\n",
       "            var tags = this.$el.find(\".\"+cid);\n",
       "\n",
       "            // Clear color classes\n",
       "            tags.removeClass(\"candidate-h\");\n",
       "            tags.removeClass(\"true-candidate\");\n",
       "            tags.removeClass(\"true-candidate-h\");\n",
       "            tags.removeClass(\"false-candidate\");\n",
       "            tags.removeClass(\"false-candidate-h\");\n",
       "            tags.removeClass(\"highlighted\");\n",
       "\n",
       "            if (highlight) {\n",
       "                if (cid in this.labels) {\n",
       "                    tags.addClass(String(this.labels[cid]) + \"-candidate-h\");\n",
       "                } else {\n",
       "                    tags.addClass(\"candidate-h\");\n",
       "                }\n",
       "            \n",
       "            // If un-highlighting, leave with first non-null coloring\n",
       "            } else {\n",
       "                var that = this;\n",
       "                tags.each(function() {\n",
       "                    var cids = $(this).attr('class').split(/\\s+/).map(function(item) {\n",
       "                        return parseInt(item);\n",
       "                    });\n",
       "                    cids.sort();\n",
       "                    for (var i in cids) {\n",
       "                        if (cids[i] in that.labels) {\n",
       "                            var label = that.labels[cids[i]];\n",
       "                            $(this).addClass(String(label) + \"-candidate\");\n",
       "                            $(this).removeClass(String(!label) + \"-candidate\");\n",
       "                            break;\n",
       "                        }\n",
       "                    }\n",
       "                });\n",
       "            }\n",
       "\n",
       "            // Extra highlighting css\n",
       "            if (highlight) {\n",
       "                tags.addClass(\"highlighted\");\n",
       "            }\n",
       "\n",
       "            // Classes for showing direction of relation\n",
       "            if (highlight) {\n",
       "                this.$el.find(\".\"+cid+\"-0\").addClass(\"left-candidate\");\n",
       "                this.$el.find(\".\"+cid+\"-1\").addClass(\"right-candidate\");\n",
       "            } else {\n",
       "                this.$el.find(\".\"+cid+\"-0\").removeClass(\"left-candidate\");\n",
       "                this.$el.find(\".\"+cid+\"-1\").removeClass(\"right-candidate\");\n",
       "            }\n",
       "        },\n",
       "\n",
       "        // Cycle through candidates and highlight, by increment inc\n",
       "        switchCandidate: function(inc) {\n",
       "            var N = this.cids[this.pid].length\n",
       "            var M = this.cids[this.pid][this.cxid].length;\n",
       "            if (N == 0 || M == 0) { return false; }\n",
       "\n",
       "            // Clear highlighting from previous candidate\n",
       "            if (inc != 0) {\n",
       "                this.markCurrentCandidate(false);\n",
       "\n",
       "                // Increment the cid counter\n",
       "\n",
       "                // Move to next context\n",
       "                if (this.cid + inc >= M) {\n",
       "                    while (this.cid + inc >= M) {\n",
       "                        \n",
       "                        // At last context on page, halt\n",
       "                        if (this.cxid == N - 1) {\n",
       "                            this.cid = M - 1;\n",
       "                            inc = 0;\n",
       "                            break;\n",
       "                        \n",
       "                        // Increment to next context\n",
       "                        } else {\n",
       "                            inc -= M - this.cid;\n",
       "                            this.cxid += 1;\n",
       "                            M = this.cids[this.pid][this.cxid].length;\n",
       "                            this.cid = 0;\n",
       "                        }\n",
       "                    }\n",
       "\n",
       "                // Move to previous context\n",
       "                } else if (this.cid + inc < 0) {\n",
       "                    while (this.cid + inc < 0) {\n",
       "                        \n",
       "                        // At first context on page, halt\n",
       "                        if (this.cxid == 0) {\n",
       "                            this.cid = 0;\n",
       "                            inc = 0;\n",
       "                            break;\n",
       "                        \n",
       "                        // Increment to previous context\n",
       "                        } else {\n",
       "                            inc += this.cid + 1;\n",
       "                            this.cxid -= 1;\n",
       "                            M = this.cids[this.pid][this.cxid].length;\n",
       "                            this.cid = M - 1;\n",
       "                        }\n",
       "                    }\n",
       "                }\n",
       "\n",
       "                // Move within current context\n",
       "                this.cid += inc;\n",
       "            }\n",
       "            this.markCurrentCandidate(true);\n",
       "\n",
       "            // Push this new cid to the model\n",
       "            this.model.set('_selected_cid', this.cids[this.pid][this.cxid][this.cid]);\n",
       "            this.touch();\n",
       "        },\n",
       "\n",
       "        // Switch through contexts\n",
       "        switchContext: function(inc) {\n",
       "            this.markCurrentCandidate(false);\n",
       "\n",
       "            // Iterate context on this page\n",
       "            var M = this.cids[this.pid].length;\n",
       "            if (this.cxid + inc < 0) {\n",
       "                this.cxid = 0;\n",
       "            } else if (this.cxid + inc >= M) {\n",
       "                this.cxid = M - 1;\n",
       "            } else {\n",
       "                this.cxid += inc;\n",
       "            }\n",
       "\n",
       "            // Reset cid and set to first candidate\n",
       "            this.cid = 0;\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Switch through pages\n",
       "        switchPage: function(inc) {\n",
       "            this.markCurrentCandidate(false);\n",
       "            this.$el.find(\".viewer-page\").hide();\n",
       "            if (this.pid + inc < 0) {\n",
       "                this.pid = 0;\n",
       "            } else if (this.pid + inc > this.nPages - 1) {\n",
       "                this.pid = this.nPages - 1;\n",
       "            } else {\n",
       "                this.pid += inc;\n",
       "            }\n",
       "            this.$el.find(\"#viewer-page-\"+this.pid).show();\n",
       "\n",
       "            // Show pagination\n",
       "            this.$el.find(\"#page\").html(this.pid);\n",
       "\n",
       "            // Reset cid and set to first candidate\n",
       "            this.cid = 0;\n",
       "            this.cxid = 0;\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Label currently-selected candidate\n",
       "        labelCandidate: function(label, highlighted) {\n",
       "            var c    = this.getCandidate();\n",
       "            var cid  = this.cids[this.pid][this.cxid][this.cid];\n",
       "            var cl   = String(label) + \"-candidate\";\n",
       "            var clh  = String(label) + \"-candidate-h\";\n",
       "            var cln  = String(!label) + \"-candidate\";\n",
       "            var clnh = String(!label) + \"-candidate-h\";\n",
       "\n",
       "            // Toggle label highlighting\n",
       "            if (c.hasClass(cl) || c.hasClass(clh)) {\n",
       "                c.removeClass(cl);\n",
       "                c.removeClass(clh);\n",
       "                if (highlighted) {\n",
       "                    c.addClass(\"candidate-h\");\n",
       "                }\n",
       "                this.labels[cid] = null;\n",
       "                this.send({event: 'delete_label', cid: cid});\n",
       "            } else {\n",
       "                c.removeClass(cln);\n",
       "                c.removeClass(clnh);\n",
       "                if (highlighted) {\n",
       "                    c.addClass(clh);\n",
       "                } else {\n",
       "                    c.addClass(cl);\n",
       "                }\n",
       "                this.labels[cid] = label;\n",
       "                this.send({event: 'set_label', cid: cid, value: label});\n",
       "            }\n",
       "\n",
       "            // Set the label and pass back to the model\n",
       "            this.model.set('_labels_serialized', this.serializeDict(this.labels));\n",
       "            this.touch();\n",
       "        },\n",
       "\n",
       "        // Serialization of hash maps, because traitlets Dict doesn't seem to work...\n",
       "        serializeDict: function(d) {\n",
       "            var s = [];\n",
       "            for (var key in d) {\n",
       "                s.push(key+\"~~\"+d[key]);\n",
       "            }\n",
       "            return s.join();\n",
       "        },\n",
       "\n",
       "        // Deserialization of hash maps\n",
       "        deserializeDict: function(s) {\n",
       "            var d = {};\n",
       "            var entries = s.split(/,/);\n",
       "            var kv;\n",
       "            for (var i in entries) {\n",
       "                kv = entries[i].split(/~~/);\n",
       "                if (kv[1] == \"true\") {\n",
       "                    d[kv[0]] = true;\n",
       "                } else if (kv[1] == \"false\") {\n",
       "                    d[kv[0]] = false;\n",
       "                }\n",
       "            }\n",
       "            return d;\n",
       "        },\n",
       "    });\n",
       "\n",
       "    return {\n",
       "        ViewerView: ViewerView\n",
       "    };\n",
       "});\n"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3fa5b6a5d78e4bf1824acd0f03c3ae59",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "SentenceNgramViewer(cids=[[[31]], [[23]], [[52]], [[45, 56]], [[92]], [[73, 74, 75]], [[87]], [[91, 104]], [[2…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tp, fp, tn, fn = error_analysis(session, LF_marriage, split=1, gold=L_gold_dev)\n",
    "\n",
    "# now let's view what this LF labeled\n",
    "SentenceNgramViewer(fp, session, n_per_page=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Other Search Contexts\n",
    "We can also search other sentence contexts, such as a window of text to the left or right of our candidate spans. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Coverage: 0.39% (11/2811)\n"
     ]
    },
    {
     "data": {
      "application/javascript": [
       "require.undef('viewer');\n",
       "\n",
       "// NOTE: all elements should be selected using this.$el.find to avoid collisions with other Viewers\n",
       "\n",
       "define('viewer', [\"@jupyter-widgets/base\"], function(widgets) {\n",
       "    var ViewerView = widgets.DOMWidgetView.extend({\n",
       "        render: function() {\n",
       "            this.cids   = this.model.get('cids');\n",
       "            this.nPages = this.cids.length;\n",
       "            this.pid  = 0;\n",
       "            this.cxid = 0;\n",
       "            this.cid  = 0;\n",
       "\n",
       "            // Insert the html payload\n",
       "            this.$el.append(this.model.get('html'));\n",
       "\n",
       "            // Initialize all labels from previous sessions\n",
       "            this.labels = this.deserializeDict(this.model.get('_labels_serialized'));\n",
       "            for (var i=0; i < this.nPages; i++) {\n",
       "                this.pid = i;\n",
       "                for (var j=0; j < this.cids[i].length; j++) {\n",
       "                    this.cxid = j;\n",
       "                    for (var k=0; k < this.cids[i][j].length; k++) {\n",
       "                        this.cid = k;\n",
       "                        if (this.cids[i][j][k] in this.labels) {\n",
       "                            this.markCurrentCandidate(false);\n",
       "                        }\n",
       "                    }\n",
       "                }\n",
       "            }\n",
       "            this.pid  = 0;\n",
       "            this.cxid = 0;\n",
       "            this.cid  = 0;\n",
       "\n",
       "            // Enable button functionality for navigation\n",
       "            var that = this;\n",
       "            this.$el.find(\"#next-cand\").click(function() {\n",
       "                that.switchCandidate(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-cand\").click(function() {\n",
       "                that.switchCandidate(-1);\n",
       "            });\n",
       "            this.$el.find(\"#next-context\").click(function() {\n",
       "                that.switchContext(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-context\").click(function() {\n",
       "                that.switchContext(-1);\n",
       "            });\n",
       "            this.$el.find(\"#next-page\").click(function() {\n",
       "                that.switchPage(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-page\").click(function() {\n",
       "                that.switchPage(-1);\n",
       "            });\n",
       "            this.$el.find(\"#label-true\").click(function() {\n",
       "                that.labelCandidate(true, true);\n",
       "            });\n",
       "            this.$el.find(\"#label-false\").click(function() {\n",
       "                that.labelCandidate(false, true);\n",
       "            });\n",
       "\n",
       "            // Arrow key functionality\n",
       "            this.$el.keydown(function(e) {\n",
       "                switch(e.which) {\n",
       "                    case 74: // j\n",
       "                    that.switchCandidate(-1);\n",
       "                    break;\n",
       "\n",
       "                    case 73: // i\n",
       "                    that.switchPage(-1);\n",
       "                    break;\n",
       "\n",
       "                    case 76: // l\n",
       "                    that.switchCandidate(1);\n",
       "                    break;\n",
       "\n",
       "                    case 75: // k\n",
       "                    that.switchPage(1);\n",
       "                    break;\n",
       "\n",
       "                    case 84: // t\n",
       "                    that.labelCandidate(true, true);\n",
       "                    break;\n",
       "\n",
       "                    case 70: // f\n",
       "                    that.labelCandidate(false, true);\n",
       "                    break;\n",
       "                }\n",
       "            });\n",
       "\n",
       "            // Show the first page and highlight the first candidate\n",
       "            this.$el.find(\"#viewer-page-0\").show();\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Get candidate selector for currently selected candidate, escaping id properly\n",
       "        getCandidate: function() {\n",
       "            return this.$el.find(\".\"+this.cids[this.pid][this.cxid][this.cid]);\n",
       "        },  \n",
       "\n",
       "        // Color the candidate correctly according to registered label, as well as set highlighting\n",
       "        markCurrentCandidate: function(highlight) {\n",
       "            var cid  = this.cids[this.pid][this.cxid][this.cid];\n",
       "            var tags = this.$el.find(\".\"+cid);\n",
       "\n",
       "            // Clear color classes\n",
       "            tags.removeClass(\"candidate-h\");\n",
       "            tags.removeClass(\"true-candidate\");\n",
       "            tags.removeClass(\"true-candidate-h\");\n",
       "            tags.removeClass(\"false-candidate\");\n",
       "            tags.removeClass(\"false-candidate-h\");\n",
       "            tags.removeClass(\"highlighted\");\n",
       "\n",
       "            if (highlight) {\n",
       "                if (cid in this.labels) {\n",
       "                    tags.addClass(String(this.labels[cid]) + \"-candidate-h\");\n",
       "                } else {\n",
       "                    tags.addClass(\"candidate-h\");\n",
       "                }\n",
       "            \n",
       "            // If un-highlighting, leave with first non-null coloring\n",
       "            } else {\n",
       "                var that = this;\n",
       "                tags.each(function() {\n",
       "                    var cids = $(this).attr('class').split(/\\s+/).map(function(item) {\n",
       "                        return parseInt(item);\n",
       "                    });\n",
       "                    cids.sort();\n",
       "                    for (var i in cids) {\n",
       "                        if (cids[i] in that.labels) {\n",
       "                            var label = that.labels[cids[i]];\n",
       "                            $(this).addClass(String(label) + \"-candidate\");\n",
       "                            $(this).removeClass(String(!label) + \"-candidate\");\n",
       "                            break;\n",
       "                        }\n",
       "                    }\n",
       "                });\n",
       "            }\n",
       "\n",
       "            // Extra highlighting css\n",
       "            if (highlight) {\n",
       "                tags.addClass(\"highlighted\");\n",
       "            }\n",
       "\n",
       "            // Classes for showing direction of relation\n",
       "            if (highlight) {\n",
       "                this.$el.find(\".\"+cid+\"-0\").addClass(\"left-candidate\");\n",
       "                this.$el.find(\".\"+cid+\"-1\").addClass(\"right-candidate\");\n",
       "            } else {\n",
       "                this.$el.find(\".\"+cid+\"-0\").removeClass(\"left-candidate\");\n",
       "                this.$el.find(\".\"+cid+\"-1\").removeClass(\"right-candidate\");\n",
       "            }\n",
       "        },\n",
       "\n",
       "        // Cycle through candidates and highlight, by increment inc\n",
       "        switchCandidate: function(inc) {\n",
       "            var N = this.cids[this.pid].length\n",
       "            var M = this.cids[this.pid][this.cxid].length;\n",
       "            if (N == 0 || M == 0) { return false; }\n",
       "\n",
       "            // Clear highlighting from previous candidate\n",
       "            if (inc != 0) {\n",
       "                this.markCurrentCandidate(false);\n",
       "\n",
       "                // Increment the cid counter\n",
       "\n",
       "                // Move to next context\n",
       "                if (this.cid + inc >= M) {\n",
       "                    while (this.cid + inc >= M) {\n",
       "                        \n",
       "                        // At last context on page, halt\n",
       "                        if (this.cxid == N - 1) {\n",
       "                            this.cid = M - 1;\n",
       "                            inc = 0;\n",
       "                            break;\n",
       "                        \n",
       "                        // Increment to next context\n",
       "                        } else {\n",
       "                            inc -= M - this.cid;\n",
       "                            this.cxid += 1;\n",
       "                            M = this.cids[this.pid][this.cxid].length;\n",
       "                            this.cid = 0;\n",
       "                        }\n",
       "                    }\n",
       "\n",
       "                // Move to previous context\n",
       "                } else if (this.cid + inc < 0) {\n",
       "                    while (this.cid + inc < 0) {\n",
       "                        \n",
       "                        // At first context on page, halt\n",
       "                        if (this.cxid == 0) {\n",
       "                            this.cid = 0;\n",
       "                            inc = 0;\n",
       "                            break;\n",
       "                        \n",
       "                        // Increment to previous context\n",
       "                        } else {\n",
       "                            inc += this.cid + 1;\n",
       "                            this.cxid -= 1;\n",
       "                            M = this.cids[this.pid][this.cxid].length;\n",
       "                            this.cid = M - 1;\n",
       "                        }\n",
       "                    }\n",
       "                }\n",
       "\n",
       "                // Move within current context\n",
       "                this.cid += inc;\n",
       "            }\n",
       "            this.markCurrentCandidate(true);\n",
       "\n",
       "            // Push this new cid to the model\n",
       "            this.model.set('_selected_cid', this.cids[this.pid][this.cxid][this.cid]);\n",
       "            this.touch();\n",
       "        },\n",
       "\n",
       "        // Switch through contexts\n",
       "        switchContext: function(inc) {\n",
       "            this.markCurrentCandidate(false);\n",
       "\n",
       "            // Iterate context on this page\n",
       "            var M = this.cids[this.pid].length;\n",
       "            if (this.cxid + inc < 0) {\n",
       "                this.cxid = 0;\n",
       "            } else if (this.cxid + inc >= M) {\n",
       "                this.cxid = M - 1;\n",
       "            } else {\n",
       "                this.cxid += inc;\n",
       "            }\n",
       "\n",
       "            // Reset cid and set to first candidate\n",
       "            this.cid = 0;\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Switch through pages\n",
       "        switchPage: function(inc) {\n",
       "            this.markCurrentCandidate(false);\n",
       "            this.$el.find(\".viewer-page\").hide();\n",
       "            if (this.pid + inc < 0) {\n",
       "                this.pid = 0;\n",
       "            } else if (this.pid + inc > this.nPages - 1) {\n",
       "                this.pid = this.nPages - 1;\n",
       "            } else {\n",
       "                this.pid += inc;\n",
       "            }\n",
       "            this.$el.find(\"#viewer-page-\"+this.pid).show();\n",
       "\n",
       "            // Show pagination\n",
       "            this.$el.find(\"#page\").html(this.pid);\n",
       "\n",
       "            // Reset cid and set to first candidate\n",
       "            this.cid = 0;\n",
       "            this.cxid = 0;\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Label currently-selected candidate\n",
       "        labelCandidate: function(label, highlighted) {\n",
       "            var c    = this.getCandidate();\n",
       "            var cid  = this.cids[this.pid][this.cxid][this.cid];\n",
       "            var cl   = String(label) + \"-candidate\";\n",
       "            var clh  = String(label) + \"-candidate-h\";\n",
       "            var cln  = String(!label) + \"-candidate\";\n",
       "            var clnh = String(!label) + \"-candidate-h\";\n",
       "\n",
       "            // Toggle label highlighting\n",
       "            if (c.hasClass(cl) || c.hasClass(clh)) {\n",
       "                c.removeClass(cl);\n",
       "                c.removeClass(clh);\n",
       "                if (highlighted) {\n",
       "                    c.addClass(\"candidate-h\");\n",
       "                }\n",
       "                this.labels[cid] = null;\n",
       "                this.send({event: 'delete_label', cid: cid});\n",
       "            } else {\n",
       "                c.removeClass(cln);\n",
       "                c.removeClass(clnh);\n",
       "                if (highlighted) {\n",
       "                    c.addClass(clh);\n",
       "                } else {\n",
       "                    c.addClass(cl);\n",
       "                }\n",
       "                this.labels[cid] = label;\n",
       "                this.send({event: 'set_label', cid: cid, value: label});\n",
       "            }\n",
       "\n",
       "            // Set the label and pass back to the model\n",
       "            this.model.set('_labels_serialized', this.serializeDict(this.labels));\n",
       "            this.touch();\n",
       "        },\n",
       "\n",
       "        // Serialization of hash maps, because traitlets Dict doesn't seem to work...\n",
       "        serializeDict: function(d) {\n",
       "            var s = [];\n",
       "            for (var key in d) {\n",
       "                s.push(key+\"~~\"+d[key]);\n",
       "            }\n",
       "            return s.join();\n",
       "        },\n",
       "\n",
       "        // Deserialization of hash maps\n",
       "        deserializeDict: function(s) {\n",
       "            var d = {};\n",
       "            var entries = s.split(/,/);\n",
       "            var kv;\n",
       "            for (var i in entries) {\n",
       "                kv = entries[i].split(/~~/);\n",
       "                if (kv[1] == \"true\") {\n",
       "                    d[kv[0]] = true;\n",
       "                } else if (kv[1] == \"false\") {\n",
       "                    d[kv[0]] = false;\n",
       "                }\n",
       "            }\n",
       "            return d;\n",
       "        },\n",
       "    });\n",
       "\n",
       "    return {\n",
       "        ViewerView: ViewerView\n",
       "    };\n",
       "});\n"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bf771193495546189055637fef2969ff",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "SentenceNgramViewer(cids=[[[3, 4, 5]], [[9, 10]], [[0, 1, 2]], [[6, 7, 8]]], html='<head>\\n<style>\\nspan.candi…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "other_relationship  = {'boyfriend', 'girlfriend'}\n",
    "\n",
    "LF_other_relationship = MatchTerms(name='other_relationship', terms=other_relationship, \n",
    "                                   label=-1, search='left', window=1).lf()\n",
    "labeled = coverage(session, LF_other_relationship, split=1)\n",
    "\n",
    "# now let's view what this LF labeled\n",
    "SentenceNgramViewer(labeled, session, n_per_page=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Regular Expression Factory\n",
    "\n",
    "Sometimes we want to express more generic textual patterns to match against candidates. Perhaps we want to match a specific phrase like 'power couple' or look for modifier prefixes like 'ex' wife, husband, etc. \n",
    "\n",
    "We can generate this supervision in the same way as above using sets of [regular expressions](https://en.wikipedia.org/wiki/Regular_expression) -- a formal language for string matching."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Coverage: 0.43% (12/2811)\n"
     ]
    },
    {
     "data": {
      "application/javascript": [
       "require.undef('viewer');\n",
       "\n",
       "// NOTE: all elements should be selected using this.$el.find to avoid collisions with other Viewers\n",
       "\n",
       "define('viewer', [\"@jupyter-widgets/base\"], function(widgets) {\n",
       "    var ViewerView = widgets.DOMWidgetView.extend({\n",
       "        render: function() {\n",
       "            this.cids   = this.model.get('cids');\n",
       "            this.nPages = this.cids.length;\n",
       "            this.pid  = 0;\n",
       "            this.cxid = 0;\n",
       "            this.cid  = 0;\n",
       "\n",
       "            // Insert the html payload\n",
       "            this.$el.append(this.model.get('html'));\n",
       "\n",
       "            // Initialize all labels from previous sessions\n",
       "            this.labels = this.deserializeDict(this.model.get('_labels_serialized'));\n",
       "            for (var i=0; i < this.nPages; i++) {\n",
       "                this.pid = i;\n",
       "                for (var j=0; j < this.cids[i].length; j++) {\n",
       "                    this.cxid = j;\n",
       "                    for (var k=0; k < this.cids[i][j].length; k++) {\n",
       "                        this.cid = k;\n",
       "                        if (this.cids[i][j][k] in this.labels) {\n",
       "                            this.markCurrentCandidate(false);\n",
       "                        }\n",
       "                    }\n",
       "                }\n",
       "            }\n",
       "            this.pid  = 0;\n",
       "            this.cxid = 0;\n",
       "            this.cid  = 0;\n",
       "\n",
       "            // Enable button functionality for navigation\n",
       "            var that = this;\n",
       "            this.$el.find(\"#next-cand\").click(function() {\n",
       "                that.switchCandidate(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-cand\").click(function() {\n",
       "                that.switchCandidate(-1);\n",
       "            });\n",
       "            this.$el.find(\"#next-context\").click(function() {\n",
       "                that.switchContext(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-context\").click(function() {\n",
       "                that.switchContext(-1);\n",
       "            });\n",
       "            this.$el.find(\"#next-page\").click(function() {\n",
       "                that.switchPage(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-page\").click(function() {\n",
       "                that.switchPage(-1);\n",
       "            });\n",
       "            this.$el.find(\"#label-true\").click(function() {\n",
       "                that.labelCandidate(true, true);\n",
       "            });\n",
       "            this.$el.find(\"#label-false\").click(function() {\n",
       "                that.labelCandidate(false, true);\n",
       "            });\n",
       "\n",
       "            // Arrow key functionality\n",
       "            this.$el.keydown(function(e) {\n",
       "                switch(e.which) {\n",
       "                    case 74: // j\n",
       "                    that.switchCandidate(-1);\n",
       "                    break;\n",
       "\n",
       "                    case 73: // i\n",
       "                    that.switchPage(-1);\n",
       "                    break;\n",
       "\n",
       "                    case 76: // l\n",
       "                    that.switchCandidate(1);\n",
       "                    break;\n",
       "\n",
       "                    case 75: // k\n",
       "                    that.switchPage(1);\n",
       "                    break;\n",
       "\n",
       "                    case 84: // t\n",
       "                    that.labelCandidate(true, true);\n",
       "                    break;\n",
       "\n",
       "                    case 70: // f\n",
       "                    that.labelCandidate(false, true);\n",
       "                    break;\n",
       "                }\n",
       "            });\n",
       "\n",
       "            // Show the first page and highlight the first candidate\n",
       "            this.$el.find(\"#viewer-page-0\").show();\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Get candidate selector for currently selected candidate, escaping id properly\n",
       "        getCandidate: function() {\n",
       "            return this.$el.find(\".\"+this.cids[this.pid][this.cxid][this.cid]);\n",
       "        },  \n",
       "\n",
       "        // Color the candidate correctly according to registered label, as well as set highlighting\n",
       "        markCurrentCandidate: function(highlight) {\n",
       "            var cid  = this.cids[this.pid][this.cxid][this.cid];\n",
       "            var tags = this.$el.find(\".\"+cid);\n",
       "\n",
       "            // Clear color classes\n",
       "            tags.removeClass(\"candidate-h\");\n",
       "            tags.removeClass(\"true-candidate\");\n",
       "            tags.removeClass(\"true-candidate-h\");\n",
       "            tags.removeClass(\"false-candidate\");\n",
       "            tags.removeClass(\"false-candidate-h\");\n",
       "            tags.removeClass(\"highlighted\");\n",
       "\n",
       "            if (highlight) {\n",
       "                if (cid in this.labels) {\n",
       "                    tags.addClass(String(this.labels[cid]) + \"-candidate-h\");\n",
       "                } else {\n",
       "                    tags.addClass(\"candidate-h\");\n",
       "                }\n",
       "            \n",
       "            // If un-highlighting, leave with first non-null coloring\n",
       "            } else {\n",
       "                var that = this;\n",
       "                tags.each(function() {\n",
       "                    var cids = $(this).attr('class').split(/\\s+/).map(function(item) {\n",
       "                        return parseInt(item);\n",
       "                    });\n",
       "                    cids.sort();\n",
       "                    for (var i in cids) {\n",
       "                        if (cids[i] in that.labels) {\n",
       "                            var label = that.labels[cids[i]];\n",
       "                            $(this).addClass(String(label) + \"-candidate\");\n",
       "                            $(this).removeClass(String(!label) + \"-candidate\");\n",
       "                            break;\n",
       "                        }\n",
       "                    }\n",
       "                });\n",
       "            }\n",
       "\n",
       "            // Extra highlighting css\n",
       "            if (highlight) {\n",
       "                tags.addClass(\"highlighted\");\n",
       "            }\n",
       "\n",
       "            // Classes for showing direction of relation\n",
       "            if (highlight) {\n",
       "                this.$el.find(\".\"+cid+\"-0\").addClass(\"left-candidate\");\n",
       "                this.$el.find(\".\"+cid+\"-1\").addClass(\"right-candidate\");\n",
       "            } else {\n",
       "                this.$el.find(\".\"+cid+\"-0\").removeClass(\"left-candidate\");\n",
       "                this.$el.find(\".\"+cid+\"-1\").removeClass(\"right-candidate\");\n",
       "            }\n",
       "        },\n",
       "\n",
       "        // Cycle through candidates and highlight, by increment inc\n",
       "        switchCandidate: function(inc) {\n",
       "            var N = this.cids[this.pid].length\n",
       "            var M = this.cids[this.pid][this.cxid].length;\n",
       "            if (N == 0 || M == 0) { return false; }\n",
       "\n",
       "            // Clear highlighting from previous candidate\n",
       "            if (inc != 0) {\n",
       "                this.markCurrentCandidate(false);\n",
       "\n",
       "                // Increment the cid counter\n",
       "\n",
       "                // Move to next context\n",
       "                if (this.cid + inc >= M) {\n",
       "                    while (this.cid + inc >= M) {\n",
       "                        \n",
       "                        // At last context on page, halt\n",
       "                        if (this.cxid == N - 1) {\n",
       "                            this.cid = M - 1;\n",
       "                            inc = 0;\n",
       "                            break;\n",
       "                        \n",
       "                        // Increment to next context\n",
       "                        } else {\n",
       "                            inc -= M - this.cid;\n",
       "                            this.cxid += 1;\n",
       "                            M = this.cids[this.pid][this.cxid].length;\n",
       "                            this.cid = 0;\n",
       "                        }\n",
       "                    }\n",
       "\n",
       "                // Move to previous context\n",
       "                } else if (this.cid + inc < 0) {\n",
       "                    while (this.cid + inc < 0) {\n",
       "                        \n",
       "                        // At first context on page, halt\n",
       "                        if (this.cxid == 0) {\n",
       "                            this.cid = 0;\n",
       "                            inc = 0;\n",
       "                            break;\n",
       "                        \n",
       "                        // Increment to previous context\n",
       "                        } else {\n",
       "                            inc += this.cid + 1;\n",
       "                            this.cxid -= 1;\n",
       "                            M = this.cids[this.pid][this.cxid].length;\n",
       "                            this.cid = M - 1;\n",
       "                        }\n",
       "                    }\n",
       "                }\n",
       "\n",
       "                // Move within current context\n",
       "                this.cid += inc;\n",
       "            }\n",
       "            this.markCurrentCandidate(true);\n",
       "\n",
       "            // Push this new cid to the model\n",
       "            this.model.set('_selected_cid', this.cids[this.pid][this.cxid][this.cid]);\n",
       "            this.touch();\n",
       "        },\n",
       "\n",
       "        // Switch through contexts\n",
       "        switchContext: function(inc) {\n",
       "            this.markCurrentCandidate(false);\n",
       "\n",
       "            // Iterate context on this page\n",
       "            var M = this.cids[this.pid].length;\n",
       "            if (this.cxid + inc < 0) {\n",
       "                this.cxid = 0;\n",
       "            } else if (this.cxid + inc >= M) {\n",
       "                this.cxid = M - 1;\n",
       "            } else {\n",
       "                this.cxid += inc;\n",
       "            }\n",
       "\n",
       "            // Reset cid and set to first candidate\n",
       "            this.cid = 0;\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Switch through pages\n",
       "        switchPage: function(inc) {\n",
       "            this.markCurrentCandidate(false);\n",
       "            this.$el.find(\".viewer-page\").hide();\n",
       "            if (this.pid + inc < 0) {\n",
       "                this.pid = 0;\n",
       "            } else if (this.pid + inc > this.nPages - 1) {\n",
       "                this.pid = this.nPages - 1;\n",
       "            } else {\n",
       "                this.pid += inc;\n",
       "            }\n",
       "            this.$el.find(\"#viewer-page-\"+this.pid).show();\n",
       "\n",
       "            // Show pagination\n",
       "            this.$el.find(\"#page\").html(this.pid);\n",
       "\n",
       "            // Reset cid and set to first candidate\n",
       "            this.cid = 0;\n",
       "            this.cxid = 0;\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Label currently-selected candidate\n",
       "        labelCandidate: function(label, highlighted) {\n",
       "            var c    = this.getCandidate();\n",
       "            var cid  = this.cids[this.pid][this.cxid][this.cid];\n",
       "            var cl   = String(label) + \"-candidate\";\n",
       "            var clh  = String(label) + \"-candidate-h\";\n",
       "            var cln  = String(!label) + \"-candidate\";\n",
       "            var clnh = String(!label) + \"-candidate-h\";\n",
       "\n",
       "            // Toggle label highlighting\n",
       "            if (c.hasClass(cl) || c.hasClass(clh)) {\n",
       "                c.removeClass(cl);\n",
       "                c.removeClass(clh);\n",
       "                if (highlighted) {\n",
       "                    c.addClass(\"candidate-h\");\n",
       "                }\n",
       "                this.labels[cid] = null;\n",
       "                this.send({event: 'delete_label', cid: cid});\n",
       "            } else {\n",
       "                c.removeClass(cln);\n",
       "                c.removeClass(clnh);\n",
       "                if (highlighted) {\n",
       "                    c.addClass(clh);\n",
       "                } else {\n",
       "                    c.addClass(cl);\n",
       "                }\n",
       "                this.labels[cid] = label;\n",
       "                this.send({event: 'set_label', cid: cid, value: label});\n",
       "            }\n",
       "\n",
       "            // Set the label and pass back to the model\n",
       "            this.model.set('_labels_serialized', this.serializeDict(this.labels));\n",
       "            this.touch();\n",
       "        },\n",
       "\n",
       "        // Serialization of hash maps, because traitlets Dict doesn't seem to work...\n",
       "        serializeDict: function(d) {\n",
       "            var s = [];\n",
       "            for (var key in d) {\n",
       "                s.push(key+\"~~\"+d[key]);\n",
       "            }\n",
       "            return s.join();\n",
       "        },\n",
       "\n",
       "        // Deserialization of hash maps\n",
       "        deserializeDict: function(s) {\n",
       "            var d = {};\n",
       "            var entries = s.split(/,/);\n",
       "            var kv;\n",
       "            for (var i in entries) {\n",
       "                kv = entries[i].split(/~~/);\n",
       "                if (kv[1] == \"true\") {\n",
       "                    d[kv[0]] = true;\n",
       "                } else if (kv[1] == \"false\") {\n",
       "                    d[kv[0]] = false;\n",
       "                }\n",
       "            }\n",
       "            return d;\n",
       "        },\n",
       "    });\n",
       "\n",
       "    return {\n",
       "        ViewerView: ViewerView\n",
       "    };\n",
       "});\n"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0ed93c4136624ce1a0d543d92a54dceb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "SentenceNgramViewer(cids=[[[3]], [[0, 2, 9, 10]], [[6, 8]], [[7, 11]], [[1, 4, 5]]], html='<head>\\n<style>\\nsp…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "exes_rgxs = {' ex[- ](husband|wife)'}\n",
    "\n",
    "LF_exes = MatchRegex(name='exes', rgxs=exes_rgxs, label=-1, search='between').lf()\n",
    "labeled = coverage(session, LF_exes, split=1)\n",
    "\n",
    "# now let's view what this LF labeled\n",
    "SentenceNgramViewer(labeled, session, n_per_page=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## B. Distant Supervision Labeling Functions\n",
    "\n",
    "In addition to using factories that encode pattern matching heuristics, we can also write labeling functions that _distantly supervise_ examples. Here, we'll load in a list of known spouse pairs and check to see if the candidate pair matches one of these.\n",
    "\n",
    "**DBpedia**\n",
    "http://wiki.dbpedia.org/\n",
    "Out database of known spouses comes from DBpedia, which is a community-driven resource similar to Wikipedia but for curating structured data. We'll use a preprocessed snapshot as our knowledge base for all labeling function development.\n",
    "\n",
    "We can look at some of the example entries from DBPedia and use them in a simple distant supervision labeling function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Matthew Cutler', 'Nicole Cutler'),\n",
       " ('Bharathi Vishnuvardhan', 'Dr.Vishnuvardhan'),\n",
       " ('Eileen Ryan', 'Leo Penn'),\n",
       " ('Judith Malina', 'Julian Beck'),\n",
       " ('Lorenza Ramírez de Arellano', 'Luis A. Ferré')]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from lib.dbpedia import known_spouses \n",
    "\n",
    "list(known_spouses)[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Coverage: 0.11% (3/2811)\n",
      "========================================\n",
      "LF Score\n",
      "========================================\n",
      "Pos. class accuracy: 1.0\n",
      "Neg. class accuracy: 0.0\n",
      "Precision            0.667\n",
      "Recall               1.0\n",
      "F1                   0.8\n",
      "----------------------------------------\n",
      "TP: 2 | FP: 1 | TN: 0 | FN: 0\n",
      "========================================\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/javascript": [
       "require.undef('viewer');\n",
       "\n",
       "// NOTE: all elements should be selected using this.$el.find to avoid collisions with other Viewers\n",
       "\n",
       "define('viewer', [\"@jupyter-widgets/base\"], function(widgets) {\n",
       "    var ViewerView = widgets.DOMWidgetView.extend({\n",
       "        render: function() {\n",
       "            this.cids   = this.model.get('cids');\n",
       "            this.nPages = this.cids.length;\n",
       "            this.pid  = 0;\n",
       "            this.cxid = 0;\n",
       "            this.cid  = 0;\n",
       "\n",
       "            // Insert the html payload\n",
       "            this.$el.append(this.model.get('html'));\n",
       "\n",
       "            // Initialize all labels from previous sessions\n",
       "            this.labels = this.deserializeDict(this.model.get('_labels_serialized'));\n",
       "            for (var i=0; i < this.nPages; i++) {\n",
       "                this.pid = i;\n",
       "                for (var j=0; j < this.cids[i].length; j++) {\n",
       "                    this.cxid = j;\n",
       "                    for (var k=0; k < this.cids[i][j].length; k++) {\n",
       "                        this.cid = k;\n",
       "                        if (this.cids[i][j][k] in this.labels) {\n",
       "                            this.markCurrentCandidate(false);\n",
       "                        }\n",
       "                    }\n",
       "                }\n",
       "            }\n",
       "            this.pid  = 0;\n",
       "            this.cxid = 0;\n",
       "            this.cid  = 0;\n",
       "\n",
       "            // Enable button functionality for navigation\n",
       "            var that = this;\n",
       "            this.$el.find(\"#next-cand\").click(function() {\n",
       "                that.switchCandidate(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-cand\").click(function() {\n",
       "                that.switchCandidate(-1);\n",
       "            });\n",
       "            this.$el.find(\"#next-context\").click(function() {\n",
       "                that.switchContext(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-context\").click(function() {\n",
       "                that.switchContext(-1);\n",
       "            });\n",
       "            this.$el.find(\"#next-page\").click(function() {\n",
       "                that.switchPage(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-page\").click(function() {\n",
       "                that.switchPage(-1);\n",
       "            });\n",
       "            this.$el.find(\"#label-true\").click(function() {\n",
       "                that.labelCandidate(true, true);\n",
       "            });\n",
       "            this.$el.find(\"#label-false\").click(function() {\n",
       "                that.labelCandidate(false, true);\n",
       "            });\n",
       "\n",
       "            // Arrow key functionality\n",
       "            this.$el.keydown(function(e) {\n",
       "                switch(e.which) {\n",
       "                    case 74: // j\n",
       "                    that.switchCandidate(-1);\n",
       "                    break;\n",
       "\n",
       "                    case 73: // i\n",
       "                    that.switchPage(-1);\n",
       "                    break;\n",
       "\n",
       "                    case 76: // l\n",
       "                    that.switchCandidate(1);\n",
       "                    break;\n",
       "\n",
       "                    case 75: // k\n",
       "                    that.switchPage(1);\n",
       "                    break;\n",
       "\n",
       "                    case 84: // t\n",
       "                    that.labelCandidate(true, true);\n",
       "                    break;\n",
       "\n",
       "                    case 70: // f\n",
       "                    that.labelCandidate(false, true);\n",
       "                    break;\n",
       "                }\n",
       "            });\n",
       "\n",
       "            // Show the first page and highlight the first candidate\n",
       "            this.$el.find(\"#viewer-page-0\").show();\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Get candidate selector for currently selected candidate, escaping id properly\n",
       "        getCandidate: function() {\n",
       "            return this.$el.find(\".\"+this.cids[this.pid][this.cxid][this.cid]);\n",
       "        },  \n",
       "\n",
       "        // Color the candidate correctly according to registered label, as well as set highlighting\n",
       "        markCurrentCandidate: function(highlight) {\n",
       "            var cid  = this.cids[this.pid][this.cxid][this.cid];\n",
       "            var tags = this.$el.find(\".\"+cid);\n",
       "\n",
       "            // Clear color classes\n",
       "            tags.removeClass(\"candidate-h\");\n",
       "            tags.removeClass(\"true-candidate\");\n",
       "            tags.removeClass(\"true-candidate-h\");\n",
       "            tags.removeClass(\"false-candidate\");\n",
       "            tags.removeClass(\"false-candidate-h\");\n",
       "            tags.removeClass(\"highlighted\");\n",
       "\n",
       "            if (highlight) {\n",
       "                if (cid in this.labels) {\n",
       "                    tags.addClass(String(this.labels[cid]) + \"-candidate-h\");\n",
       "                } else {\n",
       "                    tags.addClass(\"candidate-h\");\n",
       "                }\n",
       "            \n",
       "            // If un-highlighting, leave with first non-null coloring\n",
       "            } else {\n",
       "                var that = this;\n",
       "                tags.each(function() {\n",
       "                    var cids = $(this).attr('class').split(/\\s+/).map(function(item) {\n",
       "                        return parseInt(item);\n",
       "                    });\n",
       "                    cids.sort();\n",
       "                    for (var i in cids) {\n",
       "                        if (cids[i] in that.labels) {\n",
       "                            var label = that.labels[cids[i]];\n",
       "                            $(this).addClass(String(label) + \"-candidate\");\n",
       "                            $(this).removeClass(String(!label) + \"-candidate\");\n",
       "                            break;\n",
       "                        }\n",
       "                    }\n",
       "                });\n",
       "            }\n",
       "\n",
       "            // Extra highlighting css\n",
       "            if (highlight) {\n",
       "                tags.addClass(\"highlighted\");\n",
       "            }\n",
       "\n",
       "            // Classes for showing direction of relation\n",
       "            if (highlight) {\n",
       "                this.$el.find(\".\"+cid+\"-0\").addClass(\"left-candidate\");\n",
       "                this.$el.find(\".\"+cid+\"-1\").addClass(\"right-candidate\");\n",
       "            } else {\n",
       "                this.$el.find(\".\"+cid+\"-0\").removeClass(\"left-candidate\");\n",
       "                this.$el.find(\".\"+cid+\"-1\").removeClass(\"right-candidate\");\n",
       "            }\n",
       "        },\n",
       "\n",
       "        // Cycle through candidates and highlight, by increment inc\n",
       "        switchCandidate: function(inc) {\n",
       "            var N = this.cids[this.pid].length\n",
       "            var M = this.cids[this.pid][this.cxid].length;\n",
       "            if (N == 0 || M == 0) { return false; }\n",
       "\n",
       "            // Clear highlighting from previous candidate\n",
       "            if (inc != 0) {\n",
       "                this.markCurrentCandidate(false);\n",
       "\n",
       "                // Increment the cid counter\n",
       "\n",
       "                // Move to next context\n",
       "                if (this.cid + inc >= M) {\n",
       "                    while (this.cid + inc >= M) {\n",
       "                        \n",
       "                        // At last context on page, halt\n",
       "                        if (this.cxid == N - 1) {\n",
       "                            this.cid = M - 1;\n",
       "                            inc = 0;\n",
       "                            break;\n",
       "                        \n",
       "                        // Increment to next context\n",
       "                        } else {\n",
       "                            inc -= M - this.cid;\n",
       "                            this.cxid += 1;\n",
       "                            M = this.cids[this.pid][this.cxid].length;\n",
       "                            this.cid = 0;\n",
       "                        }\n",
       "                    }\n",
       "\n",
       "                // Move to previous context\n",
       "                } else if (this.cid + inc < 0) {\n",
       "                    while (this.cid + inc < 0) {\n",
       "                        \n",
       "                        // At first context on page, halt\n",
       "                        if (this.cxid == 0) {\n",
       "                            this.cid = 0;\n",
       "                            inc = 0;\n",
       "                            break;\n",
       "                        \n",
       "                        // Increment to previous context\n",
       "                        } else {\n",
       "                            inc += this.cid + 1;\n",
       "                            this.cxid -= 1;\n",
       "                            M = this.cids[this.pid][this.cxid].length;\n",
       "                            this.cid = M - 1;\n",
       "                        }\n",
       "                    }\n",
       "                }\n",
       "\n",
       "                // Move within current context\n",
       "                this.cid += inc;\n",
       "            }\n",
       "            this.markCurrentCandidate(true);\n",
       "\n",
       "            // Push this new cid to the model\n",
       "            this.model.set('_selected_cid', this.cids[this.pid][this.cxid][this.cid]);\n",
       "            this.touch();\n",
       "        },\n",
       "\n",
       "        // Switch through contexts\n",
       "        switchContext: function(inc) {\n",
       "            this.markCurrentCandidate(false);\n",
       "\n",
       "            // Iterate context on this page\n",
       "            var M = this.cids[this.pid].length;\n",
       "            if (this.cxid + inc < 0) {\n",
       "                this.cxid = 0;\n",
       "            } else if (this.cxid + inc >= M) {\n",
       "                this.cxid = M - 1;\n",
       "            } else {\n",
       "                this.cxid += inc;\n",
       "            }\n",
       "\n",
       "            // Reset cid and set to first candidate\n",
       "            this.cid = 0;\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Switch through pages\n",
       "        switchPage: function(inc) {\n",
       "            this.markCurrentCandidate(false);\n",
       "            this.$el.find(\".viewer-page\").hide();\n",
       "            if (this.pid + inc < 0) {\n",
       "                this.pid = 0;\n",
       "            } else if (this.pid + inc > this.nPages - 1) {\n",
       "                this.pid = this.nPages - 1;\n",
       "            } else {\n",
       "                this.pid += inc;\n",
       "            }\n",
       "            this.$el.find(\"#viewer-page-\"+this.pid).show();\n",
       "\n",
       "            // Show pagination\n",
       "            this.$el.find(\"#page\").html(this.pid);\n",
       "\n",
       "            // Reset cid and set to first candidate\n",
       "            this.cid = 0;\n",
       "            this.cxid = 0;\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Label currently-selected candidate\n",
       "        labelCandidate: function(label, highlighted) {\n",
       "            var c    = this.getCandidate();\n",
       "            var cid  = this.cids[this.pid][this.cxid][this.cid];\n",
       "            var cl   = String(label) + \"-candidate\";\n",
       "            var clh  = String(label) + \"-candidate-h\";\n",
       "            var cln  = String(!label) + \"-candidate\";\n",
       "            var clnh = String(!label) + \"-candidate-h\";\n",
       "\n",
       "            // Toggle label highlighting\n",
       "            if (c.hasClass(cl) || c.hasClass(clh)) {\n",
       "                c.removeClass(cl);\n",
       "                c.removeClass(clh);\n",
       "                if (highlighted) {\n",
       "                    c.addClass(\"candidate-h\");\n",
       "                }\n",
       "                this.labels[cid] = null;\n",
       "                this.send({event: 'delete_label', cid: cid});\n",
       "            } else {\n",
       "                c.removeClass(cln);\n",
       "                c.removeClass(clnh);\n",
       "                if (highlighted) {\n",
       "                    c.addClass(clh);\n",
       "                } else {\n",
       "                    c.addClass(cl);\n",
       "                }\n",
       "                this.labels[cid] = label;\n",
       "                this.send({event: 'set_label', cid: cid, value: label});\n",
       "            }\n",
       "\n",
       "            // Set the label and pass back to the model\n",
       "            this.model.set('_labels_serialized', this.serializeDict(this.labels));\n",
       "            this.touch();\n",
       "        },\n",
       "\n",
       "        // Serialization of hash maps, because traitlets Dict doesn't seem to work...\n",
       "        serializeDict: function(d) {\n",
       "            var s = [];\n",
       "            for (var key in d) {\n",
       "                s.push(key+\"~~\"+d[key]);\n",
       "            }\n",
       "            return s.join();\n",
       "        },\n",
       "\n",
       "        // Deserialization of hash maps\n",
       "        deserializeDict: function(s) {\n",
       "            var d = {};\n",
       "            var entries = s.split(/,/);\n",
       "            var kv;\n",
       "            for (var i in entries) {\n",
       "                kv = entries[i].split(/~~/);\n",
       "                if (kv[1] == \"true\") {\n",
       "                    d[kv[0]] = true;\n",
       "                } else if (kv[1] == \"false\") {\n",
       "                    d[kv[0]] = false;\n",
       "                }\n",
       "            }\n",
       "            return d;\n",
       "        },\n",
       "    });\n",
       "\n",
       "    return {\n",
       "        ViewerView: ViewerView\n",
       "    };\n",
       "});\n"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7b402e09731b45d9a26963de2064c9cc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "SentenceNgramViewer(cids=[[[0]], [[1]], [[2]]], html='<head>\\n<style>\\nspan.candidate {\\n    background-color:…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "LF_distant_supervision = DistantSupervision(\"dbpedia\", kb=known_spouses).lf()\n",
    "labeled = coverage(session, LF_distant_supervision, split=1)\n",
    "\n",
    "# score out LF against dev set labels\n",
    "score(session, LF_distant_supervision, split=1, gold=L_gold_dev)\n",
    "\n",
    "SentenceNgramViewer(labeled, session, n_per_page=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## C. Writing Custom Labeling Functions\n",
    "\n",
    "The strength of LFs is that you can write any arbitrary function and use it to supervise a classification task. This approach can combine many of the same strategies discussed above or encode other information. \n",
    "\n",
    "For example, we observe that when mentions of person names occur far apart in a sentence, this is a good indicator that the candidate's label is False."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "def LF_too_far_apart(c):\n",
    "    \"\"\"Person mentions occur at a distance > 30 words\"\"\"\n",
    "    return -1 if len(list(get_between_tokens(c))) > 30 else 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "labeled = coverage(session, LF_too_far_apart, split=1)\n",
    "score(session, LF_too_far_apart, split=1, gold=L_gold_dev)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## D. Composing Labeling Functions\n",
    "\n",
    "Another useful technique for writing LFs is composing multiple, weaker LFs together. For example, our `LF_marriage` example above has low precision.  Instead of modifying `LF_marriage`, we'll compose it with our `LF_too_far_apart` from above.\n",
    "\n",
    "    LF_marriage                                TP: 63 | FP: 114\n",
    "    LF_marriage AND NOT LF_too_far_apart       TP: 60 | FP: 86\n",
    "\n",
    "We missed 3 true candidates, but we cut our false positive rate by 28 candidates!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========================================\n",
      "LF Score\n",
      "========================================\n",
      "Pos. class accuracy: 1.0\n",
      "Neg. class accuracy: 0.0\n",
      "Precision            0.458\n",
      "Recall               1.0\n",
      "F1                   0.628\n",
      "----------------------------------------\n",
      "TP: 54 | FP: 64 | TN: 0 | FN: 0\n",
      "========================================\n",
      "\n"
     ]
    }
   ],
   "source": [
    "def LF_marriage_and_too_far_apart(c):\n",
    "    return 1 if LF_too_far_apart(c) != -1 and LF_marriage(c) == 1 else 0\n",
    "\n",
    "LF_marriage_and_not_same_person = lambda c: LF_too_far_apart(c) != -1 and LF_marriage(c)\n",
    "\n",
    "score(session, LF_marriage_and_too_far_apart, split=1, gold=L_gold_dev)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# VI. Development Sandbox\n",
    "----\n",
    "\n",
    "## A. Writing Your Own Labeling Functions\n",
    "\n",
    "Using the information above, write your own labeling functions for this task. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "spouses = {'spouse', 'wife', 'husband', 'ex-wife', 'ex-husband'}\n",
    "family = {'father', 'mother', 'sister', 'brother', 'son', 'daughter',\n",
    "              'grandfather', 'grandmother', 'uncle', 'aunt', 'cousin'}\n",
    "family = family | {f + '-in-law' for f in family}\n",
    "\n",
    "\n",
    "def last_name(s):\n",
    "    name_parts = s.split(' ')\n",
    "    return name_parts[-1] if len(name_parts) > 1 else None    \n",
    "\n",
    "def LF_same_last_name(c):\n",
    "    p1_last_name = last_name(c.person1.get_span())\n",
    "    p2_last_name = last_name(c.person2.get_span())\n",
    "    if p1_last_name and p2_last_name and p1_last_name == p2_last_name:\n",
    "        if c.person1.get_span() != c.person2.get_span():\n",
    "            return 1\n",
    "    return 0\n",
    "\n",
    "def LF_no_spouse_in_sentence(c):\n",
    "    return -1 if np.random.rand() < 0.75 and len(spouses.intersection(c.get_parent().words)) == 0 else 0\n",
    "\n",
    "def LF_familial_relationship(c):\n",
    "    return -1 if len(family.intersection(get_between_tokens(c))) > 0 else 0\n",
    "\n",
    "def LF_family_left_window(c):\n",
    "    if len(family.intersection(get_left_tokens(c[0], window=3))) > 0:\n",
    "        return -1\n",
    "    elif len(family.intersection(get_left_tokens(c[1], window=3))) > 0:\n",
    "        return -1\n",
    "    else:\n",
    "        return 0\n",
    "    \n",
    "def LF_husband_wife_left_window(c):\n",
    "    if len(spouses.intersection(get_left_tokens(c[0], window=2))) > 0:\n",
    "        return 1\n",
    "    elif len(spouses.intersection(get_left_tokens(c[1], window=2))) > 0:\n",
    "        return 1\n",
    "    else:\n",
    "        return 0\n",
    "    \n",
    "# Function to remove special characters from text\n",
    "def strip_special(s):\n",
    "    return ''.join(c for c in s if ord(c) < 128)\n",
    "\n",
    "# Read in known spouse pairs and save as set of tuples\n",
    "with bz2.BZ2File('data/spouses_dbpedia.csv.bz2', 'rb') as f:\n",
    "    known_spouses = set(\n",
    "        tuple(strip_special(x.decode('utf-8')).strip().split(',')) for x in f.readlines()\n",
    "    )\n",
    "# Last name pairs for known spouses\n",
    "last_names = set([(last_name(x), last_name(y)) for x, y in known_spouses if last_name(x) and last_name(y)])\n",
    "    \n",
    "def LF_distant_supervision(c):\n",
    "    p1, p2 = c.person1.get_span(), c.person2.get_span()\n",
    "    return 1 if (p1, p2) in known_spouses or (p2, p1) in known_spouses else 0\n",
    "\n",
    "def LF_distant_supervision_last_names(c):\n",
    "    p1, p2 = c.person1.get_span(), c.person2.get_span()\n",
    "    p1n, p2n = last_name(p1), last_name(p2)\n",
    "    return 1 if (p1 != p2) and ((p1n, p2n) in last_names or (p2n, p1n) in last_names) else 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## B. Applying Labeling Functions\n",
    "---\n",
    "\n",
    "Next, we need to actually run the LFs over **all** of our training candidates, producing a set of `Labels` and `LabelKeys` (just the names of the LFs) in the database.  We'll do this using the `LabelAnnotator` class, a UDF which we will again run with `UDFRunner`.\n",
    "\n",
    "### 1. Preparing your Labeling Functions\n",
    "\n",
    "First we put all our labeling functions into list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "LFs = [\n",
    "    LF_no_spouse_in_sentence,\n",
    "    LF_marriage,\n",
    "    LF_other_relationship,\n",
    "    LF_distant_supervision,\n",
    "    LF_too_far_apart,\n",
    "    LF_marriage_and_too_far_apart,\n",
    "    LF_familial_relationship,\n",
    "    LF_family_left_window,\n",
    "    LF_husband_wife_left_window,\n",
    "    LF_distant_supervision_last_names,\n",
    "    LF_distant_supervision\n",
    "]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we setup the label annotator class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "from snorkel.annotations import LabelAnnotator\n",
    "labeler = LabelAnnotator(lfs=LFs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Generating the Label Matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Clearing existing...\n",
      "Running UDF...\n",
      "[========================================] 100%\n",
      "\n",
      "CPU times: user 1min 36s, sys: 184 ms, total: 1min 37s\n",
      "Wall time: 1min 36s\n",
      "(22254, 10)\n",
      "Clearing existing...\n",
      "Running UDF...\n",
      "[========================================] 100%\n",
      "\n",
      "CPU times: user 15.9 s, sys: 90.3 ms, total: 15.9 s\n",
      "Wall time: 15.7 s\n",
      "(2811, 10)\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(1701)\n",
    "\n",
    "%time L_train = labeler.apply(split=0, parallelism=1)\n",
    "print(L_train.shape)\n",
    "\n",
    "%time L_dev = labeler.apply_existing(split=1, parallelism=1)\n",
    "print(L_dev.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>j</th>\n",
       "      <th>Coverage</th>\n",
       "      <th>Overlaps</th>\n",
       "      <th>Conflicts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>LF_no_spouse_in_sentence</th>\n",
       "      <td>0</td>\n",
       "      <td>0.604521</td>\n",
       "      <td>0.152557</td>\n",
       "      <td>0.003864</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_TERMS_marriage_[between|words]_TRUE</th>\n",
       "      <td>1</td>\n",
       "      <td>0.077469</td>\n",
       "      <td>0.077469</td>\n",
       "      <td>0.029208</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_TERMS_other_relationship_[left|words|window=1]_FALSE</th>\n",
       "      <td>2</td>\n",
       "      <td>0.002786</td>\n",
       "      <td>0.002427</td>\n",
       "      <td>0.000135</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_distant_supervision</th>\n",
       "      <td>3</td>\n",
       "      <td>0.001438</td>\n",
       "      <td>0.001393</td>\n",
       "      <td>0.000404</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_too_far_apart</th>\n",
       "      <td>4</td>\n",
       "      <td>0.170666</td>\n",
       "      <td>0.132066</td>\n",
       "      <td>0.028220</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_marriage_and_too_far_apart</th>\n",
       "      <td>5</td>\n",
       "      <td>0.055586</td>\n",
       "      <td>0.055586</td>\n",
       "      <td>0.007325</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_familial_relationship</th>\n",
       "      <td>6</td>\n",
       "      <td>0.104386</td>\n",
       "      <td>0.098005</td>\n",
       "      <td>0.016806</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_family_left_window</th>\n",
       "      <td>7</td>\n",
       "      <td>0.078188</td>\n",
       "      <td>0.072796</td>\n",
       "      <td>0.010290</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_husband_wife_left_window</th>\n",
       "      <td>8</td>\n",
       "      <td>0.077964</td>\n",
       "      <td>0.050193</td>\n",
       "      <td>0.018738</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_distant_supervision_last_names</th>\n",
       "      <td>9</td>\n",
       "      <td>0.008043</td>\n",
       "      <td>0.006875</td>\n",
       "      <td>0.005033</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    j  Coverage  Overlaps  \\\n",
       "LF_no_spouse_in_sentence                            0  0.604521  0.152557   \n",
       "LF_TERMS_marriage_[between|words]_TRUE              1  0.077469  0.077469   \n",
       "LF_TERMS_other_relationship_[left|words|window=...  2  0.002786  0.002427   \n",
       "LF_distant_supervision                              3  0.001438  0.001393   \n",
       "LF_too_far_apart                                    4  0.170666  0.132066   \n",
       "LF_marriage_and_too_far_apart                       5  0.055586  0.055586   \n",
       "LF_familial_relationship                            6  0.104386  0.098005   \n",
       "LF_family_left_window                               7  0.078188  0.072796   \n",
       "LF_husband_wife_left_window                         8  0.077964  0.050193   \n",
       "LF_distant_supervision_last_names                   9  0.008043  0.006875   \n",
       "\n",
       "                                                    Conflicts  \n",
       "LF_no_spouse_in_sentence                             0.003864  \n",
       "LF_TERMS_marriage_[between|words]_TRUE               0.029208  \n",
       "LF_TERMS_other_relationship_[left|words|window=...   0.000135  \n",
       "LF_distant_supervision                               0.000404  \n",
       "LF_too_far_apart                                     0.028220  \n",
       "LF_marriage_and_too_far_apart                        0.007325  \n",
       "LF_familial_relationship                             0.016806  \n",
       "LF_family_left_window                                0.010290  \n",
       "LF_husband_wife_left_window                          0.018738  \n",
       "LF_distant_supervision_last_names                    0.005033  "
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L_train.lf_stats(session)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Label Matrix Empirical Accuracies\n",
    "\n",
    "If we have a small set of human-labeled data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>j</th>\n",
       "      <th>Coverage</th>\n",
       "      <th>Overlaps</th>\n",
       "      <th>Conflicts</th>\n",
       "      <th>TP</th>\n",
       "      <th>FP</th>\n",
       "      <th>FN</th>\n",
       "      <th>TN</th>\n",
       "      <th>Empirical Acc.</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>LF_no_spouse_in_sentence</th>\n",
       "      <td>0</td>\n",
       "      <td>0.625756</td>\n",
       "      <td>0.165422</td>\n",
       "      <td>0.002490</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>51</td>\n",
       "      <td>1637</td>\n",
       "      <td>0.969787</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_TERMS_marriage_[between|words]_TRUE</th>\n",
       "      <td>1</td>\n",
       "      <td>0.063323</td>\n",
       "      <td>0.063323</td>\n",
       "      <td>0.026325</td>\n",
       "      <td>60</td>\n",
       "      <td>109</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.355030</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_TERMS_other_relationship_[left|words|window=1]_FALSE</th>\n",
       "      <td>2</td>\n",
       "      <td>0.003913</td>\n",
       "      <td>0.003202</td>\n",
       "      <td>0.001067</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>11</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_distant_supervision</th>\n",
       "      <td>3</td>\n",
       "      <td>0.001067</td>\n",
       "      <td>0.000711</td>\n",
       "      <td>0.000356</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_too_far_apart</th>\n",
       "      <td>4</td>\n",
       "      <td>0.179651</td>\n",
       "      <td>0.144077</td>\n",
       "      <td>0.030238</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>22</td>\n",
       "      <td>463</td>\n",
       "      <td>0.954639</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_marriage_and_too_far_apart</th>\n",
       "      <td>5</td>\n",
       "      <td>0.044112</td>\n",
       "      <td>0.044112</td>\n",
       "      <td>0.007115</td>\n",
       "      <td>54</td>\n",
       "      <td>64</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.457627</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_familial_relationship</th>\n",
       "      <td>6</td>\n",
       "      <td>0.118107</td>\n",
       "      <td>0.109214</td>\n",
       "      <td>0.024191</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>15</td>\n",
       "      <td>301</td>\n",
       "      <td>0.952532</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_family_left_window</th>\n",
       "      <td>7</td>\n",
       "      <td>0.077552</td>\n",
       "      <td>0.069726</td>\n",
       "      <td>0.010317</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>206</td>\n",
       "      <td>0.990385</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_husband_wife_left_window</th>\n",
       "      <td>8</td>\n",
       "      <td>0.069726</td>\n",
       "      <td>0.047314</td>\n",
       "      <td>0.021700</td>\n",
       "      <td>81</td>\n",
       "      <td>107</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.430851</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_distant_supervision_last_names</th>\n",
       "      <td>9</td>\n",
       "      <td>0.006403</td>\n",
       "      <td>0.005336</td>\n",
       "      <td>0.003913</td>\n",
       "      <td>7</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.388889</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    j  Coverage  Overlaps  \\\n",
       "LF_no_spouse_in_sentence                            0  0.625756  0.165422   \n",
       "LF_TERMS_marriage_[between|words]_TRUE              1  0.063323  0.063323   \n",
       "LF_TERMS_other_relationship_[left|words|window=...  2  0.003913  0.003202   \n",
       "LF_distant_supervision                              3  0.001067  0.000711   \n",
       "LF_too_far_apart                                    4  0.179651  0.144077   \n",
       "LF_marriage_and_too_far_apart                       5  0.044112  0.044112   \n",
       "LF_familial_relationship                            6  0.118107  0.109214   \n",
       "LF_family_left_window                               7  0.077552  0.069726   \n",
       "LF_husband_wife_left_window                         8  0.069726  0.047314   \n",
       "LF_distant_supervision_last_names                   9  0.006403  0.005336   \n",
       "\n",
       "                                                    Conflicts  TP   FP  FN  \\\n",
       "LF_no_spouse_in_sentence                             0.002490   0    0  51   \n",
       "LF_TERMS_marriage_[between|words]_TRUE               0.026325  60  109   0   \n",
       "LF_TERMS_other_relationship_[left|words|window=...   0.001067   0    0   0   \n",
       "LF_distant_supervision                               0.000356   2    1   0   \n",
       "LF_too_far_apart                                     0.030238   0    0  22   \n",
       "LF_marriage_and_too_far_apart                        0.007115  54   64   0   \n",
       "LF_familial_relationship                             0.024191   0    0  15   \n",
       "LF_family_left_window                                0.010317   0    0   2   \n",
       "LF_husband_wife_left_window                          0.021700  81  107   0   \n",
       "LF_distant_supervision_last_names                    0.003913   7   11   0   \n",
       "\n",
       "                                                      TN  Empirical Acc.  \n",
       "LF_no_spouse_in_sentence                            1637        0.969787  \n",
       "LF_TERMS_marriage_[between|words]_TRUE                 0        0.355030  \n",
       "LF_TERMS_other_relationship_[left|words|window=...    11        1.000000  \n",
       "LF_distant_supervision                                 0        0.666667  \n",
       "LF_too_far_apart                                     463        0.954639  \n",
       "LF_marriage_and_too_far_apart                          0        0.457627  \n",
       "LF_familial_relationship                             301        0.952532  \n",
       "LF_family_left_window                                206        0.990385  \n",
       "LF_husband_wife_left_window                            0        0.430851  \n",
       "LF_distant_supervision_last_names                      0        0.388889  "
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L_dev.lf_stats(session, labels=L_gold_dev.toarray().ravel())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 3. Iterating on Labeling Function Design\n",
    "\n",
    "When writing labeling functions, you will want to iterate on the process outlined above several times. You should focus on tuning individual LFs, based on emprical accuracy metrics, and adding new LFs to improve coverage. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-butt": {},
  "kernelspec": {
   "display_name": "Python (snorkel)",
   "language": "python",
   "name": "snorkel"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}