example.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from pathlib import Path\n",
    "from modelpicker import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example Task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As an example use case, we will select the best pretrained model for a task of contextual emotion detection from text. The collection of pretrained models is formed from the development history of a participant of the [EmoContext](https://www.humanizing-ai.com/emocontext.html) task in SemEval 2019. Changes at each development step include adding various word representations such as ELMo and GloVe and leveraging speaker embeddings and/or universal sentence encoder, which creates performance differences among the models. \n",
    "\n",
    "Our goal is to select the best pretrained model to make predictions on the unlabelled instances by only partially labelling a very few of 5,509 of them via ```modelpicker```."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Picker"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run this command on terminal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Please enter the label for the instance with ID 1346:\n"
     ]
    }
   ],
   "source": [
    "# Modelpicker takes the following arguments in order: --path to prediction file (CSV) --path to labelspace file (CSV) --an integer budget\n",
    "%run modelpicker data/emocontext/predictions data/emocontext/labelspace 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Or take numpy arrays as inputs in your code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we load the predictions matrix and labelspace contained in the ```data/emocontext/``` path. They are both ```CSV``` files and the labels are coded by integers. Map your labels to integers before you proceed.\n",
    "\n",
    "In the example we have below, we have 8 different models. The data consists of 8 model predictions on 5,509 unlabeled data instances. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set filenames\n",
    "filename_predictions = 'predictions'\n",
    "filename_labelspace = 'labelspace'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Model collections and label set\n",
    "\n",
    "datapath = Path(r'data/emocontext') # set path\n",
    "\n",
    "file_predictions = open(str(datapath)+'/'+str(filename_predictions)+'.csv') # read predictions\n",
    "mypredictions = np.loadtxt(file_predictions, delimiter=\",\")\n",
    "\n",
    "file_labelspace = open(str(datapath)+'/'+str(filename_labelspace)+'.csv') # read label space\n",
    "mylabelspace = np.loadtxt(file_labelspace, delimiter=\",\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run Model Picker"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ```modelpicker``` algorithm takes the following inputs\n",
    "\n",
    "**ARGUMENTS**\n",
    "\n",
    "- _predictions_ The name of your CSV file consisting of model predictions. This is a 2D array of model predictions on your freshly collected data with size 𝑁×𝑘 where 𝑁 is the amount of unlabeled instances available at time 𝑡, and 𝑘 is the number of models. Each prediction is mapped to an integer.\n",
    "\n",
    "- _labelset_ The name of your CSV file consisting of elements of label space. For instance, for a dataset consisting of 4 classes, a possible label space can be {0,1,2,3}. These labels should be consistent with the mapping used for prediction matrix as well.\n",
    "\n",
    "- _budget_ An integer that indicates number of labeling the user wants to do.\n",
    "\n",
    "At the output, the algorithm returns the following:\n",
    "\n",
    "**OUTPUTS**\n",
    "\n",
    "- _bestmodel_ ID of the winner model based on the requested labels\n",
    "\n",
    "- _beliefs_ An array of size $k$ that quantifies the posterior belief on each model being the best one. The posterior belief also hints the ranking of models. That is, the higher nominal value will indicate a higher belief on that model being the best."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Set budget\n",
    "budget = 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Please enter the label for the instance with ID 3377:\n",
      "3\n",
      "Please enter the label for the instance with ID 1225:\n",
      "3\n",
      "Please enter the label for the instance with ID 1288:\n",
      "0\n",
      "Please enter the label for the instance with ID 1320:\n",
      "3\n",
      "Please enter the label for the instance with ID 1183:\n",
      "3\n"
     ]
    }
   ],
   "source": [
    "## Run model picker\n",
    "\n",
    "(bestmodel, beliefs) = modelpicker(mypredictions, mylabelspace, budget)\n",
    "\n",
    "# Note: for the sake of completeness, we added the ground truth labels for this dataset (see data/emocontext/oracle.csv).\n",
    "# For your own dataset, labeling is left to the user. The labeling shows below is based on the ground truths."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('ID of best model: '+ bestmodel)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}