umd-coherence/coherence.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Is Automated Topic Model Evaluation Broken? The Incoherence of Coherence<sup>1</sup>\n",
    "\n",
    "### Authors: Alexander Hoyle, Pranav Goel, Denis Peskov, Andrew Hian-Cheong, Jordan Boyd-Graber and Philip Resnik (University of Maryland)\n",
    "\n",
    "Link to paper: https://openreview.net/forum?id=tjdHCnPqoo\n",
    "\n",
    "\n",
    "Paper walkthrough by Armin Catovic, Sr. Data Scientist, Schibsted\n",
    "\n",
    "<sub><sup>1</sup> Presented in a poster session at NeurIPS 2021</sub>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "> ___When a measure becomes a target, it ceases to be a good measure___ _(Goodharts' Law)_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Background\n",
    "\n",
    "* The goal of topic models is to facilitate human understanding of a text<sup>2</sup> corpus\n",
    "* While real-world users evaluate topics based on their specific needs, research/academia has gravitated towards automated proxies of human judgment, i.e. automated evaluation metrics\n",
    "* __Perplexity__ based evaluations are now seldom used since they are negatively correlated with human interpretability\n",
    "* The current standard for automated evaluation of topic models is using __coherence__ metrics - most popular being __Normalized Pointwise Mutual Information (NPMI)__\n",
    "* Over the last few years most of the topic modelling has migrated to neural based approaches, with ever-increasing NPMI scores and state-of-the-art (SOTA) claims\n",
    "* However, __very few papers have used human evaluations__ when reporting results<sup>3</sup>\n",
    "* Neural models seem to manifest qualitatively distinct topics compared to classical approaches such as Latent Dirichlet Allocation (LDA) - __can a single coherence metric equally apply to both neural based as well as classical approaches?__\n",
    "* There seems to be a wide __standardization gap__ when it comes to pre-processing of text, as well as in the calculation of the NPMI itself\n",
    "\n",
    "<sub><sup>2</sup> Topic modelling has also been applied to non-textual datasets, such as images and genome sequences.</sub>\n",
    "\n",
    "<sub><sup>3</sup> See <em>Table 6</em> in <em>Appendix A</em> of the original paper. Outside the core method-development literature, human evaluations have been used to develop new metrics and improve understanding of existing model behaviour.</sub>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Approach\n",
    "\n",
    "Hoyle et al make the following contributions in the paper:\n",
    "\n",
    "* Present a meta-analysis of neural topic model evaluation and the current state of affairs\n",
    "* Develop a standardized, re-producable, pre-processed versions of two widely used evaluation datasets - __Wikitext-103__ (Wikipedia corpus) and __LDC2008T19__ (New York Times corpus)\n",
    "* Optimize three topic models - one classical (LDA with Gibbs sampling) and two neural (ETM and D-VAE) - using identical pre-processing, model selection criteria, and hyperparameter tuning\n",
    "* Obtain human evaluations of these models using ratings and word intrusion tasks\n",
    "* Provide new evaluations of the correlation between automated and human evaluations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Motivation\n",
    "\n",
    "* Question the validity of fully automated evaluations\n",
    "* Define a standard for text pre-processing, model tuning, and model evluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Topic Modelling in Perspective\n",
    "\n",
    "* Topic models, such as __Latent Dirichlet Allocation (LDA)__, are probabilistic generative models of text - they \"tell a (somewhat simplified) story\" of how a document came to be\n",
    "* Topic models assume that each document is a sparse mixture of topics, and that each topic is a specific categorical distribution over the words/vocabulary\n",
    "* When we evaluate topic models, we typically look at (1) most probable $N$ words (e.g. $N=10$) in each topic, and (2) most probable topic assignments for each document\n",
    "* With the emergence of deep learning, topic modelling methods have mostly migrated towards __Neural Topic Models (NTMs)__, where instead of using sparse representations and Markov Chain Monte Carlo (MCMC) based estimators, NTMs use dense continuous word representations and gradient optimization to fit the parameters\n",
    "* NTMs have gained in popularity because of ___results suggesting they produce more interpretable topics compared to \"classical\" methods such as LDA___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## NPMI - The Standard Topic Model Coherence Evaluation\n",
    "\n",
    "* __Coherence__ stems from a Latin _cohaerere_, meaning _\"to cling/stick (closely) together\"_\n",
    "* A topic is said to be coherent, if given its set of $N$ most probable terms, when viewed together, enables one to recognise the topic as an identifiable category<sup>4</sup>\n",
    "* Today, the evalutation consensus is to use __pairwise normalized pointwise mutual information (NPMI)__, as defined below:\n",
    "\n",
    "![NPMI](./figures/npmi.png \"NPMI\")\n",
    "\n",
    "* NPMI scores highly if top $N$ words - summed over all pairs of $w_i$ and $w_j$ - have high joint probability $P(w_i, w_j)$ compared to their marginal probability\n",
    "* The probabilities are estimated using word co-occurance counts from a ___reference corpus___ for a specific ___context window___, which can range from ten words, up to the entire document\n",
    "* Therefore, the choice of a reference corpus and corresponding context window determine the NPMI score\n",
    "* Furthermore, since neural word representations are intimately connected to NPMI, there is a potential for NTMs to generate topics with high NPMI scores, without (qualitatively) explaining the corpus well to the user\n",
    "\n",
    "<sub><sup>4</sup> Doogan and Buntine state that \"an interpretable topic is one that can be easily labeled\".</sub>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Human Metrics of Topic Coherence\n",
    "\n",
    "\n",
    "* Hoyle et al obtain human evaluations for three different topic models\n",
    "* They use __Prolific.co__ to recruit crowdworkers and collect the data via __Qualtrics__ survey platform\n",
    "* The authors apply two well established human evaluation tasks - __word intrusion__ and __topic rating__\n",
    "* __Intrusion__ - behavioural way to assess topic coherence\n",
    "  * Each topic is represented by its top $N$ words plus one _\"intruder\"_ word, which has a low probability of belonging to that topic, but a high probability of belonging to a different topic\n",
    "  * Topic coherence is judged by how well human annotators detect the _\"intruder\"_ word\n",
    "* __Rating__ - human evaluators are presented with a topic and its corresponding $N$ most probable words; they have to rate the _\"quality\"_ of the topic using a three-point ordinal scale"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Human Evaluation and Statistical Power\n",
    "\n",
    "\n",
    "* Many NLP experiments, including those relying on human evaluation, are insufficiently powered to detect model differences at reported levels\n",
    "* To select the number of crowdworkers, the authors conducted a power analysis with simulated data by formulating a generative model of annotations<sup>5</sup>\n",
    "* This approach - ensuring statistical power of at least $1 - \\beta = 0.9$, and significance at $\\alpha = 0.05$ - resulted in a minimum of __15 crowdworkers per topic__, for both intrusion as well as rating tasks\n",
    "* The authors also used pairwise significance tests to establish model differences using human ratings - using one-tailed tests for each pair in both directions\n",
    "  * A __proportion test__ was used for the intrusion scores\n",
    "  * A __Mann–Whitney $U$ test__ was used for the ratings\n",
    "  * A __Student's $t$-test__ was used for automated metrics\n",
    "\n",
    "<sub><sup>5</sup> See Appendix A.5 Power Analysis for Human Evaluation Tasks.</sub>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Human Evaluation - Word Intrusion\n",
    "\n",
    "![Figure 1a](./figures/human_evaluation_intrusion.png \"Figure 1a\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Human Evaluation - Topic Rating\n",
    "\n",
    "![Figure 1b](./figures/human_evaluation_rating.png \"Figure 1b\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## A Meta-Analysis of Neural Topic Modeling\n",
    "\n",
    "While analysing a large body of NTM literature, related to forty different models, all claiming superior topic coherence scores, the authors have found the following:\n",
    "\n",
    "* Variance in all areas subject to authors' analysis\n",
    "* Pre-processing, which can significantly affect model quality, is 30% of the time inconsistent across datasets __within the same paper__\n",
    "* Some pre-processing details are often omitted making it difficult/impossible to replicate the pipeline\n",
    "* Datasets used to establish relationships between human annotations and automated metrics, are different from datasets used to train the models\n",
    "* 40% of papers fail to clearly specify their model tuning and selection procedure\n",
    "* NPMI evaluation itself is ambiguous:\n",
    "  * Reference corpus is often not specified\n",
    "  * Co-occurance window size and top-$N$ most probable words per topic often not specified\n",
    "  * Often not clear which __implementation__ of coherence metric is actually used"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Closing the Standardization Gap for Topic Models\n",
    "\n",
    "* __Dataset selection__:\n",
    "  * The authors choose Wikipedia (__Wikitext-103__) and NY Times (__LDC2008T19__) for both training and as a reference (evaluation) corpus\n",
    "* __Pre-processing pipeline__:\n",
    "  * The authors specify in great detail<sup>6</sup>, all aspects of document processing, vocabulary creation, and vocabulary filtering\n",
    "    * The use of Zipf law for vocabulary pruning, i.e. by removing tokens that appear in fewer than $2(0.02|D|)^{-log10}$ documents ($|D|$ is the size of corpus), is particularly enlightening\n",
    "* __Model selection__:\n",
    "  * Three different models are used for comparison purposes\n",
    "    * Gibbs-LDA (G-LDA) is a \"classic\" LDA topic model, estimated using partially collapsed Gibbs sampler implemented in a Mallet Java package\n",
    "    * Dirichlet-VAE (D-VAE) is a SOTA NTM\n",
    "    * Embedded Topic Model (ETM) is similar to LDA, except it uses dense word embeddings in its generative model\n",
    "    * Fixed computational budget per model is maintained, and same hyperparameter selection process is applied to all three models<sup>7</sup>\n",
    "    * Authors eliminate models with highly redundant topics as follows:\n",
    "      * Models in which any of the top-5 words of one topic overlap with another\n",
    "      * Models that have a topic uniqueness score above 0.7\n",
    "* __Topic coherence metric__:\n",
    "  * Authors use NPMI, estimated using the reference corpus with a 10-word window over the top-10 topic words\n",
    "\n",
    "\n",
    "<sub><sup>6</sup> See Appendix A.2 Preprocessing Details.</sub>\n",
    "\n",
    "<sub><sup>7</sup> See Appendix A.3 Training Details.</sub>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Results - Human Judgment Differs from Automated Metrics\n",
    "\n",
    "![Figure 2](./figures/results_automated_vs_human_evaluation.png \"Figure 2\")\n",
    "\n",
    "__Note:__ Coloured circles correspond to pairwise one-tailed significance tests between model scores at $\\alpha = 0.05$; for example, the right-most circle at bottom right shows that human evaluation of topic ratings for D-VAE are significantly higher than ETM for topics derived from Wikipedia."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Results - Human Judgment Differs from Automated Metrics (2)\n",
    "\n",
    "![Figure 3](./figures/results_human_evaluation_all_vs_filtered.png \"Figure 3\")\n",
    "\n",
    "__Note:__ Mean human evaluation on the ratings and word intrusion tasks, after filtering out respondents who reported a lack of familiarity with the topic words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Discussion and Conclusion\n",
    "\n",
    "* While automated evaluations (using NPMI) suggest D-VAE as a clear winner between models, human evaluation is more nuanced\n",
    "  * Human judgments exhibit greater variability\n",
    "  * Although D-VAE fares better on the intrusion task, evaluation using topic ratings favours \"good old\" G-LDA\n",
    "* While human evaluators report familiarity with terms over 90% of the time for both G-LDA and ETM, D-VAE has a notably lower average term familiarity (70%)\n",
    "  * This difference, as well as the validation by filtering out respondents who report lack of term familiarity, indicate that D-VAE, by-default, produces more _\"esoteric\"_ topics that are narrower in scope than those of other models\n",
    "* Automated coherence metrics, such as NPMI, merely provide good _\"guidance\"_, but should not be solely used to judge topic quality, nor to make superiority claims\n",
    "  * __Human evaluations (with statistically significant number of human evaluators) should be an integral part of evaluating topic models, and possibly all unsupervised models__\n",
    "* Standardizing datasets, model tuning, and pre-processing pipelines, should be considered a high priority for topic modelling researchers\n",
    "  * Hoyle et al do a good job of formulating one such evaluation standard\n",
    "* The findings, i.e. lack of consistency, reproducability, and the use of Goodhart's law, are in a way reflective of the __general trend in machine learning, and particularly where unsupervised learning is concerned.__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Personal Reflections...\n",
    "\n",
    "* We use topic models - currently LDA with Gibbs sampling, i.e. same as G-LDA specified in the paper - as part of our contextual advertising product, so this paper is highly relevant to us\n",
    "* We 100% agree that text pre-processing is one of the most influential factors on model performance (and topic coherence)\n",
    "  * We put constraints on minimum article/document length, we perform vocabulary pruning, and we use a custom list of stop words that was developed together with our product specialists\n",
    "  * The idea from Hoyle et al. of constraining vocabulary according to power-law distribution makes a lot of sense\n",
    "* We perform usability tests and topic evaluation together with our product specialists and sales reps\n",
    "  * In this regard, we are also 100% in agreement with the paper\n",
    "  * The use of formalized intrusion/rating tasks could be an improvement and could be applied on larger scale across Schibsted Marketing Services\n",
    "* The authors demonstrated scientific/statistical rigour when it came to crowdworker selection, and model performance comparisons - this level of statistical significance reporting/transparancy is rare in ML papers\n",
    "* Overall a very important paper, and very happy that it got some spotlight at NeurIPS 2021, as this standardization gap in topic modelling and ML as a whole, is becoming increasingly important"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}