Introduction to Text as Data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Text as Data\n",
    "\n",
    "## UNM Workshop, April 9, 2018"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learning Objectives\n",
    "\n",
    "In today's workshop, we're going to combine the [tidyverse workflow](https://cran.r-project.org/web/packages/tidyverse/vignettes/manifesto.html) with a set of common TAD goals. The tidyverse is a group of R packages created by Hadley Wickham and others which prioritizes a common representation of data and data workflows. You should have gotten at least a little bit of exposure to the tidyverse prior to this workshop. If not, that's still fine. But make sure to look up any particular verbs which aren't familiar. \n",
    "\n",
    "By the end of the workshop, you should be able to:\n",
    "\n",
    "- understand the data flow of `tidyverse` and `tidytext`\n",
    "- count the most common words from a particular text\n",
    "- compare two texts in word proportions\n",
    "- apply a structural topic model to a set of documents using `stm`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What are Text as Data methods?\n",
    "\n",
    "Text as data (TAD) methods are a body of different computational methods for doing analysis of natural language. TAD is drawn from a number of different fields, including computational linguistics, natural language processing, and machine learning.\n",
    "\n",
    "TAD as a body of methods has grown as a means of dealing with large amounts of political and social text. Within sociology, TAD has been largely used as a means of studying culture and institutional logics ([Mohr and Bogdanov 2013](https://www.sciencedirect.com/journal/poetics/vol/41); [Nelson 2017](http://journals.sagepub.com/doi/abs/10.1177/0049124117729703)). There's been a more widespread adoption of TAD within political science and political methodology more specifically. Earlier interventions into this space focused on ideological scaling ([Laver, Benoit, and Garry 2003](http://faculty.washington.edu/jwilker/tft/Laver.pdf); [Slapin and Proksch 2008](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.704.3894&rep=rep1&type=pdf)), i.e. identifying ideological positions of political actors via some kind of written or spoken text. More recent work has focused on classification of text, either based on known categories or unknown categories. Categories can be diverse as political sentiment towards a particular topic or politician ([Hopkins and King 2010](https://gking.harvard.edu/files/words.pdf)) to types of protest speech ([Hanna 2013](http://alex-hanna.com/static/pdf/Hanna.Moby2013.pdf)) to political credit taking ([Grimmer 2013](http://www.cambridge.org/us/academic/subjects/politics-international-relations/american-government-politics-and-policy/representational-style-congress-what-legislators-say-and-why-it-matters#SlBUdh8bBkEg8eGI.97)). \n",
    "\n",
    "There's a separate but related strain of TAD which is less concerned with ideological scaling or categorization and more concerned with event data creation. There's a long history of this methodology, largely pioneered by Phil Schrodt ([2012](https://www.researchgate.net/profile/Philip_Schrodt/publication/254242323_Precedents_Progress_and_Prospects_in_Political_Event_Data/links/543fc29a0cf21227a11b77af/Precedents-Progress-and-Prospects-in-Political-Event-Data.pdf)). This has been a space of intervention for both Dr. Osorio and myself ([Osorio and Reyes 2016](https://docs.wixstatic.com/ugd/76b255_12d39b7bb99d402687359d72c8a769b9.pdf); [Hanna 2017](https://osf.io/preprints/socarxiv/xuqmv)).\n",
    "\n",
    "The figure below (slightly amended to include event data) displays directives of how to choose a particular TAD method for your own purposes. The article from which it comes ([Grimmer and Stewart 2013](https://web.stanford.edu/~jgrimmer/tad2.pdf)) provides an excellent set of procedures for performing TAD transparently, replicably, and validly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![overview of TAD](img/Grimmer-Stewart-mod.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 1\n",
    "\n",
    "1. I want to classify Congressional press releases based on a classification scheme I've generated through a literature review. Which method(s) could I consider using?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using `tidytext`\n",
    "\n",
    "[`tidytext`](https://cran.r-project.org/web/packages/tidytext/vignettes/tidytext.html) is an R package develped by Julia Silge and David Robinson, created for the purpose of integrating text processing methods into the tidyverse. This package allows us to preprocess text in a number of ways, such as converting it to lowercase, removing stopwords, and tokenizing it. It also allows us to convert between different methods for importing into other text packages. We'll get into what this all means below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start with single text, in this case, Mary Wollenstonecraft's *A Vindication of the Rights of Woman*. This work is considered a cornerstone of political philosophy in first-wave feminism. Let's start digging into some of its language."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First let's load the `tidyverse`, `tidytext`, and `gutenbergr`, the last of which is a package developed by David Robinson which allows us to easily load in books from Project Gutenberg with only an numerical ID or a title."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(tidyverse)\n",
    "library(tidytext)\n",
    "library(gutenbergr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we're going to download *Vindication* and convert it to tidy format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## A Vindication of the Rights of Woman by Mary Wollstonecraft\n",
    "vindication <- gutenberg_download(3420)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Project Gutenberg gives us a text as a set of lines. Next, we're going to use one of the workhorse \"verbs\" of `tidytext` -- `unnest_tokens` -- to convert text to lowercase and *tokenize* the text.  \n",
    "\n",
    "On the level of meaning, the lowercase and uppercase of a word generally mean the same thing (sometimes they don't, though. For instance, an SMS saying 'Thank you.' means something different from one saying 'THANK YOU.'). But for the most part, we will want to convert to lowercase.\n",
    "\n",
    "Secondly, we want to split the text in \"tokens\" by some kind of rule. This is the process of tokenization. For many applications, we mostly want to split by individual words and remove punctuation. \n",
    "\n",
    "`unnest_tokens` takes two mandatory arguments in this format -- the output column name and the input column name. We can call the output column `word`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tidy_vindication <- vindication %>% \n",
    "    unnest_tokens(word, text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of `unnest_tokens` with no other arguments is a list of each word (or token) in order. For some purposes, we may also want to use conjunctions of multiple words, what are called *ngrams*. This allows us to have a little more context for the word. For instance, there's more meaning in \"white house\" than in \"white\" or \"house\" alone. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vindication %>% unnest_tokens(word, text, token = \"ngrams\", n = 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This one-per-row format lends itself nicely to other functions of the tidyverse. We can now do things like count unique items, join words with other data, and summarize across useful common variables.\n",
    "\n",
    "The most basic thing we can begin to do is simply count the top words in the text. Using the `count` verb makes this straightforward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tidy_vindication %>% count(word, sort = TRUE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nothing too exciting here. Most of the top words are common words, like \"and\" or \"the\". These are what are known as *stop words* and we typically want to remove them from analysis. Luckily, there's another verb we can use to this, called `anti_join`. Simply put, this will join a dataset on common elements and remove those elements from the dataset. In this case, we'll join the list of English stop words with the list of single words, then we'll count them again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tidy_vindication <- tidy_vindication %>%\n",
    "    anti_join(stop_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tidy_vindication %>% count(word, sort = TRUE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we're getting somewhere. We're getting some important words from this text, such as *women, reason, mind*, and *virtue*.\n",
    "\n",
    "Lastly, let's look if there are some important bigrams (these are n-grams with length 2) which may be important in this text. We can't just use `anti_join` to remove stop words in this instance. We have to first split each bigram with `separate`, check if each word is in the stop word list with `filter`, then count the word pairs. One of the great things about the tidyverse is that we can stop after each verb and check the output to see if it makes sense."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## bigram stopword removal\n",
    "bigrams <- vindication %>%\n",
    "    unnest_tokens(bigram, text, token = \"ngrams\", n = 2) %>%\n",
    "    separate(bigram, c(\"word1\", \"word2\"), sep = \" \") %>%\n",
    "    filter(!word1 %in% stop_words$word) %>%\n",
    "    filter(!word2 %in% stop_words$word) %>%\n",
    "    count(word1, word2, sort = TRUE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this analysis, we see that in the most common bigrams, Wollstonecraft is using more universalist language, such as *fellow creatures*, *sufficient strength*, and *common sense*.\n",
    "\n",
    "Now it's your turn!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 2\n",
    "\n",
    "1. Load another text, *Souls of Black Folks* by W.E.B. DuBois from `gutenbergr`. The ID is 408. Store it in a variable called `souls`.\n",
    "2. Convert `souls` to a list of words using tidytext and remove the stop words. Store this in `tidy_souls`.\n",
    "3. Generate a count of the top words. Now, select only the top 20 words in the counted word list. You'll need to use the tidyverse verb `top_n`, which takes a single argument, the number of rows to display."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparing texts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We rarely want to look at a text in isolation. Typically, we want to compare texts across time (e.g. see the change in the prevalence of a topic), across individual authors (e.g. across politicians), or some other variable of interest. For this reason, it's helpful to know how to compare different texts to each other. \n",
    "\n",
    "We could do this through a number of methods. We could, for instance, look at common words and compare the proportion of each author's text. Laura K. Nelson, for instance, performs this procedure [as a first step](https://github.com/lknelson/computational-grounded-theory) in comparing texts from different feminist organizations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's load our second first-wave feminist text, *The Subjection of Women* by John Stuart Mill from `gutenbergr` and convert it to tidy format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "subjection <- gutenberg_download(27083)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tidy_subjection <- subjection %>% \n",
    "    unnest_tokens(word, text) %>%\n",
    "    anti_join(stop_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next thing we're going to do is smush together the tidy data frames for each author. The `mutate` verb adds a new column to a data frame. `bind_rows` adds two data frames together sequentially."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tidy_books <- bind_rows(mutate(tidy_subjection, author = 'Mills'), \n",
    "                        mutate(tidy_vindication, author = 'Wollstonecraft'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we're going to go through a few different tidy data procedures to compare the two authors. We are first going to count each word but now grouped by author. We then generate a new column with `mutate` which will result in a proportion. We then must remove the column `n` from the analysis for the next few parts to work correctly using `select`. Next we move from a long format to a wide format using `spread`. Spread moves the proportions from a single column to two columns, one for each author. We lastly take the difference between the two proportions and sort them by descending order using `arrange`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_proportions <- tidy_books %>% \n",
    "    count(word, author) %>%\n",
    "    group_by(author) %>%\n",
    "    mutate(proportion = n / sum(n)) %>%\n",
    "    select(-n) %>%\n",
    "    spread(author, proportion, fill = 0) %>%\n",
    "    mutate(diffprop = Mills - Wollstonecraft) %>%\n",
    "    arrange(desc(diffprop))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the top (and bottom) 10 in difference of proportion with `head` and `tail`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_proportions %>% head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_proportions %>% tail(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What we can begin to see here is that Mills uses the word *women* more than Wollstonecraft, but also uses more legalistic language: *law, power, human*, and *social*. Meanwhile, Wollstonecraft uses much more gendered language: *virtue, love, heart*, and *understanding*, although *mind* and *reason* are important exceptions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's visualize these results with `ggplot2`, which will allow us to more readily see the scale of difference. We're going to take the top 10 positive and top 10 negative words. `mutate` here reorders them such that the list is ordered by `diffprop`. We'll draw a bar graph with `geom_col` and flip the graph on its side with `coord_flip`. We'll add a label for good measure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_proportions %>%\n",
    "    group_by(diffprop < 0) %>%\n",
    "    top_n(10, abs(diffprop)) %>%\n",
    "    ungroup() %>%\n",
    "    mutate(word = reorder(word, diffprop)) %>%\n",
    "    ggplot(aes(word, diffprop, fill = diffprop < 0)) +\n",
    "    geom_col(show.legend = FALSE) + \n",
    "    coord_flip() + \n",
    "    ylab(\"Difference of proportions (Mills/Wollstonecraft)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see similar results but see that the scale of the use of *women* and *reason* is larger than other words which differentiate these two authors.\n",
    "\n",
    "Your turn again!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise 3\n",
    "\n",
    "1. Let's load *Up from Slavery* by Booker T. Washington using `gutenbergr`. The ID is 2376. Store it in a variable called `slavery`.\n",
    "2. Generate the tidy data frame without stop words. Store in `tidy_slavery`.\n",
    "3. Merge the two data frames and created a new data frame for author. Store in `tidy_compare`.\n",
    "4. Generate a difference of proportions and plot this difference. What differs the most in DuBois? And what differs the most in Washington?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generating topic models with `stm`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a last exercise, we'll be working with topic models. Topic models have gained a lot of traction within the sociology of culture, political science, and the digital humanities. They allow us to generate a probability distribution of topics over both our words and documents. Unlike ideological scaling methods, topics are interpreted *post hoc*, so it's up to the researcher to interpret what each topic is about. Topic models are a mixed membership model, which means that words and documents can belong to mulitple topics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this exercise, we're going to use the [structural topic model](https://www.structuraltopicmodel.com/) or `stm` package, which was developed by political scientists Molly Roberts, Brandon Stewart, and Dustin Tingley. The structured topic model provides a fast implementation of topic models but also provides the added bonus in that you can incorporate document-level covariates into the model itself and associated topics with those covariates. Imagine incorporating the effect of calendar time, party, and ideology into document analysis of Congressional press releases or floor speeches."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We most likely won't get into analyzing covariates today, however. But I do want to demonstrate how to use `stm` with `tidytext` and how to begin interpreting topic models. We'll begin by loading `stm`, as well as two other packages we need for the analysis: `stringr` (for string manipulation) and `RCurl` (allows us to download from the web). If neither are installed, you can do `install.packages` on them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(stm)\n",
    "library(stringr)\n",
    "\n",
    "## if this doesn't exist: install.packages('RCurl')\n",
    "library(RCurl)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As for data, we're pulling a sample of tweets made during the 2018 State of the Union address and using the hashtag *#sotu*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets <- getURL('http://alex-hanna.com/research/sotu-text.csv', .encoding = 'UTF-8')\n",
    "tweets <- read_csv(tweets)\n",
    "tweets <- tweets %>% mutate(id = as.character(id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To preprocess these tweets, we're going to do a little more cleaning on the front end. First we're going to use `stringr` to remove any links (strings which begin with https) from the tweets. The default way that `unnest_tokens` tokenizes makes the results a bit weird. We're also going to remove *sotu* and *trump*, because both end up being pretty meaningless words in the analysis but end up being prevalent across many topics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tidy_tweets <- tweets %>% \n",
    "    mutate(text = str_replace(text, 'https://.+', '')) %>%\n",
    "    unnest_tokens(word, text) %>%\n",
    "    anti_join(stop_words) %>%\n",
    "    filter(!word %in% c(\"sotu\", \"trump\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we're going to convert the words from tidy format into a format more familiar to `stm`, which is a document-frequency matrix (dfm), using `cast_dfm`. In a word, each row of the dfm represents a document, while each column is a frequency count of a particular token. We feed into `cast_dfm` a document ID (in this case our tweet ID) plus a count for each word in that document.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweet_dfm <- tidy_tweets %>% \n",
    "    count(id, word) %>%\n",
    "    cast_dfm(id, word, n)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, we are able to use the workhorse function of the `stm`, conveniently called `stm`. We provide a number of topics for the model to fit. You can play around a bit with this, depending on your domain knowledge. There's a literature on how to pick topic numbers which is beyond the scope of this workshop, however. For simplicity, we'll consider 8."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "k <- 8\n",
    "topic_model <- stm(tweet_dfm, K = k, verbose = FALSE, init.type = \"Spectral\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After generating the topic model, we can convert it back to tidy format with `tidy`. Without any additional arguments, this will provide a data frame with three columns: topic, term, and beta, which is the probability that the word should go with that particular topic. We'll store in this in `td_beta`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "td_beta <- tidy(topic_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once stored, we can plot the probabilities for each topic. The code below takes the first 10 words within each topic and plots their beta within each topic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "td_beta %>%\n",
    "    group_by(topic) %>%\n",
    "    top_n(10, beta) %>%\n",
    "    ungroup() %>%\n",
    "    mutate(topic = paste0(\"Topic \", topic), term) %>%\n",
    "    ggplot(aes(term, beta, fill = as.factor(topic))) +\n",
    "    geom_col(show.legend = FALSE) +\n",
    "    facet_wrap(~ topic, scales = \"free\") +\n",
    "    coord_flip() +\n",
    "    labs(x = NULL, y = expression(beta),\n",
    "         title = \"Highest word probabilities for each topic\",\n",
    "         subtitle = \"Different words are associated with different topics\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a second to try to interpret each of these topics. What could we tenatively call each of these topics?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also retrieve the probability that each document belongs to each topic. With `tidy` and the keyword argument `matrix = \"gamma\"`, we can get that probability, also called \"gamma\". Using the keyword argument `document_names` allows us to associate it with each particular document ID."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "td_gamma <- tidy(topic_model, matrix = \"gamma\",\n",
    "                document_names = rownames(tweet_dfm))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, we can use `summarize` to get the topic which has the highest probability and associate with the document more definitively. Then, we will join the result to our original tweet data frame. Let's look at the each tweet. Did it seem to track the topics we outlined earlier?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweet_topics <- td_gamma %>% \n",
    "    group_by(document) %>%\n",
    "    summarize(max_topic = topic[which(gamma == max(gamma))]) %>%\n",
    "    ungroup() %>%\n",
    "    inner_join(tweets, by = c(\"document\" = \"id\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion\n",
    "\n",
    "This is a very, very brief introduction to Text as Data methods in R. We covered counting words, comparing documents, and touched on topic modeling. You should have a sense of how to begin counting words and developing quantitive insights from text.\n",
    "\n",
    "For further reading, there is an incredibly [large number of tools](https://cran.r-project.org/web/views/NaturalLanguageProcessing.html) in R for doing TAD. For another intervention from within political science, you should check out the [`quanteda` project](https://docs.quanteda.io/) by Kenneth Benoit and others. For more reading on `tidytext`, check out the [*Text Mining for R* book](https://www.tidytextmining.com/) by Julia Silge and David Robinson. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.3.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}