data_cleaning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Parsing and Cleaning PHEME Rumor Dataset Events\n",
    "\n",
    "Although data from the Twitter API is at the level of individual tweets, we're interested in analysis and prediction at the thread-level because the data is richer at this level, in terms of meta-data and graph structure.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "This notebook cleans tweet level data generated from `lib/pheme_parsing.py` and aggregates this tabular, individual data to tabular thread-level data. It also provides a useful sanity check after making modifications to `lib/pheme_parsing.py`. \n",
    "\n",
    "## Instructions\n",
    "1. Update the variable `event` in the cell below with one of the following events:\n",
    "    1. germanwings-crash\n",
    "    1. ferguson\n",
    "    1. ottawashooting\n",
    "    1. sydneysiege\n",
    "    1. charliehebdo\n",
    "1. Run all the cells in this notebook to generate thread-level CSV files in the `data/threads` directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [],
   "source": [
    "# Load dependencies for this Jupyter Notebook\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import networkx as nx\n",
    "from functools import reduce\n",
    "from lib.util import fetch_tweets\n",
    "\n",
    "event = \"germanwings-crash\"  # Change this value to clear different PHEME datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Parsing and Cleaning Data\n",
    "\n",
    "We've including a `lib/pheme_parsing.py` script for extracting data from directories of JSON file to tabular, tweet-level CSV files.\n",
    "\n",
    "The original PHEME dataset consists of JSON files organized into directories by event and category (rumor or non-rumor). These \"cached\" CSV files saved in `data/tweets` are tabularized versions of this data and include all the fields listed below. \n",
    "\n",
    "At the tweet level we've extracted 58 features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "data = fetch_tweets(event)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "##  Tweet Level Features\n",
    "\n",
    "| Name/Column       | Description                   | Type   | Notes  |\n",
    "|-------------------|-------------------------------|--------| ------ |\n",
    "| Adjective         | Number of Adjectives          | `int`  |   |\n",
    "| Adverb            | Number of Adverbs             | `int`  |   |\n",
    "| Noun              | Number of Nouns               | `int`  |   |\n",
    "| Pronoun           | Number of Pronouns            | `int`  |   |\n",
    "| Verb              | Number of Verbs               | `int`  |   |\n",
    "| capitalratio      | Ratio of capital leters       | `float`|   |\n",
    "| contentlength     | Length of content             | `int`  |   |\n",
    "| contentlength     | Length of content             | `int`  |   |\n",
    "| created           | Datetime Tweet was created    | `int`    | In Unix Epoch Time |\n",
    "| event             | the name of the event         | `string` | | \n",
    "| has_exclaim       | Text has an exclaimation mark | `int` | |\n",
    "| has_place         | Tweet has place location | `int` | |\n",
    "| has_quest         | Tweet has question mark | `int` | |\n",
    "| has_quest_or_exclaim | Tweet has question mark or has exclaimation point | `int` | |\n",
    "| has_url_in_text   | Does the tweet have a url in the text | `int` | Either 0 for False or 1 for True |\n",
    "| is_rumor          | Was this classified as rumor  | \"bool\" (`int`) | *Classification done by journalists* |\n",
    "| thread            | Source tweet id               | `str`  |                                                   |\n",
    "| in_reply_tweet    | Tweet ID in reply to          | `str`  |                                                   |\n",
    "| event             | Name of the PHEME event       | `str`  | Corresponds to event in the PHEME dataset         |\n",
    "| tweet_id          | Unique ID for tweet           | `str`  | This field is the ID referenced in `in_reply_tweet`     |\n",
    "| is_source_tweet   | Was this classified as rumor  | \"bool\" (`int`) |                                                   |\n",
    "| in_reply_user     | User ID in reply to           | `str`  |                                                   |\n",
    "| user_id           | Twitter User's ID             | `str`  | This field is the ID referenced in `in_reply_user` |\n",
    "| tweet_length      | Number of characters in tweet | `int`  |                                                   |\n",
    "| urls_count        | Number of URLS in tweet       | `int`  |                                                   |\n",
    "| hashtags_count    | Number of hashtags in tweet   | `int`  |                                                   |\n",
    "| retweet_count     | Times the tweet was retweeted | `int`  |                                                   |\n",
    "| favorite_count    | Number of times favorited     | `int`  |                                                   |\n",
    "| mentions_count    | Number of users mentioned     | `int`  |                                                   |\n",
    "| is_truncated      | Is this tweet truncated       | \"bool\" (`int`) | Did User type > 140 characters. [See Tweet updates](https://developer.twitter.com/en/docs/tweets/tweet-updates) |\n",
    "| has_smile_emoji      | Does Tweet contain \"\"?        | \"bool\" (`int`) | 😊 is the smile emoji |\n",
    "| user.tweets_count    | User's tweet total, currently | `int`  | |\n",
    "| user.verified        | Is Twitter user verified?     | \"bool\" (`int`) |                                                   |\n",
    "| user.followers_count | Total number of followers  | `int` | |\n",
    "| user.listed_count    | Number of lists for this user | `int` | | \n",
    "| user.friends_count   | Count of user's Friends | `int` | |\n",
    "| user.time_zone       | Timezone of the user's Twitter account | `str` | |\n",
    "| user.desc_length     | Length of the user's biographic description | `int` |\n",
    "| user.has_bg_img      | Does user have a profile background image?  | \"bool\" (`int`) |\n",
    "| user.default_pric    | Does the user have the default profile picture | \"bool\" (`int`) |\n",
    "| user.created_at      | Date and time Twitter account was activated | `datetime` | |\n",
    "| user.utc_dist        | TK | `int` | See [this blog post time and the Twitter API](https://zacharyst.com/2017/04/05/assigning-the-correct-time-to-a-twee) |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 4489 entries, 0 to 4488\n",
      "Data columns (total 59 columns):\n",
      "is_rumor                4489 non-null int64\n",
      "thread                  4489 non-null object\n",
      "in_reply_tweet          4489 non-null object\n",
      "event                   4489 non-null object\n",
      "tweet_id                4489 non-null object\n",
      "is_source_tweet         4489 non-null int64\n",
      "in_reply_user           4489 non-null object\n",
      "user_id                 4489 non-null object\n",
      "tweet_length            4489 non-null int64\n",
      "symbol_count            4489 non-null int64\n",
      "user_mentions           4489 non-null int64\n",
      "urls_count              4489 non-null int64\n",
      "media_count             4489 non-null int64\n",
      "hashtags_count          4489 non-null int64\n",
      "retweet_count           4489 non-null int64\n",
      "favorite_count          4489 non-null int64\n",
      "mentions_count          4489 non-null int64\n",
      "is_truncated            4489 non-null int64\n",
      "created                 4489 non-null float64\n",
      "has_smile_emoji         4489 non-null int64\n",
      "sensitive               4489 non-null int64\n",
      "has_place               4489 non-null int64\n",
      "has_coords              4489 non-null int64\n",
      "has_quest               4489 non-null int64\n",
      "has_exclaim             4489 non-null int64\n",
      "has_quest_or_exclaim    4489 non-null int64\n",
      "user.tweets_count       4489 non-null int64\n",
      "user.verified           4489 non-null int64\n",
      "user.followers_count    4489 non-null int64\n",
      "user.listed_count       4489 non-null int64\n",
      "user.desc_length        4489 non-null int64\n",
      "user.handle_length      4489 non-null int64\n",
      "user.name_length        4489 non-null int64\n",
      "user.notifications      4489 non-null int64\n",
      "user.friends_count      4489 non-null int64\n",
      "user.time_zone          3079 non-null object\n",
      "user.has_bg_img         4489 non-null int64\n",
      "user.default_pic        4489 non-null int64\n",
      "user.created_at         4489 non-null float64\n",
      "user.location           4489 non-null int64\n",
      "user.profile_sbcolor    4489 non-null int64\n",
      "user.profile_bgcolor    4489 non-null int64\n",
      "user.utc_dist           2969 non-null float64\n",
      "hasperiod               4489 non-null int64\n",
      "number_punct            4489 non-null int64\n",
      "negativewordcount       4489 non-null int64\n",
      "positivewordcount       4489 non-null int64\n",
      "capitalratio            4489 non-null float64\n",
      "contentlength           4489 non-null int64\n",
      "sentimentscore          4489 non-null float64\n",
      "Noun                    4489 non-null int64\n",
      "Verb                    4489 non-null int64\n",
      "Adjective               4489 non-null int64\n",
      "Pronoun                 4489 non-null int64\n",
      "FirstPersonPronoun      4489 non-null int64\n",
      "SecondPersonPronoun     4489 non-null int64\n",
      "ThirdPersonPronoun      4489 non-null int64\n",
      "Adverb                  4489 non-null int64\n",
      "has_url_in_text         4489 non-null int64\n",
      "dtypes: float64(5), int64(47), object(7)\n",
      "memory usage: 2.0+ MB\n"
     ]
    }
   ],
   "source": [
    "data.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Unlike previous projects, our contribution provides a list of every feature we used with code of how we extracted it. If someone in the future is ever interested in feature engineering with this dataset, we hope this notebook would be helpful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>is_rumor</th>\n",
       "      <th>thread</th>\n",
       "      <th>in_reply_tweet</th>\n",
       "      <th>event</th>\n",
       "      <th>tweet_id</th>\n",
       "      <th>is_source_tweet</th>\n",
       "      <th>in_reply_user</th>\n",
       "      <th>user_id</th>\n",
       "      <th>tweet_length</th>\n",
       "      <th>symbol_count</th>\n",
       "      <th>...</th>\n",
       "      <th>sentimentscore</th>\n",
       "      <th>Noun</th>\n",
       "      <th>Verb</th>\n",
       "      <th>Adjective</th>\n",
       "      <th>Pronoun</th>\n",
       "      <th>FirstPersonPronoun</th>\n",
       "      <th>SecondPersonPronoun</th>\n",
       "      <th>ThirdPersonPronoun</th>\n",
       "      <th>Adverb</th>\n",
       "      <th>has_url_in_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>580333314415919104</td>\n",
       "      <td>nan</td>\n",
       "      <td>germanwings-crash</td>\n",
       "      <td>580333314415919104</td>\n",
       "      <td>1</td>\n",
       "      <td>nan</td>\n",
       "      <td>7587032</td>\n",
       "      <td>135</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>580333314415919104</td>\n",
       "      <td>5.803333144159191e+17</td>\n",
       "      <td>germanwings-crash</td>\n",
       "      <td>580333739445764096</td>\n",
       "      <td>0</td>\n",
       "      <td>7587032.0</td>\n",
       "      <td>716785466</td>\n",
       "      <td>63</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.195</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>580333314415919104</td>\n",
       "      <td>5.803344525026181e+17</td>\n",
       "      <td>germanwings-crash</td>\n",
       "      <td>580372174659313665</td>\n",
       "      <td>0</td>\n",
       "      <td>2668764799.0</td>\n",
       "      <td>288730262</td>\n",
       "      <td>121</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000</td>\n",
       "      <td>6</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>580333314415919104</td>\n",
       "      <td>5.803721746593137e+17</td>\n",
       "      <td>germanwings-crash</td>\n",
       "      <td>580380418337345537</td>\n",
       "      <td>0</td>\n",
       "      <td>288730262.0</td>\n",
       "      <td>2668764799</td>\n",
       "      <td>56</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.200</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>580333314415919104</td>\n",
       "      <td>5.803333144159191e+17</td>\n",
       "      <td>germanwings-crash</td>\n",
       "      <td>580334092207722496</td>\n",
       "      <td>0</td>\n",
       "      <td>7587032.0</td>\n",
       "      <td>367907778</td>\n",
       "      <td>139</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.600</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 59 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   is_rumor              thread         in_reply_tweet              event  \\\n",
       "0         0  580333314415919104                    nan  germanwings-crash   \n",
       "1         0  580333314415919104  5.803333144159191e+17  germanwings-crash   \n",
       "2         0  580333314415919104  5.803344525026181e+17  germanwings-crash   \n",
       "3         0  580333314415919104  5.803721746593137e+17  germanwings-crash   \n",
       "4         0  580333314415919104  5.803333144159191e+17  germanwings-crash   \n",
       "\n",
       "             tweet_id  is_source_tweet in_reply_user     user_id  \\\n",
       "0  580333314415919104                1           nan     7587032   \n",
       "1  580333739445764096                0     7587032.0   716785466   \n",
       "2  580372174659313665                0  2668764799.0   288730262   \n",
       "3  580380418337345537                0   288730262.0  2668764799   \n",
       "4  580334092207722496                0     7587032.0   367907778   \n",
       "\n",
       "   tweet_length  symbol_count       ...         sentimentscore  Noun  Verb  \\\n",
       "0           135             0       ...                  0.000     8     0   \n",
       "1            63             0       ...                 -0.195     4     1   \n",
       "2           121             0       ...                  0.000     6     2   \n",
       "3            56             0       ...                  0.200     5     2   \n",
       "4           139             0       ...                 -0.600     8     1   \n",
       "\n",
       "   Adjective  Pronoun  FirstPersonPronoun  SecondPersonPronoun  \\\n",
       "0          2        0                   0                    0   \n",
       "1          1        0                   0                    0   \n",
       "2          4        0                   0                    0   \n",
       "3          0        0                   0                    0   \n",
       "4          2        0                   0                    0   \n",
       "\n",
       "   ThirdPersonPronoun  Adverb  has_url_in_text  \n",
       "0                   0       0                1  \n",
       "1                   0       0                0  \n",
       "2                   0       0                0  \n",
       "3                   0       0                0  \n",
       "4                   0       0                1  \n",
       "\n",
       "[5 rows x 59 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Boolean Features\n",
    "\n",
    "We choose to keep columns with values of either 1 or 0 as integers instead of casting them to Boolean types in python to facilitate reading and writing to CSV files.\n",
    "\n",
    "Depending upon the dataset, some boolean columns only had one unique value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "Just for a sanity check. The cell below converts these boolean columns into value of type `bool` and describes them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>is_rumor</th>\n",
       "      <th>is_source_tweet</th>\n",
       "      <th>is_truncated</th>\n",
       "      <th>has_smile_emoji</th>\n",
       "      <th>user.verified</th>\n",
       "      <th>user.has_bg_img</th>\n",
       "      <th>user.default_pic</th>\n",
       "      <th>sensitive</th>\n",
       "      <th>has_place</th>\n",
       "      <th>has_coords</th>\n",
       "      <th>user.notifications</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>4489</td>\n",
       "      <td>4489</td>\n",
       "      <td>4489</td>\n",
       "      <td>4489</td>\n",
       "      <td>4489</td>\n",
       "      <td>4489</td>\n",
       "      <td>4489</td>\n",
       "      <td>4489</td>\n",
       "      <td>4489</td>\n",
       "      <td>4489</td>\n",
       "      <td>4489</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>unique</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>top</th>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>freq</th>\n",
       "      <td>2494</td>\n",
       "      <td>4020</td>\n",
       "      <td>4489</td>\n",
       "      <td>4487</td>\n",
       "      <td>4109</td>\n",
       "      <td>3992</td>\n",
       "      <td>2827</td>\n",
       "      <td>4455</td>\n",
       "      <td>4228</td>\n",
       "      <td>4363</td>\n",
       "      <td>4489</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       is_rumor is_source_tweet is_truncated has_smile_emoji user.verified  \\\n",
       "count      4489            4489         4489            4489          4489   \n",
       "unique        2               2            1               2             2   \n",
       "top        True           False        False           False         False   \n",
       "freq       2494            4020         4489            4487          4109   \n",
       "\n",
       "       user.has_bg_img user.default_pic sensitive has_place has_coords  \\\n",
       "count             4489             4489      4489      4489       4489   \n",
       "unique               2                2         2         2          2   \n",
       "top               True            False     False     False      False   \n",
       "freq              3992             2827      4455      4228       4363   \n",
       "\n",
       "       user.notifications  \n",
       "count                4489  \n",
       "unique                  1  \n",
       "top                 False  \n",
       "freq                 4489  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bool_columns = [\"is_rumor\", \"is_source_tweet\", \"is_truncated\", \n",
    "                \"has_smile_emoji\", \"user.verified\", \"user.has_bg_img\", \n",
    "                \"user.default_pic\", \"sensitive\", \"has_place\", \"has_coords\", \"user.notifications\"]\n",
    "\n",
    "data[bool_columns].astype(bool).describe(include=\"bool\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Some columns in some PHEME events have only one unique value for all tweets. Instead of dropping them, we'll just be aware of them because they may vary across PHEME datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Warning, column `event` only has one unique value \"germanwings-crash\"\n",
      "Warning, column `symbol_count` only has one unique value \"0\"\n",
      "Warning, column `is_truncated` only has one unique value \"0\"\n",
      "Warning, column `user.notifications` only has one unique value \"0\"\n",
      "Warning, column `Adverb` only has one unique value \"0\"\n"
     ]
    }
   ],
   "source": [
    "for col in data.columns:\n",
    "    if len(data[col].unique()) == 1:\n",
    "        print(\"Warning, column `%s` only has one unique value \\\"%s\\\"\" % (col, data[col][0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Thead Level Data\n",
    "\n",
    "Every tweet belongs to a thread, indexed by the tweet id of the source tweet. We'll aggregate this tweet-level data into thread-level and use this dataset for prediction.\n",
    "\n",
    "Before aggregating, here are so summary statistics of our tweet-level data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>is_rumor</th>\n",
       "      <th>is_source_tweet</th>\n",
       "      <th>tweet_length</th>\n",
       "      <th>symbol_count</th>\n",
       "      <th>user_mentions</th>\n",
       "      <th>urls_count</th>\n",
       "      <th>media_count</th>\n",
       "      <th>hashtags_count</th>\n",
       "      <th>retweet_count</th>\n",
       "      <th>favorite_count</th>\n",
       "      <th>...</th>\n",
       "      <th>sentimentscore</th>\n",
       "      <th>Noun</th>\n",
       "      <th>Verb</th>\n",
       "      <th>Adjective</th>\n",
       "      <th>Pronoun</th>\n",
       "      <th>FirstPersonPronoun</th>\n",
       "      <th>SecondPersonPronoun</th>\n",
       "      <th>ThirdPersonPronoun</th>\n",
       "      <th>Adverb</th>\n",
       "      <th>has_url_in_text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>4489.000000</td>\n",
       "      <td>4489.000000</td>\n",
       "      <td>4489.000000</td>\n",
       "      <td>4489.0</td>\n",
       "      <td>4489.000000</td>\n",
       "      <td>4489.000000</td>\n",
       "      <td>4489.000000</td>\n",
       "      <td>4489.000000</td>\n",
       "      <td>4489.000000</td>\n",
       "      <td>4489.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>4489.000000</td>\n",
       "      <td>4489.000000</td>\n",
       "      <td>4489.000000</td>\n",
       "      <td>4489.000000</td>\n",
       "      <td>4489.000000</td>\n",
       "      <td>4489.000000</td>\n",
       "      <td>4489.000000</td>\n",
       "      <td>4489.000000</td>\n",
       "      <td>4489.0</td>\n",
       "      <td>4489.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.555580</td>\n",
       "      <td>0.104478</td>\n",
       "      <td>90.788372</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.506572</td>\n",
       "      <td>0.185787</td>\n",
       "      <td>0.127423</td>\n",
       "      <td>0.298062</td>\n",
       "      <td>25.290933</td>\n",
       "      <td>9.049677</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.015889</td>\n",
       "      <td>5.637113</td>\n",
       "      <td>1.897082</td>\n",
       "      <td>0.873914</td>\n",
       "      <td>0.417688</td>\n",
       "      <td>0.108710</td>\n",
       "      <td>0.087325</td>\n",
       "      <td>0.291156</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.239474</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.496957</td>\n",
       "      <td>0.305913</td>\n",
       "      <td>39.920378</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.020736</td>\n",
       "      <td>0.400827</td>\n",
       "      <td>0.334150</td>\n",
       "      <td>0.693938</td>\n",
       "      <td>163.969869</td>\n",
       "      <td>70.400399</td>\n",
       "      <td>...</td>\n",
       "      <td>0.289531</td>\n",
       "      <td>3.006968</td>\n",
       "      <td>1.630293</td>\n",
       "      <td>0.979131</td>\n",
       "      <td>0.774458</td>\n",
       "      <td>0.348459</td>\n",
       "      <td>0.340967</td>\n",
       "      <td>0.594175</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.426810</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>56.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>97.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>130.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>151.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>4388.000000</td>\n",
       "      <td>2541.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>22.000000</td>\n",
       "      <td>9.000000</td>\n",
       "      <td>7.000000</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 52 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          is_rumor  is_source_tweet  tweet_length  symbol_count  \\\n",
       "count  4489.000000      4489.000000   4489.000000        4489.0   \n",
       "mean      0.555580         0.104478     90.788372           0.0   \n",
       "std       0.496957         0.305913     39.920378           0.0   \n",
       "min       0.000000         0.000000      5.000000           0.0   \n",
       "25%       0.000000         0.000000     56.000000           0.0   \n",
       "50%       1.000000         0.000000     97.000000           0.0   \n",
       "75%       1.000000         0.000000    130.000000           0.0   \n",
       "max       1.000000         1.000000    151.000000           0.0   \n",
       "\n",
       "       user_mentions   urls_count  media_count  hashtags_count  retweet_count  \\\n",
       "count    4489.000000  4489.000000  4489.000000     4489.000000    4489.000000   \n",
       "mean        1.506572     0.185787     0.127423        0.298062      25.290933   \n",
       "std         1.020736     0.400827     0.334150        0.693938     163.969869   \n",
       "min         0.000000     0.000000     0.000000        0.000000       0.000000   \n",
       "25%         1.000000     0.000000     0.000000        0.000000       0.000000   \n",
       "50%         1.000000     0.000000     0.000000        0.000000       0.000000   \n",
       "75%         2.000000     0.000000     0.000000        0.000000       0.000000   \n",
       "max        12.000000     2.000000     2.000000        8.000000    4388.000000   \n",
       "\n",
       "       favorite_count       ...         sentimentscore         Noun  \\\n",
       "count     4489.000000       ...            4489.000000  4489.000000   \n",
       "mean         9.049677       ...              -0.015889     5.637113   \n",
       "std         70.400399       ...               0.289531     3.006968   \n",
       "min          0.000000       ...              -1.000000     0.000000   \n",
       "25%          0.000000       ...               0.000000     3.000000   \n",
       "50%          0.000000       ...               0.000000     5.000000   \n",
       "75%          1.000000       ...               0.000000     8.000000   \n",
       "max       2541.000000       ...               1.000000    22.000000   \n",
       "\n",
       "              Verb    Adjective      Pronoun  FirstPersonPronoun  \\\n",
       "count  4489.000000  4489.000000  4489.000000         4489.000000   \n",
       "mean      1.897082     0.873914     0.417688            0.108710   \n",
       "std       1.630293     0.979131     0.774458            0.348459   \n",
       "min       0.000000     0.000000     0.000000            0.000000   \n",
       "25%       1.000000     0.000000     0.000000            0.000000   \n",
       "50%       2.000000     1.000000     0.000000            0.000000   \n",
       "75%       3.000000     1.000000     1.000000            0.000000   \n",
       "max       9.000000     7.000000     8.000000            3.000000   \n",
       "\n",
       "       SecondPersonPronoun  ThirdPersonPronoun  Adverb  has_url_in_text  \n",
       "count          4489.000000         4489.000000  4489.0      4489.000000  \n",
       "mean              0.087325            0.291156     0.0         0.239474  \n",
       "std               0.340967            0.594175     0.0         0.426810  \n",
       "min               0.000000            0.000000     0.0         0.000000  \n",
       "25%               0.000000            0.000000     0.0         0.000000  \n",
       "50%               0.000000            0.000000     0.0         0.000000  \n",
       "75%               0.000000            0.000000     0.0         0.000000  \n",
       "max               4.000000            5.000000     0.0         1.000000  \n",
       "\n",
       "[8 rows x 52 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def agg_tweets_by_thread(df):\n",
    "    \n",
    "    shared = lambda x: 1 - len(set(x)) / len(x)\n",
    "    shared.__name__ = \"shared\"\n",
    "\n",
    "    funcs = [np.mean, sum, np.var]\n",
    "    agg_props = {\n",
    "        \"favorite_count\": funcs,\n",
    "        \"user_mentions\": funcs,\n",
    "        \"media_count\": funcs,\n",
    "        \"sensitive\": funcs,\n",
    "        \"has_place\": funcs,\n",
    "        \"has_coords\": funcs,\n",
    "        \"retweet_count\": funcs,\n",
    "        \"hashtags_count\": funcs + [shared],\n",
    "        \"urls_count\": funcs,\n",
    "        \"user.tweets_count\": funcs,\n",
    "        \"is_rumor\": max,\n",
    "        \"tweet_id\": len,\n",
    "        \"user.has_bg_img\": funcs,\n",
    "        \"has_quest\": funcs,\n",
    "        \"has_exclaim\": funcs,\n",
    "        \"has_quest_or_exclaim\": funcs,\n",
    "        \"user.default_pic\": funcs,\n",
    "        \"has_smile_emoji\": funcs,\n",
    "        \"user.verified\": funcs,\n",
    "        \"user.name_length\": funcs,\n",
    "        \"user.handle_length\": funcs,\n",
    "        \"user.profile_sbcolor\": funcs,\n",
    "        \"user.profile_bgcolor\": funcs,\n",
    "        \n",
    "        \"hasperiod\": funcs,\n",
    "        \"number_punct\": funcs,\n",
    "        \"negativewordcount\" : funcs,\n",
    "        \"positivewordcount\" : funcs,\n",
    "        \"capitalratio\" : funcs,\n",
    "        \"contentlength\" : funcs,\n",
    "        \"sentimentscore\" : funcs,\n",
    "        \"Noun\" : funcs,\n",
    "        \"Verb\" : funcs,\n",
    "        \"Adjective\" : funcs,\n",
    "        \"Pronoun\" : funcs,\n",
    "        \"Adverb\": funcs,\n",
    "    }\n",
    "    rename = {\n",
    "        \"tweet_id\": \"thread_length\"\n",
    "    }\n",
    "\n",
    "    def g(x):\n",
    "        # Add size of largest user-to-user conversation component in each thread        \n",
    "        d = []\n",
    "        thread_tweets = list(x[\"tweet_id\"])\n",
    "        G = nx.from_pandas_edgelist(df[df.tweet_id.isin(thread_tweets)], \"user_id\", \"in_reply_user\")\n",
    "        Gc = max(nx.connected_component_subgraphs(G), key=len)\n",
    "        d.append(nx.number_connected_components(G))\n",
    "        d.append(nx.diameter(Gc))\n",
    "        return pd.Series(d, index=[\"component_count\", \"largest_cc_diameter\"])\n",
    "    \n",
    "    # Step 0: Build graph-based features\n",
    "    graph = df.groupby(\"thread\").apply(g)\n",
    "    \n",
    "    # Step 1: Build simple aggregate features\n",
    "    agg = df.groupby(\"thread\")\\\n",
    "        .agg(agg_props)\\\n",
    "        .rename(columns=rename)\n",
    "    \n",
    "    agg.columns = [ \"_\".join(x) for x in agg.columns.ravel() ]\n",
    "    agg = agg.rename(columns={\"is_rumor_max\": \"is_rumor\", \"thread_length_len\": \"thread_length\"})\n",
    "    \n",
    "    # Step 2: Builds some features off the source tweet, which has tweet_id == thread            \n",
    "    src = df[df[\"is_source_tweet\"] == 1][[\"thread\",\n",
    "                                          \"user.followers_count\", \n",
    "                                          \"user.listed_count\",\n",
    "                                          \"user.verified\",\n",
    "                                          \"created\",\n",
    "                                          \"user.created_at\",\n",
    "                                          \"user.tweets_count\"]] \\\n",
    "                         .rename(columns={\"user.followers_count\": \"src.followers_count\",\n",
    "                                          \"user.listed_count\": \"src.listed_count\",\n",
    "                                          \"user.verified\": \"src.user_verified\",\n",
    "                                          \"user.created_at\": \"src.created_at\",\n",
    "                                          \"user.tweets_count\": \"src.tweets_total\"})\n",
    "    \n",
    "    # Step 3: Build features off of the reply tweets\n",
    "    def f(x):\n",
    "        d = []\n",
    "        \n",
    "        # Get various features from the distribution of times of reply tweet\n",
    "        d.append(min(x[\"created\"]))\n",
    "        d.append(max(x[\"created\"]))\n",
    "        d.append(np.var(x[\"created\"]))\n",
    "                \n",
    "        return pd.Series(d, index=[\"first_resp\", \"last_resp\",\"resp_var\"])\n",
    "        \n",
    "    replies = df[df[\"is_source_tweet\"] == False] \\\n",
    "        .groupby(\"thread\") \\\n",
    "        .apply(f)\n",
    "\n",
    "    graph_features = df.groupby(\"thread\").apply(g)\n",
    "    \n",
    "    dfs = [agg, src, replies, graph]\n",
    "    thrd_data = reduce(lambda left, right: pd.merge(left,right, on=\"thread\"), dfs)\n",
    "    \n",
    "    # Step 3: Add miscelaneous features\n",
    "    # Remember timestamps increase as time progresses\n",
    "    # src.created_at < created < first_resp < last_resp\n",
    "    thrd_data[\"time_to_first_resp\"] = thrd_data[\"first_resp\"] - thrd_data[\"created\"]\n",
    "    thrd_data[\"time_to_last_resp\"] = thrd_data[\"last_resp\"] - thrd_data[\"created\"]\n",
    "    \n",
    "    return thrd_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 405 entries, 0 to 404\n",
      "Columns: 116 entries, thread to time_to_last_resp\n",
      "dtypes: float64(75), int64(40), object(1)\n",
      "memory usage: 370.2+ KB\n"
     ]
    }
   ],
   "source": [
    "thrds = agg_tweets_by_thread(data)\n",
    "thrds.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Thread Level Feature Overview\n",
    "\n",
    "**Bold features** represent high performing features identified in C. Buntain and J. Golbeck, [\"Automatically Identifying Fake News in Popular Twitter Threads\"](http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8118443&isnumber=8118402) \n",
    "\n",
    "This table is just a description of the dimensions we aggregated. Our actual thread-level data includes many of these concepts aggregated by mean, sum, and variances.\n",
    "\n",
    "\n",
    "| Name                | Description                               | Type    | Notes |\n",
    "| ---                 | ---                                       | ---     | ----- |\n",
    "| thread              | Tweet ID of the source tweet              | `str`   | |\n",
    "| favorite_count      | Normalized favorite total                 | `float` | |\n",
    "| retweet_count       | Normlaized retweet total                  | `float` | |\n",
    "| **hashtags_count**  | Normlaized hashtag total                  | `float` | |\n",
    "| urls_count          | URL total normalized by thread length     | `float`  | |\n",
    "| user.tweets_count   | Total tweets by thread users              | `float` | |\n",
    "| event               | Name of PHEME event                       | `str`  | |\n",
    "| is_rumor            | Either rumor or nonrumor                  | `bool` | |\n",
    "| thread_length       | Number of tweets in the thread            | `int`  | |\n",
    "| user.has_bg_img     | Ratio of users who have bg image          | `float`| |\n",
    "| user.default_pic    | Ratio of users with default profile pic   | `float`| |\n",
    "| **has_smile_emoji** | Number of smile emojis in the thread      | `int`  | 😊 is the smile emoji |\n",
    "| user.verified       | Count of verified users in the thread normalized by thread length     | `float`  | |\n",
    "| **src.followers_count** | The number of followers of the original poster of the thread. | `int` | |\n",
    "| src.listed_count    | How many lists did source user belong to | `int` | |\n",
    "| src.user_verified   | Was the source user verified | `int` | |\n",
    "| src.tweets_total    | How many tweets had the source user issued to that point | `int` | |\n",
    "| reply_var           | The variance in the timestamps of responses to the source tweet | `float` |\n",
    "| src_age             | Difference in src user's creation and tweet creation            | `int`   | Measured in seconds |\n",
    "| time_to_first_resp  | The difference between tweet creation datetime and 1st reply    | `int`   | Measured in seconds |\n",
    "| time_to_last_resp   | The difference between tweet creation datetime and last reply   | `int`   | Measured in seconds |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>thread</th>\n",
       "      <th>favorite_count_mean</th>\n",
       "      <th>favorite_count_sum</th>\n",
       "      <th>favorite_count_var</th>\n",
       "      <th>user_mentions_mean</th>\n",
       "      <th>user_mentions_sum</th>\n",
       "      <th>user_mentions_var</th>\n",
       "      <th>media_count_mean</th>\n",
       "      <th>media_count_sum</th>\n",
       "      <th>media_count_var</th>\n",
       "      <th>...</th>\n",
       "      <th>created</th>\n",
       "      <th>src.created_at</th>\n",
       "      <th>src.tweets_total</th>\n",
       "      <th>first_resp</th>\n",
       "      <th>last_resp</th>\n",
       "      <th>resp_var</th>\n",
       "      <th>component_count</th>\n",
       "      <th>largest_cc_diameter</th>\n",
       "      <th>time_to_first_resp</th>\n",
       "      <th>time_to_last_resp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>580317556516483072</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>40</td>\n",
       "      <td>167.714286</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>16</td>\n",
       "      <td>8.285714</td>\n",
       "      <td>0.125</td>\n",
       "      <td>1</td>\n",
       "      <td>0.125</td>\n",
       "      <td>...</td>\n",
       "      <td>1.427193e+12</td>\n",
       "      <td>1.389095e+12</td>\n",
       "      <td>14408</td>\n",
       "      <td>1.427193e+12</td>\n",
       "      <td>1.427194e+12</td>\n",
       "      <td>2.810739e+10</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>36000.0</td>\n",
       "      <td>590000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>580317998147325952</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>2</td>\n",
       "      <td>1.333333</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>2</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>...</td>\n",
       "      <td>1.427194e+12</td>\n",
       "      <td>1.250783e+12</td>\n",
       "      <td>87411</td>\n",
       "      <td>1.427194e+12</td>\n",
       "      <td>1.427194e+12</td>\n",
       "      <td>1.600000e+09</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>101000.0</td>\n",
       "      <td>181000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>580318020192571392</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>1</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>1</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>...</td>\n",
       "      <td>1.427194e+12</td>\n",
       "      <td>1.178640e+12</td>\n",
       "      <td>22201</td>\n",
       "      <td>1.427198e+12</td>\n",
       "      <td>1.427198e+12</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>4476000.0</td>\n",
       "      <td>4476000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>580318210609696769</td>\n",
       "      <td>4.500000</td>\n",
       "      <td>9</td>\n",
       "      <td>40.500000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>...</td>\n",
       "      <td>1.427194e+12</td>\n",
       "      <td>1.238071e+12</td>\n",
       "      <td>13875</td>\n",
       "      <td>1.427194e+12</td>\n",
       "      <td>1.427194e+12</td>\n",
       "      <td>0.000000e+00</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>193000.0</td>\n",
       "      <td>193000.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>580318669483413504</td>\n",
       "      <td>1.800000</td>\n",
       "      <td>9</td>\n",
       "      <td>16.200000</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>4</td>\n",
       "      <td>0.200000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>...</td>\n",
       "      <td>1.427194e+12</td>\n",
       "      <td>1.254230e+12</td>\n",
       "      <td>5426</td>\n",
       "      <td>1.427198e+12</td>\n",
       "      <td>1.427218e+12</td>\n",
       "      <td>5.194105e+13</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>4258000.0</td>\n",
       "      <td>24336000.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 116 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               thread  favorite_count_mean  favorite_count_sum  \\\n",
       "0  580317556516483072             5.000000                  40   \n",
       "1  580317998147325952             0.666667                   2   \n",
       "2  580318020192571392             0.500000                   1   \n",
       "3  580318210609696769             4.500000                   9   \n",
       "4  580318669483413504             1.800000                   9   \n",
       "\n",
       "   favorite_count_var  user_mentions_mean  user_mentions_sum  \\\n",
       "0          167.714286            2.000000                 16   \n",
       "1            1.333333            0.666667                  2   \n",
       "2            0.500000            0.500000                  1   \n",
       "3           40.500000            1.000000                  2   \n",
       "4           16.200000            0.800000                  4   \n",
       "\n",
       "   user_mentions_var  media_count_mean  media_count_sum  media_count_var  \\\n",
       "0           8.285714             0.125                1            0.125   \n",
       "1           0.333333             0.000                0            0.000   \n",
       "2           0.500000             0.000                0            0.000   \n",
       "3           2.000000             0.000                0            0.000   \n",
       "4           0.200000             0.000                0            0.000   \n",
       "\n",
       "         ...               created  src.created_at  src.tweets_total  \\\n",
       "0        ...          1.427193e+12    1.389095e+12             14408   \n",
       "1        ...          1.427194e+12    1.250783e+12             87411   \n",
       "2        ...          1.427194e+12    1.178640e+12             22201   \n",
       "3        ...          1.427194e+12    1.238071e+12             13875   \n",
       "4        ...          1.427194e+12    1.254230e+12              5426   \n",
       "\n",
       "     first_resp     last_resp      resp_var  component_count  \\\n",
       "0  1.427193e+12  1.427194e+12  2.810739e+10                3   \n",
       "1  1.427194e+12  1.427194e+12  1.600000e+09                2   \n",
       "2  1.427198e+12  1.427198e+12  0.000000e+00                2   \n",
       "3  1.427194e+12  1.427194e+12  0.000000e+00                2   \n",
       "4  1.427198e+12  1.427218e+12  5.194105e+13                2   \n",
       "\n",
       "   largest_cc_diameter  time_to_first_resp  time_to_last_resp  \n",
       "0                    2             36000.0           590000.0  \n",
       "1                    2            101000.0           181000.0  \n",
       "2                    1           4476000.0          4476000.0  \n",
       "3                    1            193000.0           193000.0  \n",
       "4                    2           4258000.0         24336000.0  \n",
       "\n",
       "[5 rows x 116 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "thrds.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(405, 116)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "thrds.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Wrote data to data/threads/germanwings-crash.csv'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fn = \"data/threads/%s.csv\" % event\n",
    "thrds.to_csv(fn, index=False)\n",
    "\"Wrote data to %s\" % fn"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}