MCC Project 1 Notebook-Twitter.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Case Study 1 : Collecting Data from Twitter\n",
    "\n",
    "* ------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TEAM Members:**\n",
    "\n",
    "    Matt Curcio\n",
    "    \n",
    "    Tingting Meng\n",
    "    \n",
    "    Wenlu Du\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Required Readings:** \n",
    "* Chapter 1 and Chapter 9 of the book [Mining the Social Web](http://bit.ly/1pC5ujw) \n",
    "* The codes for [Chapter 1](http://bit.ly/1qCtMrr) and [Chapter 9](http://bit.ly/1u7eP33)\n",
    "\n",
    "\n",
    "** NOTE **\n",
    "* Please don't forget to save the notebook frequently when working in IPython Notebook, otherwise the changes you made can be lost.\n",
    "\n",
    "*----------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problem 1: Sampling Twitter Data with Streaming API about a certain topic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Select a topic that you are interested in, for example, \"WPI\" or \"Lady Gaga\"\n",
    "* Use Twitter Streaming API to sample a collection of tweets about this topic in real time. (It would be recommended that the number of tweets should be larger than 200, but smaller than 1 million.\n",
    "* Store the tweets you downloaded into a local file (txt file or json file) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "import twitter\n",
    "#---------------------------------------------\n",
    "# Define a Function to Login Twitter API\n",
    "def oauth_login():\n",
    "    # Go to http://twitter.com/apps/new to create an app and get values\n",
    "    # for these credentials that you'll need to provide in place of these\n",
    "    # empty string values that are defined as placeholders.\n",
    "    # See https://dev.twitter.com/docs/auth/oauth for more information \n",
    "    # on Twitter's OAuth implementation.\n",
    "    \n",
    "    CONSUMER_KEY = ck\n",
    "    CONSUMER_SECRET = cs\n",
    "    OAUTH_TOKEN = ot\n",
    "    OAUTH_TOKEN_SECRET = ots\n",
    "    \n",
    "    auth = twitter.oauth.OAuth(OAUTH_TOKEN, OAUTH_TOKEN_SECRET,\n",
    "                               CONSUMER_KEY, CONSUMER_SECRET)\n",
    "    \n",
    "    twitter_api = twitter.Twitter(auth=auth)\n",
    "    return twitter_api\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Searching for tweets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import twitter\n",
    "\n",
    "def twitter_search(twitter_api, q, max_results=100, **kw):\n",
    "\n",
    "    # See https://dev.twitter.com/docs/api/1.1/get/search/tweets and \n",
    "    # https://dev.twitter.com/docs/using-search for details on advanced \n",
    "    # search criteria that may be useful for keyword arguments\n",
    "    \n",
    "    # See https://dev.twitter.com/docs/api/1.1/get/search/tweets    \n",
    "    search_results = twitter_api.search.tweets(q=q, count=100, **kw)\n",
    "    \n",
    "    statuses = search_results['statuses']\n",
    "    \n",
    "    # Iterate through batches of results by following the cursor until we\n",
    "    # reach the desired number of results, keeping in mind that OAuth users\n",
    "    # can \"only\" make 180 search queries per 15-minute interval. See\n",
    "    # https://dev.twitter.com/docs/rate-limiting/1.1/limits\n",
    "    # for details. A reasonable number of results is ~1000, although\n",
    "    # that number of results may not exist for all queries.\n",
    "    \n",
    "    # Enforce a reasonable limit\n",
    "    max_results = min(1000, max_results)\n",
    "    \n",
    "    for _ in range(10): # 10*100 = 1000\n",
    "        try:\n",
    "            next_results = search_results['search_metadata']['next_results']\n",
    "        except KeyError, e: # No more results when next_results doesn't exist\n",
    "            break\n",
    "            \n",
    "        # Create a dictionary from next_results, which has the following form:\n",
    "        # ?max_id=313519052523986943&q=NCAA&include_entities=1\n",
    "        kwargs = dict([ kv.split('=') \n",
    "                        for kv in next_results[1:].split(\"&\") ])\n",
    "        \n",
    "        search_results = twitter_api.search.tweets(**kwargs)\n",
    "        statuses += search_results['statuses']\n",
    "        \n",
    "        if len(statuses) > max_results: \n",
    "            break\n",
    "            \n",
    "    return statuses\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Store the tweets into  a local file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import twitter\n",
    "\n",
    "def save_file(search_results):\n",
    "    f=open('workfile','w')\n",
    "    json.dump(search_results,f)\n",
    "    f.close\n",
    "    \n",
    "#usage\n",
    "twitter_api = oauth_login()\n",
    "\n",
    "q = \"iPhone 6\"\n",
    "results = twitter_search(twitter_api, q, max_results=10)\n",
    "save_file(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Report some statistics about the tweets you collected \n",
    "\n",
    "*The topic of interest: iPhone 6\n",
    "\n",
    "*The total number of tweets collected:  100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*-----------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Problem 2: Analyzing Tweets and Tweet Entities with Frequency Analysis\n",
    "\n",
    "**1. Word Count:** \n",
    "* Use the tweets you collected in Problem 1, and compute the frequencies of the words being used in these tweets. \n",
    "* Plot a table of the top 30 words with their counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------------+-------+\n",
      "| Word                   | Count |\n",
      "+------------------------+-------+\n",
      "| RT                     |    66 |\n",
      "| 6                      |    63 |\n",
      "| iPhone                 |    63 |\n",
      "| u                      |    26 |\n",
      "| thank                  |    25 |\n",
      "| 6!!                    |    25 |\n",
      "| mom                    |    24 |\n",
      "| My                     |    21 |\n",
      "| the                    |    21 |\n",
      "| 😍😍                   |    20 |\n",
      "| IPhone                 |    20 |\n",
      "| http://t.co/N3eTuyN7k2 |    18 |\n",
      "| @traplxrde:            |    18 |\n",
      "| @RelatableQuote:       |    10 |\n",
      "| my                     |     8 |\n",
      "| I                      |     8 |\n",
      "| http://t.co/MeJQ20XIkg |     8 |\n",
      "| http://t.co/qPBqfMLuuP |     8 |\n",
      "| @iPhoneTeam:           |     8 |\n",
      "| iphone                 |     7 |\n",
      "| is                     |     7 |\n",
      "| a                      |     6 |\n",
      "| Plus                   |     5 |\n",
      "| on                     |     5 |\n",
      "| The                    |     5 |\n",
      "| 6?                     |     5 |\n",
      "| an                     |     5 |\n",
      "| you                    |     5 |\n",
      "| to                     |     4 |\n",
      "| Urban                  |     4 |\n",
      "+------------------------+-------+\n"
     ]
    }
   ],
   "source": [
    "from prettytable import PrettyTable\n",
    "from collections import Counter\n",
    "import twitter\n",
    "import json\n",
    "\n",
    "#Frequency analysis\n",
    "def count_freq():\n",
    "    statuses = json.loads(open('workfile').read())\n",
    "   \n",
    "   # statuses = search_results['statuses']\n",
    "\n",
    "    status_texts = [ status['text'] \n",
    "                     for status in statuses ]\n",
    "    \n",
    "    words = [ w \n",
    "              for t in status_texts \n",
    "                  for w in t.split() ]\n",
    "    \n",
    "    pt = PrettyTable(field_names=['Word','Count'])\n",
    "    c = Counter(words)\n",
    "    \n",
    "    [ pt.add_row(kv) for kv in c.most_common()[:30] ]\n",
    "    pt.align['Word'],pt.align['Count']='l','r'\n",
    "    print pt\n",
    "\n",
    "    \n",
    "#usage\n",
    "count_freq()\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2. Find the most popular tweets in your collection of tweets**\n",
    "\n",
    "Please plot a table of the top 10 tweets that are the most popular among your collection, i.e., the tweets with the largest number of retweet counts.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+----------------+----------------------------------------------------+\n",
      "| Count | Screen Name    | Text                                               |\n",
      "+-------+----------------+----------------------------------------------------+\n",
      "| 36785 | justinbieber   | RT @justinbieber: Shots looks amazing on the new   |\n",
      "|       |                | iPhone 6 Plus http://t.co/wzHgUWU6ih               |\n",
      "| 9830  | GIF_A_GOGO     | RT @GIF_A_GOGO: Comment Apple se console de vos    |\n",
      "|       |                | critiques sur l'Iphone 6 ...                       |\n",
      "|       |                | http://t.co/6fqMWjxrxY                             |\n",
      "| 6993  | yoshiiiiii_    | RT @yoshiiiiii_: rt pour la tablette milka         |\n",
      "|       |                | fav pour l'iphone 6 http://t.co/Y2vQZpgPJJ         |\n",
      "| 5782  | RelatableQuote | RT @RelatableQuote: The first guy to buy an iPhone |\n",
      "|       |                | 6 😂 https://t.co/5fVpi4eQCF                       |\n",
      "| 5632  | Dory           | RT @Dory: iPhone 6!! thank u mom                   |\n",
      "|       |                | http://t.co/XQiwFv4XjR                             |\n",
      "| 5632  | Dory           | RT @Dory: iPhone 6!! thank u mom                   |\n",
      "|       |                | http://t.co/XQiwFv4XjR                             |\n",
      "| 5544  | TheTumblrPosts | RT @TheTumblrPosts: iPhone 6!! thank u dad         |\n",
      "|       |                | http://t.co/ffJDD4iWoN                             |\n",
      "| 5074  | ohnahcarter    | RT @ohnahcarter: my iphone 6 OMFG 😻               |\n",
      "|       |                | http://t.co/hJHmW5JTDY                             |\n",
      "| 5074  | ohnahcarter    | RT @ohnahcarter: my iphone 6 OMFG 😻               |\n",
      "|       |                | http://t.co/hJHmW5JTDY                             |\n",
      "| 5074  | ohnahcarter    | RT @ohnahcarter: my iphone 6 OMFG 😻               |\n",
      "|       |                | http://t.co/hJHmW5JTDY                             |\n",
      "+-------+----------------+----------------------------------------------------+\n"
     ]
    }
   ],
   "source": [
    "from prettytable import PrettyTable\n",
    "from collections import Counter\n",
    "import twitter\n",
    "import json\n",
    "\n",
    "#Find the most popular tweets\n",
    "def retweets():\n",
    "    statuses = json.loads(open('workfile').read())\n",
    "    \n",
    "    retweets = [\n",
    "            # Store out a tuple of these three values ...\n",
    "            (status['retweet_count'], \n",
    "             status['retweeted_status']['user']['screen_name'],\n",
    "             status['text']) \n",
    "            \n",
    "            # ... for each status ...\n",
    "            for status in statuses \n",
    "            \n",
    "            # ... so long as the status meets this condition.\n",
    "                if status.has_key('retweeted_status')\n",
    "           ]\n",
    "\n",
    "    # Slice off the first 10 from the sorted results and display each item in the tuple\n",
    "\n",
    "    pt = PrettyTable(field_names=['Count', 'Screen Name', 'Text'])\n",
    "    [ pt.add_row(row) for row in sorted(retweets, reverse=True)[:10] ]\n",
    "    pt.max_width['Text'] = 50\n",
    "    pt.align= 'l'\n",
    "    print pt\n",
    "\n",
    "#usage\n",
    "retweets()\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**3. Find the most popular Tweet Entities in your collection of tweets**\n",
    "\n",
    "Please plot a table of the top 10 hashtags, top 10 user mentions that are the most popular in your collection of tweets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------+-------+\n",
      "| Hashtag | Count |\n",
      "+---------+-------+\n",
      "| iPhone6 |    12 |\n",
      "| iOS8    |    12 |\n",
      "| iPhone  |     4 |\n",
      "| The     |     2 |\n",
      "| gadgets |     2 |\n",
      "| tech    |     2 |\n",
      "| iPad    |     1 |\n",
      "| iTunes  |     1 |\n",
      "| Also    |     1 |\n",
      "| Got     |     1 |\n",
      "+---------+-------+\n",
      "+-----------------+-------+\n",
      "| User_mention    | Count |\n",
      "+-----------------+-------+\n",
      "| traplxrde       |    14 |\n",
      "| FITNESSSLEGEND  |    12 |\n",
      "| ohnahcarter     |     7 |\n",
      "| iPhoneTeam      |     7 |\n",
      "| santosfcinforma |     6 |\n",
      "| RelatableQuote  |     6 |\n",
      "| pedhsm          |     5 |\n",
      "| TheRealMikeEpps |     5 |\n",
      "| BrianSinSuerte  |     5 |\n",
      "| GobbiGustavo    |     4 |\n",
      "+-----------------+-------+\n"
     ]
    }
   ],
   "source": [
    "from prettytable import PrettyTable\n",
    "from collections import Counter\n",
    "import twitter\n",
    "import json\n",
    "\n",
    "#The most popular tweet entites \n",
    "def top_entities():\n",
    "    statuses = json.loads(open('workfile').read())\n",
    "   \n",
    "    #hashtages\n",
    "    hashtags = [ hashtag['text'] \n",
    "                 for status in statuses\n",
    "                     for hashtag in status['entities']['hashtags'] ]\n",
    "    \n",
    "    #usar mentions\n",
    "    user_mentions = [ user_mention['screen_name']\n",
    "                      for status in statuses\n",
    "                          for user_mention in status['entities']['user_mentions'] ]\n",
    "    \n",
    "    #plot the results in table \n",
    "    for label, data in ( ('Hashtag', hashtags ),\n",
    "                          ('User_mention', user_mentions)): \n",
    "        pt = PrettyTable(field_names=[label,'Count'])\n",
    "        c = Counter(data)          \n",
    "        [ pt.add_row(kv) for kv in c.most_common()[:10] ]\n",
    "        pt.align[label], pt.align['Count'] = 'l', 'r' # Set column alignment\n",
    "        print pt\n",
    "\n",
    "    \n",
    "#usage\n",
    "top_entities()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*------------------------\n",
    "\n",
    "## Problem 3: Getting \"All\" friends and \"All\" followers of a popular user in twitter\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* choose a popular twitter user who has many followers, such as \"ladygaga\".\n",
    "* Get the list of all friends and all followers of the twitter user.\n",
    "* Plot 20 out of the followers, plot their ID numbers and screen names in a table.\n",
    "* Plot 20 out of the friends (if the user has more than 20 friends), plot their ID numbers and screen names in a table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Followers\n",
      "\n",
      "+------------+-----------------+\n",
      "| ID         |     Screen name |\n",
      "+------------+-----------------+\n",
      "| 17917640   |           20Ent |\n",
      "| 1928730974 |   Lupine_Lights |\n",
      "| 9587902    |     carborocket |\n",
      "| 2671440313 |        RotorUsa |\n",
      "| 260277853  |       neilpryde |\n",
      "| 983738660  |  InsiderPeloton |\n",
      "| 2776570843 |   ChrisKemp_sac |\n",
      "| 2820506090 |   HernanEliezer |\n",
      "| 37143075   |    TheFixStudio |\n",
      "| 14869462   |       r_ashwell |\n",
      "| 80411378   |    nikola_innov |\n",
      "| 82917971   |          ahhdyu |\n",
      "| 353799500  |     DonkeyLabel |\n",
      "| 39188622   | NorCalBikeSport |\n",
      "| 91023883   |     FormigliUSA |\n",
      "| 20367168   |            RQCG |\n",
      "| 193291202  |       Jonny_05_ |\n",
      "| 39942436   |          ekitah |\n",
      "| 81378085   | oldfashionedtoy |\n",
      "| 2787793003 |         KaiSaiz |\n",
      "+------------+-----------------+\n",
      "\n",
      "Friends\n",
      "\n",
      "+------------+-----------------+\n",
      "| ID         |     Screen name |\n",
      "+------------+-----------------+\n",
      "| 18227861   |      piraterace |\n",
      "| 983738660  |  InsiderPeloton |\n",
      "| 2345404443 |          Gritcx |\n",
      "| 1059552948 |      dirtwiretv |\n",
      "| 95023423   |       UberFacts |\n",
      "| 2795373043 |  FatChanceBikes |\n",
      "| 101871654  |    daniellloyd1 |\n",
      "| 1653442136 |   _youhadonejob |\n",
      "| 443215941  |       Sethrogen |\n",
      "| 48416511   |  Schwinnchester |\n",
      "| 1332778028 |     DasPretzels |\n",
      "| 2530297842 |    HiddenCashCO |\n",
      "| 598443842  |      NWSBoulder |\n",
      "| 174198926  |       Dave_Rome |\n",
      "| 72003616   |       Jackos016 |\n",
      "| 119724014  |        SadhbhOS |\n",
      "| 847940155  |     RobWeaver14 |\n",
      "| 107492667  |   WeightWeenies |\n",
      "| 414228963  |        ronkoch1 |\n",
      "| 765593660  | CiclismoEspreso |\n",
      "+------------+-----------------+\n"
     ]
    }
   ],
   "source": [
    "#Problem 3.1\n",
    "from prettytable import PrettyTable\n",
    "from collections import Counter\n",
    "import twitter\n",
    "import json\n",
    "\n",
    "#Find all the followers and friends of a user and plot 20 of them into table.\n",
    "def user_info():\n",
    "    followers = oauth_login().followers.list(screen_name='angryasian',count = 20 )\n",
    "    friends = oauth_login().friends.list(screen_name='angryasian',count = 20 )\n",
    "    \n",
    "    print 'Followers'\n",
    "    print\n",
    "    pt_follower = PrettyTable(field_names = ['ID', 'Screen name'])\n",
    "    \n",
    "    for follower in followers['users']:\n",
    "        pt_follower.add_row((follower['id'],follower['screen_name']))\n",
    "        pt_follower.align['ID'], pt_follower.align['Screen name'] = 'l', 'r' # Set column alignment\n",
    "    print pt_follower\n",
    "    \n",
    "    print\n",
    "    print 'Friends'\n",
    "    print\n",
    "    pt_friends = PrettyTable(field_names = ['ID', 'Screen name'])\n",
    "    \n",
    "    for friend in friends['users']:\n",
    "        pt_friends.add_row((friend['id'],friend['screen_name']))\n",
    "        pt_friends.align['ID'], pt_friends.align['Screen name'] = 'l', 'r' # Set column alignment\n",
    "    print pt_friends\n",
    "   \n",
    "    \n",
    "#usage\n",
    "user_info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Compute the mutual friends within the two groups, i.e., the users who are in both friend list and follower list, plot their ID numbers and screen names in a table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----------+----------------+\n",
      "| ID        |    Screen name |\n",
      "+-----------+----------------+\n",
      "| 983738660 | InsiderPeloton |\n",
      "+-----------+----------------+\n"
     ]
    }
   ],
   "source": [
    "# Problem 3.2\n",
    "from prettytable import PrettyTable\n",
    "from collections import Counter\n",
    "import twitter\n",
    "import json\n",
    "\n",
    "\n",
    "#find mutual friend of an user.\n",
    "def mutual_friend():\n",
    "    followers = oauth_login().followers.list(screen_name='angryasian', count= 100 )\n",
    "    friends = oauth_login().friends.list(screen_name='angryasian', count = 100)\n",
    "    \n",
    "    #Followers' set\n",
    "    followers_set = set([ follower['id']\n",
    "                         for follower in followers['users']])\n",
    "\n",
    "    #Friends' set\n",
    "    friends_set = set([ friend['id']\n",
    "                         for friend in friends['users']])\n",
    "    \n",
    "    #find the results of the mutual friends\n",
    "    mutual_friends = followers_set.intersection(friends_set)\n",
    "    \n",
    "    #Create table to show the results\n",
    "    pt_friends = PrettyTable(field_names = ['ID', 'Screen name'])\n",
    "    \n",
    "    for friend in friends['users']:\n",
    "        if friend['id'] in mutual_friends:\n",
    "            pt_friends.add_row((friend['id'],friend['screen_name']))\n",
    "            pt_friends.align['ID'], pt_friends.align['Screen name'] = 'l', 'r' # Set column alignment\n",
    "    \n",
    "    print pt_friends\n",
    "\n",
    "\n",
    "#usage\n",
    "mutual_friend()\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* -----------------\n",
    "# Done\n",
    "\n",
    "All set! \n",
    "\n",
    "** What do you need to submit?**\n",
    "\n",
    "* **Notebook File**: Save this IPython notebook, and find the notebook file in your folder (for example, \"filename.ipynb\"). This is the file you need to submit. Please make sure all the plotted tables and figures are in the notebook. If you used \"ipython notebook --pylab=inline\" to open the notebook, all the figures and tables should have shown up in the notebook.\n",
    "\n",
    "\n",
    "* **PPT Slides**: please prepare PPT slides (for 10 minutes' talk) to present about the case study . We will ask two teams which are randomly selected to present their case studies in class for this case study. \n",
    "\n",
    "* ** Report**: please prepare a report (less than 10 pages) to report what you found in the data.\n",
    "    * What data you collected? \n",
    "    * Why this topic is interesting or important to you? (Motivations)\n",
    "    * How did you analyse the data?\n",
    "    * What did you find in the data? \n",
    " \n",
    "     (please include figures or tables in the report, but no source code)\n",
    "\n",
    "Please compress all the files in a zipped file.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}