3 - Data Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Analysis (Implementing RAKE Algorythm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this final section of the workshop, we will apply an algorythm called, Rapid Automated Keyword Extraction.    __[RAKE](https://www.researchgate.net/publication/227988510_Automatic_Keyword_Extraction_from_Individual_Documents)__ is an unsupersived, language neutral, and domain independent algorythm developed by Stuart Stuart,  Dave Engel, Nick Cramer, and Wendy Cowley (2010).  Since our dateset consists of untagged news articles, RAKE can provide us the most important keywords emerging from our dataset representing the articles in a condensed form. \n",
    "\n",
    "First, we need to __import__ pandas to access dataset as a dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "tw_news = pd.read_csv('tw_dataset.csv', date_parser='date')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now need the create a function to concanate the articles in a string object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def sum_text(dataset):\n",
    "    '''\n",
    "    takes pd dataset returns connotated text\n",
    "    '''\n",
    "    sumoftxt = ''\n",
    "    for i in dataset['fulltext']:\n",
    "        sumoftxt += i\n",
    "    return sumoftxt\n",
    "\n",
    "raw_text = sum_text(tw_news)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "In order to under take the analysis, we need to __import__ the RAKE module. Before that you can install the module by typing to an empty cell:\n",
    "\n",
    "___```!pip install python-rake```___."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import RAKE\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can apply RAKE to our data. Before that, for a further reference, please consider this information regarding the algorythm. \n",
    "- You can find the open source code of the module __[here!](https://github.com/fabianvf/python-rake/blob/master/RAKE/RAKE.py)__\n",
    "\n",
    "- RAKE calculates the word scores, first creating a word occurence graph of the content, and:\n",
    "\n",
    "$$\n",
    "\\begin{equation}\n",
    "\\frac{degree(word)}{frequency(word)}\n",
    "\\end{equation}\n",
    "$$\n",
    "\n",
    "__The score of the keyword is the sum of all of its member words.__ For more detailed information please refer to the paper.\n",
    "\n",
    "Please be aware that both the number of selected keywords, object parameters and attributions stems from subject/domain knowledge. Exploring our data in the previous sections, I found out after a several trials that passing 3 for each _minimum number of characters, maximum number of words in the keyword, and minimum number of word appearance._ I also choose to use in-built __FoxStopList__. I also choose the retrive top __20__ candidates.\n",
    "\n",
    "Now, we can find the key words and save the key words in a __pandas dataframe__.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "kwobjct = RAKE.Rake(stop_words = RAKE.FoxStopList())\n",
    "\n",
    "candidateKWords = kwobjct.run(raw_text, 3, 3, 3)\n",
    "\n",
    "keywordsData = pd.DataFrame(candidateKWords[:20], columns=['Keyword','Score'], index=range(1,len(candidateKWords[:21])))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Let's see the results by calling the top five items in the new data frame!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Keyword</th>\n",
       "      <th>Score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>president tsai ing-wen</td>\n",
       "      <td>13.060656</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>hong kong-listed company</td>\n",
       "      <td>9.800372</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>china-friendly kuomintang party</td>\n",
       "      <td>9.635668</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>han kuo-yu</td>\n",
       "      <td>8.050000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>golden horse awards</td>\n",
       "      <td>8.008333</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           Keyword      Score\n",
       "1           president tsai ing-wen  13.060656\n",
       "2         hong kong-listed company   9.800372\n",
       "3  china-friendly kuomintang party   9.635668\n",
       "4                       han kuo-yu   8.050000\n",
       "5              golden horse awards   8.008333"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "keywordsData.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems that we retrived some useful keywords for our discussion. Let's save them in a csv file and finish our small data analysis project. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "keywordsData.to_csv('twnews_keywords.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stepping stone for Advanced Knowledge Extraction\n",
    "\n",
    "So far, our case study used simple but powerful methods of Natural Language Processing. Before finalizing this study, you should be reminded that Python is capable of more advance methods. We will not be able to cover them in this workshop. For instance, Python's __[spaCy](https://spacy.io)__ module provides an extensive tools, pre-trained model and interoperability with Python's other deep learning libraries, such as __[scikit-learn](https://scikit-learn.org)__. Just to note the comprehensibility of _spaCy_'s models, one model called __'en_core_web_lg'__ can size around 800 MB.\n",
    "\n",
    "Just to demonstrate the power of _spaCy_, please see the graph below that I created using a _spaCy_ based module that can be found __[here](https://github.com/BrambleXu/news-graph)__. After downloading the module, I customized it and run the __raw_text__ variable holding the full text of the articles. This graph provided me the information about and the relationship between the key actors, locations and organizations. I wanted to particularly see the dynamic relationship between the United States and other actors in our dataset. This simple code below provided the overview that I needed.\n",
    "\n",
    "```Python\n",
    "from news_graph import NewsMining\n",
    "Miner = NewsMining()\n",
    "Miner.main(raw_text)\n",
    "```\n",
    "\n",
    "<img src='images/Tw_elec_nlp.gif' style=\"width: 850px\" align=\"middle\" />\n",
    "\n",
    "__As demonstrated above, Python can provide extremely useful methods for the social scientists. I hope that this workshop helped you to have a brief impression of Python and its possible usage for your research.__ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- __[Previous: Data Exploration](2 - Data Exploration.ipynb)__"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}