1 - Finding the Data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finding the Data\n",
    "\n",
    "We need to install [newsapi-python](https://github.com/mattlisiv/newsapi-python) package. We can do this by entering ! in the beginning of a cell to directly access to the system terminal. Using exclamation mark is an easy way to access system terminal and install required packages as well undertake other work such as finding paths for working directory or other files.\n",
    "```\n",
    "$ pip install newsapi-python\n",
    "```\n",
    "After installing the package, we can start to send queries to retrive data. First we need to import NewsApiClient from  _newsapi_ module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from newsapi import NewsApiClient"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "We need to use the key from the News Api application that we earlier created. In order to not to show your secret and key, it is a good practice to save them as in a different python file. Then  we can import that here and attacht to variables to prevent exposure. I save mine in a file called __'nws_token.py'__. Using the code below, I __import__ key and secret string objects __from__ nws_token module that I created.\n",
    "\n",
    "In Python there are various ways to __import__ a module, here are some examples.\n",
    "\n",
    "```Python\n",
    "import module #method 1\n",
    "from module import something #method 2\n",
    "from module import * #method 3 imports all\n",
    "```\n",
    "If you use the first method, later you need to use the syntax below by first calling the module name then the function/variable name later on:\n",
    "```Python\n",
    "x = module.function_name() #if you use the first method\n",
    "```\n",
    "Otherwise, you can just call the method/variable from that module by its name. Here, we use the second method to import a variable from a module since there will not be any other variables with the same name that might cause bugs.\n",
    "\n",
    "After importing the key, we will create an instance of the NewsApiClient object by passing our individual key as a parameter.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from nws_token import key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "api = NewsApiClient(api_key=key)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we created an instance of _NewsApiClient_ object, we are now ready find the data we are looking for. It is always a good practice to refer to the official documentation to find out what parametres we can pass, and what kind of data we can retrive. You can reach the official documentation of News API [here!](https://newsapi.org/docs) After reading through the documentation, we have a better understanding of the parameters we want to use. \n",
    "\n",
    "Now, let's try to retrive all 100 most recent news articles mentioning 2020 Taiwan Presidential Elections and save all into a __dictionary__ object called _articles_. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "articles = {}\n",
    "for i in range(1,6):\n",
    "    articles.update({'page'+str(i): (api.get_everything(q='Taiwan AND Elections', \n",
    "                                                        language= 'en', page = i))})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the information of the articles are now saved in our dictionary object called _articles_. It has a nested data structure that the iteration above saved every 20 articles for each page. As it stands, _articles_ does not have much use for us. It is complex, hard to read data object with numerous information for each article(i.e. date posted, author, source, abstract, full content,). If you want to take a look just run this code in an empty cell:\n",
    "```Python\n",
    "print(articles)\n",
    "```\n",
    "Looks complex and hard to read! As an example, let's take a look at the data on one article."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'urlToImage': 'https://static01.nyt.com/images/2019/12/02/opinion/02kassam/merlin_164544267_d1bde649-259b-4ea5-a9ad-161b056b48db-facebookJumbo.jpg', 'source': {'name': 'The New York Times', 'id': 'the-new-york-times'}, 'title': 'China Has Lost Taiwan, and It Knows It', 'url': 'https://www.nytimes.com/2019/12/01/opinion/china-taiwan-election.html', 'publishedAt': '2019-12-02T00:00:07Z', 'author': 'Natasha Kassam', 'content': 'The Sunflower Movement of 2014, a series of protests led by a coalition of students and civil-society activists, marked the rejection of close relations with China by Taiwans younger generations. So did the election of the pro-sovereignty Ms. Tsai in 2016.\\r\\nM… [+2354 chars]', 'description': 'So it is attacking democracy on the island from within.'}\n"
     ]
    }
   ],
   "source": [
    "print(articles['page1']['articles'][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__It is still complicated but gives a better view on the available data. Given that we have 100 of such a data, we need to manipulate and filter these information into a more useful form.__\n",
    "News API does not provide the full content of the articles. We need to use webscrapping to retrive the full content of each article. For now, we can use a function to parse the results to only save the fields we need. We need Title, Source, Publication Date,description and the URL. \n",
    "\n",
    "#### Functions in Python\n",
    "\n",
    "Functions are the fundamental programming tools that enables to wrap several statements and procudes the values that we desire. They make it easy for the code reusability and recyclability. For this workshop, it is sufficient just to grasp the basics of the functions in Python. A Function code usually basically looks like this:\n",
    "```Python\n",
    "def func_name(args):\n",
    "    statement\n",
    "    return result\n",
    "```\n",
    "You can also use 'yield' instead of return if your function is a generator. But it is a more advanced technique that we will not use in this workshop. After you define the function, you need to call it by its name, and if required you can bind the returned object to variable.\n",
    "```Python\n",
    "func_name() ## calls the function\n",
    "x = func_name() ## binds the returned object to a variable called x\n",
    "```\n",
    "Functions are a rich and powerful way in Python, and I recommend you to read more about them. \n",
    "\n",
    "We will now use a function to grap the information we need from the articles."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's __```import datetime```__  and __```dateutil.parser```__ modules for formatting existing publication date into a more readable format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from dateutil.parser import parse\n",
    "from datetime import datetime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's first create a helper function to make the publication date more readable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def reformat_date(date):\n",
    "    '''takes a string and returns a reformatted string'''\n",
    "    newdate = parse(date)\n",
    "    return newdate.strftime(\"%d-%B-%Y\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will create another helper function to prevent duplicate articles appearing in our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def check_duplicate(dataset,title):\n",
    "    '''\n",
    "    takes a list of dictionaries and a title string\n",
    "    to check for duplication of same articles\n",
    "    '''\n",
    "    for i in dataset:\n",
    "        if i['title'] == title:\n",
    "            return True\n",
    "        \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the article News API does not provide the full text of the articles, we need a web scrabbing function to retrive the full text of the each articles. We need to import __```requests```__ and __```BeautifulSoup```__ packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can write another helper function to retrive the full text of the articles. Since we might face errors and exceptions while retriving the full text from a website, it is important to catch the possible exceptions and handle them to prevent our application from breaking. We can do this using this syntax:\n",
    "```Python\n",
    "try:\n",
    "    some_code()\n",
    "except:\n",
    "    some_exception_handling()\n",
    "```\n",
    "Below, we use __\"```Exception as e```\"__ expression so that we can print the properties of the error to be able to handle it better next time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_fulltext(url):\n",
    "    '''\n",
    "    Takes the URL and returns \n",
    "    article full text. \n",
    "    '''\n",
    "    HEADERS = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5)'\n",
    "                ' AppleWebKit/537.36 (KHTML, like Gecko) Cafari/537.36'}\n",
    "    try:\n",
    "        page = requests.get(url,headers = HEADERS)\n",
    "        soup = BeautifulSoup(page.content, 'html.parser')\n",
    "        texts = soup.find_all('p')\n",
    "        article = ''\n",
    "        for i in texts:\n",
    "            article += str(i.get_text())\n",
    "        return article\n",
    "    except Exception as e:\n",
    "        print(e)\n",
    "        return None\n",
    "        \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now create a function to extract the information we want in a readable way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def article_extract(articles):\n",
    "    '''\n",
    "    takes a dictionary object returned from News API and\n",
    "    returns a list of dictionary with the required fields\n",
    "    '''\n",
    "    news_data = []\n",
    "    for i in articles.keys():\n",
    "        for n in range(0,len(articles[i]['articles'])):\n",
    "            if not check_duplicate(news_data,articles[i]['articles'][n]['title']):\n",
    "                news_data.append({'title':articles[i]['articles'][n]['title'],\n",
    "                                  'source': articles[i]['articles'][n]['source']['name'],\n",
    "                                  'URL': articles[i]['articles'][n]['url'],\n",
    "                                  'description': articles[i]['articles'][n]['description'],\n",
    "                                  'date': reformat_date(articles[i]['articles'][n]['publishedAt']),\n",
    "                                 'fulltext': get_fulltext(articles[i]['articles'][n]['url'])})\n",
    "        return news_data\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Now our function is ready for operation. Let's call it and see the first item of the dataset created by our function. It must be more readible with only required fields.__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'source': 'Reuters', 'date': '25-November-2019', 'URL': 'https://www.reuters.com/article/us-taiwan-election-idUSKBN1XZ0AP', 'title': \"Taiwan ruling party says China 'enemy of democracy' after meddling allegations\", 'fulltext': '4 Min ReadTAIPEI (Reuters) - Taiwan President Tsai Ing-wen’s ruling party denounced China as an “enemy of democracy” on Monday following fresh claims of Chinese interference in the island’s politics ahead of presidential and legislative elections on Jan. 11. The allegations, reported by Australian media, were made by a Chinese asylum seeker in Australia who said he was a Chinese spy. China, which claims Taiwan as its sacred territory, to be brought under Beijing’s control by force if necessary, has branded the asylum seeker a fraud. The Chinese man, Wang Liqiang, also provided details of Chinese efforts to infiltrate universities and media in the Chinese territory of Hong Kong, which has been rocked by months of anti-government protests.  Cho Jung-tai, chairman of Tsai’s Democratic Progressive Party, which favors Taiwan’s formal independence, said there needed to be further investigations, noting that a lot of fake news came from China. “The enemy of democracy is China. At present Taiwan’s most ambitious opponent, competitor, is also China,” Cho told a news conference in Taipei. Taiwan’s presidential office cited Tsai as saying on Monday that the allegations were being probed, and that people should not reach conclusions before a complete investigation was done. Among several allegations leveled, the would-be defector said he had helped guide positive media attention toward certain Taiwanese politicians, including President Tsai’s main opponent, Han Kuo-yu of the China-friendly Kuomintang party. The Kuomintang’s Han said he would drop out of the election if he has taken any money from the Chinese Communist Party.  Speaking at a separate news conference, Kuomintang’s spokeswoman Wang Hong-wei said the issue was one of “blundering Communist espionage” that should be investigated immediately, and accused the government of seeking to use the matter to “manipulate elections”. China’s state-backed Global Times tabloid said in a Monday opinion piece that Wang Liqiang sounded like an “opportunistic liar, probably a swindler”. Three Taipei-based diplomatic and security sources, speaking on condition of anonymity, told Reuters they also doubted whether Wang was who he claimed to be, though his allegations  were plausible. “Our assessment is he is most likely not who is says he is,” said one of the sources. Separately, Taiwan is investigating two directors of a Hong Kong-listed company which was named in the Australian reports as being involved in Communist Party infiltration of Hong Kong universities and media, the Justice Ministry’s Investigation Bureau said on Monday, without giving details. Wang said he was part of an intelligence operation working within Hong Kong-listed China Innovation Investment Limited to infiltrate Hong Kong universities and media with pro-Communist Party operatives to counter the territory’s democracy movement. In a statement to the Hong Kong Stock Exchange earlier on Monday, the company said the reports were “all fictitious and forged” and that Wang had never worked there. The company said that it had received notification from Executive Director Xiang Xin and alternate Director Kung Ching that they had been about to leave Taipei Taoyuan Airport when Taiwanese investigators requested their cooperation in an “investigation on the matter of the news reports”. “In fact, Mr. Xiang and Mrs. Kung knew nothing about the issues exposed in the news reports,” the statement said. It said both directors had engaged Taiwan lawyers to provide assistance. It was not immediately clear who their lawyers were.  The company declined to provide further details beyond what was in the statement. Reporting by Ben Blanchard and Yimou Lee; editing by Simon Cameron-MooreAll quotes delayed a minimum of 15 minutes. See here for a complete list of exchanges and delays.© 2019 Reuters. All Rights Reserved.', 'description': 'Taiwan President Tsai Ing-wen\\'s ruling party denounced China as an \"enemy of democracy\" on Monday following fresh claims of Chinese interference in the island\\'s politics ahead of presidential and legislative elections on Jan. 11.'}\n"
     ]
    }
   ],
   "source": [
    "data_set = article_extract(articles)\n",
    "print(data_set[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems from the results that we managed to create our data set. Now we can save it in a commo seperated value file to start our analysis. For this, we need to __```import csv```__ module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import csv\n",
    "with open(\"tw_dataset.csv\", 'w') as file:\n",
    "    tw_dt= csv.DictWriter(file,data_set[0].keys())\n",
    "    tw_dt.writeheader()\n",
    "    tw_dt.writerows(data_set)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Our Data Set is saved in our working directory and now ready for exploration and analysis!__\n",
    "\n",
    "<img src=\"images/dataset.gif\" style=\"width: 650px\" align=\"middle\" /> \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- __[Previous: Setting the Scene](0 - Setting the Scene.ipynb)__\n",
    "- __[Next: Data Exploration](2 - Data Exploration.ipynb)__"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}