T7 - Web Scraping - Nobel Prize.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Web Scraping\n",
    "\n",
    "**Examples**\n",
    "1. Nobel Prizes - https://github.com/SBalas/2019-CS109A/blob/master/content/sections/section1/solutions/section_1_solutions.ipynb\n",
    "2. Bourbon - https://harvard-iacs.github.io/2020-CS109A/lectures/lecture27/notebook/\n",
    "3. Article - https://medium.com/@oguss/web-scraping-a-beginners-tips-on-how-to-inspect-websites-using-google-chrome-and-extract-required-768d24661ca0\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 279,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url = 'https://www.nytimes.com/'\n",
    "\n",
    "# Get the webpage\n",
    "homepage = requests.get(url)\n",
    "\n",
    "# check status of getting: 200 is success, 400 is page not found\n",
    "homepage.status_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use beautiful soup to access contents\n",
    "soup = BeautifulSoup(homepage.text, 'html.parser')\n",
    "soup = BeautifulSoup(homepage.content, 'html.parser')\n",
    "\n",
    "# print raw data\n",
    "#soup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<title data-rh=\"true\">The New York Times - Breaking News, US News, World News and Videos</title>"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.title  # print raw title"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The New York Times - Breaking News, US News, World News and Videos'"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.title.string  # extrac title"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "#soup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<p class=\"css-5n6ag3\">Steve Bannon, who was a top aide to former President Trump, had refused to comply with subpoenas investigating the Jan. 6 attack on Congress.</p>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<a class=\"css-1f8er69\" href=\"#site-content\">Skip to content</a>"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.findAll('a') # all tags\n",
    "soup.find('a')    # first a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['#site-content',\n",
       " '#site-index',\n",
       " '/',\n",
       " '/',\n",
       " '/international/?action=click&region=Editions&pgtype=Homepage',\n",
       " '/ca/?action=click&region=Editions&pgtype=Homepage',\n",
       " 'https://www.nytimes.com/es/',\n",
       " 'https://cn.nytimes.com',\n",
       " 'https://www.nytimes.com/section/todayspaper',\n",
       " 'https://myaccount.nytimes.com/auth/login?response_type=cookie&client_id=vi&redirect_uri=https%3A%2F%2Fwww.nytimes.com%2Fsubscription%2Fmultiproduct%2Flp8KQUS.html%3FcampaignId%3D7JFJX&asset=masthead',\n",
       " '/',\n",
       " '/',\n",
       " '/section/world',\n",
       " '/section/us',\n",
       " '/section/politics',\n",
       " '/section/nyregion',\n",
       " '/section/business',\n",
       " '/section/opinion',\n",
       " '/section/technology',\n",
       " '/section/science',\n",
       " '/section/health',\n",
       " '/section/sports',\n",
       " '/section/arts',\n",
       " '/section/books',\n",
       " '/section/style',\n",
       " '/section/food',\n",
       " '/section/travel',\n",
       " '/section/magazine',\n",
       " '/section/t-magazine',\n",
       " '/section/realestate',\n",
       " '/video',\n",
       " 'https://www.nytimes.com/live/2021/11/13/climate/cop26-glasgow-climate-summit',\n",
       " 'https://www.nytimes.com/live/2021/11/13/climate/cop26-glasgow-climate-summit',\n",
       " 'https://www.nytimes.com/live/2021/11/13/climate/cop26-glasgow-climate-summit',\n",
       " '#after-dfp-ad-mid1',\n",
       " 'https://www.nytimes.com/2021/11/12/us/politics/bannon-indicted.html',\n",
       " 'https://www.nytimes.com/2021/11/12/us/politics/bannon-indicted.html',\n",
       " 'https://www.nytimes.com/2021/11/12/us/politics/republican-violent-rhetoric.html',\n",
       " 'https://www.nytimes.com/2021/11/12/us/politics/republican-violent-rhetoric.html',\n",
       " 'https://www.nytimes.com/2021/11/12/us/politics/republican-violent-rhetoric.html',\n",
       " 'https://www.nytimes.com/2021/11/13/us/politics/chris-christie-book-interview.html',\n",
       " 'https://www.nytimes.com/2021/11/13/us/politics/chris-christie-book-interview.html',\n",
       " 'https://www.nytimes.com/2021/11/12/us/politics/republican-violent-rhetoric.html',\n",
       " 'https://www.nytimes.com/2021/11/13/world/middleeast/belarus-migrants-iraq-kurds.html',\n",
       " 'https://www.nytimes.com/2021/11/13/world/middleeast/belarus-migrants-iraq-kurds.html',\n",
       " 'https://www.nytimes.com/2021/11/13/world/middleeast/belarus-migrants-iraq-kurds.html',\n",
       " 'https://www.nytimes.com/2021/11/12/world/europe/lukashenko-putin-belarus-russia.html',\n",
       " 'https://www.nytimes.com/2021/11/12/world/europe/lukashenko-putin-belarus-russia.html',\n",
       " 'https://www.nytimes.com/2021/11/13/health/coronavirus-school-nurses.html',\n",
       " 'https://www.nytimes.com/2021/11/13/health/coronavirus-school-nurses.html',\n",
       " 'https://www.nytimes.com/2021/11/13/health/coronavirus-school-nurses.html',\n",
       " 'https://www.nytimes.com/live/2021/11/13/world/covid-vaccine-boosters-mandate?type=styln-live-updates&label=coronavirus updates&index=0',\n",
       " 'https://www.nytimes.com/live/2021/11/13/world/covid-vaccine-boosters-mandate?type=styln-live-updates&label=coronavirus%20updates&index=0#portugal-remote-work-law-pandemic',\n",
       " 'https://www.nytimes.com/live/2021/11/13/world/covid-vaccine-boosters-mandate?type=styln-live-updates&label=coronavirus%20updates&index=0#appeals-court-upholds-block-on-bidens-vaccine-mandate-for-big-employers',\n",
       " 'https://www.nytimes.com/interactive/2021/us/covid-cases.html',\n",
       " 'https://www.nytimes.com/interactive/2021/us/covid-cases.html',\n",
       " 'https://www.nytimes.com/interactive/2021/us/covid-cases.html',\n",
       " 'https://www.nytimes.com/interactive/2020/us/covid-19-vaccine-doses.html',\n",
       " 'https://www.nytimes.com/interactive/2021/world/covid-cases.html',\n",
       " 'https://www.nytimes.com/interactive/2021/us/covid-cases-deaths-tracker.html',\n",
       " 'https://www.nytimes.com/interactive/2021/world/covid-vaccinations-tracker.html',\n",
       " 'https://www.nytimes.com/interactive/2021/world/united-kingdom-covid-cases.html',\n",
       " 'https://www.nytimes.com/interactive/2021/us/alaska-covid-cases.html',\n",
       " 'https://www.nytimes.com/interactive/2021/us/colorado-covid-cases.html',\n",
       " 'https://www.nytimes.com/interactive/2021/us/new-mexico-covid-cases.html',\n",
       " 'https://www.nytimes.com/interactive/2020/us/covid-hospitals-near-you.html',\n",
       " 'https://www.nytimes.com/interactive/2020/science/coronavirus-vaccine-tracker.html',\n",
       " 'https://www.nytimes.com/interactive/2021/world/covid-vaccinations-tracker.html',\n",
       " 'https://www.nytimes.com/interactive/2021/world/united-kingdom-covid-cases.html',\n",
       " 'https://www.nytimes.com/interactive/2021/us/alaska-covid-cases.html',\n",
       " 'https://www.nytimes.com/interactive/2021/us/colorado-covid-cases.html',\n",
       " 'https://www.nytimes.com/interactive/2021/us/new-mexico-covid-cases.html',\n",
       " 'https://www.nytimes.com/interactive/2020/us/covid-hospitals-near-you.html',\n",
       " 'https://www.nytimes.com/interactive/2020/science/coronavirus-vaccine-tracker.html',\n",
       " 'https://www.nytimes.com/interactive/2021/us/covid-cases-deaths-tracker.html',\n",
       " 'https://www.nytimes.com/2021/11/12/arts/music/britney-spears-conservatorship-ends.html',\n",
       " 'https://www.nytimes.com/2021/11/12/arts/music/britney-spears-conservatorship-ends.html',\n",
       " 'https://www.nytimes.com/2021/11/12/arts/music/britney-spears-conservatorship-ends.html',\n",
       " 'https://www.nytimes.com/2021/11/11/t-magazine/scorched-burnt-rice.html',\n",
       " 'https://www.nytimes.com/2021/11/11/t-magazine/scorched-burnt-rice.html',\n",
       " 'https://www.nytimes.com/2021/11/11/t-magazine/mexico-rice-conquest.html',\n",
       " 'https://www.nytimes.com/2021/11/11/t-magazine/mexico-rice-conquest.html',\n",
       " 'https://www.nytimes.com/2021/11/11/t-magazine/senegal-homegrown-rice.html',\n",
       " 'https://www.nytimes.com/2021/11/11/t-magazine/senegal-homegrown-rice.html',\n",
       " 'https://www.nytimes.com/2021/11/11/t-magazine/mansaf-bedouin-rice.html',\n",
       " 'https://www.nytimes.com/2021/11/11/t-magazine/mansaf-bedouin-rice.html',\n",
       " 'https://www.nytimes.com/interactive/2021/11/12/briefing/the-weekender.html',\n",
       " 'https://www.nytimes.com/interactive/2021/11/12/briefing/the-weekender.html',\n",
       " 'https://www.nytimes.com/interactive/2021/11/12/briefing/the-weekender.html',\n",
       " 'https://www.nytimes.com/interactive/2021/11/12/briefing/news-quiz-inflation-aaron-rodgers-poland-belarus.html',\n",
       " '#after-dfp-ad-mid2',\n",
       " 'https://www.nytimes.com/2021/11/12/realestate/disaster-proof-housing.html',\n",
       " 'https://www.nytimes.com/2021/11/12/realestate/disaster-proof-housing.html',\n",
       " 'https://www.nytimes.com/2021/11/12/realestate/disaster-proof-housing.html',\n",
       " 'https://www.nytimes.com/2021/11/09/well/move/exercise-chronic-pain.html',\n",
       " 'https://www.nytimes.com/article/recipes-for-kids.html',\n",
       " 'https://www.nytimes.com/2021/11/09/arts/television/jimmy-kimmel-ted-cruz-big-bird.html',\n",
       " 'https://www.nytimes.com/2021/11/10/arts/television/jimmy-kimmel-biden-is-steady-but-slow-like-grandpa-at-the-wheel.html',\n",
       " 'https://www.nytimes.com/2021/11/11/arts/television/stephen-colbert-trump-ruling.html',\n",
       " 'https://www.nytimes.com/2021/11/12/arts/television/jimmy-kimmel-qanon-shaman.html',\n",
       " 'https://www.nytimes.com/2021/11/12/arts/television/jimmy-kimmel-qanon-shaman.html',\n",
       " 'https://www.nytimes.com/2021/11/11/arts/television/stephen-colbert-trump-ruling.html',\n",
       " 'https://www.nytimes.com/section/opinion',\n",
       " 'https://www.nytimes.com/2021/11/12/opinion/biden-foreign-policy.html',\n",
       " 'https://www.nytimes.com/2021/11/12/opinion/kyrsten-sinema-clothing.html',\n",
       " 'https://www.nytimes.com/2021/11/12/opinion/kyrsten-sinema-clothing.html',\n",
       " 'https://www.nytimes.com/2021/11/12/opinion/book-bans.html',\n",
       " 'https://www.nytimes.com/2021/11/12/opinion/inflation-consumers.html',\n",
       " 'https://www.nytimes.com/2021/11/12/opinion/writing-spoken-argument.html',\n",
       " 'https://www.nytimes.com/2021/11/12/opinion/facebook-privacy.html',\n",
       " 'https://www.nytimes.com/2021/11/12/opinion/facebook-privacy.html',\n",
       " 'https://www.nytimes.com/2021/11/12/opinion/build-back-better-joe-manchin.html',\n",
       " 'https://www.nytimes.com/2021/11/12/opinion/ezra-klein-podcast-jay-rosen.html',\n",
       " 'https://www.nytimes.com/2021/11/12/opinion/william-holden-death-anxiety.html',\n",
       " 'https://www.nytimes.com/2021/11/12/opinion/letters/democrats-elections.html',\n",
       " 'https://www.nytimes.com/2021/11/12/opinion/climate-change-facebook-glasgow.html',\n",
       " 'https://www.nytimes.com/2021/11/12/opinion/inflation-us-europe-covid.html',\n",
       " 'https://www.nytimes.com/2021/11/12/opinion/gerrymandering-guarantee-clause.html',\n",
       " 'https://www.nytimes.com/2021/11/12/opinion/ethiopia-tigray-abiy-war.html',\n",
       " '#after-dfp-ad-mid3',\n",
       " 'https://www.nytimes.com/2021/11/12/dining/best-pumpkin-apple-pecan-pie-recipes-thanksgiving.html',\n",
       " 'https://cooking.nytimes.com/recipes/1022737-florentine-dip',\n",
       " 'https://cooking.nytimes.com/recipes/1022737-florentine-dip',\n",
       " 'https://cooking.nytimes.com/recipes/1022707-trinidadian-macaroni-pie',\n",
       " 'https://cooking.nytimes.com/recipes/1022707-trinidadian-macaroni-pie',\n",
       " 'https://cooking.nytimes.com/topics/vegan-thanksgiving',\n",
       " 'https://cooking.nytimes.com/topics/vegan-thanksgiving',\n",
       " 'https://cooking.nytimes.com/recipes/1022001-yuca-con-mojo',\n",
       " 'https://cooking.nytimes.com/recipes/1022001-yuca-con-mojo',\n",
       " 'https://cooking.nytimes.com/recipes/1022669-french-onion-stuffing',\n",
       " 'https://cooking.nytimes.com/recipes/1022669-french-onion-stuffing',\n",
       " 'https://www.nytimes.com/2021/11/13/nyregion/city-council-speaker-nyc.html',\n",
       " 'https://www.nytimes.com/2021/11/13/nyregion/city-council-speaker-nyc.html',\n",
       " 'https://www.nytimes.com/2021/11/13/world/canada/newfoundland-migrants.html',\n",
       " 'https://www.nytimes.com/2021/11/13/world/canada/newfoundland-migrants.html',\n",
       " 'https://www.nytimes.com/2021/11/13/world/canada/newfoundland-migrants.html',\n",
       " 'https://www.nytimes.com/2021/11/13/world/europe/france-2015-attacks-trial-victims.html',\n",
       " 'https://www.nytimes.com/2021/11/13/world/europe/france-2015-attacks-trial-victims.html',\n",
       " 'https://www.nytimes.com/2021/11/12/us/plessy-ferguson-pardon.html',\n",
       " 'https://www.nytimes.com/2021/11/12/us/plessy-ferguson-pardon.html',\n",
       " 'https://www.nytimes.com/2021/11/12/us/ahmaud-arbery-trial-lawyer-kevin-gough.html',\n",
       " 'https://www.nytimes.com/2021/11/13/world/africa/sudan-protests.html',\n",
       " 'https://www.nytimes.com/2021/11/13/us/doctor-denali-false-report-helicopter.html',\n",
       " 'https://www.nytimes.com/2021/11/13/sports/soccer/usmnt-mexico-pulisic-mckennie.html',\n",
       " 'https://www.nytimes.com/2021/11/12/nyregion/glen-de-vries-plane-crash.html',\n",
       " 'https://www.nytimes.com/2021/11/13/world/asia/philippines-sara-duterte-marcos.html',\n",
       " 'https://www.nytimes.com/2021/11/12/business/helene-fortunoff-dead.html',\n",
       " '#after-dfp-ad-mid4',\n",
       " 'https://www.nytimes.com/2021/11/12/podcasts/aaron-rodgers-coronavirus-apple-narrated-articles.html',\n",
       " 'https://www.nytimes.com/2021/11/12/podcasts/aaron-rodgers-coronavirus-apple-narrated-articles.html',\n",
       " 'https://www.nytimes.com/2021/11/12/podcasts/aaron-rodgers-coronavirus-apple-narrated-articles.html',\n",
       " 'https://www.nytimes.com/2021/11/12/books/review/podcast-alan-cumming-baggage-allen-guelzo-robert-e-lee-biography.html',\n",
       " 'https://www.nytimes.com/2021/11/12/books/review/podcast-alan-cumming-baggage-allen-guelzo-robert-e-lee-biography.html',\n",
       " 'https://www.nytimes.com/2021/11/12/books/review/podcast-alan-cumming-baggage-allen-guelzo-robert-e-lee-biography.html',\n",
       " 'https://www.nytimes.com/2021/10/20/technology/healthy-online-conversations-event.html',\n",
       " 'https://www.nytimes.com/2021/10/20/technology/healthy-online-conversations-event.html',\n",
       " 'https://www.nytimes.com/2021/10/20/technology/healthy-online-conversations-event.html',\n",
       " 'https://www.nytimes.com/newsletters/rory-smith',\n",
       " 'https://www.nytimes.com/newsletters/rory-smith',\n",
       " 'https://www.nytimes.com/newsletters/rory-smith',\n",
       " 'https://www.nytimes.com/2021/11/12/arts/design/nick-relph-eclipse-body-soul-syntax.html',\n",
       " 'https://www.nytimes.com/2021/11/12/arts/design/nick-relph-eclipse-body-soul-syntax.html',\n",
       " 'https://www.nytimes.com/2021/11/11/arts/design/barbara-kruger-art-institute-review.html',\n",
       " 'https://www.nytimes.com/2021/11/11/arts/design/barbara-kruger-art-institute-review.html',\n",
       " 'https://www.nytimes.com/2021/11/11/style/parents-financial-support-social-qs.html',\n",
       " 'https://www.nytimes.com/2021/11/11/style/parents-financial-support-social-qs.html',\n",
       " 'https://www.nytimes.com/2021/11/12/books/review/artists-best-illustrated-childrens-books.html',\n",
       " 'https://www.nytimes.com/2021/11/12/books/review/artists-best-illustrated-childrens-books.html',\n",
       " 'https://www.nytimes.com/2021/11/09/style/grand-prospect-hall-brooklyn-demolition.html',\n",
       " 'https://www.nytimes.com/2021/11/09/style/grand-prospect-hall-brooklyn-demolition.html',\n",
       " 'https://www.nytimes.com/2020/11/19/at-home/winter-running.html',\n",
       " 'https://www.nytimes.com/2020/11/19/at-home/winter-running.html',\n",
       " 'https://www.nytimes.com/2020/09/22/well/family/childhood-trauma-family.html',\n",
       " 'https://www.nytimes.com/2020/09/22/well/family/childhood-trauma-family.html',\n",
       " 'https://www.nytimes.com/2017/09/19/well/mind/how-honesty-could-make-you-happier.html',\n",
       " 'https://www.nytimes.com/2017/09/19/well/mind/how-honesty-could-make-you-happier.html',\n",
       " 'https://www.nytimes.com/2020/06/30/parenting/introduce-new-baby-sibling.html',\n",
       " 'https://www.nytimes.com/2020/06/30/parenting/introduce-new-baby-sibling.html',\n",
       " 'https://www.nytimes.com/interactive/2021/11/11/well/live/11healthquiz-11112021.html',\n",
       " 'https://www.nytimes.com/interactive/2021/11/11/well/live/11healthquiz-11112021.html',\n",
       " 'https://www.nytimes.com/wirecutter/',\n",
       " 'https://www.nytimes.com/wirecutter/reviews/best-bed-pillows/',\n",
       " 'https://www.nytimes.com/wirecutter/reviews/best-bed-pillows/',\n",
       " 'https://www.nytimes.com/wirecutter/reviews/best-usb-car-charger/',\n",
       " 'https://www.nytimes.com/wirecutter/reviews/best-usb-car-charger/',\n",
       " 'https://www.nytimes.com/wirecutter/gifts/gifts-for-people-who-have-everything/',\n",
       " 'https://www.nytimes.com/wirecutter/gifts/gifts-for-people-who-have-everything/',\n",
       " 'https://www.nytimes.com/wirecutter/gifts/gifts-for-people-who-have-everything/',\n",
       " 'https://www.nytimes.com/wirecutter/reviews/best-automatic-feeder-for-cats-and-small-dogs/',\n",
       " 'https://www.nytimes.com/wirecutter/reviews/best-automatic-feeder-for-cats-and-small-dogs/',\n",
       " '#after-dfp-ad-bottom',\n",
       " 'https://www.nytimes.com/puzzles/spelling-bee',\n",
       " 'https://www.nytimes.com/puzzles/spelling-bee',\n",
       " 'https://www.nytimes.com/puzzles/spelling-bee',\n",
       " 'https://www.nytimes.com/crosswords',\n",
       " 'https://www.nytimes.com/crosswords',\n",
       " 'https://www.nytimes.com/crosswords',\n",
       " 'https://www.nytimes.com/puzzles/letter-boxed',\n",
       " 'https://www.nytimes.com/puzzles/letter-boxed',\n",
       " 'https://www.nytimes.com/puzzles/letter-boxed',\n",
       " 'https://www.nytimes.com/puzzles/tiles',\n",
       " 'https://www.nytimes.com/puzzles/tiles',\n",
       " 'https://www.nytimes.com/puzzles/tiles',\n",
       " 'https://www.nytimes.com/puzzles/vertex',\n",
       " 'https://www.nytimes.com/puzzles/vertex',\n",
       " 'https://www.nytimes.com/puzzles/vertex',\n",
       " 'https://help.nytimes.com/hc/en-us/articles/115014792127-Copyright-notice',\n",
       " 'https://www.nytco.com/',\n",
       " 'https://help.nytimes.com/hc/en-us/articles/115015385887-Contact-Us',\n",
       " 'https://help.nytimes.com/hc/en-us/articles/115015727108-Accessibility',\n",
       " 'https://www.nytco.com/careers/',\n",
       " 'https://nytmediakit.com/',\n",
       " 'http://www.tbrandstudio.com/',\n",
       " 'https://www.nytimes.com/privacy/cookie-policy#how-do-i-manage-trackers',\n",
       " 'https://www.nytimes.com/privacy/privacy-policy',\n",
       " 'https://help.nytimes.com/hc/en-us/articles/115014893428-Terms-of-service',\n",
       " 'https://help.nytimes.com/hc/en-us/articles/115014893968-Terms-of-sale',\n",
       " '/sitemap/',\n",
       " 'https://www.nytimes.com/ca/?action=click&region=Footer&pgtype=Homepage',\n",
       " 'https://www.nytimes.com/international/?action=click&region=Footer&pgtype=Homepage',\n",
       " 'https://help.nytimes.com/hc/en-us',\n",
       " 'https://www.nytimes.com/subscription?campaignId=37WXW']"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get all the links in the page\n",
    "link_list = [l.get('href') for l in soup.findAll('a')]\n",
    "link_list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#site-content\n",
      "#site-index\n",
      "/\n",
      "/\n",
      "/international/?action=click&region=Editions&pgtype=Homepage\n",
      "/ca/?action=click&region=Editions&pgtype=Homepage\n",
      "https://www.nytimes.com/es/\n",
      "https://cn.nytimes.com\n",
      "https://www.nytimes.com/section/todayspaper\n",
      "https://myaccount.nytimes.com/auth/login?response_type=cookie&client_id=vi&redirect_uri=https%3A%2F%2Fwww.nytimes.com%2Fsubscription%2Fmultiproduct%2Flp8KQUS.html%3FcampaignId%3D7JFJX&asset=masthead\n",
      "/\n",
      "/\n",
      "/section/world\n",
      "/section/us\n",
      "/section/politics\n",
      "/section/nyregion\n",
      "/section/business\n",
      "/section/opinion\n",
      "/section/technology\n",
      "/section/science\n",
      "/section/health\n",
      "/section/sports\n",
      "/section/arts\n",
      "/section/books\n",
      "/section/style\n",
      "/section/food\n",
      "/section/travel\n",
      "/section/magazine\n",
      "/section/t-magazine\n",
      "/section/realestate\n",
      "/video\n",
      "https://www.nytimes.com/live/2021/11/13/climate/cop26-glasgow-climate-summit\n",
      "https://www.nytimes.com/live/2021/11/13/climate/cop26-glasgow-climate-summit\n",
      "https://www.nytimes.com/live/2021/11/13/climate/cop26-glasgow-climate-summit\n",
      "#after-dfp-ad-mid1\n",
      "https://www.nytimes.com/2021/11/12/us/politics/bannon-indicted.html\n",
      "https://www.nytimes.com/2021/11/12/us/politics/bannon-indicted.html\n",
      "https://www.nytimes.com/2021/11/12/us/politics/republican-violent-rhetoric.html\n",
      "https://www.nytimes.com/2021/11/12/us/politics/republican-violent-rhetoric.html\n",
      "https://www.nytimes.com/2021/11/12/us/politics/republican-violent-rhetoric.html\n",
      "https://www.nytimes.com/2021/11/13/us/politics/chris-christie-book-interview.html\n",
      "https://www.nytimes.com/2021/11/13/us/politics/chris-christie-book-interview.html\n",
      "https://www.nytimes.com/2021/11/12/us/politics/republican-violent-rhetoric.html\n",
      "https://www.nytimes.com/2021/11/13/world/middleeast/belarus-migrants-iraq-kurds.html\n",
      "https://www.nytimes.com/2021/11/13/world/middleeast/belarus-migrants-iraq-kurds.html\n",
      "https://www.nytimes.com/2021/11/13/world/middleeast/belarus-migrants-iraq-kurds.html\n",
      "https://www.nytimes.com/2021/11/12/world/europe/lukashenko-putin-belarus-russia.html\n",
      "https://www.nytimes.com/2021/11/12/world/europe/lukashenko-putin-belarus-russia.html\n",
      "https://www.nytimes.com/2021/11/13/health/coronavirus-school-nurses.html\n",
      "https://www.nytimes.com/2021/11/13/health/coronavirus-school-nurses.html\n",
      "https://www.nytimes.com/2021/11/13/health/coronavirus-school-nurses.html\n",
      "https://www.nytimes.com/live/2021/11/13/world/covid-vaccine-boosters-mandate?type=styln-live-updates&label=coronavirus updates&index=0\n",
      "https://www.nytimes.com/live/2021/11/13/world/covid-vaccine-boosters-mandate?type=styln-live-updates&label=coronavirus%20updates&index=0#portugal-remote-work-law-pandemic\n",
      "https://www.nytimes.com/live/2021/11/13/world/covid-vaccine-boosters-mandate?type=styln-live-updates&label=coronavirus%20updates&index=0#appeals-court-upholds-block-on-bidens-vaccine-mandate-for-big-employers\n",
      "https://www.nytimes.com/interactive/2021/us/covid-cases.html\n",
      "https://www.nytimes.com/interactive/2021/us/covid-cases.html\n",
      "https://www.nytimes.com/interactive/2021/us/covid-cases.html\n",
      "https://www.nytimes.com/interactive/2020/us/covid-19-vaccine-doses.html\n",
      "https://www.nytimes.com/interactive/2021/world/covid-cases.html\n",
      "https://www.nytimes.com/interactive/2021/us/covid-cases-deaths-tracker.html\n",
      "https://www.nytimes.com/interactive/2021/world/covid-vaccinations-tracker.html\n",
      "https://www.nytimes.com/interactive/2021/world/united-kingdom-covid-cases.html\n",
      "https://www.nytimes.com/interactive/2021/us/alaska-covid-cases.html\n",
      "https://www.nytimes.com/interactive/2021/us/colorado-covid-cases.html\n",
      "https://www.nytimes.com/interactive/2021/us/new-mexico-covid-cases.html\n",
      "https://www.nytimes.com/interactive/2020/us/covid-hospitals-near-you.html\n",
      "https://www.nytimes.com/interactive/2020/science/coronavirus-vaccine-tracker.html\n",
      "https://www.nytimes.com/interactive/2021/world/covid-vaccinations-tracker.html\n",
      "https://www.nytimes.com/interactive/2021/world/united-kingdom-covid-cases.html\n",
      "https://www.nytimes.com/interactive/2021/us/alaska-covid-cases.html\n",
      "https://www.nytimes.com/interactive/2021/us/colorado-covid-cases.html\n",
      "https://www.nytimes.com/interactive/2021/us/new-mexico-covid-cases.html\n",
      "https://www.nytimes.com/interactive/2020/us/covid-hospitals-near-you.html\n",
      "https://www.nytimes.com/interactive/2020/science/coronavirus-vaccine-tracker.html\n",
      "https://www.nytimes.com/interactive/2021/us/covid-cases-deaths-tracker.html\n",
      "https://www.nytimes.com/2021/11/12/arts/music/britney-spears-conservatorship-ends.html\n",
      "https://www.nytimes.com/2021/11/12/arts/music/britney-spears-conservatorship-ends.html\n",
      "https://www.nytimes.com/2021/11/12/arts/music/britney-spears-conservatorship-ends.html\n",
      "https://www.nytimes.com/2021/11/11/t-magazine/scorched-burnt-rice.html\n",
      "https://www.nytimes.com/2021/11/11/t-magazine/scorched-burnt-rice.html\n",
      "https://www.nytimes.com/2021/11/11/t-magazine/mexico-rice-conquest.html\n",
      "https://www.nytimes.com/2021/11/11/t-magazine/mexico-rice-conquest.html\n",
      "https://www.nytimes.com/2021/11/11/t-magazine/senegal-homegrown-rice.html\n",
      "https://www.nytimes.com/2021/11/11/t-magazine/senegal-homegrown-rice.html\n",
      "https://www.nytimes.com/2021/11/11/t-magazine/mansaf-bedouin-rice.html\n",
      "https://www.nytimes.com/2021/11/11/t-magazine/mansaf-bedouin-rice.html\n",
      "https://www.nytimes.com/interactive/2021/11/12/briefing/the-weekender.html\n",
      "https://www.nytimes.com/interactive/2021/11/12/briefing/the-weekender.html\n",
      "https://www.nytimes.com/interactive/2021/11/12/briefing/the-weekender.html\n",
      "https://www.nytimes.com/interactive/2021/11/12/briefing/news-quiz-inflation-aaron-rodgers-poland-belarus.html\n",
      "#after-dfp-ad-mid2\n",
      "https://www.nytimes.com/2021/11/12/realestate/disaster-proof-housing.html\n",
      "https://www.nytimes.com/2021/11/12/realestate/disaster-proof-housing.html\n",
      "https://www.nytimes.com/2021/11/12/realestate/disaster-proof-housing.html\n",
      "https://www.nytimes.com/2021/11/09/well/move/exercise-chronic-pain.html\n",
      "https://www.nytimes.com/article/recipes-for-kids.html\n",
      "https://www.nytimes.com/2021/11/09/arts/television/jimmy-kimmel-ted-cruz-big-bird.html\n",
      "https://www.nytimes.com/2021/11/10/arts/television/jimmy-kimmel-biden-is-steady-but-slow-like-grandpa-at-the-wheel.html\n",
      "https://www.nytimes.com/2021/11/11/arts/television/stephen-colbert-trump-ruling.html\n",
      "https://www.nytimes.com/2021/11/12/arts/television/jimmy-kimmel-qanon-shaman.html\n",
      "https://www.nytimes.com/2021/11/12/arts/television/jimmy-kimmel-qanon-shaman.html\n",
      "https://www.nytimes.com/2021/11/11/arts/television/stephen-colbert-trump-ruling.html\n",
      "https://www.nytimes.com/section/opinion\n",
      "https://www.nytimes.com/2021/11/12/opinion/biden-foreign-policy.html\n",
      "https://www.nytimes.com/2021/11/12/opinion/kyrsten-sinema-clothing.html\n",
      "https://www.nytimes.com/2021/11/12/opinion/kyrsten-sinema-clothing.html\n",
      "https://www.nytimes.com/2021/11/12/opinion/book-bans.html\n",
      "https://www.nytimes.com/2021/11/12/opinion/inflation-consumers.html\n",
      "https://www.nytimes.com/2021/11/12/opinion/writing-spoken-argument.html\n",
      "https://www.nytimes.com/2021/11/12/opinion/facebook-privacy.html\n",
      "https://www.nytimes.com/2021/11/12/opinion/facebook-privacy.html\n",
      "https://www.nytimes.com/2021/11/12/opinion/build-back-better-joe-manchin.html\n",
      "https://www.nytimes.com/2021/11/12/opinion/ezra-klein-podcast-jay-rosen.html\n",
      "https://www.nytimes.com/2021/11/12/opinion/william-holden-death-anxiety.html\n",
      "https://www.nytimes.com/2021/11/12/opinion/letters/democrats-elections.html\n",
      "https://www.nytimes.com/2021/11/12/opinion/climate-change-facebook-glasgow.html\n",
      "https://www.nytimes.com/2021/11/12/opinion/inflation-us-europe-covid.html\n",
      "https://www.nytimes.com/2021/11/12/opinion/gerrymandering-guarantee-clause.html\n",
      "https://www.nytimes.com/2021/11/12/opinion/ethiopia-tigray-abiy-war.html\n",
      "#after-dfp-ad-mid3\n",
      "https://www.nytimes.com/2021/11/12/dining/best-pumpkin-apple-pecan-pie-recipes-thanksgiving.html\n",
      "https://cooking.nytimes.com/recipes/1022737-florentine-dip\n",
      "https://cooking.nytimes.com/recipes/1022737-florentine-dip\n",
      "https://cooking.nytimes.com/recipes/1022707-trinidadian-macaroni-pie\n",
      "https://cooking.nytimes.com/recipes/1022707-trinidadian-macaroni-pie\n",
      "https://cooking.nytimes.com/topics/vegan-thanksgiving\n",
      "https://cooking.nytimes.com/topics/vegan-thanksgiving\n",
      "https://cooking.nytimes.com/recipes/1022001-yuca-con-mojo\n",
      "https://cooking.nytimes.com/recipes/1022001-yuca-con-mojo\n",
      "https://cooking.nytimes.com/recipes/1022669-french-onion-stuffing\n",
      "https://cooking.nytimes.com/recipes/1022669-french-onion-stuffing\n",
      "https://www.nytimes.com/2021/11/13/nyregion/city-council-speaker-nyc.html\n",
      "https://www.nytimes.com/2021/11/13/nyregion/city-council-speaker-nyc.html\n",
      "https://www.nytimes.com/2021/11/13/world/canada/newfoundland-migrants.html\n",
      "https://www.nytimes.com/2021/11/13/world/canada/newfoundland-migrants.html\n",
      "https://www.nytimes.com/2021/11/13/world/canada/newfoundland-migrants.html\n",
      "https://www.nytimes.com/2021/11/13/world/europe/france-2015-attacks-trial-victims.html\n",
      "https://www.nytimes.com/2021/11/13/world/europe/france-2015-attacks-trial-victims.html\n",
      "https://www.nytimes.com/2021/11/12/us/plessy-ferguson-pardon.html\n",
      "https://www.nytimes.com/2021/11/12/us/plessy-ferguson-pardon.html\n",
      "https://www.nytimes.com/2021/11/12/us/ahmaud-arbery-trial-lawyer-kevin-gough.html\n",
      "https://www.nytimes.com/2021/11/13/world/africa/sudan-protests.html\n",
      "https://www.nytimes.com/2021/11/13/us/doctor-denali-false-report-helicopter.html\n",
      "https://www.nytimes.com/2021/11/13/sports/soccer/usmnt-mexico-pulisic-mckennie.html\n",
      "https://www.nytimes.com/2021/11/12/nyregion/glen-de-vries-plane-crash.html\n",
      "https://www.nytimes.com/2021/11/13/world/asia/philippines-sara-duterte-marcos.html\n",
      "https://www.nytimes.com/2021/11/12/business/helene-fortunoff-dead.html\n",
      "#after-dfp-ad-mid4\n",
      "https://www.nytimes.com/2021/11/12/podcasts/aaron-rodgers-coronavirus-apple-narrated-articles.html\n",
      "https://www.nytimes.com/2021/11/12/podcasts/aaron-rodgers-coronavirus-apple-narrated-articles.html\n",
      "https://www.nytimes.com/2021/11/12/podcasts/aaron-rodgers-coronavirus-apple-narrated-articles.html\n",
      "https://www.nytimes.com/2021/11/12/books/review/podcast-alan-cumming-baggage-allen-guelzo-robert-e-lee-biography.html\n",
      "https://www.nytimes.com/2021/11/12/books/review/podcast-alan-cumming-baggage-allen-guelzo-robert-e-lee-biography.html\n",
      "https://www.nytimes.com/2021/11/12/books/review/podcast-alan-cumming-baggage-allen-guelzo-robert-e-lee-biography.html\n",
      "https://www.nytimes.com/2021/10/20/technology/healthy-online-conversations-event.html\n",
      "https://www.nytimes.com/2021/10/20/technology/healthy-online-conversations-event.html\n",
      "https://www.nytimes.com/2021/10/20/technology/healthy-online-conversations-event.html\n",
      "https://www.nytimes.com/newsletters/rory-smith\n",
      "https://www.nytimes.com/newsletters/rory-smith\n",
      "https://www.nytimes.com/newsletters/rory-smith\n",
      "https://www.nytimes.com/2021/11/12/arts/design/nick-relph-eclipse-body-soul-syntax.html\n",
      "https://www.nytimes.com/2021/11/12/arts/design/nick-relph-eclipse-body-soul-syntax.html\n",
      "https://www.nytimes.com/2021/11/11/arts/design/barbara-kruger-art-institute-review.html\n",
      "https://www.nytimes.com/2021/11/11/arts/design/barbara-kruger-art-institute-review.html\n",
      "https://www.nytimes.com/2021/11/11/style/parents-financial-support-social-qs.html\n",
      "https://www.nytimes.com/2021/11/11/style/parents-financial-support-social-qs.html\n",
      "https://www.nytimes.com/2021/11/12/books/review/artists-best-illustrated-childrens-books.html\n",
      "https://www.nytimes.com/2021/11/12/books/review/artists-best-illustrated-childrens-books.html\n",
      "https://www.nytimes.com/2021/11/09/style/grand-prospect-hall-brooklyn-demolition.html\n",
      "https://www.nytimes.com/2021/11/09/style/grand-prospect-hall-brooklyn-demolition.html\n",
      "https://www.nytimes.com/2020/11/19/at-home/winter-running.html\n",
      "https://www.nytimes.com/2020/11/19/at-home/winter-running.html\n",
      "https://www.nytimes.com/2020/09/22/well/family/childhood-trauma-family.html\n",
      "https://www.nytimes.com/2020/09/22/well/family/childhood-trauma-family.html\n",
      "https://www.nytimes.com/2017/09/19/well/mind/how-honesty-could-make-you-happier.html\n",
      "https://www.nytimes.com/2017/09/19/well/mind/how-honesty-could-make-you-happier.html\n",
      "https://www.nytimes.com/2020/06/30/parenting/introduce-new-baby-sibling.html\n",
      "https://www.nytimes.com/2020/06/30/parenting/introduce-new-baby-sibling.html\n",
      "https://www.nytimes.com/interactive/2021/11/11/well/live/11healthquiz-11112021.html\n",
      "https://www.nytimes.com/interactive/2021/11/11/well/live/11healthquiz-11112021.html\n",
      "https://www.nytimes.com/wirecutter/\n",
      "https://www.nytimes.com/wirecutter/reviews/best-bed-pillows/\n",
      "https://www.nytimes.com/wirecutter/reviews/best-bed-pillows/\n",
      "https://www.nytimes.com/wirecutter/reviews/best-usb-car-charger/\n",
      "https://www.nytimes.com/wirecutter/reviews/best-usb-car-charger/\n",
      "https://www.nytimes.com/wirecutter/gifts/gifts-for-people-who-have-everything/\n",
      "https://www.nytimes.com/wirecutter/gifts/gifts-for-people-who-have-everything/\n",
      "https://www.nytimes.com/wirecutter/gifts/gifts-for-people-who-have-everything/\n",
      "https://www.nytimes.com/wirecutter/reviews/best-automatic-feeder-for-cats-and-small-dogs/\n",
      "https://www.nytimes.com/wirecutter/reviews/best-automatic-feeder-for-cats-and-small-dogs/\n",
      "#after-dfp-ad-bottom\n",
      "https://www.nytimes.com/puzzles/spelling-bee\n",
      "https://www.nytimes.com/puzzles/spelling-bee\n",
      "https://www.nytimes.com/puzzles/spelling-bee\n",
      "https://www.nytimes.com/crosswords\n",
      "https://www.nytimes.com/crosswords\n",
      "https://www.nytimes.com/crosswords\n",
      "https://www.nytimes.com/puzzles/letter-boxed\n",
      "https://www.nytimes.com/puzzles/letter-boxed\n",
      "https://www.nytimes.com/puzzles/letter-boxed\n",
      "https://www.nytimes.com/puzzles/tiles\n",
      "https://www.nytimes.com/puzzles/tiles\n",
      "https://www.nytimes.com/puzzles/tiles\n",
      "https://www.nytimes.com/puzzles/vertex\n",
      "https://www.nytimes.com/puzzles/vertex\n",
      "https://www.nytimes.com/puzzles/vertex\n",
      "https://help.nytimes.com/hc/en-us/articles/115014792127-Copyright-notice\n",
      "https://www.nytco.com/\n",
      "https://help.nytimes.com/hc/en-us/articles/115015385887-Contact-Us\n",
      "https://help.nytimes.com/hc/en-us/articles/115015727108-Accessibility\n",
      "https://www.nytco.com/careers/\n",
      "https://nytmediakit.com/\n",
      "http://www.tbrandstudio.com/\n",
      "https://www.nytimes.com/privacy/cookie-policy#how-do-i-manage-trackers\n",
      "https://www.nytimes.com/privacy/privacy-policy\n",
      "https://help.nytimes.com/hc/en-us/articles/115014893428-Terms-of-service\n",
      "https://help.nytimes.com/hc/en-us/articles/115014893968-Terms-of-sale\n",
      "/sitemap/\n",
      "https://www.nytimes.com/ca/?action=click&region=Footer&pgtype=Homepage\n",
      "https://www.nytimes.com/international/?action=click&region=Footer&pgtype=Homepage\n",
      "https://help.nytimes.com/hc/en-us\n",
      "https://www.nytimes.com/subscription?campaignId=37WXW\n"
     ]
    }
   ],
   "source": [
    "for link in soup.find_all('a'):\n",
    "    print(link.get('href'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print all the text\n",
    "# print(soup.get_text())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example with Nobel prize archive\n",
    "\n",
    "Use [2018 version of the Nobel website](http://web.archive.org/web/20180820111639/https://www.nobelprize.org/prizes/lists/all-nobel-prizes/)\n",
    "to extract list of nobel recipients by year and category\n",
    "\n",
    "**HTML tags**\n",
    "\n",
    "**\\<h3> : header 3 tag** tag is a header size 3 tag (header 1 is the largest tag). This tag will contain the title and year of the nobel prize, which we will parse out. <br>\n",
    "**\\<h6> : header 6 tag** tag (smaller than header 3) will contain the prize recipients <br>\n",
    "**\\<p> : paragraph tag** tags used for text, contains the prize motivation <br>\n",
    "**\\<div class =\"Class: by year\">** \"Content Division element ( \\<div> ) is the generic container for flow content.\" What we care about here is the class attribute, which we will use with beautiful soup to quickly parse information which we want. The class attribute could be attatched to any tag.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Response [200]>"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url = 'http://web.archive.org/web/20180820111639/https://www.nobelprize.org/prizes/lists/all-nobel-prizes/'\n",
    "snapshot = requests.get(url)\n",
    "snapshot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<!DOCTYPE html>\\n<html class=\"no-js\" lang=\"en-US\" prefix=\"og: http://ogp.me/ns#\">\\n <head>\\n  <script src=\"//archive.org/includes/analytics.js?v=cf34f82\" type=\"text/javascript\">\\n  </script>\\n  <script type=\"text/javascript\">\\n   window.addEventListener(\\'DOMContentLoaded\\',function(){var v=archive_analytics.values;v.service=\\'wb\\';v.server_name=\\'wwwb-app227.us.archive.org\\';v.server_ms=947;archive_analytics.send_pageview({});});\\n  </script>\\n  <script charset=\"utf-8\" src=\"/_static/js/bundle-playback.js?v=U'"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get all the text and print the first 500 chars\n",
    "soup = BeautifulSoup(snapshot.text, 'html.parser')\n",
    "all_text = soup.prettify()\n",
    "all_text[:500]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'All Nobel Prizes'"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get title\n",
    "soup.title.string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 266,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<a href=\"http://web.archive.org/web/20180820111639/https://www.nobelprize.org/nobel_prizes/physics/laureates/2018/\">The Nobel Prize in Physics 2018</a>,\n",
       " <a href=\"http://web.archive.org/web/20180820111639/https://www.nobelprize.org/nobel_prizes/chemistry/laureates/2018/\">The Nobel Prize in Chemistry 2018</a>,\n",
       " <a href=\"http://web.archive.org/web/20180820111639/https://www.nobelprize.org/nobel_prizes/medicine/laureates/2018/\">The Nobel Prize in Physiology or Medicine 2018</a>,\n",
       " <a href=\"http://web.archive.org/web/20180820111639/https://www.nobelprize.org/nobel_prizes/literature/laureates/2018/\">The Nobel Prize in Literature 2018</a>,\n",
       " <a href=\"http://web.archive.org/web/20180820111639/https://www.nobelprize.org/nobel_prizes/peace/laureates/2018/\">The Nobel Peace Prize 2018</a>]"
      ]
     },
     "execution_count": 266,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# returns the objects in first titles\n",
    "soup.select(\"h3 a\")[0:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extract award data\n",
    "Each award is in a ```div``` with a ```by_year``` class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "640"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "award_nodes = soup.select('.by_year')\n",
    "len(award_nodes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<div class=\"by_year\">\n",
       "<h3><a href=\"http://web.archive.org/web/20180820111639/https://www.nobelprize.org/nobel_prizes/physics/laureates/2017/\">The Nobel Prize in Physics 2017</a></h3>\n",
       "<h6><a href=\"http://web.archive.org/web/20180820111639/https://www.nobelprize.org/nobel_prizes/physics/laureates/2017/weiss-facts.html\">Rainer Weiss</a>, <a href=\"http://web.archive.org/web/20180820111639/https://www.nobelprize.org/nobel_prizes/physics/laureates/2017/barish-facts.html\">Barry C. Barish</a> and <a href=\"http://web.archive.org/web/20180820111639/https://www.nobelprize.org/nobel_prizes/physics/laureates/2017/thorne-facts.html\">Kip S. Thorne</a></h6>\n",
       "<p>“for decisive contributions to the LIGO detector and the observation of gravitational waves”</p>\n",
       "</div>"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This lists all the awardees for Physics in 2017 and the description\n",
    "award_node = award_nodes[6]\n",
    "award_node"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The Nobel Prize in Physics 2017'"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "award_node.select(\"h3\")    # gives array in h3 --- size 1\n",
    "award_node.select(\"h3\")[0] # first object\n",
    "award_node.select(\"h3\")[0].text   # gives the text in it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The Nobel Prize in Physics'"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# strip the year from the title\n",
    "award_node.select(\"h3\")[0].text[:-5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Functions to extract title, year, reason, and recipients"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 455,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_award_title(node):\n",
    "    return node.select(\"h3\")[0].text[:-5]\n",
    "\n",
    "def get_award_year(node):\n",
    "    return node.select(\"h3\")[0].text[-4:]\n",
    "\n",
    "def get_award_motivation(node):\n",
    "    '''check if this section exits'''\n",
    "    if not node.select('p') :\n",
    "        return None\n",
    "    return node.select('p')[0].text\n",
    "\n",
    "def get_recipients(node):\n",
    "    '''check if this section exits'''\n",
    "    return [i.text for i in node.select('h6 a')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get award title and year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Nobel Prize in Physics\n",
      "2017\n"
     ]
    }
   ],
   "source": [
    "print(get_award_title(award_node))\n",
    "print(get_award_year(award_node))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [],
   "source": [
    "# extract all the awards\n",
    "list_awards = []\n",
    "for award_node in award_nodes:\n",
    "    list_awards.append(get_award_title(award_node))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 262,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['The Nobel Prize in Chemistry',\n",
       " 'The Nobel Prize in Physiology or Medicine',\n",
       " 'The Nobel Prize in Literature',\n",
       " 'The Nobel Peace Prize',\n",
       " 'The Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel',\n",
       " 'The Nobel Prize in Physics',\n",
       " 'The Nobel Prize in Chemistry',\n",
       " 'The Nobel Prize in Physiology or Medicine',\n",
       " 'The Nobel Prize in Literature']"
      ]
     },
     "execution_count": 262,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# one line code of the same\n",
    "list_awards = [get_award_title(award_node) for award_node in award_nodes ]\n",
    "list_awards[1:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the recipients\n",
    "- For each division class (year in this case) , **h3** gives the name of the prize, **h6** the list of recipients, and **p** a brief explanation of the work\n",
    "- Note the differences between the methods used below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 380,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<a href=\"http://web.archive.org/web/20180820111639/https://www.nobelprize.org/nobel_prizes/physics/laureates/2017/weiss-facts.html\">Rainer Weiss</a>,\n",
       " <a href=\"http://web.archive.org/web/20180820111639/https://www.nobelprize.org/nobel_prizes/physics/laureates/2017/barish-facts.html\">Barry C. Barish</a>,\n",
       " <a href=\"http://web.archive.org/web/20180820111639/https://www.nobelprize.org/nobel_prizes/physics/laureates/2017/thorne-facts.html\">Kip S. Thorne</a>]"
      ]
     },
     "execution_count": 380,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "award_nodes[6].select('h6 a')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 381,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<h6><a href=\"http://web.archive.org/web/20180820111639/https://www.nobelprize.org/nobel_prizes/physics/laureates/2017/weiss-facts.html\">Rainer Weiss</a>, <a href=\"http://web.archive.org/web/20180820111639/https://www.nobelprize.org/nobel_prizes/physics/laureates/2017/barish-facts.html\">Barry C. Barish</a> and <a href=\"http://web.archive.org/web/20180820111639/https://www.nobelprize.org/nobel_prizes/physics/laureates/2017/thorne-facts.html\">Kip S. Thorne</a></h6>]"
      ]
     },
     "execution_count": 381,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "award_nodes[6].select('h6')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 386,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Rainer Weiss, Barry C. Barish and Kip S. Thorne'"
      ]
     },
     "execution_count": 386,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "award_nodes[6].select('h6')[0].text   # gets all the names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 440,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Rainer Weiss'"
      ]
     },
     "execution_count": 440,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "award_nodes[6].select('h6 a')[0].text  # gets only the first name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 450,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rainer Weiss\n",
      "Barry C. Barish\n",
      "Kip S. Thorne\n"
     ]
    }
   ],
   "source": [
    "for i in award_nodes[6].select('h6 a'):\n",
    "    print(i.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 456,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Rainer Weiss', 'Barry C. Barish', 'Kip S. Thorne']"
      ]
     },
     "execution_count": 456,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recipients(award_nodes[6])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get Links"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['http://web.archive.org/web/20180820111639/https://www.nobelprize.org/nobel_prizes/peace/laureates/1901/',\n",
       " None,\n",
       " 'http://web.archive.org/web/20180820111639/https://www.nobelprize.org/nobel_prizes/peace/laureates/1901/dunant-facts.html',\n",
       " None,\n",
       " 'http://web.archive.org/web/20180820111639/https://www.nobelprize.org/nobel_prizes/peace/laureates/1901/passy-facts.html']"
      ]
     },
     "execution_count": 170,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get the relevant links\n",
    "len(award_nodes)  # 640 \n",
    "recipient_links = [l.get(\"href\") for l in award_nodes[639].select('h6, a')]\n",
    "recipient_links"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the reason for the prize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'“for their discoveries concerning the regulation of cholesterol metabolism”'"
      ]
     },
     "execution_count": 182,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "award_nodes[200].select('p')             # the array\n",
    "award_nodes[200].select('p')[0]          # first element\n",
    "award_nodes[200].select('p')[0].text     # just the text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 231,
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to get the reason\n",
    "def get_award_motivation(node):\n",
    "    if not node.select('p') :\n",
    "        return None\n",
    "    return node.select('p')[0].text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 296,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['“for its work to draw attention to the catastrophic humanitarian consequences of any use of nuclear weapons and for its ground-breaking efforts to achieve a treaty-based prohibition of such weapons”',\n",
       " '“for his contributions to behavioural economics”',\n",
       " '“for theoretical discoveries of topological phase transitions and topological phases of matter”',\n",
       " '“for the design and synthesis of molecular machines”',\n",
       " '“for his discoveries of mechanisms for autophagy”']"
      ]
     },
     "execution_count": 296,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list_motivation = []\n",
    "list_motivation = [get_award_motivation(node) for node in award_nodes]\n",
    "list_motivation[10:15]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Pandas dataframe to store the extracted info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 457,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a set of dictionaries\n",
    "\n",
    "awards = []\n",
    "\n",
    "for node in soup.select(\" .by_year\"):\n",
    "    ''' initialize dictionary to store attributes and values for each year'''\n",
    "    dict = {} \n",
    "    \n",
    "    recipients = get_recipients(node)\n",
    "    \n",
    "    dict['year'] = get_award_year(node)\n",
    "    dict['title'] = get_award_title(node)\n",
    "    dict['recipients'] = recipients\n",
    "    dict['num_recipients'] = len(recipients)\n",
    "    dict['motivation'] = get_award_motivation(node)\n",
    "    \n",
    "    \n",
    "    awards.append(dict)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 485,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>year</th>\n",
       "      <th>title</th>\n",
       "      <th>recipients</th>\n",
       "      <th>num_recipients</th>\n",
       "      <th>motivation</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2018</td>\n",
       "      <td>The Nobel Prize in Physics</td>\n",
       "      <td>[]</td>\n",
       "      <td>0</td>\n",
       "      <td>The 2018 Nobel Prize in Physics has not been a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2018</td>\n",
       "      <td>The Nobel Prize in Chemistry</td>\n",
       "      <td>[]</td>\n",
       "      <td>0</td>\n",
       "      <td>The 2018 Nobel Prize in Chemistry has not been...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2018</td>\n",
       "      <td>The Nobel Prize in Physiology or Medicine</td>\n",
       "      <td>[]</td>\n",
       "      <td>0</td>\n",
       "      <td>The 2018 Nobel Prize in Physiology or Medicine...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2018</td>\n",
       "      <td>The Nobel Prize in Literature</td>\n",
       "      <td>[]</td>\n",
       "      <td>0</td>\n",
       "      <td>The 2018 Nobel Prize in Literature has been po...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2018</td>\n",
       "      <td>The Nobel Peace Prize</td>\n",
       "      <td>[]</td>\n",
       "      <td>0</td>\n",
       "      <td>The 2018 Nobel Peace Prize has not been awarde...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   year                                      title recipients  num_recipients  \\\n",
       "0  2018                 The Nobel Prize in Physics         []               0   \n",
       "1  2018               The Nobel Prize in Chemistry         []               0   \n",
       "2  2018  The Nobel Prize in Physiology or Medicine         []               0   \n",
       "3  2018              The Nobel Prize in Literature         []               0   \n",
       "4  2018                      The Nobel Peace Prize         []               0   \n",
       "\n",
       "                                          motivation  \n",
       "0  The 2018 Nobel Prize in Physics has not been a...  \n",
       "1  The 2018 Nobel Prize in Chemistry has not been...  \n",
       "2  The 2018 Nobel Prize in Physiology or Medicine...  \n",
       "3  The 2018 Nobel Prize in Literature has been po...  \n",
       "4  The 2018 Nobel Peace Prize has not been awarde...  "
      ]
     },
     "execution_count": 485,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create dataframe from the dictionary\n",
    "df_awards = pd.DataFrame(awards)\n",
    "df_awards.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 462,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 640 entries, 0 to 639\n",
      "Data columns (total 5 columns):\n",
      " #   Column          Non-Null Count  Dtype \n",
      "---  ------          --------------  ----- \n",
      " 0   year            640 non-null    object\n",
      " 1   title           640 non-null    object\n",
      " 2   recipients      640 non-null    object\n",
      " 3   num_recipients  640 non-null    int64 \n",
      " 4   motivation      571 non-null    object\n",
      "dtypes: int64(1), object(4)\n",
      "memory usage: 25.1+ KB\n"
     ]
    }
   ],
   "source": [
    "df_awards.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 467,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "69"
      ]
     },
     "execution_count": 467,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# where no motivation present\n",
    "sum(df_awards.motivation.isnull())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 469,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    347\n",
       "2    138\n",
       "3    100\n",
       "0     55\n",
       "Name: num_recipients, dtype: int64"
      ]
     },
     "execution_count": 469,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# number of times there were 0, 1, 2, 3 rcipients\n",
    "df_awards.num_recipients.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example with Bourbon reviews\n",
    "\n",
    "[Distiller.com](https://distiller.com/) is a site with reviews of various drinks. We pick Bourbon by searching it to get this url (https://distiller.com/search?term=bourbon) and then scrape info from it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 490,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'year': '2018',\n",
       " 'title': 'The Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel',\n",
       " 'recipients': [],\n",
       " 'num_recipients': 0,\n",
       " 'motivation': 'The 2018 Prize in Economic Sciences has not been awarded yet. It will be announced on Monday 8 October, 11:45 a.m. at the earliest.'}"
      ]
     },
     "execution_count": 490,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "awards[5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}