Homework-18-00 - Scraping basics for Selenium-Tomita.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scraping basics for Selenium\n",
    "\n",
    "If you feel comfortable with scraping, you're free to skip this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 0: Imports\n",
    "\n",
    "Import what you need to use Selenium, and start up a new Chrome to use for scraping. You might want to copy from the [Selenium snippets](http://jonathansoma.com/lede/foundations-2018/classes/selenium/selenium-snippets/) page.\n",
    "\n",
    "**You only need to do `driver = webdriver.Chrome(...)` once,** every time you do it you'll open a new Chrome instance. You'll only need to run it again if you close the window (or want another Chrome, for some reason)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import time\n",
    "\n",
    "from selenium import webdriver\n",
    "from selenium.webdriver.common.by import By\n",
    "from webdriver_manager.chrome import ChromeDriverManager"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/m5/5y2tv80j62v6m1f84pdbl2vm0000gn/T/ipykernel_80316/1503906442.py:1: DeprecationWarning: executable_path has been deprecated, please pass in a Service object\n",
      "  driver = webdriver.Chrome(ChromeDriverManager().install())\n"
     ]
    }
   ],
   "source": [
    "driver = webdriver.Chrome(ChromeDriverManager().install())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1: Scraping by class\n",
    "\n",
    "Scrape the content at http://jonathansoma.com/lede/static/by-class.html, printing out the title, subhead, and byline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.get(\"http://jonathansoma.com/lede/static/by-class.html\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<html><head></head><body><h1 class=\"title\">How to Scrape Things</h1>\\n<h3 class=\"subhead\">Some Supplemental Materials</h3>\\n<p class=\"byline\">By Jonathan Soma</p></body></html>'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "driver.page_source"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "title: How to Scrape Things\n",
      "subhead: How to Scrape Things\n",
      "byline: How to Scrape Things\n"
     ]
    }
   ],
   "source": [
    "title = driver.find_element(By.CSS_SELECTOR, \".title\").text\n",
    "subhead = driver.find_element(By.CSS_SELECTOR, \".subhead\").text\n",
    "byline = driver.find_element(By.CSS_SELECTOR, \".byline\").text\n",
    "\n",
    "print(\"title:\", title)\n",
    "print(\"subhead:\", title)\n",
    "print(\"byline:\", title)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2: Scraping using tags\n",
    "\n",
    "Scrape the content at http://jonathansoma.com/lede/static/by-tag.html, printing out the title, subhead, and byline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.get(\"http://jonathansoma.com/lede/static/by-tag.html\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<html><head></head><body><h1>How to Scrape Things</h1>\\n<h3>Some Supplemental Materials</h3>\\n<p>By Jonathan Soma</p></body></html>'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "driver.page_source"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "title: How to Scrape Things\n",
      "subhead: How to Scrape Things\n",
      "byline: How to Scrape Things\n"
     ]
    }
   ],
   "source": [
    "title = driver.find_element(By.CSS_SELECTOR, \"h1\").text\n",
    "subhead = driver.find_element(By.CSS_SELECTOR, \"h3\").text\n",
    "byline = driver.find_element(By.CSS_SELECTOR, \"p\").text\n",
    "\n",
    "print(\"title:\", title)\n",
    "print(\"subhead:\", title)\n",
    "print(\"byline:\", title)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 3: Scraping using a single tag\n",
    "\n",
    "Scrape the content at http://jonathansoma.com/lede/static/by-list.html, printing out the title, subhead, and byline.\n",
    "\n",
    "> **This will be important for the next few:** if you scrape multiples, you have a list. Even though it's Seleninum, you can use things like `[0]`, `[1]`, `[-1]` etc just like you would for a normal list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.get(\"http://jonathansoma.com/lede/static/by-list.html\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<html><head></head><body><p>How to Scrape Things</p>\\n<p>Some Supplemental Materials</p>\\n<p>By Jonathan Soma</p></body></html>'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "driver.page_source"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "title: How to Scrape Things\n",
      "subhead: Some Supplemental Materials\n",
      "byline: By Jonathan Soma\n"
     ]
    }
   ],
   "source": [
    "tags = driver.find_elements(By.CSS_SELECTOR, \"p\")\n",
    "texts = [tag.text for tag in tags]\n",
    "\n",
    "print(\"title:\", texts[0])\n",
    "print(\"subhead:\", texts[1])\n",
    "print(\"byline:\", texts[-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 4: Scraping a single table row\n",
    "\n",
    "Scrape the content at http://jonathansoma.com/lede/static/single-table-row.html, printing out the title, subhead, and byline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.get(\"http://jonathansoma.com/lede/static/single-table-row.html\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<html><head></head><body><table>\\n  <tbody><tr>\\n    <td>How to Scrape Things</td>\\n    <td>Some Supplemental Materials</td>\\n    <td>By Jonathan Soma</td>\\n  </tr>\\n</tbody></table></body></html>'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "driver.page_source"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "title: How to Scrape Things\n",
      "subhead: Some Supplemental Materials\n",
      "byline: By Jonathan Soma\n"
     ]
    }
   ],
   "source": [
    "elements = driver.find_elements(By.CSS_SELECTOR, \"td\")\n",
    "texts = [element.text for element in elements]\n",
    "\n",
    "print(\"title:\", texts[0])\n",
    "print(\"subhead:\", texts[1])\n",
    "print(\"byline:\", texts[-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 5: Saving into a dictionary\n",
    "\n",
    "Scrape the content at http://jonathansoma.com/lede/static/single-table-row.html, saving the title, subhead, and byline into a single dictionary called `book`.\n",
    "\n",
    "> Don't use pandas for this one!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "items = [\"title\", \"subhead\", \"byline\"]\n",
    "\n",
    "book = dict(zip(items, texts))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'title': 'How to Scrape Things',\n",
       " 'subhead': 'Some Supplemental Materials',\n",
       " 'byline': 'By Jonathan Soma'}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "book"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 6: Scraping multiple table rows\n",
    "\n",
    "Scrape the content at http://jonathansoma.com/lede/static/multiple-table-rows.html, printing out each title, subhead, and byline.\n",
    "\n",
    "> You won't use pandas for this one, either!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.get(\"https://jonathansoma.com/lede/static/multiple-table-rows.html\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"<html><head></head><body><table>\\n  <tbody><tr>\\n    <td>How to Scrape Things</td>\\n    <td>Some Supplemental Materials</td>\\n    <td>By Jonathan Soma</td>\\n  </tr>\\n  <tr>\\n    <td>How to Scrape Many Things</td>\\n    <td>But, Is It Even Possible?</td>\\n    <td>By Sonathan Joma</td>\\n  </tr>\\n  <tr>\\n    <td>The End of Scraping</td>\\n    <td>Let's All Use CSV Files</td>\\n    <td>By Amos Nathanos</td>\\n  </tr>\\n</tbody></table></body></html>\""
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "driver.page_source"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "title: How to Scrape Things\n",
      "subhead: Some Supplemental Materials\n",
      "byline: By Jonathan Soma\n",
      "------\n",
      "title: How to Scrape Many Things\n",
      "subhead: But, Is It Even Possible?\n",
      "byline: By Sonathan Joma\n",
      "------\n",
      "title: The End of Scraping\n",
      "subhead: Let's All Use CSV Files\n",
      "byline: By Amos Nathanos\n",
      "------\n"
     ]
    }
   ],
   "source": [
    "items = [\"title\", \"subhead\", \"byline\"]\n",
    "\n",
    "rows = driver.find_elements(By.CSS_SELECTOR, \"tr\")\n",
    "\n",
    "for row in rows:\n",
    "    elements = row.find_elements(By.CSS_SELECTOR, \"td\")\n",
    "    texts = [element.text for element in elements]\n",
    "    book = dict(zip(items, texts))\n",
    "    for key, value in book.items():\n",
    "        print(f\"{key}: {value}\")\n",
    "    print('------')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 7: Scraping an actual table\n",
    "\n",
    "Scrape the content at http://jonathansoma.com/lede/static/the-actual-table.html, creating a list of dictionaries.\n",
    "\n",
    "> Don't use pandas here, either!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.get(\"https://jonathansoma.com/lede/static/the-actual-table.html\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<html><head></head><body><table id=\"booklist\">\\n  <tbody><tr>\\n    <td>How to Scrape Things</td>\\n    <td>Some Supplemental Materials</td>\\n    <td>By Jonathan Soma</td>\\n  </tr>\\n  <tr>\\n    <td>How to Scrape Many Things</td>\\n    <td>But, Is It Even Possible?</td>\\n    <td>By Sonathan Joma</td>\\n  </tr>\\n  <tr>\\n    <td>The End of Scraping</td>\\n    <td>Let\\'s All Use CSV Files</td>\\n    <td>By Amos Nathanos</td>\\n  </tr>\\n</tbody></table></body></html>'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "driver.page_source"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'title': 'How to Scrape Things',\n",
       "  'subhead': 'Some Supplemental Materials',\n",
       "  'byline': 'By Jonathan Soma'},\n",
       " {'title': 'How to Scrape Many Things',\n",
       "  'subhead': 'But, Is It Even Possible?',\n",
       "  'byline': 'By Sonathan Joma'},\n",
       " {'title': 'The End of Scraping',\n",
       "  'subhead': \"Let's All Use CSV Files\",\n",
       "  'byline': 'By Amos Nathanos'}]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "items = [\"title\", \"subhead\", \"byline\"]\n",
    "books = []\n",
    "\n",
    "rows = driver.find_elements(By.CSS_SELECTOR, \"tr\")\n",
    "\n",
    "for row in rows:\n",
    "    elements = row.find_elements(By.CSS_SELECTOR, \"td\")\n",
    "    texts = [element.text for element in elements]\n",
    "    book = dict(zip(items, texts))\n",
    "    books.append(book)\n",
    "    \n",
    "books"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 8: Scraping multiple table rows into a list of dictionaries\n",
    "\n",
    "Scrape the content at http://jonathansoma.com/lede/static/the-actual-table.html, creating a pandas DataFrame.\n",
    "\n",
    "> There are two ways to do this one! One uses just pandas, the other one uses the result from Part 7."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>subhead</th>\n",
       "      <th>byline</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>How to Scrape Things</td>\n",
       "      <td>Some Supplemental Materials</td>\n",
       "      <td>By Jonathan Soma</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>How to Scrape Many Things</td>\n",
       "      <td>But, Is It Even Possible?</td>\n",
       "      <td>By Sonathan Joma</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>The End of Scraping</td>\n",
       "      <td>Let's All Use CSV Files</td>\n",
       "      <td>By Amos Nathanos</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       title                      subhead            byline\n",
       "0       How to Scrape Things  Some Supplemental Materials  By Jonathan Soma\n",
       "1  How to Scrape Many Things    But, Is It Even Possible?  By Sonathan Joma\n",
       "2        The End of Scraping      Let's All Use CSV Files  By Amos Nathanos"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(books)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 9: Scraping into a file\n",
    "\n",
    "Scrape the content at http://jonathansoma.com/lede/static/the-actual-table.html and save it as `output.csv`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv(\"output.csv\", index=False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv310",
   "language": "python",
   "name": "venv310"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}