scraping-with-htmlparser.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple parsing with HTMLParser\n",
    "\n",
    "In this notebook you will practice one of the workflows for using `HTMLParser` effectively. As you already know, `HTMLParser` is a streaming parser, where data comes in with chunks. Each chunk of data has delimeters like tags. \n",
    "\n",
    "It might feel a bit complicated to have special methods to look at tags, and others to process data - this is one of the caveats of using a streaming parser.\n",
    "\n",
    "For this exercise, you will use predefined HTML variables with raw content that can be parsed. Instead of requesting the data from the web, the content is already defined and available to be processed. The process is the same to scrape the html."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "content = \"\"\"\n",
    "<!DOCTYPE html>\n",
    "<html class=\"client-nojs\" lang=\"en\" dir=\"ltr\">\n",
    "<head>\n",
    "<meta charset=\"UTF-8\"/>\n",
    "<title>1992 World Junior Championships in Athletics – Men's high jump - Wikipedia</title>\n",
    "\"\"\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the data is available, import the html modules so that you can write the class next. The class has to have the `__init__()` method and set some class attributes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from html.parser import HTMLParser\n",
    "\n",
    "class Parser(HTMLParser):\n",
    "\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.recording = False\n",
    "\n",
    "    def handle_starttag(self, tag, attrs):\n",
    "        if tag == \"title\":\n",
    "            self.recording = True\n",
    "        else:\n",
    "            self.recording = False\n",
    "            \n",
    "    def handle_data(self, data):\n",
    "        if self.recording:\n",
    "            print(f\"Found data for tag: {data}\")\n",
    "            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found data for tag: 1992 World Junior Championships in Athletics – Men's high jump - Wikipedia\n",
      "Found data for tag: \n",
      "\n"
     ]
    }
   ],
   "source": [
    "p = Parser()\n",
    "p.feed(content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why is `handled_data()` printing twice? The second line appears to have an _empty_ data. Here is one way to find out: update the `handle_data()` method so that it displays the string with the `repr()` built-in function:\n",
    "\n",
    "```python\n",
    "    def handle_data(self, data):\n",
    "        if self.recording:\n",
    "            print(f\"Found data for tag: {repr(data)}\")\n",
    "```\n",
    "\n",
    "Run the cell where the class lives and re-run the Parser cell again to see if you spot the problem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A string with an empty string var wouldn't show the variable: \n",
      "A string with an empty string var wouldn't show the variable: ''\n"
     ]
    }
   ],
   "source": [
    "# repr() helps when there are hidden characters that `print()` wouldn't show. \n",
    "empty = \"\"\n",
    "print(f\"A string with an empty string var wouldn't show the variable: {empty}\")\n",
    "print(f\"A string with an empty string var wouldn't show the variable: {repr(empty)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Think about what changes could you make to prevent two lines showing in the output. There are several approaches you could take to improve the quality of the data gathering, and the previous cells showed one. But what if you are also dealing with newline characters? Or other non-visible characters? An alternative you could try is to append the data found to a list instead of printing, and when the parsing is completed, joining the data found.\n",
    "Here is how that would look with an example data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1992 World Junior Championships in Athletics – Men's high jump\n",
      "\n",
      "Wikipedia\n"
     ]
    }
   ],
   "source": [
    "captured_data = [\"1992 World Junior Championships in Athletics – Men's high jump\", \"\\n\", \"\\n\", \"Wikipedia\"]\n",
    "print(''.join(captured_data))"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "bd2f83ef6aaf4a37d879bac4506a61ecf83930e56ca6bae2ddb43fbf873d7559"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}