URL Scraper.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# External Libraries\n",
    "import pandas as pd\n",
    "from recipe_scrapers import scrape_me\n",
    "import time\n",
    "import random\n",
    "import json\n",
    "import pymongo\n",
    "from tqdm.notebook import tqdm\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Scripts\n",
    "from scripts import scrapers, parsers, db_funcs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Db Information\n",
    "urls, recipes = db_funcs.get_scraper_dbs()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scraper\n",
    "Take the url database and start retrieving the actual recipe contents to store in a new database dedicated to the recipes. Given that this database will form the basis for an actual app and analysis the key unique id will be generated by Mongo and keep close tabs with the url database to ensure no read recipes are reread."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Retriever Workflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "code_folding": [],
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No URL's to Read.\n"
     ]
    }
   ],
   "source": [
    "scrapers.scrape_unread_urls(urls, recipes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Collector Workflow\n",
    "Below is a set of crawlers dedicated to different recipe sources."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Formulated Flavors\n",
    "crawler = scrapers.RecipeCollector(\"https://formulatedflavors.com/\", \n",
    "                          \"formulatedflavors.com\", \n",
    "                          \"formulated_flavors\", \n",
    "                          recipes, \n",
    "                          source_ref,\n",
    "                          utm_pages=True)\n",
    "crawler.recursive_page_scrape(\"https://formulatedflavors.com/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "code_folding": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recipes Found: 1620 | Scraping https://sallysbakingaddiction.com/\r"
     ]
    }
   ],
   "source": [
    "# Sallys Baking Addiction\n",
    "crawler = scrapers.RecipeCollector(\"https://sallysbakingaddiction.com/\", \n",
    "                          \"sallysbakingaddiction.com\", \n",
    "                          \"sallys_baking_addiction\", \n",
    "                          recipes, \n",
    "                          source_ref,\n",
    "                          utm_pages=True)\n",
    "\n",
    "# Crawling will resume more or less where it left off\n",
    "crawler.recursive_page_scrape(\"https://sallysbakingaddiction.com/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "code_folding": [
     0
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recipes Found: 542 | Scraping https://realsimplegood.com/wprm_print/recipe/27858                                                                                                                        \r"
     ]
    }
   ],
   "source": [
    "# Real Simple\n",
    "crawler = scrapers.RecipeCollector(\"https://realsimplegood.com/\",\n",
    "                          \"realsimplegood.com\",\n",
    "                          \"real_simple_good\",\n",
    "                          recipes,\n",
    "                          source_ref,\n",
    "                          utm_pages=True)\n",
    "\n",
    "# Crawling will resume more or less where it left off\n",
    "crawler.recursive_page_scrape(\"https://realsimplegood.com/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "code_folding": [
     0
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recipes Found: 8238 | Scraping https://pinchofyum.com                                                                                                                                                   \r"
     ]
    }
   ],
   "source": [
    "# Pinch of Yum\n",
    "crawler = scrapers.RecipeCollector(\"https://pinchofyum.com\",\n",
    "                          \"pinchofyum.com\",\n",
    "                          \"pinch_of_yum\",\n",
    "                          recipes,\n",
    "                          source_ref,\n",
    "                          utm_pages=True)\n",
    "\n",
    "# Crawling will resume more or less where it left off\n",
    "crawler.recursive_page_scrape(\"https://pinchofyum.com\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "code_folding": [
     0
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recipes Found: 1723 | Scraping https://www.twopeasandtheirpod.com/recipes/?fwp_paged=146                                                                                                                \r"
     ]
    }
   ],
   "source": [
    "# Two Peas and their pod\n",
    "crawler = scrapers.RecipeCollector(\"https://www.twopeasandtheirpod.com\",\n",
    "                          \"https://www.twopeasandtheirpod.com\",\n",
    "                          \"two_peas_and_their_pod\",\n",
    "                          recipes,\n",
    "                          source_ref,\n",
    "                          utm_pages=True)\n",
    "\n",
    "# Crawling will resume more or less where it left off\n",
    "pages = [\"https://www.twopeasandtheirpod.com/recipes/?fwp_paged=\"+str(i) for i in range(2, 147)]\n",
    "for page in pages:\n",
    "   # Crawling will resume more or less where it left off\n",
    "    crawler.recursive_page_scrape(page) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "code_folding": [
     0
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recipes Found: 346 | Scraping https://www.foxandbriar.com/page/46/                                                                                                                                      \r"
     ]
    }
   ],
   "source": [
    "# Fox and Briar\n",
    "crawler = scrapers.RecipeCollector(\"https://www.foxandbriar.com/\",\n",
    "                          \"https://www.foxandbriar.com/\",\n",
    "                          \"fox_and_briar\",\n",
    "                          recipes,\n",
    "                          source_ref,\n",
    "                          utm_pages=True)\n",
    "\n",
    "# Crawling will resume more or less where it left off\n",
    "crawler.recursive_page_scrape(\"https://www.foxandbriar.com/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "code_folding": [
     0
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recipes Found: 301 | Scraping https://saltandbaker.com/page/29/                                                                                                                                         \r"
     ]
    }
   ],
   "source": [
    "# Salt and Baker\n",
    "crawler = scrapers.RecipeCollector(\"https://saltandbaker.com/\",\n",
    "                          \"https://saltandbaker.com/\",\n",
    "                          \"salt_and_baker\",\n",
    "                          recipes,\n",
    "                          source_ref,\n",
    "                          utm_pages=False)\n",
    "\n",
    "# Crawling will resume more or less where it left off\n",
    "crawler.recursive_page_scrape(\"https://saltandbaker.com/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "code_folding": [
     0
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recipes Found: 271 | Scraping https://thecleaneatingcouple.com/category/recipes/breakfast/page/13/                                                                                                      \r"
     ]
    }
   ],
   "source": [
    "# Clean Eating Couple\n",
    "crawler = scrapers.RecipeCollector(\"https://thecleaneatingcouple.com/\",\n",
    "                          \"https://thecleaneatingcouple.com/\",\n",
    "                          \"clean_eating_couple\",\n",
    "                          recipes,\n",
    "                          source_ref,\n",
    "                          utm_pages=False)\n",
    "\n",
    "# Crawling will resume more or less where it left off\n",
    "crawler.recursive_page_scrape(\"https://thecleaneatingcouple.com/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "code_folding": [
     0
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recipes Found: 1468 | Scraping https://dinnerthendessert.com/recent-recipes/page/2                                                                                                                      \r"
     ]
    }
   ],
   "source": [
    "# Dinner then Dessert\n",
    "crawler = scrapers.RecipeCollector(\"https://dinnerthendessert.com/\",\n",
    "                          \"https://dinnerthendessert.com/\",\n",
    "                          \"dinner_then_dessert\",\n",
    "                          recipes,\n",
    "                          source_ref,\n",
    "                          utm_pages=False)\n",
    "\n",
    "# Crawling will resume more or less where it left off\n",
    "crawler.recursive_page_scrape(\"https://dinnerthendessert.com/recipe-index\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "code_folding": [
     0
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recipes Found: 499 | Scraping https://thesaltymarshmallow.com/category/dinner/page/20/                                                                                                                  \r"
     ]
    }
   ],
   "source": [
    "# Salty Marshmellow\n",
    "crawler = scrapers.RecipeCollector(\"https://thesaltymarshmallow.com/\",\n",
    "                          \"https://thesaltymarshmallow.com/\",\n",
    "                          \"the_salty_marshmellow\",\n",
    "                          recipes,\n",
    "                          source_ref,\n",
    "                          utm_pages=False)\n",
    "\n",
    "# Crawling will resume more or less where it left off\n",
    "crawler.recursive_page_scrape(\"https://thesaltymarshmallow.com/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "code_folding": [
     0
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recipes Found: 145 | Scraping https://www.afarmgirlsdabbles.com/downloads/MerryChristmasTags.pdf                                                                                                        \r"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some characters could not be decoded, and were replaced with REPLACEMENT CHARACTER.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recipes Found: 268 | Scraping https://www.afarmgirlsdabbles.com/downloads/FourthOfJulyTags.pdf                                                                                                          \r"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some characters could not be decoded, and were replaced with REPLACEMENT CHARACTER.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recipes Found: 407 | Scraping https://www.afarmgirlsdabbles.com/tag/zucchini/                                                                                                                           \r"
     ]
    }
   ],
   "source": [
    "# A Farm Girl Dabbles\n",
    "crawler = scrapers.RecipeCollector(\"https://www.afarmgirlsdabbles.com/\",\n",
    "                          \"https://www.afarmgirlsdabbles.com/\",\n",
    "                          \"a_farm_girl_dabbles\",\n",
    "                          recipes,\n",
    "                          source_ref,\n",
    "                          utm_pages=False)\n",
    "\n",
    "# Crawling will resume more or less where it left off\n",
    "crawler.recursive_page_scrape(\"https://www.afarmgirlsdabbles.com/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "code_folding": [
     0
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recipes Found: 17 | Scraping https://cakescottage.com/                                                                                                                                                  \r"
     ]
    }
   ],
   "source": [
    "# Cakes Cottage\n",
    "crawler = scrapers.RecipeCollector(\"https://cakescottage.com/\",\n",
    "                          \"https://cakescottage.com/\",\n",
    "                          \"cakes_cottage\",\n",
    "                          recipes,\n",
    "                          source_ref,\n",
    "                          utm_pages=False)\n",
    "\n",
    "# Crawling will resume more or less where it left off\n",
    "crawler.recursive_page_scrape(\"https://cakescottage.com/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "code_folding": [
     0
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recipes Found: 874 | Scraping https://www.yourcupofcake.com/category/yeast-breads                                                                                                                       entines-dayhttps:/www.yourcupofcake.com/category/holiday/valentines-dayhttps:/www.yourcupofcake.com/category/holiday/valentines-day/page/3/\r"
     ]
    }
   ],
   "source": [
    "# Your Cup of Cake\n",
    "crawler = scrapers.RecipeCollector(\"https://www.yourcupofcake.com/\",\n",
    "                          \"https://www.yourcupofcake.com/\",\n",
    "                          \"your_cup_of_cake\",\n",
    "                          recipes,\n",
    "                          source_ref,\n",
    "                          utm_pages=False)\n",
    "\n",
    "# Crawling will resume more or less where it left off\n",
    "crawler.recursive_page_scrape(\"https://www.yourcupofcake.com/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "code_folding": [
     0
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recipes Found: 535 | Scraping https://www.howsweeteats.com/                                                                                                                                             \r"
     ]
    }
   ],
   "source": [
    "# How Sweet Eats\n",
    "crawler = scrapers.RecipeCollector(\"https://www.howsweeteats.com/\",\n",
    "                          \"https://www.howsweeteats.com/\",\n",
    "                          \"how_sweet_eats\",\n",
    "                          recipes,\n",
    "                          source_ref,\n",
    "                          utm_pages=False)\n",
    "\n",
    "# Crawling will resume more or less where it left off\n",
    "crawler.recursive_page_scrape(\"https://www.howsweeteats.com/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [
     0
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recipes Found: 278 | Scraping https://bakerbynature.com/nutella-cinnamon-sugar-doughnuts/                                                                                                               \r"
     ]
    }
   ],
   "source": [
    "# Baker by Nature\n",
    "crawler = scrapers.RecipeCollector(\"https://bakerbynature.com/\",\n",
    "                          \"https://bakerbynature.com/\",\n",
    "                          \"baker_by_nature\",\n",
    "                          recipes,\n",
    "                          source_ref,\n",
    "                          utm_pages=False)\n",
    "\n",
    "# Crawling will resume more or less where it left off\n",
    "crawler.recursive_page_scrape(\"https://bakerbynature.com/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [
     0
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recipes Found: 19 | Scraping https://www.foxandbriar.com/slow-cooker-beef-burrito-bowls/                                                                                                                \r"
     ]
    }
   ],
   "source": [
    "# Feels Like Home Blog\n",
    "crawler = scrapers.RecipeCollector(\"https://feelslikehomeblog.com/category/home-cooking\",\n",
    "                          \"https://feelslikehomeblog.com/\",\n",
    "                          \"feels_like_home_blog\",\n",
    "                          recipes,\n",
    "                          source_ref,\n",
    "                          utm_pages=False)\n",
    "\n",
    "# Crawling will resume more or less where it left off\n",
    "crawler.recursive_page_scrape(\"https://feelslikehomeblog.com/category/home-cooking\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "code_folding": [
     0
    ],
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recipes Found: 222 | Scraping https://carlsbadcravings.com/slow-cooker-salsa-verde-honey-lime-chicken-recipe/                                                                                           \r"
     ]
    },
    {
     "ename": "JSONDecodeError",
     "evalue": "Expecting ',' delimiter: line 1 column 2109 (char 2108)",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mJSONDecodeError\u001b[0m                           Traceback (most recent call last)",
      "\u001b[1;32mc:\\users\\atawz\\appdata\\local\\programs\\python\\python37\\lib\\site-packages\\extruct\\jsonld.py\u001b[0m in \u001b[0;36m_extract_items\u001b[1;34m(self, node)\u001b[0m\n\u001b[0;32m     33\u001b[0m             \u001b[1;31m# TODO: `strict=False` can be configurable if needed\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 34\u001b[1;33m             \u001b[0mdata\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mjson\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mloads\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mscript\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mstrict\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mFalse\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     35\u001b[0m         \u001b[1;32mexcept\u001b[0m \u001b[0mValueError\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\users\\atawz\\appdata\\local\\programs\\python\\python37\\lib\\json\\__init__.py\u001b[0m in \u001b[0;36mloads\u001b[1;34m(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)\u001b[0m\n\u001b[0;32m    360\u001b[0m         \u001b[0mkw\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'parse_constant'\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mparse_constant\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 361\u001b[1;33m     \u001b[1;32mreturn\u001b[0m \u001b[0mcls\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdecode\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0ms\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32mc:\\users\\atawz\\appdata\\local\\programs\\python\\python37\\lib\\json\\decoder.py\u001b[0m in \u001b[0;36mdecode\u001b[1;34m(self, s, _w)\u001b[0m\n\u001b[0;32m    336\u001b[0m         \"\"\"\n\u001b[1;32m--> 337\u001b[1;33m         \u001b[0mobj\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mend\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mraw_decode\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0ms\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0midx\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0m_w\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0ms\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;36m0\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    338\u001b[0m         \u001b[0mend\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0m_w\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0ms\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mend\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\users\\atawz\\appdata\\local\\programs\\python\\python37\\lib\\json\\decoder.py\u001b[0m in \u001b[0;36mraw_decode\u001b[1;34m(self, s, idx)\u001b[0m\n\u001b[0;32m    352\u001b[0m         \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 353\u001b[1;33m             \u001b[0mobj\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mend\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mscan_once\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0ms\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0midx\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    354\u001b[0m         \u001b[1;32mexcept\u001b[0m \u001b[0mStopIteration\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0merr\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mJSONDecodeError\u001b[0m: Expecting ',' delimiter: line 1 column 2109 (char 2108)",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mJSONDecodeError\u001b[0m                           Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-8-26d16c9ac0a1>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      8\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      9\u001b[0m \u001b[1;31m# Crawling will resume more or less where it left off\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 10\u001b[1;33m \u001b[0mcrawler\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrecursive_page_scrape\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"https://carlsbadcravings.com/\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32m<ipython-input-4-bb004e8045bb>\u001b[0m in \u001b[0;36mrecursive_page_scrape\u001b[1;34m(self, page)\u001b[0m\n\u001b[0;32m     53\u001b[0m                 \u001b[1;32mif\u001b[0m \u001b[0mlink\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mlink_library\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mkeys\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     54\u001b[0m                     \u001b[0mwait\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 55\u001b[1;33m                     \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mrecursive_page_scrape\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mlink\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     56\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     57\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mscrape_recipe\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mlink_data\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mpage\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m<ipython-input-4-bb004e8045bb>\u001b[0m in \u001b[0;36mrecursive_page_scrape\u001b[1;34m(self, page)\u001b[0m\n\u001b[0;32m     42\u001b[0m         \u001b[0msoup\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mBeautifulSoup\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mr\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcontent\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;34m\"html.parser\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     43\u001b[0m         \u001b[0mbase_url\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mget_base_url\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mr\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtext\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mr\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0murl\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 44\u001b[1;33m         \u001b[0mdata\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mextruct\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mextract\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mr\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtext\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mbase_url\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mbase_url\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     45\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     46\u001b[0m         \u001b[1;31m# Recipe\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\users\\atawz\\appdata\\local\\programs\\python\\python37\\lib\\site-packages\\extruct\\_extruct.py\u001b[0m in \u001b[0;36mextract\u001b[1;34m(htmlstring, base_url, encoding, syntaxes, errors, uniform, return_html_node, schema_context, **kwargs)\u001b[0m\n\u001b[0;32m     99\u001b[0m     \u001b[1;32mfor\u001b[0m \u001b[0msyntax\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mextract\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdocument\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mprocessors\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    100\u001b[0m         \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 101\u001b[1;33m             \u001b[0moutput\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0msyntax\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mlist\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mextract\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdocument\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mbase_url\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mbase_url\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    102\u001b[0m         \u001b[1;32mexcept\u001b[0m \u001b[0mException\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    103\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[0merrors\u001b[0m \u001b[1;33m==\u001b[0m \u001b[1;34m'log'\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\users\\atawz\\appdata\\local\\programs\\python\\python37\\lib\\site-packages\\extruct\\jsonld.py\u001b[0m in \u001b[0;36mextract_items\u001b[1;34m(self, document, base_url)\u001b[0m\n\u001b[0;32m     24\u001b[0m         return [\n\u001b[0;32m     25\u001b[0m             \u001b[0mitem\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 26\u001b[1;33m             \u001b[1;32mfor\u001b[0m \u001b[0mitems\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mmap\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_extract_items\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_xp_jsonld\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdocument\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     27\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[0mitems\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mitem\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mitems\u001b[0m \u001b[1;32mif\u001b[0m \u001b[0mitem\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     28\u001b[0m         ]\n",
      "\u001b[1;32mc:\\users\\atawz\\appdata\\local\\programs\\python\\python37\\lib\\site-packages\\extruct\\jsonld.py\u001b[0m in \u001b[0;36m<listcomp>\u001b[1;34m(.0)\u001b[0m\n\u001b[0;32m     23\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mextract_items\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdocument\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mbase_url\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mNone\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     24\u001b[0m         return [\n\u001b[1;32m---> 25\u001b[1;33m             \u001b[0mitem\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     26\u001b[0m             \u001b[1;32mfor\u001b[0m \u001b[0mitems\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mmap\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_extract_items\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_xp_jsonld\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdocument\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     27\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[0mitems\u001b[0m \u001b[1;32mfor\u001b[0m \u001b[0mitem\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mitems\u001b[0m \u001b[1;32mif\u001b[0m \u001b[0mitem\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\users\\atawz\\appdata\\local\\programs\\python\\python37\\lib\\site-packages\\extruct\\jsonld.py\u001b[0m in \u001b[0;36m_extract_items\u001b[1;34m(self, node)\u001b[0m\n\u001b[0;32m     36\u001b[0m             \u001b[1;31m# sometimes JSON-decoding errors are due to leading HTML or JavaScript comments\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     37\u001b[0m             data = json.loads(\n\u001b[1;32m---> 38\u001b[1;33m                 HTML_OR_JS_COMMENTLINE.sub('', script), strict=False)\n\u001b[0m\u001b[0;32m     39\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mlist\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     40\u001b[0m             \u001b[1;32mreturn\u001b[0m \u001b[0mdata\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\users\\atawz\\appdata\\local\\programs\\python\\python37\\lib\\json\\__init__.py\u001b[0m in \u001b[0;36mloads\u001b[1;34m(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)\u001b[0m\n\u001b[0;32m    359\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[0mparse_constant\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    360\u001b[0m         \u001b[0mkw\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'parse_constant'\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mparse_constant\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 361\u001b[1;33m     \u001b[1;32mreturn\u001b[0m \u001b[0mcls\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m**\u001b[0m\u001b[0mkw\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdecode\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0ms\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32mc:\\users\\atawz\\appdata\\local\\programs\\python\\python37\\lib\\json\\decoder.py\u001b[0m in \u001b[0;36mdecode\u001b[1;34m(self, s, _w)\u001b[0m\n\u001b[0;32m    335\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    336\u001b[0m         \"\"\"\n\u001b[1;32m--> 337\u001b[1;33m         \u001b[0mobj\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mend\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mraw_decode\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0ms\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0midx\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0m_w\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0ms\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;36m0\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    338\u001b[0m         \u001b[0mend\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0m_w\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0ms\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mend\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mend\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    339\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mend\u001b[0m \u001b[1;33m!=\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0ms\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mc:\\users\\atawz\\appdata\\local\\programs\\python\\python37\\lib\\json\\decoder.py\u001b[0m in \u001b[0;36mraw_decode\u001b[1;34m(self, s, idx)\u001b[0m\n\u001b[0;32m    351\u001b[0m         \"\"\"\n\u001b[0;32m    352\u001b[0m         \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 353\u001b[1;33m             \u001b[0mobj\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mend\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mscan_once\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0ms\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0midx\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    354\u001b[0m         \u001b[1;32mexcept\u001b[0m \u001b[0mStopIteration\u001b[0m \u001b[1;32mas\u001b[0m \u001b[0merr\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    355\u001b[0m             \u001b[1;32mraise\u001b[0m \u001b[0mJSONDecodeError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"Expecting value\"\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0ms\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0merr\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mvalue\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;32mfrom\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mJSONDecodeError\u001b[0m: Expecting ',' delimiter: line 1 column 2109 (char 2108)"
     ]
    }
   ],
   "source": [
    "# Carlsbad Cravings\n",
    "crawler = scrapers.RecipeCollector(\"https://carlsbadcravings.com/\",\n",
    "                          \"https://carlsbadcravings.com/\",\n",
    "                          \"carlsbad_cravings\",\n",
    "                          recipes,\n",
    "                          source_ref,\n",
    "                          utm_pages=False)\n",
    "\n",
    "# Crawling will resume more or less where it left off\n",
    "crawler.recursive_page_scrape(\"https://carlsbadcravings.com/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [
     0
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Recipes Found: 19 | Scraping https://www.foxandbriar.com/slow-cooker-beef-burrito-bowls/                                                                                                                \r"
     ]
    }
   ],
   "source": [
    "# Kitchn\n",
    "crawler = scrapers.RecipeCollector(\"https://www.thekitchn.com/sitemap\",\n",
    "                          \"https://www.thekitchn.com\",\n",
    "                          \"the_kitchn\",\n",
    "                          recipes,\n",
    "                          source_ref,\n",
    "                          utm_pages=True)\n",
    "\n",
    "# Crawling will resume more or less where it left off\n",
    "crawler.recursive_page_scrape(\"https://www.thekitchn.com/sitemap\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Database Cleaning and Maintenance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "URL Database: 226844\n",
      "Read: 226825\n",
      "To Read: 19\n",
      "Errors: 105\n",
      "Recipes Database: 235817\n"
     ]
    }
   ],
   "source": [
    "# Print Overview\n",
    "db_funcs.print_db_metadata(urls, recipes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Duplicate Cleaning\n",
    "At this stage should be true duplicates, if urls are found more than once it keeps first, which is arbitrary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "db_funcs.recipes_clean_duplicates(recipes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Marking Read\n",
    "Depending on certai issues some documents don't get marked as read. If they are in recipes then they should be updated and this quick fix will clean that up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5bc34ab5f7254d85a2adc9b3393d327c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=29245.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "# Mark everything as read\n",
    "db_funcs.clean_read_recipes_urls()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}