01_scrape_yellowstoneai_images.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3363dd30",
   "metadata": {},
   "source": [
    "# Scraping our Yellowstone.ai trailcam images"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed10b5bc",
   "metadata": {},
   "source": [
    "The goal of the scraper is to create a textfile of image links which we'll download using the widely used `requests` library. We have to do this because yellowstone.ai doesn't offer a way to download images in bulk.\n",
    "\n",
    "ChromeDriver is a browser that we can control with scripting. We'll use the Selenium API to \"drive\" it – aka issue browser commands such as \"click here\", \"enter my email address there\", and importantly, \"copy that image link\".\n",
    "\n",
    "We'll build our scraper by issuing commands one-at-a-time in a trial and error fashion until we're confident that we have all the pieces to put them together and create the final scraper."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16f466ce",
   "metadata": {},
   "source": [
    "# Building the scraper step-by-step"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d012fb6",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a258651f",
   "metadata": {},
   "source": [
    "Go `https://chromedriver.chromium.org/downloads` and download the version of ChromeDriver that matches your Chrome version. Move it to your project's root folder. Right-click + open the downloaded file to let your OS know that it's safe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b3e1413c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-29T01:03:18.602603Z",
     "start_time": "2021-12-29T01:03:18.599933Z"
    }
   },
   "outputs": [],
   "source": [
    "# !pip install selenium"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efcd881c",
   "metadata": {},
   "source": [
    "Verify it works by importing Selenium's webdriver and creating a driver – it should open an empty browser window. Note that importing Options and manually setting options is only required when running this on linux machines (useful for running on servers)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "23be3814",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-29T01:03:19.281447Z",
     "start_time": "2021-12-29T01:03:18.603941Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-2-870fad7f7734>:7: DeprecationWarning: executable_path has been deprecated, please pass in a Service object\n",
      "  driver = webdriver.Chrome('/usr/local/bin/chromedriver', options=chrome_options)\n"
     ]
    }
   ],
   "source": [
    "from selenium import webdriver\n",
    "from selenium.webdriver.chrome.options import Options\n",
    "\n",
    "chrome_options = Options()\n",
    "chrome_options.add_argument('--headless')\n",
    "\n",
    "driver = webdriver.Chrome('/usr/local/bin/chromedriver', options=chrome_options)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0fc25471",
   "metadata": {},
   "source": [
    "Selenium is the library we use to give ChromeDriver commands. We'll also use the time library to issue `sleep` commands so that we can wait for Yellowstone's website updates that take place in response to our commands. We'll use the `csv` library to save the results, which will be a list of image urls."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e2eab3c3",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-29T01:03:19.286647Z",
     "start_time": "2021-12-29T01:03:19.282944Z"
    }
   },
   "outputs": [],
   "source": [
    "from time import sleep\n",
    "import csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2dbe8a97",
   "metadata": {},
   "source": [
    "## Login"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3947de74",
   "metadata": {},
   "source": [
    "Now we'll start issuing browser commands. Before we start issuing web driver commands, let's store our yellowstone.ai email and password. I keep mine in a text file called `creds.txt`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f3c592ef",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-29T01:03:19.292236Z",
     "start_time": "2021-12-29T01:03:19.287819Z"
    }
   },
   "outputs": [],
   "source": [
    "file = open('creds.txt', \"r\")\n",
    "my_email, my_password = file.read().split(\"\\n\")\n",
    "file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f551642",
   "metadata": {},
   "source": [
    "Now we can start issuing commands to the web driver. bring up the page that contains the images and wait one second for the page to load. After running, we should now be looking at an email/password login page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "ed157e11",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-29T01:03:20.760606Z",
     "start_time": "2021-12-29T01:03:19.292980Z"
    }
   },
   "outputs": [],
   "source": [
    "driver.get(\"https://my.yellowstone.ai/media\")\n",
    "sleep(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "793e504f",
   "metadata": {},
   "source": [
    "We need to identify the email input html element and the password input element. Ideally, we would like to use an `id` tag b/c those are guaranteed to be unique. A `name` isn't always unique, but that may work. `class` is a last choice.\n",
    "\n",
    "This page has `id`s for the two forms we want to grab, so we'll use `driver.find_element_by_id` to grab those elements.\n",
    "\n",
    "(Tip for using non-id attributes: if multiple objects match the search, the first on the page will be returned.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "af142781",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-29T01:03:20.775708Z",
     "start_time": "2021-12-29T01:03:20.762085Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-6-9707a9d4b7d5>:1: DeprecationWarning: find_element_by_* commands are deprecated. Please use find_element() instead\n",
      "  email_input = driver.find_element_by_id(\"email\")\n"
     ]
    }
   ],
   "source": [
    "email_input = driver.find_element_by_id(\"email\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "866e4b98",
   "metadata": {},
   "source": [
    "Now we can use the `send_keys` method of this search result to send our email."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "5e0a9c03",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-29T01:03:20.815283Z",
     "start_time": "2021-12-29T01:03:20.777258Z"
    }
   },
   "outputs": [],
   "source": [
    "email_input.send_keys(my_email)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cc77935",
   "metadata": {},
   "source": [
    "We'll do the same thing for the password."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "bed334e5",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-29T01:03:20.850781Z",
     "start_time": "2021-12-29T01:03:20.816522Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-8-66a27d1e3eea>:1: DeprecationWarning: find_element_by_* commands are deprecated. Please use find_element() instead\n",
      "  password_input = driver.find_element_by_id(\"password\")\n"
     ]
    }
   ],
   "source": [
    "password_input = driver.find_element_by_id(\"password\")\n",
    "password_input.send_keys(my_password)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2090a108",
   "metadata": {},
   "source": [
    "Now we can hit \"return\" on the password input. To do that, we send the `RETURN` key by using selenium's built-in keys."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ff511d27",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-29T01:03:20.853509Z",
     "start_time": "2021-12-29T01:03:20.851754Z"
    }
   },
   "outputs": [],
   "source": [
    "from selenium.webdriver.common.keys import Keys"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "7836132f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-29T01:03:22.438638Z",
     "start_time": "2021-12-29T01:03:20.854454Z"
    }
   },
   "outputs": [],
   "source": [
    "password_input.send_keys(Keys.RETURN)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34363ed0",
   "metadata": {},
   "source": [
    "It worked!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "815faebe",
   "metadata": {},
   "source": [
    "## Scrape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b71c7a1e",
   "metadata": {},
   "source": [
    "Before we start scraping, I want to show you that you have access to the page's source code at any time. We probably won't use `driver.page_source` often, but it's good to know that it exists and should shed some light on what's going on!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "66e9e1d5",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-29T01:03:22.441804Z",
     "start_time": "2021-12-29T01:03:22.439955Z"
    }
   },
   "outputs": [],
   "source": [
    "# print(driver.page_source) # warning: large output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d882ab38",
   "metadata": {},
   "source": [
    "Let's begin.\n",
    "\n",
    "This time we want to get all the images instead of one specific image. Upon inspection, we can use a class name with the method `find_elements_by_class_name` (notice that elements is plural)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "4f8b4168",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-29T01:03:22.483869Z",
     "start_time": "2021-12-29T01:03:22.442907Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-12-bd69c85a8624>:1: DeprecationWarning: find_elements_by_* commands are deprecated. Please use find_elements() instead\n",
      "  imgs = driver.find_elements_by_class_name(\"shadow-lg\")\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[<selenium.webdriver.remote.webelement.WebElement (session=\"284d90a3c3215aaf129c2b971fa869ff\", element=\"ec8807d9-d71a-4040-8bcd-219515941eb8\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"284d90a3c3215aaf129c2b971fa869ff\", element=\"1dfcb59a-8e4e-48f9-8ca2-910fe06a23c9\")>,\n",
       " <selenium.webdriver.remote.webelement.WebElement (session=\"284d90a3c3215aaf129c2b971fa869ff\", element=\"1ccf3ce0-c84a-404c-b51d-cf68e63a3e44\")>]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imgs = driver.find_elements_by_class_name(\"shadow-lg\")\n",
    "imgs[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db460e2f",
   "metadata": {},
   "source": [
    "To access an attribute from an element, we need to use a try/except. So we'll first instantiate an empty list `links`, then we'll try to append the `src` attribute from each element in `imgs`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "fd95f30a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-29T01:03:23.780044Z",
     "start_time": "2021-12-29T01:03:22.485036Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([None,\n",
       "  'https://d1xrbm8v2c14yb.cloudfront.net/1468/thumb_1639164201_SYFW1279.JPG',\n",
       "  'https://d1xrbm8v2c14yb.cloudfront.net/1468/thumb_1639164201_SYFW1277.JPG'],\n",
       " 201)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "links = []\n",
    "\n",
    "for img in imgs:\n",
    "    try: links.append(img.get_attribute(\"src\"))\n",
    "    except: print(\"failed to get src\")\n",
    "        \n",
    "links[:3] , len(links)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b63f9ea",
   "metadata": {},
   "source": [
    "## Scrape next page"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e10cd85",
   "metadata": {},
   "source": [
    "Excellent! We scraped image links from one page. Now we need to move to the next page and scrape the next batch of links."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77933333",
   "metadata": {},
   "source": [
    "Let's find the page buttons at the bottom of the page and try to individually select the \">\" button which brings us to the next page.\n",
    "\n",
    "I think this will work because I went to the very last page, and this button doesn't exist there. So, we can click the \">\" button and save the links until \">\" stops working!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea77b062",
   "metadata": {},
   "source": [
    "To select a `button` element by specifying the value of one of its attributes, xpath comes in handy. The syntax for xpath accepted by `find_element_by_xpath` is `'//element[@attr=\"value\"]'`.\n",
    "\n",
    "Examples:\n",
    "- `driver.find_elements_by_xpath('//img[@src=\"https://www.rorymm.com/fun\"]')\n",
    "- `driver.find_elements_by_xpath('//div[@class=\"abcde\" or @class=\"zyxwv\"]')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "527c55d3",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-29T01:03:23.815110Z",
     "start_time": "2021-12-29T01:03:23.781105Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-14-0df71d069e7d>:1: DeprecationWarning: find_element_by_* commands are deprecated. Please use find_element() instead\n",
      "  next_page_button = driver.find_element_by_xpath('//button[@dusk=\"nextPage.after\"]')\n"
     ]
    }
   ],
   "source": [
    "next_page_button = driver.find_element_by_xpath('//button[@dusk=\"nextPage.after\"]')\n",
    "next_page_button.send_keys(Keys.RETURN)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54fdd1cb",
   "metadata": {},
   "source": [
    "Since we're loading a new page with new imgs, we'll sleep for a few seconds (1 second is fine for black and white ims but color imgs need longer – the later pages need the full 2)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "026b0d5d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-29T01:03:25.820586Z",
     "start_time": "2021-12-29T01:03:23.816193Z"
    }
   },
   "outputs": [],
   "source": [
    "sleep(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1eecda4",
   "metadata": {},
   "source": [
    "Now we'll scrape this page's images. The code is identical to last time EXCEPT we already have `links`, so we don't want to accidentally reinstantiate it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "fa7bc4eb",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-29T01:03:27.355569Z",
     "start_time": "2021-12-29T01:03:25.821815Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-16-ba03bf67a6d6>:1: DeprecationWarning: find_elements_by_* commands are deprecated. Please use find_elements() instead\n",
      "  imgs = driver.find_elements_by_class_name(\"shadow-lg\")\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[None,\n",
       " 'https://d1xrbm8v2c14yb.cloudfront.net/1468/thumb_1639164201_SYFW1279.JPG',\n",
       " 'https://d1xrbm8v2c14yb.cloudfront.net/1468/thumb_1639164201_SYFW1277.JPG',\n",
       " 'https://d1xrbm8v2c14yb.cloudfront.net/1468/thumb_1639148040_SYFW1276.JPG',\n",
       " 'https://d1xrbm8v2c14yb.cloudfront.net/1468/thumb_1639148039_SYFW1274.JPG']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imgs = driver.find_elements_by_class_name(\"shadow-lg\")\n",
    "\n",
    "for img in imgs:\n",
    "    try: links.append(img.get_attribute(\"src\"))\n",
    "    except: print(\"failed to get src\")\n",
    "        \n",
    "links[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "85736f32",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-29T01:03:27.359523Z",
     "start_time": "2021-12-29T01:03:27.356707Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "402"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(links)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7ce7d86",
   "metadata": {},
   "source": [
    "Very close – each page should have 200 imgs, so we have two extra."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4da59e0e",
   "metadata": {},
   "source": [
    "## Cleaning and saving the links"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3747614",
   "metadata": {},
   "source": [
    "Let's see what shouldn't be here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "13d71b05",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-29T01:03:27.380984Z",
     "start_time": "2021-12-29T01:03:27.360474Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[None,\n",
       " 'https://d1xrbm8v2c14yb.cloudfront.net/1468/thumb_1639164201_SYFW1279.JPG',\n",
       " 'https://d1xrbm8v2c14yb.cloudfront.net/1468/thumb_1639164201_SYFW1277.JPG']"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "links [:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af093435",
   "metadata": {},
   "source": [
    "Well wha-da-ya-know, the very first item is a `None`. Let's remove all the `None`s and see if that works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "5796403e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-29T01:03:27.399012Z",
     "start_time": "2021-12-29T01:03:27.382031Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "400"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "for i,l in enumerate(links):\n",
    "    if l == None: links.pop(i)\n",
    "\n",
    "len(links)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abfb8ac6",
   "metadata": {},
   "source": [
    "Perfect.\n",
    "\n",
    "These are thumbnails of size 400x300, which isn't a bad size for deep learning, but I want to get the full sized images. I tried removing \"thumb_\" from the link, and that worked! I'll edit the links so they all are full sized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "3a7f0ae1",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-29T01:03:27.417883Z",
     "start_time": "2021-12-29T01:03:27.399841Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['https://d1xrbm8v2c14yb.cloudfront.net/1468/1639164201_SYFW1279.JPG',\n",
       " 'https://d1xrbm8v2c14yb.cloudfront.net/1468/1639164201_SYFW1277.JPG',\n",
       " 'https://d1xrbm8v2c14yb.cloudfront.net/1468/1639148040_SYFW1276.JPG']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "links_big = [link.replace('thumb_', '') for link in links]\n",
    "links_big[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4a449e0",
   "metadata": {},
   "source": [
    "Now we're ready to put it all together and save the links to a text file. Saving to a text file will help me make sure I don't spend time downloading imgs I already have in the future!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97e5e931",
   "metadata": {},
   "source": [
    "# Putting it all together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "46fb9d93",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-12-29T01:12:31.536007Z",
     "start_time": "2021-12-29T01:12:07.650887Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "STARTING Yellowstone.ai image scraper.\n",
      "Starting headless webdriver ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-24-38c453efd8c7>:42: DeprecationWarning: executable_path has been deprecated, please pass in a Service object\n",
      "  driver = webdriver.Chrome(DRIVER_BIN, options=options)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Navigating to https://my.yellowstone.ai/media ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-24-38c453efd8c7>:52: DeprecationWarning: find_element_by_* commands are deprecated. Please use find_element() instead\n",
      "  email_input = driver.find_element_by_id(\"email\")\n",
      "<ipython-input-24-38c453efd8c7>:57: DeprecationWarning: find_element_by_* commands are deprecated. Please use find_element() instead\n",
      "  password_input = driver.find_element_by_id(\"password\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Logging into mccallionr+yellowstoneai@gmail.com's account ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-24-38c453efd8c7>:81: DeprecationWarning: find_elements_by_* commands are deprecated. Please use find_elements() instead\n",
      "  imgs = driver.find_elements_by_class_name(\"shadow-lg\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scraping image links from page 1 ...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-24-38c453efd8c7>:111: DeprecationWarning: find_element_by_* commands are deprecated. Please use find_element() instead\n",
      "  driver.find_element_by_xpath(xpath_str).send_keys(Keys.RETURN)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scraping image links from page 2 ...\n",
      "Scraping image links from page 3 ...\n",
      "Scraping image links from page 4 ...\n",
      "Scraping image links from page 5 ...\n",
      "Scraping image links from page 6 ...\n",
      "Stopping on page 6: this page has fewer than 200 links.\n",
      "Total links scraped: 1018\n",
      "Current trailcam images: 1018.\n",
      "New images to download: 0.\n",
      "Downloaded 0 images successfully, 0 failed.\n",
      "FINISHED scraping & downloading Yellowstone.ai trailcam images.\n"
     ]
    }
   ],
   "source": [
    "from fastai.vision.all import *\n",
    "from selenium import webdriver\n",
    "from selenium.webdriver.chrome.options import Options\n",
    "from selenium.webdriver.common.keys import Keys\n",
    "from selenium.webdriver.common.action_chains import ActionChains\n",
    "from time import sleep\n",
    "import requests\n",
    "import shutil\n",
    "import csv\n",
    "\n",
    "\n",
    "\n",
    "##-----------------------------------------------------------------------------\n",
    "##----- Params\n",
    "\n",
    "CREDS = 'creds.txt'\n",
    "URL = 'https://my.yellowstone.ai/media'\n",
    "LINUX = True # change to False if running on a Mac (Windows untested)\n",
    "\n",
    "if LINUX:\n",
    "    DRIVER_BIN = '/usr/local/bin/chromedriver'\n",
    "    IMAGES = '/home/rory/data/trailcam'\n",
    "else:\n",
    "    DRIVER_BIN = '/Users/rorymccallion/repos/scrapers/chromedriver96'\n",
    "    IMAGES = '/Users/rorymccallion/repos/trailcam/imgs'\n",
    "\n",
    "\n",
    "\n",
    "##-----------------------------------------------------------------------------\n",
    "##----- Webdriver Setup\n",
    "\n",
    "print(f\"STARTING Yellowstone.ai image scraper.\")\n",
    "\n",
    "file = open(CREDS, \"r\")\n",
    "my_email, my_password = file.read().split(\"\\n\")\n",
    "file.close()\n",
    "\n",
    "# Summon chrome webdriver.\n",
    "print(\"Starting headless webdriver ...\")\n",
    "options = Options()\n",
    "options.add_argument('--headless')\n",
    "driver = webdriver.Chrome(DRIVER_BIN, options=options)\n",
    "\n",
    "    \n",
    "# Open chrome, go to url, wait for page to load.\n",
    "driver.get(URL)\n",
    "sleep(1)\n",
    "print(f\"Navigating to {URL} ...\")\n",
    "\n",
    "\n",
    "# Find email input and enter email.\n",
    "email_input = driver.find_element_by_id(\"email\")\n",
    "email_input.send_keys(my_email)\n",
    "\n",
    "\n",
    "# Same for password, then \"hit enter\" to submit the form.\n",
    "password_input = driver.find_element_by_id(\"password\")\n",
    "password_input.send_keys(my_password)\n",
    "password_input.send_keys(Keys.RETURN)\n",
    "print(f\"Logging into {my_email}'s account ...\")\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "##-----------------------------------------------------------------------------\n",
    "##----- Scrape Image Links\n",
    "\n",
    "# We're about to start scraping image links and storing them in a list. When\n",
    "#  we've finished scraping image links, we'll compare this list of links to the\n",
    "#  list of image files we've already downloaded to determine which links have\n",
    "#  new images. We'll then download those new images.\n",
    "\n",
    "links = []\n",
    "\n",
    "# Do the following on each page of images:\n",
    "pagenum = 1\n",
    "while True:\n",
    "    \n",
    "    # Wait for the imgs to load, then find them into a list.\n",
    "    sleep(2)\n",
    "    imgs = driver.find_elements_by_class_name(\"shadow-lg\")\n",
    "    print(f\"Scraping image links from page {pagenum} ...\")\n",
    "    \n",
    "    \n",
    "    # Store their src attribute (their link) in links, else break.\n",
    "    for img in imgs:\n",
    "        try:\n",
    "            links.append(img.get_attribute(\"src\"))\n",
    "        except:\n",
    "            print(\"Couldn't get src attribute; moving on...\")\n",
    "            break\n",
    "    \n",
    "    \n",
    "    # Remove `None` values so we can test that we grabbed exactly 200.\n",
    "    for i,l in enumerate(links):\n",
    "        if l == None:\n",
    "            links.pop(i)\n",
    "            \n",
    "            \n",
    "    # Do the test.\n",
    "    if len(links) % 200 != 0:\n",
    "        # Note on the next line: the print says \"fewer\" but it COULD be greater than!\n",
    "        print(f\"Stopping on page {pagenum}: this page has fewer than 200 links.\")\n",
    "        print(f\"Total links scraped: {len(links)}\")\n",
    "        break\n",
    "    \n",
    "    \n",
    "    # Find the next page button and click it.\n",
    "    xpath_str = '//button[@dusk=\"nextPage.after\"]'\n",
    "    try:\n",
    "        driver.find_element_by_xpath(xpath_str).send_keys(Keys.RETURN)\n",
    "        pagenum += 1\n",
    "    except:\n",
    "        print(f\"Stopping on page {pagenum}: couldn't find next page.\")\n",
    "        print(f\"Total links scraped: {len(links)}\")\n",
    "        break\n",
    "\n",
    "        \n",
    "        \n",
    "                \n",
    "##-----------------------------------------------------------------------------\n",
    "##----- Find New Links\n",
    "\n",
    "\n",
    "# Get fullsized image links by changing URLs to remove \"thumb_\".\n",
    "links = [link.replace('thumb_', '') for link in links]\n",
    "\n",
    "# Get filenames from links.\n",
    "link_fns = [link.split(\"/\")[-1].replace('.JPG','.jpg') for link in links]\n",
    "\n",
    "# Get filenames from already downloaded images.\n",
    "image_fns = [path.name for path in get_image_files(IMAGES)]\n",
    "\n",
    "# If a link's fn isn't in the image fns, it's new and should be downloaded.\n",
    "new_links = [l for fn,l in zip(link_fns, links) if fn not in image_fns]\n",
    "\n",
    "print(f\"Current trailcam images: {len(image_fns)}.\")\n",
    "print(f\"New images to download: {len(new_links)}.\")\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "##-----------------------------------------------------------------------------\n",
    "##----- Download New Images\n",
    "\n",
    "\n",
    "succ, fail = 0, 0\n",
    "for link in new_links:\n",
    "    \n",
    "    r = requests.get(link, stream = True)\n",
    "    \n",
    "    if r.status_code == 200:\n",
    "        r.raw.decode_content = True\n",
    "        filename = link.split(\"/\")[-1].replace('.JPG','.jpg')\n",
    "        with open(IMAGES + \"/\" + filename, 'wb') as f:\n",
    "            shutil.copyfileobj(r.raw, f)\n",
    "        print('Sucessfully downloaded', link)\n",
    "        succ += 1\n",
    "    else:\n",
    "        print('Failed to download:', link)\n",
    "        fail += 1\n",
    "\n",
    "print(f\"Downloaded {succ} images successfully, {fail} failed.\")\n",
    "print(f\"FINISHED scraping & downloading Yellowstone.ai trailcam images.\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Headings",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}