01_get_submissions_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "dfef3756-6a1d-472c-ba07-8cb6c8ac1406",
   "metadata": {},
   "source": [
    "# Get Reddit submissions data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4020d33b-96ae-4336-b017-de6e9bce70f9",
   "metadata": {},
   "source": [
    "We'll use [psaw](https://psaw.readthedocs.io/en/latest/) Python wrapper for [Pushshift API](https://github.com/pushshift/api) to query [AskDocs subreddit](https://www.reddit.com/r/AskDocs/) for submissions posted in 2017 through Jan 21st 2022."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "6e194aa7-10de-4402-bd31-88b9e32ad515",
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime as dt\n",
    "import pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "f465f5b3-ae49-400b-8039-e38e0813ecbd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# for chime alerts for query success/fails\n",
    "import chime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "bd5fd370-e8a2-4b53-a409-c68315cbac29",
   "metadata": {},
   "outputs": [],
   "source": [
    "chime.theme('zelda')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "3a2e86b4-e36c-436a-9bdb-15ca852b0828",
   "metadata": {},
   "outputs": [],
   "source": [
    "from psaw import PushshiftAPI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "08bcfeca-23a8-4ca7-8f32-72e5b9c340e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "api = PushshiftAPI()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "44d189bf-69a4-453f-bbb3-9d4be16e1a9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "start_year = 2017\n",
    "end_year = 2022"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1004c1e9-dadb-48fd-91d8-de6b93436b21",
   "metadata": {},
   "source": [
    "The data we want to query is pretty big and it will take some time to pull all of it. Let's query for 2017 first, because it is the smaller data chunk by year, and we can use it to do some EDA while the rest of the querying finishes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a72551e2-9fe9-40be-9d8a-14878428e461",
   "metadata": {},
   "outputs": [],
   "source": [
    "d = {}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05f24365-3cba-4dbe-9824-3faf7d31193c",
   "metadata": {},
   "source": [
    "Note: not sure whether the end_epoch is inclusive or exclusive in Pushshift API, so we'll assume it's exclusive here and check for dups in EDA."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "449d78a0-1954-4c1e-8773-533432aff3bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "year = 2017\n",
    "\n",
    "start_epoch=int(dt.datetime(year, 1, 1, 0, 0, 0, 0, tzinfo=dt.timezone.utc).timestamp())\n",
    "end_epoch=int(dt.datetime(year+1, 1, 1, 0, 0, 0, 0, tzinfo=dt.timezone.utc).timestamp())\n",
    "\n",
    "gen = api.search_submissions(\n",
    "    subreddit='askdocs', \n",
    "    after=start_epoch,\n",
    "    before=end_epoch\n",
    ")\n",
    "\n",
    "d[year] = [thing.d_ for thing in gen]\n",
    "\n",
    "print(f'=== Finished ====')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "adef50c9-eb1b-4607-8cf9-bf101850889c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "62438"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(d[2017])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "58944fa2-2691-405f-8803-7ca08dd7f38d",
   "metadata": {},
   "outputs": [],
   "source": [
    "fname = 'reddit_askdocs_submissions_2017.pkl'\n",
    "\n",
    "with open(fname, 'wb') as outfile:\n",
    "    pickle.dump(d[2017], outfile)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83a1031d-4f49-44e4-896c-c3ea4d06bea3",
   "metadata": {},
   "source": [
    "Now let's query for the remaining years of the data sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "id": "67f6121a-aea9-4c1c-9a8f-ee27ce63cf55",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "=== Collecting data for year 2018 ====\n",
      "-- Attempt #1 to get the data for 2018-01.\n",
      "-- Got the data for 2018-01 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2018-02.\n",
      "-- Got the data for 2018-02 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2018-03.\n",
      "-- Got the data for 2018-03 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2018-04.\n",
      "-- Got the data for 2018-04 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2018-05.\n",
      "-- Got the data for 2018-05 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2018-06.\n",
      "-- Got the data for 2018-06 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2018-07.\n",
      "-- Got the data for 2018-07 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2018-08.\n",
      "-- Got the data for 2018-08 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2018-09.\n",
      "-- Got the data for 2018-09 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2018-10.\n",
      "-- Got the data for 2018-10 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2018-11.\n",
      "-- Got the data for 2018-11 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2018-12.\n",
      "-- Got the data for 2018-12 after 1 attempts.\n",
      "=== Finished ====\n",
      "\n",
      "\n",
      "=== Collecting data for year 2019 ====\n",
      "-- Attempt #1 to get the data for 2019-01.\n",
      "-- Got the data for 2019-01 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2019-02.\n",
      "-- Got the data for 2019-02 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2019-03.\n",
      "-- Got the data for 2019-03 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2019-04.\n",
      "-- Got the data for 2019-04 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2019-05.\n",
      "-- Got the data for 2019-05 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2019-06.\n",
      "-- Got the data for 2019-06 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2019-07.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/miniconda3/lib/python3.9/site-packages/psaw/PushshiftAPI.py:192: UserWarning: Got non 200 code 525\n",
      "  warnings.warn(\"Got non 200 code %s\" % response.status_code)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-- Got the data for 2019-07 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2019-08.\n",
      "-- Got the data for 2019-08 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2019-09.\n",
      "-- Got the data for 2019-09 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2019-10.\n",
      "-- Got the data for 2019-10 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2019-11.\n",
      "-- Got the data for 2019-11 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2019-12.\n",
      "-- Got the data for 2019-12 after 1 attempts.\n",
      "=== Finished ====\n",
      "\n",
      "\n",
      "=== Collecting data for year 2020 ====\n",
      "-- Attempt #1 to get the data for 2020-01.\n",
      "-- Got the data for 2020-01 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2020-02.\n",
      "-- Got the data for 2020-02 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2020-03.\n",
      "-- Got the data for 2020-03 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2020-04.\n",
      "-- Got the data for 2020-04 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2020-05.\n",
      "-- Got the data for 2020-05 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2020-06.\n",
      "-- Got the data for 2020-06 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2020-07.\n",
      "-- Got the data for 2020-07 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2020-08.\n",
      "-- Got the data for 2020-08 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2020-09.\n",
      "-- Got the data for 2020-09 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2020-10.\n",
      "-- Got the data for 2020-10 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2020-11.\n",
      "-- Got the data for 2020-11 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2020-12.\n",
      "-- Got the data for 2020-12 after 1 attempts.\n",
      "=== Finished ====\n",
      "\n",
      "\n",
      "=== Collecting data for year 2021 ====\n",
      "-- Attempt #1 to get the data for 2021-01.\n",
      "-- Got the data for 2021-01 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2021-02.\n",
      "-- Got the data for 2021-02 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2021-03.\n",
      "-- Got the data for 2021-03 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2021-04.\n",
      "-- Got the data for 2021-04 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2021-05.\n",
      "-- Got the data for 2021-05 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2021-06.\n",
      "(\"Connection broken: InvalidChunkLength(got length b'', 0 bytes read)\", InvalidChunkLength(got length b'', 0 bytes read))\n",
      "-- Attempt #2 to get the data for 2021-06.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/miniconda3/lib/python3.9/site-packages/psaw/PushshiftAPI.py:192: UserWarning: Got non 200 code 522\n",
      "  warnings.warn(\"Got non 200 code %s\" % response.status_code)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-- Got the data for 2021-06 after 2 attempts.\n",
      "-- Attempt #1 to get the data for 2021-07.\n",
      "-- Got the data for 2021-07 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2021-08.\n",
      "-- Got the data for 2021-08 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2021-09.\n",
      "-- Got the data for 2021-09 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2021-10.\n",
      "-- Got the data for 2021-10 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2021-11.\n",
      "-- Got the data for 2021-11 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2021-12.\n",
      "-- Got the data for 2021-12 after 1 attempts.\n",
      "=== Finished ====\n",
      "\n",
      "\n",
      "=== Collecting data for year 2022 ====\n",
      "-- Attempt #1 to get the data for 2022-01.\n",
      "-- Got the data for 2022-01 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2022-02.\n",
      "-- Got the data for 2022-02 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2022-03.\n",
      "-- Got the data for 2022-03 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2022-04.\n",
      "-- Got the data for 2022-04 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2022-05.\n",
      "-- Got the data for 2022-05 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2022-06.\n",
      "-- Got the data for 2022-06 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2022-07.\n",
      "-- Got the data for 2022-07 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2022-08.\n",
      "-- Got the data for 2022-08 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2022-09.\n",
      "-- Got the data for 2022-09 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2022-10.\n",
      "-- Got the data for 2022-10 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2022-11.\n",
      "-- Got the data for 2022-11 after 1 attempts.\n",
      "-- Attempt #1 to get the data for 2022-12.\n",
      "-- Got the data for 2022-12 after 1 attempts.\n",
      "=== Finished ====\n",
      "CPU times: user 12min 2s, sys: 1min 12s, total: 13min 14s\n",
      "Wall time: 4h 28min 59s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "for year in range(start_year+1, end_year+1):\n",
    "    print(f'\\n\\n=== Collecting data for year {year} ====')\n",
    "    chime.info()\n",
    "    d[year] = []\n",
    "\n",
    "    # Getting a lot of connection errors when querying by year, so querying by month instead\n",
    "    for month in range(1, 13):\n",
    "        if month < 12:\n",
    "            start_epoch=int(dt.datetime(year, month, 1, 0, 0, 0, 0, tzinfo=dt.timezone.utc).timestamp())\n",
    "            end_epoch=int(dt.datetime(year, month+1, 1, 0, 0, 0, 0, tzinfo=dt.timezone.utc).timestamp())\n",
    "        else:\n",
    "            start_epoch=int(dt.datetime(year, 12, 1, 0, 0, 0, 0, tzinfo=dt.timezone.utc).timestamp())\n",
    "            end_epoch=int(dt.datetime(year+1, 1, 1, 0, 0, 0, 0, tzinfo=dt.timezone.utc).timestamp())\n",
    "\n",
    "        got_the_data = False\n",
    "        n_tries = 0\n",
    "        while not got_the_data:\n",
    "            n_tries += 1\n",
    "            print(f'-- Attempt #{n_tries} to get the data for {year}-{month:02}.')\n",
    "            try: \n",
    "                gen = api.search_submissions(\n",
    "                    subreddit='askdocs', \n",
    "                    after=start_epoch,\n",
    "                    before=end_epoch\n",
    "                )\n",
    "\n",
    "                data_ym = [thing.d_ for thing in gen]\n",
    "                \n",
    "                got_the_data = True\n",
    "                print(f'-- Got the data for {year}-{month:02} after {n_tries} attempts.')\n",
    "                chime.success()\n",
    "                \n",
    "            except Exception as e:\n",
    "                chime.error()\n",
    "                print(e)\n",
    "        \n",
    "        d[year].extend(data_ym)\n",
    "\n",
    "    print(f'=== Finished {year} ====')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "fa8fac34-e485-4fb2-8d10-bd40dcfd0554",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys([2018, 2019, 2020, 2021, 2022])"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "id": "29dad815-c2cc-439c-ad02-0dde740cb83e",
   "metadata": {},
   "outputs": [],
   "source": [
    "fname = 'reddit_askdocs_submissions_2018_to_20220121.pkl'\n",
    "\n",
    "with open(fname, 'wb') as outfile:\n",
    "    pickle.dump(d, outfile)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}