03_fields_selection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a3f56a88-7e76-474b-a19d-7a06c36e13e1",
   "metadata": {},
   "source": [
    "# Fields selection  \n",
    "\n",
    "#### Notebook objectives:  \n",
    "- **Review the descriptive stats for columns in 2017 dataset and select the fields relevant to analyses**\n",
    "  - Note that the reddit data for the years after 2017 contains more fields. For this project, we're only interested in submissions text data and basic info about it that is available throught the datasets. So, we're interested in selecting the relevant fields available since 2017.  \n",
    "- **Output a list of fields selected for analyses**  \n",
    "\n",
    "#### Steps:  \n",
    "1. [Load data](#Load-data)  \n",
    "2. [Review fields and assign potential analysis roles](#Review-fields-and-assign-potential-analysis-roles)  \n",
    "3. [Final fields selection](#Final-fields-selection)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1e36df90-cf50-419b-ae58-13ebb4079a67",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from IPython.display import display, HTML, Markdown, clear_output\n",
    "import ipywidgets as widgets\n",
    "import time \n",
    "import chime\n",
    "\n",
    "import plotly.graph_objects as go"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d440fae5-4cfd-4891-9a7e-e066fc53f41b",
   "metadata": {},
   "outputs": [],
   "source": [
    "chime.theme('zelda')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "588783d0-aef5-4931-a499-d93a003d3489",
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_PATH = 'data/'\n",
    "OUTPUT_PATH = 'output/'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33c25a95-3002-4812-a96e-685837ba00c2",
   "metadata": {},
   "source": [
    "## Load data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03066e30-c0b5-49e9-9aed-f7d3feb8b336",
   "metadata": {},
   "source": [
    "Load the 2017 data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1a43c0b7-d706-46e3-a331-4e4a760d7f41",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(DATA_PATH + 'reddit_askdocs_submissions_2017.pkl', 'rb') as f:\n",
    "    d_2017 = pickle.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "bffc75c2-dee9-4c15-83cf-002704f2e9ac",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "62438"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(d_2017)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ffa0e619-a5e8-4219-8202-7aa9a9698224",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_2017 = pd.DataFrame(d_2017)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "1ab31337-fba7-4523-a81c-d4225be34694",
   "metadata": {},
   "outputs": [],
   "source": [
    "del d_2017"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "4b55f7ae-f634-476c-a2a3-318a8c73e783",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Index: 716072 entries, 0 to 15884\n",
      "Columns: 106 entries, author to url_overridden_by_dest\n",
      "dtypes: bool(6), float64(19), int64(4), object(77)\n",
      "memory usage: 555.9+ MB\n"
     ]
    }
   ],
   "source": [
    "df_2017.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "564c7fd5-21d3-48c1-84e2-0fb0530d56bd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>author</th>\n",
       "      <th>author_flair_css_class</th>\n",
       "      <th>author_flair_text</th>\n",
       "      <th>brand_safe</th>\n",
       "      <th>can_mod_post</th>\n",
       "      <th>contest_mode</th>\n",
       "      <th>created_utc</th>\n",
       "      <th>domain</th>\n",
       "      <th>full_link</th>\n",
       "      <th>id</th>\n",
       "      <th>...</th>\n",
       "      <th>removed_by_category</th>\n",
       "      <th>updated_utc</th>\n",
       "      <th>steward_reports</th>\n",
       "      <th>og_description</th>\n",
       "      <th>og_title</th>\n",
       "      <th>removed_by</th>\n",
       "      <th>media_metadata</th>\n",
       "      <th>is_created_from_ads_ui</th>\n",
       "      <th>author_is_blocked</th>\n",
       "      <th>url_overridden_by_dest</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[deleted]</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>1514764452</td>\n",
       "      <td>self.AskDocs</td>\n",
       "      <td>https://www.reddit.com/r/AskDocs/comments/7nbw...</td>\n",
       "      <td>7nbwtn</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>XenonCSGO</td>\n",
       "      <td>default</td>\n",
       "      <td>This user has not yet been verified.</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>1514764122</td>\n",
       "      <td>self.AskDocs</td>\n",
       "      <td>https://www.reddit.com/r/AskDocs/comments/7nbv...</td>\n",
       "      <td>7nbvsv</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[deleted]</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>1514764055</td>\n",
       "      <td>self.AskDocs</td>\n",
       "      <td>https://www.reddit.com/r/AskDocs/comments/7nbv...</td>\n",
       "      <td>7nbvln</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>DavisTheMagicSheep</td>\n",
       "      <td>default</td>\n",
       "      <td>This user has not yet been verified.</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>1514763799</td>\n",
       "      <td>self.AskDocs</td>\n",
       "      <td>https://www.reddit.com/r/AskDocs/comments/7nbu...</td>\n",
       "      <td>7nburb</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Dontgetscooped</td>\n",
       "      <td>default</td>\n",
       "      <td>This user has not yet been verified.</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>1514763188</td>\n",
       "      <td>self.AskDocs</td>\n",
       "      <td>https://www.reddit.com/r/AskDocs/comments/7nbs...</td>\n",
       "      <td>7nbsw2</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 106 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "               author author_flair_css_class  \\\n",
       "0           [deleted]                   None   \n",
       "1           XenonCSGO                default   \n",
       "2           [deleted]                   None   \n",
       "3  DavisTheMagicSheep                default   \n",
       "4      Dontgetscooped                default   \n",
       "\n",
       "                      author_flair_text brand_safe can_mod_post  contest_mode  \\\n",
       "0                                  None       True        False         False   \n",
       "1  This user has not yet been verified.       True        False         False   \n",
       "2                                  None       True        False         False   \n",
       "3  This user has not yet been verified.       True        False         False   \n",
       "4  This user has not yet been verified.       True        False         False   \n",
       "\n",
       "   created_utc        domain  \\\n",
       "0   1514764452  self.AskDocs   \n",
       "1   1514764122  self.AskDocs   \n",
       "2   1514764055  self.AskDocs   \n",
       "3   1514763799  self.AskDocs   \n",
       "4   1514763188  self.AskDocs   \n",
       "\n",
       "                                           full_link      id  ...  \\\n",
       "0  https://www.reddit.com/r/AskDocs/comments/7nbw...  7nbwtn  ...   \n",
       "1  https://www.reddit.com/r/AskDocs/comments/7nbv...  7nbvsv  ...   \n",
       "2  https://www.reddit.com/r/AskDocs/comments/7nbv...  7nbvln  ...   \n",
       "3  https://www.reddit.com/r/AskDocs/comments/7nbu...  7nburb  ...   \n",
       "4  https://www.reddit.com/r/AskDocs/comments/7nbs...  7nbsw2  ...   \n",
       "\n",
       "  removed_by_category updated_utc  steward_reports og_description  og_title  \\\n",
       "0                 NaN         NaN              NaN            NaN       NaN   \n",
       "1                 NaN         NaN              NaN            NaN       NaN   \n",
       "2                 NaN         NaN              NaN            NaN       NaN   \n",
       "3                 NaN         NaN              NaN            NaN       NaN   \n",
       "4                 NaN         NaN              NaN            NaN       NaN   \n",
       "\n",
       "   removed_by  media_metadata  is_created_from_ads_ui author_is_blocked  \\\n",
       "0         NaN             NaN                     NaN               NaN   \n",
       "1         NaN             NaN                     NaN               NaN   \n",
       "2         NaN             NaN                     NaN               NaN   \n",
       "3         NaN             NaN                     NaN               NaN   \n",
       "4         NaN             NaN                     NaN               NaN   \n",
       "\n",
       "  url_overridden_by_dest  \n",
       "0                    NaN  \n",
       "1                    NaN  \n",
       "2                    NaN  \n",
       "3                    NaN  \n",
       "4                    NaN  \n",
       "\n",
       "[5 rows x 106 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_2017.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "651ad446-d7e6-4bb3-8357-411cdb44c290",
   "metadata": {},
   "source": [
    "## Review fields and assign potential analysis roles"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b56df971-c83b-4b5f-a2c6-21171eebb000",
   "metadata": {},
   "source": [
    "Some of the fields in our 2017 dataset might not be useful for our analyses.  \n",
    "Let's hand pick a starter set of fields that might be useful. We'll review and refine this list in the next steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa71d2c0-d80d-41dc-ba89-154e57a7a6d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hand-picking fields\n",
    "selected_fields = []\n",
    "\n",
    "for c in df_2017.columns:\n",
    "    clear_output(wait=True)\n",
    "    display(HTML(f'<h2>Field: {c}</h2>'))\n",
    "    time.sleep(3)\n",
    "    \n",
    "    forward = False\n",
    "    while not forward in ['y', 'n']:\n",
    "        forward = input('Use this field? [y/n]')\n",
    "        if forward == 'y':\n",
    "            selected_fields.append(c)\n",
    "\n",
    "clear_output(wait=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22946722-64ef-43d6-b262-0cad21ea31b1",
   "metadata": {},
   "source": [
    "Here's the list of fields we've hand picked:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "8d337fd1-e6aa-4395-abfd-942449326caa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['author',\n",
       " 'author_flair_text',\n",
       " 'created_utc',\n",
       " 'domain',\n",
       " 'full_link',\n",
       " 'id',\n",
       " 'is_reddit_media_domain',\n",
       " 'is_video',\n",
       " 'locked',\n",
       " 'num_comments',\n",
       " 'num_crossposts',\n",
       " 'over_18',\n",
       " 'permalink',\n",
       " 'pinned',\n",
       " 'score',\n",
       " 'selftext',\n",
       " 'spoiler',\n",
       " 'stickied',\n",
       " 'title',\n",
       " 'url',\n",
       " 'banned_by',\n",
       " 'edited',\n",
       " 'crosspost_parent',\n",
       " 'crosspost_parent_list',\n",
       " 'distinguished',\n",
       " 'author_created_utc',\n",
       " 'author_fullname']"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "selected_fields"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4eba92d6-a484-4015-af19-0aca45308f19",
   "metadata": {},
   "source": [
    "['author',\n",
    " 'author_flair_text',\n",
    " 'created_utc',\n",
    " 'domain',\n",
    " 'full_link',\n",
    " 'id',\n",
    " 'is_reddit_media_domain',\n",
    " 'is_video',\n",
    " 'locked',\n",
    " 'num_comments',\n",
    " 'num_crossposts',\n",
    " 'over_18',\n",
    " 'permalink',\n",
    " 'pinned',\n",
    " 'score',\n",
    " 'selftext',\n",
    " 'spoiler',\n",
    " 'stickied',\n",
    " 'title',\n",
    " 'url',\n",
    " 'banned_by',\n",
    " 'edited',\n",
    " 'crosspost_parent',\n",
    " 'crosspost_parent_list',\n",
    " 'distinguished',\n",
    " 'author_created_utc',\n",
    " 'author_fullname']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "991243d7-b137-4f46-9d1d-16996b398146",
   "metadata": {},
   "source": [
    "The fields dropped by hand in the step above were either uninformative (e.g. all nulls or all same value), data used for UI (e.g. css class, thumbnails, image size, etc.) or redundant (e.g. created timestamp in query local time)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0894059a-7182-4cae-b474-603b933d2a33",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save the list of hand-picked fields\n",
    "fname = 'reddit_submissions_selected_fields_list.pkl'\n",
    "with open(fname, 'wb') as f:\n",
    "    pickle.dump(selected_fields, f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "df7e9ac0-e5bc-49b1-bce3-aa5832b8ff4a",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(fname, 'rb') as f:\n",
    "    selected_fields = pickle.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "id": "32821fad-5926-4960-8ee0-20c43b7b1bdf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# selected_fields"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0ed8e52-39b2-4105-abd6-32feda61f73f",
   "metadata": {},
   "source": [
    "Having explored individual fields and entries a bit, let's run through the starter set of selected fields again, looking at sample values, stats and value counts, to review and refine the selection, and add some helpful notes, like how the fields will likely be used in analysis, what it's for on reddit, the type of data and any additional notes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "b59168ba-976f-4e54-9f7a-6ae7fc04f95b",
   "metadata": {},
   "outputs": [],
   "source": [
    "field_roles_dict = {\n",
    "    'reddit': {\n",
    "        'a': 'author info',\n",
    "        'p': 'post details',\n",
    "        'aa': 'author actions on the post (other than commenting)',\n",
    "        'ga': 'general subreddit users reactions to the post',\n",
    "        'ma': 'mod reactions to either post content or comments activity on it'\n",
    "    },\n",
    "    'analysis': {\n",
    "        'i': 'id',\n",
    "        'r': 'reference',\n",
    "        'a': 'analysis',\n",
    "        'f': 'filtering',\n",
    "        't': 'transform',\n",
    "        'd': 'drop'\n",
    "    },\n",
    "    'type': {\n",
    "        't': 'timestamp',\n",
    "        's': 'short text',\n",
    "        'l': 'long text',\n",
    "        'u': 'url',\n",
    "        'c': 'categorical',\n",
    "        'b': 'binary flag',\n",
    "        'n': 'numeric'\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "id": "cc3a82d8-bffb-4ecb-a278-d0fd000c6f41",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== DONE! ====\n",
      "Selected fields:\n",
      "['author', 'author_flair_text', 'created_utc', 'domain', 'full_link', 'id', 'is_reddit_media_domain', 'is_video', 'locked', 'num_comments', 'num_crossposts', 'over_18', 'permalink', 'pinned', 'score', 'selftext', 'spoiler', 'stickied', 'title', 'url', 'banned_by', 'edited', 'crosspost_parent', 'crosspost_parent_list', 'distinguished', 'author_created_utc', 'author_fullname']\n"
     ]
    }
   ],
   "source": [
    "# Display sample stats, univariate stats and value counts for each field and collect user input\n",
    "# on whether to select the field as potentially useful for analysis.\n",
    "\n",
    "selected_fields_dict = {\n",
    "    'field_name': [],\n",
    "    'type': [],\n",
    "    'reddit_role': [],\n",
    "    'analysis_role': [],\n",
    "    'notes': []\n",
    "}\n",
    "\n",
    "for c in selected_fields:\n",
    "    selected_fields_dict['field_name'].append(c)\n",
    "    \n",
    "    clear_output(wait=True)\n",
    "    \n",
    "    display(HTML(f'<h2>Field: {c}</h2>'))\n",
    "    display(HTML(f'<h3>Sample values:</h3>'))\n",
    "    display(df_2017[~df_2017[c].isna()][c].head(10))\n",
    "    display(HTML(f'<hr>'))\n",
    "    \n",
    "    display(HTML(f'<h3>Stats:</h3>'))\n",
    "    display(df_2017[c].describe())\n",
    "    display(HTML(f'<hr>'))\n",
    "    \n",
    "    display(HTML(f'<h3>Value counts:</h3>'))\n",
    "    vc1 = df_2017[c].value_counts(dropna=False)\n",
    "    vc2 = df_2017[c].value_counts(dropna=False, normalize=True)\n",
    "    display(pd.concat([vc1, vc2], axis=1))\n",
    "    display(HTML(f'<hr>'))\n",
    "    \n",
    "    time.sleep(5)\n",
    "    \n",
    "    forward = False\n",
    "    while forward == False:\n",
    "        # ask reddit role\n",
    "        role_type = 'reddit'\n",
    "        q1_options_str = '\\n'.join(\n",
    "            [f\"{x} = {field_roles_dict[role_type][x]}\" for x in field_roles_dict[role_type]]\n",
    "        )\n",
    "\n",
    "        q1r = input(f'''\n",
    "Assign {role_type} role:\n",
    "\n",
    "{q1_options_str}\n",
    "\n",
    "        ''')\n",
    "        selected_fields_dict[f'{role_type}_role'].append(q1r)\n",
    "        \n",
    "        # ask analysis role\n",
    "        role_type = 'analysis'\n",
    "        q2_options_str = '\\n'.join(\n",
    "            [f\"{x} = {field_roles_dict[role_type][x]}\" for x in field_roles_dict[role_type]]\n",
    "        )\n",
    "\n",
    "        q2r = input(f'''\n",
    "Assign {role_type} role:\n",
    "\n",
    "{q2_options_str}\n",
    "\n",
    "        ''')\n",
    "        selected_fields_dict[f'{role_type}_role'].append(q2r)\n",
    "        \n",
    "\n",
    "        # ask field type\n",
    "        role_type = 'type'\n",
    "        q3_options_str = '\\n'.join(\n",
    "            [f\"{x} = {field_roles_dict[role_type][x]}\" for x in field_roles_dict[role_type]]\n",
    "        )\n",
    "\n",
    "        q3r = input(f'''\n",
    "Assign field {role_type}:\n",
    "\n",
    "{q3_options_str}\n",
    "\n",
    "        ''')\n",
    "        selected_fields_dict[f'{role_type}'].append(q3r)\n",
    "        \n",
    "        \n",
    "        # ask for any notes\n",
    "        q4r = input('Any notes on the field?:')\n",
    "        selected_fields_dict['notes'].append(q4r)\n",
    "        \n",
    "        \n",
    "        forward = input('Continue? [hit enter]')\n",
    "\n",
    "clear_output(wait=True)\n",
    "\n",
    "print('=== DONE! ====')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "id": "72f9e138-94e9-4692-bc70-51d538c6ee4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "selected_fields_df = pd.DataFrame(selected_fields_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "id": "8be73764-61d7-4149-a4a9-16a85b72c9b7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>field_name</th>\n",
       "      <th>type</th>\n",
       "      <th>reddit_role</th>\n",
       "      <th>analysis_role</th>\n",
       "      <th>notes</th>\n",
       "      <th>type_long</th>\n",
       "      <th>reddit_role_long</th>\n",
       "      <th>analysis_role_long</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>author</td>\n",
       "      <td>s</td>\n",
       "      <td>a</td>\n",
       "      <td>i</td>\n",
       "      <td>Note the [deleted] and [removed] entries.</td>\n",
       "      <td>short text</td>\n",
       "      <td>author info</td>\n",
       "      <td>id</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>author_flair_text</td>\n",
       "      <td>c</td>\n",
       "      <td>a</td>\n",
       "      <td>a</td>\n",
       "      <td>31% NaN values.</td>\n",
       "      <td>categorical</td>\n",
       "      <td>author info</td>\n",
       "      <td>analysis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>created_utc</td>\n",
       "      <td>t</td>\n",
       "      <td>p</td>\n",
       "      <td>a</td>\n",
       "      <td>Dups present.</td>\n",
       "      <td>timestamp</td>\n",
       "      <td>post details</td>\n",
       "      <td>analysis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>domain</td>\n",
       "      <td>s</td>\n",
       "      <td>p</td>\n",
       "      <td>a</td>\n",
       "      <td>Domain where the post originated from.</td>\n",
       "      <td>short text</td>\n",
       "      <td>post details</td>\n",
       "      <td>analysis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>full_link</td>\n",
       "      <td>u</td>\n",
       "      <td>p</td>\n",
       "      <td>r</td>\n",
       "      <td>A link to the post on Reddit.</td>\n",
       "      <td>url</td>\n",
       "      <td>post details</td>\n",
       "      <td>reference</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>id</td>\n",
       "      <td>s</td>\n",
       "      <td>p</td>\n",
       "      <td>i</td>\n",
       "      <td>Post id</td>\n",
       "      <td>short text</td>\n",
       "      <td>post details</td>\n",
       "      <td>id</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>is_reddit_media_domain</td>\n",
       "      <td>b</td>\n",
       "      <td>p</td>\n",
       "      <td>d</td>\n",
       "      <td>All values are either empty or false.</td>\n",
       "      <td>binary flag</td>\n",
       "      <td>post details</td>\n",
       "      <td>drop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>is_video</td>\n",
       "      <td>b</td>\n",
       "      <td>p</td>\n",
       "      <td>d</td>\n",
       "      <td>All values are either empty or false.</td>\n",
       "      <td>binary flag</td>\n",
       "      <td>post details</td>\n",
       "      <td>drop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>locked</td>\n",
       "      <td>b</td>\n",
       "      <td>ma</td>\n",
       "      <td>a</td>\n",
       "      <td>Only 6 values are true, everything else false.</td>\n",
       "      <td>binary flag</td>\n",
       "      <td>mod reactions to either post content or commen...</td>\n",
       "      <td>analysis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>num_comments</td>\n",
       "      <td>n</td>\n",
       "      <td>ga</td>\n",
       "      <td>a</td>\n",
       "      <td></td>\n",
       "      <td>numeric</td>\n",
       "      <td>general subreddit users reactions to the post</td>\n",
       "      <td>analysis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>num_crossposts</td>\n",
       "      <td>n</td>\n",
       "      <td>aa</td>\n",
       "      <td>a</td>\n",
       "      <td>Both NaN and zeros present. Few values &gt;0.</td>\n",
       "      <td>numeric</td>\n",
       "      <td>author actions on the post (other than comment...</td>\n",
       "      <td>analysis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>over_18</td>\n",
       "      <td>b</td>\n",
       "      <td>p</td>\n",
       "      <td>a</td>\n",
       "      <td>98% false. Looks like a NSFW-type label on the...</td>\n",
       "      <td>binary flag</td>\n",
       "      <td>post details</td>\n",
       "      <td>analysis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>permalink</td>\n",
       "      <td>s</td>\n",
       "      <td>p</td>\n",
       "      <td>d</td>\n",
       "      <td>Do not need this, redundant to the full_link f...</td>\n",
       "      <td>short text</td>\n",
       "      <td>post details</td>\n",
       "      <td>drop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>pinned</td>\n",
       "      <td>b</td>\n",
       "      <td>aa</td>\n",
       "      <td>a</td>\n",
       "      <td>Users can pin up to 4 posts to their profile.</td>\n",
       "      <td>binary flag</td>\n",
       "      <td>author actions on the post (other than comment...</td>\n",
       "      <td>analysis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>score</td>\n",
       "      <td>n</td>\n",
       "      <td>ga</td>\n",
       "      <td>a</td>\n",
       "      <td>The score is based on up and down votes.</td>\n",
       "      <td>numeric</td>\n",
       "      <td>general subreddit users reactions to the post</td>\n",
       "      <td>analysis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>selftext</td>\n",
       "      <td>l</td>\n",
       "      <td>p</td>\n",
       "      <td>a</td>\n",
       "      <td>Can have [deleted] as values.</td>\n",
       "      <td>long text</td>\n",
       "      <td>post details</td>\n",
       "      <td>analysis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>spoiler</td>\n",
       "      <td>b</td>\n",
       "      <td>aa</td>\n",
       "      <td>a</td>\n",
       "      <td>Spoiler tags are used to mark spoiler content,...</td>\n",
       "      <td>binary flag</td>\n",
       "      <td>author actions on the post (other than comment...</td>\n",
       "      <td>analysis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>stickied</td>\n",
       "      <td>b</td>\n",
       "      <td>aa</td>\n",
       "      <td>f</td>\n",
       "      <td>Mods can pin up to 2 of their own posts to the...</td>\n",
       "      <td>binary flag</td>\n",
       "      <td>author actions on the post (other than comment...</td>\n",
       "      <td>filtering</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>title</td>\n",
       "      <td>l</td>\n",
       "      <td>p</td>\n",
       "      <td>a</td>\n",
       "      <td>Title of the post, can be very long.</td>\n",
       "      <td>long text</td>\n",
       "      <td>post details</td>\n",
       "      <td>analysis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>url</td>\n",
       "      <td>u</td>\n",
       "      <td>p</td>\n",
       "      <td>r</td>\n",
       "      <td>Url to the original post if crossposted or fro...</td>\n",
       "      <td>url</td>\n",
       "      <td>post details</td>\n",
       "      <td>reference</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>banned_by</td>\n",
       "      <td>c</td>\n",
       "      <td>ma</td>\n",
       "      <td>f</td>\n",
       "      <td>The only values are NaN and moderators.</td>\n",
       "      <td>categorical</td>\n",
       "      <td>mod reactions to either post content or commen...</td>\n",
       "      <td>filtering</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>edited</td>\n",
       "      <td>t</td>\n",
       "      <td>aa</td>\n",
       "      <td>a</td>\n",
       "      <td>86% NaNs.</td>\n",
       "      <td>timestamp</td>\n",
       "      <td>author actions on the post (other than comment...</td>\n",
       "      <td>analysis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>crosspost_parent</td>\n",
       "      <td>s</td>\n",
       "      <td>p</td>\n",
       "      <td>t</td>\n",
       "      <td>Cross-post parent post id.</td>\n",
       "      <td>short text</td>\n",
       "      <td>post details</td>\n",
       "      <td>transform</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>crosspost_parent_list</td>\n",
       "      <td>l</td>\n",
       "      <td>p</td>\n",
       "      <td>t</td>\n",
       "      <td>This ultimately contains the body text of a cr...</td>\n",
       "      <td>long text</td>\n",
       "      <td>post details</td>\n",
       "      <td>transform</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>distinguished</td>\n",
       "      <td>c</td>\n",
       "      <td>aa</td>\n",
       "      <td>f</td>\n",
       "      <td>Mods can tag posts as distinguished, usually u...</td>\n",
       "      <td>categorical</td>\n",
       "      <td>author actions on the post (other than comment...</td>\n",
       "      <td>filtering</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>author_created_utc</td>\n",
       "      <td>t</td>\n",
       "      <td>a</td>\n",
       "      <td>d</td>\n",
       "      <td>Unclear what this is.</td>\n",
       "      <td>timestamp</td>\n",
       "      <td>author info</td>\n",
       "      <td>drop</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>author_fullname</td>\n",
       "      <td>s</td>\n",
       "      <td>a</td>\n",
       "      <td>i</td>\n",
       "      <td>Unclear what this is, and lots of NaNs, but de...</td>\n",
       "      <td>short text</td>\n",
       "      <td>author info</td>\n",
       "      <td>id</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                field_name type reddit_role analysis_role  \\\n",
       "0                   author    s           a             i   \n",
       "1        author_flair_text    c           a             a   \n",
       "2              created_utc    t           p             a   \n",
       "3                   domain    s           p             a   \n",
       "4                full_link    u           p             r   \n",
       "5                       id    s           p             i   \n",
       "6   is_reddit_media_domain    b           p             d   \n",
       "7                 is_video    b           p             d   \n",
       "8                   locked    b          ma             a   \n",
       "9             num_comments    n          ga             a   \n",
       "10          num_crossposts    n          aa             a   \n",
       "11                 over_18    b           p             a   \n",
       "12               permalink    s           p             d   \n",
       "13                  pinned    b          aa             a   \n",
       "14                   score    n          ga             a   \n",
       "15                selftext    l           p             a   \n",
       "16                 spoiler    b          aa             a   \n",
       "17                stickied    b          aa             f   \n",
       "18                   title    l           p             a   \n",
       "19                     url    u           p             r   \n",
       "20               banned_by    c          ma             f   \n",
       "21                  edited    t          aa             a   \n",
       "22        crosspost_parent    s           p             t   \n",
       "23   crosspost_parent_list    l           p             t   \n",
       "24           distinguished    c          aa             f   \n",
       "25      author_created_utc    t           a             d   \n",
       "26         author_fullname    s           a             i   \n",
       "\n",
       "                                                notes    type_long  \\\n",
       "0           Note the [deleted] and [removed] entries.   short text   \n",
       "1                                     31% NaN values.  categorical   \n",
       "2                                       Dups present.    timestamp   \n",
       "3              Domain where the post originated from.   short text   \n",
       "4                       A link to the post on Reddit.          url   \n",
       "5                                             Post id   short text   \n",
       "6               All values are either empty or false.  binary flag   \n",
       "7               All values are either empty or false.  binary flag   \n",
       "8      Only 6 values are true, everything else false.  binary flag   \n",
       "9                                                          numeric   \n",
       "10         Both NaN and zeros present. Few values >0.      numeric   \n",
       "11  98% false. Looks like a NSFW-type label on the...  binary flag   \n",
       "12  Do not need this, redundant to the full_link f...   short text   \n",
       "13      Users can pin up to 4 posts to their profile.  binary flag   \n",
       "14           The score is based on up and down votes.      numeric   \n",
       "15                      Can have [deleted] as values.    long text   \n",
       "16  Spoiler tags are used to mark spoiler content,...  binary flag   \n",
       "17  Mods can pin up to 2 of their own posts to the...  binary flag   \n",
       "18               Title of the post, can be very long.    long text   \n",
       "19  Url to the original post if crossposted or fro...          url   \n",
       "20            The only values are NaN and moderators.  categorical   \n",
       "21                                          86% NaNs.    timestamp   \n",
       "22                         Cross-post parent post id.   short text   \n",
       "23  This ultimately contains the body text of a cr...    long text   \n",
       "24  Mods can tag posts as distinguished, usually u...  categorical   \n",
       "25                              Unclear what this is.    timestamp   \n",
       "26  Unclear what this is, and lots of NaNs, but de...   short text   \n",
       "\n",
       "                                     reddit_role_long analysis_role_long  \n",
       "0                                         author info                 id  \n",
       "1                                         author info           analysis  \n",
       "2                                        post details           analysis  \n",
       "3                                        post details           analysis  \n",
       "4                                        post details          reference  \n",
       "5                                        post details                 id  \n",
       "6                                        post details               drop  \n",
       "7                                        post details               drop  \n",
       "8   mod reactions to either post content or commen...           analysis  \n",
       "9       general subreddit users reactions to the post           analysis  \n",
       "10  author actions on the post (other than comment...           analysis  \n",
       "11                                       post details           analysis  \n",
       "12                                       post details               drop  \n",
       "13  author actions on the post (other than comment...           analysis  \n",
       "14      general subreddit users reactions to the post           analysis  \n",
       "15                                       post details           analysis  \n",
       "16  author actions on the post (other than comment...           analysis  \n",
       "17  author actions on the post (other than comment...          filtering  \n",
       "18                                       post details           analysis  \n",
       "19                                       post details          reference  \n",
       "20  mod reactions to either post content or commen...          filtering  \n",
       "21  author actions on the post (other than comment...           analysis  \n",
       "22                                       post details          transform  \n",
       "23                                       post details          transform  \n",
       "24  author actions on the post (other than comment...          filtering  \n",
       "25                                        author info               drop  \n",
       "26                                        author info                 id  "
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "selected_fields_df['type_long'] = selected_fields_df['type'].apply(lambda x: field_roles_dict['type'][x])\n",
    "selected_fields_df['reddit_role_long'] = selected_fields_df['reddit_role']\\\n",
    "    .apply(lambda x: field_roles_dict['reddit'][x])\n",
    "selected_fields_df['analysis_role_long'] = selected_fields_df['analysis_role']\\\n",
    "    .apply(lambda x: field_roles_dict['analysis'][x])\n",
    "selected_fields_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "id": "ed7dec67-96c7-475f-95ee-fd0fb2ea4b28",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>field_name</th>\n",
       "      <th>type</th>\n",
       "      <th>reddit_role</th>\n",
       "      <th>analysis_role</th>\n",
       "      <th>notes</th>\n",
       "      <th>type_long</th>\n",
       "      <th>reddit_role_long</th>\n",
       "      <th>analysis_role_long</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>author</td>\n",
       "      <td>s</td>\n",
       "      <td>a</td>\n",
       "      <td>i</td>\n",
       "      <td>Note the [deleted] and [removed] entries.</td>\n",
       "      <td>short text</td>\n",
       "      <td>author info</td>\n",
       "      <td>id</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>author_flair_text</td>\n",
       "      <td>c</td>\n",
       "      <td>a</td>\n",
       "      <td>a</td>\n",
       "      <td>31% NaN values.</td>\n",
       "      <td>categorical</td>\n",
       "      <td>author info</td>\n",
       "      <td>analysis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>created_utc</td>\n",
       "      <td>t</td>\n",
       "      <td>p</td>\n",
       "      <td>a</td>\n",
       "      <td>Dups present.</td>\n",
       "      <td>timestamp</td>\n",
       "      <td>post details</td>\n",
       "      <td>analysis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>domain</td>\n",
       "      <td>s</td>\n",
       "      <td>p</td>\n",
       "      <td>a</td>\n",
       "      <td>Domain where the post originated from.</td>\n",
       "      <td>short text</td>\n",
       "      <td>post details</td>\n",
       "      <td>analysis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>full_link</td>\n",
       "      <td>u</td>\n",
       "      <td>p</td>\n",
       "      <td>r</td>\n",
       "      <td>A link to the post on Reddit.</td>\n",
       "      <td>url</td>\n",
       "      <td>post details</td>\n",
       "      <td>reference</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>id</td>\n",
       "      <td>s</td>\n",
       "      <td>p</td>\n",
       "      <td>i</td>\n",
       "      <td>Post id</td>\n",
       "      <td>short text</td>\n",
       "      <td>post details</td>\n",
       "      <td>id</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>locked</td>\n",
       "      <td>b</td>\n",
       "      <td>ma</td>\n",
       "      <td>a</td>\n",
       "      <td>Only 6 values are true, everything else false.</td>\n",
       "      <td>binary flag</td>\n",
       "      <td>mod reactions to either post content or commen...</td>\n",
       "      <td>analysis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>num_comments</td>\n",
       "      <td>n</td>\n",
       "      <td>ga</td>\n",
       "      <td>a</td>\n",
       "      <td></td>\n",
       "      <td>numeric</td>\n",
       "      <td>general subreddit users reactions to the post</td>\n",
       "      <td>analysis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>num_crossposts</td>\n",
       "      <td>n</td>\n",
       "      <td>aa</td>\n",
       "      <td>a</td>\n",
       "      <td>Both NaN and zeros present. Few values &gt;0.</td>\n",
       "      <td>numeric</td>\n",
       "      <td>author actions on the post (other than comment...</td>\n",
       "      <td>analysis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>over_18</td>\n",
       "      <td>b</td>\n",
       "      <td>p</td>\n",
       "      <td>a</td>\n",
       "      <td>98% false. Looks like a NSFW-type label on the...</td>\n",
       "      <td>binary flag</td>\n",
       "      <td>post details</td>\n",
       "      <td>analysis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>pinned</td>\n",
       "      <td>b</td>\n",
       "      <td>aa</td>\n",
       "      <td>a</td>\n",
       "      <td>Users can pin up to 4 posts to their profile.</td>\n",
       "      <td>binary flag</td>\n",
       "      <td>author actions on the post (other than comment...</td>\n",
       "      <td>analysis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>score</td>\n",
       "      <td>n</td>\n",
       "      <td>ga</td>\n",
       "      <td>a</td>\n",
       "      <td>The score is based on up and down votes.</td>\n",
       "      <td>numeric</td>\n",
       "      <td>general subreddit users reactions to the post</td>\n",
       "      <td>analysis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>selftext</td>\n",
       "      <td>l</td>\n",
       "      <td>p</td>\n",
       "      <td>a</td>\n",
       "      <td>Can have [deleted] as values.</td>\n",
       "      <td>long text</td>\n",
       "      <td>post details</td>\n",
       "      <td>analysis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>spoiler</td>\n",
       "      <td>b</td>\n",
       "      <td>aa</td>\n",
       "      <td>a</td>\n",
       "      <td>Spoiler tags are used to mark spoiler content,...</td>\n",
       "      <td>binary flag</td>\n",
       "      <td>author actions on the post (other than comment...</td>\n",
       "      <td>analysis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>stickied</td>\n",
       "      <td>b</td>\n",
       "      <td>aa</td>\n",
       "      <td>f</td>\n",
       "      <td>Mods can pin up to 2 of their own posts to the...</td>\n",
       "      <td>binary flag</td>\n",
       "      <td>author actions on the post (other than comment...</td>\n",
       "      <td>filtering</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>title</td>\n",
       "      <td>l</td>\n",
       "      <td>p</td>\n",
       "      <td>a</td>\n",
       "      <td>Title of the post, can be very long.</td>\n",
       "      <td>long text</td>\n",
       "      <td>post details</td>\n",
       "      <td>analysis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>url</td>\n",
       "      <td>u</td>\n",
       "      <td>p</td>\n",
       "      <td>r</td>\n",
       "      <td>Url to the original post if crossposted or fro...</td>\n",
       "      <td>url</td>\n",
       "      <td>post details</td>\n",
       "      <td>reference</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>banned_by</td>\n",
       "      <td>c</td>\n",
       "      <td>ma</td>\n",
       "      <td>f</td>\n",
       "      <td>The only values are NaN and moderators.</td>\n",
       "      <td>categorical</td>\n",
       "      <td>mod reactions to either post content or commen...</td>\n",
       "      <td>filtering</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>edited</td>\n",
       "      <td>t</td>\n",
       "      <td>aa</td>\n",
       "      <td>a</td>\n",
       "      <td>86% NaNs.</td>\n",
       "      <td>timestamp</td>\n",
       "      <td>author actions on the post (other than comment...</td>\n",
       "      <td>analysis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>crosspost_parent</td>\n",
       "      <td>s</td>\n",
       "      <td>p</td>\n",
       "      <td>t</td>\n",
       "      <td>Cross-post parent post id.</td>\n",
       "      <td>short text</td>\n",
       "      <td>post details</td>\n",
       "      <td>transform</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>crosspost_parent_list</td>\n",
       "      <td>l</td>\n",
       "      <td>p</td>\n",
       "      <td>t</td>\n",
       "      <td>This ultimately contains the body text of a cr...</td>\n",
       "      <td>long text</td>\n",
       "      <td>post details</td>\n",
       "      <td>transform</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>distinguished</td>\n",
       "      <td>c</td>\n",
       "      <td>aa</td>\n",
       "      <td>f</td>\n",
       "      <td>Mods can tag posts as distinguished, usually u...</td>\n",
       "      <td>categorical</td>\n",
       "      <td>author actions on the post (other than comment...</td>\n",
       "      <td>filtering</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>author_fullname</td>\n",
       "      <td>s</td>\n",
       "      <td>a</td>\n",
       "      <td>i</td>\n",
       "      <td>Unclear what this is, and lots of NaNs, but de...</td>\n",
       "      <td>short text</td>\n",
       "      <td>author info</td>\n",
       "      <td>id</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               field_name type reddit_role analysis_role  \\\n",
       "0                  author    s           a             i   \n",
       "1       author_flair_text    c           a             a   \n",
       "2             created_utc    t           p             a   \n",
       "3                  domain    s           p             a   \n",
       "4               full_link    u           p             r   \n",
       "5                      id    s           p             i   \n",
       "8                  locked    b          ma             a   \n",
       "9            num_comments    n          ga             a   \n",
       "10         num_crossposts    n          aa             a   \n",
       "11                over_18    b           p             a   \n",
       "13                 pinned    b          aa             a   \n",
       "14                  score    n          ga             a   \n",
       "15               selftext    l           p             a   \n",
       "16                spoiler    b          aa             a   \n",
       "17               stickied    b          aa             f   \n",
       "18                  title    l           p             a   \n",
       "19                    url    u           p             r   \n",
       "20              banned_by    c          ma             f   \n",
       "21                 edited    t          aa             a   \n",
       "22       crosspost_parent    s           p             t   \n",
       "23  crosspost_parent_list    l           p             t   \n",
       "24          distinguished    c          aa             f   \n",
       "26        author_fullname    s           a             i   \n",
       "\n",
       "                                                notes    type_long  \\\n",
       "0           Note the [deleted] and [removed] entries.   short text   \n",
       "1                                     31% NaN values.  categorical   \n",
       "2                                       Dups present.    timestamp   \n",
       "3              Domain where the post originated from.   short text   \n",
       "4                       A link to the post on Reddit.          url   \n",
       "5                                             Post id   short text   \n",
       "8      Only 6 values are true, everything else false.  binary flag   \n",
       "9                                                          numeric   \n",
       "10         Both NaN and zeros present. Few values >0.      numeric   \n",
       "11  98% false. Looks like a NSFW-type label on the...  binary flag   \n",
       "13      Users can pin up to 4 posts to their profile.  binary flag   \n",
       "14           The score is based on up and down votes.      numeric   \n",
       "15                      Can have [deleted] as values.    long text   \n",
       "16  Spoiler tags are used to mark spoiler content,...  binary flag   \n",
       "17  Mods can pin up to 2 of their own posts to the...  binary flag   \n",
       "18               Title of the post, can be very long.    long text   \n",
       "19  Url to the original post if crossposted or fro...          url   \n",
       "20            The only values are NaN and moderators.  categorical   \n",
       "21                                          86% NaNs.    timestamp   \n",
       "22                         Cross-post parent post id.   short text   \n",
       "23  This ultimately contains the body text of a cr...    long text   \n",
       "24  Mods can tag posts as distinguished, usually u...  categorical   \n",
       "26  Unclear what this is, and lots of NaNs, but de...   short text   \n",
       "\n",
       "                                     reddit_role_long analysis_role_long  \n",
       "0                                         author info                 id  \n",
       "1                                         author info           analysis  \n",
       "2                                        post details           analysis  \n",
       "3                                        post details           analysis  \n",
       "4                                        post details          reference  \n",
       "5                                        post details                 id  \n",
       "8   mod reactions to either post content or commen...           analysis  \n",
       "9       general subreddit users reactions to the post           analysis  \n",
       "10  author actions on the post (other than comment...           analysis  \n",
       "11                                       post details           analysis  \n",
       "13  author actions on the post (other than comment...           analysis  \n",
       "14      general subreddit users reactions to the post           analysis  \n",
       "15                                       post details           analysis  \n",
       "16  author actions on the post (other than comment...           analysis  \n",
       "17  author actions on the post (other than comment...          filtering  \n",
       "18                                       post details           analysis  \n",
       "19                                       post details          reference  \n",
       "20  mod reactions to either post content or commen...          filtering  \n",
       "21  author actions on the post (other than comment...           analysis  \n",
       "22                                       post details          transform  \n",
       "23                                       post details          transform  \n",
       "24  author actions on the post (other than comment...          filtering  \n",
       "26                                        author info                 id  "
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Drop the fields marked for it\n",
    "selected_fields_df = selected_fields_df[selected_fields_df['analysis_role'] != 'd']\n",
    "selected_fields_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8cafe05d-a1ee-4eb8-8543-b392f708961e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate an easier to read markdown version of the selected fields table\n",
    "\n",
    "# print(\n",
    "# selected_fields_df[['field_name', 'reddit_role_long', 'analysis_role_long', 'type_long', 'notes']]\n",
    "# .to_markdown()\n",
    "# )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea32e578-7ff8-4eb0-91f9-ff525a28b5e0",
   "metadata": {},
   "source": [
    "|    | field_name            | reddit_role_long                                                | analysis_role_long   | type_long   | notes                                                                                                                                                                                                                   |\n",
    "|---:|:----------------------|:----------------------------------------------------------------|:---------------------|:------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n",
    "|  0 | author                | author info                                                     | id                   | short text  | Note the [deleted] and [removed] entries.                                                                                                                                                                               |\n",
    "|  1 | author_flair_text     | author info                                                     | analysis             | categorical | 31% NaN values.                                                                                                                                                                                                         |\n",
    "|  2 | created_utc           | post details                                                    | analysis             | timestamp   | Dups present.                                                                                                                                                                                                           |\n",
    "|  3 | domain                | post details                                                    | analysis             | short text  | Domain where the post originated from.                                                                                                                                                                                  |\n",
    "|  4 | full_link             | post details                                                    | reference            | url         | A link to the post on Reddit.                                                                                                                                                                                           |\n",
    "|  5 | id                    | post details                                                    | id                   | short text  | Post id                                                                                                                                                                                                                 |\n",
    "|  8 | locked                | mod reactions to either post content or comments activity on it | analysis             | binary flag | Only 6 values are true, everything else false.                                                                                                                                                                          |\n",
    "|  9 | num_comments          | general subreddit users reactions to the post                   | analysis             | numeric     |                                                                                                                                                                                                                         |\n",
    "| 10 | num_crossposts        | author actions on the post (other than commenting)              | analysis             | numeric     | Both NaN and zeros present. Few values >0.                                                                                                                                                                              |\n",
    "| 11 | over_18               | post details                                                    | analysis             | binary flag | 98% false. Looks like a NSFW-type label on the post content.                                                                                                                                                            |\n",
    "| 13 | pinned                | author actions on the post (other than commenting)              | analysis             | binary flag | Users can pin up to 4 posts to their profile.                                                                                                                                                                           |\n",
    "| 14 | score                 | general subreddit users reactions to the post                   | analysis             | numeric     | The score is based on up and down votes.                                                                                                                                                                                |\n",
    "| 15 | selftext              | post details                                                    | analysis             | long text   | Can have [deleted] as values.                                                                                                                                                                                           |\n",
    "| 16 | spoiler               | author actions on the post (other than commenting)              | analysis             | binary flag | Spoiler tags are used to mark spoiler content, and they can blur the preview or thumbnails. Both mods and post authors can add a spoiler tag on a post. There were 30 true values in the sample, so decided to keep it. |\n",
    "| 17 | stickied              | author actions on the post (other than commenting)              | filtering            | binary flag | Mods can pin up to 2 of their own posts to the top of the subreddit. This tag used to be called announcements.                                                                                                          |\n",
    "| 18 | title                 | post details                                                    | analysis             | long text   | Title of the post, can be very long.                                                                                                                                                                                    |\n",
    "| 19 | url                   | post details                                                    | reference            | url         | Url to the original post if crossposted or from other source.                                                                                                                                                           |\n",
    "| 20 | banned_by             | mod reactions to either post content or comments activity on it | filtering            | categorical | The only values are NaN and moderators.                                                                                                                                                                                 |\n",
    "| 21 | edited                | author actions on the post (other than commenting)              | analysis             | timestamp   | 86% NaNs.                                                                                                                                                                                                               |\n",
    "| 22 | crosspost_parent      | post details                                                    | transform            | short text  | Cross-post parent post id.                                                                                                                                                                                              |\n",
    "| 23 | crosspost_parent_list | post details                                                    | transform            | long text   | This ultimately contains the body text of a crossposted post. Just have to pull if out of the list of dicts.                                                                                                            |\n",
    "| 24 | distinguished         | author actions on the post (other than commenting)              | filtering            | categorical | Mods can tag posts as distinguished, usually used for subreddit management. Use this field for filtering out these posts.                                                                                               |\n",
    "| 26 | author_fullname       | author info                                                     | id                   | short text  | Unclear what this is, and lots of NaNs, but decided to keep for now.                                                                                                                                                    |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f76ad753-d082-4192-bf1b-d398d521d320",
   "metadata": {},
   "source": [
    "## Final fields selection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "id": "826653e7-bde1-4062-a474-e42a0327b7ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "fname = 'reddit_submissions_selected_fields.csv'\n",
    "selected_fields_df.to_csv(fname, index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e162bcc5-c697-4d97-b44c-c327e55c9890",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}