lg_data_pub.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data resources on academic LibGuides\n",
    "\n",
    "Code used to analyze data for the paper \"Taking count: A computational analysis of data resources on academic LibGuides,\" by Cody Hennesy, Alicia Kubas, and Jenny McBurney (currently being prepared for submission).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import re\n",
    "import pickle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import scraped data\n",
    "Details about the collection of data from LibGuides is available in the full paper and the code used to scrape the original guides is available upon request (email chennesy@umn.edu).\n",
    "\n",
    "The data used below is available from the Data Repository for U of M (DRUM). (Link forthcoming)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#adjust the line below to point to the CSV file that you've downloaded from DRUM.\n",
    "df_links = pd.read_csv('drum/lg_data.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "resource_name_normalized    object\n",
       "resource_name_scraped       object\n",
       "resource_url                object\n",
       "resource_url_simple         object\n",
       "resource_domain             object\n",
       "guide_url                   object\n",
       "institution_site_id          int64\n",
       "dtype: object"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_links.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Descriptive statistics\n",
    "\n",
    "#### Number of resources\n",
    "\"Resources\" refers to the links collected from all guides. They include \"link assets\" and \"database assets\" (according to the terms used in LibGuides) as well as any other links added in the WYSIWYG tool in LibGuides. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of non-unique resources in dataset: 186,952 \n",
      "\n",
      "Unique resource names \n",
      "---------------------------------------------------------------------- \n",
      "Unique resource names, after initial cleaning but before normalization: 77094 \n",
      "Unique resource names after cleaning and normalization: 64131 \n",
      "Resource names normalized through cleaning: 12963 (16.81%)\n",
      "----------------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "print('Number of non-unique resources in dataset:', '{:,}'.format(len(df_links)), '\\n')\n",
    "print('Unique resource names',\n",
    "      '\\n----------------------------------------------------------------------',\n",
    "      '\\nUnique resource names, after initial cleaning but before normalization:', df_links['resource_name_scraped'].nunique(), \n",
    "      '\\nUnique resource names after cleaning and normalization:', df_links['resource_name_normalized'].nunique(), \n",
    "     '\\nResource names normalized through cleaning:', df_links['resource_name_scraped'].nunique() -  df_links['resource_name_normalized'].nunique(), '('\n",
    "     '{:.2%}'.format((df_links['resource_name_scraped'].nunique() -  df_links['resource_name_normalized'].nunique())/ df_links['resource_name_scraped'].nunique())+')'\n",
    "       '\\n----------------------------------------------------------------------')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Number of guide pages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of guide pages included in study: 10448\n"
     ]
    }
   ],
   "source": [
    "n_links_per_guide_page = df_links['guide_url'].value_counts()\n",
    "\n",
    "n_guides = df_links['guide_url'].nunique()\n",
    "print('Number of guide pages included in study:', n_guides)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Resources per institution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of institutions: 123 \n",
      "-------------------------------------------------------------\n",
      "\n",
      "Non-unique links per institution:\n",
      "------------------------------------------------------------- \n",
      "Maximum: 10799 \n",
      "Minimum: 21 \n",
      "Mean: 1519.93 \n",
      "Standard deviation 1395.24 \n",
      "-------------------------------------------------------------\n",
      "Unique links per institution:\n",
      "------------------------------------------------------------- \n",
      "Maximum: 3284 \n",
      "Minimum: 21 \n",
      "Mean: 791.96 \n",
      "Standard deviation 638.44 \n",
      "-------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "links_per_site = df_links.groupby('institution_site_id')['resource_name_normalized'].count().sort_values(ascending=False)\n",
    "unique_links_per_site = df_links.groupby('institution_site_id')['resource_name_normalized'].nunique().sort_values(ascending=False)\n",
    "\n",
    "print('Number of institutions:',\n",
    "      df_links['institution_site_id'].nunique(),\n",
    "     '\\n-------------------------------------------------------------\\n')\n",
    "\n",
    "for i, df in enumerate([links_per_site, unique_links_per_site]):\n",
    "    if i == 0:\n",
    "        print('Non-unique links per institution:')\n",
    "    else:\n",
    "        print('Unique links per institution:')\n",
    "    print(\n",
    "      '-------------------------------------------------------------',\n",
    "      '\\nMaximum:', df.max(),\n",
    "      '\\nMinimum:', df.min(),\n",
    "      '\\nMean:', '{:.2f}'.format(df.mean()), \n",
    "      '\\nStandard deviation', '{:.2f}'.format(df.std()),\n",
    "     '\\n-------------------------------------------------------------')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Guide pages per site"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Guide pages per institution:\n",
      "------------------------------------------------------------- \n",
      "Maximum: 336 \n",
      "Minimum: 1 \n",
      "Mean: 84.94 \n",
      "Standard deviation 64.15 \n",
      "-------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "unique_pages_per_site = df_links.groupby('institution_site_id')['guide_url'].nunique().sort_values(ascending=False)\n",
    "\n",
    "print('Guide pages per institution:')\n",
    "print(\n",
    "  '-------------------------------------------------------------',\n",
    "  '\\nMaximum:', unique_pages_per_site.max(),\n",
    "  '\\nMinimum:', unique_pages_per_site.min(),\n",
    "  '\\nMean:', '{:.2f}'.format(unique_pages_per_site.mean()), \n",
    "  '\\nStandard deviation', '{:.2f}'.format(unique_pages_per_site.std()),\n",
    " '\\n-------------------------------------------------------------')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Top 500 resources\n",
    "Creates a link_site_df dataframe containing the 500 most common resources by the 'resource_name_normalized' column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "#save the top unique resource names to new dataframe\n",
    "unique_names = df_links['resource_name_normalized'].value_counts()\n",
    "top_names = unique_names.to_frame()\n",
    "\n",
    "#unique site_ids to list\n",
    "site_ids = df_links['institution_site_id'].unique().tolist()\n",
    "\n",
    "#create empty df to populate with names and site_id counts and\n",
    "#assign site_ids as column names\n",
    "link_site_df = pd.DataFrame(columns=site_ids)\n",
    "\n",
    "#add name column and add top 500 databases as values\n",
    "link_site_df['name'] = top_names.index[0:500]\n",
    "\n",
    "#move name column to beginning of df\n",
    "cols = list(link_site_df.columns)\n",
    "cols = [cols[-1]] + cols[:-1]\n",
    "link_site_df = link_site_df[cols]\n",
    "\n",
    "#set name column as index\n",
    "link_site_df = link_site_df.set_index(['name'])\n",
    "\n",
    "#add counts of each resource to df\n",
    "\n",
    "#add value of 1 if name is found in site_id and 0 if name is not found for site_id\n",
    "for site in site_ids:\n",
    "    current_site = df_links[df_links['institution_site_id'] == site]\n",
    "    for name in top_names.index[0:500]:\n",
    "        \n",
    "        matches = current_site[current_site['resource_name_normalized'] == name]\n",
    "        if len(matches) >= 1:\n",
    "            link_site_df.loc[name, site] = 1\n",
    "        else:\n",
    "            link_site_df.loc[name, site] = 0\n",
    "\n",
    "#create columns for sum and percent of how many sites include each link name\n",
    "link_site_df['sum'] = link_site_df.sum(axis=1)\n",
    "link_site_df['percent'] = link_site_df['sum']/len(site_ids)\n",
    "link_site_df['percent_of_sites'] = (100. * link_site_df['percent']).round(2).astype(str) + '%'\n",
    "link_site_df = link_site_df.sort_values(by=['percent'], ascending=False)\n",
    "\n",
    "#add column with full counts per db\n",
    "link_site_df['count'] = ''\n",
    "\n",
    "#add counts for top 500 links to df\n",
    "for i,row in top_names[0:500].iterrows():\n",
    "    name = i\n",
    "    count = row['resource_name_normalized']\n",
    "    link_site_df.loc[name, 'count'] = count\n",
    "\n",
    "#add column with an example url that equals the most common (simplified) url found per resource name\n",
    "\n",
    "#create empty column\n",
    "link_site_df['url'] = ''\n",
    "\n",
    "#for each link in top_names, create a subset dataframe and then count the most common simple urls\n",
    "#then add the simple url next to the appropriate row in link_site_df\n",
    "\n",
    "for i,row in top_names[0:500].iterrows():\n",
    "    name = i\n",
    "    current_site = df_links[df_links['resource_name_normalized'] == i]\n",
    "    most_common_url = current_site['resource_url_simple'].value_counts()\n",
    "    mc_url = most_common_url.index[0]\n",
    "    link_site_df.loc[name, 'url'] = mc_url\n",
    "\n",
    "link_site_df.reset_index(inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this stage the 'name', 'sum', 'percent_of_sites', 'count', and 'url' columns were exported to CSV and added to a Google Sheet where co-authors annotated the sheet to note whether each of the top resources was a 'free', 'paid', or 'hybrid' resource."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Top 20 Resources (Sorted by percentage)\n",
    "Table of the twenty resources most commonly included across institutional sites."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Resource name</th>\n",
       "      <th>Percentage of institutions</th>\n",
       "      <th>Overall count</th>\n",
       "      <th>Example URL</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>icpsr</td>\n",
       "      <td>94.31%</td>\n",
       "      <td>1834</td>\n",
       "      <td>www.icpsr.umich.edu</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>data.gov</td>\n",
       "      <td>94.31%</td>\n",
       "      <td>954</td>\n",
       "      <td>www.data.gov</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>u.s. census bureau</td>\n",
       "      <td>93.5%</td>\n",
       "      <td>956</td>\n",
       "      <td>www.census.gov</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>u.s. census: data</td>\n",
       "      <td>90.24%</td>\n",
       "      <td>1145</td>\n",
       "      <td>data.census.gov/cedsci</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>undata (united nations)</td>\n",
       "      <td>88.62%</td>\n",
       "      <td>1054</td>\n",
       "      <td>data.un.org</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>nces (u.s. national center for education stati...</td>\n",
       "      <td>86.99%</td>\n",
       "      <td>546</td>\n",
       "      <td>nces.ed.gov</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>nchs (u.s. national center for health statistics)</td>\n",
       "      <td>86.99%</td>\n",
       "      <td>856</td>\n",
       "      <td>www.cdc.gov/nchs</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>u.s. census: american factfinder</td>\n",
       "      <td>83.74%</td>\n",
       "      <td>720</td>\n",
       "      <td>factfinder.census.gov/faces/nav/jsf/pages/inde...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>bls (u.s. bureau of labor statistics)</td>\n",
       "      <td>82.93%</td>\n",
       "      <td>535</td>\n",
       "      <td>www.bls.gov</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>roper center: ipoll</td>\n",
       "      <td>80.49%</td>\n",
       "      <td>1090</td>\n",
       "      <td>proxy2.library.illinois.edu/login?url=ropercen...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>eurostat (european commission)</td>\n",
       "      <td>80.49%</td>\n",
       "      <td>501</td>\n",
       "      <td>ec.europa.eu/eurostat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>bjs (u.s. bureau of justice statistics)</td>\n",
       "      <td>78.86%</td>\n",
       "      <td>437</td>\n",
       "      <td>www.bjs.gov</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>social explorer</td>\n",
       "      <td>78.86%</td>\n",
       "      <td>1430</td>\n",
       "      <td>www.socialexplorer.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>who: global health observatory (world health o...</td>\n",
       "      <td>78.05%</td>\n",
       "      <td>491</td>\n",
       "      <td>www.who.int/gho/en</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>statistical abstract of the united states (pro...</td>\n",
       "      <td>78.05%</td>\n",
       "      <td>1058</td>\n",
       "      <td>www.libraries.rutgers.edu/indexes/statabus</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>world bank: data</td>\n",
       "      <td>76.42%</td>\n",
       "      <td>545</td>\n",
       "      <td>data.worldbank.org</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>historical statistics of the united states (ca...</td>\n",
       "      <td>74.8%</td>\n",
       "      <td>684</td>\n",
       "      <td>hsus.cambridge.org/HSUSWeb</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>bea (u.s. bureau of economic analysis)</td>\n",
       "      <td>74.8%</td>\n",
       "      <td>422</td>\n",
       "      <td>www.bea.gov</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>imf data (international monetary fund)</td>\n",
       "      <td>74.8%</td>\n",
       "      <td>445</td>\n",
       "      <td>data.imf.org</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>cdc: data and statistics</td>\n",
       "      <td>73.17%</td>\n",
       "      <td>361</td>\n",
       "      <td>www.cdc.gov/DataStatistics</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        Resource name  \\\n",
       "0                                               icpsr   \n",
       "1                                            data.gov   \n",
       "2                                  u.s. census bureau   \n",
       "3                                   u.s. census: data   \n",
       "4                             undata (united nations)   \n",
       "5   nces (u.s. national center for education stati...   \n",
       "6   nchs (u.s. national center for health statistics)   \n",
       "7                    u.s. census: american factfinder   \n",
       "8               bls (u.s. bureau of labor statistics)   \n",
       "9                                 roper center: ipoll   \n",
       "10                     eurostat (european commission)   \n",
       "11            bjs (u.s. bureau of justice statistics)   \n",
       "12                                    social explorer   \n",
       "13  who: global health observatory (world health o...   \n",
       "14  statistical abstract of the united states (pro...   \n",
       "15                                   world bank: data   \n",
       "16  historical statistics of the united states (ca...   \n",
       "17             bea (u.s. bureau of economic analysis)   \n",
       "18             imf data (international monetary fund)   \n",
       "19                           cdc: data and statistics   \n",
       "\n",
       "   Percentage of institutions Overall count  \\\n",
       "0                      94.31%          1834   \n",
       "1                      94.31%           954   \n",
       "2                       93.5%           956   \n",
       "3                      90.24%          1145   \n",
       "4                      88.62%          1054   \n",
       "5                      86.99%           546   \n",
       "6                      86.99%           856   \n",
       "7                      83.74%           720   \n",
       "8                      82.93%           535   \n",
       "9                      80.49%          1090   \n",
       "10                     80.49%           501   \n",
       "11                     78.86%           437   \n",
       "12                     78.86%          1430   \n",
       "13                     78.05%           491   \n",
       "14                     78.05%          1058   \n",
       "15                     76.42%           545   \n",
       "16                      74.8%           684   \n",
       "17                      74.8%           422   \n",
       "18                      74.8%           445   \n",
       "19                     73.17%           361   \n",
       "\n",
       "                                          Example URL  \n",
       "0                                 www.icpsr.umich.edu  \n",
       "1                                        www.data.gov  \n",
       "2                                      www.census.gov  \n",
       "3                              data.census.gov/cedsci  \n",
       "4                                         data.un.org  \n",
       "5                                         nces.ed.gov  \n",
       "6                                    www.cdc.gov/nchs  \n",
       "7   factfinder.census.gov/faces/nav/jsf/pages/inde...  \n",
       "8                                         www.bls.gov  \n",
       "9   proxy2.library.illinois.edu/login?url=ropercen...  \n",
       "10                              ec.europa.eu/eurostat  \n",
       "11                                        www.bjs.gov  \n",
       "12                             www.socialexplorer.com  \n",
       "13                                 www.who.int/gho/en  \n",
       "14         www.libraries.rutgers.edu/indexes/statabus  \n",
       "15                                 data.worldbank.org  \n",
       "16                         hsus.cambridge.org/HSUSWeb  \n",
       "17                                        www.bea.gov  \n",
       "18                                       data.imf.org  \n",
       "19                         www.cdc.gov/DataStatistics  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "table_top_resources = link_site_df[['name','percent_of_sites', 'count', 'url']][0:20]\n",
    "table_top_resources = table_top_resources.rename(columns ={'name': 'Resource name', 'percent_of_sites':'Percentage of institutions', 'count':'Overall count', 'url':'Example URL'})\n",
    "table_top_resources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Top 10 Resources (Sorted by overall count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>percent_of_sites</th>\n",
       "      <th>count</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>icpsr</td>\n",
       "      <td>94.31%</td>\n",
       "      <td>1834</td>\n",
       "      <td>www.icpsr.umich.edu</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>statistical insight (proquest)</td>\n",
       "      <td>61.79%</td>\n",
       "      <td>1532</td>\n",
       "      <td>proxy2.library.illinois.edu/login?url=search.p...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>social explorer</td>\n",
       "      <td>78.86%</td>\n",
       "      <td>1430</td>\n",
       "      <td>www.socialexplorer.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53</th>\n",
       "      <td>data planet (sage)</td>\n",
       "      <td>50.41%</td>\n",
       "      <td>1318</td>\n",
       "      <td>go.openathens.net/redirector/fiu.edu?url=datap...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>u.s. census: data</td>\n",
       "      <td>90.24%</td>\n",
       "      <td>1145</td>\n",
       "      <td>data.census.gov/cedsci</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>statista</td>\n",
       "      <td>64.23%</td>\n",
       "      <td>1104</td>\n",
       "      <td>go.openathens.net/redirector/fiu.edu?url=http%...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>roper center: ipoll</td>\n",
       "      <td>80.49%</td>\n",
       "      <td>1090</td>\n",
       "      <td>proxy2.library.illinois.edu/login?url=ropercen...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>statistical abstract of the united states (pro...</td>\n",
       "      <td>78.05%</td>\n",
       "      <td>1058</td>\n",
       "      <td>www.libraries.rutgers.edu/indexes/statabus</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>undata (united nations)</td>\n",
       "      <td>88.62%</td>\n",
       "      <td>1054</td>\n",
       "      <td>data.un.org</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>u.s. census bureau</td>\n",
       "      <td>93.5%</td>\n",
       "      <td>956</td>\n",
       "      <td>www.census.gov</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 name percent_of_sites count  \\\n",
       "0                                               icpsr           94.31%  1834   \n",
       "31                     statistical insight (proquest)           61.79%  1532   \n",
       "12                                    social explorer           78.86%  1430   \n",
       "53                                 data planet (sage)           50.41%  1318   \n",
       "3                                   u.s. census: data           90.24%  1145   \n",
       "28                                           statista           64.23%  1104   \n",
       "9                                 roper center: ipoll           80.49%  1090   \n",
       "14  statistical abstract of the united states (pro...           78.05%  1058   \n",
       "4                             undata (united nations)           88.62%  1054   \n",
       "2                                  u.s. census bureau            93.5%   956   \n",
       "\n",
       "                                                  url  \n",
       "0                                 www.icpsr.umich.edu  \n",
       "31  proxy2.library.illinois.edu/login?url=search.p...  \n",
       "12                             www.socialexplorer.com  \n",
       "53  go.openathens.net/redirector/fiu.edu?url=datap...  \n",
       "3                              data.census.gov/cedsci  \n",
       "28  go.openathens.net/redirector/fiu.edu?url=http%...  \n",
       "9   proxy2.library.illinois.edu/login?url=ropercen...  \n",
       "14         www.libraries.rutgers.edu/indexes/statabus  \n",
       "4                                         data.un.org  \n",
       "2                                      www.census.gov  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_10 = link_site_df[['name','percent_of_sites', 'count', 'url']].sort_values(by='count', ascending=False)[0:10]\n",
    "top_10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Top 10 (full) Domains"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Domain</th>\n",
       "      <th>resource_domain</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>census.gov</td>\n",
       "      <td>7656</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>cdc.gov</td>\n",
       "      <td>5382</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>nces.ed.gov</td>\n",
       "      <td>2546</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>icpsr.umich.edu</td>\n",
       "      <td>2222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>bls.gov</td>\n",
       "      <td>1928</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>go.openathens.net</td>\n",
       "      <td>1837</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>who.int</td>\n",
       "      <td>1202</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>utk.primo.exlibrisgroup.com</td>\n",
       "      <td>1128</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>data.worldbank.org</td>\n",
       "      <td>1120</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>nrs.harvard.edu</td>\n",
       "      <td>1120</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        Domain  resource_domain\n",
       "0                   census.gov             7656\n",
       "1                      cdc.gov             5382\n",
       "2                  nces.ed.gov             2546\n",
       "3              icpsr.umich.edu             2222\n",
       "4                      bls.gov             1928\n",
       "5            go.openathens.net             1837\n",
       "6                      who.int             1202\n",
       "7  utk.primo.exlibrisgroup.com             1128\n",
       "8           data.worldbank.org             1120\n",
       "9              nrs.harvard.edu             1120"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unique_domain_counts = df_links['resource_domain'].value_counts()\n",
    "domains_top = unique_domain_counts.to_frame()\n",
    "domains_top = domains_top.reset_index()\n",
    "domains_top = domains_top.rename(columns = {'index': 'Domain', 'domain': 'Count'})\n",
    "domains_top[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Top 20 Root Domains\n",
    "Note that root domains such as harvard.edu, exlibrisgroup.com, upenn.edu, cornell.edu, openathens.net, and msu.edu are not included in the final paper Table since the most common resources included from those domains were not relevant to our analysis. See full output below for example subdomains for each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Root domain</th>\n",
       "      <th>Count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>census.gov</td>\n",
       "      <td>9670</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>cdc.gov</td>\n",
       "      <td>6072</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>harvard.edu</td>\n",
       "      <td>4480</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>umich.edu</td>\n",
       "      <td>3691</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ed.gov</td>\n",
       "      <td>3122</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>exlibrisgroup.com</td>\n",
       "      <td>2744</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>worldbank.org</td>\n",
       "      <td>2705</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>un.org</td>\n",
       "      <td>2636</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>upenn.edu</td>\n",
       "      <td>2319</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>cornell.edu</td>\n",
       "      <td>2155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>bls.gov</td>\n",
       "      <td>2066</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>openathens.net</td>\n",
       "      <td>1861</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>usda.gov</td>\n",
       "      <td>1853</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>nih.gov</td>\n",
       "      <td>1570</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>msu.edu</td>\n",
       "      <td>1421</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>who.int</td>\n",
       "      <td>1411</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>princeton.edu</td>\n",
       "      <td>1347</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>data.gov</td>\n",
       "      <td>1242</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>oclc.org</td>\n",
       "      <td>1240</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>unc.edu</td>\n",
       "      <td>1238</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Root domain  Count\n",
       "0          census.gov   9670\n",
       "1             cdc.gov   6072\n",
       "2         harvard.edu   4480\n",
       "3           umich.edu   3691\n",
       "4              ed.gov   3122\n",
       "5   exlibrisgroup.com   2744\n",
       "6       worldbank.org   2705\n",
       "7              un.org   2636\n",
       "8           upenn.edu   2319\n",
       "9         cornell.edu   2155\n",
       "10            bls.gov   2066\n",
       "11     openathens.net   1861\n",
       "12           usda.gov   1853\n",
       "13            nih.gov   1570\n",
       "14            msu.edu   1421\n",
       "15            who.int   1411\n",
       "16      princeton.edu   1347\n",
       "17           data.gov   1242\n",
       "18           oclc.org   1240\n",
       "19            unc.edu   1238"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#separate the top level domain (this is slightly coarse but works for the top sites)\n",
    "df_links['top_level_domain'] = df_links.resource_domain.str.split(r'([^.\\s]+\\.[^.\\s]+)$').str[1]\n",
    "top_level_domain_counts = df_links['top_level_domain'].value_counts()\n",
    "\n",
    "root_domains_top = top_level_domain_counts.to_frame()\n",
    "root_domains_top = root_domains_top.reset_index()\n",
    "root_domains_top = root_domains_top.rename(columns = {'index': 'Root domain', 'top_level_domain': 'Count'})\n",
    "root_domains_top[0:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Print the ten most common domains for each of the top 20 root domains"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "census.gov \n",
      "------------------------------------------------\n",
      " census.gov total count: 9670 \n",
      " census.gov # of domains: 42 \n",
      " census.gov mean count per domain: 230.24 \n",
      "\n",
      "                       Domain  resource_domain\n",
      "0                 census.gov             7656\n",
      "14           data.census.gov              945\n",
      "64     factfinder.census.gov              343\n",
      "75    factfinder2.census.gov              316\n",
      "466   dataferrett.census.gov               61\n",
      "626      usatrade.census.gov               45\n",
      "674      lehd.ces.census.gov               41\n",
      "680    quickfacts.census.gov               41\n",
      "814           ask.census.gov               33\n",
      "965  onthemap.ces.census.gov               27 \n",
      "------------------------------------------------\n",
      "\n",
      "cdc.gov \n",
      "------------------------------------------------\n",
      " cdc.gov total count: 6072 \n",
      " cdc.gov # of domains: 26 \n",
      " cdc.gov mean count per domain: 233.54 \n",
      "\n",
      "                              Domain  resource_domain\n",
      "1                           cdc.gov             5382\n",
      "65                   wonder.cdc.gov              341\n",
      "335                    wwwn.cdc.gov               83\n",
      "414                    data.cdc.gov               68\n",
      "757                   atsdr.cdc.gov               36\n",
      "787             chronicdata.cdc.gov               34\n",
      "793                    nccd.cdc.gov               34\n",
      "1588  www-cdc-gov.proxy.lib.fsu.edu               15\n",
      "1737                    svi.cdc.gov               14\n",
      "2537            ephtracking.cdc.gov                9 \n",
      "------------------------------------------------\n",
      "\n",
      "harvard.edu \n",
      "------------------------------------------------\n",
      " harvard.edu total count: 4480 \n",
      " harvard.edu # of domains: 243 \n",
      " harvard.edu mean count per domain: 18.44 \n",
      "\n",
      "                                        Domain  resource_domain\n",
      "9                             nrs.harvard.edu             1120\n",
      "11                         id.lib.harvard.edu             1069\n",
      "33                      dataverse.harvard.edu              478\n",
      "49                 guides.library.harvard.edu              382\n",
      "225                        hollis.harvard.edu              123\n",
      "416                       library.harvard.edu               68\n",
      "429                          wiki.harvard.edu               67\n",
      "542                   projects.iq.harvard.edu               51\n",
      "564  heinonline.org.ezp-prod1.hul.harvard.edu               50\n",
      "631                           law.harvard.edu               45 \n",
      "------------------------------------------------\n",
      "\n",
      "umich.edu \n",
      "------------------------------------------------\n",
      " umich.edu total count: 3691 \n",
      " umich.edu # of domains: 91 \n",
      " umich.edu mean count per domain: 40.56 \n",
      "\n",
      "                                   Domain  resource_domain\n",
      "3                        icpsr.umich.edu             2222\n",
      "45                         lib.umich.edu              395\n",
      "78                  guides.lib.umich.edu              304\n",
      "110                   apps.lib.umich.edu              239\n",
      "207                 search.lib.umich.edu              133\n",
      "411             psidonline.isr.umich.edu               69\n",
      "513                  proxy.lib.umich.edu               54\n",
      "552   www-icpsr-umich-edu.ezproxy.bu.edu               51\n",
      "709              hrsonline.isr.umich.edu               39\n",
      "1084                       law.umich.edu               24 \n",
      "------------------------------------------------\n",
      "\n",
      "ed.gov \n",
      "------------------------------------------------\n",
      " ed.gov total count: 3122 \n",
      " ed.gov # of domains: 31 \n",
      " ed.gov mean count per domain: 100.71 \n",
      "\n",
      "                        Domain  resource_domain\n",
      "2                 nces.ed.gov             2546\n",
      "158                    ed.gov              184\n",
      "260                ope.ed.gov              109\n",
      "433                ies.ed.gov               66\n",
      "517      eddataexpress.ed.gov               54\n",
      "711            ocrdata.ed.gov               39\n",
      "782   collegescorecard.ed.gov               34\n",
      "840          dashboard.ed.gov               32\n",
      "1427              eric.ed.gov               17\n",
      "2198            p12.nysed.gov               11 \n",
      "------------------------------------------------\n",
      "\n",
      "exlibrisgroup.com \n",
      "------------------------------------------------\n",
      " exlibrisgroup.com total count: 2744 \n",
      " exlibrisgroup.com # of domains: 74 \n",
      " exlibrisgroup.com mean count per domain: 37.08 \n",
      "\n",
      "                                              Domain  resource_domain\n",
      "7                       utk.primo.exlibrisgroup.com             1128\n",
      "228  galileo-usg-gsu-primo.hosted.exlibrisgroup.com              122\n",
      "233                    pitt.primo.exlibrisgroup.com              120\n",
      "238             i-share-uiu.primo.exlibrisgroup.com              119\n",
      "268         alliance-primo.hosted.exlibrisgroup.com              106\n",
      "322               ou-primo.hosted.exlibrisgroup.com               87\n",
      "347               bc-primo.hosted.exlibrisgroup.com               80\n",
      "383          pascal-clemson.primo.exlibrisgroup.com               72\n",
      "421                buprimo.hosted.exlibrisgroup.com               68\n",
      "505         utah-primoprod.hosted.exlibrisgroup.com               55 \n",
      "------------------------------------------------\n",
      "\n",
      "worldbank.org \n",
      "------------------------------------------------\n",
      " worldbank.org total count: 2705 \n",
      " worldbank.org # of domains: 56 \n",
      " worldbank.org mean count per domain: 48.30 \n",
      "\n",
      "                           Domain  resource_domain\n",
      "8             data.worldbank.org             1120\n",
      "41        databank.worldbank.org              423\n",
      "125     datatopics.worldbank.org              220\n",
      "136                worldbank.org              205\n",
      "137    datacatalog.worldbank.org              205\n",
      "271            web.worldbank.org              106\n",
      "561  openknowledge.worldbank.org               50\n",
      "571      microdata.worldbank.org               50\n",
      "594             go.worldbank.org               48\n",
      "610           econ.worldbank.org               46 \n",
      "------------------------------------------------\n",
      "\n",
      "un.org \n",
      "------------------------------------------------\n",
      " un.org total count: 2636 \n",
      " un.org # of domains: 40 \n",
      " un.org mean count per domain: 65.90 \n",
      "\n",
      "                      Domain  resource_domain\n",
      "10           unstats.un.org             1076\n",
      "13              data.un.org              977\n",
      "131                  un.org              214\n",
      "190         comtrade.un.org              154\n",
      "736             mdgs.un.org               37\n",
      "784              esa.un.org               34\n",
      "1295       dataunodc.un.org               19\n",
      "1447  digitallibrary.un.org               17\n",
      "1735        research.un.org               14\n",
      "1857         library.un.org               13 \n",
      "------------------------------------------------\n",
      "\n",
      "upenn.edu \n",
      "------------------------------------------------\n",
      " upenn.edu total count: 2319 \n",
      " upenn.edu # of domains: 54 \n",
      " upenn.edu mean count per domain: 42.94 \n",
      "\n",
      "                           Domain  resource_domain\n",
      "15         hdl.library.upenn.edu              882\n",
      "80       proxy.library.upenn.edu              300\n",
      "107     guides.library.upenn.edu              245\n",
      "114   franklin.library.upenn.edu              235\n",
      "127   wrds-www.wharton.upenn.edu              218\n",
      "144   wrds-web.wharton.upenn.edu              197\n",
      "460       wrds.wharton.upenn.edu               63\n",
      "1262       dla.library.upenn.edu               20\n",
      "1308       catalog.ldc.upenn.edu               19\n",
      "1421               ldc.upenn.edu               18 \n",
      "------------------------------------------------\n",
      "\n",
      "cornell.edu \n",
      "------------------------------------------------\n",
      " cornell.edu total count: 2155 \n",
      " cornell.edu # of domains: 86 \n",
      " cornell.edu mean count per domain: 25.06 \n",
      "\n",
      "                              Domain  resource_domain\n",
      "19       guides.library.cornell.edu              661\n",
      "51     resolver.library.cornell.edu              380\n",
      "73         usda.mannlib.cornell.edu              320\n",
      "134         ropercenter.cornell.edu              208\n",
      "203  newcatalog.library.cornell.edu              140\n",
      "515               ciser.cornell.edu               54\n",
      "609              ndacan.cornell.edu               46\n",
      "613        usda.library.cornell.edu               46\n",
      "703                 law.cornell.edu               40\n",
      "887       archive.ciser.cornell.edu               30 \n",
      "------------------------------------------------\n",
      "\n",
      "bls.gov \n",
      "------------------------------------------------\n",
      " bls.gov total count: 2066 \n",
      " bls.gov # of domains: 4 \n",
      " bls.gov mean count per domain: 516.50 \n",
      "\n",
      "              Domain  resource_domain\n",
      "4           bls.gov             1928\n",
      "232   stats.bls.gov              121\n",
      "1997   data.bls.gov               12\n",
      "4190   beta.bls.gov                5 \n",
      "------------------------------------------------\n",
      "\n",
      "openathens.net \n",
      "------------------------------------------------\n",
      " openathens.net total count: 1861 \n",
      " openathens.net # of domains: 8 \n",
      " openathens.net mean count per domain: 232.62 \n",
      "\n",
      "                                                   Domain  resource_domain\n",
      "5                                      go.openathens.net             1837\n",
      "1863                              connect.openathens.net               13\n",
      "6524        www-icpsr-umich-edu.eu1.proxy.openathens.net                3\n",
      "8343    app-simplyanalytics-com.eu1.proxy.openathens.net                2\n",
      "8660   statisticaldatasets-data--planet-com.eu1.proxy...                2\n",
      "9950    ropercenter-cornell-edu.eu1.proxy.openathens.net                2\n",
      "12409        www-ccdc-cam-ac-uk.eu1.proxy.openathens.net                1\n",
      "18576              www-rkma-com.eu1.proxy.openathens.net                1 \n",
      "------------------------------------------------\n",
      "\n",
      "usda.gov \n",
      "------------------------------------------------\n",
      " usda.gov total count: 1853 \n",
      " usda.gov # of domains: 69 \n",
      " usda.gov mean count per domain: 26.86 \n",
      "\n",
      "                         Domain  resource_domain\n",
      "26               nass.usda.gov              530\n",
      "67                ers.usda.gov              329\n",
      "183          agcensus.usda.gov              165\n",
      "258               fas.usda.gov              110\n",
      "280  datagateway.nrcs.usda.gov              103\n",
      "340                   usda.gov               82\n",
      "458               ars.usda.gov               63\n",
      "493               fns.usda.gov               57\n",
      "590              nrcs.usda.gov               48\n",
      "776               nal.usda.gov               35 \n",
      "------------------------------------------------\n",
      "\n",
      "nih.gov \n",
      "------------------------------------------------\n",
      " nih.gov total count: 1570 \n",
      " nih.gov # of domains: 117 \n",
      " nih.gov mean count per domain: 13.42 \n",
      "\n",
      "                         Domain  resource_domain\n",
      "34            ncbi.nlm.nih.gov              477\n",
      "47                 nlm.nih.gov              392\n",
      "226               nimh.nih.gov              123\n",
      "596   pubchem.ncbi.nlm.nih.gov               47\n",
      "683          hsric.nlm.nih.gov               41\n",
      "734                    nih.gov               37\n",
      "836              niddk.nih.gov               32\n",
      "875          wwwcf.nlm.nih.gov               30\n",
      "1132        toxnet.nlm.nih.gov               23\n",
      "1181            grants.nih.gov               22 \n",
      "------------------------------------------------\n",
      "\n",
      "msu.edu \n",
      "------------------------------------------------\n",
      " msu.edu total count: 1421 \n",
      " msu.edu # of domains: 78 \n",
      " msu.edu mean count per domain: 18.22 \n",
      "\n",
      "                      Domain  resource_domain\n",
      "52      catalog.lib.msu.edu              379\n",
      "85          ezproxy.msu.edu              288\n",
      "117     archive.lib.msu.edu              234\n",
      "129   libguides.lib.msu.edu              214\n",
      "358      globaledge.msu.edu               77\n",
      "789           ippsr.msu.edu               34\n",
      "1020         er.lib.msu.edu               26\n",
      "1137            lib.msu.edu               23\n",
      "2035            opb.msu.edu               11\n",
      "2607      staff.lib.msu.edu                9 \n",
      "------------------------------------------------\n",
      "\n",
      "who.int \n",
      "------------------------------------------------\n",
      " who.int total count: 1411 \n",
      " who.int # of domains: 14 \n",
      " who.int mean count per domain: 100.79 \n",
      "\n",
      "                     Domain  resource_domain\n",
      "6                  who.int             1202\n",
      "182           apps.who.int              168\n",
      "2007          euro.who.int               12\n",
      "2912   gamapserver.who.int                7\n",
      "3053       covid19.who.int                7\n",
      "4952          www3.who.int                4\n",
      "6074          afro.who.int                3\n",
      "7278      rho.emro.who.int                2\n",
      "11891     extranet.who.int                1\n",
      "12871         iarc.who.int                1 \n",
      "------------------------------------------------\n",
      "\n",
      "princeton.edu \n",
      "------------------------------------------------\n",
      " princeton.edu total count: 1347 \n",
      " princeton.edu # of domains: 46 \n",
      " princeton.edu mean count per domain: 29.28 \n",
      "\n",
      "                                          Domain  resource_domain\n",
      "31                        library.princeton.edu              481\n",
      "118                     libguides.princeton.edu              233\n",
      "188                       catalog.princeton.edu              155\n",
      "514                           opr.princeton.edu               54\n",
      "578                   faq.library.princeton.edu               49\n",
      "689                  mediacentral.princeton.edu               41\n",
      "753                       mmp.opr.princeton.edu               36\n",
      "764                          dss2.princeton.edu               35\n",
      "1028                     popindex.princeton.edu               26\n",
      "1060  methods-sagepub-com.ezproxy.princeton.edu               25 \n",
      "------------------------------------------------\n",
      "\n",
      "data.gov \n",
      "------------------------------------------------\n",
      " data.gov total count: 1242 \n",
      " data.gov # of domains: 52 \n",
      " data.gov mean count per domain: 23.88 \n",
      "\n",
      "                 Domain  resource_domain\n",
      "12            data.gov             1032\n",
      "143     healthdata.gov              198\n",
      "162   catalog.data.gov              183\n",
      "702        data.gov.uk               40\n",
      "1442       data.gov.au               17\n",
      "1681       data.gov.in               14\n",
      "2645      geo.data.gov                8\n",
      "3123   health.data.gov                7\n",
      "4330       data.gov.ua                4\n",
      "4569  gos2.geodata.gov                4 \n",
      "------------------------------------------------\n",
      "\n",
      "oclc.org \n",
      "------------------------------------------------\n",
      " oclc.org total count: 1240 \n",
      " oclc.org # of domains: 48 \n",
      " oclc.org mean count per domain: 25.83 \n",
      "\n",
      "                       Domain  resource_domain\n",
      "53          cmu.idm.oclc.org              375\n",
      "123          uc.idm.oclc.org              224\n",
      "139        pitt.idm.oclc.org              200\n",
      "196        yale.idm.oclc.org              147\n",
      "263         unh.idm.oclc.org              108\n",
      "937   dartmouth.idm.oclc.org               28\n",
      "1096        wvu.idm.oclc.org               24\n",
      "1538                oclc.org               16\n",
      "1564   colorado.idm.oclc.org               16\n",
      "1715       utep.idm.oclc.org               14 \n",
      "------------------------------------------------\n",
      "\n",
      "unc.edu \n",
      "------------------------------------------------\n",
      " unc.edu total count: 1238 \n",
      " unc.edu # of domains: 81 \n",
      " unc.edu mean count per domain: 15.28 \n",
      "\n",
      "                      Domain  resource_domain\n",
      "25       guides.lib.unc.edu              547\n",
      "283     catalog.lib.unc.edu              102\n",
      "407       dataverse.unc.edu               69\n",
      "422         library.unc.edu               67\n",
      "476    libproxy.lib.unc.edu               60\n",
      "481             cpc.unc.edu               59\n",
      "684        arc.irss.unc.edu               41\n",
      "893            odum.unc.edu               30\n",
      "1291           irss.unc.edu               20\n",
      "1408  garymarks.web.unc.edu               18 \n",
      "------------------------------------------------\n",
      "\n",
      "europa.eu \n",
      "------------------------------------------------\n",
      " europa.eu total count: 1233 \n",
      " europa.eu # of domains: 52 \n",
      " europa.eu mean count per domain: 23.71 \n",
      "\n",
      "                          Domain  resource_domain\n",
      "17                 ec.europa.eu              766\n",
      "227   epp.eurostat.ec.europa.eu              122\n",
      "496              data.europa.eu               57\n",
      "511                   europa.eu               55\n",
      "642               eea.europa.eu               44\n",
      "936          eacea.ec.europa.eu               28\n",
      "1904          eur-lex.europa.eu               12\n",
      "1995             ecdc.europa.eu               12\n",
      "2092            curia.europa.eu               11\n",
      "2241              ecb.europa.eu               10 \n",
      "------------------------------------------------\n",
      "\n",
      "google.com \n",
      "------------------------------------------------\n",
      " google.com total count: 1190 \n",
      " google.com # of domains: 32 \n",
      " google.com mean count per domain: 37.19 \n",
      "\n",
      "                                  Domain  resource_domain\n",
      "32                           google.com              481\n",
      "157                  toolbox.google.com              185\n",
      "296                     docs.google.com               97\n",
      "401   datasetsearch.research.google.com               70\n",
      "474                    drive.google.com               60\n",
      "486                    sites.google.com               59\n",
      "544                  support.google.com               51\n",
      "713                    books.google.com               39\n",
      "716                  scholar.google.com               38\n",
      "1174                    maps.google.com               22 \n",
      "------------------------------------------------\n",
      "\n",
      "missouri.edu \n",
      "------------------------------------------------\n",
      " missouri.edu total count: 1183 \n",
      " missouri.edu # of domains: 39 \n",
      " missouri.edu mean count per domain: 30.33 \n",
      "\n",
      "                           Domain  resource_domain\n",
      "24       laurel.lso.missouri.edu              549\n",
      "133   libraryguides.missouri.edu              211\n",
      "236       proxy.mul.missouri.edu              119\n",
      "332            mcdc.missouri.edu               85\n",
      "386         library.missouri.edu               72\n",
      "1027          mcdc2.missouri.edu               26\n",
      "1321          msdis.missouri.edu               19\n",
      "1889          oseda.missouri.edu               13\n",
      "2428         census.missouri.edu                9\n",
      "2632         mobius.missouri.edu                8 \n",
      "------------------------------------------------\n",
      "\n",
      "usgs.gov \n",
      "------------------------------------------------\n",
      " usgs.gov total count: 1148 \n",
      " usgs.gov # of domains: 100 \n",
      " usgs.gov mean count per domain: 11.48 \n",
      "\n",
      "                       Domain  resource_domain\n",
      "71                  usgs.gov              323\n",
      "264   earthexplorer.usgs.gov              108\n",
      "338           ngmdb.usgs.gov               83\n",
      "436           water.usgs.gov               65\n",
      "477         pubs.er.usgs.gov               60\n",
      "484            pubs.usgs.gov               59\n",
      "708       waterdata.usgs.gov               39\n",
      "802            eros.usgs.gov               33\n",
      "1023         energy.usgs.gov               26\n",
      "1280      landcover.usgs.gov               20 \n",
      "------------------------------------------------\n",
      "\n",
      "berkeley.edu \n",
      "------------------------------------------------\n",
      " berkeley.edu total count: 1118 \n",
      " berkeley.edu # of domains: 67 \n",
      " berkeley.edu mean count per domain: 16.69 \n",
      "\n",
      "                            Domain  resource_domain\n",
      "38          libproxy.berkeley.edu              440\n",
      "148       guides.lib.berkeley.edu              191\n",
      "275          oskicat.berkeley.edu              104\n",
      "352              sda.berkeley.edu               79\n",
      "501              lib.berkeley.edu               56\n",
      "688   search.library.berkeley.edu               41\n",
      "902             dlab.berkeley.edu               29\n",
      "1600             gif.berkeley.edu               15\n",
      "1613          ucdata.berkeley.edu               15\n",
      "1700    researchdata.berkeley.edu               14 \n",
      "------------------------------------------------\n",
      "\n",
      "arcgis.com \n",
      "------------------------------------------------\n",
      " arcgis.com total count: 1109 \n",
      " arcgis.com # of domains: 273 \n",
      " arcgis.com mean count per domain: 4.06 \n",
      "\n",
      "                                      Domain  resource_domain\n",
      "189                              arcgis.com              154\n",
      "562                      desktop.arcgis.com               50\n",
      "623                  livingatlas.arcgis.com               45\n",
      "845                          hub.arcgis.com               31\n",
      "955                    storymaps.arcgis.com               28\n",
      "1000  hifld-geoplatform.opendata.arcgis.com               26\n",
      "1089                         bao.arcgis.com               24\n",
      "1221            communityanalyst.arcgis.com               21\n",
      "1390               yalemaps.maps.arcgis.com               18\n",
      "1480          gis-txdot.opendata.arcgis.com               17 \n",
      "------------------------------------------------\n",
      "\n",
      "bu.edu \n",
      "------------------------------------------------\n",
      " bu.edu total count: 1104 \n",
      " bu.edu # of domains: 40 \n",
      " bu.edu mean count per domain: 27.60 \n",
      "\n",
      "                                     Domain  resource_domain\n",
      "16                          library.bu.edu              862\n",
      "343                                 bu.edu               81\n",
      "552     www-icpsr-umich-edu.ezproxy.bu.edu               51\n",
      "1151               lawlibraryguides.bu.edu               22\n",
      "1622                           bumc.bu.edu               15\n",
      "2957        icpsr.umich.edu.ezproxy.bu.edu                7\n",
      "3972                           open.bu.edu                5\n",
      "4043      link-springer-com.ezproxy.bu.edu                5\n",
      "4773                           math.bu.edu                4\n",
      "4787  oxfordscholarship.com.ezproxy.bu.edu                4 \n",
      "------------------------------------------------\n",
      "\n",
      "wisc.edu \n",
      "------------------------------------------------\n",
      " wisc.edu total count: 1104 \n",
      " wisc.edu # of domains: 64 \n",
      " wisc.edu mean count per domain: 17.25 \n",
      "\n",
      "                                Domain  resource_domain\n",
      "43    researchguides.library.wisc.edu              415\n",
      "145           search.library.wisc.edu              195\n",
      "167          digital.library.wisc.edu              178\n",
      "570                      ssc.wisc.edu               50\n",
      "611                     disc.wisc.edu               46\n",
      "872                  library.wisc.edu               30\n",
      "1039                     irp.wisc.edu               25\n",
      "1423                     sco.wisc.edu               18\n",
      "1734                  nelson.wisc.edu               14\n",
      "2006               dpls.dacc.wisc.edu               12 \n",
      "------------------------------------------------\n",
      "\n",
      "illinois.edu \n",
      "------------------------------------------------\n",
      " illinois.edu total count: 984 \n",
      " illinois.edu # of domains: 60 \n",
      " illinois.edu mean count per domain: 16.40 \n",
      "\n",
      "                                Domain  resource_domain\n",
      "27        proxy2.library.illinois.edu              514\n",
      "191              library.illinois.edu              152\n",
      "355       guides.library.illinois.edu               78\n",
      "843                 stat.illinois.edu               31\n",
      "1256    archives.library.illinois.edu               20\n",
      "1445         clinecenter.illinois.edu               17\n",
      "1845           sfx.carli.illinois.edu               13\n",
      "1877                isws.illinois.edu               13\n",
      "2186     search.grainger.illinois.edu               11\n",
      "2271  clearinghouse.isgs.illinois.edu               10 \n",
      "------------------------------------------------\n",
      "\n",
      "nyu.edu \n",
      "------------------------------------------------\n",
      " nyu.edu total count: 945 \n",
      " nyu.edu # of domains: 37 \n",
      " nyu.edu mean count per domain: 25.54 \n",
      "\n",
      "                           Domain  resource_domain\n",
      "35    persistent.library.nyu.edu              461\n",
      "105               guides.nyu.edu              246\n",
      "434       bobcat.library.nyu.edu               66\n",
      "998        getit.library.nyu.edu               27\n",
      "1068             library.nyu.edu               24\n",
      "1097       proxy.library.nyu.edu               23\n",
      "1101        arch.library.nyu.edu               23\n",
      "2008                 law.nyu.edu               12\n",
      "2343     datacatalog.med.nyu.edu               10\n",
      "2556                 geo.nyu.edu                9 \n",
      "------------------------------------------------\n",
      "\n",
      "noaa.gov \n",
      "------------------------------------------------\n",
      " noaa.gov total count: 886 \n",
      " noaa.gov # of domains: 74 \n",
      " noaa.gov mean count per domain: 11.97 \n",
      "\n",
      "                        Domain  resource_domain\n",
      "82              ncdc.noaa.gov              297\n",
      "214             ngdc.noaa.gov              128\n",
      "427            coast.noaa.gov               67\n",
      "777             nodc.noaa.gov               35\n",
      "829                  noaa.gov               32\n",
      "1015            ncei.noaa.gov               26\n",
      "1159             nws.noaa.gov               22\n",
      "1274  nauticalcharts.noaa.gov               20\n",
      "1316       maps.ngdc.noaa.gov               19\n",
      "1484             nhc.noaa.gov               17 \n",
      "------------------------------------------------\n",
      "\n",
      "ucf.edu \n",
      "------------------------------------------------\n",
      " ucf.edu total count: 877 \n",
      " ucf.edu # of domains: 47 \n",
      " ucf.edu mean count per domain: 18.66 \n",
      "\n",
      "                              Domain  resource_domain\n",
      "22                   guides.ucf.edu              595\n",
      "279                 library.ucf.edu              104\n",
      "620           stars.library.ucf.edu               46\n",
      "1149                  ucftv.ucf.edu               22\n",
      "2279                    ikm.ucf.edu               10\n",
      "2283                 police.ucf.edu               10\n",
      "2977  login.ezproxy.library.ucf.edu                7\n",
      "2992                     fp.ucf.edu                7\n",
      "4450          libguides.lib.ucf.edu                4\n",
      "4453               graduate.ucf.edu                4 \n",
      "------------------------------------------------\n",
      "\n",
      "utexas.edu \n",
      "------------------------------------------------\n",
      " utexas.edu total count: 874 \n",
      " utexas.edu # of domains: 50 \n",
      " utexas.edu mean count per domain: 17.48 \n",
      "\n",
      "                       Domain  resource_domain\n",
      "37     guides.lib.utexas.edu              453\n",
      "330           lib.utexas.edu               85\n",
      "478         lanic.utexas.edu               60\n",
      "693          stat.utexas.edu               40\n",
      "712    search.lib.utexas.edu               39\n",
      "1107  ezproxy.lib.utexas.edu               23\n",
      "1220  catalog.lib.utexas.edu               21\n",
      "1494   legacy.lib.utexas.edu               16\n",
      "1693     utip.gov.utexas.edu               14\n",
      "1888      utlists.utexas.edu               13 \n",
      "------------------------------------------------\n",
      "\n",
      "ucla.edu \n",
      "------------------------------------------------\n",
      " ucla.edu total count: 867 \n",
      " ucla.edu # of domains: 66 \n",
      " ucla.edu mean count per domain: 13.14 \n",
      "\n",
      "                               Domain  resource_domain\n",
      "108          search.library.ucla.edu              244\n",
      "147          guides.library.ucla.edu              192\n",
      "356                 library.ucla.edu               78\n",
      "403         catalog.library.ucla.edu               70\n",
      "804   williamsinstitute.law.ucla.edu               33\n",
      "810                anderson.ucla.edu               33\n",
      "1047                     it.ucla.edu               25\n",
      "1115             stats.idre.ucla.edu               23\n",
      "1456           healthpolicy.ucla.edu               17\n",
      "1540                   heri.ucla.edu               16 \n",
      "------------------------------------------------\n",
      "\n",
      "libguides.com \n",
      "------------------------------------------------\n",
      " libguides.com total count: 867 \n",
      " libguides.com # of domains: 57 \n",
      " libguides.com mean count per domain: 15.21 \n",
      "\n",
      "                            Domain  resource_domain\n",
      "77             uark.libguides.com              305\n",
      "115            ucsd.libguides.com              235\n",
      "381     data-planet.libguides.com               72\n",
      "607   instr.iastate.libguides.com               47\n",
      "962          godort.libguides.com               27\n",
      "1037        clemson.libguides.com               25\n",
      "1273       proquest.libguides.com               20\n",
      "1306           pitt.libguides.com               19\n",
      "1789           tamu.libguides.com               13\n",
      "2233           uiuc.libguides.com               10 \n",
      "------------------------------------------------\n",
      "\n",
      "vt.edu \n",
      "------------------------------------------------\n",
      " vt.edu total count: 862 \n",
      " vt.edu # of domains: 75 \n",
      " vt.edu mean count per domain: 11.49 \n",
      "\n",
      "                                    Domain  resource_domain\n",
      "30                      guides.lib.vt.edu              485\n",
      "830                            lib.vt.edu               32\n",
      "851   vt-policymap-com.ezproxy.lib.vt.edu               31\n",
      "1248                      data.lib.vt.edu               20\n",
      "1369            informaticslab.lib.vt.edu               18\n",
      "1392                vtechworks.lib.vt.edu               18\n",
      "1523                           gis.vt.edu               16\n",
      "1844           eiu.com.ezproxy.lib.vt.edu               13\n",
      "1848                  app.nli.tlos.vt.edu               13\n",
      "1861                   catalog.lib.vt.edu               13 \n",
      "------------------------------------------------\n",
      "\n",
      "gmu.edu \n",
      "------------------------------------------------\n",
      " gmu.edu total count: 831 \n",
      " gmu.edu # of domains: 57 \n",
      " gmu.edu mean count per domain: 14.58 \n",
      "\n",
      "                      Domain  resource_domain\n",
      "62       infoguides.gmu.edu              345\n",
      "84            mutex.gmu.edu              292\n",
      "939    dataservices.gmu.edu               28\n",
      "1228    nassgeodata.gmu.edu               21\n",
      "1255          magik.gmu.edu               20\n",
      "1366  writingcenter.gmu.edu               18\n",
      "1849        library.gmu.edu               13\n",
      "2926          schar.gmu.edu                7\n",
      "3180            its.gmu.edu                7\n",
      "3279        infowiz.gmu.edu                6 \n",
      "------------------------------------------------\n",
      "\n",
      "rutgers.edu \n",
      "------------------------------------------------\n",
      " rutgers.edu total count: 829 \n",
      " rutgers.edu # of domains: 53 \n",
      " rutgers.edu mean count per domain: 15.64 \n",
      "\n",
      "                                  Domain  resource_domain\n",
      "44                libraries.rutgers.edu              405\n",
      "205               libguides.rutgers.edu              136\n",
      "695                    cawp.rutgers.edu               40\n",
      "856                     scc.rutgers.edu               31\n",
      "867   login.proxy.libraries.rutgers.edu               30\n",
      "1030                  oirap.rutgers.edu               26\n",
      "1406  quicksearch.libraries.rutgers.edu               18\n",
      "1608              ndbserver.rutgers.edu               15\n",
      "2001             njdatabook.rutgers.edu               12\n",
      "2037      catalog.libraries.rutgers.edu               11 \n",
      "------------------------------------------------\n",
      "\n",
      "uchicago.edu \n",
      "------------------------------------------------\n",
      " uchicago.edu total count: 827 \n",
      " uchicago.edu # of domains: 46 \n",
      " uchicago.edu mean count per domain: 17.98 \n",
      "\n",
      "                         Domain  resource_domain\n",
      "42            lib.uchicago.edu              421\n",
      "221    guides.lib.uchicago.edu              124\n",
      "276        pi.lib.uchicago.edu              104\n",
      "1050         dsal.uchicago.edu               25\n",
      "1249         norc.uchicago.edu               20\n",
      "1534        proxy.uchicago.edu               16\n",
      "1725  catalog.lib.uchicago.edu               14\n",
      "1939      rcc.lib.uchicago.edu               12\n",
      "2496    popcenter.uchicago.edu                9\n",
      "2693    knowledge.uchicago.edu                8 \n",
      "------------------------------------------------\n",
      "\n",
      "virginia.edu \n",
      "------------------------------------------------\n",
      " virginia.edu total count: 818 \n",
      " virginia.edu # of domains: 36 \n",
      " virginia.edu mean count per domain: 22.72 \n",
      "\n",
      "                           Domain  resource_domain\n",
      "106     proxy01.its.virginia.edu              246\n",
      "132       proxy.its.virginia.edu              213\n",
      "360   mapserver.lib.virginia.edu               77\n",
      "378      guides.lib.virginia.edu               73\n",
      "523      search.lib.virginia.edu               53\n",
      "733      fisher.lib.virginia.edu               37\n",
      "823             lib.virginia.edu               32\n",
      "1104        lib.law.virginia.edu               23\n",
      "1782        library.virginia.edu               13\n",
      "3250   data.library.virginia.edu                6 \n",
      "------------------------------------------------\n",
      "\n",
      "usc.edu \n",
      "------------------------------------------------\n",
      " usc.edu total count: 782 \n",
      " usc.edu # of domains: 34 \n",
      " usc.edu mean count per domain: 23.00 \n",
      "\n",
      "                       Domain  resource_domain\n",
      "28          libproxy.usc.edu              502\n",
      "171        libguides.usc.edu              177\n",
      "1323         spatial.usc.edu               19\n",
      "1548                 usc.edu               16\n",
      "1641         library.usc.edu               15\n",
      "2713       libraries.usc.edu                8\n",
      "3791        ida.loni.usc.edu                5\n",
      "4474        dornsife.usc.edu                4\n",
      "5116        software.usc.edu                3\n",
      "5145  digitallibrary.usc.edu                3 \n",
      "------------------------------------------------\n",
      "\n",
      "bea.gov \n",
      "------------------------------------------------\n",
      " bea.gov total count: 780 \n",
      " bea.gov # of domains: 2 \n",
      " bea.gov mean count per domain: 390.00 \n",
      "\n",
      "             Domain  resource_domain\n",
      "18         bea.gov              760\n",
      "1281  apps.bea.gov               20 \n",
      "------------------------------------------------\n",
      "\n",
      "georgetown.edu \n",
      "------------------------------------------------\n",
      " georgetown.edu total count: 761 \n",
      " georgetown.edu # of domains: 38 \n",
      " georgetown.edu mean count per domain: 20.03 \n",
      "\n",
      "                                   Domain  resource_domain\n",
      "50         guides.library.georgetown.edu              380\n",
      "161         proxy.library.georgetown.edu              183\n",
      "437               library.georgetown.edu               65\n",
      "767       catalog.library.georgetown.edu               35\n",
      "1114                 pdba.georgetown.edu               23\n",
      "2081              faculty.georgetown.edu               11\n",
      "3120                 cara.georgetown.edu                7\n",
      "3211                  uis.georgetown.edu                6\n",
      "3239  findingaids.library.georgetown.edu                6\n",
      "3570            guides.ll.georgetown.edu                6 \n",
      "------------------------------------------------\n",
      "\n",
      "ipums.org \n",
      "------------------------------------------------\n",
      " ipums.org total count: 751 \n",
      " ipums.org # of domains: 14 \n",
      " ipums.org mean count per domain: 53.64 \n",
      "\n",
      "                        Domain  resource_domain\n",
      "98                  ipums.org              265\n",
      "113             usa.ipums.org              237\n",
      "218   international.ipums.org              126\n",
      "410             cps.ipums.org               69\n",
      "1989           nhis.ipums.org               12\n",
      "2214          terra.ipums.org               11\n",
      "2417  healthsurveys.ipums.org                9\n",
      "3419       highered.ipums.org                6\n",
      "4223          ihgis.ipums.org                5\n",
      "4237           meps.ipums.org                4 \n",
      "------------------------------------------------\n",
      "\n",
      "ahrq.gov \n",
      "------------------------------------------------\n",
      " ahrq.gov total count: 734 \n",
      " ahrq.gov # of domains: 15 \n",
      " ahrq.gov mean count per domain: 48.93 \n",
      "\n",
      "                             Domain  resource_domain\n",
      "60                        ahrq.gov              348\n",
      "224               hcupnet.ahrq.gov              123\n",
      "278                  meps.ahrq.gov              104\n",
      "473               hcup-us.ahrq.gov               60\n",
      "656               nhqrnet.ahrq.gov               43\n",
      "1594                cahps.ahrq.gov               15\n",
      "2420  effectivehealthcare.ahrq.gov                9\n",
      "3071          innovations.ahrq.gov                7\n",
      "3084      qualitymeasures.ahrq.gov                7\n",
      "3159              archive.ahrq.gov                7 \n",
      "------------------------------------------------\n",
      "\n",
      "ac.uk \n",
      "------------------------------------------------\n",
      " ac.uk total count: 707 \n",
      " ac.uk # of domains: 170 \n",
      " ac.uk mean count per domain: 4.16 \n",
      "\n",
      "                             Domain  resource_domain\n",
      "385                      ebi.ac.uk               72\n",
      "400             data-archive.ac.uk               70\n",
      "497            ukdataservice.ac.uk               56\n",
      "910                      ucl.ac.uk               29\n",
      "928   discover.ukdataservice.ac.uk               28\n",
      "1055               v2.sherpa.ac.uk               25\n",
      "1078               blogs.lse.ac.uk               24\n",
      "1229                     dcc.ac.uk               21\n",
      "1377                     bgs.ac.uk               18\n",
      "1662  archaeologydataservice.ac.uk               15 \n",
      "------------------------------------------------\n",
      "\n",
      "gsu.edu \n",
      "------------------------------------------------\n",
      " gsu.edu total count: 706 \n",
      " gsu.edu # of domains: 21 \n",
      " gsu.edu mean count per domain: 33.62 \n",
      "\n",
      "                                   Domain  resource_domain\n",
      "23              research.library.gsu.edu              575\n",
      "781              campusdirectory.gsu.edu               34\n",
      "1065               ursa.research.gsu.edu               24\n",
      "1266                  technology.gsu.edu               20\n",
      "1425                policecrime.bgsu.edu               17\n",
      "1529                     gilfind.gsu.edu               16\n",
      "1945  digitalcollections.library.gsu.edu               12\n",
      "2839                     library.gsu.edu                8\n",
      "6578                       acrdc.gsu.edu                3\n",
      "8677             collegetocareer.gsu.edu                2 \n",
      "------------------------------------------------\n",
      "\n",
      "mit.edu \n",
      "------------------------------------------------\n",
      " mit.edu total count: 700 \n",
      " mit.edu # of domains: 28 \n",
      " mit.edu mean count per domain: 25.00 \n",
      "\n",
      "                       Domain  resource_domain\n",
      "90         libguides.mit.edu              282\n",
      "141        libraries.mit.edu              199\n",
      "463      electionlab.mit.edu               62\n",
      "817          library.mit.edu               33\n",
      "827              web.mit.edu               32\n",
      "1294  login.libproxy.mit.edu               20\n",
      "1315             ocw.mit.edu               19\n",
      "1790      livingwage.mit.edu               13\n",
      "2870           wikis.mit.edu                8\n",
      "3597         geodata.mit.edu                6 \n",
      "------------------------------------------------\n",
      "\n",
      "duke.edu \n",
      "------------------------------------------------\n",
      " duke.edu total count: 692 \n",
      " duke.edu # of domains: 40 \n",
      " duke.edu mean count per domain: 17.30 \n",
      "\n",
      "                             Domain  resource_domain\n",
      "176          find.library.duke.edu              174\n",
      "193               library.duke.edu              149\n",
      "244        guides.library.duke.edu              118\n",
      "281       login.proxy.lib.duke.edu              103\n",
      "925             proxy.lib.duke.edu               28\n",
      "1297                sites.duke.edu               19\n",
      "1492           db.library.duke.edu               16\n",
      "2285        library.fuqua.duke.edu               10\n",
      "2523  getitatduke.library.duke.edu                9\n",
      "2606       search.library.duke.edu                9 \n",
      "------------------------------------------------\n",
      "\n",
      "imf.org \n",
      "------------------------------------------------\n",
      " imf.org total count: 689 \n",
      " imf.org # of domains: 17 \n",
      " imf.org mean count per domain: 40.53 \n",
      "\n",
      "                                                Domain  resource_domain\n",
      "69                                            imf.org              328\n",
      "79                                       data.imf.org              300\n",
      "798                                  elibrary.imf.org               34\n",
      "1209                    data.imf.org.ezp2.lib.umn.edu               21\n",
      "2076                            elibrary-data.imf.org               11\n",
      "2115           data.imf.org.ezp-prod1.hul.harvard.edu               11\n",
      "3766                                 datahelp.imf.org                5\n",
      "4454  elibrary-data.imf.org.ezp-prod1.hul.harvard.edu                4\n",
      "4584                                      fsi.imf.org                4\n",
      "6289                                     dsbb.imf.org                3 \n",
      "------------------------------------------------\n",
      "\n"
     ]
    }
   ],
   "source": [
    "root_domains_20 = {}\n",
    "for i, row in root_domains_top[0:50].iterrows():\n",
    "    root_domains_20[row['Root domain']] = row['Count']\n",
    "\n",
    "for k,v in root_domains_20.items():\n",
    "    domain_check = domains_top[domains_top['Domain'].str.contains(k)]\n",
    "    print(k, '\\n------------------------------------------------\\n',\n",
    "          k, 'total count:', v, '\\n', \n",
    "          k, '# of domains:', len(domain_check), '\\n',\n",
    "          k, 'mean count per domain:', '{:.2f}'.format(v/len(domain_check)), '\\n\\n', \n",
    "          domain_check[0:10],\n",
    "         '\\n------------------------------------------------\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Top 20 Free and Paid/Hybrid Resources\n",
    "The top_dbs_annotated.csv file was created by adding an 'access' column noting whether each of the top 500 resources identified and exported above is 'free', 'paid', or 'hybrid'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Percentage of top 500 resources that are paid or hybrid: 20.28% \n",
      "Number that are paid or hybrid: 101 \n",
      "\tPercentage of top 500 resources that are paid: 17.47% \n",
      "\tNumber that are paid: 87 \n",
      "\tPercentage of top 500 resources that are hybrid: 2.81% \n",
      "\tNumber that are hybrid: 14 \n",
      "Percentage of top 500 resources that are free: 79.72% \n",
      "Number that are free: 397\n",
      "\n",
      "\n",
      "Number of links for paid/hybrid resources: 19627 \n",
      "Mean number of links per paid/hybrid resource: 194.33 \n",
      "\n",
      "\tNumer of links to paid resources: 15175 \n",
      "\tMean number of links per paid resource: 174.43 \n",
      "\tMedian number of links per paid resource: 66.00 \n",
      "\n",
      "\tNumer of links to hybrid resources: 4452 \n",
      "\tMean number of links per hybrid resource: 318.00 \n",
      "\tMedian number of links per hybrid resource: 67.50 \n",
      "\n",
      "Number of links for free resources: 41361 \n",
      "Mean number of links per free resource: 104.18 \n",
      "Median number of links per free resource: 53.00\n"
     ]
    }
   ],
   "source": [
    "free_paid_df = pd.read_csv('drum/lg_data_top_annotated.csv')\n",
    "free_paid_df = free_paid_df.sort_values(by='count', ascending=False)\n",
    "free_paid_df = free_paid_df.drop(columns=['sum'])\n",
    "\n",
    "paid_df = free_paid_df[free_paid_df['access'] == 'paid']\n",
    "free_df = free_paid_df[free_paid_df['access'] == 'free']\n",
    "hybrid_df = free_paid_df[free_paid_df['access'] == 'hybrid']\n",
    "paid_hybrid_df = free_paid_df[free_paid_df['access'] != 'free']\n",
    "\n",
    "print(\n",
    "    'Percentage of top 500 resources that are paid or hybrid:', \n",
    "    '{:.2%}'.format((len(paid_hybrid_df)/len(free_paid_df))),\n",
    "    '\\nNumber that are paid or hybrid:', \n",
    "    len(paid_hybrid_df),\n",
    "    '\\n\\tPercentage of top 500 resources that are paid:', \n",
    "    '{:.2%}'.format((len(paid_df)/len(free_paid_df))),\n",
    "    '\\n\\tNumber that are paid:', \n",
    "    len(paid_df),\n",
    "    '\\n\\tPercentage of top 500 resources that are hybrid:', \n",
    "    '{:.2%}'.format((len(hybrid_df)/len(free_paid_df))),\n",
    "    '\\n\\tNumber that are hybrid:', \n",
    "    len(hybrid_df),\n",
    "     '\\nPercentage of top 500 resources that are free:',               \n",
    "      '{:.2%}'.format((len(free_df)/len(free_paid_df))), \n",
    "    '\\nNumber that are free:', \n",
    "    len(free_df),\n",
    "                   )\n",
    "\n",
    "print(\n",
    "    '\\n\\nNumber of links for paid/hybrid resources:', paid_hybrid_df['count'].sum(),\n",
    "    '\\nMean number of links per paid/hybrid resource:', '{:.2f}'.format(paid_hybrid_df['count'].mean()),\n",
    "    '\\n\\n\\tNumer of links to paid resources:', paid_df['count'].sum(),\n",
    "    '\\n\\tMean number of links per paid resource:', '{:.2f}'.format(paid_df['count'].mean()),\n",
    "    '\\n\\tMedian number of links per paid resource:', '{:.2f}'.format(paid_df['count'].median()),\n",
    "    '\\n\\n\\tNumer of links to hybrid resources:', hybrid_df['count'].sum(),\n",
    "    '\\n\\tMean number of links per hybrid resource:', '{:.2f}'.format(hybrid_df['count'].mean()),\n",
    "    '\\n\\tMedian number of links per hybrid resource:', '{:.2f}'.format(hybrid_df['count'].median()),\n",
    "    '\\n\\nNumber of links for free resources:', free_df['count'].sum(),\n",
    "    '\\nMean number of links per free resource:', '{:.2f}'.format(free_df['count'].mean()),\n",
    "    '\\nMedian number of links per free resource:', '{:.2f}'.format(free_df['count'].median())\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>percent_of_sites</th>\n",
       "      <th>count</th>\n",
       "      <th>url</th>\n",
       "      <th>access</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>icpsr</td>\n",
       "      <td>94.31%</td>\n",
       "      <td>1834</td>\n",
       "      <td>www.icpsr.umich.edu</td>\n",
       "      <td>hybrid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>statistical insight (proquest)</td>\n",
       "      <td>61.79%</td>\n",
       "      <td>1532</td>\n",
       "      <td>proxy2.library.illinois.edu/login?url=search.p...</td>\n",
       "      <td>paid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>social explorer</td>\n",
       "      <td>78.86%</td>\n",
       "      <td>1430</td>\n",
       "      <td>www.socialexplorer.com</td>\n",
       "      <td>paid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>data planet (sage)</td>\n",
       "      <td>50.41%</td>\n",
       "      <td>1318</td>\n",
       "      <td>go.openathens.net/redirector/fiu.edu?url=datap...</td>\n",
       "      <td>paid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>statista</td>\n",
       "      <td>64.23%</td>\n",
       "      <td>1104</td>\n",
       "      <td>go.openathens.net/redirector/fiu.edu?url=http%...</td>\n",
       "      <td>hybrid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>roper center: ipoll</td>\n",
       "      <td>80.49%</td>\n",
       "      <td>1090</td>\n",
       "      <td>proxy2.library.illinois.edu/login?url=ropercen...</td>\n",
       "      <td>paid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>statistical abstract of the united states (pro...</td>\n",
       "      <td>78.05%</td>\n",
       "      <td>1058</td>\n",
       "      <td>www.libraries.rutgers.edu/indexes/statabus</td>\n",
       "      <td>paid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>oecd ilibrary</td>\n",
       "      <td>69.11%</td>\n",
       "      <td>851</td>\n",
       "      <td>www.oecd-ilibrary.org</td>\n",
       "      <td>hybrid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>historical statistics of the united states (ca...</td>\n",
       "      <td>74.80%</td>\n",
       "      <td>684</td>\n",
       "      <td>hsus.cambridge.org/HSUSWeb</td>\n",
       "      <td>paid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66</th>\n",
       "      <td>simplyanalytics</td>\n",
       "      <td>44.72%</td>\n",
       "      <td>672</td>\n",
       "      <td>go.openathens.net/redirector/fiu.edu?url=app.s...</td>\n",
       "      <td>paid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>policymap</td>\n",
       "      <td>43.09%</td>\n",
       "      <td>645</td>\n",
       "      <td>library.bu.edu/policymap</td>\n",
       "      <td>paid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>wharton research data services (wrds)</td>\n",
       "      <td>43.90%</td>\n",
       "      <td>470</td>\n",
       "      <td>nrs.harvard.edu/urn-3:hul.eresource:WRDS</td>\n",
       "      <td>paid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63</th>\n",
       "      <td>polling the nations</td>\n",
       "      <td>45.53%</td>\n",
       "      <td>313</td>\n",
       "      <td>nrs.harvard.edu/urn-3:hul.eresource:pollnati</td>\n",
       "      <td>paid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>103</th>\n",
       "      <td>passport (euromonitor international)</td>\n",
       "      <td>34.96%</td>\n",
       "      <td>249</td>\n",
       "      <td>ezproxy.fiu.edu/login?url=www.portal.euromonit...</td>\n",
       "      <td>paid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110</th>\n",
       "      <td>china data online</td>\n",
       "      <td>33.33%</td>\n",
       "      <td>209</td>\n",
       "      <td>ezproxy.fiu.edu/login?url=china-data-online.com</td>\n",
       "      <td>paid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>105</th>\n",
       "      <td>business source complete (ebsco)</td>\n",
       "      <td>34.15%</td>\n",
       "      <td>191</td>\n",
       "      <td>search.ebscohost.com/login.aspx?authtype=shib,...</td>\n",
       "      <td>paid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>114</th>\n",
       "      <td>data citation index (web of science/clarivate)</td>\n",
       "      <td>32.52%</td>\n",
       "      <td>177</td>\n",
       "      <td>isiknowledge.com/drci</td>\n",
       "      <td>paid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>220</th>\n",
       "      <td>economist intelligence unit (eiu)</td>\n",
       "      <td>21.14%</td>\n",
       "      <td>175</td>\n",
       "      <td>id.lib.harvard.edu/alma/990081655110203941/cat...</td>\n",
       "      <td>paid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>74</th>\n",
       "      <td>ibisworld</td>\n",
       "      <td>43.09%</td>\n",
       "      <td>168</td>\n",
       "      <td>ezproxy.fiu.edu/login?url=www.ibisworld.com</td>\n",
       "      <td>paid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>156</th>\n",
       "      <td>europa world plus</td>\n",
       "      <td>26.83%</td>\n",
       "      <td>161</td>\n",
       "      <td>uclibs.org/PID/115872</td>\n",
       "      <td>paid</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  name percent_of_sites  \\\n",
       "0                                                icpsr           94.31%   \n",
       "33                      statistical insight (proquest)           61.79%   \n",
       "11                                     social explorer           78.86%   \n",
       "51                                  data planet (sage)           50.41%   \n",
       "28                                            statista           64.23%   \n",
       "10                                 roper center: ipoll           80.49%   \n",
       "14   statistical abstract of the united states (pro...           78.05%   \n",
       "22                                       oecd ilibrary           69.11%   \n",
       "16   historical statistics of the united states (ca...           74.80%   \n",
       "66                                     simplyanalytics           44.72%   \n",
       "73                                           policymap           43.09%   \n",
       "67               wharton research data services (wrds)           43.90%   \n",
       "63                                 polling the nations           45.53%   \n",
       "103               passport (euromonitor international)           34.96%   \n",
       "110                                  china data online           33.33%   \n",
       "105                   business source complete (ebsco)           34.15%   \n",
       "114     data citation index (web of science/clarivate)           32.52%   \n",
       "220                  economist intelligence unit (eiu)           21.14%   \n",
       "74                                           ibisworld           43.09%   \n",
       "156                                  europa world plus           26.83%   \n",
       "\n",
       "     count                                                url  access  \n",
       "0     1834                                www.icpsr.umich.edu  hybrid  \n",
       "33    1532  proxy2.library.illinois.edu/login?url=search.p...    paid  \n",
       "11    1430                             www.socialexplorer.com    paid  \n",
       "51    1318  go.openathens.net/redirector/fiu.edu?url=datap...    paid  \n",
       "28    1104  go.openathens.net/redirector/fiu.edu?url=http%...  hybrid  \n",
       "10    1090  proxy2.library.illinois.edu/login?url=ropercen...    paid  \n",
       "14    1058         www.libraries.rutgers.edu/indexes/statabus    paid  \n",
       "22     851                              www.oecd-ilibrary.org  hybrid  \n",
       "16     684                         hsus.cambridge.org/HSUSWeb    paid  \n",
       "66     672  go.openathens.net/redirector/fiu.edu?url=app.s...    paid  \n",
       "73     645                           library.bu.edu/policymap    paid  \n",
       "67     470           nrs.harvard.edu/urn-3:hul.eresource:WRDS    paid  \n",
       "63     313       nrs.harvard.edu/urn-3:hul.eresource:pollnati    paid  \n",
       "103    249  ezproxy.fiu.edu/login?url=www.portal.euromonit...    paid  \n",
       "110    209    ezproxy.fiu.edu/login?url=china-data-online.com    paid  \n",
       "105    191  search.ebscohost.com/login.aspx?authtype=shib,...    paid  \n",
       "114    177                              isiknowledge.com/drci    paid  \n",
       "220    175  id.lib.harvard.edu/alma/990081655110203941/cat...    paid  \n",
       "74     168        ezproxy.fiu.edu/login?url=www.ibisworld.com    paid  \n",
       "156    161                              uclibs.org/PID/115872    paid  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "paid_hybrid_df[0:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>percent_of_sites</th>\n",
       "      <th>count</th>\n",
       "      <th>url</th>\n",
       "      <th>access</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>u.s. census: data</td>\n",
       "      <td>90.24%</td>\n",
       "      <td>1145</td>\n",
       "      <td>data.census.gov/cedsci</td>\n",
       "      <td>free</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>undata (united nations)</td>\n",
       "      <td>88.62%</td>\n",
       "      <td>1054</td>\n",
       "      <td>data.un.org</td>\n",
       "      <td>free</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>u.s. census bureau</td>\n",
       "      <td>93.50%</td>\n",
       "      <td>956</td>\n",
       "      <td>www.census.gov</td>\n",
       "      <td>free</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>data.gov</td>\n",
       "      <td>94.31%</td>\n",
       "      <td>954</td>\n",
       "      <td>www.data.gov</td>\n",
       "      <td>free</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>nchs (u.s. national center for health statistics)</td>\n",
       "      <td>86.99%</td>\n",
       "      <td>856</td>\n",
       "      <td>www.cdc.gov/nchs</td>\n",
       "      <td>free</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>u.s. census: american factfinder</td>\n",
       "      <td>83.74%</td>\n",
       "      <td>720</td>\n",
       "      <td>factfinder.census.gov/faces/nav/jsf/pages/inde...</td>\n",
       "      <td>free</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>world bank: world development indicators</td>\n",
       "      <td>65.85%</td>\n",
       "      <td>599</td>\n",
       "      <td>data.worldbank.org/data-catalog/world-developm...</td>\n",
       "      <td>free</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>nces (u.s. national center for education stati...</td>\n",
       "      <td>86.99%</td>\n",
       "      <td>546</td>\n",
       "      <td>nces.ed.gov</td>\n",
       "      <td>free</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>world bank: data</td>\n",
       "      <td>76.42%</td>\n",
       "      <td>545</td>\n",
       "      <td>data.worldbank.org</td>\n",
       "      <td>free</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>bls (u.s. bureau of labor statistics)</td>\n",
       "      <td>82.93%</td>\n",
       "      <td>535</td>\n",
       "      <td>www.bls.gov</td>\n",
       "      <td>free</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>eurostat (european commission)</td>\n",
       "      <td>80.49%</td>\n",
       "      <td>501</td>\n",
       "      <td>ec.europa.eu/eurostat</td>\n",
       "      <td>free</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>who: global health observatory (world health o...</td>\n",
       "      <td>78.05%</td>\n",
       "      <td>491</td>\n",
       "      <td>www.who.int/gho/en</td>\n",
       "      <td>free</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>imf data (international monetary fund)</td>\n",
       "      <td>74.80%</td>\n",
       "      <td>445</td>\n",
       "      <td>data.imf.org</td>\n",
       "      <td>free</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>u.s. census: statistical abstract series</td>\n",
       "      <td>71.54%</td>\n",
       "      <td>442</td>\n",
       "      <td>www.census.gov/library/publications/time-serie...</td>\n",
       "      <td>free</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>bjs (u.s. bureau of justice statistics)</td>\n",
       "      <td>78.86%</td>\n",
       "      <td>437</td>\n",
       "      <td>www.bjs.gov</td>\n",
       "      <td>free</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>bea (u.s. bureau of economic analysis)</td>\n",
       "      <td>74.80%</td>\n",
       "      <td>422</td>\n",
       "      <td>www.bea.gov</td>\n",
       "      <td>free</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>pew research center</td>\n",
       "      <td>60.16%</td>\n",
       "      <td>386</td>\n",
       "      <td>www.pewresearch.org</td>\n",
       "      <td>free</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>usa.gov: data and statistics</td>\n",
       "      <td>69.92%</td>\n",
       "      <td>385</td>\n",
       "      <td>www.usa.gov/statistics</td>\n",
       "      <td>free</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>cdc: data and statistics</td>\n",
       "      <td>73.17%</td>\n",
       "      <td>361</td>\n",
       "      <td>www.cdc.gov/DataStatistics</td>\n",
       "      <td>free</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>56</th>\n",
       "      <td>world bank: databank</td>\n",
       "      <td>49.59%</td>\n",
       "      <td>348</td>\n",
       "      <td>databank.worldbank.org/data/home.aspx</td>\n",
       "      <td>free</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 name percent_of_sites  count  \\\n",
       "3                                   u.s. census: data           90.24%   1145   \n",
       "4                             undata (united nations)           88.62%   1054   \n",
       "2                                  u.s. census bureau           93.50%    956   \n",
       "1                                            data.gov           94.31%    954   \n",
       "6   nchs (u.s. national center for health statistics)           86.99%    856   \n",
       "7                    u.s. census: american factfinder           83.74%    720   \n",
       "25           world bank: world development indicators           65.85%    599   \n",
       "5   nces (u.s. national center for education stati...           86.99%    546   \n",
       "15                                   world bank: data           76.42%    545   \n",
       "8               bls (u.s. bureau of labor statistics)           82.93%    535   \n",
       "9                      eurostat (european commission)           80.49%    501   \n",
       "13  who: global health observatory (world health o...           78.05%    491   \n",
       "18             imf data (international monetary fund)           74.80%    445   \n",
       "20           u.s. census: statistical abstract series           71.54%    442   \n",
       "12            bjs (u.s. bureau of justice statistics)           78.86%    437   \n",
       "17             bea (u.s. bureau of economic analysis)           74.80%    422   \n",
       "38                                pew research center           60.16%    386   \n",
       "21                       usa.gov: data and statistics           69.92%    385   \n",
       "19                           cdc: data and statistics           73.17%    361   \n",
       "56                               world bank: databank           49.59%    348   \n",
       "\n",
       "                                                  url access  \n",
       "3                              data.census.gov/cedsci   free  \n",
       "4                                         data.un.org   free  \n",
       "2                                      www.census.gov   free  \n",
       "1                                        www.data.gov   free  \n",
       "6                                    www.cdc.gov/nchs   free  \n",
       "7   factfinder.census.gov/faces/nav/jsf/pages/inde...   free  \n",
       "25  data.worldbank.org/data-catalog/world-developm...   free  \n",
       "5                                         nces.ed.gov   free  \n",
       "15                                 data.worldbank.org   free  \n",
       "8                                         www.bls.gov   free  \n",
       "9                               ec.europa.eu/eurostat   free  \n",
       "13                                 www.who.int/gho/en   free  \n",
       "18                                       data.imf.org   free  \n",
       "20  www.census.gov/library/publications/time-serie...   free  \n",
       "12                                        www.bjs.gov   free  \n",
       "17                                        www.bea.gov   free  \n",
       "38                                www.pewresearch.org   free  \n",
       "21                             www.usa.gov/statistics   free  \n",
       "19                         www.cdc.gov/DataStatistics   free  \n",
       "56              databank.worldbank.org/data/home.aspx   free  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "free_df[0:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>percent_of_sites</th>\n",
       "      <th>count</th>\n",
       "      <th>url</th>\n",
       "      <th>access</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>icpsr</td>\n",
       "      <td>94.31%</td>\n",
       "      <td>1834</td>\n",
       "      <td>www.icpsr.umich.edu</td>\n",
       "      <td>hybrid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>statista</td>\n",
       "      <td>64.23%</td>\n",
       "      <td>1104</td>\n",
       "      <td>go.openathens.net/redirector/fiu.edu?url=http%...</td>\n",
       "      <td>hybrid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>oecd ilibrary</td>\n",
       "      <td>69.11%</td>\n",
       "      <td>851</td>\n",
       "      <td>www.oecd-ilibrary.org</td>\n",
       "      <td>hybrid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195</th>\n",
       "      <td>oecd ilibrary: statistics</td>\n",
       "      <td>22.76%</td>\n",
       "      <td>110</td>\n",
       "      <td>www.oecd-ilibrary.org/statistics</td>\n",
       "      <td>hybrid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>446</th>\n",
       "      <td>rand state statistics</td>\n",
       "      <td>10.57%</td>\n",
       "      <td>83</td>\n",
       "      <td>www.randstatestats.org</td>\n",
       "      <td>hybrid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>130</th>\n",
       "      <td>oecd (organisation for economic co-operation a...</td>\n",
       "      <td>30.08%</td>\n",
       "      <td>78</td>\n",
       "      <td>www.oecd.org</td>\n",
       "      <td>hybrid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>170</th>\n",
       "      <td>pollingreport.com</td>\n",
       "      <td>23.58%</td>\n",
       "      <td>68</td>\n",
       "      <td>www.pollingreport.com</td>\n",
       "      <td>hybrid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>184</th>\n",
       "      <td>icpsr: health and medical care archive (hmca)</td>\n",
       "      <td>22.76%</td>\n",
       "      <td>67</td>\n",
       "      <td>www.icpsr.umich.edu/web/pages/HMCA/index.html</td>\n",
       "      <td>hybrid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>447</th>\n",
       "      <td>wolframalpha</td>\n",
       "      <td>10.57%</td>\n",
       "      <td>58</td>\n",
       "      <td>www.wolframalpha.com</td>\n",
       "      <td>hybrid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>497</th>\n",
       "      <td>byu law and corpus linguistics</td>\n",
       "      <td>0.81%</td>\n",
       "      <td>55</td>\n",
       "      <td>lawcorpus.byu.edu</td>\n",
       "      <td>hybrid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>209</th>\n",
       "      <td>icpsr: resource center for minority data (rcmd)</td>\n",
       "      <td>21.14%</td>\n",
       "      <td>42</td>\n",
       "      <td>www.icpsr.umich.edu/icpsrweb/RCMD</td>\n",
       "      <td>hybrid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>276</th>\n",
       "      <td>icpsr: national archive of computerized data o...</td>\n",
       "      <td>17.89%</td>\n",
       "      <td>35</td>\n",
       "      <td>www.icpsr.umich.edu/icpsrweb/NACDA</td>\n",
       "      <td>hybrid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>344</th>\n",
       "      <td>cpanda (cultural policy and the arts national ...</td>\n",
       "      <td>14.63%</td>\n",
       "      <td>35</td>\n",
       "      <td>www.cpanda.org/cpanda</td>\n",
       "      <td>hybrid</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>292</th>\n",
       "      <td>icpsr: national archive of data on arts and cu...</td>\n",
       "      <td>17.07%</td>\n",
       "      <td>32</td>\n",
       "      <td>www.icpsr.umich.edu/icpsrweb/NADAC</td>\n",
       "      <td>hybrid</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  name percent_of_sites  \\\n",
       "0                                                icpsr           94.31%   \n",
       "28                                            statista           64.23%   \n",
       "22                                       oecd ilibrary           69.11%   \n",
       "195                          oecd ilibrary: statistics           22.76%   \n",
       "446                              rand state statistics           10.57%   \n",
       "130  oecd (organisation for economic co-operation a...           30.08%   \n",
       "170                                  pollingreport.com           23.58%   \n",
       "184      icpsr: health and medical care archive (hmca)           22.76%   \n",
       "447                                       wolframalpha           10.57%   \n",
       "497                     byu law and corpus linguistics            0.81%   \n",
       "209    icpsr: resource center for minority data (rcmd)           21.14%   \n",
       "276  icpsr: national archive of computerized data o...           17.89%   \n",
       "344  cpanda (cultural policy and the arts national ...           14.63%   \n",
       "292  icpsr: national archive of data on arts and cu...           17.07%   \n",
       "\n",
       "     count                                                url  access  \n",
       "0     1834                                www.icpsr.umich.edu  hybrid  \n",
       "28    1104  go.openathens.net/redirector/fiu.edu?url=http%...  hybrid  \n",
       "22     851                              www.oecd-ilibrary.org  hybrid  \n",
       "195    110                   www.oecd-ilibrary.org/statistics  hybrid  \n",
       "446     83                             www.randstatestats.org  hybrid  \n",
       "130     78                                       www.oecd.org  hybrid  \n",
       "170     68                              www.pollingreport.com  hybrid  \n",
       "184     67      www.icpsr.umich.edu/web/pages/HMCA/index.html  hybrid  \n",
       "447     58                               www.wolframalpha.com  hybrid  \n",
       "497     55                                  lawcorpus.byu.edu  hybrid  \n",
       "209     42                  www.icpsr.umich.edu/icpsrweb/RCMD  hybrid  \n",
       "276     35                 www.icpsr.umich.edu/icpsrweb/NACDA  hybrid  \n",
       "344     35                              www.cpanda.org/cpanda  hybrid  \n",
       "292     32                 www.icpsr.umich.edu/icpsrweb/NADAC  hybrid  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hybrid_df[0:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Output the Top 200 resources (annotated as free/paid/hybrid) to HTML Table\n",
    "To view the rendered table of 200 top resources online, visit (GitHub pages link forthcoming)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "for_html = free_paid_df.sort_values(by='count', ascending=False)[0:200]\n",
    "\n",
    "for_html = for_html.reset_index()\n",
    "for_html.index += 1 \n",
    "for_html = for_html[['name', 'percent_of_sites', 'count', 'access', 'url']]\n",
    "\n",
    "html = for_html.to_html()\n",
    "  \n",
    "# write html to file\n",
    "text_file = open(\"table.html\", \"w\")\n",
    "text_file.write(html)\n",
    "text_file.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}