sample_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Demonstration of Expand-T\n",
    "In this Jupyter Notebook, we show all functionalities of Expand-T. \n",
    "Given a Source Table and a data lake, Expand-T uses [Gen-T](https://github.com/northeastern-datalab/gen-t) to first reclaim the source table. To do so, Expand-T first finds the set of originating tables from the data lake whose data values contribute to source values. Then, Expand-T performs integration in a targeted manner and reproduces the Source Table. Afterwards, the user can choose to expand their reclaimed table and see what additional attributes and tuples they can add to their source table, and what non-null values they can use to replace nulls. Lastly, users can sort and refine tuples in the expanded table to then input into their downstream task."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parameters and Imports\n",
    "Users can import a source table and a data lake (benchmark). For now, we use a table from the [T2D Gold benchmark](https://webdatacommons.org/webtables/goldstandard.html).\n",
    "User can also set a parameter for timeout at integration time. For now, we set the integration timeout to 1hr."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import display\n",
    "import sys\n",
    "import os\n",
    "os.chdir('/home/gfan/gen-t-demo/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Parameters\n",
    "benchmark = \"t2d_gold\"\n",
    "source_table_name = \"table_471_1.csv\"\n",
    "BENCHMARKPATH = '/home/gfan/Datasets/%s/' % (benchmark)  \n",
    "integration_timeout = 3600"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finding Candidate Tables\n",
    "Expand-T uses Gen-T to find a set of candidate tables from the data lake. Gen-T uses an efficient off-the-shelf table discovery method to do so, like [Starmie](https://www.vldb.org/pvldb/vol16/p1726-fan.pdf). Then, Gen-T filters out the retrieved set of relevant tables to those that have high set overlap with the source table, and return these tables as **Candidate Tables**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "12 Candidate Tables found for Source Table table_471_1.csv\n",
      "['table_536.csv', 'table_50.csv', 'table_187.csv', 'table_267.csv', 'table_594.csv', 'table_25.csv', 'table_244.csv', 'table_95.csv', 'table_640.csv', 'table_292.csv', 'table_674.csv', 'table_760.csv']\n"
     ]
    }
   ],
   "source": [
    "sys.path.append(\"discovery/\")\n",
    "from discovery_utils import get_lake, get_starmie_candidates\n",
    "from discover_candidates import CandidateTables\n",
    "\n",
    "runStarmie = 0\n",
    "lake_dfs, all_lake_table_cols = get_lake(benchmark)\n",
    "source_candidates = []\n",
    "if runStarmie: \n",
    "    starmie_candidates = get_starmie_candidates(benchmark)\n",
    "    source_candidates = starmie_candidates[source_table_name.replace('.csv', '')]\n",
    "\n",
    "# Call CandidateTables to find candidates\n",
    "candidate_table_finder = CandidateTables(benchmark,lake_dfs, all_lake_table_cols, source_candidates)\n",
    "candidateTablesFound, _ = candidate_table_finder.find_candidates(source_table_name)\n",
    "print(f\"{len(candidateTablesFound)} Candidate Tables found for Source Table {source_table_name}\")\n",
    "print(list(candidateTablesFound.keys()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finding Originating Tables\n",
    "Gen-T prunes the set of candidate tables to only include tables that when integrated, reproduce the source table. We call this pruned set, **Originating Tables**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "From 12 candidate tables, found 6 originating tables\n",
      "['table_50.csv', 'table_267.csv', 'table_95.csv', 'table_674.csv', 'table_25.csv', 'table_244.csv']\n"
     ]
    }
   ],
   "source": [
    "\n",
    "from prune_candidates import OriginatingTables\n",
    "# Call OriginatingTables to prune candidates to a set of originating tables\n",
    "originating_tables_finder = OriginatingTables(benchmark, candidateTablesFound)\n",
    "originating_tables, matrix_traversal_runtimes = originating_tables_finder.find_originating_tables(source_table_name)\n",
    "source_df = originating_tables_finder.source_table\n",
    "print(f\"From {len(candidateTablesFound)} candidate tables, found {len(originating_tables)} originating tables\")\n",
    "print(originating_tables)\n",
    "origin_tables_matched_cols = {}\n",
    "if originating_tables: origin_tables_matched_cols = {t: candidateTablesFound[t] for t in originating_tables}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reclaiming the Source Table\n",
    "Gen-T now integrates the set of originating tables with the goal of reproducing or reclaiming the source table. We show the output of Table Reclamation, a **reclaimed Source Table**. This reclaimed table has bolded values highlighted in yellow for reproduced source values. For non-null values that are suggested to replace missing values in the source table, we underline them and highlight them in blue."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.chdir('integration/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-----x---------x--------x---\n",
      "Source Table:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Title</th>\n",
       "      <th>Fans' Rank</th>\n",
       "      <th>Year</th>\n",
       "      <th>Director(s)</th>\n",
       "      <th>Overall Rank</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Pulp Fiction</td>\n",
       "      <td>2</td>\n",
       "      <td>1994</td>\n",
       "      <td>Quentin Tarantino</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Citizen Kane</td>\n",
       "      <td>3</td>\n",
       "      <td>1941</td>\n",
       "      <td>Orson Welles</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Star Wars</td>\n",
       "      <td>5</td>\n",
       "      <td>1977</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Title  Fans' Rank  Year        Director(s)  Overall Rank\n",
       "0  Pulp Fiction           2  1994  Quentin Tarantino            15\n",
       "1  Citizen Kane           3  1941       Orson Welles             2\n",
       "2     Star Wars           5  1977                NaN             8"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-----x---------x--------x---\n",
      "Reclaimed Source Table:\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_8bc7b_row0_col0, #T_8bc7b_row0_col1, #T_8bc7b_row0_col2, #T_8bc7b_row0_col3, #T_8bc7b_row0_col4, #T_8bc7b_row1_col0, #T_8bc7b_row1_col1, #T_8bc7b_row1_col2, #T_8bc7b_row1_col3, #T_8bc7b_row1_col4, #T_8bc7b_row2_col0, #T_8bc7b_row2_col1, #T_8bc7b_row2_col2, #T_8bc7b_row2_col4 {\n",
       "  background-color: khaki;\n",
       "  font-weight: bold;\n",
       "}\n",
       "#T_8bc7b_row2_col3 {\n",
       "  background-color: lightskyblue;\n",
       "  text-decoration: underline;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_8bc7b_\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th class=\"col_heading level0 col0\" >Title</th>\n",
       "      <th class=\"col_heading level0 col1\" >Fans' Rank</th>\n",
       "      <th class=\"col_heading level0 col2\" >Year</th>\n",
       "      <th class=\"col_heading level0 col3\" >Director(s)</th>\n",
       "      <th class=\"col_heading level0 col4\" >Overall Rank</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_8bc7b_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_8bc7b_row0_col0\" class=\"data row0 col0\" >Pulp Fiction</td>\n",
       "      <td id=\"T_8bc7b_row0_col1\" class=\"data row0 col1\" >2</td>\n",
       "      <td id=\"T_8bc7b_row0_col2\" class=\"data row0 col2\" >1994</td>\n",
       "      <td id=\"T_8bc7b_row0_col3\" class=\"data row0 col3\" >Quentin Tarantino</td>\n",
       "      <td id=\"T_8bc7b_row0_col4\" class=\"data row0 col4\" >15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_8bc7b_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "      <td id=\"T_8bc7b_row1_col0\" class=\"data row1 col0\" >Citizen Kane</td>\n",
       "      <td id=\"T_8bc7b_row1_col1\" class=\"data row1 col1\" >3</td>\n",
       "      <td id=\"T_8bc7b_row1_col2\" class=\"data row1 col2\" >1941</td>\n",
       "      <td id=\"T_8bc7b_row1_col3\" class=\"data row1 col3\" >Orson Welles</td>\n",
       "      <td id=\"T_8bc7b_row1_col4\" class=\"data row1 col4\" >2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_8bc7b_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "      <td id=\"T_8bc7b_row2_col0\" class=\"data row2 col0\" >Star Wars</td>\n",
       "      <td id=\"T_8bc7b_row2_col1\" class=\"data row2 col1\" >5</td>\n",
       "      <td id=\"T_8bc7b_row2_col2\" class=\"data row2 col2\" >1977</td>\n",
       "      <td id=\"T_8bc7b_row2_col3\" class=\"data row2 col3\" >George Lucas</td>\n",
       "      <td id=\"T_8bc7b_row2_col4\" class=\"data row2 col4\" >8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f642f76bed0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from targeted_integration import TableIntegration\n",
    "import integration_utils as utils \n",
    "sys.path.append('../discovery/')\n",
    "\n",
    "table_integrator = TableIntegration(benchmark, origin_tables_matched_cols, integration_timeout)\n",
    "timed_out, noCandidates, numOutputVals = table_integrator.integrate_tables(source_table_name)\n",
    "integration_result = table_integrator.reproducedSourceTable\n",
    "integration_result = table_integrator.order_rows_cols(integration_result)\n",
    "integration_result_highlighted = table_integrator.highlight_overlap_values(integration_result)\n",
    "display(integration_result_highlighted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Expanding the Source Table\n",
    "Expand-T can also expand the reclaimed source table. Expand-T can suggest new data as follows:\n",
    "* expand tables in a 'horizontal' direction: Expand-T adds more columns\n",
    "* expand tables in a 'vertical' direction: Expand-T adds more rows\n",
    "* expand tables in 'all' directions: Expand-T adds more columns and more rows\n",
    "\n",
    "When expanding the table, the user can also specify the number $N$ of new columns and/or rows they would like to see. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Integrated Table has 2 new columns: ['Religion', 'Actor']\n",
      "Integrated Table has 414 new tuples\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_8c2a7_row0_col0, #T_8c2a7_row0_col1, #T_8c2a7_row0_col2, #T_8c2a7_row0_col3, #T_8c2a7_row0_col4, #T_8c2a7_row1_col0, #T_8c2a7_row1_col1, #T_8c2a7_row1_col2, #T_8c2a7_row1_col3, #T_8c2a7_row1_col4, #T_8c2a7_row2_col0, #T_8c2a7_row2_col1, #T_8c2a7_row2_col2, #T_8c2a7_row2_col4 {\n",
       "  background-color: khaki;\n",
       "  font-weight: bold;\n",
       "}\n",
       "#T_8c2a7_row0_col5, #T_8c2a7_row1_col5, #T_8c2a7_row2_col5, #T_8c2a7_row3_col5, #T_8c2a7_row4_col5, #T_8c2a7_row5_col5 {\n",
       "  background-color: coral;\n",
       "  font-style: italic;\n",
       "}\n",
       "#T_8c2a7_row2_col3 {\n",
       "  background-color: lightskyblue;\n",
       "  text-decoration: underline;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_8c2a7_\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th class=\"col_heading level0 col0\" >Title</th>\n",
       "      <th class=\"col_heading level0 col1\" >Fans' Rank</th>\n",
       "      <th class=\"col_heading level0 col2\" >Year</th>\n",
       "      <th class=\"col_heading level0 col3\" >Director(s)</th>\n",
       "      <th class=\"col_heading level0 col4\" >Overall Rank</th>\n",
       "      <th class=\"col_heading level0 col5\" >Religion</th>\n",
       "      <th class=\"col_heading level0 col6\" >Actor</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_8c2a7_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_8c2a7_row0_col0\" class=\"data row0 col0\" >Pulp Fiction</td>\n",
       "      <td id=\"T_8c2a7_row0_col1\" class=\"data row0 col1\" >2</td>\n",
       "      <td id=\"T_8c2a7_row0_col2\" class=\"data row0 col2\" >1994</td>\n",
       "      <td id=\"T_8c2a7_row0_col3\" class=\"data row0 col3\" >Quentin Tarantino</td>\n",
       "      <td id=\"T_8c2a7_row0_col4\" class=\"data row0 col4\" >15</td>\n",
       "      <td id=\"T_8c2a7_row0_col5\" class=\"data row0 col5\" >Atheist</td>\n",
       "      <td id=\"T_8c2a7_row0_col6\" class=\"data row0 col6\" >John Travolta</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_8c2a7_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "      <td id=\"T_8c2a7_row1_col0\" class=\"data row1 col0\" >Citizen Kane</td>\n",
       "      <td id=\"T_8c2a7_row1_col1\" class=\"data row1 col1\" >3</td>\n",
       "      <td id=\"T_8c2a7_row1_col2\" class=\"data row1 col2\" >1941</td>\n",
       "      <td id=\"T_8c2a7_row1_col3\" class=\"data row1 col3\" >Orson Welles</td>\n",
       "      <td id=\"T_8c2a7_row1_col4\" class=\"data row1 col4\" >2</td>\n",
       "      <td id=\"T_8c2a7_row1_col5\" class=\"data row1 col5\" >nan</td>\n",
       "      <td id=\"T_8c2a7_row1_col6\" class=\"data row1 col6\" >Orson Welles</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_8c2a7_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "      <td id=\"T_8c2a7_row2_col0\" class=\"data row2 col0\" >Star Wars</td>\n",
       "      <td id=\"T_8c2a7_row2_col1\" class=\"data row2 col1\" >5</td>\n",
       "      <td id=\"T_8c2a7_row2_col2\" class=\"data row2 col2\" >1977</td>\n",
       "      <td id=\"T_8c2a7_row2_col3\" class=\"data row2 col3\" >George Lucas</td>\n",
       "      <td id=\"T_8c2a7_row2_col4\" class=\"data row2 col4\" >8</td>\n",
       "      <td id=\"T_8c2a7_row2_col5\" class=\"data row2 col5\" >nan</td>\n",
       "      <td id=\"T_8c2a7_row2_col6\" class=\"data row2 col6\" >nan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_8c2a7_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "      <td id=\"T_8c2a7_row3_col0\" class=\"data row3 col0\" >Kill Bill: Vol. 2</td>\n",
       "      <td id=\"T_8c2a7_row3_col1\" class=\"data row3 col1\" >224</td>\n",
       "      <td id=\"T_8c2a7_row3_col2\" class=\"data row3 col2\" >2004</td>\n",
       "      <td id=\"T_8c2a7_row3_col3\" class=\"data row3 col3\" >Quentin Tarantino</td>\n",
       "      <td id=\"T_8c2a7_row3_col4\" class=\"data row3 col4\" >701</td>\n",
       "      <td id=\"T_8c2a7_row3_col5\" class=\"data row3 col5\" >nan</td>\n",
       "      <td id=\"T_8c2a7_row3_col6\" class=\"data row3 col6\" >nan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_8c2a7_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "      <td id=\"T_8c2a7_row4_col0\" class=\"data row4 col0\" >Night of the Living Dead</td>\n",
       "      <td id=\"T_8c2a7_row4_col1\" class=\"data row4 col1\" >229</td>\n",
       "      <td id=\"T_8c2a7_row4_col2\" class=\"data row4 col2\" >1968</td>\n",
       "      <td id=\"T_8c2a7_row4_col3\" class=\"data row4 col3\" >George A. Romero</td>\n",
       "      <td id=\"T_8c2a7_row4_col4\" class=\"data row4 col4\" >139</td>\n",
       "      <td id=\"T_8c2a7_row4_col5\" class=\"data row4 col5\" >Christian</td>\n",
       "      <td id=\"T_8c2a7_row4_col6\" class=\"data row4 col6\" >nan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_8c2a7_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
       "      <td id=\"T_8c2a7_row5_col0\" class=\"data row5 col0\" >The Piano</td>\n",
       "      <td id=\"T_8c2a7_row5_col1\" class=\"data row5 col1\" >307</td>\n",
       "      <td id=\"T_8c2a7_row5_col2\" class=\"data row5 col2\" >1993</td>\n",
       "      <td id=\"T_8c2a7_row5_col3\" class=\"data row5 col3\" >Jane Campion</td>\n",
       "      <td id=\"T_8c2a7_row5_col4\" class=\"data row5 col4\" >247</td>\n",
       "      <td id=\"T_8c2a7_row5_col5\" class=\"data row5 col5\" >Christian</td>\n",
       "      <td id=\"T_8c2a7_row5_col6\" class=\"data row5 col6\" >nan</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f642dec8110>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "expanded_integ_result, new_cols, new_tuples = table_integrator.expand_tables('all', N=3)\n",
    "expanded_integration_result_highlighted = table_integrator.highlight_overlap_values(expanded_integ_result)\n",
    "display(expanded_integration_result_highlighted)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring the Expanded Table\n",
    "User can also sort and refine tuples in the expanded table. \n",
    "\n",
    "To sort tuples, they can call ``sort_tuples()`` and specify the column they would like to sort by (``sortCol``).\n",
    "\n",
    "To refine or filter tuples, users can create a dictionary of column name and column values that they would like to filter by. Then, they can call ``filter_tuples()`` and pass in the dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_3d2e7_row0_col0, #T_3d2e7_row0_col1, #T_3d2e7_row0_col2, #T_3d2e7_row0_col3, #T_3d2e7_row0_col4, #T_3d2e7_row1_col0, #T_3d2e7_row1_col1, #T_3d2e7_row1_col2, #T_3d2e7_row1_col4, #T_3d2e7_row2_col0, #T_3d2e7_row2_col1, #T_3d2e7_row2_col2, #T_3d2e7_row2_col3, #T_3d2e7_row2_col4 {\n",
       "  background-color: khaki;\n",
       "  font-weight: bold;\n",
       "}\n",
       "#T_3d2e7_row0_col5, #T_3d2e7_row1_col5, #T_3d2e7_row2_col5, #T_3d2e7_row3_col5, #T_3d2e7_row4_col5, #T_3d2e7_row5_col5 {\n",
       "  background-color: coral;\n",
       "  font-style: italic;\n",
       "}\n",
       "#T_3d2e7_row1_col3 {\n",
       "  background-color: lightskyblue;\n",
       "  text-decoration: underline;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_3d2e7_\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th class=\"col_heading level0 col0\" >Title</th>\n",
       "      <th class=\"col_heading level0 col1\" >Fans' Rank</th>\n",
       "      <th class=\"col_heading level0 col2\" >Year</th>\n",
       "      <th class=\"col_heading level0 col3\" >Director(s)</th>\n",
       "      <th class=\"col_heading level0 col4\" >Overall Rank</th>\n",
       "      <th class=\"col_heading level0 col5\" >Religion</th>\n",
       "      <th class=\"col_heading level0 col6\" >Actor</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_3d2e7_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_3d2e7_row0_col0\" class=\"data row0 col0\" >Citizen Kane</td>\n",
       "      <td id=\"T_3d2e7_row0_col1\" class=\"data row0 col1\" >3</td>\n",
       "      <td id=\"T_3d2e7_row0_col2\" class=\"data row0 col2\" >1941</td>\n",
       "      <td id=\"T_3d2e7_row0_col3\" class=\"data row0 col3\" >Orson Welles</td>\n",
       "      <td id=\"T_3d2e7_row0_col4\" class=\"data row0 col4\" >2</td>\n",
       "      <td id=\"T_3d2e7_row0_col5\" class=\"data row0 col5\" >nan</td>\n",
       "      <td id=\"T_3d2e7_row0_col6\" class=\"data row0 col6\" >Orson Welles</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_3d2e7_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "      <td id=\"T_3d2e7_row1_col0\" class=\"data row1 col0\" >Star Wars</td>\n",
       "      <td id=\"T_3d2e7_row1_col1\" class=\"data row1 col1\" >5</td>\n",
       "      <td id=\"T_3d2e7_row1_col2\" class=\"data row1 col2\" >1977</td>\n",
       "      <td id=\"T_3d2e7_row1_col3\" class=\"data row1 col3\" >George Lucas</td>\n",
       "      <td id=\"T_3d2e7_row1_col4\" class=\"data row1 col4\" >8</td>\n",
       "      <td id=\"T_3d2e7_row1_col5\" class=\"data row1 col5\" >nan</td>\n",
       "      <td id=\"T_3d2e7_row1_col6\" class=\"data row1 col6\" >nan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_3d2e7_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "      <td id=\"T_3d2e7_row2_col0\" class=\"data row2 col0\" >Pulp Fiction</td>\n",
       "      <td id=\"T_3d2e7_row2_col1\" class=\"data row2 col1\" >2</td>\n",
       "      <td id=\"T_3d2e7_row2_col2\" class=\"data row2 col2\" >1994</td>\n",
       "      <td id=\"T_3d2e7_row2_col3\" class=\"data row2 col3\" >Quentin Tarantino</td>\n",
       "      <td id=\"T_3d2e7_row2_col4\" class=\"data row2 col4\" >15</td>\n",
       "      <td id=\"T_3d2e7_row2_col5\" class=\"data row2 col5\" >Atheist</td>\n",
       "      <td id=\"T_3d2e7_row2_col6\" class=\"data row2 col6\" >John Travolta</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_3d2e7_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "      <td id=\"T_3d2e7_row3_col0\" class=\"data row3 col0\" >Night of the Living Dead</td>\n",
       "      <td id=\"T_3d2e7_row3_col1\" class=\"data row3 col1\" >229</td>\n",
       "      <td id=\"T_3d2e7_row3_col2\" class=\"data row3 col2\" >1968</td>\n",
       "      <td id=\"T_3d2e7_row3_col3\" class=\"data row3 col3\" >George A. Romero</td>\n",
       "      <td id=\"T_3d2e7_row3_col4\" class=\"data row3 col4\" >139</td>\n",
       "      <td id=\"T_3d2e7_row3_col5\" class=\"data row3 col5\" >Christian</td>\n",
       "      <td id=\"T_3d2e7_row3_col6\" class=\"data row3 col6\" >nan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_3d2e7_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "      <td id=\"T_3d2e7_row4_col0\" class=\"data row4 col0\" >The Piano</td>\n",
       "      <td id=\"T_3d2e7_row4_col1\" class=\"data row4 col1\" >307</td>\n",
       "      <td id=\"T_3d2e7_row4_col2\" class=\"data row4 col2\" >1993</td>\n",
       "      <td id=\"T_3d2e7_row4_col3\" class=\"data row4 col3\" >Jane Campion</td>\n",
       "      <td id=\"T_3d2e7_row4_col4\" class=\"data row4 col4\" >247</td>\n",
       "      <td id=\"T_3d2e7_row4_col5\" class=\"data row4 col5\" >Christian</td>\n",
       "      <td id=\"T_3d2e7_row4_col6\" class=\"data row4 col6\" >nan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_3d2e7_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
       "      <td id=\"T_3d2e7_row5_col0\" class=\"data row5 col0\" >Kill Bill: Vol. 2</td>\n",
       "      <td id=\"T_3d2e7_row5_col1\" class=\"data row5 col1\" >224</td>\n",
       "      <td id=\"T_3d2e7_row5_col2\" class=\"data row5 col2\" >2004</td>\n",
       "      <td id=\"T_3d2e7_row5_col3\" class=\"data row5 col3\" >Quentin Tarantino</td>\n",
       "      <td id=\"T_3d2e7_row5_col4\" class=\"data row5 col4\" >701</td>\n",
       "      <td id=\"T_3d2e7_row5_col5\" class=\"data row5 col5\" >nan</td>\n",
       "      <td id=\"T_3d2e7_row5_col6\" class=\"data row5 col6\" >nan</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f642f800a90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sorted_df = table_integrator.sort_tuples(expanded_integ_result, sortCol='Overall Rank')\n",
    "sorted_df = table_integrator.highlight_overlap_values(sorted_df)\n",
    "display(sorted_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_b0ae0_row0_col0, #T_b0ae0_row0_col1, #T_b0ae0_row0_col2, #T_b0ae0_row0_col3, #T_b0ae0_row0_col4 {\n",
       "  background-color: khaki;\n",
       "  font-weight: bold;\n",
       "}\n",
       "#T_b0ae0_row0_col5, #T_b0ae0_row1_col5 {\n",
       "  background-color: coral;\n",
       "  font-style: italic;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_b0ae0_\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th class=\"col_heading level0 col0\" >Title</th>\n",
       "      <th class=\"col_heading level0 col1\" >Fans' Rank</th>\n",
       "      <th class=\"col_heading level0 col2\" >Year</th>\n",
       "      <th class=\"col_heading level0 col3\" >Director(s)</th>\n",
       "      <th class=\"col_heading level0 col4\" >Overall Rank</th>\n",
       "      <th class=\"col_heading level0 col5\" >Religion</th>\n",
       "      <th class=\"col_heading level0 col6\" >Actor</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_b0ae0_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_b0ae0_row0_col0\" class=\"data row0 col0\" >Pulp Fiction</td>\n",
       "      <td id=\"T_b0ae0_row0_col1\" class=\"data row0 col1\" >2</td>\n",
       "      <td id=\"T_b0ae0_row0_col2\" class=\"data row0 col2\" >1994</td>\n",
       "      <td id=\"T_b0ae0_row0_col3\" class=\"data row0 col3\" >Quentin Tarantino</td>\n",
       "      <td id=\"T_b0ae0_row0_col4\" class=\"data row0 col4\" >15</td>\n",
       "      <td id=\"T_b0ae0_row0_col5\" class=\"data row0 col5\" >Atheist</td>\n",
       "      <td id=\"T_b0ae0_row0_col6\" class=\"data row0 col6\" >John Travolta</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_b0ae0_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "      <td id=\"T_b0ae0_row1_col0\" class=\"data row1 col0\" >Kill Bill: Vol. 2</td>\n",
       "      <td id=\"T_b0ae0_row1_col1\" class=\"data row1 col1\" >224</td>\n",
       "      <td id=\"T_b0ae0_row1_col2\" class=\"data row1 col2\" >2004</td>\n",
       "      <td id=\"T_b0ae0_row1_col3\" class=\"data row1 col3\" >Quentin Tarantino</td>\n",
       "      <td id=\"T_b0ae0_row1_col4\" class=\"data row1 col4\" >701</td>\n",
       "      <td id=\"T_b0ae0_row1_col5\" class=\"data row1 col5\" >nan</td>\n",
       "      <td id=\"T_b0ae0_row1_col6\" class=\"data row1 col6\" >nan</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f642f800a50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "filter_dict = {'Director(s)': ['Quentin Tarantino'], \n",
    "               'Overall Rank': [i for i in range(1000)]}\n",
    "filtered_df = table_integrator.filter_tuples(expanded_integ_result, filter_dict)\n",
    "filtered_df = table_integrator.highlight_overlap_values(filtered_df)\n",
    "display(filtered_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Expand-T vs. Outer Join\n",
    "Another method to reclaiming and expanding a source table is performing Outer Join on all candidate tables retrieved from the data lake. Instead of applying Gen-T to reclaim the table and expanding the table, we can directly integrate all candidate tables. However, as seen below, this integration method leads to many noisy tuples with nulls. Specifically, using outer join produces almost 2X as many tuples as Expand-T. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style type=\"text/css\">\n",
       "#T_e1be9_row0_col0, #T_e1be9_row0_col2, #T_e1be9_row1_col0, #T_e1be9_row1_col1, #T_e1be9_row1_col2, #T_e1be9_row1_col3, #T_e1be9_row1_col4, #T_e1be9_row2_col0, #T_e1be9_row2_col2, #T_e1be9_row2_col3, #T_e1be9_row2_col4, #T_e1be9_row3_col0, #T_e1be9_row3_col1, #T_e1be9_row3_col2, #T_e1be9_row3_col3, #T_e1be9_row3_col4, #T_e1be9_row4_col0, #T_e1be9_row4_col2, #T_e1be9_row5_col0, #T_e1be9_row5_col2, #T_e1be9_row5_col3, #T_e1be9_row5_col4, #T_e1be9_row6_col0, #T_e1be9_row6_col2, #T_e1be9_row6_col4, #T_e1be9_row7_col0, #T_e1be9_row7_col1, #T_e1be9_row7_col2, #T_e1be9_row7_col4 {\n",
       "  background-color: khaki;\n",
       "  font-weight: bold;\n",
       "}\n",
       "#T_e1be9_row0_col5, #T_e1be9_row1_col5, #T_e1be9_row2_col5, #T_e1be9_row3_col5, #T_e1be9_row4_col5, #T_e1be9_row5_col5, #T_e1be9_row6_col5, #T_e1be9_row7_col5, #T_e1be9_row8_col5, #T_e1be9_row9_col5, #T_e1be9_row10_col5 {\n",
       "  background-color: coral;\n",
       "  font-style: italic;\n",
       "}\n",
       "#T_e1be9_row6_col3, #T_e1be9_row7_col3 {\n",
       "  background-color: lightskyblue;\n",
       "  text-decoration: underline;\n",
       "}\n",
       "</style>\n",
       "<table id=\"T_e1be9_\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th class=\"blank level0\" >&nbsp;</th>\n",
       "      <th class=\"col_heading level0 col0\" >Title</th>\n",
       "      <th class=\"col_heading level0 col1\" >Fans' Rank</th>\n",
       "      <th class=\"col_heading level0 col2\" >Year</th>\n",
       "      <th class=\"col_heading level0 col3\" >Director(s)</th>\n",
       "      <th class=\"col_heading level0 col4\" >Overall Rank</th>\n",
       "      <th class=\"col_heading level0 col5\" >Religion</th>\n",
       "      <th class=\"col_heading level0 col6\" >Actor</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th id=\"T_e1be9_level0_row0\" class=\"row_heading level0 row0\" >0</th>\n",
       "      <td id=\"T_e1be9_row0_col0\" class=\"data row0 col0\" >Pulp Fiction</td>\n",
       "      <td id=\"T_e1be9_row0_col1\" class=\"data row0 col1\" >nan</td>\n",
       "      <td id=\"T_e1be9_row0_col2\" class=\"data row0 col2\" >1994</td>\n",
       "      <td id=\"T_e1be9_row0_col3\" class=\"data row0 col3\" >nan</td>\n",
       "      <td id=\"T_e1be9_row0_col4\" class=\"data row0 col4\" >nan</td>\n",
       "      <td id=\"T_e1be9_row0_col5\" class=\"data row0 col5\" >nan</td>\n",
       "      <td id=\"T_e1be9_row0_col6\" class=\"data row0 col6\" >nan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_e1be9_level0_row1\" class=\"row_heading level0 row1\" >1</th>\n",
       "      <td id=\"T_e1be9_row1_col0\" class=\"data row1 col0\" >Pulp Fiction</td>\n",
       "      <td id=\"T_e1be9_row1_col1\" class=\"data row1 col1\" >2</td>\n",
       "      <td id=\"T_e1be9_row1_col2\" class=\"data row1 col2\" >1994</td>\n",
       "      <td id=\"T_e1be9_row1_col3\" class=\"data row1 col3\" >Quentin Tarantino</td>\n",
       "      <td id=\"T_e1be9_row1_col4\" class=\"data row1 col4\" >15</td>\n",
       "      <td id=\"T_e1be9_row1_col5\" class=\"data row1 col5\" >Atheist</td>\n",
       "      <td id=\"T_e1be9_row1_col6\" class=\"data row1 col6\" >nan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_e1be9_level0_row2\" class=\"row_heading level0 row2\" >2</th>\n",
       "      <td id=\"T_e1be9_row2_col0\" class=\"data row2 col0\" >Pulp Fiction</td>\n",
       "      <td id=\"T_e1be9_row2_col1\" class=\"data row2 col1\" >nan</td>\n",
       "      <td id=\"T_e1be9_row2_col2\" class=\"data row2 col2\" >1994</td>\n",
       "      <td id=\"T_e1be9_row2_col3\" class=\"data row2 col3\" >Quentin Tarantino</td>\n",
       "      <td id=\"T_e1be9_row2_col4\" class=\"data row2 col4\" >15</td>\n",
       "      <td id=\"T_e1be9_row2_col5\" class=\"data row2 col5\" >nan</td>\n",
       "      <td id=\"T_e1be9_row2_col6\" class=\"data row2 col6\" >John Travolta</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_e1be9_level0_row3\" class=\"row_heading level0 row3\" >3</th>\n",
       "      <td id=\"T_e1be9_row3_col0\" class=\"data row3 col0\" >Citizen Kane</td>\n",
       "      <td id=\"T_e1be9_row3_col1\" class=\"data row3 col1\" >3</td>\n",
       "      <td id=\"T_e1be9_row3_col2\" class=\"data row3 col2\" >1941</td>\n",
       "      <td id=\"T_e1be9_row3_col3\" class=\"data row3 col3\" >Orson Welles</td>\n",
       "      <td id=\"T_e1be9_row3_col4\" class=\"data row3 col4\" >2</td>\n",
       "      <td id=\"T_e1be9_row3_col5\" class=\"data row3 col5\" >nan</td>\n",
       "      <td id=\"T_e1be9_row3_col6\" class=\"data row3 col6\" >nan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_e1be9_level0_row4\" class=\"row_heading level0 row4\" >4</th>\n",
       "      <td id=\"T_e1be9_row4_col0\" class=\"data row4 col0\" >Citizen Kane</td>\n",
       "      <td id=\"T_e1be9_row4_col1\" class=\"data row4 col1\" >nan</td>\n",
       "      <td id=\"T_e1be9_row4_col2\" class=\"data row4 col2\" >1941</td>\n",
       "      <td id=\"T_e1be9_row4_col3\" class=\"data row4 col3\" >nan</td>\n",
       "      <td id=\"T_e1be9_row4_col4\" class=\"data row4 col4\" >nan</td>\n",
       "      <td id=\"T_e1be9_row4_col5\" class=\"data row4 col5\" >nan</td>\n",
       "      <td id=\"T_e1be9_row4_col6\" class=\"data row4 col6\" >nan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_e1be9_level0_row5\" class=\"row_heading level0 row5\" >5</th>\n",
       "      <td id=\"T_e1be9_row5_col0\" class=\"data row5 col0\" >Citizen Kane</td>\n",
       "      <td id=\"T_e1be9_row5_col1\" class=\"data row5 col1\" >nan</td>\n",
       "      <td id=\"T_e1be9_row5_col2\" class=\"data row5 col2\" >1941</td>\n",
       "      <td id=\"T_e1be9_row5_col3\" class=\"data row5 col3\" >Orson Welles</td>\n",
       "      <td id=\"T_e1be9_row5_col4\" class=\"data row5 col4\" >2</td>\n",
       "      <td id=\"T_e1be9_row5_col5\" class=\"data row5 col5\" >nan</td>\n",
       "      <td id=\"T_e1be9_row5_col6\" class=\"data row5 col6\" >Orson Welles</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_e1be9_level0_row6\" class=\"row_heading level0 row6\" >6</th>\n",
       "      <td id=\"T_e1be9_row6_col0\" class=\"data row6 col0\" >Star Wars</td>\n",
       "      <td id=\"T_e1be9_row6_col1\" class=\"data row6 col1\" >nan</td>\n",
       "      <td id=\"T_e1be9_row6_col2\" class=\"data row6 col2\" >1977</td>\n",
       "      <td id=\"T_e1be9_row6_col3\" class=\"data row6 col3\" >George Lucas</td>\n",
       "      <td id=\"T_e1be9_row6_col4\" class=\"data row6 col4\" >8</td>\n",
       "      <td id=\"T_e1be9_row6_col5\" class=\"data row6 col5\" >nan</td>\n",
       "      <td id=\"T_e1be9_row6_col6\" class=\"data row6 col6\" >nan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_e1be9_level0_row7\" class=\"row_heading level0 row7\" >7</th>\n",
       "      <td id=\"T_e1be9_row7_col0\" class=\"data row7 col0\" >Star Wars</td>\n",
       "      <td id=\"T_e1be9_row7_col1\" class=\"data row7 col1\" >5</td>\n",
       "      <td id=\"T_e1be9_row7_col2\" class=\"data row7 col2\" >1977</td>\n",
       "      <td id=\"T_e1be9_row7_col3\" class=\"data row7 col3\" >George Lucas</td>\n",
       "      <td id=\"T_e1be9_row7_col4\" class=\"data row7 col4\" >8</td>\n",
       "      <td id=\"T_e1be9_row7_col5\" class=\"data row7 col5\" >nan</td>\n",
       "      <td id=\"T_e1be9_row7_col6\" class=\"data row7 col6\" >nan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_e1be9_level0_row8\" class=\"row_heading level0 row8\" >8</th>\n",
       "      <td id=\"T_e1be9_row8_col0\" class=\"data row8 col0\" >The Piano</td>\n",
       "      <td id=\"T_e1be9_row8_col1\" class=\"data row8 col1\" >307</td>\n",
       "      <td id=\"T_e1be9_row8_col2\" class=\"data row8 col2\" >1993</td>\n",
       "      <td id=\"T_e1be9_row8_col3\" class=\"data row8 col3\" >Jane Campion</td>\n",
       "      <td id=\"T_e1be9_row8_col4\" class=\"data row8 col4\" >247</td>\n",
       "      <td id=\"T_e1be9_row8_col5\" class=\"data row8 col5\" >Christian</td>\n",
       "      <td id=\"T_e1be9_row8_col6\" class=\"data row8 col6\" >nan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_e1be9_level0_row9\" class=\"row_heading level0 row9\" >9</th>\n",
       "      <td id=\"T_e1be9_row9_col0\" class=\"data row9 col0\" >Night of the Living Dead</td>\n",
       "      <td id=\"T_e1be9_row9_col1\" class=\"data row9 col1\" >229</td>\n",
       "      <td id=\"T_e1be9_row9_col2\" class=\"data row9 col2\" >1968</td>\n",
       "      <td id=\"T_e1be9_row9_col3\" class=\"data row9 col3\" >George A. Romero</td>\n",
       "      <td id=\"T_e1be9_row9_col4\" class=\"data row9 col4\" >139</td>\n",
       "      <td id=\"T_e1be9_row9_col5\" class=\"data row9 col5\" >Christian</td>\n",
       "      <td id=\"T_e1be9_row9_col6\" class=\"data row9 col6\" >nan</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th id=\"T_e1be9_level0_row10\" class=\"row_heading level0 row10\" >10</th>\n",
       "      <td id=\"T_e1be9_row10_col0\" class=\"data row10 col0\" >Kill Bill: Vol. 2</td>\n",
       "      <td id=\"T_e1be9_row10_col1\" class=\"data row10 col1\" >224</td>\n",
       "      <td id=\"T_e1be9_row10_col2\" class=\"data row10 col2\" >2004</td>\n",
       "      <td id=\"T_e1be9_row10_col3\" class=\"data row10 col3\" >Quentin Tarantino</td>\n",
       "      <td id=\"T_e1be9_row10_col4\" class=\"data row10 col4\" >701</td>\n",
       "      <td id=\"T_e1be9_row10_col5\" class=\"data row10 col5\" >nan</td>\n",
       "      <td id=\"T_e1be9_row10_col6\" class=\"data row10 col6\" >nan</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n"
      ],
      "text/plain": [
       "<pandas.io.formats.style.Styler at 0x7f642d5a2ed0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "joined_df = utils.outerjoin(list(originating_tables_finder.candidate_table_dfs.values()))\n",
    "joined_df = table_integrator.order_rows_cols(joined_df)\n",
    "joined_df = table_integrator.highlight_overlap_values(joined_df)\n",
    "display(joined_df)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "sdd",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}