04_refutation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "08ea43f7-a118-44bf-9e8f-0a64ca254932",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "This solution accelerator notebook is available at https://github.com/databricks-industry-solutions/causal-incentive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "5252e041-9be1-4fae-b0d2-d1ea362b1e74",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "%run ./util/notebook-config"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "715d7031-369f-4b0c-a7d2-5a9d6db62340",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "##Refutation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "44a73459-20fa-468d-b0c6-8491949a976c",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "- <b>How much should we trust the estimators used by the recommender?</b>\n",
    "\n",
    "- <b>Would they break if new features are added?</b>\n",
    "\n",
    "- <b>What if there was a subset of the data driving the results, but the estimation does not applied to other subsets?</b>\n",
    "\n",
    "To questions related to the level of trust we should have in the estimators, we will execute a series of tests to determine the degree of sensitivity the models have to deviations in our dataset and assumptions\n",
    "\n",
    "The [DoWhy](https://www.pywhy.org/dowhy/v0.8/user_guide/effect_inference/refute.html) package of [PyWhy](https://github.com/py-why) provides us with a battery of predefine \"refutation\" tests we can easily use for these purpose. [The approach taken by DoWhy](https://github.com/py-why/dowhy/issues/312) is based on [statistical hypothesis testing](https://en.wikipedia.org/wiki/Statistical_hypothesis_testing), where the null hypothesis is that the test did not detect a problem in the model.  In other words, if the test p-value is less than 0.05, you would conclude with a 95% confidence level that the model should not be trusted if a similar deviation as the one simluated would actually take place.\n",
    "\n",
    "Please note this phase is computationally expensive as it simulates many scenarios.  In order to keep the tests simple and easy to follow, no distributed computing approach have been.  Some multithreading capabilities already provided by the package as been leverage.  This notebook will take longer time than the previous ones.  To avoid even a longer execution time we will focus on one of the estimators: the ```Discount``` effect estimator.  These test don't need to be continuesly execute,  instead they should be executed when when new estimators are trained."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "4c90b52c-6de2-4625-92c4-b9742865b275",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "First, lets load the ```Discount``` effect estimator model from [MLflow](https://www.databricks.com/product/managed-mlflow)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "f86ad090-bbe1-4d34-92bc-7487b746135d",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "wrapped_model = get_registered_wrapped_model(model_name=f\"{catalog}.{db}.discount_dowhy_model\")\n",
    "\n",
    "model = wrapped_model.get_model()\n",
    "estimand = wrapped_model.get_estimand()\n",
    "estimate = wrapped_model.get_estimate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "bd614eae-c2f1-4e20-aea3-d089e985fb0f",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "Let's now test if adding an artificial feature which influences both the the probability of giving a ```Discount``` and the ```Revenue```, would yield a significant different result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "ec92f977-642e-4f60-8ee1-4ed65ca61916",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "res_random_common_cause = model.refute_estimate(\n",
    "    estimand=estimand,\n",
    "    estimate=estimate,\n",
    "    show_progress_bar=True,\n",
    "    method_name=\"random_common_cause\",\n",
    "    num_simulations=100,\n",
    "    n_jobs=16,\n",
    ")\n",
    "\n",
    "refutation_random_common_cause_df = pd.DataFrame(\n",
    "    [\n",
    "        {\n",
    "            \"Refutation Type\": res_random_common_cause.refutation_type,\n",
    "            \"Estimated Effect\": res_random_common_cause.estimated_effect,\n",
    "            \"New Effect\": res_random_common_cause.new_effect,\n",
    "            \"Refutation Result (p value)\": res_random_common_cause.refutation_result[\n",
    "                \"p_value\"\n",
    "            ],\n",
    "        }\n",
    "    ]\n",
    ")\n",
    "\n",
    "refutation_random_common_cause_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "fd8c2bb9-4725-46f4-95a6-944618c51585",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "Let's now execute a similar test, but this time the new feature will be similated but not included in the estimation.  This mimics an scenario where a factor that influences both ```Discount``` and ```Revenue``` exists but we are unaware of it,  in other words the factor is \"un-observed\".  \n",
    "\n",
    "The test reports back a plot showing the effect of different \"unobserved\" factor values in the probability of applying \"treatment\" or incentive (```Discount```),  the outcome value (```Revenue```), and the estimated incentive effect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "703c0184-37c6-4c1e-9321-9c2e6c9a200a",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "mlflow.autolog(disable=True)\n",
    "\n",
    "res_unobserved_common_cause = model.refute_estimate(\n",
    "    estimand=estimand,\n",
    "    estimate=estimate,\n",
    "    show_progress_bar=True,\n",
    "    method_name=\"add_unobserved_common_cause\",\n",
    "    confounders_effect_on_treatment=\"binary_flip\",\n",
    "    confounders_effect_on_outcome=\"linear\",\n",
    "    effect_fraction_on_treatment=0.05,\n",
    "    effect_fraction_on_outcome=0.05,\n",
    ")\n",
    "\n",
    "refutation_unobserved_common_cause_df = pd.DataFrame(\n",
    "    [\n",
    "        {\n",
    "            \"Refutation Type\": res_unobserved_common_cause.refutation_type,\n",
    "            \"Estimated Effect\": res_unobserved_common_cause.estimated_effect,\n",
    "            \"New Effect\": res_unobserved_common_cause.new_effect,\n",
    "            \"Refutation Result (p value)\": None,\n",
    "        }\n",
    "    ]\n",
    ")\n",
    "\n",
    "refutation_unobserved_common_cause_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "e3a0d131-1e6e-486b-bc61-909fa138573f",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "The following test changes the order of the ```Discount``` values in the dataset,  braking the relation between the ```Discount``` and the ```Revenue``` in a given account.  As a result the model should not predict a good estimation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "bf97f2c7-7a2d-4d9c-8655-e07b589411bb",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "res_placebo = model.refute_estimate(\n",
    "    estimand=estimand,\n",
    "    estimate=estimate,\n",
    "    show_progress_bar=True,\n",
    "    method_name=\"placebo_treatment_refuter\",\n",
    "    placebo_type=\"permute\",\n",
    "    num_simulations=100,\n",
    "    n_jobs=16,\n",
    ")\n",
    "\n",
    "refutation_placebo_df = pd.DataFrame(\n",
    "    [\n",
    "        {\n",
    "            \"Refutation Type\": res_placebo.refutation_type,\n",
    "            \"Estimated Effect\": res_placebo.estimated_effect,\n",
    "            \"New Effect\": res_placebo.new_effect,\n",
    "            \"Refutation Result (p value)\": res_placebo.refutation_result[\"p_value\"],\n",
    "        }\n",
    "    ]\n",
    ")\n",
    "\n",
    "refutation_placebo_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "f786d7ca-774f-4d3f-84bd-ff6ad2b377c8",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "Next, we will apply the estimation on many non-overlaping subsets of the dataset.  The average of the estimations should be close enough to the estimation done with the full dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "59810965-5ed9-47f9-ad0a-d396984b6dea",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "res_subset = model.refute_estimate(\n",
    "    estimand=estimand,\n",
    "    estimate=estimate,\n",
    "    show_progress_bar=True,\n",
    "    method_name=\"data_subset_refuter\",\n",
    "    subset_fraction=0.8,\n",
    "    num_simulations=100,\n",
    "    n_jobs=16,\n",
    ")\n",
    "\n",
    "refutation_subset_df = pd.DataFrame(\n",
    "    [\n",
    "        {\n",
    "            \"Refutation Type\": res_subset.refutation_type,\n",
    "            \"Estimated Effect\": res_subset.estimated_effect,\n",
    "            \"New Effect\": res_subset.new_effect,\n",
    "            \"Refutation Result (p value)\": res_subset.refutation_result[\"p_value\"],\n",
    "        }\n",
    "    ]\n",
    ")\n",
    "\n",
    "refutation_subset_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "fd7bc428-3ce6-45b6-8342-05ee3cbfba28",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "Lastely,  we will replace the ```Revenue``` values with artifitial randomly generated values. The estimation should show no effect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "44ab4124-32ed-4938-876e-584c33358c9a",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "mlflow.autolog(disable=True)\n",
    "\n",
    "coefficients = np.array([10, 0.02])\n",
    "bias = 1000\n",
    "\n",
    "\n",
    "def linear_gen(df):\n",
    "    y_new = np.dot(df[[\"W0\", \"W1\"]].values, coefficients) + bias\n",
    "    return y_new\n",
    "\n",
    "\n",
    "res_dummy_outcome = model.refute_estimate(\n",
    "    estimand=estimand,\n",
    "    estimate=estimate,\n",
    "    show_progress_bar=True,\n",
    "    method_name=\"dummy_outcome_refuter\",\n",
    "    outcome_function=linear_gen,\n",
    ")[0]\n",
    "\n",
    "refutation_dummy_outcome_df = pd.DataFrame(\n",
    "    [\n",
    "        {\n",
    "            \"Refutation Type\": res_dummy_outcome.refutation_type,\n",
    "            \"Estimated Effect\": res_dummy_outcome.estimated_effect,\n",
    "            \"New Effect\": res_dummy_outcome.new_effect,\n",
    "            \"Refutation Result (p value)\": res_dummy_outcome.refutation_result[\n",
    "                \"p_value\"\n",
    "            ],\n",
    "        }\n",
    "    ]\n",
    ")\n",
    "\n",
    "refutation_dummy_outcome_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "eac0f339-0fbb-49db-b7f0-3d981d499596",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "When comparing all the tests results we can see the estimator is most sensitive to unobserved factors. This is in a way expected, as the test breaks one of the [assumptions](https://journals.lww.com/epidem/fulltext/2009/01000/the_consistency_statement_in_causal_inference__a.3.aspx#:~:text=Three%20assumptions%20sufficient%20to%20identify,measurement%20of%20the%20outcome%E2%80%9D).) in which the approaches presented are based.  The value obtain by applying that specific test to understanding the degree of impact of unobserved factors in the estimation.\n",
    "\n",
    "The rest of the tests have p-values higher than 0.05"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "065c5f30-7bfb-4b30-b0b7-6e45ed7fe2df",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "refutation_df = pd.concat(\n",
    "    [\n",
    "        refutation_random_common_cause_df,\n",
    "        refutation_unobserved_common_cause_df,\n",
    "        refutation_subset_df,\n",
    "        refutation_placebo_df,\n",
    "        refutation_dummy_outcome_df,\n",
    "    ]\n",
    ")\n",
    "refutation_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "bea61eee-89be-465c-a2e2-2350902cbdde",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "© 2023 Databricks, Inc. All rights reserved. The source in this notebook is provided subject to the Databricks License. All included or referenced third party libraries are subject to the licenses set forth below.\n",
    "\n",
    "| library                                | description             | license    | source                                              |\n",
    "|----------------------------------------|-------------------------|------------|-----------------------------------------------------|\n",
    "| dowhy   | Python library for causal inference that supports explicit modeling and testing of causal assumptions | MIT   | https://pypi.org/project/dowhy/          \n",
    "| econml   |  contains several methods for calculating Conditional Average Treatment Effects | MIT    | https://pypi.org/project/econml/  \n",
    "| causal-learn   | python package for causal discovery  | MIT    | https://pypi.org/project/causal-learn/"
   ]
  }
 ],
 "metadata": {
  "application/vnd.databricks.v1+notebook": {
   "dashboards": [],
   "environmentMetadata": null,
   "language": "python",
   "notebookMetadata": {
    "pythonIndentUnit": 2
   },
   "notebookName": "04_refutation",
   "widgets": {}
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}