XOR_TCAV.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyNCE0c8khhPZ2vzCf8ckzoG"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "V3VFHzxkE_WL",
        "outputId": "f4d35de4-296b-4b2b-de14-c39e3f89cf10"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%cd /content/drive/MyDrive/Data_Science_Projects/TCAV_Investigation"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lXczKha_E_ej",
        "outputId": "208c8c47-6420-4982-c46e-81c5cfb49adf"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/drive/MyDrive/Data_Science_Projects/TCAV_Investigation\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## TCAV\n",
        "\n"
      ],
      "metadata": {
        "id": "4VUr_lGlqo_y"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "TCAV, or *Testing with Concept Activation Vectors*, is a well-known tool for interpreting deep learning models first proposed by Kim *et. al* [1]. TCAV is intended to test whether a model relies on human-understandable concepts in its hidden layers when those concepts aren't explicitly among the input features.\n",
        "\n",
        "[1] https://arxiv.org/abs/1711.11279\n",
        "\n",
        "For instance, in many tasks we don't want the model's reasoning about us to rely on race or gender (Kim *et. al* discuss image classification examples). Even if race or gender aren't found in the model inputs, though, our model might be \"learning\" them somewhere in its hidden layers, possibly leading to biased outputs.\n",
        "\n",
        "An in-depth description of TCAV can be found in the original paper, but here's a broad overview of how the method checks whether concept *C* is important for a model's reasoning about output class *k* in dataset *X* at hidden layer *l*:\n",
        "\n",
        " 1. Retrieve the activations from the class-*k* members of *X* at *l*.\n",
        " 2. Fit a linear classifier that separates the activations based on whether they are from a *C* example or a non-*C* example.\n",
        " 3. Define the Concept Activation Vector (CAV) for *C* as the unit vector orthogonal to the boundary hyperplane in the direction of *C*.\n",
        " 4. Calculate gradients of the model outputs with respect to the layer *l* activations.\n",
        " 5. Calculate directional derivatives of the gradients in the direction of *C* by taking the dot products of the gradients and the CAV. These are the *sensitivities* of the model outputs to concept *C* at layer *l*.\n",
        " 6. Calculate the *TCAV score* of concept *C* at layer *l* by finding the proportion of the dataset with positive sensitivity.\n",
        "\n",
        "TCAV scores range from 0 to 1. According to Kim *et. al*, a TCAV score away from 0.5 in ether direction indicates that \"we can consider the resulting concept as related to the class prediction\".\n",
        "\n",
        "This project tries to gain a better understanding of how TCAV works. I give examples of the method failing to identify important concepts and identifying unimportant concepts. I conclude that TCAV score is a measure of the *directional uniformity* of a concept's effect on the model predictions."
      ],
      "metadata": {
        "id": "GAeiXF8QYZD5"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Implementation"
      ],
      "metadata": {
        "id": "2_GkmBDs-Af6"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "I use TensorFlow to implement a basic TCAV class that performs every step listed above."
      ],
      "metadata": {
        "id": "FKVkMwC7-DNp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "from sklearn.linear_model import SGDClassifier\n",
        "import graphviz\n",
        "import copy\n",
        "\n",
        "tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR)"
      ],
      "metadata": {
        "id": "3L6-IvE_FNf1"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class TCAV():\n",
        "    def __init__(self, model, layer_idx=0, random_state=0):\n",
        "        self.layer_activations = None\n",
        "        self.cav = None\n",
        "\n",
        "        self.model = model\n",
        "        self.lm = SGDClassifier(penalty='l1', alpha=0.01, random_state=random_state)\n",
        "        self._set_layer(layer_idx)\n",
        "        self._set_submodels()\n",
        "\n",
        "    def _set_layer(self, layer_idx):\n",
        "        self.layer_idx = layer_idx\n",
        "        self.layer = self.model.layers[layer_idx]\n",
        "\n",
        "    def _set_submodels(self):\n",
        "        # Submodel taking inputs to layer activations\n",
        "        self.model_f = tf.keras.Model(inputs=self.model.input,\n",
        "                                      outputs=self.model.layers[self.layer_idx].output)\n",
        "        # Submodel taking layer activations to outputs\n",
        "        model_h_input = copy.deepcopy(tf.keras.layers.Input(self.model.layers[self.layer_idx+1].input_shape[1:]))\n",
        "        output = model_h_input\n",
        "        for layer in self.model.layers[self.layer_idx + 1:]:\n",
        "            output = copy.deepcopy(layer)(output)\n",
        "        self.model_h = tf.keras.Model(inputs=model_h_input, outputs=output)\n",
        "\n",
        "    def _set_layer_activations(self, X):\n",
        "        self.layer_activations = tf.convert_to_tensor(self.model_f(X))\n",
        "\n",
        "    def _set_gradients(self): # requires self.layer_activations\n",
        "        with tf.GradientTape() as tape:\n",
        "            tape.watch(self.layer_activations)\n",
        "            y_hat = self.model_h(self.layer_activations)\n",
        "        self.grad = tf.keras.backend.get_value(tape.gradient(y_hat, self.layer_activations))\n",
        "\n",
        "    def _set_sensitivity(self, concept_labels, y, y_value): # requires self.cav\n",
        "        self._set_gradients()\n",
        "        self.sensitivity = np.dot(self.grad[y==y_value], self.cav)\n",
        "\n",
        "    def train_cav(self, X, concept_labels):\n",
        "        # Calculate hidden layer activations\n",
        "        self._set_layer_activations(X)\n",
        "        # Fit hidden layer activations\n",
        "        self.lm.fit(self.layer_activations, concept_labels)\n",
        "        self.cav = self.lm.coef_.T/np.linalg.norm(self.lm.coef_)\n",
        "        return self.cav\n",
        "\n",
        "    def get_layer_prediction_accuracy(self, X, concept_labels):\n",
        "        # fit model\n",
        "        if self.cav is None:\n",
        "            self.train_cav(X, concept_labels)\n",
        "        # return accuracy score\n",
        "        return self.lm.score(self.layer_activations, concept_labels)\n",
        "\n",
        "    def tcav_score(self, X, concept_labels, y, y_value):\n",
        "        # fit model\n",
        "        if self.cav is None:\n",
        "            self.train_cav(X, concept_labels)\n",
        "        # calculate input sensitivities\n",
        "        self._set_sensitivity(concept_labels, y, y_value)\n",
        "        # return TCAV score\n",
        "        tcav_score = (self.sensitivity > 0).sum()/(y == y_value).sum()\n",
        "        return tcav_score"
      ],
      "metadata": {
        "id": "bzC_QYitGB-N"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Model"
      ],
      "metadata": {
        "id": "xdF-rjphGReb"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "To investigate TCAV, I'll see how it works on a relatively simple example in which we're able to know exactly how different concepts affect the model's reasoning. To this end, I define a \"dataset\" that contains 3 Boolean features A, B, and C and a $y$ variable representing (A XOR B) XOR C."
      ],
      "metadata": {
        "id": "9Q_UBVo7Yi1Q"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "zX4prilHEuBp"
      },
      "outputs": [],
      "source": [
        "df = pd.DataFrame({'A': [1,0,1,0,1,0,1,0],\n",
        "                   'B': [1,1,0,0,1,1,0,0],\n",
        "                   'C': [1,1,1,1,0,0,0,0],\n",
        "                   'A OR B': [1,1,1,0,1,1,1,0],\n",
        "                   '~(A AND B)': [0,1,1,1,0,1,1,1],\n",
        "                   'A XOR B': [0,1,1,0,0,1,1,0],\n",
        "                   '(A XOR B) XOR C': [1,0,0,1,0,1,1,0]},\n",
        "                  index=range(1,9))\n",
        "\n",
        "input_columns = ['A', 'B', 'C']\n",
        "X = tf.Variable(df[input_columns].values, dtype=tf.float64)\n",
        "y = df['(A XOR B) XOR C'].values"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "XOR problems are interesting because the target classes aren't linearly separable. Solving them with neural networks requires hidden layers. The diagram below shows a solution for a 2-input XOR problem that takes advantage of the fact that A XOR B is logically equivalent to (A OR B) AND ~(A AND B). Each gate contains an activation function that goes to 1 given positive input and 0 otherwise."
      ],
      "metadata": {
        "id": "GyKFF5mL3jcV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "graphviz.Source.from_file('graphs/saved_graphs/network1.gv')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 302
        },
        "id": "XdMZ6qEeeZ31",
        "outputId": "835badc2-d4b2-4396-8b9a-a86b2c5b5465"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "image/svg+xml": "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Generated by graphviz version 2.43.0 (0)\n -->\n<!-- Title: graph Pages: 1 -->\n<svg width=\"307pt\" height=\"211pt\"\n viewBox=\"0.00 0.00 306.69 210.92\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 206.92)\">\n<title>graph</title>\n<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-206.92 302.69,-206.92 302.69,4 -4,4\"/>\n<!-- a -->\n<g id=\"node1\" class=\"node\">\n<title>a</title>\n<ellipse fill=\"none\" stroke=\"black\" cx=\"27\" cy=\"-97.92\" rx=\"27\" ry=\"18\"/>\n<text text-anchor=\"middle\" x=\"27\" y=\"-94.22\" font-family=\"Times,serif\" font-size=\"14.00\">A</text>\n</g>\n<!-- b -->\n<g id=\"node2\" class=\"node\">\n<title>b</title>\n<ellipse fill=\"none\" stroke=\"black\" cx=\"27\" cy=\"-25.92\" rx=\"27\" ry=\"18\"/>\n<text text-anchor=\"middle\" x=\"27\" y=\"-22.22\" font-family=\"Times,serif\" font-size=\"14.00\">B</text>\n</g>\n<!-- a&#45;&gt;b -->\n<!-- d -->\n<g id=\"node4\" class=\"node\">\n<title>d</title>\n<ellipse fill=\"none\" stroke=\"black\" cx=\"146.45\" cy=\"-112.92\" rx=\"27\" ry=\"18\"/>\n<text text-anchor=\"middle\" x=\"146.45\" y=\"-109.22\" font-family=\"Times,serif\" font-size=\"14.00\">OR</text>\n</g>\n<!-- a&#45;&gt;d -->\n<g id=\"edge4\" class=\"edge\">\n<title>a&#45;&gt;d</title>\n<path fill=\"none\" stroke=\"black\" d=\"M53.77,-101.21C70.14,-103.3 91.56,-106.04 109.58,-108.34\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"109.37,-111.84 119.73,-109.64 110.26,-104.9 109.37,-111.84\"/>\n<text text-anchor=\"middle\" x=\"83\" y=\"-109.72\" font-family=\"Times,serif\" font-size=\"14.00\">1</text>\n</g>\n<!-- e -->\n<g id=\"node5\" class=\"node\">\n<title>e</title>\n<ellipse fill=\"none\" stroke=\"black\" cx=\"146.45\" cy=\"-40.92\" rx=\"34.39\" ry=\"18\"/>\n<text text-anchor=\"middle\" x=\"146.45\" y=\"-37.22\" font-family=\"Times,serif\" font-size=\"14.00\">~AND</text>\n</g>\n<!-- a&#45;&gt;e -->\n<g id=\"edge5\" class=\"edge\">\n<title>a&#45;&gt;e</title>\n<path fill=\"none\" stroke=\"black\" d=\"M35.65,-80.66C42.83,-66.82 55.02,-48.35 72,-39.92 81.78,-35.07 93.24,-33.57 104.15,-33.73\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"104.19,-37.24 114.38,-34.34 104.6,-30.25 104.19,-37.24\"/>\n<text text-anchor=\"middle\" x=\"83\" y=\"-43.72\" font-family=\"Times,serif\" font-size=\"14.00\">&#45;1</text>\n</g>\n<!-- b&#45;&gt;d -->\n<g id=\"edge7\" class=\"edge\">\n<title>b&#45;&gt;d</title>\n<path fill=\"none\" stroke=\"black\" d=\"M45.84,-39.14C65.53,-53.72 97.39,-77.32 119.76,-93.89\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"117.71,-96.73 127.83,-99.87 121.87,-91.1 117.71,-96.73\"/>\n<text text-anchor=\"middle\" x=\"83\" y=\"-78.72\" font-family=\"Times,serif\" font-size=\"14.00\">1</text>\n</g>\n<!-- b&#45;&gt;e -->\n<g id=\"edge8\" class=\"edge\">\n<title>b&#45;&gt;e</title>\n<path fill=\"none\" stroke=\"black\" d=\"M45.57,-12.77C58.82,-4.51 77.47,3.63 94,-1.92 104.2,-5.35 113.98,-11.69 122.23,-18.28\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"120.05,-21.02 129.92,-24.85 124.59,-15.69 120.05,-21.02\"/>\n<text text-anchor=\"middle\" x=\"83\" y=\"-5.72\" font-family=\"Times,serif\" font-size=\"14.00\">&#45;1</text>\n</g>\n<!-- const1 -->\n<g id=\"node3\" class=\"node\">\n<title>const1</title>\n<ellipse fill=\"none\" stroke=\"black\" cx=\"27\" cy=\"-169.92\" rx=\"27\" ry=\"18\"/>\n<text text-anchor=\"middle\" x=\"27\" y=\"-166.22\" font-family=\"Times,serif\" font-size=\"14.00\">1</text>\n</g>\n<!-- const1&#45;&gt;a -->\n<!-- const1&#45;&gt;d -->\n<g id=\"edge1\" class=\"edge\">\n<title>const1&#45;&gt;d</title>\n<path fill=\"none\" stroke=\"black\" d=\"M53.47,-174.14C66.17,-175.12 81.45,-174.55 94,-168.92 108.54,-162.4 120.81,-149.58 129.71,-137.85\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"132.63,-139.78 135.58,-129.61 126.93,-135.72 132.63,-139.78\"/>\n<text text-anchor=\"middle\" x=\"83\" y=\"-177.72\" font-family=\"Times,serif\" font-size=\"14.00\">&#45;0.5</text>\n</g>\n<!-- const1&#45;&gt;e -->\n<g id=\"edge2\" class=\"edge\">\n<title>const1&#45;&gt;e</title>\n<path fill=\"none\" stroke=\"black\" d=\"M49.26,-159.36C63.42,-151.47 81.64,-139.53 94,-124.92 106.33,-110.35 102.25,-102.34 112,-85.92 116.03,-79.13 120.87,-72.13 125.59,-65.72\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"128.62,-67.52 131.88,-57.44 123.04,-63.29 128.62,-67.52\"/>\n<text text-anchor=\"middle\" x=\"83\" y=\"-147.72\" font-family=\"Times,serif\" font-size=\"14.00\">1.5</text>\n</g>\n<!-- d&#45;&gt;e -->\n<!-- g -->\n<g id=\"node7\" class=\"node\">\n<title>g</title>\n<ellipse fill=\"none\" stroke=\"black\" cx=\"268.79\" cy=\"-112.92\" rx=\"29.8\" ry=\"18\"/>\n<text text-anchor=\"middle\" x=\"268.79\" y=\"-109.22\" font-family=\"Times,serif\" font-size=\"14.00\">AND</text>\n</g>\n<!-- d&#45;&gt;g -->\n<g id=\"edge11\" class=\"edge\">\n<title>d&#45;&gt;g</title>\n<path fill=\"none\" stroke=\"black\" d=\"M173.56,-112.92C189.62,-112.92 210.52,-112.92 228.55,-112.92\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"228.78,-116.42 238.78,-112.92 228.78,-109.42 228.78,-116.42\"/>\n<text text-anchor=\"middle\" x=\"209.89\" y=\"-116.72\" font-family=\"Times,serif\" font-size=\"14.00\">1</text>\n</g>\n<!-- e&#45;&gt;g -->\n<g id=\"edge13\" class=\"edge\">\n<title>e&#45;&gt;g</title>\n<path fill=\"none\" stroke=\"black\" d=\"M169.8,-54.3C189.07,-65.83 217.04,-82.56 238.17,-95.2\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"236.39,-98.21 246.77,-100.34 239.98,-92.21 236.39,-98.21\"/>\n<text text-anchor=\"middle\" x=\"209.89\" y=\"-87.72\" font-family=\"Times,serif\" font-size=\"14.00\">1</text>\n</g>\n<!-- const2 -->\n<g id=\"node6\" class=\"node\">\n<title>const2</title>\n<ellipse fill=\"none\" stroke=\"black\" cx=\"146.45\" cy=\"-184.92\" rx=\"27\" ry=\"18\"/>\n<text text-anchor=\"middle\" x=\"146.45\" y=\"-181.22\" font-family=\"Times,serif\" font-size=\"14.00\">1</text>\n</g>\n<!-- const2&#45;&gt;d -->\n<!-- const2&#45;&gt;g -->\n<g id=\"edge9\" class=\"edge\">\n<title>const2&#45;&gt;g</title>\n<path fill=\"none\" stroke=\"black\" d=\"M167.05,-173.19C186.37,-161.64 216.13,-143.83 238.3,-130.57\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"240.2,-133.51 246.98,-125.37 236.61,-127.5 240.2,-133.51\"/>\n<text text-anchor=\"middle\" x=\"209.89\" y=\"-156.72\" font-family=\"Times,serif\" font-size=\"14.00\">&#45;1.5</text>\n</g>\n</g>\n</svg>\n",
            "text/plain": [
              "<graphviz.sources.Source at 0x7f332bdcd4b0>"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df.loc[:4, ['A','B','A OR B','~(A AND B)','A XOR B']]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 175
        },
        "id": "TgHTGufQITYf",
        "outputId": "89711b28-e225-4f4b-8c45-f105776ce9e8"
      },
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   A  B  A OR B  ~(A AND B)  A XOR B\n",
              "1  1  1       1           0        0\n",
              "2  0  1       1           1        1\n",
              "3  1  0       1           1        1\n",
              "4  0  0       0           1        0"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-2cc85a66-c461-4a99-9559-fc7fd32da2a6\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>A</th>\n",
              "      <th>B</th>\n",
              "      <th>A OR B</th>\n",
              "      <th>~(A AND B)</th>\n",
              "      <th>A XOR B</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-2cc85a66-c461-4a99-9559-fc7fd32da2a6')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-2cc85a66-c461-4a99-9559-fc7fd32da2a6 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-2cc85a66-c461-4a99-9559-fc7fd32da2a6');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-a0eb4f1f-a237-4ac8-b17e-c21dc65f77f5\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-a0eb4f1f-a237-4ac8-b17e-c21dc65f77f5')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-a0eb4f1f-a237-4ac8-b17e-c21dc65f77f5 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 4,\n  \"fields\": [\n    {\n      \"column\": \"A\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"num_unique_values\": 2,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"B\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"num_unique_values\": 2,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"A OR B\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"num_unique_values\": 2,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"~(A AND B)\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"samples\": [\n          1,\n          0\n        ],\n        \"num_unique_values\": 2,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"A XOR B\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"samples\": [\n          1,\n          0\n        ],\n        \"num_unique_values\": 2,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "To solve the 3-input XOR problem ((A XOR B) XOR C), we can embed the A XOR B architecture inside a network that takes its output and XORs it with C."
      ],
      "metadata": {
        "id": "IywESggA6vZN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "graphviz.Source.from_file('graphs/saved_graphs/network2.gv')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 322
        },
        "id": "HfzOS0wgpVT0",
        "outputId": "e74dd9d5-14bc-44ea-a4fc-f11954048588"
      },
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "image/svg+xml": "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Generated by graphviz version 2.43.0 (0)\n -->\n<!-- Title: graph Pages: 1 -->\n<svg width=\"551pt\" height=\"226pt\"\n viewBox=\"0.00 0.00 551.38 225.92\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 221.92)\">\n<title>graph</title>\n<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-221.92 547.38,-221.92 547.38,4 -4,4\"/>\n<!-- a -->\n<g id=\"node1\" class=\"node\">\n<title>a</title>\n<ellipse fill=\"none\" stroke=\"black\" cx=\"27\" cy=\"-97.92\" rx=\"27\" ry=\"18\"/>\n<text text-anchor=\"middle\" x=\"27\" y=\"-94.22\" font-family=\"Times,serif\" font-size=\"14.00\">A</text>\n</g>\n<!-- b -->\n<g id=\"node2\" class=\"node\">\n<title>b</title>\n<ellipse fill=\"none\" stroke=\"black\" cx=\"27\" cy=\"-25.92\" rx=\"27\" ry=\"18\"/>\n<text text-anchor=\"middle\" x=\"27\" y=\"-22.22\" font-family=\"Times,serif\" font-size=\"14.00\">B</text>\n</g>\n<!-- a&#45;&gt;b -->\n<!-- d -->\n<g id=\"node4\" class=\"node\">\n<title>d</title>\n<ellipse fill=\"none\" stroke=\"black\" cx=\"146.45\" cy=\"-112.92\" rx=\"27\" ry=\"18\"/>\n<text text-anchor=\"middle\" x=\"146.45\" y=\"-109.22\" font-family=\"Times,serif\" font-size=\"14.00\">OR</text>\n</g>\n<!-- a&#45;&gt;d -->\n<g id=\"edge4\" class=\"edge\">\n<title>a&#45;&gt;d</title>\n<path fill=\"none\" stroke=\"black\" d=\"M53.77,-101.21C70.14,-103.3 91.56,-106.04 109.58,-108.34\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"109.37,-111.84 119.73,-109.64 110.26,-104.9 109.37,-111.84\"/>\n<text text-anchor=\"middle\" x=\"83\" y=\"-110.72\" font-family=\"Times,serif\" font-size=\"14.00\">1</text>\n</g>\n<!-- e -->\n<g id=\"node5\" class=\"node\">\n<title>e</title>\n<ellipse fill=\"none\" stroke=\"black\" cx=\"146.45\" cy=\"-40.92\" rx=\"34.39\" ry=\"18\"/>\n<text text-anchor=\"middle\" x=\"146.45\" y=\"-37.22\" font-family=\"Times,serif\" font-size=\"14.00\">~AND</text>\n</g>\n<!-- a&#45;&gt;e -->\n<g id=\"edge5\" class=\"edge\">\n<title>a&#45;&gt;e</title>\n<path fill=\"none\" stroke=\"black\" d=\"M35.65,-80.66C42.83,-66.82 55.02,-48.35 72,-39.92 81.78,-35.07 93.24,-33.57 104.15,-33.73\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"104.19,-37.24 114.38,-34.34 104.6,-30.25 104.19,-37.24\"/>\n<text text-anchor=\"middle\" x=\"83\" y=\"-43.72\" font-family=\"Times,serif\" font-size=\"14.00\">&#45;1</text>\n</g>\n<!-- b&#45;&gt;d -->\n<g id=\"edge7\" class=\"edge\">\n<title>b&#45;&gt;d</title>\n<path fill=\"none\" stroke=\"black\" d=\"M45.84,-39.14C65.53,-53.72 97.39,-77.32 119.76,-93.89\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"117.71,-96.73 127.83,-99.87 121.87,-91.1 117.71,-96.73\"/>\n<text text-anchor=\"middle\" x=\"83\" y=\"-78.72\" font-family=\"Times,serif\" font-size=\"14.00\">1</text>\n</g>\n<!-- b&#45;&gt;e -->\n<g id=\"edge8\" class=\"edge\">\n<title>b&#45;&gt;e</title>\n<path fill=\"none\" stroke=\"black\" d=\"M45.57,-12.77C58.82,-4.51 77.47,3.63 94,-1.92 104.2,-5.35 113.98,-11.69 122.23,-18.28\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"120.05,-21.02 129.92,-24.85 124.59,-15.69 120.05,-21.02\"/>\n<text text-anchor=\"middle\" x=\"83\" y=\"-5.72\" font-family=\"Times,serif\" font-size=\"14.00\">&#45;1</text>\n</g>\n<!-- const1 -->\n<g id=\"node3\" class=\"node\">\n<title>const1</title>\n<ellipse fill=\"none\" stroke=\"black\" cx=\"27\" cy=\"-169.92\" rx=\"27\" ry=\"18\"/>\n<text text-anchor=\"middle\" x=\"27\" y=\"-166.22\" font-family=\"Times,serif\" font-size=\"14.00\">1</text>\n</g>\n<!-- const1&#45;&gt;a -->\n<!-- const1&#45;&gt;d -->\n<g id=\"edge1\" class=\"edge\">\n<title>const1&#45;&gt;d</title>\n<path fill=\"none\" stroke=\"black\" d=\"M53.47,-174.14C66.17,-175.12 81.45,-174.55 94,-168.92 108.54,-162.4 120.81,-149.58 129.71,-137.85\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"132.63,-139.78 135.58,-129.61 126.93,-135.72 132.63,-139.78\"/>\n<text text-anchor=\"middle\" x=\"83\" y=\"-177.72\" font-family=\"Times,serif\" font-size=\"14.00\">&#45;0.5</text>\n</g>\n<!-- const1&#45;&gt;e -->\n<g id=\"edge2\" class=\"edge\">\n<title>const1&#45;&gt;e</title>\n<path fill=\"none\" stroke=\"black\" d=\"M49.31,-159.74C63.5,-152.08 81.72,-140.42 94,-125.92 106.6,-111.05 102.13,-102.73 112,-85.92 116,-79.11 120.82,-72.1 125.55,-65.7\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"128.57,-67.5 131.84,-57.42 123,-63.26 128.57,-67.5\"/>\n<text text-anchor=\"middle\" x=\"83\" y=\"-147.72\" font-family=\"Times,serif\" font-size=\"14.00\">1.5</text>\n</g>\n<!-- d&#45;&gt;e -->\n<!-- g -->\n<g id=\"node7\" class=\"node\">\n<title>g</title>\n<ellipse fill=\"none\" stroke=\"black\" cx=\"268.79\" cy=\"-112.92\" rx=\"29.8\" ry=\"18\"/>\n<text text-anchor=\"middle\" x=\"268.79\" y=\"-109.22\" font-family=\"Times,serif\" font-size=\"14.00\">AND</text>\n</g>\n<!-- d&#45;&gt;g -->\n<g id=\"edge11\" class=\"edge\">\n<title>d&#45;&gt;g</title>\n<path fill=\"none\" stroke=\"black\" d=\"M173.56,-112.92C189.62,-112.92 210.52,-112.92 228.55,-112.92\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"228.78,-116.42 238.78,-112.92 228.78,-109.42 228.78,-116.42\"/>\n<text text-anchor=\"middle\" x=\"209.89\" y=\"-116.72\" font-family=\"Times,serif\" font-size=\"14.00\">1</text>\n</g>\n<!-- e&#45;&gt;g -->\n<g id=\"edge13\" class=\"edge\">\n<title>e&#45;&gt;g</title>\n<path fill=\"none\" stroke=\"black\" d=\"M169.8,-54.3C189.07,-65.83 217.04,-82.56 238.17,-95.2\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"236.39,-98.21 246.77,-100.34 239.98,-92.21 236.39,-98.21\"/>\n<text text-anchor=\"middle\" x=\"209.89\" y=\"-87.72\" font-family=\"Times,serif\" font-size=\"14.00\">1</text>\n</g>\n<!-- const2 -->\n<g id=\"node6\" class=\"node\">\n<title>const2</title>\n<ellipse fill=\"none\" stroke=\"black\" cx=\"146.45\" cy=\"-184.92\" rx=\"27\" ry=\"18\"/>\n<text text-anchor=\"middle\" x=\"146.45\" y=\"-181.22\" font-family=\"Times,serif\" font-size=\"14.00\">1</text>\n</g>\n<!-- const2&#45;&gt;d -->\n<!-- const2&#45;&gt;g -->\n<g id=\"edge9\" class=\"edge\">\n<title>const2&#45;&gt;g</title>\n<path fill=\"none\" stroke=\"black\" d=\"M167.05,-173.19C186.37,-161.64 216.13,-143.83 238.3,-130.57\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"240.2,-133.51 246.98,-125.37 236.61,-127.5 240.2,-133.51\"/>\n<text text-anchor=\"middle\" x=\"209.89\" y=\"-156.72\" font-family=\"Times,serif\" font-size=\"14.00\">&#45;1.5</text>\n</g>\n<!-- h -->\n<g id=\"node8\" class=\"node\">\n<title>h</title>\n<ellipse fill=\"none\" stroke=\"black\" cx=\"268.79\" cy=\"-40.92\" rx=\"27\" ry=\"18\"/>\n<text text-anchor=\"middle\" x=\"268.79\" y=\"-37.22\" font-family=\"Times,serif\" font-size=\"14.00\">C</text>\n</g>\n<!-- g&#45;&gt;h -->\n<!-- i -->\n<g id=\"node10\" class=\"node\">\n<title>i</title>\n<ellipse fill=\"none\" stroke=\"black\" cx=\"391.14\" cy=\"-127.92\" rx=\"27\" ry=\"18\"/>\n<text text-anchor=\"middle\" x=\"391.14\" y=\"-124.22\" font-family=\"Times,serif\" font-size=\"14.00\">OR</text>\n</g>\n<!-- g&#45;&gt;i -->\n<g id=\"edge17\" class=\"edge\">\n<title>g&#45;&gt;i</title>\n<path fill=\"none\" stroke=\"black\" d=\"M298.31,-116.48C315.04,-118.56 336.34,-121.22 354.22,-123.44\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"353.93,-126.94 364.28,-124.7 354.79,-119.99 353.93,-126.94\"/>\n<text text-anchor=\"middle\" x=\"327.69\" y=\"-125.72\" font-family=\"Times,serif\" font-size=\"14.00\">1</text>\n</g>\n<!-- j -->\n<g id=\"node11\" class=\"node\">\n<title>j</title>\n<ellipse fill=\"none\" stroke=\"black\" cx=\"391.14\" cy=\"-55.92\" rx=\"34.39\" ry=\"18\"/>\n<text text-anchor=\"middle\" x=\"391.14\" y=\"-52.22\" font-family=\"Times,serif\" font-size=\"14.00\">~AND</text>\n</g>\n<!-- g&#45;&gt;j -->\n<g id=\"edge18\" class=\"edge\">\n<title>g&#45;&gt;j</title>\n<path fill=\"none\" stroke=\"black\" d=\"M278.24,-95.63C286.07,-81.76 299.16,-63.29 316.69,-54.92 326.46,-50.26 337.82,-48.8 348.65,-48.93\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"348.62,-52.43 358.8,-49.49 349,-45.44 348.62,-52.43\"/>\n<text text-anchor=\"middle\" x=\"327.69\" y=\"-58.72\" font-family=\"Times,serif\" font-size=\"14.00\">&#45;1</text>\n</g>\n<!-- h&#45;&gt;i -->\n<g id=\"edge20\" class=\"edge\">\n<title>h&#45;&gt;i</title>\n<path fill=\"none\" stroke=\"black\" d=\"M287.81,-53.95C307.93,-68.49 340.68,-92.17 363.7,-108.81\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"361.84,-111.79 372,-114.81 365.94,-106.11 361.84,-111.79\"/>\n<text text-anchor=\"middle\" x=\"327.69\" y=\"-93.72\" font-family=\"Times,serif\" font-size=\"14.00\">1</text>\n</g>\n<!-- h&#45;&gt;j -->\n<g id=\"edge21\" class=\"edge\">\n<title>h&#45;&gt;j</title>\n<path fill=\"none\" stroke=\"black\" d=\"M287.6,-28C301.49,-19.55 321.28,-11.07 338.69,-16.92 348.89,-20.35 358.67,-26.69 366.92,-33.28\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"364.73,-36.02 374.61,-39.85 369.28,-30.69 364.73,-36.02\"/>\n<text text-anchor=\"middle\" x=\"327.69\" y=\"-20.72\" font-family=\"Times,serif\" font-size=\"14.00\">&#45;1</text>\n</g>\n<!-- const3 -->\n<g id=\"node9\" class=\"node\">\n<title>const3</title>\n<ellipse fill=\"none\" stroke=\"black\" cx=\"268.79\" cy=\"-184.92\" rx=\"27\" ry=\"18\"/>\n<text text-anchor=\"middle\" x=\"268.79\" y=\"-181.22\" font-family=\"Times,serif\" font-size=\"14.00\">1</text>\n</g>\n<!-- const3&#45;&gt;g -->\n<!-- const3&#45;&gt;i -->\n<g id=\"edge14\" class=\"edge\">\n<title>const3&#45;&gt;i</title>\n<path fill=\"none\" stroke=\"black\" d=\"M295.06,-189.24C308.56,-190.41 325.15,-189.99 338.69,-183.92 353.23,-177.41 365.5,-164.59 374.4,-152.86\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"377.32,-154.79 380.27,-144.61 371.62,-150.72 377.32,-154.79\"/>\n<text text-anchor=\"middle\" x=\"327.69\" y=\"-192.72\" font-family=\"Times,serif\" font-size=\"14.00\">&#45;0.5</text>\n</g>\n<!-- const3&#45;&gt;j -->\n<g id=\"edge15\" class=\"edge\">\n<title>const3&#45;&gt;j</title>\n<path fill=\"none\" stroke=\"black\" d=\"M291.73,-175.1C306.59,-167.54 325.79,-155.85 338.69,-140.92 351.44,-126.17 346.82,-117.73 356.69,-100.92 360.69,-94.11 365.51,-87.1 370.23,-80.7\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"373.26,-82.5 376.53,-72.42 367.69,-78.26 373.26,-82.5\"/>\n<text text-anchor=\"middle\" x=\"327.69\" y=\"-163.72\" font-family=\"Times,serif\" font-size=\"14.00\">1.5</text>\n</g>\n<!-- i&#45;&gt;j -->\n<!-- k -->\n<g id=\"node13\" class=\"node\">\n<title>k</title>\n<ellipse fill=\"none\" stroke=\"black\" cx=\"513.48\" cy=\"-127.92\" rx=\"29.8\" ry=\"18\"/>\n<text text-anchor=\"middle\" x=\"513.48\" y=\"-124.22\" font-family=\"Times,serif\" font-size=\"14.00\">AND</text>\n</g>\n<!-- i&#45;&gt;k -->\n<g id=\"edge24\" class=\"edge\">\n<title>i&#45;&gt;k</title>\n<path fill=\"none\" stroke=\"black\" d=\"M418.25,-127.92C434.31,-127.92 455.21,-127.92 473.24,-127.92\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"473.47,-131.42 483.47,-127.92 473.47,-124.42 473.47,-131.42\"/>\n<text text-anchor=\"middle\" x=\"454.58\" y=\"-131.72\" font-family=\"Times,serif\" font-size=\"14.00\">1</text>\n</g>\n<!-- j&#45;&gt;k -->\n<g id=\"edge26\" class=\"edge\">\n<title>j&#45;&gt;k</title>\n<path fill=\"none\" stroke=\"black\" d=\"M414.49,-69.3C433.76,-80.83 461.73,-97.56 482.86,-110.2\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"481.08,-113.21 491.46,-115.34 484.67,-107.21 481.08,-113.21\"/>\n<text text-anchor=\"middle\" x=\"454.58\" y=\"-102.72\" font-family=\"Times,serif\" font-size=\"14.00\">1</text>\n</g>\n<!-- const4 -->\n<g id=\"node12\" class=\"node\">\n<title>const4</title>\n<ellipse fill=\"none\" stroke=\"black\" cx=\"391.14\" cy=\"-199.92\" rx=\"27\" ry=\"18\"/>\n<text text-anchor=\"middle\" x=\"391.14\" y=\"-196.22\" font-family=\"Times,serif\" font-size=\"14.00\">1</text>\n</g>\n<!-- const4&#45;&gt;i -->\n<!-- const4&#45;&gt;k -->\n<g id=\"edge22\" class=\"edge\">\n<title>const4&#45;&gt;k</title>\n<path fill=\"none\" stroke=\"black\" d=\"M411.74,-188.19C431.06,-176.64 460.82,-158.83 482.99,-145.57\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"484.89,-148.51 491.67,-140.37 481.29,-142.5 484.89,-148.51\"/>\n<text text-anchor=\"middle\" x=\"454.58\" y=\"-171.72\" font-family=\"Times,serif\" font-size=\"14.00\">&#45;1.5</text>\n</g>\n</g>\n</svg>\n",
            "text/plain": [
              "<graphviz.sources.Source at 0x7f3328d1ad10>"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 300
        },
        "id": "-lrKCzJlJ7GX",
        "outputId": "97fd15ad-4d4c-47c8-b941-62f5d7b720ea"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   A  B  C  A OR B  ~(A AND B)  A XOR B  (A XOR B) XOR C\n",
              "1  1  1  1       1           0        0                1\n",
              "2  0  1  1       1           1        1                0\n",
              "3  1  0  1       1           1        1                0\n",
              "4  0  0  1       0           1        0                1\n",
              "5  1  1  0       1           0        0                0\n",
              "6  0  1  0       1           1        1                1\n",
              "7  1  0  0       1           1        1                1\n",
              "8  0  0  0       0           1        0                0"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-500168a3-8682-418a-b7d9-ce2de07eed25\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>A</th>\n",
              "      <th>B</th>\n",
              "      <th>C</th>\n",
              "      <th>A OR B</th>\n",
              "      <th>~(A AND B)</th>\n",
              "      <th>A XOR B</th>\n",
              "      <th>(A XOR B) XOR C</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-500168a3-8682-418a-b7d9-ce2de07eed25')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-500168a3-8682-418a-b7d9-ce2de07eed25 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-500168a3-8682-418a-b7d9-ce2de07eed25');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-2483f873-9f83-46ce-aa4d-799c59d5462d\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-2483f873-9f83-46ce-aa4d-799c59d5462d')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-2483f873-9f83-46ce-aa4d-799c59d5462d button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "  <div id=\"id_1fae4544-a354-4dee-8997-7e6abdb7f17f\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('df')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_1fae4544-a354-4dee-8997-7e6abdb7f17f button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('df');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 8,\n  \"fields\": [\n    {\n      \"column\": \"A\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"num_unique_values\": 2,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"B\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"num_unique_values\": 2,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"C\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"num_unique_values\": 2,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"A OR B\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"num_unique_values\": 2,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"~(A AND B)\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"samples\": [\n          1,\n          0\n        ],\n        \"num_unique_values\": 2,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"A XOR B\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"samples\": [\n          1,\n          0\n        ],\n        \"num_unique_values\": 2,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"(A XOR B) XOR C\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"num_unique_values\": 2,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "If my implementation of the network actually used the flat stepwise activation function describe above, the gradients would all be 0. To closely simulate this function while maintaining meaningful gradients, I use a compressed sigmoid as the activation function at each layer."
      ],
      "metadata": {
        "id": "0rtRuhjp7zJ7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def layer_activation(x):\n",
        "    return tf.nn.sigmoid(10*x)"
      ],
      "metadata": {
        "id": "2CyYC42QGB1E"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "My implementation feeds all three features to the input layer, keeping the \"C\" values isolated until they're fed into Hidden Layer 3. These are the network weights we'll need:\n"
      ],
      "metadata": {
        "id": "urB-No24GXhv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "weights1 = tf.Variable(np.array([[1,-1,0],[1,-1,0],[0,0,2]]), dtype=tf.float64)\n",
        "bias1 = tf.Variable(np.array([-0.5,1.5,-1]), dtype=tf.float64)\n",
        "\n",
        "weights2 = tf.Variable(np.array([[1,0],[1,0],[0,2]]), dtype=tf.float64)\n",
        "bias2 = tf.Variable(np.array([-1.5,-1]), dtype=tf.float64)\n",
        "\n",
        "weights3 = tf.Variable(np.array([[1,-1],[1,-1]]), dtype=tf.float64)\n",
        "bias3 = tf.Variable(np.array([-0.5,1.5]), dtype=tf.float64)\n",
        "\n",
        "weights4 = tf.Variable(np.array([[1],[1]]), dtype=tf.float64)\n",
        "bias4 = tf.Variable(np.array([-1.5]), dtype=tf.float64)"
      ],
      "metadata": {
        "id": "h1KLv4FsFNkM"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "I define the model and confirm that it works."
      ],
      "metadata": {
        "id": "dFlIemJr83bm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model = tf.keras.Sequential(\n",
        "    [\n",
        "        tf.keras.layers.Dense(3, activation=layer_activation, weights=[weights1,bias1], name='hidden1'),\n",
        "        tf.keras.layers.Dense(2, activation=layer_activation, weights=[weights2,bias2], name='hidden2'),\n",
        "        tf.keras.layers.Dense(2, activation=layer_activation, weights=[weights3,bias3], name='hidden3'),\n",
        "        tf.keras.layers.Dense(1, activation=layer_activation, weights=[weights4,bias4], name='output'),\n",
        "    ],\n",
        "    name='my_sequential'\n",
        "    )"
      ],
      "metadata": {
        "id": "VVSA0hp0GB28"
      },
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model.compile(optimizer=tf.keras.optimizers.Adam(),\n",
        "              loss=tf.keras.losses.BinaryCrossentropy(from_logits=False),\n",
        "              metrics=['accuracy'])"
      ],
      "metadata": {
        "id": "zP06yUwMGmlz"
      },
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "model(X)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "F-RnTdltGB5c",
        "outputId": "38fee041-3f0c-420b-c281-2ffdf7573bf1"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(8, 1), dtype=float32, numpy=\n",
              "array([[0.9923533 ],\n",
              "       [0.0071906 ],\n",
              "       [0.0071906 ],\n",
              "       [0.9923533 ],\n",
              "       [0.00718809],\n",
              "       [0.992353  ],\n",
              "       [0.992353  ],\n",
              "       [0.00718809]], dtype=float32)>"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model.evaluate(X, y)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mG9VgCqbGB77",
        "outputId": "8ae075f0-a251-4416-a004-53426ecb4c49"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1/1 [==============================] - 0s 336ms/step - loss: 0.0074 - accuracy: 1.0000\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[0.007445765659213066, 1.0]"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The network correctly classifies each example with low cross-entropy loss."
      ],
      "metadata": {
        "id": "Ay9yhyJ9G5LL"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Testing Concepts\n",
        "\n"
      ],
      "metadata": {
        "id": "q1Q21dpY9_xE"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### A OR B"
      ],
      "metadata": {
        "id": "MZpzLq2aLnoO"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "An advantage of using a simple, hard-coded model is that we know exactly where certain concepts are represented in the model and how the model uses those concepts in its reasoning. I'll first look at the case of A OR B."
      ],
      "metadata": {
        "id": "KA3YHtBGY0qR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "concept = 'A OR B'\n",
        "concept_labels = df[concept].values"
      ],
      "metadata": {
        "id": "2ZWChpOaGCCk"
      },
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The model represents A OR B in the first neuron of Hidden Layer 1, as we can see below. It then applies an AND operation to conjoin the concept with ~(A AND B)."
      ],
      "metadata": {
        "id": "Hq2_cVt4aLML"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tcav = TCAV(model, layer_idx=0)\n",
        "tcav.train_cav(X, concept_labels)\n",
        "print('Hidden Layer 1 Activations:')\n",
        "print(tcav.layer_activations)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yy2vPAWTQo1M",
        "outputId": "b5fb55ee-bdc0-43d7-9780-c3db82a884ff"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Hidden Layer 1 Activations:\n",
            "tf.Tensor(\n",
            "[[9.999997e-01 6.692851e-03 9.999546e-01]\n",
            " [9.933072e-01 9.933072e-01 9.999546e-01]\n",
            " [9.933072e-01 9.933072e-01 9.999546e-01]\n",
            " [6.692851e-03 9.999997e-01 9.999546e-01]\n",
            " [9.999997e-01 6.692851e-03 4.539787e-05]\n",
            " [9.933072e-01 9.933072e-01 4.539787e-05]\n",
            " [9.933072e-01 9.933072e-01 4.539787e-05]\n",
            " [6.692851e-03 9.999997e-01 4.539787e-05]], shape=(8, 3), dtype=float32)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "A linear classifier fit to these activations is of course able to perfectly distinguish A OR B examples from ~(A OR B) examples."
      ],
      "metadata": {
        "id": "v-fIwuN8aXjR"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "acc = tcav.get_layer_prediction_accuracy(X, concept_labels)\n",
        "print('Prediction Accuracy:', acc)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ex8K3XhKQ4EU",
        "outputId": "d77c1d51-a5f1-4441-d44c-fcea77b9c7d6"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Prediction Accuracy: 1.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The CAV is defined as the coefficients of the linear classifier. It is roughly in the direction of A OR B in the activation space, though it gets a bit distracted by the second dimension and the imperfect activation function."
      ],
      "metadata": {
        "id": "6wjD0Hl4cCvg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print('CAV:')\n",
        "print(tcav.cav)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6Jx8XJ0eQ3aM",
        "outputId": "0618bd5d-a284-4bfd-a105-bd35944cdd99"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CAV:\n",
            "[[ 0.91887672]\n",
            " [-0.39454476]\n",
            " [ 0.        ]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We know that the A OR B concept is important at this layer for the model's predictions. We'll check whether this is reflected in the TCAV score and sensitivities for the $y = 1$ class. Note that this gives the sensitivities for rows 1, 4, 6, and 7."
      ],
      "metadata": {
        "id": "vPIDeNCYgjb3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df.loc[[1,4,6,7]]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 175
        },
        "id": "nuONUPXeKPDP",
        "outputId": "95d1004e-8dcc-4ba9-f00d-46e3a6a72fd8"
      },
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   A  B  C  A OR B  ~(A AND B)  A XOR B  (A XOR B) XOR C\n",
              "1  1  1  1       1           0        0                1\n",
              "4  0  0  1       0           1        0                1\n",
              "6  0  1  0       1           1        1                1\n",
              "7  1  0  0       1           1        1                1"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-08b63100-7faf-4eb6-abe6-b41f2a754761\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>A</th>\n",
              "      <th>B</th>\n",
              "      <th>C</th>\n",
              "      <th>A OR B</th>\n",
              "      <th>~(A AND B)</th>\n",
              "      <th>A XOR B</th>\n",
              "      <th>(A XOR B) XOR C</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-08b63100-7faf-4eb6-abe6-b41f2a754761')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-08b63100-7faf-4eb6-abe6-b41f2a754761 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-08b63100-7faf-4eb6-abe6-b41f2a754761');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-4aa84626-9a83-4d7d-a168-331afd499d6a\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-4aa84626-9a83-4d7d-a168-331afd499d6a')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-4aa84626-9a83-4d7d-a168-331afd499d6a button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 4,\n  \"fields\": [\n    {\n      \"column\": \"A\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"num_unique_values\": 2,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"B\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"num_unique_values\": 2,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"C\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"num_unique_values\": 2,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"A OR B\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"num_unique_values\": 2,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"~(A AND B)\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"samples\": [\n          1,\n          0\n        ],\n        \"num_unique_values\": 2,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"A XOR B\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"samples\": [\n          1,\n          0\n        ],\n        \"num_unique_values\": 2,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"(A XOR B) XOR C\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 1,\n        \"max\": 1,\n        \"samples\": [\n          1\n        ],\n        \"num_unique_values\": 1,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tcav_score = tcav.tcav_score(X, concept_labels, y, 1)\n",
        "print('TCAV Score:', tcav_score)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4IPINfBnQ4B0",
        "outputId": "4d85e8a7-6061-477b-9591-de3ae740c774"
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "TCAV Score: 0.5\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print('Sensitivity:')\n",
        "tcav.sensitivity"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IZDN--pfQ3-7",
        "outputId": "c809de99-1012-4e7b-9cb1-b1d0fcbbadb1"
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Sensitivity:\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[-2.63639714e-05],\n",
              "       [-2.63639714e-05],\n",
              "       [ 3.01125423e-05],\n",
              "       [ 3.01125423e-05]])"
            ]
          },
          "metadata": {},
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The sensitivity values are negative for rows 1 and 4 but positive for rows 6 and 7, so the TCAV score is 0.5. TCAV fails to flag A OR B as an important concept at this layer even though it clearly is; why is this the case?\n",
        "\n",
        "For the first and second $y=1$ examples (rows 1 and 4 of the entire dataset), moving in the A OR B direction in Hidden Layer 1 means moving in the A XOR B direction in Hidden Layer 2. Since C = 1 in those rows, a move in the A XOR B direction is a move *away from* (A XOR B) XOR C, so the sensitivity values are negative."
      ],
      "metadata": {
        "id": "7ik5ho-xXafE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df.loc[[1,4]]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 112
        },
        "id": "Z89k1LAvKsSD",
        "outputId": "b9a5adc0-6e73-47ed-ba44-9d3886a0a74e"
      },
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   A  B  C  A OR B  ~(A AND B)  A XOR B  (A XOR B) XOR C\n",
              "1  1  1  1       1           0        0                1\n",
              "4  0  0  1       0           1        0                1"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-f70a4543-d0d6-4b93-930e-79de445f1d9a\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>A</th>\n",
              "      <th>B</th>\n",
              "      <th>C</th>\n",
              "      <th>A OR B</th>\n",
              "      <th>~(A AND B)</th>\n",
              "      <th>A XOR B</th>\n",
              "      <th>(A XOR B) XOR C</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-f70a4543-d0d6-4b93-930e-79de445f1d9a')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-f70a4543-d0d6-4b93-930e-79de445f1d9a button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-f70a4543-d0d6-4b93-930e-79de445f1d9a');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-3b953626-5f00-4f2c-adcc-c7f8f2debc55\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-3b953626-5f00-4f2c-adcc-c7f8f2debc55')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-3b953626-5f00-4f2c-adcc-c7f8f2debc55 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 2,\n  \"fields\": [\n    {\n      \"column\": \"A\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"num_unique_values\": 2,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"B\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"num_unique_values\": 2,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"C\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 1,\n        \"max\": 1,\n        \"samples\": [\n          1\n        ],\n        \"num_unique_values\": 1,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"A OR B\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"samples\": [\n          0\n        ],\n        \"num_unique_values\": 2,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"~(A AND B)\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"samples\": [\n          1\n        ],\n        \"num_unique_values\": 2,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"A XOR B\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 0,\n        \"samples\": [\n          0\n        ],\n        \"num_unique_values\": 1,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"(A XOR B) XOR C\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 1,\n        \"max\": 1,\n        \"samples\": [\n          1\n        ],\n        \"num_unique_values\": 1,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "For the third and fourth $y=1$ examples (rows 6 and 7 of the entire dataset), moving in the A OR B direction in Hidden Layer 1 again means moving in the A XOR B direction in Hidden Layer 2. However, since C = 0 in those rows, a move in the A XOR B direction is a move *towards* the (A XOR B) XOR C direction, so the sensitivity values are positive."
      ],
      "metadata": {
        "id": "vxW-jBONKrBh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df.loc[[6,7]]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 112
        },
        "id": "bmoj3iCTKzzS",
        "outputId": "284065ab-9c51-4c24-9d39-a23dabd9ffa2"
      },
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   A  B  C  A OR B  ~(A AND B)  A XOR B  (A XOR B) XOR C\n",
              "6  0  1  0       1           1        1                1\n",
              "7  1  0  0       1           1        1                1"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-1ea42ef8-2f16-407b-8615-e148b3849bc0\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>A</th>\n",
              "      <th>B</th>\n",
              "      <th>C</th>\n",
              "      <th>A OR B</th>\n",
              "      <th>~(A AND B)</th>\n",
              "      <th>A XOR B</th>\n",
              "      <th>(A XOR B) XOR C</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-1ea42ef8-2f16-407b-8615-e148b3849bc0')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-1ea42ef8-2f16-407b-8615-e148b3849bc0 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-1ea42ef8-2f16-407b-8615-e148b3849bc0');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-0d97feb9-d4d2-47f1-8722-dd9fc989003b\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-0d97feb9-d4d2-47f1-8722-dd9fc989003b')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-0d97feb9-d4d2-47f1-8722-dd9fc989003b button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 2,\n  \"fields\": [\n    {\n      \"column\": \"A\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"samples\": [\n          1,\n          0\n        ],\n        \"num_unique_values\": 2,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"B\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"num_unique_values\": 2,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"C\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 0,\n        \"samples\": [\n          0\n        ],\n        \"num_unique_values\": 1,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"A OR B\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 1,\n        \"max\": 1,\n        \"samples\": [\n          1\n        ],\n        \"num_unique_values\": 1,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"~(A AND B)\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 1,\n        \"max\": 1,\n        \"samples\": [\n          1\n        ],\n        \"num_unique_values\": 1,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"A XOR B\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 1,\n        \"max\": 1,\n        \"samples\": [\n          1\n        ],\n        \"num_unique_values\": 1,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"(A XOR B) XOR C\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 1,\n        \"max\": 1,\n        \"samples\": [\n          1\n        ],\n        \"num_unique_values\": 1,\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### A XOR B"
      ],
      "metadata": {
        "id": "yj4OjuF0Lssm"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We'll look at another example before moving on. A XOR B is an important concept in Hidden Layer 2. It's represented in the layer's first neuron, and the next two layers XOR it with C to give the final output."
      ],
      "metadata": {
        "id": "v94nns8HwYZ5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "concept = 'A XOR B'\n",
        "concept_labels = df[concept].values"
      ],
      "metadata": {
        "id": "YjnE00tDSGks"
      },
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tcav = TCAV(model, layer_idx=1)\n",
        "tcav.train_cav(X, concept_labels)\n",
        "print('Hidden Layer 2 Activations:')\n",
        "print(tcav.layer_activations)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8c1e5590-59f4-486b-a3ea-4e15bcf4a0fb",
        "id": "1ksyMSlbwt-P"
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Hidden Layer 2 Activations:\n",
            "tf.Tensor(\n",
            "[[7.1527860e-03 9.9995458e-01]\n",
            " [9.9235588e-01 9.9995458e-01]\n",
            " [9.9235588e-01 9.9995458e-01]\n",
            " [7.1527860e-03 9.9995458e-01]\n",
            " [7.1527860e-03 4.5439105e-05]\n",
            " [9.9235588e-01 4.5439105e-05]\n",
            " [9.9235588e-01 4.5439105e-05]\n",
            " [7.1527860e-03 4.5439105e-05]], shape=(8, 2), dtype=float32)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Again, a linear classifier fit to these activations of course has perfect accuracy."
      ],
      "metadata": {
        "id": "ydEe6FWQwt-S"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "acc = tcav.get_layer_prediction_accuracy(X, concept_labels)\n",
        "print('Prediction Accuracy:', acc)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "48a1fe01-30e7-4a59-c9cc-2bdabab13757",
        "id": "0TsTqy5Dwt-S"
      },
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Prediction Accuracy: 1.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The CAV is in the direction of A XOR B in the activation space."
      ],
      "metadata": {
        "id": "XTZ2P0NXwt-T"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print('CAV:')\n",
        "print(tcav.cav)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4d054794-ad8d-4bd7-e280-ccf87ea6e714",
        "id": "RKwUhFw6wt-T"
      },
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CAV:\n",
            "[[1.]\n",
            " [0.]]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We know that the A XOR B concept is important at Hidden Layer 2 for the model's predictions. We'll again check whether this is reflected in the TCAV score and sensitivities for the $y = 1$ class."
      ],
      "metadata": {
        "id": "bF5pr69Dwt-U"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tcav_score = tcav.tcav_score(X, concept_labels, y, 1)\n",
        "print('TCAV Score:', tcav_score)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4a3a4acf-c7e2-4105-c7a6-becef558564a",
        "id": "hua0q-Zlwt-V"
      },
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "TCAV Score: 0.5\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "print('Sensitivity:')\n",
        "tcav.sensitivity"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "cb727cc3-8e33-4613-a9ec-ac44784b6285",
        "id": "SdiYYTj9wt-V"
      },
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Sensitivity:\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[-0.00070802],\n",
              "       [-0.00070802],\n",
              "       [ 0.00075709],\n",
              "       [ 0.00075709]])"
            ]
          },
          "metadata": {},
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We again find a TCAV score of 0.5 for this clearly important concept. For the row 1 and 4 examples, moving in the A XOR B direction means moving *away from* the (A XOR B) XOR C direction since C = 1. For the row 6 and 7 examples, moving in the A XOR B direction means moving *towards* the (A XOR B) XOR C direction since C = 0.  "
      ],
      "metadata": {
        "id": "Y5euyFUHx1-G"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Discussion\n",
        "\n",
        "According to Kim *et. al*, TCAV score \"uses directional derivatives to quantify the degree to which a user-defined concept is important to a classification result\" (*ibid.* p. 1). After working through these examples, I think this claim needs some qualification.\n",
        "\n",
        "TCAV score only measures a particular type of importance: the *directional consistency* of a concept's impact on class predictions at a given layer. An extreme TCAV score (towards either 0 or 1) arises when moving in the direction of a concept generally has the same type of effect on class predictions across the dataset. For instance, Kim *et. al* show that the concept RED gets a high TCAV score across most layers of a CNN trained to classify FIRE ENGINE images. This result is expected since (at least to my knowledge) RED has a pretty straightforward positive relationship with FIRE ENGINE.\n",
        "\n",
        "TCAV score struggles to capture the importance of concepts with a more complex impact on model predictions. In the above example, A OR B is clearly an important concept for the model's predictions of the $y=1$ class. One neuron recreates it (nearly) exactly, and for rows 4, 6, and 7, flipping its value in that layer flips the prediction. However, the concept's importance isn't captured by the TCAV score because its effect is inconsistent, varying due to interactions with other concepts. Moving in the A OR B direction has a positive effect on the model's prediction if C = 0 and a negative effect if C = 1."
      ],
      "metadata": {
        "id": "fH3JCoGAyxJ9"
      }
    }
  ]
}