notebooks/02 Clustering the Predictions of the Metr-LA Dataset.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<center>\n",
        "    <h1>Verbal Explanation of Spatial Temporal GNNs for Traffic Forecasting</h1>\n",
        "    <h2>Clustering the Predictions of the Metr-LA dataset</h2>\n",
        "</center>\n",
        "\n",
        "---\n",
        "\n",
        "In this notebook the predictions of the *STGNN* on the *Metr-LA* dataset are clustered in order to obtain events to explain such as congestions or free-flows.\n",
        "\n",
        "Firstly, a distance matrix is obtained for each predicted instance of the datasets in order to compute the spatio-temporal and speed distance among all nodes in the prediction. The distance matrix is obtained throught the method explained in *Revealing the day-to-day regularity of urban congestion patterns with\n",
        "3d speed maps* <a name=\"cite_paper\"></a>[<sup>[1]</sup>](#note_paper)\n",
        "\n",
        "Finally, the predictions are clustered through *DBSCAN*, while considering the distance matrix as a dissimilarity measure.\n",
        "\n",
        "For more detailed informations about the used functions, look into the corresponding docstrings inside the python files, inside the `src` folder.\n",
        "\n",
        "---\n",
        "<small>\n",
        "\n",
        "<a name=\"note_paper\"></a>[1] \n",
        "C. Lopez et al. “Revealing the day-to-day regularity of urban congestion patterns with\n",
        "3d speed maps”. In: *Scientific Reports, 7(1):14029*, September 2017. ISSN:\n",
        "2045-2322. DOI: 10.1038/s41598-017-14237-8. URL: https://doi.org/10.1038/s41598-017-14237-8.\n",
        "</small>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "vLMMte_HKFqA"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "import os\n",
        "\n",
        "# Set the main path in the root folder of the project.\n",
        "sys.path.append(os.path.join('..'))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "HnmplGx2KFqD"
      },
      "outputs": [],
      "source": [
        "# Settings for autoreloading.\n",
        "%load_ext autoreload\n",
        "%autoreload 2"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "BVThP1k1KFqD"
      },
      "outputs": [],
      "source": [
        "from src.utils.seed import set_random_seed\n",
        "\n",
        "# Set the random seed for deterministic operations.\n",
        "SEED = 42\n",
        "set_random_seed(SEED)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "E9lFKnFUKFqE",
        "outputId": "325c9a3f-fb58-4e6f-a50d-53b544763d67"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The selected device is: \"cuda\"\n"
          ]
        }
      ],
      "source": [
        "import torch\n",
        "\n",
        "# Set the device for training and querying the model.\n",
        "DEVICE = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
        "print(f'The selected device is: \"{DEVICE}\"')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WkJzAnB5KFqF"
      },
      "source": [
        "# 1 Loading the Data\n",
        "In this section the data is loaded."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "sLOMzmxnKFqG"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "BASE_DATA_DIR = os.path.join('..', 'data', 'metr-la')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "7xeJlGK_KFqG"
      },
      "outputs": [],
      "source": [
        "import pickle\n",
        "with open(os.path.join(BASE_DATA_DIR, 'processed', 'scaler.pkl'), 'rb') as f:\n",
        "    scaler = pickle.load(f)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "q9OqSPOJKFqH"
      },
      "outputs": [],
      "source": [
        "from src.spatial_temporal_gnn.model import SpatialTemporalGNN\n",
        "from src.data.data_extraction import get_adjacency_matrix\n",
        "\n",
        "# Get the adjacency matrix\n",
        "adj_matrix_structure = get_adjacency_matrix(\n",
        "    os.path.join(BASE_DATA_DIR, 'raw', 'adj_mx_metr_la.pkl'))\n",
        "\n",
        "# Get the header of the adjacency matrix, the node indices and the\n",
        "# matrix itself.\n",
        "header, node_ids_dict, adj_matrix = adj_matrix_structure\n",
        "\n",
        "# Get the STGNN and load the checkpoints.\n",
        "spatial_temporal_gnn = SpatialTemporalGNN(9, 1, 12, 12, adj_matrix, DEVICE, 64)\n",
        "\n",
        "stgnn_checkpoints_path = os.path.join('..', 'models', 'checkpoints',\n",
        "                                      'st_gnn_metr_la.pth')\n",
        "\n",
        "stgnn_checkpoints = torch.load(stgnn_checkpoints_path)\n",
        "spatial_temporal_gnn.load_state_dict(stgnn_checkpoints['model_state_dict'])\n",
        "\n",
        "# Set the model in evaluation mode.\n",
        "spatial_temporal_gnn.eval();"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "V_2BKaDPKFqH"
      },
      "outputs": [],
      "source": [
        "from src.data.data_extraction import get_locations_dataframe\n",
        "\n",
        "# Get the dataframe containing the latitude and longitude of each sensor.\n",
        "locations_df = get_locations_dataframe(\n",
        "    os.path.join(BASE_DATA_DIR, 'raw', 'graph_sensor_locations_metr_la.csv'),\n",
        "    has_header=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "eiWj_ujNKFqI"
      },
      "outputs": [],
      "source": [
        "# Get the node positions dictionary.\n",
        "node_pos_dict = { i: id for id, i in node_ids_dict.items() }"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "ioKQfKUrKFqI"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import numpy as np\n",
        "from src.spatial_temporal_gnn.prediction import predict\n",
        "\n",
        "# Get the data and the values predicted by the STGNN.\n",
        "x_train = np.load(os.path.join(BASE_DATA_DIR, 'predicted', 'x_train.npy'))\n",
        "y_train = np.load(os.path.join(BASE_DATA_DIR, 'predicted', 'y_train.npy'))\n",
        "x_val = np.load(os.path.join(BASE_DATA_DIR, 'predicted', 'x_val.npy'))\n",
        "y_val = np.load(os.path.join(BASE_DATA_DIR, 'predicted', 'y_val.npy'))\n",
        "x_test = np.load(os.path.join(BASE_DATA_DIR, 'predicted', 'x_test.npy'))\n",
        "y_test = np.load(os.path.join(BASE_DATA_DIR, 'predicted', 'y_test.npy'))\n",
        "\n",
        "# Get the time information of the train, validation and test sets.\n",
        "x_train_time = np.load(\n",
        "    os.path.join(BASE_DATA_DIR, 'predicted', 'x_train_time.npy'))\n",
        "y_train_time = np.load(\n",
        "    os.path.join(BASE_DATA_DIR, 'predicted', 'y_train_time.npy'))\n",
        "x_val_time = np.load(\n",
        "    os.path.join(BASE_DATA_DIR, 'predicted', 'x_val_time.npy'))\n",
        "y_val_time = np.load(\n",
        "    os.path.join(BASE_DATA_DIR, 'predicted', 'y_val_time.npy'))\n",
        "x_test_time = np.load(\n",
        "    os.path.join(BASE_DATA_DIR, 'predicted', 'x_test_time.npy'))\n",
        "y_test_time = np.load(\n",
        "    os.path.join(BASE_DATA_DIR, 'predicted', 'y_test_time.npy'))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The predictions are turned into km/h"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "7277uo5lKFqI"
      },
      "outputs": [],
      "source": [
        "# Turn the results in kilometers per hour.\n",
        "from src.utils.config import MPH_TO_KMH_FACTOR\n",
        "\n",
        "\n",
        "y_train = y_train * MPH_TO_KMH_FACTOR\n",
        "y_val = y_val * MPH_TO_KMH_FACTOR\n",
        "y_test = y_test * MPH_TO_KMH_FACTOR"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "EE1HKIB6KFqI"
      },
      "outputs": [],
      "source": [
        "_, n_timesteps, n_nodes, _ = y_train.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# 2 Distance matrix\n",
        "The distance matrix $M$ used as a metric of dissimilarity in DBSCAN is computed separately for each instance and is composed of:\n",
        "* A *spatial distance matrix* $M_d$;\n",
        "* A *temporal distance matrix* $M_t$;\n",
        "* A *speed distance matrix* $M_s$\n",
        "\n",
        "The matrices $M_d$, $M_t$ and $M_s$ are each scaled through *min-max scaling* between $0$ and $1$ and summed together in order to obtain $M$ as follows: \n",
        "$$M = W \\cdot M_s + M_d + M_t$$\n",
        "where the speed distance is overweighted by multiplying $M_s$ by a factor $W \\geq 1$ because the speed variable is expected to play a predominant role during the clustering process. $M$ is next normalized again between $0$ and $1$ through min-max scaling."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rcmu0JKxKFqJ"
      },
      "source": [
        "## 2.1 Spatial Distance Matrix\n",
        "The *spatial distance matrix* $M_d$ is an $(N \\cdot T') \\times (N \\cdot T')$ matrix derived from the adjacency matrix of the traffic network $A \\in \\mathbb{R}^{N \\times N}$ and describing the spatial distant of each nodes regardless of their timestep."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "ZYeghX61KFqJ"
      },
      "outputs": [],
      "source": [
        "from src.explanation.clustering.clustering import (\n",
        "    get_adjacency_distance_matrix)\n",
        "\n",
        "adj_distance_matrix = get_adjacency_distance_matrix(adj_matrix, n_timesteps)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "d-8kWCYYKFqJ",
        "outputId": "43f04230-a150-47c0-b461-4018695d2b0c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Shape of the Adjacency Distance Matrix: (2484, 2484)\n"
          ]
        }
      ],
      "source": [
        "print(f'Shape of the Adjacency Distance Matrix: {adj_distance_matrix.shape}')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vGI6lnJZKFqJ"
      },
      "source": [
        "# 2.2 Temporal Distance Matrix\n",
        "The *temporal distance matrix* $M_t$ is an $(N \\cdot T') \\times (N \\cdot T')$ matrix describing the temporal distance of the nodes at each timestep."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "id": "CrwG-PfaKFqJ"
      },
      "outputs": [],
      "source": [
        "from src.explanation.clustering.clustering import (\n",
        "    get_temporal_distance_matrix)\n",
        "\n",
        "temporal_distance_matrix = get_temporal_distance_matrix(n_nodes, n_timesteps)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "B7dSerYZKFqJ",
        "outputId": "690b58be-692c-45f1-9320-157a0b905c0f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Shape of the Temporal Distance Matrix: (2484, 2484)\n"
          ]
        }
      ],
      "source": [
        "print('Shape of the Temporal Distance Matrix:',\n",
        "      f'{temporal_distance_matrix.shape}')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# 2.3 Speed Distance Matrix\n",
        "The *speed distance matrix* $M_s$ is an $(N \\cdot T') \\times (N \\cdot T')$ matrix describing the speed distance of the nodes at each timestep.\n",
        "\n",
        "It is computed for each instance separately, before DBSCAN is performed. The value $W$, overweighting it is set as $3$."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ggPKt03fKFqK"
      },
      "source": [
        "# 3 Clustering Function\n",
        "Next, the clustering technique *DBSCAN* is used on the distance matrix $M$. It identifies whether each node of the predicted traffic network at a given time, specified by an index in $M$, is part of a distinct traffic cluster or categorized as noise."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 3.1 Grid Search\n",
        "Grid search is performed on a portion of the train dataset in order to find the best set of hyperparameters of DBSCAN:\n",
        "* `eps`\n",
        "* `min_samples`\n",
        "\n",
        "The results are evaluated in terms of *Within Cluster Variance*, *Cluster Dissimilarity* and *noise ratio*.\n",
        "\n",
        "* **Within Cluster Variance:**\n",
        "    $$WCV = \\frac{1}{\\sum_{i = 1}^N n_i} \\frac{\\sum_{i = 1}^N n_i \\sigma^2_i}{\\sigma^2} $$\n",
        "    with $N$ the number of clusters, $n_i$ the nodes of the $i^{th}$ cluster in the predicted network and $\\sigma_i^2$ the speed variance among its nodes. $\\sigma^2$ is the variance among all nodes of the prediction.\n",
        "\n",
        "* **Cluster Dissimilarity:**\n",
        "    $$ CD = \\frac{\\sum_{i = 1}^N \\sum_{k = i + 1}^N \\sqrt{n_i \\cdot n_k} \\cdot |\\mu_i - \\mu_k|}{\\sum_{i = 1}^N \\sum_{k = i + 1}^N \\sqrt{n_i \\cdot n_k}} $$\n",
        "    with $\\mu_i$ the mean speed value of cluster $i$.\n",
        "\n",
        "* **Noise Ratio:** It measures the ratio of the nodes classified as outliers by DBSCAN in a predicted traffic network."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jLiV_PdkKFqK",
        "outputId": "be7b28cc-9a33-4374-f5ac-28d6166b2aca"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "eps: 0.1 min_samples: 5\n",
            "\tWithin-Cluster Variance: 0.999 Connected Cluster Dissimilarity: 5.77 Noise points ratio: 0.997\n",
            "\n",
            "eps: 0.1 min_samples: 7\n",
            "\tWithin-Cluster Variance: 1 Connected Cluster Dissimilarity: 0 Noise points ratio: 1\n",
            "\n",
            "eps: 0.1 min_samples: 10\n",
            "\tWithin-Cluster Variance: 1 Connected Cluster Dissimilarity: 0 Noise points ratio: 1\n",
            "\n",
            "eps: 0.1 min_samples: 12\n",
            "\tWithin-Cluster Variance: 1 Connected Cluster Dissimilarity: 0 Noise points ratio: 1\n",
            "\n",
            "eps: 0.1 min_samples: 15\n",
            "\tWithin-Cluster Variance: 1 Connected Cluster Dissimilarity: 0 Noise points ratio: 1\n",
            "\n",
            "eps: 0.1 min_samples: 17\n",
            "\tWithin-Cluster Variance: 1 Connected Cluster Dissimilarity: 0 Noise points ratio: 1\n",
            "\n",
            "eps: 0.1 min_samples: 20\n",
            "\tWithin-Cluster Variance: 1 Connected Cluster Dissimilarity: 0 Noise points ratio: 1\n",
            "\n",
            "eps: 0.15 min_samples: 5\n",
            "\tWithin-Cluster Variance: 0.95 Connected Cluster Dissimilarity: 11.1 Noise points ratio: 0.917\n",
            "\n",
            "eps: 0.15 min_samples: 7\n",
            "\tWithin-Cluster Variance: 0.994 Connected Cluster Dissimilarity: 7.73 Noise points ratio: 0.989\n",
            "\n",
            "eps: 0.15 min_samples: 10\n",
            "\tWithin-Cluster Variance: 1 Connected Cluster Dissimilarity: 0.169 Noise points ratio: 1\n",
            "\n",
            "eps: 0.15 min_samples: 12\n",
            "\tWithin-Cluster Variance: 1 Connected Cluster Dissimilarity: 0 Noise points ratio: 1\n",
            "\n",
            "eps: 0.15 min_samples: 15\n",
            "\tWithin-Cluster Variance: 1 Connected Cluster Dissimilarity: 0 Noise points ratio: 1\n",
            "\n",
            "eps: 0.15 min_samples: 17\n",
            "\tWithin-Cluster Variance: 1 Connected Cluster Dissimilarity: 0 Noise points ratio: 1\n",
            "\n",
            "eps: 0.15 min_samples: 20\n",
            "\tWithin-Cluster Variance: 1 Connected Cluster Dissimilarity: 0 Noise points ratio: 1\n",
            "\n",
            "eps: 0.2 min_samples: 5\n",
            "\tWithin-Cluster Variance: 0.496 Connected Cluster Dissimilarity: 11.7 Noise points ratio: 0.357\n",
            "\n",
            "eps: 0.2 min_samples: 7\n",
            "\tWithin-Cluster Variance: 0.905 Connected Cluster Dissimilarity: 10.2 Noise points ratio: 0.83\n",
            "\n",
            "eps: 0.2 min_samples: 10\n",
            "\tWithin-Cluster Variance: 0.991 Connected Cluster Dissimilarity: 8.44 Noise points ratio: 0.983\n",
            "\n",
            "eps: 0.2 min_samples: 12\n",
            "\tWithin-Cluster Variance: 0.999 Connected Cluster Dissimilarity: 1.46 Noise points ratio: 0.999\n",
            "\n",
            "eps: 0.2 min_samples: 15\n",
            "\tWithin-Cluster Variance: 1 Connected Cluster Dissimilarity: 0 Noise points ratio: 1\n",
            "\n",
            "eps: 0.2 min_samples: 17\n",
            "\tWithin-Cluster Variance: 1 Connected Cluster Dissimilarity: 0 Noise points ratio: 1\n",
            "\n",
            "eps: 0.2 min_samples: 20\n",
            "\tWithin-Cluster Variance: 1 Connected Cluster Dissimilarity: 0 Noise points ratio: 1\n",
            "\n",
            "eps: 0.25 min_samples: 5\n",
            "\tWithin-Cluster Variance: 0.11 Connected Cluster Dissimilarity: 14.7 Noise points ratio: 0.0829\n",
            "\n",
            "eps: 0.25 min_samples: 7\n",
            "\tWithin-Cluster Variance: 0.722 Connected Cluster Dissimilarity: 12.3 Noise points ratio: 0.644\n",
            "\n",
            "eps: 0.25 min_samples: 10\n",
            "\tWithin-Cluster Variance: 0.949 Connected Cluster Dissimilarity: 10.6 Noise points ratio: 0.907\n",
            "\n",
            "eps: 0.25 min_samples: 12\n",
            "\tWithin-Cluster Variance: 0.985 Connected Cluster Dissimilarity: 10.3 Noise points ratio: 0.973\n",
            "\n",
            "eps: 0.25 min_samples: 15\n",
            "\tWithin-Cluster Variance: 0.997 Connected Cluster Dissimilarity: 4.57 Noise points ratio: 0.994\n",
            "\n",
            "eps: 0.25 min_samples: 17\n",
            "\tWithin-Cluster Variance: 1 Connected Cluster Dissimilarity: 0.477 Noise points ratio: 0.999\n",
            "\n",
            "eps: 0.25 min_samples: 20\n",
            "\tWithin-Cluster Variance: 1 Connected Cluster Dissimilarity: 0 Noise points ratio: 1\n",
            "\n",
            "eps: 0.3 min_samples: 5\n",
            "\tWithin-Cluster Variance: 0.0629 Connected Cluster Dissimilarity: 15.1 Noise points ratio: 0.0432\n",
            "\n",
            "eps: 0.3 min_samples: 7\n",
            "\tWithin-Cluster Variance: 0.298 Connected Cluster Dissimilarity: 12.9 Noise points ratio: 0.215\n",
            "\n",
            "eps: 0.3 min_samples: 10\n",
            "\tWithin-Cluster Variance: 0.759 Connected Cluster Dissimilarity: 11.1 Noise points ratio: 0.645\n",
            "\n",
            "eps: 0.3 min_samples: 12\n",
            "\tWithin-Cluster Variance: 0.911 Connected Cluster Dissimilarity: 9.72 Noise points ratio: 0.826\n",
            "\n",
            "eps: 0.3 min_samples: 15\n",
            "\tWithin-Cluster Variance: 0.98 Connected Cluster Dissimilarity: 10.5 Noise points ratio: 0.961\n",
            "\n",
            "eps: 0.3 min_samples: 17\n",
            "\tWithin-Cluster Variance: 0.992 Connected Cluster Dissimilarity: 8.12 Noise points ratio: 0.983\n",
            "\n",
            "eps: 0.3 min_samples: 20\n",
            "\tWithin-Cluster Variance: 0.999 Connected Cluster Dissimilarity: 1.52 Noise points ratio: 0.998\n",
            "\n",
            "eps: 0.35 min_samples: 5\n",
            "\tWithin-Cluster Variance: 0.0602 Connected Cluster Dissimilarity: 15.4 Noise points ratio: 0.0287\n",
            "\n",
            "eps: 0.35 min_samples: 7\n",
            "\tWithin-Cluster Variance: 0.109 Connected Cluster Dissimilarity: 14.6 Noise points ratio: 0.0764\n",
            "\n",
            "eps: 0.35 min_samples: 10\n",
            "\tWithin-Cluster Variance: 0.587 Connected Cluster Dissimilarity: 12.5 Noise points ratio: 0.492\n",
            "\n",
            "eps: 0.35 min_samples: 12\n",
            "\tWithin-Cluster Variance: 0.767 Connected Cluster Dissimilarity: 11.3 Noise points ratio: 0.657\n",
            "\n",
            "eps: 0.35 min_samples: 15\n",
            "\tWithin-Cluster Variance: 0.944 Connected Cluster Dissimilarity: 10.4 Noise points ratio: 0.897\n",
            "\n",
            "eps: 0.35 min_samples: 17\n",
            "\tWithin-Cluster Variance: 0.971 Connected Cluster Dissimilarity: 10.7 Noise points ratio: 0.945\n",
            "\n",
            "eps: 0.35 min_samples: 20\n",
            "\tWithin-Cluster Variance: 0.992 Connected Cluster Dissimilarity: 7.89 Noise points ratio: 0.983\n",
            "\n",
            "eps: 0.4 min_samples: 5\n",
            "\tWithin-Cluster Variance: 0.0664 Connected Cluster Dissimilarity: 15.5 Noise points ratio: 0.0185\n",
            "\n",
            "eps: 0.4 min_samples: 7\n",
            "\tWithin-Cluster Variance: 0.0794 Connected Cluster Dissimilarity: 15.3 Noise points ratio: 0.0407\n",
            "\n",
            "eps: 0.4 min_samples: 10\n",
            "\tWithin-Cluster Variance: 0.464 Connected Cluster Dissimilarity: 13 Noise points ratio: 0.345\n",
            "\n",
            "eps: 0.4 min_samples: 12\n",
            "\tWithin-Cluster Variance: 0.557 Connected Cluster Dissimilarity: 12.3 Noise points ratio: 0.444\n",
            "\n",
            "eps: 0.4 min_samples: 15\n",
            "\tWithin-Cluster Variance: 0.803 Connected Cluster Dissimilarity: 10.7 Noise points ratio: 0.673\n",
            "\n",
            "eps: 0.4 min_samples: 17\n",
            "\tWithin-Cluster Variance: 0.901 Connected Cluster Dissimilarity: 10.3 Noise points ratio: 0.818\n",
            "\n",
            "eps: 0.4 min_samples: 20\n",
            "\tWithin-Cluster Variance: 0.962 Connected Cluster Dissimilarity: 10.5 Noise points ratio: 0.925\n",
            "\n",
            "eps: 0.45 min_samples: 5\n",
            "\tWithin-Cluster Variance: 0.081 Connected Cluster Dissimilarity: 15.8 Noise points ratio: 0.0119\n",
            "\n",
            "eps: 0.45 min_samples: 7\n",
            "\tWithin-Cluster Variance: 0.0854 Connected Cluster Dissimilarity: 15.7 Noise points ratio: 0.028\n",
            "\n",
            "eps: 0.45 min_samples: 10\n",
            "\tWithin-Cluster Variance: 0.402 Connected Cluster Dissimilarity: 13.5 Noise points ratio: 0.269\n",
            "\n",
            "eps: 0.45 min_samples: 12\n",
            "\tWithin-Cluster Variance: 0.468 Connected Cluster Dissimilarity: 13 Noise points ratio: 0.349\n",
            "\n",
            "eps: 0.45 min_samples: 15\n",
            "\tWithin-Cluster Variance: 0.639 Connected Cluster Dissimilarity: 11.9 Noise points ratio: 0.518\n",
            "\n",
            "eps: 0.45 min_samples: 17\n",
            "\tWithin-Cluster Variance: 0.803 Connected Cluster Dissimilarity: 11 Noise points ratio: 0.675\n",
            "\n",
            "eps: 0.45 min_samples: 20\n",
            "\tWithin-Cluster Variance: 0.915 Connected Cluster Dissimilarity: 10.4 Noise points ratio: 0.843\n",
            "\n",
            "eps: 0.5 min_samples: 5\n",
            "\tWithin-Cluster Variance: 0.106 Connected Cluster Dissimilarity: 16.2 Noise points ratio: 0.00795\n",
            "\n",
            "eps: 0.5 min_samples: 7\n",
            "\tWithin-Cluster Variance: 0.106 Connected Cluster Dissimilarity: 16 Noise points ratio: 0.0195\n",
            "\n",
            "eps: 0.5 min_samples: 10\n",
            "\tWithin-Cluster Variance: 0.133 Connected Cluster Dissimilarity: 15.3 Noise points ratio: 0.0645\n",
            "\n",
            "eps: 0.5 min_samples: 12\n",
            "\tWithin-Cluster Variance: 0.397 Connected Cluster Dissimilarity: 13.6 Noise points ratio: 0.25\n",
            "\n",
            "eps: 0.5 min_samples: 15\n",
            "\tWithin-Cluster Variance: 0.494 Connected Cluster Dissimilarity: 12.6 Noise points ratio: 0.361\n",
            "\n",
            "eps: 0.5 min_samples: 17\n",
            "\tWithin-Cluster Variance: 0.603 Connected Cluster Dissimilarity: 11.8 Noise points ratio: 0.457\n",
            "\n",
            "eps: 0.5 min_samples: 20\n",
            "\tWithin-Cluster Variance: 0.793 Connected Cluster Dissimilarity: 10.6 Noise points ratio: 0.641\n",
            "\n"
          ]
        }
      ],
      "source": [
        "from src.explanation.clustering.evaluation import apply_grid_search\n",
        "\n",
        "# Apply the grid search on a subset of the training set.\n",
        "apply_grid_search(\n",
        "    instances=y_train[:200],\n",
        "    eps_list=[.1, .15, .2, .25, .3, .35, .4, .45, .5],\n",
        "    min_samples_list=[5, 7, 10, 12, 15, 17, 20],\n",
        "    adj_distance_matrix=adj_distance_matrix,\n",
        "    temporal_distance_matrix=temporal_distance_matrix)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The best hyperparameters are expressed below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "id": "uaBIIuX6KFqK"
      },
      "outputs": [],
      "source": [
        "# Set the best parameters based on the results of the grid search.\n",
        "\n",
        "EPS = .35\n",
        "MIN_SAMPLES = 5"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 3.2 Clustering and Saving the Results\n",
        "In this section the clustering of the predictions of the train, validation and test datasets is computed using the selected hyperparameters.\n",
        "\n",
        "The results are evaluated in terms of *Within Cluster Variance*, *Cluster Dissimilarity* and *noise ratio* on the test set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vB3ggcBsKFqK",
        "outputId": "6d93cfe5-cec5-4a5f-de86-fcf8e78b4c65"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Within-Cluster Variance on the test set: 0.106 Connected Cluster Dissimilarity on the test set: 14.9 Noise points ratio on the test set: 0.0663\n"
          ]
        }
      ],
      "source": [
        "from src.explanation.clustering.evaluation import get_dataset_clustering_scores\n",
        "\n",
        "(avg_within_cluster_variance, avg_connected_cluster_dissimilarity,\n",
        " avg_noise_ratio) = get_dataset_clustering_scores(\n",
        "     y_test, adj_distance_matrix, temporal_distance_matrix, EPS, MIN_SAMPLES)\n",
        "\n",
        "print(\n",
        "    'Within-Cluster Variance on the test set:',\n",
        "    f'{avg_within_cluster_variance:.3g}',\n",
        "    'Connected Cluster Dissimilarity on the test set:',\n",
        "    f'{avg_connected_cluster_dissimilarity:.3g}',\n",
        "    'Noise points ratio on the test set:', f'{avg_noise_ratio:.3g}')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "id": "vfSDHmz_KFqL"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "DATA_DIR = os.path.join('..', 'data', 'metr-la', 'explainable')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "id": "MwxzqFElKFqM"
      },
      "outputs": [],
      "source": [
        "from numpy import save\n",
        "from src.explanation.clustering.clustering import (\n",
        "    get_dataset_for_explainability)\n",
        "\n",
        "os.makedirs(DATA_DIR, exist_ok=True)\n",
        "\n",
        "(x_train_expl, y_train_expl,\n",
        " x_train_time_expl, y_train_time_expl) = get_dataset_for_explainability(\n",
        "    x_train,\n",
        "    y_train,\n",
        "    x_train_time,\n",
        "    y_train_time,\n",
        "    EPS,\n",
        "    MIN_SAMPLES,\n",
        "    adj_distance_matrix,\n",
        "    temporal_distance_matrix,\n",
        "    total_samples=1_000)\n",
        "save(os.path.join(DATA_DIR, 'x_train.npy'), x_train_expl)\n",
        "save(os.path.join(DATA_DIR, 'y_train.npy'), y_train_expl)\n",
        "save(os.path.join(DATA_DIR, 'x_train_time.npy'), x_train_time_expl)\n",
        "save(os.path.join(DATA_DIR, 'y_train_time.npy'), y_train_time_expl)\n",
        "\n",
        "(x_val_expl, y_val_expl,\n",
        " x_val_time_expl, y_val_time_expl) = get_dataset_for_explainability(\n",
        "    x_val,\n",
        "    y_val,\n",
        "    x_val_time,\n",
        "    y_val_time,\n",
        "    EPS,\n",
        "    MIN_SAMPLES,\n",
        "    adj_distance_matrix,\n",
        "    temporal_distance_matrix,\n",
        "    total_samples=200)\n",
        "save(os.path.join(DATA_DIR, 'x_val.npy'), x_val_expl)\n",
        "save(os.path.join(DATA_DIR, 'y_val.npy'), y_val_expl)\n",
        "save(os.path.join(DATA_DIR, 'x_val_time.npy'), x_val_time_expl)\n",
        "save(os.path.join(DATA_DIR, 'y_val_time.npy'), y_val_time_expl)\n",
        "\n",
        "(x_test_expl, y_test_expl,\n",
        " x_test_time_expl, y_test_time_expl) = get_dataset_for_explainability(\n",
        "    x_test,\n",
        "    y_test,\n",
        "    x_test_time,\n",
        "    y_test_time,\n",
        "    EPS,\n",
        "    MIN_SAMPLES,\n",
        "    adj_distance_matrix,\n",
        "    temporal_distance_matrix,\n",
        "    total_samples=300)\n",
        "save(os.path.join(DATA_DIR, 'x_test.npy'), x_test_expl)\n",
        "save(os.path.join(DATA_DIR, 'y_test.npy'), y_test_expl)\n",
        "save(os.path.join(DATA_DIR, 'x_test_time.npy'), x_test_time_expl)\n",
        "save(os.path.join(DATA_DIR, 'y_test_time.npy'), y_test_time_expl)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OvMGNnt0KFqM",
        "outputId": "4559a9c5-b8fb-409b-ad2f-ee4b01ae06d6"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Train dataset for explainability shapes: (999, 12, 207, 9) (999, 12, 207, 1)\n",
            "Validation dataset for explainability shapes: (198, 12, 207, 9) (198, 12, 207, 1)\n",
            "Test dataset for explainability shapes: (300, 12, 207, 9) (300, 12, 207, 1)\n"
          ]
        }
      ],
      "source": [
        "print('Train dataset for explainability shapes:',\n",
        "      x_train_expl.shape, y_train_expl.shape)\n",
        "print('Validation dataset for explainability shapes:',\n",
        "      x_val_expl.shape, y_val_expl.shape)\n",
        "print('Test dataset for explainability shapes:',\n",
        "      x_test_expl.shape, y_test_expl.shape)"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.0"
    },
    "orig_nbformat": 4
  },
  "nbformat": 4,
  "nbformat_minor": 0
}