notebooks/02 Clustering the Predictions of the PeMS-Bay Dataset.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "<center>\n",
        "    <h1>Verbal Explanation of Spatial Temporal GNNs for Traffic Forecasting</h1>\n",
        "    <h2>Clustering the Predictions of the PeMS-Bay dataset</h2>\n",
        "</center>\n",
        "\n",
        "---\n",
        "\n",
        "In this notebook the predictions of the *STGNN* on the *PeMS-Bay* dataset are clustered in order to obtain events to explain such as congestions or free-flows.\n",
        "\n",
        "Firstly, a distance matrix is obtained for each predicted instance of the datasets in order to compute the spatio-temporal and speed distance among all nodes in the prediction. The distance matrix is obtained throught the method explained in *Revealing the day-to-day regularity of urban congestion patterns with\n",
        "3d speed maps* <a name=\"cite_paper\"></a>[<sup>[1]</sup>](#note_paper)\n",
        "\n",
        "Finally, the predictions are clustered through *DBSCAN*, while considering the distance matrix as a dissimilarity measure.\n",
        "\n",
        "For more detailed informations about the used functions, look into the corresponding docstrings inside the python files, inside the `src` folder.\n",
        "\n",
        "---\n",
        "<small>\n",
        "\n",
        "<a name=\"note_paper\"></a>[1] \n",
        "C. Lopez et al. “Revealing the day-to-day regularity of urban congestion patterns with\n",
        "3d speed maps”. In: *Scientific Reports, 7(1):14029*, September 2017. ISSN:\n",
        "2045-2322. DOI: 10.1038/s41598-017-14237-8. URL: https://doi.org/10.1038/s41598-017-14237-8.\n",
        "</small>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "EJvwmuUtNSE7"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "import os\n",
        "\n",
        "# Set the main path in the root folder of the project.\n",
        "sys.path.append(os.path.join('..'))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "mGwHPWyZNSE-"
      },
      "outputs": [],
      "source": [
        "# Settings for autoreloading.\n",
        "%load_ext autoreload\n",
        "%autoreload 2"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "8WA-blstNSE_"
      },
      "outputs": [],
      "source": [
        "from src.utils.seed import set_random_seed\n",
        "\n",
        "# Set the random seed for deterministic operations.\n",
        "SEED = 42\n",
        "set_random_seed(SEED)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "N95f6shPNSE_",
        "outputId": "43256ab3-8211-4bb6-d059-4b72258eeb93"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The selected device is: \"cuda\"\n"
          ]
        }
      ],
      "source": [
        "import torch\n",
        "\n",
        "# Set the device for training and querying the model.\n",
        "DEVICE = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
        "print(f'The selected device is: \"{DEVICE}\"')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uDozaYp5NSFA"
      },
      "source": [
        "# 1 Loading the Data\n",
        "In this section the data is loaded."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "oi_X5-kpNSFB"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "BASE_DATA_DIR = os.path.join('..', 'data', 'pems-bay')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "1zZ4_-yMNSFC"
      },
      "outputs": [],
      "source": [
        "import pickle\n",
        "with open(os.path.join(BASE_DATA_DIR, 'processed', 'scaler.pkl'), 'rb') as f:\n",
        "    scaler = pickle.load(f)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "Q7_8Ro4QNSFC"
      },
      "outputs": [],
      "source": [
        "from src.spatial_temporal_gnn.model import SpatialTemporalGNN\n",
        "from src.data.data_extraction import get_adjacency_matrix\n",
        "\n",
        "# Get the adjacency matrix\n",
        "adj_matrix_structure = get_adjacency_matrix(\n",
        "    os.path.join(BASE_DATA_DIR, 'raw', 'adj_mx_pems_bay.pkl'))\n",
        "\n",
        "# Get the header of the adjacency matrix, the node indices and the\n",
        "# matrix itself.\n",
        "header, node_ids_dict, adj_matrix = adj_matrix_structure\n",
        "\n",
        "# Get the STGNN and load the checkpoints.\n",
        "spatial_temporal_gnn = SpatialTemporalGNN(9, 1, 12, 12, adj_matrix, DEVICE, 64)\n",
        "\n",
        "stgnn_checkpoints_path = os.path.join('..', 'models', 'checkpoints',\n",
        "                                      'st_gnn_pems_bay.pth')\n",
        "\n",
        "stgnn_checkpoints = torch.load(stgnn_checkpoints_path)\n",
        "spatial_temporal_gnn.load_state_dict(stgnn_checkpoints['model_state_dict'])\n",
        "\n",
        "# Set the model in evaluation mode.\n",
        "spatial_temporal_gnn.eval();"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "TzeGJl4DNSFD"
      },
      "outputs": [],
      "source": [
        "from src.data.data_extraction import get_locations_dataframe\n",
        "\n",
        "# Get the dataframe containing the latitude and longitude of each sensor.\n",
        "locations_df = get_locations_dataframe(\n",
        "    os.path.join(BASE_DATA_DIR, 'raw', 'graph_sensor_locations_pems_bay.csv'),\n",
        "    has_header=False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "jqCGwKzNNSFE"
      },
      "outputs": [],
      "source": [
        "# Get the node positions dictionary.\n",
        "node_pos_dict = { i: id for id, i in node_ids_dict.items() }"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "L31nXfubNSFE"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import numpy as np\n",
        "from src.spatial_temporal_gnn.prediction import predict\n",
        "\n",
        "\n",
        "# Get the data and the values predicted by the STGNN.\n",
        "x_train = np.load(os.path.join(BASE_DATA_DIR, 'predicted', 'x_train.npy'))\n",
        "y_train = np.load(os.path.join(BASE_DATA_DIR, 'predicted', 'y_train.npy'))\n",
        "x_val = np.load(os.path.join(BASE_DATA_DIR, 'predicted', 'x_val.npy'))\n",
        "y_val = np.load(os.path.join(BASE_DATA_DIR, 'predicted', 'y_val.npy'))\n",
        "x_test = np.load(os.path.join(BASE_DATA_DIR, 'predicted', 'x_test.npy'))\n",
        "y_test = np.load(os.path.join(BASE_DATA_DIR, 'predicted', 'y_test.npy'))\n",
        "\n",
        "# Get the time information of the train, validation and test sets.\n",
        "x_train_time = np.load(\n",
        "    os.path.join(BASE_DATA_DIR, 'predicted', 'x_train_time.npy'))\n",
        "y_train_time = np.load(\n",
        "    os.path.join(BASE_DATA_DIR, 'predicted', 'y_train_time.npy'))\n",
        "x_val_time = np.load(\n",
        "    os.path.join(BASE_DATA_DIR, 'predicted', 'x_val_time.npy'))\n",
        "y_val_time = np.load(\n",
        "    os.path.join(BASE_DATA_DIR, 'predicted', 'y_val_time.npy'))\n",
        "x_test_time = np.load(\n",
        "    os.path.join(BASE_DATA_DIR, 'predicted', 'x_test_time.npy'))\n",
        "y_test_time = np.load(\n",
        "    os.path.join(BASE_DATA_DIR, 'predicted', 'y_test_time.npy'))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The predictions are turned into km/h"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "RfBcpAe-NSFF"
      },
      "outputs": [],
      "source": [
        "# Turn the results in kilometers per hour.\n",
        "from src.utils.config import MPH_TO_KMH_FACTOR\n",
        "\n",
        "\n",
        "y_train = y_train * MPH_TO_KMH_FACTOR\n",
        "y_val = y_val * MPH_TO_KMH_FACTOR\n",
        "y_test = y_test * MPH_TO_KMH_FACTOR"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "-CpxEtLONSFF"
      },
      "outputs": [],
      "source": [
        "_, n_timesteps, n_nodes, _ = y_train.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D_bSXgnINSFF"
      },
      "source": [
        "# 2 Distance matrix\n",
        "The distance matrix $M$ used as a metric of dissimilarity in DBSCAN is computed separately for each instance and is composed of:\n",
        "* A *spatial distance matrix* $M_d$;\n",
        "* A *temporal distance matrix* $M_t$;\n",
        "* A *speed distance matrix* $M_s$\n",
        "\n",
        "The matrices $M_d$, $M_t$ and $M_s$ are each scaled through *min-max scaling* between $0$ and $1$ and summed together in order to obtain $M$ as follows: \n",
        "$$M = W \\cdot M_s + M_d + M_t$$\n",
        "where the speed distance is overweighted by multiplying $M_s$ by a factor $W \\geq 1$ because the speed variable is expected to play a predominant role during the clustering process. $M$ is next normalized again between $0$ and $1$ through min-max scaling."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 2.1 Spatial Distance Matrix\n",
        "The *spatial distance matrix* $M_d$ is an $(N \\cdot T') \\times (N \\cdot T')$ matrix derived from the adjacency matrix of the traffic network $A \\in \\mathbb{R}^{N \\times N}$ and describing the spatial distant of each nodes regardless of their timestep."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "Im5Nr1qgNSFF"
      },
      "outputs": [],
      "source": [
        "from src.explanation.clustering.clustering import (\n",
        "    get_adjacency_distance_matrix)\n",
        "\n",
        "adj_distance_matrix = get_adjacency_distance_matrix(adj_matrix, n_timesteps)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3C4QgL1SNSFG",
        "outputId": "b8a3534f-8384-47c4-b30c-214bbd4f738f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Shape of the Adjacency Distance Matrix: (3900, 3900)\n"
          ]
        }
      ],
      "source": [
        "print(f'Shape of the Adjacency Distance Matrix: {adj_distance_matrix.shape}')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fFQO173FNSFG"
      },
      "source": [
        "# 2.2 Temporal Distance Matrix\n",
        "The *temporal distance matrix* $M_t$ is an $(N \\cdot T') \\times (N \\cdot T')$ matrix describing the temporal distance of the nodes at each timestep."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "evlNrnXZNSFG"
      },
      "outputs": [],
      "source": [
        "from src.explanation.clustering.clustering import (\n",
        "    get_temporal_distance_matrix)\n",
        "\n",
        "temporal_distance_matrix = get_temporal_distance_matrix(n_nodes, n_timesteps)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ENHh4g0XNSFG",
        "outputId": "4d42e02f-776f-465c-80c6-065b79fdeb71"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Shape of the Temporal Distance Matrix: (3900, 3900)\n"
          ]
        }
      ],
      "source": [
        "print('Shape of the Temporal Distance Matrix:',\n",
        "      f'{temporal_distance_matrix.shape}')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# 2.3 Speed Distance Matrix\n",
        "The *speed distance matrix* $M_s$ is an $(N \\cdot T') \\times (N \\cdot T')$ matrix describing the speed distance of the nodes at each timestep.\n",
        "\n",
        "It is computed for each instance separately, before DBSCAN is performed. The value $W$, overweighting it is set as $3$."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8HoLFLbwNSFG"
      },
      "source": [
        "# 3 Clustering Function\n",
        "Next, the clustering technique *DBSCAN* is used on the distance matrix $M$. It identifies whether each node of the predicted traffic network at a given time, specified by an index in $M$, is part of a distinct traffic cluster or categorized as noise."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 3.1 Grid Search\n",
        "Grid search is performed on a portion of the train dataset in order to find the best set of hyperparameters of DBSCAN:\n",
        "* `eps`\n",
        "* `min_samples`\n",
        "\n",
        "The results are evaluated in terms of *Within Cluster Variance*, *Cluster Dissimilarity* and *noise ratio*.\n",
        "\n",
        "* **Within Cluster Variance:**\n",
        "    $$WCV = \\frac{1}{\\sum_{i = 1}^N n_i} \\frac{\\sum_{i = 1}^N n_i \\sigma^2_i}{\\sigma^2} $$\n",
        "    with $N$ the number of clusters, $n_i$ the nodes of the $i^{th}$ cluster in the predicted network and $\\sigma_i^2$ the speed variance among its nodes. $\\sigma^2$ is the variance among all nodes of the prediction.\n",
        "\n",
        "* **Cluster Dissimilarity:**\n",
        "    $$ CD = \\frac{\\sum_{i = 1}^N \\sum_{k = i + 1}^N \\sqrt{n_i \\cdot n_k} \\cdot |\\mu_i - \\mu_k|}{\\sum_{i = 1}^N \\sum_{k = i + 1}^N \\sqrt{n_i \\cdot n_k}} $$\n",
        "    with $\\mu_i$ the mean speed value of cluster $i$.\n",
        "\n",
        "* **Noise Ratio:** It measures the ratio of the nodes classified as outliers by DBSCAN in a predicted traffic network."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ISnmYYA1NSFH",
        "outputId": "34d4b412-e52c-4d31-8f40-e79571ef6e00"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "eps: 0.1 min_samples: 5\n",
            "\tWithin-Cluster Variance: 0.984 Connected Cluster Dissimilarity: 4.2 Noise points ratio: 0.95\n",
            "\n",
            "eps: 0.1 min_samples: 7\n",
            "\tWithin-Cluster Variance: 0.997 Connected Cluster Dissimilarity: 4.06 Noise points ratio: 0.991\n",
            "\n",
            "eps: 0.1 min_samples: 10\n",
            "\tWithin-Cluster Variance: 1 Connected Cluster Dissimilarity: 0.827 Noise points ratio: 0.999\n",
            "\n",
            "eps: 0.1 min_samples: 12\n",
            "\tWithin-Cluster Variance: 1 Connected Cluster Dissimilarity: 0 Noise points ratio: 1\n",
            "\n",
            "eps: 0.1 min_samples: 15\n",
            "\tWithin-Cluster Variance: 1 Connected Cluster Dissimilarity: 0 Noise points ratio: 1\n",
            "\n",
            "eps: 0.1 min_samples: 17\n",
            "\tWithin-Cluster Variance: 1 Connected Cluster Dissimilarity: 0 Noise points ratio: 1\n",
            "\n",
            "eps: 0.1 min_samples: 20\n",
            "\tWithin-Cluster Variance: 1 Connected Cluster Dissimilarity: 0 Noise points ratio: 1\n",
            "\n",
            "eps: 0.15 min_samples: 5\n",
            "\tWithin-Cluster Variance: 0.839 Connected Cluster Dissimilarity: 6.12 Noise points ratio: 0.677\n",
            "\n",
            "eps: 0.15 min_samples: 7\n",
            "\tWithin-Cluster Variance: 0.964 Connected Cluster Dissimilarity: 4.47 Noise points ratio: 0.889\n",
            "\n",
            "eps: 0.15 min_samples: 10\n",
            "\tWithin-Cluster Variance: 0.995 Connected Cluster Dissimilarity: 3.85 Noise points ratio: 0.981\n",
            "\n",
            "eps: 0.15 min_samples: 12\n",
            "\tWithin-Cluster Variance: 0.999 Connected Cluster Dissimilarity: 2.15 Noise points ratio: 0.995\n",
            "\n",
            "eps: 0.15 min_samples: 15\n",
            "\tWithin-Cluster Variance: 1 Connected Cluster Dissimilarity: 0.0755 Noise points ratio: 0.999\n",
            "\n",
            "eps: 0.15 min_samples: 17\n",
            "\tWithin-Cluster Variance: 1 Connected Cluster Dissimilarity: 0.00807 Noise points ratio: 1\n",
            "\n",
            "eps: 0.15 min_samples: 20\n",
            "\tWithin-Cluster Variance: 1 Connected Cluster Dissimilarity: 0 Noise points ratio: 1\n",
            "\n",
            "eps: 0.2 min_samples: 5\n",
            "\tWithin-Cluster Variance: 0.426 Connected Cluster Dissimilarity: 7.41 Noise points ratio: 0.209\n",
            "\n",
            "eps: 0.2 min_samples: 7\n",
            "\tWithin-Cluster Variance: 0.786 Connected Cluster Dissimilarity: 6.04 Noise points ratio: 0.571\n",
            "\n",
            "eps: 0.2 min_samples: 10\n",
            "\tWithin-Cluster Variance: 0.948 Connected Cluster Dissimilarity: 4.48 Noise points ratio: 0.846\n",
            "\n",
            "eps: 0.2 min_samples: 12\n",
            "\tWithin-Cluster Variance: 0.984 Connected Cluster Dissimilarity: 4.1 Noise points ratio: 0.934\n",
            "\n",
            "eps: 0.2 min_samples: 15\n",
            "\tWithin-Cluster Variance: 0.994 Connected Cluster Dissimilarity: 4.12 Noise points ratio: 0.978\n",
            "\n",
            "eps: 0.2 min_samples: 17\n",
            "\tWithin-Cluster Variance: 0.998 Connected Cluster Dissimilarity: 2.51 Noise points ratio: 0.99\n",
            "\n",
            "eps: 0.2 min_samples: 20\n",
            "\tWithin-Cluster Variance: 1 Connected Cluster Dissimilarity: 0.595 Noise points ratio: 0.997\n",
            "\n",
            "eps: 0.25 min_samples: 5\n",
            "\tWithin-Cluster Variance: 0.122 Connected Cluster Dissimilarity: 8.75 Noise points ratio: 0.0315\n",
            "\n",
            "eps: 0.25 min_samples: 7\n",
            "\tWithin-Cluster Variance: 0.651 Connected Cluster Dissimilarity: 6.91 Noise points ratio: 0.389\n",
            "\n",
            "eps: 0.25 min_samples: 10\n",
            "\tWithin-Cluster Variance: 0.813 Connected Cluster Dissimilarity: 5.8 Noise points ratio: 0.602\n",
            "\n",
            "eps: 0.25 min_samples: 12\n",
            "\tWithin-Cluster Variance: 0.921 Connected Cluster Dissimilarity: 4.74 Noise points ratio: 0.772\n",
            "\n",
            "eps: 0.25 min_samples: 15\n",
            "\tWithin-Cluster Variance: 0.974 Connected Cluster Dissimilarity: 4.15 Noise points ratio: 0.906\n",
            "\n",
            "eps: 0.25 min_samples: 17\n",
            "\tWithin-Cluster Variance: 0.988 Connected Cluster Dissimilarity: 4.11 Noise points ratio: 0.949\n",
            "\n",
            "eps: 0.25 min_samples: 20\n",
            "\tWithin-Cluster Variance: 0.996 Connected Cluster Dissimilarity: 3.62 Noise points ratio: 0.98\n",
            "\n",
            "eps: 0.3 min_samples: 5\n",
            "\tWithin-Cluster Variance: 0.108 Connected Cluster Dissimilarity: 8.98 Noise points ratio: 0.0107\n",
            "\n",
            "eps: 0.3 min_samples: 7\n",
            "\tWithin-Cluster Variance: 0.317 Connected Cluster Dissimilarity: 7.93 Noise points ratio: 0.101\n",
            "\n",
            "eps: 0.3 min_samples: 10\n",
            "\tWithin-Cluster Variance: 0.656 Connected Cluster Dissimilarity: 6.67 Noise points ratio: 0.371\n",
            "\n",
            "eps: 0.3 min_samples: 12\n",
            "\tWithin-Cluster Variance: 0.764 Connected Cluster Dissimilarity: 5.89 Noise points ratio: 0.502\n",
            "\n",
            "eps: 0.3 min_samples: 15\n",
            "\tWithin-Cluster Variance: 0.891 Connected Cluster Dissimilarity: 4.82 Noise points ratio: 0.699\n",
            "\n",
            "eps: 0.3 min_samples: 17\n",
            "\tWithin-Cluster Variance: 0.936 Connected Cluster Dissimilarity: 4.44 Noise points ratio: 0.796\n",
            "\n",
            "eps: 0.3 min_samples: 20\n",
            "\tWithin-Cluster Variance: 0.974 Connected Cluster Dissimilarity: 4.11 Noise points ratio: 0.897\n",
            "\n",
            "eps: 0.35 min_samples: 5\n",
            "\tWithin-Cluster Variance: 0.139 Connected Cluster Dissimilarity: 8.99 Noise points ratio: 0.00462\n",
            "\n",
            "eps: 0.35 min_samples: 7\n",
            "\tWithin-Cluster Variance: 0.177 Connected Cluster Dissimilarity: 8.64 Noise points ratio: 0.0253\n",
            "\n",
            "eps: 0.35 min_samples: 10\n",
            "\tWithin-Cluster Variance: 0.573 Connected Cluster Dissimilarity: 7.06 Noise points ratio: 0.259\n",
            "\n",
            "eps: 0.35 min_samples: 12\n",
            "\tWithin-Cluster Variance: 0.656 Connected Cluster Dissimilarity: 6.63 Noise points ratio: 0.353\n",
            "\n",
            "eps: 0.35 min_samples: 15\n",
            "\tWithin-Cluster Variance: 0.801 Connected Cluster Dissimilarity: 5.69 Noise points ratio: 0.535\n",
            "\n",
            "eps: 0.35 min_samples: 17\n",
            "\tWithin-Cluster Variance: 0.855 Connected Cluster Dissimilarity: 5.21 Noise points ratio: 0.624\n",
            "\n",
            "eps: 0.35 min_samples: 20\n",
            "\tWithin-Cluster Variance: 0.925 Connected Cluster Dissimilarity: 4.44 Noise points ratio: 0.759\n",
            "\n",
            "eps: 0.4 min_samples: 5\n",
            "\tWithin-Cluster Variance: 0.178 Connected Cluster Dissimilarity: 8.94 Noise points ratio: 0.0024\n",
            "\n",
            "eps: 0.4 min_samples: 7\n",
            "\tWithin-Cluster Variance: 0.183 Connected Cluster Dissimilarity: 8.82 Noise points ratio: 0.00958\n",
            "\n",
            "eps: 0.4 min_samples: 10\n",
            "\tWithin-Cluster Variance: 0.518 Connected Cluster Dissimilarity: 7.32 Noise points ratio: 0.168\n",
            "\n",
            "eps: 0.4 min_samples: 12\n",
            "\tWithin-Cluster Variance: 0.569 Connected Cluster Dissimilarity: 6.97 Noise points ratio: 0.221\n",
            "\n",
            "eps: 0.4 min_samples: 15\n",
            "\tWithin-Cluster Variance: 0.667 Connected Cluster Dissimilarity: 6.45 Noise points ratio: 0.337\n",
            "\n",
            "eps: 0.4 min_samples: 17\n",
            "\tWithin-Cluster Variance: 0.744 Connected Cluster Dissimilarity: 5.94 Noise points ratio: 0.429\n",
            "\n",
            "eps: 0.4 min_samples: 20\n",
            "\tWithin-Cluster Variance: 0.83 Connected Cluster Dissimilarity: 5.29 Noise points ratio: 0.561\n",
            "\n",
            "eps: 0.45 min_samples: 5\n",
            "\tWithin-Cluster Variance: 0.221 Connected Cluster Dissimilarity: 8.81 Noise points ratio: 0.00134\n",
            "\n",
            "eps: 0.45 min_samples: 7\n",
            "\tWithin-Cluster Variance: 0.221 Connected Cluster Dissimilarity: 8.75 Noise points ratio: 0.00504\n",
            "\n",
            "eps: 0.45 min_samples: 10\n",
            "\tWithin-Cluster Variance: 0.509 Connected Cluster Dissimilarity: 7.42 Noise points ratio: 0.13\n",
            "\n",
            "eps: 0.45 min_samples: 12\n",
            "\tWithin-Cluster Variance: 0.536 Connected Cluster Dissimilarity: 7.18 Noise points ratio: 0.166\n",
            "\n",
            "eps: 0.45 min_samples: 15\n",
            "\tWithin-Cluster Variance: 0.61 Connected Cluster Dissimilarity: 6.78 Noise points ratio: 0.243\n",
            "\n",
            "eps: 0.45 min_samples: 17\n",
            "\tWithin-Cluster Variance: 0.665 Connected Cluster Dissimilarity: 6.45 Noise points ratio: 0.313\n",
            "\n",
            "eps: 0.45 min_samples: 20\n",
            "\tWithin-Cluster Variance: 0.762 Connected Cluster Dissimilarity: 5.87 Noise points ratio: 0.435\n",
            "\n",
            "eps: 0.5 min_samples: 5\n",
            "\tWithin-Cluster Variance: 0.263 Connected Cluster Dissimilarity: 8.66 Noise points ratio: 0.000797\n",
            "\n",
            "eps: 0.5 min_samples: 7\n",
            "\tWithin-Cluster Variance: 0.261 Connected Cluster Dissimilarity: 8.64 Noise points ratio: 0.00275\n",
            "\n",
            "eps: 0.5 min_samples: 10\n",
            "\tWithin-Cluster Variance: 0.278 Connected Cluster Dissimilarity: 8.39 Noise points ratio: 0.0199\n",
            "\n",
            "eps: 0.5 min_samples: 12\n",
            "\tWithin-Cluster Variance: 0.521 Connected Cluster Dissimilarity: 7.35 Noise points ratio: 0.118\n",
            "\n",
            "eps: 0.5 min_samples: 15\n",
            "\tWithin-Cluster Variance: 0.557 Connected Cluster Dissimilarity: 7.02 Noise points ratio: 0.165\n",
            "\n",
            "eps: 0.5 min_samples: 17\n",
            "\tWithin-Cluster Variance: 0.599 Connected Cluster Dissimilarity: 6.72 Noise points ratio: 0.206\n",
            "\n",
            "eps: 0.5 min_samples: 20\n",
            "\tWithin-Cluster Variance: 0.668 Connected Cluster Dissimilarity: 6.34 Noise points ratio: 0.287\n",
            "\n"
          ]
        }
      ],
      "source": [
        "from src.explanation.clustering.evaluation import apply_grid_search\n",
        "\n",
        "# Apply the grid search on a subset of the training set.\n",
        "apply_grid_search(\n",
        "    instances=y_train[:200],\n",
        "    eps_list=[.1, .15, .2, .25, .3, .35, .4, .45, .5],\n",
        "    min_samples_list=[5, 7, 10, 12, 15, 17, 20],\n",
        "    adj_distance_matrix=adj_distance_matrix,\n",
        "    temporal_distance_matrix=temporal_distance_matrix)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The best hyperparameters are expressed below."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "id": "yTw5SGaeNSFH"
      },
      "outputs": [],
      "source": [
        "# Set the best parameters based on the results of the grid search.\n",
        "\n",
        "EPS = .35\n",
        "MIN_SAMPLES = 5"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## 3.2 Clustering and Saving the Results\n",
        "In this section the clustering of the predictions of the train, validation and test datasets is computed using the selected hyperparameters.\n",
        "\n",
        "The results are evaluated in terms of *Within Cluster Variance*, *Cluster Dissimilarity* and *noise ratio* on the test set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Fppd8gdSNSFH",
        "outputId": "d07a6108-26c6-4902-c6ea-ddda8875af9d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Within-Cluster Variance on the test set: 0.159 Connected Cluster Dissimilarity on the test set: 9.61 Noise points ratio on the test set: 0.00601\n"
          ]
        }
      ],
      "source": [
        "from src.explanation.clustering.evaluation import get_dataset_clustering_scores\n",
        "\n",
        "(avg_within_cluster_variance, avg_connected_cluster_dissimilarity,\n",
        " avg_noise_ratio) = get_dataset_clustering_scores(\n",
        "     y_test, adj_distance_matrix, temporal_distance_matrix, EPS, MIN_SAMPLES)\n",
        "\n",
        "print(\n",
        "    'Within-Cluster Variance on the test set:',\n",
        "    f'{avg_within_cluster_variance:.3g}',\n",
        "    'Connected Cluster Dissimilarity on the test set:',\n",
        "    f'{avg_connected_cluster_dissimilarity:.3g}',\n",
        "    'Noise points ratio on the test set:', f'{avg_noise_ratio:.3g}')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "id": "rgh6eFLyNSFJ"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "DATA_DIR = os.path.join('..', 'data', 'pems-bay', 'explainable')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "id": "ehqtUhCJNSFJ"
      },
      "outputs": [],
      "source": [
        "from numpy import save\n",
        "from src.explanation.clustering.clustering import (\n",
        "    get_dataset_for_explainability)\n",
        "\n",
        "os.makedirs(DATA_DIR, exist_ok=True)\n",
        "\n",
        "(x_train_expl, y_train_expl,\n",
        " x_train_time_expl, y_train_time_expl) = get_dataset_for_explainability(\n",
        "    x_train,\n",
        "    y_train,\n",
        "    x_train_time,\n",
        "    y_train_time,\n",
        "    EPS,\n",
        "    MIN_SAMPLES,\n",
        "    adj_distance_matrix,\n",
        "    temporal_distance_matrix,\n",
        "    total_samples=1_000)\n",
        "save(os.path.join(DATA_DIR, 'x_train.npy'), x_train_expl)\n",
        "save(os.path.join(DATA_DIR, 'y_train.npy'), y_train_expl)\n",
        "save(os.path.join(DATA_DIR, 'x_train_time.npy'), x_train_time_expl)\n",
        "save(os.path.join(DATA_DIR, 'y_train_time.npy'), y_train_time_expl)\n",
        "\n",
        "(x_val_expl, y_val_expl,\n",
        " x_val_time_expl, y_val_time_expl) = get_dataset_for_explainability(\n",
        "    x_val,\n",
        "    y_val,\n",
        "    x_val_time,\n",
        "    y_val_time,\n",
        "    EPS,\n",
        "    MIN_SAMPLES,\n",
        "    adj_distance_matrix,\n",
        "    temporal_distance_matrix,\n",
        "    total_samples=200)\n",
        "save(os.path.join(DATA_DIR, 'x_val.npy'), x_val_expl)\n",
        "save(os.path.join(DATA_DIR, 'y_val.npy'), y_val_expl)\n",
        "save(os.path.join(DATA_DIR, 'x_val_time.npy'), x_val_time_expl)\n",
        "save(os.path.join(DATA_DIR, 'y_val_time.npy'), y_val_time_expl)\n",
        "\n",
        "(x_test_expl, y_test_expl,\n",
        " x_test_time_expl, y_test_time_expl) = get_dataset_for_explainability(\n",
        "    x_test,\n",
        "    y_test,\n",
        "    x_test_time,\n",
        "    y_test_time,\n",
        "    EPS,\n",
        "    MIN_SAMPLES,\n",
        "    adj_distance_matrix,\n",
        "    temporal_distance_matrix,\n",
        "    total_samples=300)\n",
        "save(os.path.join(DATA_DIR, 'x_test.npy'), x_test_expl)\n",
        "save(os.path.join(DATA_DIR, 'y_test.npy'), y_test_expl)\n",
        "save(os.path.join(DATA_DIR, 'x_test_time.npy'), x_test_time_expl)\n",
        "save(os.path.join(DATA_DIR, 'y_test_time.npy'), y_test_time_expl)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OWR1ea1HNSFJ",
        "outputId": "2ef804d9-30f3-4356-c98c-672623004363"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Train dataset for explainability shapes: (999, 12, 325, 9) (999, 12, 325, 1)\n",
            "Validation dataset for explainability shapes: (198, 12, 325, 9) (198, 12, 325, 1)\n",
            "Test dataset for explainability shapes: (300, 12, 325, 9) (300, 12, 325, 1)\n"
          ]
        }
      ],
      "source": [
        "print('Train dataset for explainability shapes:',\n",
        "      x_train_expl.shape, y_train_expl.shape)\n",
        "print('Validation dataset for explainability shapes:',\n",
        "      x_val_expl.shape, y_val_expl.shape)\n",
        "print('Test dataset for explainability shapes:',\n",
        "      x_test_expl.shape, y_test_expl.shape)"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.0"
    },
    "orig_nbformat": 4
  },
  "nbformat": 4,
  "nbformat_minor": 0
}