notebooks/06 Verbal Explanations on the PeMS-Bay Dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>\n",
    "    <h1>Verbal Explanation of Spatial Temporal GNNs for Traffic Forecasting</h1>\n",
    "    <h2>Verbal Explanations on the PeMS-Bay Dataset</h2>\n",
    "</center>\n",
    "\n",
    "---\n",
    "\n",
    "The verbal translation occurs through a template-based approach. This method involves substituting placeholders in textual templates with the chosen content to form coherent narratives.\n",
    "\n",
    "The verbal translation consists of composing a series of paragraphs by exploiting the content extracted from the graphical explanations. The first paragraph describes the predicted event and briefly sums up its causes, while the second to last paragraphs illustrate in detail each cause leading to the event which is each cluster of the important subgraph. The paragraphs describing the causes are sorted by the time they occurred."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import os\n",
    "\n",
    "# Set the main path in the root folder of the project.\n",
    "sys.path.append(os.path.join('..'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Settings for autoreloading.\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from src.utils.seed import set_random_seed\n",
    "\n",
    "# Set the random seed for deterministic operations.\n",
    "SEED = 42\n",
    "set_random_seed(SEED)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1 Loading the Data\n",
    "In this section the data is loaded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "BASE_DATA_DIR = os.path.join('..', 'data', 'pems-bay')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from src.data.data_extraction import get_adjacency_matrix\n",
    "\n",
    "# Get the adjacency matrix\n",
    "adj_matrix_structure = get_adjacency_matrix(\n",
    "    os.path.join(BASE_DATA_DIR, 'raw', 'adj_mx_pems_bay.pkl'))\n",
    "\n",
    "# Get the header of the adjacency matrix, the node indices and the\n",
    "# matrix itself.\n",
    "_, node_ids_dict, _ = adj_matrix_structure\n",
    "\n",
    "# Get the node positions dictionary.\n",
    "node_pos_dict = { i: id for id, i in node_ids_dict.items() }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "# Get the node street and kilometrage dictionary.\n",
    "with open(os.path.join(BASE_DATA_DIR, 'structured', 'node_locations.pkl'), 'rb') as f:\n",
    "    node_info = pickle.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "\n",
    "# Get the explained data.\n",
    "x_train = np.load(os.path.join(BASE_DATA_DIR, 'explained', 'x_train.npy'))[..., :1]\n",
    "y_train = np.load(os.path.join(BASE_DATA_DIR, 'explained', 'y_train.npy'))[..., :1]\n",
    "x_val = np.load(os.path.join(BASE_DATA_DIR, 'explained', 'x_val.npy'))[..., :1]\n",
    "y_val = np.load(os.path.join(BASE_DATA_DIR, 'explained', 'y_val.npy'))[..., :1]\n",
    "x_test = np.load(os.path.join(BASE_DATA_DIR, 'explained', 'x_test.npy'))[..., :1]\n",
    "y_test = np.load(os.path.join(BASE_DATA_DIR, 'explained', 'y_test.npy'))[..., :1]\n",
    "\n",
    "# Get the clustered data.\n",
    "x_train_clusters = np.load(os.path.join(BASE_DATA_DIR, 'clustered', 'x_train.npy'))\n",
    "x_val_clusters = np.load(os.path.join(BASE_DATA_DIR, 'clustered', 'x_val.npy'))\n",
    "x_test_clusters = np.load(os.path.join(BASE_DATA_DIR, 'clustered', 'x_test.npy'))\n",
    "\n",
    "\n",
    "# Get the time information of the explained data.\n",
    "x_train_times = np.load(os.path.join(BASE_DATA_DIR, 'explained', 'x_train_time.npy'))\n",
    "y_train_times = np.load(os.path.join(BASE_DATA_DIR, 'explained', 'y_train_time.npy'))\n",
    "x_val_times = np.load(os.path.join(BASE_DATA_DIR, 'explained', 'x_val_time.npy'))\n",
    "y_val_times = np.load(os.path.join(BASE_DATA_DIR, 'explained', 'y_val_time.npy'))\n",
    "x_test_times = np.load(os.path.join(BASE_DATA_DIR, 'explained', 'x_test_time.npy'))\n",
    "y_test_times = np.load(os.path.join(BASE_DATA_DIR, 'explained', 'y_test_time.npy'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The datasets are turned in km/h."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from src.utils.config import MPH_TO_KMH_FACTOR\n",
    "\n",
    "x_train = x_train * MPH_TO_KMH_FACTOR\n",
    "y_train = y_train * MPH_TO_KMH_FACTOR\n",
    "\n",
    "x_val = x_val * MPH_TO_KMH_FACTOR\n",
    "y_val = y_val * MPH_TO_KMH_FACTOR\n",
    "\n",
    "x_test = x_test * MPH_TO_KMH_FACTOR\n",
    "y_test = y_test * MPH_TO_KMH_FACTOR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2 Verbal Translation\n",
    "The translation is performed on the datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "VERBAL_TRANSLATION_DIR = os.path.join(BASE_DATA_DIR, 'translated')\n",
    "\n",
    "os.makedirs(VERBAL_TRANSLATION_DIR, exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from src.verbal_explanations.verbal_translation import translate_dataset\n",
    "\n",
    "train_translated = translate_dataset(\n",
    "    x_train,\n",
    "    x_train_times,\n",
    "    x_train_clusters,\n",
    "    y_train,\n",
    "    y_train_times,\n",
    "    node_pos_dict,\n",
    "    node_info)\n",
    "    \n",
    "\n",
    "np.save(os.path.join(VERBAL_TRANSLATION_DIR, 'train.npy'), train_translated)\n",
    "\n",
    "val_translated = translate_dataset(\n",
    "    x_val,\n",
    "    x_val_times,\n",
    "    x_val_clusters,\n",
    "    y_val,\n",
    "    y_val_times,\n",
    "    node_pos_dict,\n",
    "    node_info)\n",
    "\n",
    "np.save(os.path.join(VERBAL_TRANSLATION_DIR, 'val.npy'), val_translated)\n",
    "\n",
    "test_translated = translate_dataset(\n",
    "    x_test,\n",
    "    x_test_times,\n",
    "    x_test_clusters,\n",
    "    y_test,\n",
    "    y_test_times,\n",
    "    node_pos_dict,\n",
    "    node_info)\n",
    "\n",
    "np.save(os.path.join(VERBAL_TRANSLATION_DIR, 'test.npy'), test_translated)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Following an example of a verbal translation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A congestion was forecasted to hit I 880 at km 60 on Wednesday, 25/05/2017, from 20:20 to 21:15, with an average speed of 88.10 km/h. This was triggered by a series of congestions and free flows.\n",
      "\n",
      "An initial contributing congestion manifested on I 880 at kms 59 and 60 from 19:20 to 19:45 with an average speed of 63.09 km/h.\n",
      "\n",
      "Following this, a contributing free flow manifested from 19:20 to 20:15 on, once more, I 880 at kms 59 and 60 with an average speed of 101.28 km/h.\n",
      "\n",
      "Next, another contributing free flow materialized from 19:25 to 19:50 on Bayshore Freeway at km 63 with an average speed of 114.14 km/h.\n",
      "\n",
      "Next, yet a new contributing free flow occurred, with an average speed of 96.74 km/h, on, yet another time, I 880 at kms 58 and 59 occurring from 19:25 to 20:15.\n",
      "\n",
      "Eventually, an extra contributing congestion occurred, at 77.57 km/h, on, yet another time, I 880 at km 60 at 20:10.\n"
     ]
    }
   ],
   "source": [
    "print(test_translated[100])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}