ECML-PKDD_SCIMAI-Gym.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Notebook Info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">**TITLE:** *SCIMAI-Gym*  \n",
    "**AUTHORS:** *Francesco Stranieri*  \n",
    "**INSTITUTION:** *University of Milano-Bicocca/Polytechnic of Turin*  \n",
    "**EMAIL:** *francesco.stranieri@unimib.it*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[The MIT License (MIT)](https://github.com/frenkowski/SCIMAI-Gym/blob/main/LICENSE)\n",
    "\n",
    "Copyright (c) 2023 Francesco Stranieri"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Code inspired by hands-on tutorial ['Deep reinforcement learning for supply chain and price optimization'](https://blog.griddynamics.com/deep-reinforcement-learning-for-supply-chain-and-price-optimization/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "VEoTunnryTUu"
   },
   "source": [
    "# Environment Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "RvTz-3nAyZCV"
   },
   "source": [
    "## Install Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1630492340088
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "89GXiohzf6u8",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "OpenAI Gym is a toolkit for developing and comparing Reinforcement Learning\n",
    "algorithms.\n",
    "\"\"\"\n",
    "%pip install gym == 0.19.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1630492342731
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "yf1jCrgMp0nJ"
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Ray provides a simple, universal API for building distributed applications.\n",
    "For accelerating Machine Learning workloads, Ray is packaged with:\n",
    "- RLlib: a Scalable Reinforcement Learning library;\n",
    "- Tune: a Scalable Hyperparameter Tuning library.\n",
    "https://ray.io/\n",
    "\"\"\"\n",
    "%pip install ray == 1.5.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1630492345337
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "HGS-lL1g0y4F"
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Ax is a platform for optimizing any kind of experiment, including Machine\n",
    "Learning experiments, A/B tests, and simulations. Ax can optimize discrete\n",
    "configurations (e.g., variants of an A/B test) using multi-armed bandit\n",
    "optimization, and continuous (e.g., integer or floating point)-valued\n",
    "configurations using Bayesian optimization.\n",
    "https://ax.dev/\n",
    "\"\"\"\n",
    "%pip install ax-platform == 0.2.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1630492375917
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "S-MjqnEN_KHZ"
   },
   "outputs": [],
   "source": [
    "# for getting the GPU status\n",
    "%pip install -U GPUtil\n",
    "# for embedding pandas DataFrames as images\n",
    "%pip install -U dataframe_image\n",
    "# for working with nested data structures\n",
    "%pip install -U dm-tree\n",
    "# for generating static images\n",
    "%pip install -U kaleido\n",
    "# for processing XML and HTML\n",
    "%pip install -U lxml\n",
    "# interface for the LZ4 compression library\n",
    "%pip install -U lz4\n",
    "# for creating publication-quality figures\n",
    "%pip install matplotlib==3.4.3\n",
    "# for scientific computing\n",
    "%pip install -U numpy\n",
    "# data analysis and manipulation tool\n",
    "%pip install -U pandas\n",
    "# for making statistical graphics\n",
    "%pip install -U seaborn\n",
    "# pretty-print tabular data\n",
    "%pip install -U tabulate\n",
    "# Tune automatically outputs TensorBoard files\n",
    "%pip install -U tensorboardX"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1630492376498
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "oyoEsQcr3yLi",
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "# required to use Tune with Python >= 3.7\n",
    "import sys\n",
    "\n",
    "version_info = sys.version_info\n",
    "print(f\"Python version is {version_info}\")\n",
    "\n",
    "if sys.version_info >= (3, 7):\n",
    "    %pip uninstall -y dataclasses\n",
    "else:\n",
    "    %pip install -U dataclasses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "P5SDzcEvrbQq"
   },
   "outputs": [],
   "source": [
    "# a hack to force the runtime to restart, needed to include the above\n",
    "# dependencies\n",
    "import os\n",
    "\n",
    "os._exit(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "rjl7Ka3XyjoD"
   },
   "source": [
    "## Import Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633592872064
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "MhOyL_TDJ2iO"
   },
   "outputs": [],
   "source": [
    "# Python logging\n",
    "import logging\n",
    "\n",
    "logging.basicConfig()\n",
    "logger = logging.getLogger('LOGGING_SCIMAI-Gym_V1')\n",
    "logger.setLevel(logging.WARN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633592872889
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "spDLrcrkpn7o"
   },
   "outputs": [],
   "source": [
    "# importing Gym\n",
    "import gym\n",
    "from gym.spaces import Box"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633592874996
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "4Tqoqypcf6u_",
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "# importing Ray\n",
    "import ray\n",
    "from ray import tune\n",
    "from ray.tune.schedulers import ASHAScheduler\n",
    "\n",
    "import ray.rllib.agents.a3c as a3c\n",
    "import ray.rllib.agents.pg as pg\n",
    "import ray.rllib.agents.ppo as ppo\n",
    "\n",
    "from ray.rllib.utils import try_import_torch\n",
    "torch = try_import_torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633592877125
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "m5hYuJRv0tSF"
   },
   "outputs": [],
   "source": [
    "# importing Ax\n",
    "from ax import optimize\n",
    "\n",
    "from ax.plot.contour import interact_contour\n",
    "from ax.plot.contour import plot_contour_plotly\n",
    "from ax.plot.trace import optimization_trace_single_method_plotly\n",
    "\n",
    "from ax.utils.notebook.plotting import render\n",
    "from ax.utils.notebook.plotting import init_notebook_plotting\n",
    "init_notebook_plotting()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633592877459
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "IZflD-Ppoqfi"
   },
   "outputs": [],
   "source": [
    "# importing necessary libraries\n",
    "from datetime import datetime\n",
    "from itertools import chain\n",
    "from tabulate import tabulate\n",
    "from timeit import default_timer\n",
    "from IPython.display import display\n",
    "\n",
    "import collections\n",
    "import dataframe_image as dfi\n",
    "import glob\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.ticker as ticker\n",
    "import numpy as np\n",
    "import os\n",
    "import pandas as pd\n",
    "import random\n",
    "import seaborn as sns\n",
    "import shutil\n",
    "\n",
    "plt.style.use('seaborn')\n",
    "sns.set(context='notebook')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633592877751
    },
    "hideCode": true,
    "hidePrompt": true
   },
   "outputs": [],
   "source": [
    "# setting seed for reproducibility\n",
    "seed = 2021\n",
    "np.random.seed(seed)\n",
    "random.seed(seed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633592878023
    },
    "hideCode": true,
    "hidePrompt": true,
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "# setting output views only in case of debug\n",
    "if logger.level == 10:\n",
    "    verbose = 3\n",
    "    plt.ion()\n",
    "else:\n",
    "    verbose = 0\n",
    "    plt.ioff()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633592878307
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "ldKuxKJbf6u_"
   },
   "outputs": [],
   "source": [
    "# getting the number of CPUs\n",
    "import multiprocessing\n",
    "\n",
    "try:\n",
    "    num_cpus = multiprocessing.cpu_count()\n",
    "except Exception as e:\n",
    "    print(f\"{e.__class__} occurred!\")\n",
    "    num_cpus = 0\n",
    "\n",
    "print(f\"num cpus is {num_cpus}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": [],
    "gather": {
     "logged": 1633592878585
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "GVf7tvofDhnj"
   },
   "outputs": [],
   "source": [
    "# getting the number of GPUs\n",
    "import GPUtil as GPU\n",
    "\n",
    "try:\n",
    "    num_gpus = len(GPU.getGPUs())\n",
    "except Exception as e:\n",
    "    print(f\"{e.__class__} occurred!\")\n",
    "    num_gpus = 0\n",
    "\n",
    "print(f\"num gpus is {num_gpus}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "xh1UU-78yoU7"
   },
   "source": [
    "# Reinforcement Learning Classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "BH70DLoUzFRI"
   },
   "source": [
    "## State Class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633592878862
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "2LVOAZI7pJaB"
   },
   "outputs": [],
   "source": [
    "class State:\n",
    "    \"\"\"\n",
    "    We choose the state vector to include all current stock levels for each \n",
    "    warehouse and product type, plus the last demand values.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, product_types_num, distr_warehouses_num, T,\n",
    "                 demand_history, t=0):\n",
    "        self.product_types_num = product_types_num\n",
    "        self.factory_stocks = np.zeros(\n",
    "            (self.product_types_num,),\n",
    "            dtype=np.int32)\n",
    "        self.distr_warehouses_num = distr_warehouses_num\n",
    "        self.distr_warehouses_stocks = np.zeros(\n",
    "            (self.distr_warehouses_num, self.product_types_num),\n",
    "            dtype=np.int32)\n",
    "        self.T = T\n",
    "        self.demand_history = demand_history\n",
    "        self.t = t\n",
    "\n",
    "        logger.debug(f\"\\n--- State --- __init__\"\n",
    "                     f\"\\nproduct_types_num is \"\n",
    "                     f\"{self.product_types_num}\"\n",
    "                     f\"\\nfactory_stocks is \"\n",
    "                     f\"{self.factory_stocks}\"\n",
    "                     f\"\\ndistr_warehouses_num is \"\n",
    "                     f\"{self.distr_warehouses_num}\"\n",
    "                     f\"\\ndistr_warehouses_stocks is \"\n",
    "                     f\"{self.distr_warehouses_stocks}\"\n",
    "                     f\"\\nT is \"\n",
    "                     f\"{self.T}\"\n",
    "                     f\"\\ndemand_history is \"\n",
    "                     f\"{self.demand_history}\"\n",
    "                     f\"\\nt is \"\n",
    "                     f\"{self.t}\")\n",
    "\n",
    "    def to_array(self):\n",
    "        logger.debug(f\"\\n--- State --- to_array\"\n",
    "                     f\"\\nnp.concatenate is \"\n",
    "                     f\"\"\"{np.concatenate((\n",
    "                         self.factory_stocks,\n",
    "                         self.distr_warehouses_stocks.flatten(),\n",
    "                         np.hstack(list(chain(*chain(*self.demand_history)))),\n",
    "                         [self.t]))}\"\"\")\n",
    "\n",
    "        return np.concatenate((\n",
    "            self.factory_stocks,\n",
    "            self.distr_warehouses_stocks.flatten(),\n",
    "            np.hstack(list(chain(*chain(*self.demand_history)))),\n",
    "            [self.t]))\n",
    "\n",
    "    def stock_levels(self):\n",
    "        logger.debug(f\"\\n--- State --- stock_levels\"\n",
    "                     f\"\\nnp.concatenate is \"\n",
    "                     f\"\"\"{np.concatenate((\n",
    "                         self.factory_stocks,\n",
    "                         self.distr_warehouses_stocks.flatten()))}\"\"\")\n",
    "\n",
    "        return np.concatenate((\n",
    "            self.factory_stocks,\n",
    "            self.distr_warehouses_stocks.flatten()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "C9uyi5YKzKkB"
   },
   "source": [
    "## Action Class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633592879156
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "3vmjf1wf6J6q"
   },
   "outputs": [],
   "source": [
    "class Action:\n",
    "    \"\"\"\n",
    "    The action vector consists of production and shipping controls.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, product_types_num, distr_warehouses_num):\n",
    "        self.production_level = np.zeros(\n",
    "            (product_types_num,),\n",
    "            dtype=np.int32)\n",
    "        self.shipped_stocks = np.zeros(\n",
    "            (distr_warehouses_num, product_types_num),\n",
    "            dtype=np.int32)\n",
    "\n",
    "        logger.debug(f\"\\n--- Action --- __init__\"\n",
    "                     f\"\\nproduction_level is \"\n",
    "                     f\"{self.production_level}\"\n",
    "                     f\"\\nshipped_stocks is \"\n",
    "                     f\"{self.shipped_stocks}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "dV01Of0pzOmi"
   },
   "source": [
    "## Supply Chain Environment Class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633592879433
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "Y6z3hOGrpWFn"
   },
   "outputs": [],
   "source": [
    "class SupplyChainEnvironment:\n",
    "    \"\"\"\n",
    "    We designed a divergent two-echelon supply chain that includes a single \n",
    "    factory, multiple distribution warehouses, and multiple product types over \n",
    "    a fixed number of time steps. At each time step, the agent is asked to find \n",
    "    the number of products to be produced and preserved at the factory, as well \n",
    "    as the number of products to be shipped to different distribution \n",
    "    warehouses. To make the supply chain more realistic, we set capacity \n",
    "    constraints on warehouses (and consequently, on how many units to produce \n",
    "    at the factory), along with storage and transportation costs. \n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self):\n",
    "        # number of product types (e.g., 2 product types)\n",
    "        self.product_types_num = 2\n",
    "        # number of distribution warehouses (e.g., 2 distribution warehouses)\n",
    "        self.distr_warehouses_num = 2\n",
    "        # final time step (e.g., an episode takes 25 time steps)\n",
    "        self.T = 25\n",
    "\n",
    "        # maximum demand value, units (e.g., [3, 6])\n",
    "        self.d_max = np.array(\n",
    "            [3, 6],\n",
    "            np.int32)\n",
    "        # maximum demand variation according to a uniform distribution,\n",
    "        # units (e.g., [2, 1])\n",
    "        self.d_var = np.array(\n",
    "            [2, 1],\n",
    "            np.int32)\n",
    "\n",
    "        # sale prices, per unit (e.g., [20, 10])\n",
    "        self.sale_prices = np.array(\n",
    "            [20, 10],\n",
    "            np.int32)\n",
    "        # production costs, per unit (e.g., [2, 1])\n",
    "        self.production_costs = np.array(\n",
    "            [2, 1],\n",
    "            np.int32)\n",
    "\n",
    "        # storage capacities for each product type at each warehouse,\n",
    "        # units (e.g., [[3, 4], [6, 8], [9, 12]])\n",
    "        self.storage_capacities = np.array(\n",
    "            [[3, 4], [6, 8], [9, 12]],\n",
    "            np.int32)\n",
    "\n",
    "        # storage costs of each product type at each warehouse,\n",
    "        # per unit (e.g., [[6, 3], [4, 2], [2, 1]])\n",
    "        self.storage_costs = np.array(\n",
    "            [[6, 3], [4, 2], [2, 1]],\n",
    "            np.float32)\n",
    "        # transportation costs of each product type for each distribution\n",
    "        # warehouse, per unit (e.g., [[.1, .3], [.2, .6]])\n",
    "        self.transportation_costs = np.array(\n",
    "            [[.1, .3], [.2, .6]],\n",
    "            np.float32)\n",
    "\n",
    "        # penalty costs, per unit (e.g., [10, 5])\n",
    "        self.penalty_costs = .5*self.sale_prices\n",
    "\n",
    "        print(f\"\\n--- SupplyChainEnvironment --- __init__\"\n",
    "              f\"\\nproduct_types_num is \"\n",
    "              f\"{self.product_types_num}\"\n",
    "              f\"\\ndistr_warehouses_num is \"\n",
    "              f\"{self.distr_warehouses_num}\"\n",
    "              f\"\\nT is \"\n",
    "              f\"{self.T}\"\n",
    "              f\"\\nd_max is \"\n",
    "              f\"{self.d_max}\"\n",
    "              f\"\\nd_var is \"\n",
    "              f\"{self.d_var}\"\n",
    "              f\"\\nsale_prices is \"\n",
    "              f\"{self.sale_prices}\"\n",
    "              f\"\\nproduction_costs is \"\n",
    "              f\"{self.production_costs}\"\n",
    "              f\"\\nstorage_capacities is \"\n",
    "              f\"{self.storage_capacities}\"\n",
    "              f\"\\nstorage_costs is \"\n",
    "              f\"{self.storage_costs}\"\n",
    "              f\"\\ntransportation_costs is \"\n",
    "              f\"{self.transportation_costs}\"\n",
    "              f\"\\npenalty_costs is \"\n",
    "              f\"{self.penalty_costs}\")\n",
    "\n",
    "        self.reset()\n",
    "\n",
    "    def reset(self, demand_history_len=5):\n",
    "        # (five) demand values observed\n",
    "        self.demand_history = collections.deque(maxlen=demand_history_len)\n",
    "\n",
    "        logger.debug(f\"\\n--- SupplyChainEnvironment --- reset\"\n",
    "                     f\"\\ndemand_history is \"\n",
    "                     f\"{self.demand_history}\")\n",
    "\n",
    "        for d in range(demand_history_len):\n",
    "            self.demand_history.append(np.zeros(\n",
    "                (self.distr_warehouses_num, self.product_types_num),\n",
    "                dtype=np.int32))\n",
    "        self.t = 0\n",
    "\n",
    "        logger.debug(f\"\\ndemand_history is \"\n",
    "                     f\"{self.demand_history}\"\n",
    "                     f\"\\nt is \"\n",
    "                     f\"{self.t}\")\n",
    "\n",
    "    def demand(self, j, i, t):\n",
    "        # we simulate a seasonal behavior by representing the demand as a\n",
    "        # co-sinusoidal function with a stochastic component (a random variable\n",
    "        # assumed to be distributed according to a uniform distribution),\n",
    "        # in order to evaluate the agent\n",
    "        demand = np.round(\n",
    "            self.d_max[i-1]/2 +\n",
    "            self.d_max[i-1]/2*np.cos(4*np.pi*(2*j*i+t)/self.T) +\n",
    "            np.random.randint(0, self.d_var[i-1]+1))\n",
    "\n",
    "        logger.debug(f\"\\n--- SupplyChainEnvironment --- demand\"\n",
    "                     f\"\\nj is \"\n",
    "                     f\"{j}\"\n",
    "                     f\"\\ni is \"\n",
    "                     f\"{i}\"\n",
    "                     f\"\\nt is \"\n",
    "                     f\"{t}\"\n",
    "                     f\"\\ndemand is \"\n",
    "                     f\"{demand}\")\n",
    "\n",
    "        return demand\n",
    "\n",
    "    def initial_state(self):\n",
    "        logger.debug(f\"\\n--- SupplyChainEnvironment --- initial_state\"\n",
    "                     f\"\\nState is \"\n",
    "                     f\"\"\"{State(\n",
    "                         self.product_types_num, self.distr_warehouses_num, \n",
    "                         self.T, list(self.demand_history))}\"\"\")\n",
    "\n",
    "        return State(self.product_types_num, self.distr_warehouses_num,\n",
    "                     self.T, list(self.demand_history))\n",
    "\n",
    "    def step(self, state, action):\n",
    "        demands = np.fromfunction(\n",
    "            lambda j, i: self.demand(j+1, i+1, self.t),\n",
    "            (self.distr_warehouses_num, self.product_types_num),\n",
    "            dtype=np.int32)\n",
    "\n",
    "        logger.debug(f\"\\n--- SupplyChainEnvironment --- step\"\n",
    "                     f\"\\nstate is \"\n",
    "                     f\"{state}\"\n",
    "                     f\"\\nstate.factory_stocks is \"\n",
    "                     f\"{state.factory_stocks}\"\n",
    "                     f\"\\nstate.distr_warehouses_stocks is \"\n",
    "                     f\"{state.distr_warehouses_stocks}\"\n",
    "                     f\"\\naction is \"\n",
    "                     f\"{action}\"\n",
    "                     f\"\\naction.production_level is \"\n",
    "                     f\"{action.production_level}\"\n",
    "                     f\"\\naction.shipped_stocks is \"\n",
    "                     f\"{action.shipped_stocks}\"\n",
    "                     f\"\\ndemands is \"\n",
    "                     f\"{demands}\")\n",
    "\n",
    "        # next state\n",
    "        next_state = State(self.product_types_num, self.distr_warehouses_num,\n",
    "                           self.T, list(self.demand_history))\n",
    "\n",
    "        next_state.factory_stocks = np.minimum(\n",
    "            np.subtract(np.add(state.factory_stocks,\n",
    "                               action.production_level),\n",
    "                        np.sum(action.shipped_stocks, axis=0)\n",
    "                        ),\n",
    "            self.storage_capacities[0]\n",
    "        )\n",
    "\n",
    "        for j in range(self.distr_warehouses_num):\n",
    "            next_state.distr_warehouses_stocks[j] = np.minimum(\n",
    "                np.subtract(np.add(state.distr_warehouses_stocks[j],\n",
    "                                   action.shipped_stocks[j]),\n",
    "                            demands[j]\n",
    "                            ),\n",
    "                self.storage_capacities[j+1]\n",
    "            )\n",
    "\n",
    "        logger.debug(f\"\\n-- SupplyChainEnvironment -- next state\"\n",
    "                     f\"\\nnext_state is \"\n",
    "                     f\"{next_state}\"\n",
    "                     f\"\\nnext_state.factory_stocks is \"\n",
    "                     f\"{next_state.factory_stocks}\"\n",
    "                     f\"\\nnext_state.distr_warehouses_stocks is \"\n",
    "                     f\"{next_state.distr_warehouses_stocks}\"\n",
    "                     f\"\\nnext_state.demand_history is \"\n",
    "                     f\"{next_state.demand_history}\"\n",
    "                     f\"\\nnext_state.t is \"\n",
    "                     f\"{next_state.t}\")\n",
    "\n",
    "        # revenues\n",
    "        total_revenues = np.dot(self.sale_prices,\n",
    "                                np.sum(demands, axis=0))\n",
    "        # production costs\n",
    "        total_production_costs = np.dot(self.production_costs,\n",
    "                                        action.production_level)\n",
    "        # transportation costs\n",
    "        total_transportation_costs = np.dot(\n",
    "            self.transportation_costs.flatten(),\n",
    "            action.shipped_stocks.flatten())\n",
    "        # storage costs\n",
    "        total_storage_costs = np.dot(\n",
    "            self.storage_costs.flatten(),\n",
    "            np.maximum(next_state.stock_levels(),\n",
    "                       np.zeros(\n",
    "                           ((self.distr_warehouses_num+1) *\n",
    "                            self.product_types_num),\n",
    "                           dtype=np.int32)\n",
    "                       )\n",
    "        )\n",
    "        # penalty costs (minus sign because stock levels would be already\n",
    "        # negative in case of unfulfilled demand)\n",
    "        total_penalty_costs = -np.dot(\n",
    "            self.penalty_costs,\n",
    "            np.add(\n",
    "                np.sum(\n",
    "                    np.minimum(next_state.distr_warehouses_stocks,\n",
    "                               np.zeros(\n",
    "                                   (self.distr_warehouses_num,\n",
    "                                    self.product_types_num),\n",
    "                                   dtype=np.int32)\n",
    "                               ),\n",
    "                    axis=0),\n",
    "                np.minimum(next_state.factory_stocks,\n",
    "                           np.zeros(\n",
    "                               (self.product_types_num,),\n",
    "                               dtype=np.int32)\n",
    "                           )\n",
    "            )\n",
    "        )\n",
    "        # reward function\n",
    "        reward = total_revenues - total_production_costs - \\\n",
    "            total_transportation_costs - total_storage_costs - \\\n",
    "            total_penalty_costs\n",
    "\n",
    "        logger.debug(f\"\\n-- SupplyChainEnvironment -- reward\"\n",
    "                     f\"\\ntotal_revenues is \"\n",
    "                     f\"{total_revenues}\"\n",
    "                     f\"\\ntotal_production_costs is \"\n",
    "                     f\"{total_production_costs}\"\n",
    "                     f\"\\ntotal_transportation_costs is \"\n",
    "                     f\"{total_transportation_costs}\"\n",
    "                     f\"\\ntotal_storage_costs is \"\n",
    "                     f\"{total_storage_costs}\"\n",
    "                     f\"\\ntotal_penalty_costs is \"\n",
    "                     f\"{total_penalty_costs}\"\n",
    "                     f\"\\nreward is \"\n",
    "                     f\"{reward}\")\n",
    "\n",
    "        # the actual demand for the current time step will not be known until\n",
    "        # the next time step. This implementation choice ensures that the agent\n",
    "        # may benefit from learning the demand pattern so as to integrate a\n",
    "        # sort of demand forecasting directly into the policy\n",
    "        self.demand_history.append(demands)\n",
    "        # actual time step value is not observed (for now)\n",
    "        self.t += 1\n",
    "\n",
    "        logger.debug(f\"\\ndemand_history is \"\n",
    "                     f\"{self.demand_history}\"\n",
    "                     f\"\\nt is \"\n",
    "                     f\"{self.t}\")\n",
    "\n",
    "        logger.debug(f\"\\n-- SupplyChainEnvironment -- return\"\n",
    "                     f\"\\nnext_state is \"\n",
    "                     f\"{next_state}, \"\n",
    "                     f\"\\nreward is \"\n",
    "                     f\"{reward}, \"\n",
    "                     f\"\\ndone is \"\n",
    "                     f\"{self.t == self.T-1}\")\n",
    "\n",
    "        return next_state, reward, self.t == self.T-1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "BLU1e41QzUfw"
   },
   "source": [
    "## Supply Chain Gym Wrapper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633592879761
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "k0LeYSn2pzHC"
   },
   "outputs": [],
   "source": [
    "class SupplyChain(gym.Env):\n",
    "    \"\"\"\n",
    "    Gym environment wrapper.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, config):\n",
    "        self.reset()\n",
    "\n",
    "        # low values for action space (no negative actions)\n",
    "        low_act = np.zeros(\n",
    "            ((self.supply_chain.distr_warehouses_num+1) *\n",
    "             self.supply_chain.product_types_num),\n",
    "            dtype=np.int32)\n",
    "        # high values for action space\n",
    "        high_act = np.zeros(\n",
    "            ((self.supply_chain.distr_warehouses_num+1) *\n",
    "             self.supply_chain.product_types_num),\n",
    "            dtype=np.int32)\n",
    "        # high values for action space (factory)\n",
    "        high_act[\n",
    "            :self.supply_chain.product_types_num\n",
    "        ] = np.sum(self.supply_chain.storage_capacities, axis=0)\n",
    "        # high values for action space (distribution warehouses, according to\n",
    "        # storage capacities)\n",
    "        high_act[\n",
    "            self.supply_chain.product_types_num:\n",
    "        ] = (self.supply_chain.storage_capacities.flatten()[\n",
    "            self.supply_chain.product_types_num:])\n",
    "        # action space\n",
    "        self.action_space = Box(low=low_act,\n",
    "                                high=high_act,\n",
    "                                dtype=np.int32)\n",
    "\n",
    "        # low values for observation space\n",
    "        low_obs = np.zeros(\n",
    "            (len(self.supply_chain.initial_state().to_array()),),\n",
    "            dtype=np.int32)\n",
    "        # low values for observation space (factory, worst case scenario in\n",
    "        # case of non-production and maximum demand)\n",
    "        low_obs[\n",
    "            :self.supply_chain.product_types_num\n",
    "        ] = -np.sum(self.supply_chain.storage_capacities[1:], axis=0) * \\\n",
    "            self.supply_chain.T\n",
    "        # low values for observation space (distribution warehouses, worst case\n",
    "        # scenario in case of non-shipments and maximum demand)\n",
    "        low_obs[\n",
    "            self.supply_chain.product_types_num:\n",
    "                (self.supply_chain.distr_warehouses_num+1) *\n",
    "            self.supply_chain.product_types_num\n",
    "        ] = np.array([\n",
    "            -(self.supply_chain.d_max+self.supply_chain.d_var) *\n",
    "            self.supply_chain.T\n",
    "        ] * self.supply_chain.distr_warehouses_num).flatten()\n",
    "        # high values for observation space\n",
    "        high_obs = np.zeros(\n",
    "            (len(self.supply_chain.initial_state().to_array()),),\n",
    "            dtype=np.int32)\n",
    "        # high values for observation space (factory and distribution\n",
    "        # warehouses, according to storage capacities)\n",
    "        high_obs[\n",
    "            :(self.supply_chain.distr_warehouses_num+1) *\n",
    "            self.supply_chain.product_types_num\n",
    "        ] = self.supply_chain.storage_capacities.flatten()\n",
    "        # high values for observation space (demand, according to the maximum\n",
    "        # demand value)\n",
    "        high_obs[\n",
    "            (self.supply_chain.distr_warehouses_num+1) *\n",
    "            self.supply_chain.product_types_num:\n",
    "            len(high_obs)-1\n",
    "        ] = np.array([\n",
    "            self.supply_chain.d_max+self.supply_chain.d_var] *\n",
    "            len(list(chain(*self.supply_chain.demand_history)))).flatten()\n",
    "        # high values for observation space (episode, according to the final\n",
    "        # time step)\n",
    "        high_obs[len(high_obs)-1] = self.supply_chain.T\n",
    "        # observation space\n",
    "        self.observation_space = Box(low=low_obs,\n",
    "                                     high=high_obs,\n",
    "                                     dtype=np.int32)\n",
    "\n",
    "        logger.debug(f\"\\n--- SupplyChain --- __init__\"\n",
    "                     f\"\\nlow_act is \"\n",
    "                     f\"{low_act}\"\n",
    "                     f\"\\nhigh_act is \"\n",
    "                     f\"{high_act}\"\n",
    "                     f\"\\naction_space is \"\n",
    "                     f\"{self.action_space}\"\n",
    "                     f\"\\nlow_obs is \"\n",
    "                     f\"{low_obs}\"\n",
    "                     f\"\\nhigh_obs is \"\n",
    "                     f\"{high_obs}\"\n",
    "                     f\"\\nobservation_space is \"\n",
    "                     f\"{self.observation_space}\")\n",
    "\n",
    "    def reset(self):\n",
    "        self.supply_chain = SupplyChainEnvironment()\n",
    "        self.state = self.supply_chain.initial_state()\n",
    "\n",
    "        logger.debug(f\"\\n--- SupplyChain --- reset\"\n",
    "                     f\"\\nsupply_chain is \"\n",
    "                     f\"{self.supply_chain}\"\n",
    "                     f\"\\nstate is \"\n",
    "                     f\"{self.state}\"\n",
    "                     f\"\\nstate.to_array is \"\n",
    "                     f\"{self.state.to_array()}\")\n",
    "\n",
    "        return self.state.to_array()\n",
    "\n",
    "    def step(self, action):\n",
    "        # casting to integer actions (units of product to produce and ship)\n",
    "        action_obj = Action(\n",
    "            self.supply_chain.product_types_num,\n",
    "            self.supply_chain.distr_warehouses_num)\n",
    "        action_obj.production_level = action[\n",
    "            :self.supply_chain.product_types_num].astype(np.int32)\n",
    "        action_obj.shipped_stocks = action[\n",
    "            self.supply_chain.product_types_num:\n",
    "        ].reshape((self.supply_chain.distr_warehouses_num,\n",
    "                   self.supply_chain.product_types_num)).astype(np.int32)\n",
    "\n",
    "        logger.debug(f\"\\n--- SupplyChain --- step\"\n",
    "                     f\"\\naction is \"\n",
    "                     f\"{action}\"\n",
    "                     f\"\\naction_obj is \"\n",
    "                     f\"{action_obj}\"\n",
    "                     f\"\\naction_obj.production_level is \"\n",
    "                     f\"{action_obj.production_level}\"\n",
    "                     f\"\\naction_obj.shipped_stocks is \"\n",
    "                     f\"{action_obj.shipped_stocks}\")\n",
    "\n",
    "        self.state, reward, done = self.supply_chain.step(\n",
    "            self.state, action_obj)\n",
    "\n",
    "        logger.debug(f\"\\n-- SupplyChain -- return\"\n",
    "                     f\"\\nstate.to_array is \"\n",
    "                     f\"{self.state.to_array()}\"\n",
    "                     f\"\\nreward is \"\n",
    "                     f\"{reward}\"\n",
    "                     f\"\\ndone is \"\n",
    "                     f\"{done}\")\n",
    "\n",
    "        return self.state.to_array(), reward, done, {}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "F7PtD6l1jsry"
   },
   "source": [
    "# Global Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633592880059
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "AYTndqzPjsry"
   },
   "outputs": [],
   "source": [
    "# number of episodes for the simulations\n",
    "num_episodes = 200"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633592880368
    },
    "hideCode": true,
    "hidePrompt": true
   },
   "outputs": [],
   "source": [
    "# name of the experiment (e.g., '2P2W' stands for two product types and two\n",
    "# distribution warehouses)\n",
    "now = datetime.now()\n",
    "now_str = now.strftime('%Y-%m-%d_%H-%M-%S')\n",
    "local_dir = f\"2P2W_{now_str}\"\n",
    "# dir to save plots\n",
    "plots_dir = 'plots'\n",
    "# creating necessary dirs\n",
    "if not os.path.exists(f\"{local_dir}\"):\n",
    "    os.makedirs(f\"{local_dir}\")\n",
    "if not os.path.exists(f\"{local_dir+'/'+plots_dir}\"):\n",
    "    os.makedirs(f\"{local_dir+'/'+plots_dir}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "dWCGN9yMz8T_"
   },
   "source": [
    "# Supply Chain Environment Initialization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "b3FX1yqx5j1r"
   },
   "source": [
    "## Visualize Demand Methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633592880660
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "lcYcO8UR3yLs"
   },
   "outputs": [],
   "source": [
    "def visualize_demand(num_episodes=1, local_dir=local_dir, plots_dir=plots_dir):\n",
    "    \"\"\"\n",
    "    Visualize demand behavior for each distribution warehouses and for each \n",
    "    product type.\n",
    "    \"\"\"\n",
    "    if env.distr_warehouses_num <= 3 and env.product_types_num <= 2:\n",
    "        demands = []\n",
    "        for n in range(num_episodes):\n",
    "            # generating demands\n",
    "            demands.append(np.fromfunction(\n",
    "                lambda j, i, t: env.demand(j+1, i+1, t),\n",
    "                (env.distr_warehouses_num, env.product_types_num, env.T),\n",
    "                dtype=np.int32))\n",
    "\n",
    "        # mean of demands\n",
    "        demands_mean = np.array([np.mean(demand, axis=0)\n",
    "                                for demand in zip(*demands)])\n",
    "        # std of demands\n",
    "        demands_std = np.array([np.std(demand, axis=0)\n",
    "                               for demand in zip(*demands)])\n",
    "\n",
    "        logger.debug(f\"\\n-- visualize_demand --\"\n",
    "                     f\"\\ndemands is \"\n",
    "                     f\"{demands}\"\n",
    "                     f\"\\ndemands_mean is \"\n",
    "                     f\"{demands_mean}\"\n",
    "                     f\"\\ndemands_std is \"\n",
    "                     f\"{demands_std}\")\n",
    "\n",
    "        plt.figure(figsize=(15, 5))\n",
    "        plt.xlabel('Time Steps')\n",
    "        plt.ylabel('Demand Value')\n",
    "\n",
    "        plt.xticks(np.arange(min(range(env.T)),\n",
    "                             max(range(env.T))+1))\n",
    "        plt.tick_params(axis='x', which='both',\n",
    "                        top=True, bottom=True,\n",
    "                        labelbottom=True)\n",
    "        plt.ticklabel_format(axis='y', style='plain',\n",
    "                             useOffset=False)\n",
    "        plt.tight_layout()\n",
    "\n",
    "        # same color for the same distribution warehouse, but different line\n",
    "        # style according to the different product type\n",
    "        color = [['b', 'b'], ['g', 'g'], ['r', 'r']]\n",
    "        line_style = [['b-', 'b--'], ['g-', 'g--'], ['r-', 'r--']]\n",
    "\n",
    "        for j in range(env.distr_warehouses_num):\n",
    "            for i in range(env.product_types_num):\n",
    "                plt.plot(range(env.T),\n",
    "                         demands_mean[j][i].T,\n",
    "                         line_style[j][i])\n",
    "                plt.fill_between(range(env.T),\n",
    "                                 (demands_mean[j][i] -\n",
    "                                  demands_std[j][i]).flatten(),\n",
    "                                 (demands_mean[j][i] +\n",
    "                                  demands_std[j][i]).flatten(),\n",
    "                                 color=color[j][i], alpha=.2)\n",
    "\n",
    "        # plotting legend\n",
    "        plt.legend([f\"WH {j+1}, Prod {i+1}\"\n",
    "                    for j in range(env.distr_warehouses_num)\n",
    "                    for i in range(env.product_types_num)])\n",
    "\n",
    "        # saving plot\n",
    "        plt.savefig(f\"{local_dir}/{plots_dir}\"\n",
    "                    f\"/demand.pdf\",\n",
    "                    format='pdf', bbox_inches='tight')\n",
    "\n",
    "\n",
    "def save_env_settings(env, local_dir=local_dir, plots_dir=plots_dir):\n",
    "    \"\"\"\n",
    "    Save the Supply Chain Environment settings.\n",
    "    \"\"\"\n",
    "    f = open(f\"{local_dir}/{plots_dir}\"\n",
    "             f\"/env_settings.txt\",\n",
    "             'w', encoding='utf-8')\n",
    "    f.write(f\"--- SupplyChainEnvironment ---\"\n",
    "            f\"\\nproduct_types_num is \"\n",
    "            f\"{env.product_types_num}\"\n",
    "            f\"\\ndistr_warehouses_num is \"\n",
    "            f\"{env.distr_warehouses_num}\"\n",
    "            f\"\\nT is \"\n",
    "            f\"{env.T}\"\n",
    "            f\"\\nd_max is \"\n",
    "            f\"{env.d_max}\"\n",
    "            f\"\\nd_var is \"\n",
    "            f\"{env.d_var}\"\n",
    "            f\"\\nsale_prices is \"\n",
    "            f\"{env.sale_prices}\"\n",
    "            f\"\\nproduction_costs is \"\n",
    "            f\"{env.production_costs}\"\n",
    "            f\"\\nstorage_capacities is \"\n",
    "            f\"{env.storage_capacities}\"\n",
    "            f\"\\nstorage_costs is \"\n",
    "            f\"{env.storage_costs}\"\n",
    "            f\"\\ntransportation_costs is \"\n",
    "            f\"{env.transportation_costs}\"\n",
    "            f\"\\npenalty_costs is \"\n",
    "            f\"{env.penalty_costs}\")\n",
    "    f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "Lk4KTfmv0H30"
   },
   "source": [
    "## Initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633592881005
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "ZAIMz3nhsb0Z"
   },
   "outputs": [],
   "source": [
    "# supply chain env\n",
    "env = SupplyChainEnvironment()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633592881600
    },
    "hideCode": true,
    "hidePrompt": true,
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "# saving env settings\n",
    "save_env_settings(env)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633592884329
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "s_E3N5X5Kq49",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "visualize_demand(num_episodes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "OrcmncGdCLWl"
   },
   "source": [
    "# Methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "KB5DGFSvjsrz"
   },
   "source": [
    "## Simulator Methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633592885165
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "1RF08jdMjsrz"
   },
   "outputs": [],
   "source": [
    "def simulate_oracle(env, num_episodes):\n",
    "    \"\"\"\n",
    "    Oracle simulator.\n",
    "    \"\"\"\n",
    "    rewards_trace = []\n",
    "    for n in range(num_episodes):\n",
    "        # generating demands (oracle knows the episodes' demands a priori)\n",
    "        demands = []\n",
    "        demands = np.fromfunction(\n",
    "            lambda j, i, t: env.demand(j+1, i+1, t),\n",
    "            (env.distr_warehouses_num, env.product_types_num, env.T),\n",
    "            dtype=np.int32)\n",
    "\n",
    "        # calculating demands for each product type\n",
    "        demands_product_types = np.sum(np.sum(demands,\n",
    "                                              axis=0),\n",
    "                                       axis=1)\n",
    "        # calculating demands of each product type for each warehouse\n",
    "        demands_distr_warehouses = np.sum(demands,\n",
    "                                          axis=2)\n",
    "\n",
    "        # revenues\n",
    "        total_revenues = np.dot(env.sale_prices,\n",
    "                                demands_product_types)\n",
    "        # production costs\n",
    "        total_production_costs = np.dot(env.production_costs,\n",
    "                                        demands_product_types)\n",
    "        # transportation costs\n",
    "        total_transportation_costs = np.dot(\n",
    "            env.transportation_costs.flatten(),\n",
    "            demands_distr_warehouses.flatten())\n",
    "        # reward\n",
    "        reward = total_revenues - total_production_costs - \\\n",
    "            total_transportation_costs\n",
    "        # append reward for each episode\n",
    "        rewards_trace.append(reward)\n",
    "\n",
    "        logger.debug(f\"\\n-- simulate_oracle --\"\n",
    "                     f\"\\ndemands is \"\n",
    "                     f\"{demands}\"\n",
    "                     f\"\\ndemands_product_types is \"\n",
    "                     f\"{demands_product_types}\"\n",
    "                     f\"\\ndemands_distr_warehouses is \"\n",
    "                     f\"{demands_distr_warehouses}\"\n",
    "                     f\"\\ntotal_revenues is \"\n",
    "                     f\"{total_revenues}\"\n",
    "                     f\"\\ntotal_production_costs is \"\n",
    "                     f\"{total_production_costs}\"\n",
    "                     f\"\\ntotal_transportation_costs is \"\n",
    "                     f\"{total_transportation_costs}\"\n",
    "                     f\"\\nreward is \"\n",
    "                     f\"{reward}\")\n",
    "\n",
    "    print(f\"reward: mean \"\n",
    "          f\"{np.mean(rewards_trace)}, \"\n",
    "          f\"std \"\n",
    "          f\"{np.std(rewards_trace)}, \"\n",
    "          f\"max \"\n",
    "          f\"{np.max(rewards_trace)}, \"\n",
    "          f\"min \"\n",
    "          f\"{np.min(rewards_trace)}\")\n",
    "\n",
    "    return np.array(rewards_trace)\n",
    "\n",
    "\n",
    "def simulate_episode(env, policy):\n",
    "    \"\"\"\n",
    "    Single episode simulator.\n",
    "    \"\"\"\n",
    "    env.reset()\n",
    "    state = env.initial_state()\n",
    "    transitions = []\n",
    "\n",
    "    logger.debug(f\"\\n-- simulate_episode --\"\n",
    "                 f\"\\nstate is \"\n",
    "                 f\"{state}\"\n",
    "                 f\"\\ntransitions is \"\n",
    "                 f\"{transitions}\")\n",
    "\n",
    "    if not isinstance(policy, SQPolicy):\n",
    "        for t in range(env.T):\n",
    "            action = policy.compute_single_action(state.to_array(),\n",
    "                                                  normalize_actions=True,\n",
    "                                                  explore=False)[0].astype(\n",
    "                np.int32)\n",
    "            action_obj = Action(env.product_types_num,\n",
    "                                env.distr_warehouses_num)\n",
    "            action_obj.production_level = action[:env.product_types_num]\n",
    "            action_obj.shipped_stocks = action[\n",
    "                env.product_types_num:\n",
    "            ].reshape((env.distr_warehouses_num, env.product_types_num))\n",
    "\n",
    "            state, reward, _ = env.step(state, action_obj)\n",
    "            transitions.append(np.array(\n",
    "                [state, action_obj, reward],\n",
    "                dtype=object))\n",
    "\n",
    "            logger.debug(f\"\\naction is \"\n",
    "                         f\"{action}\"\n",
    "                         f\"\\naction_obj is \"\n",
    "                         f\"{action_obj}\"\n",
    "                         f\"\\naction_obj.production_level is \"\n",
    "                         f\"{action_obj.production_level}\"\n",
    "                         f\"\\naction_obj.shipped_stocks is \"\n",
    "                         f\"{action_obj.shipped_stocks}\"\n",
    "                         f\"\\nstate is \"\n",
    "                         f\"{state}\"\n",
    "                         f\"\\nstate.factory_stocks is \"\n",
    "                         f\"{state.factory_stocks}\"\n",
    "                         f\"\\nstate.distr_warehouses_stocks is \"\n",
    "                         f\"{state.distr_warehouses_stocks}\"\n",
    "                         f\"\\nstate.demand_history is \"\n",
    "                         f\"{state.demand_history}\"\n",
    "                         f\"\\nt is \"\n",
    "                         f\"{t}\"\n",
    "                         f\"\\nreward is \"\n",
    "                         f\"{reward}\")\n",
    "    else:\n",
    "        for t in range(env.T):\n",
    "            action = policy.select_action(state)\n",
    "            state, reward, _ = env.step(state, action)\n",
    "            transitions.append(np.array(\n",
    "                [state, action, reward],\n",
    "                dtype=object))\n",
    "\n",
    "            logger.debug(f\"\\naction is \"\n",
    "                         f\"{action}\"\n",
    "                         f\"\\naction.production_level is \"\n",
    "                         f\"{action.production_level}\"\n",
    "                         f\"\\naction.shipped_stocks is \"\n",
    "                         f\"{action.shipped_stocks}\"\n",
    "                         f\"\\nstate is \"\n",
    "                         f\"{state}\"\n",
    "                         f\"\\nstate.factory_stocks is \"\n",
    "                         f\"{state.factory_stocks}\"\n",
    "                         f\"\\nstate.distr_warehouses_stocks is \"\n",
    "                         f\"{state.distr_warehouses_stocks}\"\n",
    "                         f\"\\nstate.demand_history is \"\n",
    "                         f\"{state.demand_history}\"\n",
    "                         f\"\\nt is \"\n",
    "                         f\"{t}\"\n",
    "                         f\"\\nreward is \"\n",
    "                         f\"{reward}\")\n",
    "\n",
    "    logger.debug(f\"\\ntransitions [state, action, reward] is \"\n",
    "                 f\"{transitions}\")\n",
    "\n",
    "    return transitions\n",
    "\n",
    "\n",
    "def simulate(env, policy, num_episodes=1):\n",
    "    \"\"\"\n",
    "    Simulator.\n",
    "    \"\"\"\n",
    "    returns_trace = []\n",
    "\n",
    "    if not isinstance(policy, SQPolicy):\n",
    "        # initializing Ray\n",
    "        ray.shutdown()\n",
    "        ray.init(log_to_driver=False)\n",
    "\n",
    "    for episode in range(num_episodes):\n",
    "        returns_trace.append(np.array(\n",
    "            simulate_episode(env, policy)))\n",
    "\n",
    "    if not isinstance(policy, SQPolicy):\n",
    "        # stopping Ray\n",
    "        ray.shutdown()\n",
    "\n",
    "    logger.debug(f\"\\n-- simulate --\"\n",
    "                 f\"\\nreturns_trace is \"\n",
    "                 f\"{returns_trace}\")\n",
    "\n",
    "    return returns_trace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "iVWW73Ge5_Gy"
   },
   "source": [
    "## Visualize Transitions Methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633592885854
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "RVJAV_Pj3yLt"
   },
   "outputs": [],
   "source": [
    "def prepare_metric_plot(ylabel, n,\n",
    "                        plots_n=10 if env.product_types_num == 1 else 26):\n",
    "    \"\"\"\n",
    "    Auxiliary function.\n",
    "    \"\"\"\n",
    "    plt.subplot(plots_n, 1, n)\n",
    "    plt.ylabel(ylabel, fontsize=10)\n",
    "\n",
    "    plt.xticks(np.arange(min(range(env.T)),\n",
    "                         max(range(env.T))+1))\n",
    "    plt.tick_params(axis='x', which='both',\n",
    "                    top=True, bottom=True,\n",
    "                    labelbottom=False)\n",
    "\n",
    "\n",
    "def visualize_transitions(returns_trace, algorithm,\n",
    "                          local_dir=local_dir, plots_dir=plots_dir):\n",
    "    \"\"\"\n",
    "    Visualize transitions (stock levels, production and shipping controls,\n",
    "    reward) along the episodes.\n",
    "    \"\"\"\n",
    "    if env.distr_warehouses_num <= 3 and env.product_types_num <= 2:\n",
    "        transitions = np.array(\n",
    "            [(return_trace)\n",
    "             for return_trace in zip(*returns_trace)])\n",
    "        states_trace, actions_trace, rewards_trace = (transitions.T[0],\n",
    "                                                      transitions.T[1],\n",
    "                                                      transitions.T[2])\n",
    "\n",
    "        logger.debug(f\"\\n-- visualize_transitions --\"\n",
    "                     f\"\\nstates_trace is \"\n",
    "                     f\"{states_trace}\"\n",
    "                     f\"\\nactions_trace is \"\n",
    "                     f\"{actions_trace}\"\n",
    "                     f\"\\nrewards_trace is \"\n",
    "                     f\"{rewards_trace}\")\n",
    "\n",
    "        plt.figure(figsize=(10, 30))\n",
    "\n",
    "        # states transitions\n",
    "        states = np.array(\n",
    "            [(state_trace)\n",
    "             for state_trace in zip(*states_trace)])\n",
    "\n",
    "        logger.debug(f\"\\nstates is \"\n",
    "                     f\"{states}\")\n",
    "\n",
    "        # factory stocks\n",
    "        prepare_metric_plot('Stocks,\\nFactory',\n",
    "                            1)\n",
    "        tmp_mean = []\n",
    "        for t in range(len(states)):\n",
    "            tmp_mean.append(\n",
    "                np.mean(\n",
    "                    [np.sum(state.factory_stocks)\n",
    "                     for state in states[t]], axis=0))\n",
    "        tmp_std = []\n",
    "        for t in range(len(states)):\n",
    "            tmp_std.append(\n",
    "                np.std(\n",
    "                    [np.sum(state.factory_stocks)\n",
    "                     for state in states[t]], axis=0))\n",
    "        plt.plot(range(env.T),\n",
    "                 tmp_mean,\n",
    "                 color='purple', alpha=.5)\n",
    "        plt.fill_between(range(env.T),\n",
    "                         list(np.array(tmp_mean) -\n",
    "                              np.array(tmp_std)),\n",
    "                         list(np.array(tmp_mean) +\n",
    "                              np.array(tmp_std)),\n",
    "                         color='purple', alpha=.2)\n",
    "\n",
    "        logger.debug(f\"\\nfactory_stocks (mean) is \"\n",
    "                     f\"{tmp_mean}\"\n",
    "                     f\"\\nfactory_stocks (std) is \"\n",
    "                     f\"{tmp_std}\")\n",
    "\n",
    "        if env.product_types_num >= 2:\n",
    "            for i in range(env.product_types_num):\n",
    "                prepare_metric_plot(f\"Stocks,\\nFactory,\\nProd {i+1}\",\n",
    "                                    2+i)\n",
    "                tmp_mean = []\n",
    "                for t in range(len(states)):\n",
    "                    tmp_mean.append(\n",
    "                        np.mean(\n",
    "                            [np.sum(state.factory_stocks[i])\n",
    "                             for state in states[t]], axis=0))\n",
    "                tmp_std = []\n",
    "                for t in range(len(states)):\n",
    "                    tmp_std.append(\n",
    "                        np.std(\n",
    "                            [np.sum(state.factory_stocks[i])\n",
    "                             for state in states[t]], axis=0))\n",
    "                plt.plot(range(env.T),\n",
    "                         tmp_mean,\n",
    "                         '--',\n",
    "                         color='purple', alpha=.5)\n",
    "                plt.fill_between(range(env.T),\n",
    "                                 list(np.array(tmp_mean) -\n",
    "                                      np.array(tmp_std)),\n",
    "                                 list(np.array(tmp_mean) +\n",
    "                                      np.array(tmp_std)),\n",
    "                                 color='purple', alpha=.2)\n",
    "\n",
    "        logger.debug(f\"\\nfactory_stocks (mean for product) is \"\n",
    "                     f\"{tmp_mean}\"\n",
    "                     f\"\\nfactory_stocks (std for product) is \"\n",
    "                     f\"{tmp_std}\")\n",
    "\n",
    "        # distribution warehouses stocks\n",
    "        for j in range(env.distr_warehouses_num):\n",
    "            prepare_metric_plot(f\"Stocks,\\nWH {j+1}\",\n",
    "                                2*env.product_types_num+j)\n",
    "            tmp_mean = []\n",
    "            for t in range(len(states)):\n",
    "                tmp_mean.append(\n",
    "                    np.mean(\n",
    "                        [np.sum(state.distr_warehouses_stocks[j])\n",
    "                         for state in states[t]], axis=0))\n",
    "            tmp_std = []\n",
    "            for t in range(len(states)):\n",
    "                tmp_std.append(\n",
    "                    np.std(\n",
    "                        [np.sum(state.distr_warehouses_stocks[j])\n",
    "                         for state in states[t]], axis=0))\n",
    "            plt.plot(range(env.T),\n",
    "                     tmp_mean,\n",
    "                     color='purple', alpha=.5)\n",
    "            plt.fill_between(range(env.T),\n",
    "                             list(np.array(tmp_mean) -\n",
    "                                  np.array(tmp_std)),\n",
    "                             list(np.array(tmp_mean) +\n",
    "                                  np.array(tmp_std)),\n",
    "                             color='purple', alpha=.2)\n",
    "\n",
    "        logger.debug(f\"\\ndistr_warehouses_stocks (mean) is \"\n",
    "                     f\"{tmp_mean}\"\n",
    "                     f\"\\ndistr_warehouses_stocks (std) is \"\n",
    "                     f\"{tmp_std}\")\n",
    "\n",
    "        if env.product_types_num >= 2:\n",
    "            cont = 0\n",
    "            for j in range(env.distr_warehouses_num):\n",
    "                for i in range(env.product_types_num):\n",
    "                    prepare_metric_plot(f\"Stocks,\\nWH {j+1},\\nProd {i+1}\",\n",
    "                                        4+env.distr_warehouses_num+cont)\n",
    "                    tmp_mean = []\n",
    "                    for t in range(len(states)):\n",
    "                        tmp_mean.append(\n",
    "                            np.mean(\n",
    "                                [np.sum(state.distr_warehouses_stocks[j][i])\n",
    "                                 for state in states[t]], axis=0))\n",
    "                    tmp_std = []\n",
    "                    for t in range(len(states)):\n",
    "                        tmp_std.append(\n",
    "                            np.std(\n",
    "                                [np.sum(state.distr_warehouses_stocks[j][i])\n",
    "                                 for state in states[t]], axis=0))\n",
    "                    plt.plot(range(env.T),\n",
    "                             tmp_mean,\n",
    "                             '--',\n",
    "                             color='purple', alpha=.5)\n",
    "                    plt.fill_between(range(env.T),\n",
    "                                     list(np.array(tmp_mean) -\n",
    "                                          np.array(tmp_std)),\n",
    "                                     list(np.array(tmp_mean) +\n",
    "                                          np.array(tmp_std)),\n",
    "                                     color='purple', alpha=.2)\n",
    "                    cont += 1\n",
    "\n",
    "        logger.debug(f\"\\ndistr_warehouses_stocks (mean for product) is \"\n",
    "                     f\"{tmp_mean}\"\n",
    "                     f\"\\ndistr_warehouses_stocks (std for product) is \"\n",
    "                     f\"{tmp_std}\")\n",
    "\n",
    "        # actions (transitions)\n",
    "        actions = np.array(\n",
    "            [(action_trace)\n",
    "             for action_trace in zip(*actions_trace)])\n",
    "\n",
    "        logger.debug(f\"\\nactions is \"\n",
    "                     f\"{actions}\")\n",
    "\n",
    "        # production level\n",
    "        prepare_metric_plot('Production,\\nFactory',\n",
    "                            2+env.distr_warehouses_num if\n",
    "                            env.product_types_num == 1\n",
    "                            else\n",
    "                            4+env.distr_warehouses_num +\n",
    "                            env.distr_warehouses_num *\n",
    "                            env.product_types_num)\n",
    "        tmp_mean = []\n",
    "        for t in range(len(actions)):\n",
    "            tmp_mean.append(\n",
    "                np.mean(\n",
    "                    [np.sum(action.production_level)\n",
    "                     for action in actions[t]], axis=0))\n",
    "        tmp_std = []\n",
    "        for t in range(len(actions)):\n",
    "            tmp_std.append(\n",
    "                np.std(\n",
    "                    [np.sum(action.production_level)\n",
    "                     for action in actions[t]], axis=0))\n",
    "        plt.plot(range(env.T),\n",
    "                 tmp_mean,\n",
    "                 color='blue', alpha=.5)\n",
    "        plt.fill_between(range(env.T),\n",
    "                         list(np.array(tmp_mean) -\n",
    "                              np.array(tmp_std)),\n",
    "                         list(np.array(tmp_mean) +\n",
    "                              np.array(tmp_std)),\n",
    "                         color='blue', alpha=.2)\n",
    "\n",
    "        logger.debug(f\"\\nproduction_level (mean) is \"\n",
    "                     f\"{tmp_mean}\"\n",
    "                     f\"\\nproduction_level (std) is \"\n",
    "                     f\"{tmp_std}\")\n",
    "\n",
    "        if env.product_types_num >= 2:\n",
    "            for i in range(env.product_types_num):\n",
    "                prepare_metric_plot(f\"Production,\\nFactory,\\nProd {i+1}\",\n",
    "                                    5+env.distr_warehouses_num +\n",
    "                                    env.distr_warehouses_num *\n",
    "                                    env.product_types_num+i)\n",
    "                tmp_mean = []\n",
    "                for t in range(len(actions)):\n",
    "                    tmp_mean.append(\n",
    "                        np.mean(\n",
    "                            [np.sum(action.production_level[i])\n",
    "                             for action in actions[t]], axis=0))\n",
    "                tmp_std = []\n",
    "                for t in range(len(actions)):\n",
    "                    tmp_std.append(\n",
    "                        np.std(\n",
    "                            [np.sum(action.production_level[i])\n",
    "                             for action in actions[t]], axis=0))\n",
    "                plt.plot(range(env.T),\n",
    "                         tmp_mean,\n",
    "                         '--',\n",
    "                         color='blue', alpha=.5)\n",
    "                plt.fill_between(range(env.T),\n",
    "                                 list(np.array(tmp_mean) -\n",
    "                                      np.array(tmp_std)),\n",
    "                                 list(np.array(tmp_mean) +\n",
    "                                      np.array(tmp_std)),\n",
    "                                 color='blue', alpha=.2)\n",
    "\n",
    "        logger.debug(f\"\\nproduction_level (mean for product) is \"\n",
    "                     f\"{tmp_mean}\"\n",
    "                     f\"\\nproduction_level (std for product) is \"\n",
    "                     f\"{tmp_std}\")\n",
    "\n",
    "        # shipped stocks\n",
    "        for j in range(env.distr_warehouses_num):\n",
    "            prepare_metric_plot(f\"Shipments,\\nWH {j+1}\",\n",
    "                                3+env.distr_warehouses_num+j\n",
    "                                if env.product_types_num == 1\n",
    "                                else\n",
    "                                7+env.distr_warehouses_num +\n",
    "                                env.distr_warehouses_num *\n",
    "                                env.product_types_num+j)\n",
    "            tmp_mean = []\n",
    "            for t in range(len(actions)):\n",
    "                tmp_mean.append(\n",
    "                    np.mean(\n",
    "                        [np.sum(action.shipped_stocks[j])\n",
    "                         for action in actions[t]], axis=0))\n",
    "            tmp_std = []\n",
    "            for t in range(len(actions)):\n",
    "                tmp_std.append(\n",
    "                    np.std(\n",
    "                        [np.sum(action.shipped_stocks[j])\n",
    "                         for action in actions[t]], axis=0))\n",
    "            plt.plot(range(env.T),\n",
    "                     tmp_mean,\n",
    "                     color='blue', alpha=.5)\n",
    "            plt.fill_between(range(env.T),\n",
    "                             list(np.array(tmp_mean) -\n",
    "                                  np.array(tmp_std)),\n",
    "                             list(np.array(tmp_mean) +\n",
    "                                  np.array(tmp_std)),\n",
    "                             color='blue', alpha=.2)\n",
    "\n",
    "        logger.debug(f\"\\nshipped_stocks (mean) is \"\n",
    "                     f\"{tmp_mean}\"\n",
    "                     f\"\\nshipped_stocks (std) is \"\n",
    "                     f\"{tmp_std}\")\n",
    "\n",
    "        if env.product_types_num >= 2:\n",
    "            cont = 0\n",
    "            for j in range(env.distr_warehouses_num):\n",
    "                for i in range(env.product_types_num):\n",
    "                    prepare_metric_plot(f\"Shipments,\\nWH {j+1},\\nProd {i+1}\",\n",
    "                                        7+(2*env.distr_warehouses_num) +\n",
    "                                        env.distr_warehouses_num *\n",
    "                                        env.product_types_num+cont)\n",
    "                    tmp_mean = []\n",
    "                    for t in range(len(actions)):\n",
    "                        tmp_mean.append(\n",
    "                            np.mean(\n",
    "                                [np.sum(action.shipped_stocks[j][i])\n",
    "                                 for action in actions[t]], axis=0))\n",
    "                    tmp_std = []\n",
    "                    for t in range(len(actions)):\n",
    "                        tmp_std.append(\n",
    "                            np.std(\n",
    "                                [np.sum(action.shipped_stocks[j][i])\n",
    "                                 for action in actions[t]], axis=0))\n",
    "                    plt.plot(range(env.T),\n",
    "                             tmp_mean,\n",
    "                             '--',\n",
    "                             color='blue', alpha=.5)\n",
    "                    plt.fill_between(range(env.T),\n",
    "                                     list(np.array(tmp_mean) -\n",
    "                                          np.array(tmp_std)),\n",
    "                                     list(np.array(tmp_mean) +\n",
    "                                          np.array(tmp_std)),\n",
    "                                     color='blue', alpha=.2)\n",
    "                    cont += 1\n",
    "\n",
    "        logger.debug(f\"\\nshipped_stocks (mean for product) is \"\n",
    "                     f\"{tmp_mean}\"\n",
    "                     f\"\\nshipped_stocks (std for product) is \"\n",
    "                     f\"{tmp_std}\")\n",
    "\n",
    "        # profit\n",
    "        prepare_metric_plot('Profit',\n",
    "                            3+(2*env.distr_warehouses_num)\n",
    "                            if env.product_types_num == 1\n",
    "                            else\n",
    "                            7+(2*env.distr_warehouses_num) +\n",
    "                            2*env.distr_warehouses_num*env.product_types_num)\n",
    "        reward_mean = np.array(\n",
    "            np.mean(rewards_trace, axis=0),\n",
    "            dtype=np.int32)\n",
    "        reward_std = np.array(\n",
    "            np.std(rewards_trace.astype(np.int32), axis=0),\n",
    "            dtype=np.int32)\n",
    "        plt.plot(range(env.T),\n",
    "                 reward_mean,\n",
    "                 linewidth=2,\n",
    "                 color='red', alpha=.5)\n",
    "        plt.fill_between(range(env.T),\n",
    "                         reward_mean -\n",
    "                         reward_std,\n",
    "                         reward_mean +\n",
    "                         reward_std,\n",
    "                         color='red', alpha=.2)\n",
    "\n",
    "        logger.debug(f\"\\nprofit (mean) is \"\n",
    "                     f\"{reward_mean}\"\n",
    "                     f\"\\nprofit (std) is \"\n",
    "                     f\"{reward_std}\")\n",
    "\n",
    "        # cumulative profit\n",
    "        prepare_metric_plot('Cum\\nProfit',\n",
    "                            4+(2*env.distr_warehouses_num)\n",
    "                            if env.product_types_num == 1\n",
    "                            else\n",
    "                            8+(2*env.distr_warehouses_num) +\n",
    "                            2*env.distr_warehouses_num*env.product_types_num)\n",
    "        cum_reward = np.array(\n",
    "            [np.cumsum(reward_trace)\n",
    "             for reward_trace in rewards_trace])\n",
    "        cum_reward_mean = np.array(\n",
    "            np.mean(cum_reward, axis=0),\n",
    "            dtype=np.int32)\n",
    "        cum_reward_std = np.array(\n",
    "            np.std(cum_reward.astype(np.int32), axis=0),\n",
    "            dtype=np.int32)\n",
    "        plt.plot(range(env.T),\n",
    "                 cum_reward_mean,\n",
    "                 linewidth=2,\n",
    "                 color='red', alpha=.5)\n",
    "        plt.fill_between(range(env.T),\n",
    "                         cum_reward_mean -\n",
    "                         cum_reward_std,\n",
    "                         cum_reward_mean +\n",
    "                         cum_reward_std,\n",
    "                         color='red', alpha=.2)\n",
    "\n",
    "        logger.debug(f\"\\ncumulative profit (mean) is \"\n",
    "                     f\"{cum_reward_mean}\"\n",
    "                     f\"\\ncumulative profit (std) is \"\n",
    "                     f\"{cum_reward_std}\")\n",
    "\n",
    "        plt.xlabel('Time Steps', labelpad=10)\n",
    "        plt.ticklabel_format(axis='y', style='plain',\n",
    "                             useOffset=False)\n",
    "        plt.tight_layout()\n",
    "\n",
    "        # creating necessary subdir and saving plot\n",
    "        if not os.path.exists(f\"{local_dir}/{plots_dir}/{algorithm}\"):\n",
    "            os.makedirs(f\"{local_dir}/{plots_dir}/{algorithm}\")\n",
    "        plt.savefig(f\"{local_dir}/{plots_dir}/{algorithm}\"\n",
    "                    f\"/transitions_{algorithm}.pdf\",\n",
    "                    format='pdf', bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "WOArghzY52X-"
   },
   "source": [
    "## Visualize Cumulative Profit Methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633592886888
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "DZ75Lkfwjsry"
   },
   "outputs": [],
   "source": [
    "def calculate_cum_profit(returns_trace, print_reward=True):\n",
    "    \"\"\"\n",
    "    Calculate the cumulative profit for each episode.\n",
    "    \"\"\"\n",
    "    rewards_trace = []\n",
    "    for return_trace in returns_trace:\n",
    "        rewards_trace.append(\n",
    "            np.sum(return_trace.T[2]))\n",
    "\n",
    "    if print_reward:\n",
    "        print(f\"reward: mean \"\n",
    "              f\"{np.mean(rewards_trace)}, \"\n",
    "              f\"std \"\n",
    "              f\"{np.std(rewards_trace)}, \"\n",
    "              f\"max \"\n",
    "              f\"{np.max(rewards_trace)}, \"\n",
    "              f\"min \"\n",
    "              f\"{np.min(rewards_trace)}\")\n",
    "\n",
    "    return rewards_trace\n",
    "\n",
    "\n",
    "def visualize_cum_profit(rewards_trace, algorithm,\n",
    "                         local_dir=local_dir, plots_dir=plots_dir):\n",
    "    \"\"\"\n",
    "    Visualize the cumulative profit boxplot along the episodes.\n",
    "    \"\"\"\n",
    "    xticks = []\n",
    "    if not isinstance(algorithm, list):\n",
    "        xticks.append(algorithm)\n",
    "    else:\n",
    "        xticks = algorithm\n",
    "\n",
    "    plt.figure(figsize=(15, 5))\n",
    "    plt.boxplot(rewards_trace)\n",
    "\n",
    "    plt.ylabel('Cumulative Profit')\n",
    "    plt.xticks(np.arange(1,\n",
    "                         len(xticks)+1),\n",
    "               xticks)\n",
    "    plt.tick_params(axis='x', which='both',\n",
    "                    top=False, bottom=True,\n",
    "                    labelbottom=True)\n",
    "    plt.ticklabel_format(axis='y', style='plain',\n",
    "                         useOffset=False)\n",
    "    plt.tight_layout()\n",
    "\n",
    "    # creating necessary subdir and saving plot\n",
    "    if not os.path.exists(f\"{local_dir}/{plots_dir}/{algorithm}\"):\n",
    "        os.makedirs(f\"{local_dir}/{plots_dir}/{algorithm}\")\n",
    "    plt.savefig(f\"{local_dir}/{plots_dir}/{algorithm}\"\n",
    "                f\"/cum_profit_{algorithm}.pdf\",\n",
    "                format='pdf', bbox_inches='tight')\n",
    "\n",
    "    # saving the cumulative profit as text\n",
    "    if not isinstance(algorithm, list):\n",
    "        f = open(f\"{local_dir}/{plots_dir}/{algorithm}\"\n",
    "                 f\"/cum_profit_{algorithm}.txt\",\n",
    "                 'w', encoding='utf-8')\n",
    "        f.write(f\"reward: mean \"\n",
    "                f\"{np.mean(rewards_trace)}, \"\n",
    "                f\"std \"\n",
    "                f\"{np.std(rewards_trace)}, \"\n",
    "                f\"max \"\n",
    "                f\"{np.max(rewards_trace)}, \"\n",
    "                f\"min \"\n",
    "                f\"{np.min(rewards_trace)}\")\n",
    "        f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Oracle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633593405265
    }
   },
   "outputs": [],
   "source": [
    "# cumulative profit of the oracle\n",
    "cum_profit_oracle = simulate_oracle(env, num_episodes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633593525236
    }
   },
   "outputs": [],
   "source": [
    "visualize_cum_profit(cum_profit_oracle, 'Oracle')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "lFRT1QZFzePS"
   },
   "source": [
    "# (s, Q)-Policy Class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633593645184
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "FbQWYTYuJGFl"
   },
   "outputs": [],
   "source": [
    "class SQPolicy:\n",
    "    \"\"\"\n",
    "    To assess and compare performances achieved by the adopted DRL algorithms, \n",
    "    we implement a static reorder policy known in the specialized literature as \n",
    "    the (s, Q)-policy. This policy can be expressed by a rule, which can be \n",
    "    summarized as follows: at each time step t, the current stock level for a \n",
    "    specific warehouse and product type is compared to the reorder point s. \n",
    "    If the stock level falls below the reorder point s, then the (s, Q)-policy \n",
    "    orders Q units of product; otherwise, it does not take any action.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, factory_s, factory_Q, warehouses_s, warehouses_Q):\n",
    "        self.factory_s = factory_s\n",
    "        self.factory_Q = factory_Q\n",
    "        self.warehouses_s = warehouses_s\n",
    "        self.warehouses_Q = warehouses_Q\n",
    "\n",
    "        logger.debug(f\"\\n--- SQPolicy --- __init__\"\n",
    "                     f\"\\nfactory_s is \"\n",
    "                     f\"{self.factory_s}\"\n",
    "                     f\"\\nfactory_Q is \"\n",
    "                     f\"{self.factory_Q}\"\n",
    "                     f\"\\nwarehouses_s is \"\n",
    "                     f\"{self.warehouses_s}\"\n",
    "                     f\"\\nwarehouses_Q is \"\n",
    "                     f\"{self.warehouses_Q}\")\n",
    "\n",
    "    def select_action(self, state):\n",
    "        action = Action(state.product_types_num, state.distr_warehouses_num)\n",
    "\n",
    "        # reordering decisions are made independently for factory and\n",
    "        # distribution warehouses, so policy parameters s and Q can be\n",
    "        # different for each warehouse\n",
    "        for j in range(state.distr_warehouses_num):\n",
    "            for i in range(state.product_types_num):\n",
    "                if state.distr_warehouses_stocks[j][i] < \\\n",
    "                        self.warehouses_s[j][i]:\n",
    "                    action.shipped_stocks[j][i] = \\\n",
    "                        self.warehouses_Q[j][i]\n",
    "\n",
    "        for i in range(state.product_types_num):\n",
    "            if (state.factory_stocks[i] -\n",
    "                    np.sum(action.shipped_stocks, axis=0)[i]) < \\\n",
    "                    self.factory_s[i]:\n",
    "                action.production_level[i] = \\\n",
    "                    self.factory_Q[i]\n",
    "\n",
    "        logger.debug(f\"\\n--- SQPolicy --- select_action\"\n",
    "                     f\"\\nstate is \"\n",
    "                     f\"{state}\"\n",
    "                     f\"\\naction is \"\n",
    "                     f\"{action}\"\n",
    "                     f\"\\nstate.distr_warehouses_stocks is \"\n",
    "                     f\"{state.distr_warehouses_stocks}\"\n",
    "                     f\"\\nwarehouses_s is \"\n",
    "                     f\"{self.warehouses_s}\"\n",
    "                     f\"\\nwarehouses_Q is \"\n",
    "                     f\"{self.warehouses_Q}\"\n",
    "                     f\"\\naction.shipped_stocks is \"\n",
    "                     f\"{action.shipped_stocks}\"\n",
    "                     f\"\\nstate.factory_stocks is \"\n",
    "                     f\"{state.factory_stocks}\"\n",
    "                     f\"\\nnp.sum(action.shipped_stocks, axis=0) is \"\n",
    "                     f\"{np.sum(action.shipped_stocks, axis=0)}\"\n",
    "                     f\"\\nstate.factory_stocks - \"\n",
    "                     f\"np.sum(action.shipped_stocks, axis=0) is \"\n",
    "                     f\"\"\"{state.factory_stocks -\n",
    "                         np.sum(action.shipped_stocks, axis=0)}\"\"\"\n",
    "                     f\"\\nfactory_s is \"\n",
    "                     f\"{self.factory_s}\"\n",
    "                     f\"\\nfactory_Q is \"\n",
    "                     f\"{self.factory_Q}\"\n",
    "                     f\"\\naction.production_level is \"\n",
    "                     f\"{action.production_level}\")\n",
    "\n",
    "        return action"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "7UFb5N-K0mmA"
   },
   "source": [
    "# (s, Q)-Policy Config [Ax]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "a_xrTanB7ood"
   },
   "source": [
    "## Parameters [Ax]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633593705177
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "khgac7TcD7Go"
   },
   "outputs": [],
   "source": [
    "# total trials for Ax optimization\n",
    "total_trials_Ax = 200\n",
    "# number of episodes for each trial\n",
    "num_episodes_Ax = [25, 75, 200]\n",
    "# number of iterations for each number of episodes\n",
    "iterations_Ax = 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parameters Methods [Ax]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633593765139
    },
    "hideCode": true,
    "hidePrompt": true,
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "def create_parameters_Ax(env):\n",
    "    \"\"\"\n",
    "    Create Ax (s, Q)-policy parameters (s and Q) for the factory, the \n",
    "    distribution warehouses and for each product type.\n",
    "    \"\"\"\n",
    "    # factory parameters\n",
    "    factory_parameters = [\n",
    "        {'name': 'factory_s_',\n",
    "         'type': 'range',\n",
    "         'value_type': 'int', },\n",
    "        {'name': 'factory_Q_',\n",
    "         'type': 'range',\n",
    "         'value_type': 'int', },\n",
    "    ]\n",
    "\n",
    "    factory_parameters_Ax = []\n",
    "\n",
    "    # factory parameters (s and Q) for each product type, according to storage\n",
    "    # capacities\n",
    "    for factory_parameter in factory_parameters:\n",
    "        for i in range(env.product_types_num):\n",
    "            factory_parameters_Ax.append(\n",
    "                {**factory_parameter,\n",
    "                 'name': factory_parameter['name'] + str(i+1),\n",
    "                 'bounds': [0, env.storage_capacities[0][i].item(0)], })\n",
    "\n",
    "    # distribution warehouses parameters\n",
    "    w_parameters = [\n",
    "        {'name': 'w',\n",
    "         'type': 'range',\n",
    "         'value_type': 'int', },\n",
    "    ]\n",
    "\n",
    "    w_parameters_Ax = []\n",
    "\n",
    "    # distribution warehouses parameters (s and Q) for each product type,\n",
    "    # according to storage capacities\n",
    "    for w_parameter in w_parameters:\n",
    "        for j in range(env.distr_warehouses_num):\n",
    "            for i in range(env.product_types_num):\n",
    "                w_parameters_Ax.append(\n",
    "                    {**w_parameter,\n",
    "                     'name': w_parameter['name'] + str(j+1) + '_s_' + str(i+1),\n",
    "                     'bounds': [0, env.storage_capacities[j+1][i].item(0)], })\n",
    "                w_parameters_Ax.append(\n",
    "                    {**w_parameter,\n",
    "                     'name': w_parameter['name'] + str(j+1) + '_Q_' + str(i+1),\n",
    "                     'bounds': [0, env.storage_capacities[j+1][i].item(0)], })\n",
    "\n",
    "    # final Ax parameters\n",
    "    parameters_Ax = factory_parameters_Ax + w_parameters_Ax\n",
    "\n",
    "    logger.debug(f\"\\n-- create_parameters_Ax --\"\n",
    "                 f\"\\nparameters_Ax is \"\n",
    "                 f\"{parameters_Ax}\")\n",
    "\n",
    "    return parameters_Ax\n",
    "\n",
    "\n",
    "def save_checkpoint(checkpoint, algorithm,\n",
    "                    local_dir=local_dir, plots_dir=plots_dir):\n",
    "    \"\"\"\n",
    "    Save Ax (s, Q)-policy parameters or RLib Agent checkpoint.\n",
    "    \"\"\"\n",
    "    f = open(f\"{local_dir}/{plots_dir}/{algorithm}\"\n",
    "             f\"/best_checkpoint_{algorithm}.txt\",\n",
    "             'w', encoding='utf-8')\n",
    "    f.write(checkpoint)\n",
    "    f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true
   },
   "source": [
    "# (s, Q)-Policy Methods [Ax]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimize Methods [Ax]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633593825039
    }
   },
   "outputs": [],
   "source": [
    "def opt_func_Ax(p):\n",
    "    \"\"\"\n",
    "    Evaluation function to optimize (to maximize).\n",
    "    \"\"\"\n",
    "    args = [[],\n",
    "            [],\n",
    "            [],\n",
    "            []]\n",
    "\n",
    "    # nested list for warehouses parameters\n",
    "    args[2] = [[] for _ in range(env.distr_warehouses_num)]\n",
    "    args[3] = [[] for _ in range(env.distr_warehouses_num)]\n",
    "\n",
    "    for i in range(env.product_types_num):\n",
    "        args[0].append(p[f\"factory_s_{i+1}\"])\n",
    "        args[1].append(p[f\"factory_Q_{i+1}\"])\n",
    "        for j in range(env.distr_warehouses_num):\n",
    "            args[2][j].append(p[f\"w{j+1}_s_{i+1}\"])\n",
    "            args[3][j].append(p[f\"w{j+1}_Q_{i+1}\"])\n",
    "\n",
    "    policy = SQPolicy(*args)\n",
    "\n",
    "    return np.mean(calculate_cum_profit(simulate(env, policy, num_episodes),\n",
    "                                        print_reward=False))\n",
    "\n",
    "\n",
    "def optimize_Ax(num_episodes_Ax, iterations_Ax, parameters_Ax, total_trials_Ax,\n",
    "                seed):\n",
    "    \"\"\"\n",
    "    Brute force search through the parameter space using the Adaptive\n",
    "    Experimentation Platform developed by Facebook. This framework provides a\n",
    "    very convenient API and uses Bayesian optimization internally.\n",
    "    \"\"\"\n",
    "    best_mean_cum_profit_Ax = None\n",
    "    for num_episodes in num_episodes_Ax:\n",
    "        for iteration in range(iterations_Ax):\n",
    "            # Ax optimization runs total_trials times, each trial sees a given\n",
    "            # number of episodes and this procedure is repeated for each\n",
    "            # iteration, searching the best parameters for the Ax (s, Q)-policy\n",
    "            start_Ax = default_timer()\n",
    "            parameters, values, experiment, model = optimize(\n",
    "                parameters=parameters_Ax,\n",
    "                evaluation_function=opt_func_Ax,\n",
    "                objective_name='episode_reward_mean',\n",
    "                minimize=False,\n",
    "                total_trials=total_trials_Ax,\n",
    "                random_seed=seed)\n",
    "            end_Ax = default_timer()\n",
    "\n",
    "            # setting the optimised parameters for current num episodes and\n",
    "            # iteration\n",
    "            policy_Ax = SQPolicy(\n",
    "                [parameters[f\"factory_s_{i+1}\"]\n",
    "                 for i in range(env.product_types_num)],\n",
    "                [parameters[f\"factory_Q_{i+1}\"]\n",
    "                 for i in range(env.product_types_num)],\n",
    "                [[parameters[f\"w{j+1}_s_{i+1}\"]\n",
    "                  for i in range(env.product_types_num)]\n",
    "                 for j in range(env.distr_warehouses_num)],\n",
    "                [[parameters[f\"w{j+1}_Q_{i+1}\"]\n",
    "                  for i in range(env.product_types_num)]\n",
    "                 for j in range(env.distr_warehouses_num)])\n",
    "\n",
    "            # evaluating the Ax (s, Q)-policy for current num episodes and\n",
    "            # iteration\n",
    "            returns_trace_Ax = simulate(env, policy_Ax, num_episodes)\n",
    "\n",
    "            # printing current num episodes and iteration\n",
    "            print(f\"--num episodes: {num_episodes}, \"\n",
    "                  f\"iteration: {iteration+1}\")\n",
    "\n",
    "            # cumulative profit of the Ax (s, Q)-policy for current\n",
    "            # num episodes and iteration\n",
    "            cum_profit_Ax = calculate_cum_profit(returns_trace_Ax)\n",
    "            visualize_cum_profit(cum_profit_Ax,\n",
    "                                 f\"sQ_{num_episodes}_{iteration+1}\")\n",
    "\n",
    "            # finding the best parameters for the Ax (s, Q)-policy\n",
    "            if (best_mean_cum_profit_Ax is None or\n",
    "                    np.mean(cum_profit_Ax) > best_mean_cum_profit_Ax):\n",
    "                best_mean_cum_profit_Ax = np.mean(cum_profit_Ax)\n",
    "                best_num_episodes = num_episodes\n",
    "                best_iteration = iteration\n",
    "                best_parameters = parameters\n",
    "                best_values = values\n",
    "                best_experiment = experiment\n",
    "                best_model = model\n",
    "                time_Ax = int((end_Ax-start_Ax) // 60)\n",
    "                best_policy_Ax = policy_Ax\n",
    "                best_returns_trace_Ax = returns_trace_Ax\n",
    "                best_cum_profit_Ax = cum_profit_Ax\n",
    "\n",
    "    logger.debug(f\"\\n-- optimize_Ax --\"\n",
    "                 f\"\\nbest_mean_cum_profit_Ax is \"\n",
    "                 f\"{best_mean_cum_profit_Ax}\"\n",
    "                 f\"\\nbest_num_episodes is \"\n",
    "                 f\"{best_num_episodes}\"\n",
    "                 f\"\\nbest_iteration is \"\n",
    "                 f\"{best_iteration}\"\n",
    "                 f\"\\nbest_parameters is \"\n",
    "                 f\"{best_parameters}\"\n",
    "                 f\"\\nbest_values is \"\n",
    "                 f\"{best_values}\"\n",
    "                 f\"\\nbest_experiment is \"\n",
    "                 f\"{best_experiment}\"\n",
    "                 f\"\\nbest_model is \"\n",
    "                 f\"{best_model}\"\n",
    "                 f\"\\ntime_Ax is \"\n",
    "                 f\"{time_Ax}\"\n",
    "                 f\"\\nbest_policy_Ax is \"\n",
    "                 f\"{best_policy_Ax}\"\n",
    "                 f\"\\nbest_returns_trace_Ax is \"\n",
    "                 f\"{best_returns_trace_Ax}\"\n",
    "                 f\"\\nbest_cum_profit_Ax is \"\n",
    "                 f\"{best_cum_profit_Ax}\")\n",
    "\n",
    "    return (best_mean_cum_profit_Ax, best_num_episodes, best_iteration,\n",
    "            best_parameters, best_values, best_experiment, best_model,\n",
    "            time_Ax, best_policy_Ax, best_returns_trace_Ax, best_cum_profit_Ax)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize Rewards Methods [Ax]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633593885335
    }
   },
   "outputs": [],
   "source": [
    "def visualize_optimization_trace_Ax(experiment, verbose,\n",
    "                                    local_dir=local_dir, plots_dir=plots_dir):\n",
    "    \"\"\"\n",
    "    Plot the mean reward along the trials iterations.\n",
    "    \"\"\"\n",
    "    try:\n",
    "        best_objectives = np.array(\n",
    "            [[trial.objective_mean\n",
    "              for trial in experiment.trials.values()]])\n",
    "        best_objective_plot = optimization_trace_single_method_plotly(\n",
    "            y=np.maximum.accumulate(best_objectives, axis=1),\n",
    "            ylabel='Reward Mean',\n",
    "            title='sQ Performance vs. Trials Iterations')\n",
    "        if verbose == 3:\n",
    "            best_objective_plot.show()\n",
    "        # creating necessary subdir and saving plot\n",
    "        if not os.path.exists(f\"{local_dir}/{plots_dir}\"\n",
    "                              f\"/sQ\"):\n",
    "            os.makedirs(f\"{local_dir}/{plots_dir}\"\n",
    "                        f\"/sQ\")\n",
    "        best_objective_plot.write_image(f\"{local_dir}/{plots_dir}\"\n",
    "                                        f\"/sQ\"\n",
    "                                        f\"/optimization_trace.pdf\")\n",
    "    except Exception as e:\n",
    "        print(f\"{e.__class__} occurred!\")\n",
    "\n",
    "\n",
    "def visualize_contour_Ax(model, parameters, verbose,\n",
    "                         local_dir=local_dir, plots_dir=plots_dir):\n",
    "    \"\"\"\n",
    "    Plot the contours, showing the episode reward mean as a function of two\n",
    "    selected parameters (e.g., 'factory_s_1' and 'factory_Q_1').\n",
    "    \"\"\"\n",
    "    try:\n",
    "        # creating necessary subdir\n",
    "        if not os.path.exists(f\"{local_dir}/{plots_dir}\"\n",
    "                              f\"/sQ/contours\"):\n",
    "            os.makedirs(f\"{local_dir}/{plots_dir}\"\n",
    "                        f\"/sQ/contours\")\n",
    "        # saving plots\n",
    "        for p in range(0, len(parameters), 2):\n",
    "            contour = plot_contour_plotly(model=model,\n",
    "                                          metric_name='episode_reward_mean',\n",
    "                                          param_x=list(parameters)[p],\n",
    "                                          param_y=list(parameters)[p+1])\n",
    "            contour.write_image(f\"{local_dir}/{plots_dir}\"\n",
    "                                f\"/sQ/contours\"\n",
    "                                f\"/contour_\"\n",
    "                                f\"{list(parameters)[p]}_\"\n",
    "                                f\"{list(parameters)[p+1]}.pdf\")\n",
    "            if verbose == 3:\n",
    "                contour.show()\n",
    "\n",
    "        # interactive contour plot\n",
    "        if verbose == 3:\n",
    "            render(interact_contour(model=best_model_Ax,\n",
    "                   metric_name='episode_reward_mean'))\n",
    "    except Exception as e:\n",
    "        print(f\"{e.__class__} occurred!\")\n",
    "\n",
    "\n",
    "def move_dir_Ax(local_dir=local_dir, plots_dir=plots_dir):\n",
    "    \"\"\"\n",
    "    Move dirs whose name starts with 'sQ_' (related to all optimizations) in\n",
    "    the main sQ dir.\n",
    "    \"\"\"\n",
    "    try:\n",
    "        src_dir = f\"{local_dir}/{plots_dir}/\"\n",
    "        dst_dir = f\"{local_dir}/{plots_dir}/sQ\"\n",
    "\n",
    "        pattern = src_dir + \"sQ_*\"\n",
    "        for file in glob.iglob(pattern, recursive=True):\n",
    "            shutil.move(file, dst_dir)\n",
    "            print('moved:', file)\n",
    "    except Exception as e:\n",
    "        print(f\"{e.__class__} occurred!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "wuXG3qcKkYow"
   },
   "source": [
    "# (s, Q)-Policy Optimize [Ax]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633593945318
    },
    "hideCode": true,
    "hidePrompt": true,
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    },
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Ax parameters\n",
    "parameters_Ax = create_parameters_Ax(env)\n",
    "parameters_Ax"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633447202347
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Ax optimization\n",
    "(best_mean_cum_profit_Ax, best_num_episodes_Ax, best_iteration_Ax,\n",
    " best_parameters_Ax, best_values_Ax, best_experiment_Ax, best_model_Ax,\n",
    " time_Ax, best_policy_Ax_Ax, best_returns_trace_Ax, best_cum_profit_Ax) = \\\n",
    "    optimize_Ax(num_episodes_Ax, iterations_Ax, parameters_Ax, total_trials_Ax,\n",
    "                seed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633447202945
    },
    "hideCode": true,
    "hidePrompt": true,
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "# printing the best Ax mean cum profit with related num episodes and iteration\n",
    "print(f\"best num episodes is {best_num_episodes_Ax} \"\n",
    "      f\"at iteration {best_iteration_Ax+1} \"\n",
    "      f\"\\nmean cum profit: {best_mean_cum_profit_Ax}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633447203435
    },
    "hideCode": true,
    "hidePrompt": true
   },
   "outputs": [],
   "source": [
    "visualize_optimization_trace_Ax(best_experiment_Ax, verbose)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633447208249
    }
   },
   "outputs": [],
   "source": [
    "visualize_contour_Ax(best_model_Ax, best_parameters_Ax, verbose)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633447208700
    },
    "hideCode": true,
    "hidePrompt": true
   },
   "outputs": [],
   "source": [
    "# printing the Ax optimization time\n",
    "print(f\"sQ optimization time (in minutes) is {time_Ax}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633447209114
    },
    "hideCode": true,
    "hidePrompt": true,
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "# displaying and saving the Ax best parameters\n",
    "display(best_parameters_Ax)\n",
    "save_checkpoint(str(best_parameters_Ax), 'sQ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633447212383
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "VogEVrc1_2MR"
   },
   "outputs": [],
   "source": [
    "visualize_transitions(best_returns_trace_Ax, 'sQ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633447212699
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "Dh3QV4eyjsr2"
   },
   "outputs": [],
   "source": [
    "visualize_cum_profit(best_cum_profit_Ax, 'sQ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633447213322
    }
   },
   "outputs": [],
   "source": [
    "# moving Ax optimization dirs in the main Ax dir\n",
    "move_dir_Ax()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "oy1b_YoLVGRp"
   },
   "source": [
    "# Reinforcement Learning Config [Tune]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "EiwOPnzDpfbd"
   },
   "source": [
    "## Parameters [Tune]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633624174057
    }
   },
   "outputs": [],
   "source": [
    "# number of episodes for RLib agents\n",
    "num_episodes_ray = 50000\n",
    "# stop trials at least from this number of episodes\n",
    "grace_period_ray = num_episodes_ray / 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633624174703
    },
    "hideCode": true,
    "hidePrompt": true
   },
   "outputs": [],
   "source": [
    "# dir for saving Ray results\n",
    "ray_dir = 'ray_results'\n",
    "# creating necessary dir\n",
    "if not os.path.exists(f\"{local_dir+'/'+ray_dir}\"):\n",
    "    os.makedirs(f\"{local_dir+'/'+ray_dir}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "U_CzF14Hov7E"
   },
   "source": [
    "## Algorithms [Tune]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633624175356
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "6IIRRpP90Alc"
   },
   "outputs": [],
   "source": [
    "# https://docs.ray.io/en/latest/rllib-algorithms.html\n",
    "# https://docs.ray.io/en/master/rllib-training.html#common-parameters\n",
    "# adopted algorithms\n",
    "algorithms = {\n",
    "    'A3C': a3c.A3CTrainer,\n",
    "    'PG': pg.PGTrainer,\n",
    "    'PPO': ppo.PPOTrainer,\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "tZp2UCtYQOnh"
   },
   "source": [
    "## A3C Config [Tune]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633624175823
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "xEcIrOOvCjNy"
   },
   "outputs": [],
   "source": [
    "# https://docs.ray.io/en/master/rllib-algorithms.html#a3c\n",
    "config_A3C = a3c.DEFAULT_CONFIG.copy()\n",
    "config_A3C['seed'] = seed\n",
    "config_A3C['log_level'] = 'WARN'\n",
    "\n",
    "config_A3C['env'] = SupplyChain\n",
    "config_A3C['horizon'] = env.T-1\n",
    "\n",
    "config_A3C['model']['fcnet_hiddens'] = tune.grid_search([[64, 64],\n",
    "                                                         [128, 128]])\n",
    "config_A3C['lr'] = tune.grid_search([1e-3,\n",
    "                                     1e-4])\n",
    "config_A3C['gamma'] = .99\n",
    "\n",
    "config_A3C['rollout_fragment_length'] = tune.grid_search([10,\n",
    "                                                          100])\n",
    "config_A3C['train_batch_size'] = tune.grid_search([200,\n",
    "                                                   2000])\n",
    "\n",
    "config_A3C['grad_clip'] = tune.grid_search([20.0,\n",
    "                                            40.0])\n",
    "\n",
    "config_A3C['evaluation_num_episodes'] = 1000\n",
    "config_A3C['sample_async'] = False\n",
    "\n",
    "config_A3C['num_workers'] = num_cpus-1\n",
    "config_A3C['num_gpus'] = num_gpus\n",
    "\n",
    "config_A3C['framework'] = 'torch'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "8G1s-KiQfbhH",
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "## PG Config [Tune]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633624176368
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "1K5mUgmsfbhH",
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "# https://docs.ray.io/en/master/rllib-algorithms.html#policy-gradients\n",
    "config_PG = pg.DEFAULT_CONFIG.copy()\n",
    "config_PG['seed'] = seed\n",
    "config_PG['log_level'] = 'WARN'\n",
    "\n",
    "config_PG['env'] = SupplyChain\n",
    "config_PG['horizon'] = env.T-1\n",
    "\n",
    "config_PG['model']['fcnet_hiddens'] = tune.grid_search([[64, 64],\n",
    "                                                        [128, 128]])\n",
    "config_PG['lr'] = tune.grid_search([4e-3,\n",
    "                                    4e-4])\n",
    "config_PG['gamma'] = .99\n",
    "\n",
    "config_PG['rollout_fragment_length'] = tune.grid_search([10,\n",
    "                                                         100])\n",
    "config_PG['train_batch_size'] = tune.grid_search([200,\n",
    "                                                  2000])\n",
    "\n",
    "config_PG['evaluation_num_episodes'] = 1000\n",
    "config_PG['sample_async'] = False\n",
    "\n",
    "config_PG['num_workers'] = num_cpus-1\n",
    "config_PG['num_gpus'] = num_gpus\n",
    "\n",
    "config_PG['framework'] = 'torch'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "Znx6pGRgQX6u"
   },
   "source": [
    "## PPO Config [Tune]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633624177107
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "lF4Z33RTGHUq"
   },
   "outputs": [],
   "source": [
    "# https://docs.ray.io/en/master/rllib-algorithms.html#ppo\n",
    "config_PPO = ppo.DEFAULT_CONFIG.copy()\n",
    "config_PPO['seed'] = seed\n",
    "config_PPO['log_level'] = 'WARN'\n",
    "\n",
    "config_PPO['env'] = SupplyChain\n",
    "config_PPO['horizon'] = env.T-1\n",
    "\n",
    "config_PPO['model']['fcnet_hiddens'] = tune.grid_search([[64, 64],\n",
    "                                                         [128, 128]])\n",
    "config_PPO['lr'] = tune.grid_search([5e-3,\n",
    "                                     5e-4])\n",
    "config_PPO['gamma'] = .99\n",
    "\n",
    "config_PPO['rollout_fragment_length'] = tune.grid_search([20,\n",
    "                                                          200])\n",
    "config_PPO['train_batch_size'] = tune.grid_search([400,\n",
    "                                                   4000])\n",
    "\n",
    "config_PPO['grad_clip'] = tune.grid_search([None,\n",
    "                                            20.0])\n",
    "config_PPO['num_sgd_iter'] = tune.grid_search([15,\n",
    "                                               30])\n",
    "config_PPO['sgd_minibatch_size'] = tune.grid_search([64,\n",
    "                                                     128])\n",
    "\n",
    "config_PPO['evaluation_num_episodes'] = 1000\n",
    "config_PPO['sample_async'] = False\n",
    "\n",
    "config_PPO['num_workers'] = num_cpus-1\n",
    "config_PPO['num_gpus'] = num_gpus\n",
    "\n",
    "config_PPO['framework'] = 'torch'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "MWsHZOfjnDcj"
   },
   "source": [
    "# Reinforcement Learning Methods [Tune]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "xOvro1oF_5ME"
   },
   "source": [
    "## Train Agents Methods [Tune]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633624177645
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "qo9YSeyxwObs"
   },
   "outputs": [],
   "source": [
    "def train(algorithm, config, verbose,\n",
    "          num_episodes_ray=num_episodes_ray, grace_period_ray=grace_period_ray,\n",
    "          local_dir=local_dir, ray_dir=ray_dir):\n",
    "    \"\"\"\n",
    "    Train a RLib Agent.\n",
    "    \"\"\"\n",
    "    # initializing Ray\n",
    "    ray.shutdown()\n",
    "    ray.init(log_to_driver=False)\n",
    "\n",
    "    logger.debug(f\"\\n-- train --\"\n",
    "                 f\"\\nalgorithm is \"\n",
    "                 f\"{algorithm}\"\n",
    "                 f\"\\nconfig is \"\n",
    "                 f\"{config}\")\n",
    "\n",
    "    # https://docs.ray.io/en/latest/tune/api_docs/execution.html\n",
    "    # https://docs.ray.io/en/master/tune/api_docs/schedulers.html#summary\n",
    "    # https://docs.ray.io/en/master/tune/api_docs/analysis.html#id1\n",
    "    analysis = tune.run(algorithm,\n",
    "                        config=config,\n",
    "                        metric='episode_reward_mean',\n",
    "                        mode='max',\n",
    "                        scheduler=ASHAScheduler(\n",
    "                            time_attr='episodes_total',\n",
    "                            max_t=num_episodes_ray,\n",
    "                            grace_period=grace_period_ray,\n",
    "                            reduction_factor=5),\n",
    "                        checkpoint_freq=1,\n",
    "                        keep_checkpoints_num=1,\n",
    "                        checkpoint_score_attr='episode_reward_mean',\n",
    "                        progress_reporter=tune.JupyterNotebookReporter(\n",
    "                            overwrite=True),\n",
    "                        verbose=verbose,\n",
    "                        local_dir=os.getcwd()+'/'+local_dir+'/'+ray_dir)\n",
    "\n",
    "    trial_dataframes = analysis.trial_dataframes\n",
    "    best_result_df = analysis.best_result_df\n",
    "    best_config = analysis.best_config\n",
    "    best_checkpoint = analysis.best_checkpoint\n",
    "    print(f\"\\ncheckpoint saved at {best_checkpoint}\")\n",
    "\n",
    "    # stopping Ray\n",
    "    ray.shutdown()\n",
    "\n",
    "    return trial_dataframes, best_result_df, best_config, best_checkpoint\n",
    "\n",
    "\n",
    "def result_df_as_image(result_df, algorithm,\n",
    "                       local_dir=local_dir, plots_dir=plots_dir):\n",
    "    \"\"\"\n",
    "    Visualize the (DataFrame) RLib Agent's result as an image.\n",
    "    \"\"\"\n",
    "    # creating necessary subdir and saving plot\n",
    "    if not os.path.exists(f\"{local_dir}/{plots_dir}/{algorithm}\"):\n",
    "        os.makedirs(f\"{local_dir}/{plots_dir}/{algorithm}\")\n",
    "    dfi.export(result_df.iloc[:, np.r_[:3, 9]],\n",
    "               f\"{local_dir}/{plots_dir}/{algorithm}\"\n",
    "               f\"/best_result_{algorithm}.png\",\n",
    "               table_conversion='matplotlib')\n",
    "\n",
    "\n",
    "def calculate_training_time(result_df):\n",
    "    \"\"\"\n",
    "    Calculate a RLib Agent training time (minutes).\n",
    "    \"\"\"\n",
    "    return int(result_df.time_total_s[0]//60)\n",
    "\n",
    "\n",
    "def calculate_training_episodes(result_df):\n",
    "    \"\"\"\n",
    "    Calculate a RLib Agent training episodes (number).\n",
    "    \"\"\"\n",
    "    return round(result_df.episodes_total[0], -3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "VFPY1kX_fbhI",
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "## Policy Methods [Tune]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633624178852
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "-J4CGhsIfbhI",
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "def load_policy(algorithm, config, checkpoint):\n",
    "    \"\"\"\n",
    "    Load a RLib Agent policy.\n",
    "    \"\"\"\n",
    "    # initializing Ray\n",
    "    ray.shutdown()\n",
    "    ray.init(log_to_driver=False)\n",
    "\n",
    "    # loading policy\n",
    "    trainer = algorithm(config=config)\n",
    "    trainer.restore(f\"{checkpoint}\")\n",
    "    policy = trainer.get_policy()\n",
    "\n",
    "    # stopping Ray\n",
    "    ray.shutdown()\n",
    "\n",
    "    logger.debug(f\"\\n-- load_policy --\"\n",
    "                 f\"\\nalgorithm is \"\n",
    "                 f\"{algorithm}\"\n",
    "                 f\"\\nconfig is \"\n",
    "                 f\"{config}\"\n",
    "                 f\"\\ncheckpoint is \"\n",
    "                 f\"{checkpoint}\"\n",
    "                 f\"\\ntrainer is \"\n",
    "                 f\"{trainer}\"\n",
    "                 f\"\\npolicy is \"\n",
    "                 f\"{policy}\")\n",
    "\n",
    "    return policy\n",
    "\n",
    "\n",
    "def fix_best_checkpoint(checkpoint):\n",
    "    \"\"\"\n",
    "    Fix a RLib Agent best checkpoint path.\n",
    "    \"\"\"\n",
    "    # searching all checkpoints related to the best agent's result\n",
    "    checkpoint_dir = checkpoint.rsplit('/', 2)[0]\n",
    "    sub_dirs = [sub_dir for sub_dir in os.listdir(checkpoint_dir)\n",
    "                if os.path.isdir(os.path.join(checkpoint_dir, sub_dir))]\n",
    "    # finding the most recent checkpoint (the best one)\n",
    "    sub_dirs.sort(reverse=True)\n",
    "\n",
    "    # creating the fixed best checkpoint path\n",
    "    fixed_checkpoint_dir = checkpoint_dir + '/' + sub_dirs[0] + '/'\n",
    "    fixed_checkpoint_file = os.listdir(fixed_checkpoint_dir)[0].split('.')[0]\n",
    "    best_checkpoint = fixed_checkpoint_dir + fixed_checkpoint_file\n",
    "\n",
    "    logger.debug(f\"\\n-- fix_best_checkpoint --\"\n",
    "                 f\"\\nfixed_checkpoint_dir is \"\n",
    "                 f\"{fixed_checkpoint_dir}\"\n",
    "                 f\"\\nfixed_checkpoint_file is \"\n",
    "                 f\"{fixed_checkpoint_file}\"\n",
    "                 f\"\\nbest_checkpoint is \"\n",
    "                 f\"{best_checkpoint}\")\n",
    "\n",
    "    return best_checkpoint"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "## Visualize Rewards Methods [Tune]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633624178272
    },
    "hideCode": true,
    "hidePrompt": true,
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "def apply_style_plot(ax, episodes_total):\n",
    "    \"\"\"\n",
    "    Auxiliary function.\n",
    "    \"\"\"\n",
    "    labels = [str(0),\n",
    "              str(episodes_total//2),\n",
    "              str(episodes_total)]\n",
    "    ax.xaxis.set_major_locator(ticker.LinearLocator(len(labels)))\n",
    "    ax.set_xticklabels(labels)\n",
    "    ax.set_xlabel('Episodes')\n",
    "    ax.ticklabel_format(axis='y', style='plain',\n",
    "                        useOffset=False)\n",
    "\n",
    "\n",
    "def visualize_rewards(results, best_result, algorithm, legend=[],\n",
    "                      local_dir=local_dir, plots_dir=plots_dir):\n",
    "    \"\"\"\n",
    "    Visualize the min, mean and max rewards along the episodes.\n",
    "    \"\"\"\n",
    "    # creating necessary subdir and saving plot\n",
    "    if not os.path.exists(f\"{local_dir}/{plots_dir}/{algorithm}\"):\n",
    "        os.makedirs(f\"{local_dir}/{plots_dir}/{algorithm}\")\n",
    "    episodes_total = calculate_training_episodes(best_result)\n",
    "\n",
    "    # min reward\n",
    "    fig, ax = plt.subplots(figsize=(15, 5))\n",
    "    apply_style_plot(ax, episodes_total)\n",
    "    ax.set_ylabel('Min Reward')\n",
    "    for result in results.values():\n",
    "        ax = result.episode_reward_min.plot(ax=ax)\n",
    "    ax.legend(legend, bbox_to_anchor=(1.04, .5), borderaxespad=0,\n",
    "              frameon=False, loc='center left', fancybox=True, shadow=True)\n",
    "    plt.tight_layout()\n",
    "    plt.savefig(f\"{local_dir}/{plots_dir}/{algorithm}\"\n",
    "                f\"/episode_reward_min_{algorithm}.pdf\",\n",
    "                format='pdf', bbox_inches='tight')\n",
    "\n",
    "    # mean reward\n",
    "    fig, ax = plt.subplots(figsize=(15, 5))\n",
    "    apply_style_plot(ax, episodes_total)\n",
    "    ax.set_ylabel('Mean Reward')\n",
    "    for result in results.values():\n",
    "        ax = result.episode_reward_mean.plot(ax=ax)\n",
    "    ax.legend(legend, bbox_to_anchor=(1.04, .5), borderaxespad=0,\n",
    "              frameon=False, loc='center left', fancybox=True, shadow=True)\n",
    "    plt.tight_layout()\n",
    "    plt.savefig(f\"{local_dir}/{plots_dir}/{algorithm}\"\n",
    "                f\"/episode_reward_mean_{algorithm}.pdf\",\n",
    "                format='pdf', bbox_inches='tight')\n",
    "\n",
    "    # max reward\n",
    "    fig, ax = plt.subplots(figsize=(15, 5))\n",
    "    apply_style_plot(ax, episodes_total)\n",
    "    ax.set_ylabel('Max Reward')\n",
    "    for result in results.values():\n",
    "        ax = result.episode_reward_max.plot(ax=ax)\n",
    "    ax.legend(legend, bbox_to_anchor=(1.04, .5), borderaxespad=0,\n",
    "              frameon=False, loc='center left', fancybox=True, shadow=True)\n",
    "    plt.tight_layout()\n",
    "    plt.savefig(f\"{local_dir}/{plots_dir}/{algorithm}\"\n",
    "                f\"/episode_reward_max_{algorithm}.pdf\",\n",
    "                format='pdf', bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "Q61DceAPAJDN"
   },
   "source": [
    "# Reinforcement Learning Train Agents [Tune]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "F2cgOoJxPViF"
   },
   "source": [
    "## A3C Agent [Tune]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633634979207
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "fKStgDNZPViF",
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# training an A3C agent\n",
    "(results_A3C, best_result_A3C,\n",
    " best_config_A3C, checkpoint_A3C) = train(algorithms['A3C'],\n",
    "                                          config_A3C,\n",
    "                                          verbose)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633634979779
    },
    "hideCode": true,
    "hidePrompt": true,
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "# saving and showing the best result of the A3C agent\n",
    "result_df_as_image(best_result_A3C, 'A3C')\n",
    "best_result_A3C"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633634980414
    },
    "hideCode": true,
    "hidePrompt": true,
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "visualize_rewards(results_A3C, best_result_A3C, 'A3C')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633634980934
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "54mkghcajsr6"
   },
   "outputs": [],
   "source": [
    "# calculating and printing the A3C training time\n",
    "time_A3C = calculate_training_time(best_result_A3C)\n",
    "print(f\"total training time A3C (in minutes) is {time_A3C}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633634981450
    },
    "hideCode": true,
    "hidePrompt": true,
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "# (fixing and) saving the A3C best checkpoint\n",
    "best_checkpoint_A3C = fix_best_checkpoint(checkpoint_A3C)\n",
    "save_checkpoint(best_checkpoint_A3C, 'A3C')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633634989567
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "FTpoK2i8PViG"
   },
   "outputs": [],
   "source": [
    "# loading the best A3C agent's policy\n",
    "policy_A3C = load_policy(algorithms['A3C'],\n",
    "                         best_config_A3C,\n",
    "                         best_checkpoint_A3C)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633635015317
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "fGSeX8cVjsr6"
   },
   "outputs": [],
   "source": [
    "# evaluating the best A3C agent's policy\n",
    "returns_trace_A3C = simulate(env, policy_A3C, num_episodes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633635019409
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "WY0gK5apPViH"
   },
   "outputs": [],
   "source": [
    "visualize_transitions(returns_trace_A3C, 'A3C')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633635019733
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "5-WezYRnjsr6"
   },
   "outputs": [],
   "source": [
    "# cumulative profit of the best A3C agent's policy\n",
    "cum_profit_A3C = calculate_cum_profit(returns_trace_A3C)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633635020236
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "BSmwdPFqjsr6"
   },
   "outputs": [],
   "source": [
    "visualize_cum_profit(cum_profit_A3C, 'A3C')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "UIuC6UTWfbhN",
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "source": [
    "## PG Agent [Tune]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633330437520
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "XWwgVJ9TfbhN",
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "# training a PG agent\n",
    "(results_PG, best_result_PG,\n",
    " best_config_PG, checkpoint_PG) = train(algorithms['PG'],\n",
    "                                        config_PG,\n",
    "                                        verbose)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633330437984
    },
    "hideCode": true,
    "hidePrompt": true,
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "# saving and showing the best result of the PG agent\n",
    "result_df_as_image(best_result_PG, 'PG')\n",
    "best_result_PG"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633330438411
    },
    "hideCode": true,
    "hidePrompt": true,
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "visualize_rewards(results_PG, best_result_PG, 'PG')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633330438709
    },
    "hideCode": true,
    "hidePrompt": true
   },
   "outputs": [],
   "source": [
    "# calculating and printing the PG training time\n",
    "time_PG = calculate_training_time(best_result_PG)\n",
    "print(f\"total training time PG (in minutes) is {time_PG}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633330439000
    },
    "hideCode": true,
    "hidePrompt": true,
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "# (fixing and) saving the PG best checkpoint\n",
    "best_checkpoint_PG = fix_best_checkpoint(checkpoint_PG)\n",
    "save_checkpoint(best_checkpoint_PG, 'PG')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633330439478
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "3NbfUeCAfbhO",
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "# loading the best PG agent's policy\n",
    "policy_PG = load_policy(algorithms['PG'],\n",
    "                        best_config_PG,\n",
    "                        best_checkpoint_PG)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633330439780
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "dXcYZgthjsr8"
   },
   "outputs": [],
   "source": [
    "# evaluating the best PG agent's policy\n",
    "returns_trace_PG = simulate(env, policy_PG, num_episodes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633330440220
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "FkK1OsjmfbhO",
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "visualize_transitions(returns_trace_PG, 'PG')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633330440536
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "cZ6EqG7Pjsr8"
   },
   "outputs": [],
   "source": [
    "# cumulative profit of the best PG agent's policy\n",
    "cum_profit_PG = calculate_cum_profit(returns_trace_PG)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1633330440998
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "fQUApkdGjsr8"
   },
   "outputs": [],
   "source": [
    "visualize_cum_profit(cum_profit_PG, 'PG')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "SelC881AO8ro"
   },
   "source": [
    "## PPO Agent [Tune]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1630703897399
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "Ig01XproO8r0"
   },
   "outputs": [],
   "source": [
    "# training a PPO agent\n",
    "(results_PPO, best_result_PPO,\n",
    " best_config_PPO, checkpoint_PPO) = train(algorithms['PPO'],\n",
    "                                          config_PPO,\n",
    "                                          verbose)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1632126612616
    },
    "hideCode": true,
    "hidePrompt": true,
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "# saving and showing the best result of the PPO agent\n",
    "result_df_as_image(best_result_PPO, 'PPO')\n",
    "best_result_PPO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1632126625836
    },
    "hideCode": true,
    "hidePrompt": true,
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "visualize_rewards(results_PPO, best_result_PPO, 'PPO')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1632126629117
    },
    "hideCode": true,
    "hidePrompt": true
   },
   "outputs": [],
   "source": [
    "# calculating and printing the PPO training time\n",
    "time_PPO = calculate_training_time(best_result_PPO)\n",
    "print(f\"total training time PPO (in minutes) is {time_PPO}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1632126633520
    },
    "hideCode": true,
    "hidePrompt": true,
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "# (fixing and) saving the PPO best checkpoint\n",
    "best_checkpoint_PPO = fix_best_checkpoint(checkpoint_PPO)\n",
    "save_checkpoint(best_checkpoint_PPO, 'PPO')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1632126647379
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "o83MXxurO8r1"
   },
   "outputs": [],
   "source": [
    "# loading the best PPO agent's policy\n",
    "policy_PPO = load_policy(algorithms['PPO'],\n",
    "                         best_config_PPO,\n",
    "                         best_checkpoint_PPO)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1632126700546
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "oZ7jcRZRjsr7"
   },
   "outputs": [],
   "source": [
    "# evaluating the best PPO agent's policy\n",
    "returns_trace_PPO = simulate(env, policy_PPO, num_episodes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1632126709671
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "Pb7QLvQ_O8r2"
   },
   "outputs": [],
   "source": [
    "visualize_transitions(returns_trace_PPO, 'PPO')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1632126713957
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "6oZajepvjsr7"
   },
   "outputs": [],
   "source": [
    "# cumulative profit of the best PPO agent's policy\n",
    "cum_profit_PPO = calculate_cum_profit(returns_trace_PPO)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1632126717669
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "p-_IOhFSjsr7"
   },
   "outputs": [],
   "source": [
    "visualize_cum_profit(cum_profit_PPO, 'PPO')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "pgR6ta4sjsr9"
   },
   "source": [
    "# Final Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true
   },
   "source": [
    "## Cumulative Profit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1632126728935
    }
   },
   "outputs": [],
   "source": [
    "visualize_cum_profit([cum_profit_A3C,\n",
    "                      cum_profit_PG,\n",
    "                      cum_profit_PPO],\n",
    "                     ['A3C',\n",
    "                      'PG',\n",
    "                      'PPO'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1632126729943
    },
    "hideCode": true,
    "hidePrompt": true,
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "visualize_cum_profit([cum_profit_A3C,\n",
    "                      cum_profit_PG,\n",
    "                      cum_profit_PPO,\n",
    "                      cum_profit_oracle],\n",
    "                     ['A3C',\n",
    "                      'PG',\n",
    "                      'PPO',\n",
    "                      'Oracle'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1632126730771
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "5myPpnzejsr9",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "visualize_cum_profit([cum_profit_A3C,\n",
    "                      cum_profit_PG,\n",
    "                      cum_profit_PPO,\n",
    "                      best_cum_profit_Ax],\n",
    "                     ['A3C',\n",
    "                      'PG',\n",
    "                      'PPO',\n",
    "                      'sQ'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1632126731537
    }
   },
   "outputs": [],
   "source": [
    "visualize_cum_profit([cum_profit_A3C,\n",
    "                      cum_profit_PG,\n",
    "                      cum_profit_PPO,\n",
    "                      best_cum_profit_Ax,\n",
    "                      cum_profit_oracle],\n",
    "                     ['A3C',\n",
    "                      'PG',\n",
    "                      'PPO',\n",
    "                      'sQ',\n",
    "                      'Oracle'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true
   },
   "source": [
    "## Training Time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1632126732248
    },
    "hideCode": true,
    "hidePrompt": true
   },
   "outputs": [],
   "source": [
    "# training time of all policies\n",
    "times_total = {'Algorithm':\n",
    "               ['A3C',\n",
    "                'PG',\n",
    "                'PPO',\n",
    "                f\"sQ_{best_num_episodes_Ax}_{best_iteration_Ax+1}\"],\n",
    "               'Training Time \\n(in minutes)':\n",
    "               [time_A3C,\n",
    "                time_PG,\n",
    "                time_PPO,\n",
    "                time_Ax]}\n",
    "# creating pandas DataFrame\n",
    "times_total_df = pd.DataFrame(data=times_total)\n",
    "times_total_df.set_index('Algorithm', inplace=True)\n",
    "# saving pandas DataFrame as an image\n",
    "dfi.export(times_total_df,\n",
    "           f\"{local_dir}/{plots_dir}\"\n",
    "           f\"/times_total_df.png\",\n",
    "           table_conversion='matplotlib')\n",
    "# printing training time of all policies\n",
    "print(tabulate(times_total_df, headers='keys', tablefmt='grid'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true
   },
   "source": [
    "# Compress Final Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1632127478541
    },
    "hideCode": true,
    "hidePrompt": true,
    "id": "YuI_ela_jsr9"
   },
   "outputs": [],
   "source": [
    "# creating a tar file containing plots and Ray results\n",
    "try:\n",
    "    cmd = f\"tar -zcvf {local_dir}.tar.gz ./{local_dir}\"\n",
    "    print(f\"cmd is {cmd}\")\n",
    "    os.system(cmd)\n",
    "except Exception as e:\n",
    "    print(f\"{e.__class__} occurred!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hideCode": true,
    "hidePrompt": true,
    "id": "NkajbqUKc9wm"
   },
   "source": [
    "# TensorBoard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1630704117807
    }
   },
   "outputs": [],
   "source": [
    "# checking if A3C best checkpoint is defined\n",
    "try:\n",
    "    best_checkpoint_A3C\n",
    "except Exception as e:\n",
    "    print(f\"{e.__class__} occurred!\")\n",
    "    best_checkpoint_A3C = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1630704118158
    }
   },
   "outputs": [],
   "source": [
    "# checking if PG best checkpoint is defined\n",
    "try:\n",
    "    best_checkpoint_PG\n",
    "except Exception as e:\n",
    "    print(f\"{e.__class__} occurred!\")\n",
    "    best_checkpoint_PG = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1630704118445
    }
   },
   "outputs": [],
   "source": [
    "# checking if PPO best checkpoint is defined\n",
    "try:\n",
    "    best_checkpoint_PPO\n",
    "except Exception as e:\n",
    "    print(f\"{e.__class__} occurred!\")\n",
    "    best_checkpoint_PPO = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1630704118882
    }
   },
   "outputs": [],
   "source": [
    "# TensorBoard dir for Ray results (the first best checkpoint not None)\n",
    "tb_dir = next(checkpoint for checkpoint in [best_checkpoint_A3C,\n",
    "                                            best_checkpoint_PG,\n",
    "                                            best_checkpoint_PPO]\n",
    "              if checkpoint is not None).rsplit('/', 4)[0]\n",
    "tb_dir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "gather": {
     "logged": 1630704119654
    },
    "hideCode": true,
    "hidePrompt": true,
    "jupyter": {
     "outputs_hidden": false,
     "source_hidden": false
    },
    "nteract": {
     "transient": {
      "deleting": false
     }
    }
   },
   "outputs": [],
   "source": [
    "# loading TensorBoard\n",
    "try:\n",
    "    %load_ext tensorboard\n",
    "    %tensorboard --logdir $tb_dir\n",
    "except Exception as e:\n",
    "    print(f\"{e.__class__} occurred!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "MultiProduct_SupplyChain_V5_0.ipynb",
   "private_outputs": true,
   "provenance": [],
   "toc_visible": true
  },
  "hide_code_all_hidden": true,
  "kernel_info": {
   "name": "python38-azureml"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "microsoft": {
   "host": {
    "AzureML": {
     "notebookHasBeenCompleted": true
    }
   }
  },
  "nteract": {
   "version": "nteract-front-end@1.0.0"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "292px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "position": {
    "height": "375px",
    "left": "1059px",
    "right": "20px",
    "top": "120px",
    "width": "361px"
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}