05 - Gridworld with Value Iteration.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gridworld\n",
    "\n",
    "*Andrea Mazzolini*, andrea.mazzolini.90@gmail.com.\n",
    "\n",
    "\n",
    "Here we want to find the optimal strategy of a 2d grid-world problem having full information.\n",
    "**Value iteration** algorithm will be used."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gridworld as a Markov Decision Process\n",
    "\n",
    "### States\n",
    "The state space corresponds to the physical space of the gridworld. Therefore each state is identified by the two coordinates, and the whole space is composed of $d^2$ states:\n",
    "\n",
    "$$\n",
    "\\mathcal{S} = \\{ 0, 1, \\ldots, d-1 \\} \\times \\{ 0, 1, \\ldots, d-1 \\}\n",
    "$$\n",
    "\n",
    "### Actions\n",
    "The actions of the agent are five: he can move to nearest neighbours or stay in the cell without moving:\n",
    "\n",
    "$$\n",
    "\\mathcal{A} = \\{ \\text{up}, \\text{left}, \\text{down}, \\text{right}, \\text{stay} \\} = \\{ (0,1), (-1,0), (0,-1), (1,0), (0,0)\\}\n",
    "$$\n",
    "\n",
    "which can be expressed also translation vectors.\n",
    "Actually, these actions are not always possible in each state: the agent cannot cross boundaries. This makes actions state dependent, for example if the agent is located on the left boundary: $\\mathcal{A}(0,y) = \\{ \\text{up}, \\text{down}, \\text{right}, \\text{stay} \\}$, or in a corner: $\\mathcal{A}(d-1,d-1) = \\{ \\text{left}, \\text{down}, \\text{stay} \\}$.\n",
    "\n",
    "### Transition probabilities\n",
    "\n",
    "The transition probabilities between states are deterministic: the next state is just the old state plus the translation action chosen by the agent:\n",
    "\n",
    "$$\n",
    "p(s_{t+1} | a_t, s_t) = \\delta (s_{t+1} = a_t + s_t)\n",
    "$$\n",
    "\n",
    "### Rewards\n",
    "\n",
    "The rewards depends only on the arrival states, $r(s_{t+1})$, and are zero for all the states with the exception of some special ones chosen to contain some resource."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Value iteration algorithm\n",
    "\n",
    "* Initialise the values of all the states $V^{(0)}(s)$.\n",
    "\n",
    "* For $t$ in $[1,2,\\ldots]$ until a *convergence condition* is satisfied do:\n",
    "> * for each state $s$:\n",
    "> > * Compute the new value estimate using the Bellman equation: $ V^{(t)}(s) = \\max_{a} \\sum_{s'} p(s'|s,a)\\big[ r(s',a, s) + \\gamma\\,V^{(t-1)}(s') \\big] $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementation\n",
    "\n",
    "### Environment class: the gridworld\n",
    "\n",
    "The Gridworld class contains all the information about the environment:\n",
    "- The info about the state space (cells and obstacles).\n",
    "- The set of possible actions from each state.\n",
    "- The reward table: which reward the agent take in each cell (0 if none).\n",
    "\n",
    "The methods are:\n",
    "- `display()`: Plot the gridworld.\n",
    "- `display_best_path(values, gamma, start_cell)`: Plot the best path from the `start_cell`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from copy import copy\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib import cm\n",
    "\n",
    "\n",
    "class Gridworld:\n",
    "\n",
    "    def __init__(self, grid_size, rewards, obstacles=[]):\n",
    "        \"\"\"\n",
    "        Training environment for reinforcement learning: gridworld.\n",
    "        Args:\n",
    "        - grid_size, (int, int): defining the size of the 2d lattice\n",
    "        - init_cell, (int, int): coordinates from 0 to size-1 from which the agent starts to play\n",
    "        - rewards, list((int, int), float): list of the coordinates§ and values of the rewards\n",
    "        - obstacles, lits((int, int)): list of the coordinates of the obstacles\n",
    "        \"\"\"\n",
    "\n",
    "        # Define state space\n",
    "        self.state = None  # current state of the game\n",
    "        self.state_dim = grid_size\n",
    "        #self.init_state = init_cell\n",
    "        self.obstacles = obstacles\n",
    "        # Cells that are not obstacles\n",
    "        self.states = [(i,j) for i in range(self.state_dim[0]) for j in range(self.state_dim[1]) if (i,j) not in self.obstacles] \n",
    "\n",
    "        # Define action space\n",
    "        self.action_dim = (5,)  # up, right, down, left, stay\n",
    "        self.action_dict = {\"up\": 0, \"right\": 1, \"down\": 2, \"left\": 3, \"stay\": 4}\n",
    "        self.action_coords = [(0, 1), (1, 0), (0, -1), (-1, 0), (0, 0)]  # translations\n",
    "        self.actions_allowed = self._build_allowed_actions(obstacles)\n",
    "\n",
    "        # Define rewards table\n",
    "        self.R = self._build_rewards(rewards)\n",
    "\n",
    "\n",
    "    def _build_allowed_actions(self, obstacles):\n",
    "        actions_allowed = dict()\n",
    "        Nx, Ny = self.state_dim\n",
    "        for x in range(Nx):\n",
    "            for y in range(Ny):\n",
    "                # Actions not allowed at the boundaries\n",
    "                actions_allowed[(x,y)] = [self.action_dict[\"stay\"]] # The stay action is always allowed\n",
    "                if (y > 0):  \n",
    "                    actions_allowed[(x,y)].append(self.action_dict[\"down\"])\n",
    "                if (y < Ny - 1):  \n",
    "                    actions_allowed[(x,y)].append(self.action_dict[\"up\"])\n",
    "                if (x > 0):  \n",
    "                    actions_allowed[(x,y)].append(self.action_dict[\"left\"])\n",
    "                if (x < Nx - 1):  \n",
    "                    actions_allowed[(x,y)].append(self.action_dict[\"right\"])\n",
    "                actions_allowed[(x,y)] = np.array(actions_allowed[(x,y)], dtype=int)\n",
    "\n",
    "                # Actions not allowed because of obstacles\n",
    "                for o in obstacles:\n",
    "                    if (x+1,y) == o:\n",
    "                        actions_allowed[(x,y)] = actions_allowed[(x,y)][actions_allowed[(x,y)] != self.action_dict[\"right\"]]\n",
    "                    if (x-1,y) == o:\n",
    "                        actions_allowed[(x,y)] = actions_allowed[(x,y)][actions_allowed[(x,y)] != self.action_dict[\"left\"]]\n",
    "                    if (x,y+1) == o:\n",
    "                        actions_allowed[(x,y)] = actions_allowed[(x,y)][actions_allowed[(x,y)] != self.action_dict[\"up\"]]\n",
    "                    if (x,y-1) == o:\n",
    "                        actions_allowed[(x,y)] = actions_allowed[(x,y)][actions_allowed[(x,y)] != self.action_dict[\"down\"]]\n",
    "        return actions_allowed\n",
    "\n",
    "\n",
    "    def _build_rewards(self, rewards):\n",
    "        R = np.zeros(self.state_dim, dtype=float)\n",
    "        for rew in rewards:\n",
    "            R[rew[0]] = rew[1]\n",
    "        return R\n",
    "\n",
    "    def display(self, values=np.array([]), cmap=sns.dark_palette(\"red\", as_cmap=True), figsize=(7,6)):\n",
    "        fig = plt.figure(figsize=figsize)\n",
    "        obstacle_mask = np.zeros(self.state_dim, dtype=bool)\n",
    "        for obs in self.obstacles:\n",
    "            obstacle_mask[obs[0], obs[1]] = True\n",
    "\n",
    "        if len(values)==0:\n",
    "            ax = sns.heatmap(obstacle_mask.T, cmap=cm.get_cmap(\"Greys\"), cbar=False, \n",
    "                       linewidths=0.1, linecolor='#222222')\n",
    "        else:\n",
    "            ax = sns.heatmap(values.T, mask=obstacle_mask.T, cmap=cmap, \n",
    "                       linewidths=0.1, linecolor='#222222', vmin=np.min(values[values != 0]))\n",
    "            ax.collections[0].colorbar.set_label(\"Value\", fontsize=14)\n",
    "\n",
    "        ax.invert_yaxis()\n",
    "        ax.set_xlabel('x', fontsize=14)\n",
    "        ax.set_ylabel('y', fontsize=14)\n",
    "        #ax.scatter([start_cell[0]+0.5],[start_cell[1]+0.5], s=100, c='grey', label='Start')\n",
    "        \n",
    "        for rew in rewards:\n",
    "            ax.scatter([rew[0][0]+0.5],[rew[0][1]+0.5], s=200*rew[1], c='red', label='Reward:{}'.format(rew[1]), marker='*')\n",
    "        plt.tight_layout()\n",
    "\n",
    "        return fig, ax\n",
    "\n",
    "\n",
    "    def display_best_path(self, values, gamma, start_coord, lcolor='black', cmap=sns.dark_palette(\"red\", as_cmap=True), figsize=(7,6)):\n",
    "        fig, ax = self.display(values, cmap, figsize)\n",
    "        obstacle_mask = np.zeros(self.state_dim, dtype=bool)\n",
    "\n",
    "        s, count, best_action = start_coord, 0, 0\n",
    "        while best_action != 4 or count < self.state_dim[0]*self.state_dim[1]:\n",
    "            count += 1\n",
    "            qualities = np.zeros(5)\n",
    "            for a in self.actions_allowed[s]:\n",
    "                new_s = (self.action_coords[a][0] + s[0], self.action_coords[a][1] + s[1])\n",
    "                qualities[a] = self.R[new_s] + gamma*values[new_s]\n",
    "            best_action = np.argmax(qualities)\n",
    "            new_s = s[0] + self.action_coords[best_action][0], s[1] + self.action_coords[best_action][1]\n",
    "            ax.plot([s[0]+0.5, new_s[0]+0.5], [s[1]+0.5, new_s[1]+0.5], c='black', lw=3)\n",
    "            s = new_s\n",
    "\n",
    "        plt.tight_layout()\n",
    "\n",
    "        return fig, ax"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Figure size 864x432 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEYCAYAAABcGYHrAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAW1UlEQVR4nO3dfbBcdXnA8e9DSMK9hPAWobw2hKKC1kGM1FqLWKxEZUCqjBY6RE2JM31R2zqCQ8eacWqJbaHYjsUYtcpLbIm04AtKoAW1Eki0BINBEEEMEKDyEl6bl/v0j3Ou3IS7S2727G/v3vv9zJzZs+fsefa5e3f32fP6RGYiSdIuvU5AkjQ+WBAkSYAFQZJUsyBIkgALgiSptmuvE9hRhx9+uIdDSVID7rrrrhh1Rmb2xTBnzpxs2mSO2a24xmxev+TaTzH7KdcuxRz1e9ZNRpIkwH0IkqSaBUGSBFgQJEk1C4IkCbAgSJJqFgRJEmBBkCTVxnVBiIiFEbE6IlZv3Lix1+lI0oTW84IQEVe3mpeZSzJzbmbOnTlzZsm0JGnSKXIto4g4ptUs4OgSOUiS2it1cbtVwA1UBWB7exXKQZLURqmCsA54X2beuf2MiPh5oRwkSW2U2ofwsTbP9aeFcpAktVFkDSEzl7eZvXeJHCRJ7fX8KCNgUa8TkCSVO8ro1lazgP1L5CBJai8yu9+ZMiIeBE4EHt1+FvC9zDzwhWLYQlPSRDBjaIjl993HOw46iCd36c1Gmp620AQ+B7yuxbzLdiRGn7WnG/cxuxXXmM3rl1z7KWZPc73kkkzIvPTS5mKOQc9baGbmgsz8bot5p5fIQZLGhS9+cdvbcWQ87FSWpMnhySfh29+uxm+4AZ56qrf5bMeCIEmlXH01TJtWjU+bVt0fRywIklTKl74ETzxRjT/xRHV/HCl16QpJmviGhuDhh0eft3kzXHvtttNWrID162Hq1FEXiQJHgY5kQZCkpixfDu98J0yZ8tymoZF23e4rd8oUePGLn/+4TZtg61bm7bdfd/JswU1GktSU006Diy6C6dPh2WfhmWe2HZ58ctvHP/XU8x/z7LPV8hddxNW77140fQuCJDUlAt73Pli1Cg47DAYGxrb8wEC13KpVVZwY/fyxbhnXBcEWmpL60lFHwdq1cPrpMDi4Y8sMDsIZZ8Btt1XL90CRghARe0bEeRFxe0T8oh7W1dNaNshJW2hK6lcDA7B0KVx22QsXhcFBWLYMPvtZ2G23MvmNotQawr9RXcfo+MzcNzP3Bd5QT7u8UA6SVN5JJ1U7j9uZMgXe+tYy+bRRqiDMzszFmblheEJmbsjMxcChhXKQpPK+850de9x3R726T1GlCsLPIuLDEfHLS11HxP4RcTZgC01JE9cll2x7iYrp07e9hWr+pZeWzWsUpQrCO4F9gRsi4pGIeAS4HtgHOK1QDpJU1tAQfOUr1S3A7rvDG98Id99d3Q4fVjo0BJdf/tzjeqTU1U4fzcyzM/OlmblPPRyZmWcDbyuRgyQVt3IlbN1aHT46OAgXXABf/SrMnl3dnn9+NT0CtmyBm27qabrj4bBTW2hKmpiWLatORpszB1avhrPOeu7cgghYuLCaPmdOtdlo2bKepmsLTUnqlpUrqyJw4YWtDyc98sjqnIUPfABuvLFsftuxhaYkTTKtWmiWurjd14AZmXnL9jMi4vpCOUiS2mnVW3O8Df3UrxVodOhGzG7F7VbMbvyf+iHmcFxf02Zj9lOuXYrZu57KkqTxz4IgSQIsCJKkmgVBkgRYECRJNQuCJAkodx4CEXE4cCpwCLAFuBNYlpmPl8pBktRaqY5p7wcuAnYDXg0MUBWGGyPi+BI5SJLaK7WGcBZwdGZujYjzgW9k5vER8RngSuCVoy0UEQuBhQCzZs0qlKokTU4l9yEMF5/pwB4AmXkvMLXVAmlPZUkqptQawlJgVUSsBI4DFgNExIuARwrlIElqo0hByMwLI+Ja4Ejg/My8vZ7+MFWBkCT1WLGjjDLzNuC2Us8nSRobz0OQJAEWBElSzYIgSQIsCJKkWpGeyk2wp7IkNaNVT+Wet8bc0aGf2tPRB+0Oh3Ptl5j98JraQrN/YvZTrl2KaQtNjTNDQ73OQNIIFgT1xi9+AYccAo94oro0XlgQ1BtXXQX331/dShoXLAjqjX/5l21vJfWcBUHlbdwIK1dW4ytXVvcl9Vyxaxlpknr0Ubj33m2nXXcdTJ8OmzbBtGmwdCmccMK2jzn0UNh773J5SipTECJiGvAu4P7MvDYiTgdeC6wDlmTm5hJ5qAfOOw8++UnYbbfqy3/YE088d7toUTVAVSSefRY+/GFYvLh8vtIkVmoN4Qv1cw1GxHxgBnAFcAJwLDC/UB4q7W/+BvbaCz7+8dabhkZOHxiAT3wCzj67TH6SfqlUQfj1zHxFROwK3AccmFU7zUuANa0WsoXmBLDLLvCRj1SbhE45BR57rFoD2N706bDPPnDllfDqV5fPU1Kxncq71JuN9gAGgT3r6dOxhebkcOyx8OMfwwEHjD7/wAPh9tstBlIPlVpD+BxwOzAFOBe4PCJ+CrwG+HKhHNRr06bBAw+MPm/DhmotQVLPFFlDyMwLgNcBv5mZnwLeDnwLWJCZi0rkoHFgxYrndiwPDlbjg4PV/alTq/mSeqbYeQiZeX9m3l+PP5aZyzPz5lLPr3Hg4ourHciDg7BgQXX5ive+t9qRvHFjNV9Sz3himsrYtAm+/nWYOROuuAI+9SmYMQP+8R+r+zNnVvM3ewSy1CsWBJWxdSvMnw933AEnnrjtvHnzqulnnglbtvQmP0meqaxCBgbg059uPX///dvPl9R1riFIkgBbaErSpGMLzVFM5naPw7n2S8x+eE1todk/Mfsp1y7FtIWmJKk1C4IkCbAgSJJqFgRJEmBBkCTVLAiSJKBQQYiI90fEISWeS5K0c0qtIXwcuCkivhMRfxQRLyr0vJKkHVSqIPwUOJiqMLwK+FFEfDMi5kfEHq0WioiFEbE6IlZvbNWPV5LUiFIFITNzKDOvycwFwIHAp4F5VMWi1UK20JSkQkpd7XSb62Zk5mbgKuCqiBgolIMkqY1SawjvbDUjM58plIMkqY1SPZXvKPE8kqSd53kIkiTAgiBJqlkQJEmABUGSVLOFpiRNMrbQHMVkbvc4nGu/xOyH19QWmv0Ts59y7VJMW2hKklqzIEiSAAuCJKlmQZAkARYESVLNgiBJAiwIkqRaqZ7KvxERM+vxgYhYFBFfjYjFEbFniRwkSe2VWkP4PPB0PX4hsCewuJ72hVYL2UJTksop1TFtl8zcUo/Pzcxj6vHvRsQtrRbKzCXAEvDSFZLUbaXWENZGxHvq8TURMRcgIl4MbC6UgySpjVIF4Q+B10fEXcBRwI0R8VPgs/U8SVKPFdlklJmPA++OiD2AOfXzrs/MB0s8vyTphZXahwBAZj4BrCn5nJKkHeN5CJIkwIIgSartcEGIiP+IiJMiwiIiSRPQWL7cnwL+FVgfEZ+IiCO6lJMkqQfG1FO5vvzEGcB7gLnAd4GlwOWZ+UxXMqx5YpokNaPxnsrAy4ALgGeAx4HPAEfubLwXGvqpXyl90P+2m7n2w9/fLzGH4/qaNhuzn3LtUszmeipHxIHAKcBJwBZgOXAIcGtEfGhnYkqSemssO5WnRsQ7IuIbwM+AtwGfBA7IzAWZ+RaqzUl/2Z1UJUndNJYT0x4AArgMOCczbx3lMSuAR5tITJJU1lgKwp9R7Tx+ttUDMvNR4LCOs5IkFbfDBSEzL+5mIpKk3vIkM0kSUPjidsMi4nXAscDazLymFzlIkrZVqqfyzSPGzwL+CdgD+KuIOKdEDpKk9kptMpo6Ynwh8LuZuQh4E9WhqqOyp7IklVOqIOwSEXtHxL5Ul8t4GCAzn6I6sW1UmbkkM+dm5tyZM2cWSlWSJqdS+xD2BL5PdR5DRsSvZOaGiJhRT5Mk9VipFpqzW8waAk4tkYMkqb2eHGU0LDOfBu7uZQ6SpIrnIUiSAAuCJKlmQZAkARYESVJtTC00e8kWmpLUjMZbaJYe+qk9HX3Q7rCfcp3MMYfj+n9qNmY/5dqlmM210JQkTTwWBEkSYEGQJNUsCJIkwIIgSapZECRJQA8LQkR8qVfPLUl6viJXO42Iq7afBLwhIvYCyMyTS+QhSWqt1OWvDwZ+BCylOjEmgLnA37dbKCIWUrXcZNasWV1OUZImt1KbjOZSdUw7F3g8M68HnsnMGzLzhlYLpS00JamYUh3ThoALIuLy+vbBUs8tSdoxRb+UM3M9cFpEvBXYWPK5JUnt9eRXemZ+Hfh6L55bkjQ6z0OQJAEWBElSzYIgSQIsCJKkmi00JWmSsYXmKGyhOXn//n6JORzX17TZmP2Ua5di2kJTktSaBUGSBFgQJEk1C4IkCbAgSJJqFgRJElDw4nYRcSyQmbkqIo4C5gG3Z+Y3SuUgSWqtVAvNvwLeDOwaESuA3wCuB86JiFdm5l+XyEOS1FqpNYR3AEcD04ENwMGZuTEi/ha4CRi1INhCU5LKKbUPYUtmbs3Mp4G7MnMjQGY+Awy1WihtoSlJxZQqCJsiYrAef9XwxIjYkzYFQZJUTqlNRsdl5v/BL/srD5sKzC+UgySpjSIFYbgYjDL9f4H/LZGDJKk9z0OQJAEWBElSzYIgSQIsCJKkmi00JWmSsYXmKCZzC8nhXPslZj+8pv3WQnMyx5zM7ylbaEqSXpAFQZIEWBAkSTULgiQJsCBIkmoWBEkSYEGQJNWKFYSIeGlEnBARM7abPq9UDpKk1ooUhIh4P3Al8KfA2og4ZcTsT5TIQZLUXqkGOWcBr8rMJyNiNrA8ImZn5oXA6KdQY09lSSqpVEGYkplPAmTmPRFxPFVR+FXaFITMXAIsAa9lJEndVmofwoaIOHr4Tl0cTgJmAb9eKAdJUhulCsKZwIaREzJzS2aeCRxXKAdJUhuleiqvbzPvv0vkIElqz/MQJEmABUGSVLMgSJIAC4IkqWZPZUmaZOypPIrJHLNbcY3ZvH7JtZ9i9lOuXYppT2VJUmsWBEkSYEGQJNUsCJIkwIIgSapZECRJgAVBklTreUGIiPf0OgdJ0jgoCMCiVjMiYmFErI6I1Rs3biyZkyRNOkX6IUTEra1mAfu3Wi5toSlJxZTqqbw/cCLw6HbTA/heoRwkSW2UKghfA2Zk5i3bz4iI6wvlIElqo1QLzQVt5p1eIgdJUnvjYaeyJGkcsCBIkgALgiSpZkGQJAG20JSkSccWmqOYzDG7FdeYzeuXXPspZj/l2qWYttCUJLVmQZAkARYESVLNgiBJAiwIkqSaBUGSBJS72ikR8VLgFOAgIIH7gasyc12pHCRJrRVZQ4iIs4EvU/U/uBlYVY8vi4hzSuQgSWqv1BrCAuBlmbl55MSIOB+4DThvtIUiYiGwEGDWrFndzlGSJrVS+xCGgANHmX5APW9UmbkkM+dm5tyZM2d2LTlJUrk1hA8C10XEncDP62mHAr8G/EmhHCRJbZTqmPbNiHgxcCzVTuUA1gOrMnNriRwkSe0VO8ooM4eAlaWeT5I0Np6HIEkCLAiSpJoFQZIEWBAkSTVbaErSJNP3LTTHMgALjTn+407mmP2U62SO2U+5NhFzom4yWmjMvog7mWN2K64x+yPuuIw5UQuCJGmMLAiSJGDiFoQlxuyLuJM5ZrfiGrM/4o7LmH1zlJEkqbsm6hqCJGmMLAiSJGCCFYSImBcRP46InzTVmjMiPh8RD0XE2ibi1TEPiYj/ioh1EXFbRHyggZi7RcTNEbGmjrmoiVzr2FMi4n8i4msNxbsnIn4YEbdExOomYtZx94qI5RFxe/3a/maH8V5S5zg8bIyIDzaQ55/V/6O1EbEsInZrIOYH6ni3dZLjaO/3iNgnIlZExJ317d4NxDytznUoIuY2lOff1v/7WyPi3yNirwZifryOd0tEXBMRozX6GnPcEfM+FBEZEWNqCdki149FxH0j3q9vGWuujZ/E0asBmALcBcwBpgFrgKMaiHsccAywtsFcDwCOqcf3AO7oNFeqHhMz6vGpwE3AaxrK98+By4CvNRTvHmBWF94DXwT+sB6fBuzV8PtrA/CrHcY5CLgbGKjv/xvw7g5jvhxYCwxSXdL+WuCInYz1vPc78EngnHr8HGBxAzGPBF4CXA/MbSjPNwG71uOLG8pz5ojx9wMXNZFrPf0Q4FvAz8b6eWiR68eAD3XyXppIawjHAj/JzJ9m5ibgy8ApnQbNzG8Dj3QaZ7uYD2TmD+rxJ4B1VF8UncTMzHyyvju1Hjo+YiAiDgbeCiztNFY3RcRMqg/J5wAyc1NmPtbgU5wA3JWZP2sg1q7AQETsSvUlfn+H8Y4EVmbm05m5BbgBOHVnArV4v59CVWypb9/WaczMXJeZP96ZHNvEvKb++6HqvXJwAzE3jri7OzvxmWrzHXIB8OGGY3ZkIhWEg3iuPSdUHdk6+pItISJmA6+k+kXfaawpEXEL8BCwIjM7jgn8A9WbtmXv652QwDUR8f2IaOqMzTnAw8AX6s1bSyNi94ZiA7wLWNZpkMy8D/g74F7gAeDxzLymw7BrgeMiYt+IGATeQvXrsyn7Z+YDUP2YAfZrMHa3vBe4uolAEfHXEfFz4Azgow3FPBm4LzPXNBFvhD+pN3F9fqyb9mBiFYTRLtY0ro+pjYgZwFeAD273S2SnZObWzDya6pfRsRHx8g7zOwl4KDO/32lu2/mtzDwGeDPwxxFxXAMxd6Vahf7nzHwl8BTV5o2ORcQ04GTg8gZi7U31i/sw4EBg94j4g05iZuY6qk0kK4BvUm0u3dJ2oQksIs6l+vsvbSJeZp6bmYfU8TruAV8X7XNpqLiM8M/A4cDRVD82/n6sASZSQVjPtr+KDqbzVfGuiYipVMXg0sy8osnY9aaS64F5HYb6LeDkiLiHahPc70TEJR3GJDPvr28fAv6danNfp9YD60esFS2nKhBNeDPwg8x8sIFYbwTuzsyHM3MzcAXw2k6DZubnMvOYzDyOalPCnZ3GHOHBiDgAoL59qMHYjYqI+cBJwBlZb1hv0GXA2xuIczjVD4I19WfrYOAHEfErnQTNzAfrH4VDwGfZic/VRCoIq4AjIuKw+hfdu4CrepzTqCIiqLZ1r8vM8xuK+aLhoyoiYoDqi+f2TmJm5kcy8+DMnE31ev5nZnb0azYido+IPYbHqXYEdnwEV2ZuAH4eES+pJ50A/KjTuLXfp4HNRbV7gddExGD9PjiBah9SRyJiv/r2UOD3aC5fqD5H8+vx+cCVDcZuTETMA84GTs7MpxuKecSIuyfT4WcKIDN/mJn7Zebs+rO1nuogkw2dxB0u2rVT2ZnPVSd7pMfbQLXt9A6qo43ObSjmMqrVr831P25BAzFfR7U561bglnp4S4cxXwH8Tx1zLfDRhl/b42ngKCOqbf1r6uG2pv5PdeyjgdX1a/AfwN4NxBwEfgHs2WCei6i+WNYCFwPTG4j5HaoCuAY4oYM4z3u/A/sC11GtdVwH7NNAzFPr8f8DHgS+1UDMn1DtRxz+TI3piKAWMb9S/59uBb4KHNTEa7rd/HsY+1FGo+V6MfDDOtergAPGmquXrpAkARNrk5EkqQMWBEkSYEGQJNUsCJIkwIIgSapZECRJgAVBklSzIEiSAAuC1JH6kiEPRMRHR0x7RUQ8GxHv6GVu0lh5prLUoYg4keqyBq+numTCauDmzHxPTxOTxsiCIDUgIv6B6uJnNwC/DRydzzUskvqCBUFqQERMp7qw3BHAa7OZ5kRSUe5DkJoxm6ofR1Jd0VXqO64hSB2qmx3dSHV56Juomp2/IjPv7WVe0lhZEKQORcR5wOlUPSkep+rlOwC8IavuVVJfcJOR1IGIeD3wF8CZmflYVr+w3g0cSdW9S+obriFIkgDXECRJNQuCJAmwIEiSahYESRJgQZAk1SwIkiTAgiBJqlkQJEkA/D8HB3PuKBtm0AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 396x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "world_size = (16,12) # dimension of the gridworld\n",
    "\n",
    "#start_cell = (1,1)\n",
    "\n",
    "obstacles = [(2,2), (2,3), (2,4), (2,5), (2,6), (2,7), (2,8), (2,9), \n",
    "             (3,9), (4,9), (5,9), (6,8), (6,7), (6,6), (5,5), (4,5),\n",
    "             (4,4), (5,3), (6,2),\n",
    "             (9,2), (9,3), (9,4), (9,5), (9,6), (9,7), (9,8), (9,9), \n",
    "             (10,2), (11,2), (12,2), (13,2)]\n",
    "\n",
    "rewards = [((5, 8), 1), ((14, 10), 2)]\n",
    "\n",
    "gridworld = Gridworld(world_size, rewards, obstacles) # Building the world\n",
    "plt.figure(figsize=(12, 6))\n",
    "fig, ax = gridworld.display(figsize=(5.5,4)) # And showing it\n",
    "#ax.legend(fontsize=14)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def val_iter_gridworld(gridworld, gamma, delta_conv=0.001, max_time=1000):\n",
    "    \"\"\"\n",
    "    Value iteration algorithm applied to Gridworld. It returns the computed value \n",
    "    function as a 2d array.\n",
    "    \"\"\"\n",
    "    values = np.zeros(gridworld.state_dim)\n",
    "    for t in range(max_time):\n",
    "        old_values = copy(values)\n",
    "        delta_v = 0\n",
    "        for s in gridworld.states:\n",
    "            qualities = np.array([])\n",
    "            for a in gridworld.actions_allowed[s]:\n",
    "                new_s = (gridworld.action_coords[a][0] + s[0], gridworld.action_coords[a][1] + s[1])\n",
    "                qualities = np.append(qualities, gridworld.R[new_s] + gamma * old_values[new_s])\n",
    "            values[s] = qualities.max()\n",
    "            delta_v = max(delta_v, abs(values[s] - old_values[s]))\n",
    "        \n",
    "        if delta_v < delta_conv:\n",
    "            break\n",
    "            \n",
    "    return values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "gamma = 0.75\n",
    "\n",
    "values = val_iter_gridworld(gridworld, gamma)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<Figure size 396x288 with 2 Axes>,\n",
       " <matplotlib.axes._subplots.AxesSubplot at 0x262b10030b8>)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEYCAYAAABfgk2GAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deZxcZZ3v8c+3OwshkABZyE4WEVHBgLmoOIMiLoAMoiNXRUdkkOhrFNGRq3i547i8dEBnUGbuuARxR1QQFBEU5MoqW1CWQIIQErInhCUhAZJ01+/+cU6g0/Zyqs6pU9VV33de59XVVXV+56nuyq+efs5znp8iAjMza30djW6AmZmVwwnfzKxNOOGbmbUJJ3wzszbhhG9m1iaGNboBWc2ZM8fTicysJkuXLlXeGN1clDkHdfLe3MerhyGT8AGWLVuVa/9Zs6axfNm6XDFmzprE8uUb88WYOZ4Vj27OFWPGfmNYvXJbrhhTp49kw5ra3wITp3QB8PT6CTXH2HPfxwCIDS+pOYYmLgFg/NrX1Rxj4+QbOOixY2veH+C+CVfxd88elyvGr0ddyVkjjsoV45zt13HZnDm5Yrxj6VIW/1O+/ysHfmMSf1n6hVwxXjzns4W0wxJDKuGbmTVKpdKd+bmdTTpY7oRvZpZBRFejm5CbE76ZWQYR2Xv4zcoJ38wsg+7Kc41uQm5O+GZmGXhIx8ysTUTFCd/MrD24h29m1h48pFNnkuYD8wHGjx/f4NaYWVur7Gh0C3Jr+OUBkq7u77GIWBAR8yJi3pgxY8pslpnZLiK6Mm/NqpQevqRD+3sImFtGG8zMcvFJ28zuBG4gSfC97VVSG8zMaueEn9li4EMR8VDvByStLKkNZmY1U7cvvMrqc/R/vuD0ktpgZlY79/CziYhLB3h47zLaYGaWSwsk/IbP0gE+3+gGmJkNRtGVeWtWZc3Sube/h4B9y2iDmVkuVayH36wUUf/KgZLWA28Bnuz9EPDHiJgyWAyXODRrLXtUKly6ejXvnDqVLR31HWwoosTh1hXvypyDRs/4WVuXOLwS2CMi7u79gKTrswZxecIX5C1PCEmJwlYpTwjkKlHo8oS7Kqo84UAxdNEtdLzvm/z5S8cSJx1eU4ys7ShEC/TwSxnDj4hTI+Lmfh47qYw2mFlz0Q9uSr/e2OCWZFTpyr41qaZeS8fMWtSW5+DG5K87bngQtj4Ho3drbJsGIffwzcxqcPU9MCLtb47ohKv7m9fRPNS1PfPWrJzwzax0+uFN6OnkylU9/Rz64U0NblEGle7sW5PykI6ZFa9SYVxXF6zf9NeP7ehCv79/l7t07SJi1eMwfNeU9HyMCXtCnWfyDKYVhnSc8M2scLr0Du5YsYKYenoyZNPbsF7Ju7ODjhef+VdP++OzO+ic9BG6f346nPiqOrU2owITvqQDgJ/1uGs28NmI+HphB+mDh3TMrHBx4qs4e9w4GDkMntuBnu21bdl1WrK2bvur5/BcFzskKt86Bd55WINeSY82Vrozb4OJiAcjYm5EzAVeCTwDXF7v1+CEb2bFk/jp2LFU7vwizJpIjBpR1e4xagTMmsAJU6cSHzoK1ATXMdVvDP8oYGlEPFqHVu+iqRO+pPmSFkpauHlzvoudzKwBXjqVyqJziJNeQ+yeLenH7iOI9x5O5f5zeHhEdR8U9aRKJfvWI3el2/wBQr8buLiM11BKwpc0VtI5kpZIejzdFqf39VsAxSUOzVrAqBHEd06j8pOPDJr0Y/cRVC7+CHHBB2G35kn2QFU9/J65K90W9BVS0gjgeOCSMl5CWT38n5Oso/P6iBgXEeOAI9P7SnmhZtZgxx0CnYOknM4OeOsh5bSnWvUZ0jkG+FNErK9Tq3dRVsKfGRHnRsTzC3xExLqIOBeYUVIbzKyRblqS7Xk3P1jfdtRIUcm8VeE9lDScA+Ul/EclfUrS80shS9pX0qcBlzg0awP68S2w9YXZOTFyGJF+fd7WbeiiW8pvXBZdO7JvGUjaHXgTcFld291DWQn/XcA44AZJT0h6Arge2Ac4saQ2mFmjVCroF3eiSrLCcIweSbzx5VSWfY1448uJ0SMBUCXQJXdApapecjkqlexbBhHxTDrE3cfVafVR1mqZT0bEpyPiJRGxT7odGBGfBk4oow1m1kC3PQzdFUJKZuF87X3Erz8JMycQv/4kcd57k/sl6KrA7Usb3eK/UuQ8/EZphmmZLnFo1uJ08a3JCpmzJ1BZ+EXitCNfmFsvEfPfQGXhF2H2BNj6XPL8ZlNwD78RXOLQzOpOtz1MnHYkcf4/9D/d8sBkzr7O+BG69SGarsRdEyfyrFzi0MxaXhElDrt+NyNzDhr2lhVNcGnwXxtSJQ7NzBqmicfmsyol4UfEqQM8lrnE4bJlq3K1Y9asaYXUxV26NN8JpTlz5rREjDlp7dVWqYtbxM/TdXFfUFRd3CLaUQS1wJCOl0c2M8vCCd/MrE10NW9x8qyc8M3MsqgM/XkjTvhmZll4SMfMrE044ZuZtQkP6WQnaQ7wdmA60AU8BFxc5sJBZmY1q27Z46ZUVsWrjwHfAnYD/gcwiiTx3yrp9WW0wcwsl0pk35pUWT3804C5EdEt6Tzgqoh4vaRvA78C+ixxk9aBnA8wfvz4kppqZtaHJk7kWZW5WubOD5eRwJ4AEbECGN7fDq5pa2ZNwz38zL4D3CnpNuAI4FwASROAJ0pqg5lZzVpgCL+0tXTOl/R74EDgvIhYkt7/GMkHgJlZc+tq3p57VqXN0omI+4H7yzqemVmh3MM3M2sTQ7+D3xQlDs3Mml5UlHnLQtJeki6VtETSYkmvqfNLcA/fzCyT4od0zgd+GxHvlDQC2L3wI/TihG9mlkXGnnsWksaQTFj5AEBEbAe2F3aA/o5bRk3bIrimrZnVqoiattu/tnvmHDTiE88MeDxJc4EFwAPAK4C7gDMiYmuuRg5iSPXwlz2yMtf+s2ZPL6RMYhGl8E6LY3LFuEBX863xr8wV48Mb78oV48Mb7wJomTKJRfxeD3rs2Fwx7ptwlcsk9nDgNyYV0o5CVNHD77lKQGpBRCzo8f0w4FDg9Ii4XdL5wFnAvxTR1P4MqYRvDRABKu5PWbMhK7L/P0iT+4IBnrIKWBURt6ffX0qS8OvKs3SsX8N3bONv776SYV11H1o0a3rR1Zl5GzRWxDpgpaQD0ruOIhneqSv38K1fE55aw247nmPCk2tYO2Fmo5tj1lgFnrRNnQ5clM7QeQQ4pegD9OaEb/2avHE5AFM2LnPCt7YXVQzpZIsXdwPzCg06CCd861Nn9w722pKsa7fXlifo7N5Bd2e/C5uatb7K0B8Bd8I3hnVtZ7ftz+xy3z6bNlBRBx1RoaIOpm5YxhNjJ+7ynDHd3WzuHHy80qwVZL2CtpmVkvDTMap3A2si4veSTgIOBxaTTFfaUUY7rG8z1yxh1roH6VYHlY60FxMwrNIFJF9nr76f2WuSte86KhU6o8KHx47lK+PGNarZZuVyws/se+mxdpd0MrAHcBnJmenDgJNLaof14eHpB9E1bDiz1yxmeHdXn88ZXnnh/u6OTh6a+nK+Oryu14iYNZWix/AboayEf1BEHCxpGLAamJKWO/wxcE9/O7nEYUkklk85kCfG7Mvch25hWNd2Ovuo9tCtDrqGjeDu/V/L5j32IdILr8zaQguM4Zf1CjrSYZ09SRYIGpvePxKXOGwam/fYh1sOPpptI0b1+fi2EaO45eCj2bzHPiW3zKzxil4tsxHK6uFfCCwBOoGzgUskPQK8GvhpSW2wDEIdjNz+bJ+Pjdz+HBUN/V6OWS2ie+hPUCirxOHXJP0svb1G0g+BNwIXRMQdZbTBstln03pCHRAVujo66YigIjGs0k1IjNu0no17T2l0M81K18w996zKLHG4psftp0jWjrAmM3njo3RWuuju6GT1hFksnXYQL1p1H1MfW0ZnpYvJjz/qhG9tySdtraWoUmHCprV0dwzj3he9hsf3mgTAg/sdwsaxkzn44VsZ/9RaVKkQHR7asTbTAidtnfDteSJYM34mj0x9KduH77bLY4/vNYlbXnEMs1c/gIhWKO9pVhUP6VhLqXR0smTmof0+vn34bgM+btbKPKRjZtYuWmBIxyUOzazlFVHi8MmP7J85B+393w815Z8DQ6qH/8gjK3LtP3v2jELKJBZRCu91m/KVsbth7JUNL5NYZInDDWtqfytOnJIs+5C3TGIRv9fxa1+XK8bGyTe4TGIP71i6tJB2FMFDOmZmbcInbc3M2oSvtDUzaxMe0jEzaxMe0jEzaxMRQ39aphO+mVkGRffwJS0Hnga6ga6IqHtB87JKHH4MuDwi8s2JNDNrkDqN4R8ZERvrEbgvZf2N8kXgdkk3SfonSbVPmDYza4AIZd6aVVkJ/xFgGknifyXwgKTfSjpZ0p797SRpvqSFkhZu3ry5pKaamf21OlS8CuAaSXel5VzrrqyEHxFRiYhrIuJUYArwDeBokg+D/nZyiUMzawoRHZm3np3VdOsrob82Ig4FjgE+IumIer+Gsk7a7vKRFxE7gCuAKyT1XUDVzKyJVLqz948jYgGwYJDnrEm/bpB0OXAYcGOeNg6mrB7+u/p7ICL6LqBqZtZEihzDlzR653C2pNHAm4FFdX4JpdW0/UsZxzEzq5eCp2XuC1wuCZI8/JOI+G2RB+iL5+GbmWVQ5OybiHgEeEVhATNywjczy8BX2pqZtYlKE8+vz8oJ38wsg1ZYPM0lDs2s5RVR4vAvxx+ROQe9+Iobm/LTYUj18IsocVhEjCJK4c1e9+ZcMR6ZdE3DyyReoKuBYkocrl65reYYU6ePBPKXSSzi9xobXpIrhiYucZnEHs7Zfl0h7ShCMy+ZkNWQSvhmZo3ihG9m1ia6m6TEoaRxwBNRw3j80J9nZGZWgkaulilpuKQvS3oKWA/MSu//N0kfzhrHCd/MLIMGL4/8L8DfA6cCPU943QWckjWIE76ZWQaVUOatDk4CPhQRvwAqPe6/DzggaxCP4ZuZZdDgk7ZTgeV93N9JFXncPXwzswwaPKTzAPC3fdx/IvDnrEHKqmn7KmBxRGxO178/CziU5EV8OSI2ldEOM7NaNXhphS8A35c0haSj/g5JBwDvB/4ua5CyevjfBZ5Jb58PjAXOTe/7Xn87ucShmTWLRvbwI+JXwHuB40mGcb4EHAScEBHXZI1T1hh+R0R0pbfnpWW9AG6WdHd/O/WsGjNnzpzYuPGpOjfTzKxvjb7wKiKuAq7KE6OsHv4iSTunDt0jaR6ApBcDO0pqg5lZzborHZm3ZlVWD/+DwPmS/g+wEbhV0kpgZfqYmVlTa2QPX9KTQL9X1kbEPlnilFXicBPwgbSG4+z0uKsiYn0Zxzczy6vBJ23P7PX9cOAQ4ATg37IGKXUefkQ8DdxT5jHNzIrQyB5+RFzY1/2SFgKZl1f1hVdmZhk0acWr64Dzsj7ZCd/MLIOgKRP+icDjWZ+cOeFL+iXwHeCqiKgM9nwzs1ZSp1UwO4GFwOqI6LfSi6Q/s+tJWwGTgAnAR7Mer5oe/lbgZ8AmSd8HvhcRD1Wxv5nZkFWnIZ0zgMXAmEGe17tsVwV4DPhDRNyf9WBV1bSVNIbkaq9TgHnAzSS9/ksi4tnMgWrgmrZmVqsiatpe9+oTM+ego267ZNDjSZoG/IDkqtl/HqiHX5SqxvAjYjPwTeCbkl5GMof+28B/Sfop8PWIWFx8MxOtVNN29PpX5Yqxdd/bC2lHEfVoly/fWHOMmTPHA7Di0dqXzpixX9I5ylsXt4if59PrJ+SKsee+j7kubg+/HnVlIe0oQjU9fEnzgfk97lqQrhzQ09eBTwF75m9dNjWdtE0X8HkbcBzQBVwKTAfulfSZiPj34ppoZtZ43ZH9Ctqey8L0RdJxwIaIuEvS6/t5zoAXW/U6XrEXXkkaTpLk/xF4E8mSnF8BLo6ILelz/ifJi3TCN7OWUvBJ29cCx0s6FtgNGCPpxxHxvh7P6X2xVW7V9PDXkpwZ/glwVkTc28dzrgWeLKJhZmbNpMiTthHxGeAzAGkP/8xeyb7fi63yqCbhf4Lk5Oxz/T0hIp4kLa5rZtZKGr1aZhEyJ/yI+FE9G2Jm1szqdfFRRFwPXD/Qc9Ih9bOA9wAzgBG9Yozoa7/emncdTzOzJtLgEodfAE4D/pukAMrZJFPiN5HM5c+kIUsrSPob4DBgUTXVWszMGqXBa+m8C/hQRFwt6RzgsohYKukB4EiS6fKDKqWHL+mOHrdPA/4vydzTf5V0VhltMDPLI1DmrQ4mATuvqN0C7JXevgp4S9YgZQ3pDO9xez7wpoj4PPBmkit3++SatmbWLCqhzFsdrAQmp7eXkkyNh2SkpN+JNL2VlfA7JO0taRzJcg6PAUTEVpILt/oUEQsiYl5EzBszZrClJszM6qc7lHmrgyt4Icn/F/BFSQ+RLM3wvaxByhrDHwvcRTKPPyRNioh1kvZI7zMza2qNGMOXdFREXBcR/2vnfRHxM0mrgcOBv0TEL7PGK6vE4cx+HqoAby+jDWZmeVSxzmSRrpW0HLiQZIXiNUlb4maSxSur0tBpmRHxTEQsa2QbzMyyqKDMW4FeBlwGnA48Kuk3kk5I19Gvmufhm5ll0Ih5+BGxOCLOBKaRTM0M4BJgtaRzJR1QTTwnfDOzDBo5SyciuiLisnTN/P2A/wTeATwg6cascZzwzcwyiCq2urYjGcf/BknSf4pk5c1MXMTczCyDBl9pC4CkN5IsUX8Cyfz7i0mWWMi2fzUlDhvJJQ7NrFZFlDj8zktPy5yDPvjABYV9OkiaQVJW9gMkwzk3kiT5SwdavbgvQ6qHX0R5wmWPrMwVY9bs6YWUwtu2Yb9cMUZOfLSQdixbtqrm/WfNmgbA8mXrao4xc9akJEYTlEks4ue5YU2+/1ITp3S5TGIP9024qpB2FKG7Uv4IuKRrSdbK2UBykdWFEfFwrfGGVMI3M2uUBg0xPEtycvY3EdGdN5gTvplZBo0Yw4+I44uM54RvZpZBvQqglMkJ38wsg7YqcWhm1s7cwzczaxOt0MNv2JW2kn7YqGObmVWrEtm3wUjaTdIdku6RdL+kz9f/FZTUw5d0Re+7gCMl7QXFn4k2MytawdMytwFviIgtkoYDN0u6OiJuK/YwuyprSGca8ADJ1WFBkvDnAf8x0E6S5pOURGT8+PF1bqKZWf+KnJYZyRIHW9Jvh6db3af6lzWkM4+k4tXZwKaIuB54NiJuiIh+L4NziUMzaxbVlDjsWY873eb3jiepU9LdJFfRXhsRt9f7NZRV8aoCfE3SJenX9WUd28ysCNXM0omIBcCCQZ7TDcxNh7Yvl/TyiFiUp42DKTXpRsQq4ERJbwVqX/jEzKxk9VpnMiKeknQ9cDRQ14TfkFk6EfGbiPjfjTi2mVktiixxKGnCzkkrkkYBbwSW1PkleFjFzCyLLNMtqzAZ+EFam7YD+HlEXFnoEfrghG9mlkGRQzoRcS9wSHERs3HCNzPLIMtQTbNzwjczy2CIFAcckEscmlnLK6LE4dkzPpY5B31pxX825Z8DQ6qHX0R5wjwl/SAp61dEKbwn1+W7kGzvSZtZtzpXCCZNJddrmTNnDkDLlEks4ve6euW2XDGmTh/pMok9bJx8QyHtKEJ3C3Q5h1TCNzNrlEZUvCqaE76ZWQZDZPR7QE74ZmYZuACKmVmbcA/fzKxNuIdvZtYmCl5aoSFKS/iSDiNZ9/9OSS8lWRluSURcVVYbzMxq1QL5vrQSh/8KHAMMk3Qt8CrgeuAsSYdExJfKaIeZWa3cw8/uncBcYCSwDpgWEZslfRW4Hegz4bvEoZk1i1Y4aVvWevhdEdEdEc8ASyNiM0BEPMsA50Jc4tDMmkVXZN+aVVkJf7uk3dPbr9x5p6SxtMbJbzNrcVHF1qzKGtI5IiK2wfP1bXcaDpxcUhvMzGrmMfyMdib7Pu7fCNS+4pWZWUlaYQzf8/DNzDJohbFnJ3wzsww8pGNm1iZaIN+XNkvHzGxIq0T2bTCSpkv6g6TFku6XdEb9X4FLHJpZGyiixOFJe5+ROQf95MnzBzyepMnA5Ij4k6Q9gbuAEyLigZzNHNCQGtIpojzhsmVrcsaYUkgpvCLKE65a8WyuGNNmjOLR5U/UvP9+M/cBWqdMYhG/V8fYNUYR5QmLKNdYhK4CO8cRsRZYm95+WtJiYCpQ14TvIR0zswyqufBK0nxJC3ts8/uLK2kmcAjJMjN1NaR6+GZmjVLNLJ2IWAAsGOx5kvYAfgF8fOeSM/XkhG9mlkEUPE9H0nCSZH9RRFxWaPB+OOGbmWVQ5Dx8SQIuBBZHxHnFRR6Yx/DNzDKoVLFl8FrgH4A3SLo73Y4tvtW7cg/fzCyDIqewR8TNQO6potVywjczy6AV1tIpbUhH0kskHZWele55/9FltcHMrFYRkXlrVqUkfEkfA34FnA4skvS2Hg9/uYw2mJnl0RWReWtWZQ3pnAa8MiK2pBcZXCppZkSczwDjWK5pa2bNouhpmY1Q1pBOZ0RsAYiI5cDrgWMknccACd81bc2sWRQ8S6chykr46yTN3flNmvyPA8YDB5XUBjOzmlWIzFuzKmtI5/1AV887IqILeL+kb5fUBjOzmlWaeGw+q7Jq2va7FGJE3FJGG8zM8miFMXzPwzczy6CZh2qycsI3M8vACd/MrE14SMfMrE20Qg/fNW3NrOUVUdP2laNPzpyD7tr6g9IXRstiSPXwi6hHu3z5hlwxZs6cmKsOLCS1YBtdj3ZnO/L8PGbOnAjk+73MmjUljZG/pm3eGHlq4kJSF3f58o35Yswcz4pH8xU+mrHfGFav3JYrxtTpI9mwJl96mDilq5B6tEW0owjR1JdUZTOkEr6ZWaO0wpCOE76ZWQYVuYdvZtYWKh7SMTNrD074ZmZtwidtzczaRCuM4ZdW4tDMbCirVPFvMJK+K2mDpEUlNP15DU/4kk5pdBvMzAbTzY7MWwbfB0qv593whA98vr8HJM2XtFDSws2b812MYmaWR5E9/Ii4Ech35WQNShnDl3Rvfw8B+/a3X0QsABZAsrTC449vqUPrzMwGF3Rnfm7PetypBWk+a6iyTtruC7wFeLLX/QL+WFIbzMxqVs20zJ6d1WZSVsK/EtgjIu7u/YCk60tqg5lZzTwtM6OIOHWAx04qow1mZnlUqhjSaVbNcNLWzKzpBZXM22AkXQzcChwgaZWkfjvFRfKFV2ZmGVSiuB5+RLynsGBVcMI3M8vAY/hmZm2immmZzcolDs2s5RVR4nCf0YdkzkFPbP2zSxzmlaeEHSRl7FqpTGIR7Wia8oSPrKw9xuzphcQo4v3lMokvKKpMYhHtKEIUOIbfKEMq4ZuZNYrXwzczaxMRTvhmZm2hFU7aOuGbmWXgHr6ZWZvwPHwzszbhWTpVkPQS4G3AVCCANcAVEbG4rDaYmdWqFYZ0Slk8TdKngZ+SrH9/B3BnevtiSWeV0QYzszwq0ZV5a1Zl9fBPBV4WEbsUe5R0HnA/cE5fO/WsGjN+/Ph6t9HMrF+tMIZf1vLIFWBKH/dPTh/rU0QsiIh5ETFvzJgxdWucmdlgIrozb82qrB7+x4HrJD0E7Lz+fQbwIuCjJbXBzCyHod/DL6vi1W8lvRg4jOSkrYBVwJ3RzB+HZmapVjhpW9osnUh+WreVdTwzsyK1whi+5+GbmWXihG9m1h48pGNm1h48pGNm1i5aYH6JSxyaWcsrosShNDxzDorY0ZQlDomIltmA+Y7RejGaoQ2O0dox2mUr60rbssx3jJaM0QxtcIzWjtEWWi3hm5lZP5zwzczaRKsl/AWO0ZIxmqENjtHaMdrCkJmlY2Zm+bRaD9/MzPrhhG9m1iZaIuFLOlrSg5IerrVkoqTvStogaVGN+0+X9AdJiyXdL+mMGmLsJukOSfekMT5fS1vSWJ2S/izpyhr3Xy7pPkl3S1pYY4y9JF0qaUn6c3lNlfsfkB5/57ZZ0sdraMcn0p/nIkkXS9qthhhnpPvfn7UNfb2nJO0j6VpJD6Vf964hxolpOyqS5tXYjq+mv5d7JV0uaa8aYnwx3f9uSddI6qvI0YAxejx2pqSQNGBpu37a8TlJq3u8T44dKEZba/SFAAVcdNEJLAVmAyOAe4CX1hDnCOBQYFGN7ZgMHJre3hP4S7XtIKkTsEd6ezhwO/DqGtvzz8BPgCtr3H85MD7n7+YHwAfT2yOAvXL+ntcB+1W531RgGTAq/f7nwAeqjPFyYBGwO8lyJL8H9q/lPQV8BTgrvX0WcG4NMQ4EDgCuB+bV2I43A8PS2+fW2I4xPW5/DPhWtTHS+6cDvwMeHew91087Pgecmee92i5bK/TwDwMejohHImI7SbH0t1UbJCJuBJ6otRERsTYi/pTefhpYTJJsqokREbEl/XZ4ulV9Vl3SNOCtwHeq3bcoksaQ/Oe8ECAitkfEUzlCHgUsjYhHa9h3GDBK0jCSpL2myv0PBG6LiGciogu4AXj7YDv18556G8kHIenXE6qNERGLI+LBjG3vL8Y16WuBpE7FtBpibO7x7WgGea8O8H/sa8CnBtt/kBiWQSsk/Km8UDYRkkpaVSXaokmaCRxC0kOvdt9OSXcDG4BrI6LqGMDXSf4D5VneL4BrJN2VFpOv1mzgMeB76dDSdySNztGedwMXV7tTRKwG/h1YAawFNkXENVWGWQQcIWmcpN2BY0l6pbXYNyLWpm1bC0ysMU6R/hG4upYdJX1J0krgvcBna9j/eGB1RNxTy/F7+Gg6vPTdwYbJ2lkrJPy+Filq2FxTSXsAvwA+3qsHlElEdEfEXJIe12GSXl7l8Y8DNkTEXdUeu5fXRsShwDHARyQdUeX+w0j+9P5mRBwCbCUZwqiapBHA8cAlNey7N0mvehYwBRgt6X3VxIiIxSTDHtcCvyUZNuwacKchQtLZJK/lolr2j4izI2J6un9V9anTD8+zqeGDopdvAnOAuSQf6v+RM17LaoWEv4pde1vTqP5P9kJIGk6S7C+KiMvyxEqHP64Hjq5y19cCx0taTjK89QZJP67h+GvSrxuAy0mGzqqxCljV4y+US0k+AGpxDPCniFhfw75vBJZFxGMRsQO4DJE05gYAAALaSURBVDi82iARcWFEHBoRR5AMKTxUQ1sA1kuaDJB+3VBjnNwknQwcB7w3IvJ2kn4C/H2V+8wh+SC+J32/TgP+JGlSNUEiYn3aUaoAF1D9e7VttELCvxPYX9KstCf4buCKshshSSTj1Ysj4rwaY0zYOVtC0iiSZLWkmhgR8ZmImBYRM0l+Fv8vIqrq0UoaLWnPnbdJTvBVNXspItYBKyUdkN51FPBANTF6eA81DOekVgCvlrR7+js6iuT8SlUkTUy/zgDekaM9VwAnp7dPBn5VY5xcJB0NfBo4PiKeqTHG/j2+PZ7q36v3RcTEiJiZvl9XkUx8WFdlOyb3+PbtVPlebSuNPmtcxEYypvoXktk6Z9cY42KSPwd3kLzxTq1y/78hGUq6F7g73Y6tMsbBwJ/TGIuAz+b8ubyeGmbpkIy/35Nu9+f4mc4FFqav55fA3jXE2B14HBib4+fweZJktAj4ETCyhhg3kXxg3QMcVet7ChgHXEfyF8J1wD41xHh7ensbsB74XQ0xHiY597XzvTrYDJu+Yvwi/ZneC/wamFptjF6PL2fwWTp9teNHwH1pO64AJuf5f9PKm5dWMDNrE60wpGNmZhk44ZuZtQknfDOzNuGEb2bWJpzwzczahBO+mVmbcMI3M2sTTvhmZm3CCd+GpHQZirWSPtvjvoMlPSfpnY1sm1mz8pW2NmRJegvJJf2vI1keYCFwR0Sc0tCGmTUpJ3wb0iR9nWThrhuAvwXmxgtFZMysByd8G9IkjSRZ0Gx/4PCorWCMWVvwGL4NdTNJ6iEEySqfZtYP9/BtyEoLztxKstTw7STFrA+OiBWNbJdZs3LCtyFL0jnASSR1BDaR1GUdBRwZSfUjM+vBQzo2JEl6HfBJ4P0R8VQkPZcPAAeSVHIys17cwzczaxPu4ZuZtQknfDOzNuGEb2bWJpzwzczahBO+mVmbcMI3M2sTTvhmZm3CCd/MrE38f2VdfK1sMSyOAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 396x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "gridworld.display(values, cmap=cm.get_cmap('inferno'), figsize=(5.5,4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<Figure size 396x288 with 2 Axes>,\n",
       " <matplotlib.axes._subplots.AxesSubplot at 0x262b0cf81d0>)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEYCAYAAABfgk2GAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deZxcZZ3v8c+3OwtJIAGykJ0so4ijDmAubiOCuADDIDpyVXREB4m+BhFnxqt4ueOIvpyBWVCcO+pE0XFBVBAUGXBA7gCibEEDBBKELGRPCEtCAiTprt/945xAp+3qPlXn1Knqqu87r/Pq6qpznvpVd+VXTz/nOc9PEYGZmbW/rmYHYGZm5XDCNzPrEE74ZmYdwgnfzKxDOOGbmXWIEc0OIKv58+d7OpGZ1WXFihXK20Yvl2XOQd28N/fzNcKwSfgAq1aty3X83LkzWb1qU6425sydyurVW/O1MWcSax7dnquN2YeOZ/3aXbnamDFrNFs21P8WmDK9B4CnN0+uu40DDnkMgNjykrrb0JTlAEza+Ia629g67RZe/thJdR8PcP/k6/jTZ0/O1cbPxlzLeaOOz9XGhbtv4qr583O18Y4VK1j2l/n+rxz+lan8bsXncrXx4vmfKSQOSwyrhG9m1iyVSm/mfbtbdLDcCd/MLIOInmaHkJsTvplZBhHZe/itygnfzCyD3spzzQ4hNyd8M7MMPKRjZtYhouKEb2bWGdzDNzPrDB7SaTBJC4GFAJMmTWpyNGbW0Sp7mh1Bbk2/PEDS9dUei4hFEbEgIhaMHz++zLDMzPYR0ZN5a1Wl9PAlHVXtIeCIMmIwM8vFJ20zuxu4hSTB93dgSTGYmdXPCT+zZcCHI+Lh/g9IWltSDGZmdVOvL7zK6rNUP19wTkkxmJnVzz38bCLiykEePqiMGMzMcmmDhN/0WTrABc0OwMxsKIqezFurKmuWzn3VHgIOKSMGM7NcalgPv1UpovGVAyVtBt4KPNn/IeDXETF9qDZc4tCsvexfqXDl+vW8c8YMdnQ1drChiBKHO9e8K3MOGjf7hx1d4vBaYP+IWNL/AUk3Z23E5QlfkLc8ISQlCtulPCGQq0ShyxPuq6jyhIO1oct+Rdf7vspvv3AScfpr62ojaxyFaIMefilj+BFxZkTcVuWx08uIwcxai779y/TrrU2OJKNKT/atRbX0Wjpm1qZ2PAe3Jn/dcctDsPM5GLdfc2MagtzDNzOrw/X3wqi0vzmqG66vNq+jdahnd+atVTnhm1np9J1foqeTK1f19HPoO79sckQZVHqzby3KQzpmVrxKhYk9PbB52+8/tqcH/eKBfe7SjUuJdY/DyH1T0vNtTD4AGjyTZyjtMKTjhG9mhdOVd3HXmjXEjHOSIZv+RvRL3t1ddL34E7+326+f3UP31LPp/dE5cNqrGhRtRgUmfEmHAT/sc9c84DMR8aXCnmQAHtIxs8LFaa/i/IkTYfQIeG4PerbftmPfacnauev39uG5HvZIVL72QXjn0U16JX1irPRm3oYSEQ9FxBERcQTwSuAZ4OpGvwYnfDMrnsQPJkygcvfnYe4UYsyomg6PMaNg7mROnTGD+PDxoBa4jqlxY/jHAysi4tEGRL2Plk74khZKWixp8fbt+S52MrMmeOkMKksvJE5/DTE2W9KPsaOI976WygMX8sio2j4oGkmVSvatT+5Kt4WDNP1u4PIyXkMpCV/SBEkXSlou6fF0W5beV7UAikscmrWBMaOIb5xF5ftnD5n0Y+woKpefTXz9Q7Bf6yR7oKYeft/clW6LBmpS0ijgFOCKMl5CWT38H5Gso3NsREyMiInAcel9pbxQM2uyk4+E7iFSTncX/MmR5cRTq8YM6ZwI/CYiNjco6n2UlfDnRMRFEfH8Ah8RsSkiLgJmlxSDmTXTL5dn2++2hxobR50UlcxbDd5DScM5UF7Cf1TSJyU9vxSypEMkfQpwiUOzDqDv/Qp2vjA7J0aPINKvz9u5C132q/KDy6JnT/YtA0ljgTcDVzU07j7KSvjvAiYCt0h6QtITwM3AwcBpJcVgZs1SqaAf340qyQrDMW408aaXUVn1ReJNLyPGjQZAlUBX3AWVmnrJ5ahUsm8ZRMQz6RD3AFenNUZZq2U+GRGfioiXRMTB6XZ4RHwKOLWMGMysie54BHorhJTMwvni+4if/Q3MmUz87G+Ii9+b3C9BTwXuXNHsiH9PkfPwm6UVpmW6xKFZm9PltycrZM6bTGXx54mzjnthbr1ELHwjlcWfh3mTYedzyf6tpuAefjO4xKGZNZzueIQ46zjikj+vPt3y8GTOvs79Lrr9YVquxF0LJ/KsXOLQzNpeESUOe/5rduYcNOKta1rg0uDfN6xKHJqZNU0Lj81nVUrCj4gzB3ksc4nDVavW5Ypj7tyZhdTFXbEi3wml+fPnt0Ub89Paq+1SF7eIn6fr4r6gqLq4RcRRBLXBkI6XRzYzy8IJ38ysQ/S0bnHyrJzwzcyyqAz/eSNO+GZmWXhIx8ysQzjhm5l1CA/pZCdpPvB2YBbQAzwMXF7mwkFmZnWrbdnjllRWxauPAV8D9gP+BzCGJPHfLunYMmIwM8ulEtm3FlVWD/8s4IiI6JV0MXBdRBwr6d+BnwIDlrhJ60AuBJg0aVJJoZqZDaCFE3lWZa6WuffDZTRwAEBErAFGVjvANW3NrGW4h5/ZN4C7Jd0BHANcBCBpMvBESTGYmdWtDYbwS1tL5xJJvwAOBy6OiOXp/Y+RfACYmbW2ntbtuWdV2iydiHgAeKCs5zMzK5R7+GZmHWL4d/BbosShmVnLi4oyb1lIOlDSlZKWS1om6TUNfgnu4ZuZZVL8kM4lwM8j4p2SRgFjC3+GfpzwzcyyyNhzz0LSeJIJKx8AiIjdwO7CnqDa85ZR07YIrmlrZvUqoqbt7i+OzZyDRv3VM4M+n6QjgEXAg8AfAfcA50bEzlxBDmFY9fBXrVyb6/i582YVUiaxiFJ4Z8WJudr4uq7na5NemauNj2y9J1cbH9l6D0DblEks4vf68sdOytXG/ZOvc5nEPg7/ytRC4ihEDT38vqsEpBZFxKI+348AjgLOiYg7JV0CnAf8bRGhVjOsEr41QQSouD9lzYatyP7/IE3uiwbZZR2wLiLuTL+/kiThN5Rn6VhVI/fs4vVLrmVET8OHFs1aXvR0Z96GbCtiE7BW0mHpXceTDO80lHv4VtXkpzaw357nmPzkBjZOntPscMyaq8CTtqlzgMvSGTorgQ8W/QT9OeFbVdO2rgZg+tZVTvjW8aKGIZ1s7cUSYEGhjQ7BCd8G1N27hwN3JOvaHbjjCbp799DbXXVhU7P2Vxn+I+BO+MaInt3st/uZfe47eNsWKuqiKypU1MWMLat4YsKUffYZ39vL9u6hxyvN2kHWK2hbWSkJPx2jejewISJ+Iel04LXAMpLpSnvKiMMGNmfDcuZueohedVHpSnsxASMqPUDydd76B5i3IVn7rqtSoTsqfGTCBP5x4sRmhW1WLif8zL6VPtdYSWcA+wNXkZyZPho4o6Q4hoVPr/q3TPu9ZeXK3M+1TxtRgd6Brx+PNPkD9HZ18/CMl/FPIxt6jYhZSyl6DL8Zykr4L4+IV0gaAawHpqflDr8H3FvtIJc4bC296qJnxCiWvOh1bN//YCK98MqsI7TBGH5Zr6ArHdY5gGSBoAnp/aNxicNhY9eoMfzqFSewff+Dmx2KWemKXi2zGcrq4V8KLAe6gfOBKyStBF4N/KCkGIalf5h79oD3N2ppha5KL8fe8xO6B6jnNnr3c1Q0/Hs5ZvWI3uE/QaGsEodflPTD9PYGSd8B3gR8PSLuKiMGy+bgbZsJdUFU6OnqpiuCisSISi8hMXHbZrYeNL3ZYZqVrpV77lmVWeJwQ5/bT5GsHWEtZtrWR+mu9NDb1c36yXNZMfPl/MG6+5nx2Cq6Kz1Me/xRJ3zrSD5pa21FlQqTt22kt2sE9/3Ba3j8wKkAPHTokWydMI1XPHI7k57aiCoVostDO9Zh2uCkrRO+PU8EGybNYeWMl7J75H77PPb4gVP51R+dyLz1DyKiHcp7mtXEQzrWVipd3Syfc1TVx3eP3G/Qx83amYd0zMw6RRsM6bjEYQta2efq13nz5jUxErP2UESJwyfPflHmHHTQvz3ckn8ODKse/sqVa3IdP2/e7ELKJBZRCu8N26qXsVvJl5+/XW2/WyZc2/QyiUWWONyyof634pTpybIPecskFvF7nbTxDbna2DrtFpdJ7OMdK1YUEkcRPKRjZtYhfNLWzKxD+EpbM7MO4SEdM7MO4SEdM7MOETH8p2U64ZuZZVB0D1/SauBpoBfoiYiGFzQvq8Thx4CrIyLfnEgzsyZp0Bj+cRGxtREND6Ssv1E+D9wp6ZeS/lJS/ROmzcyaIEKZt1ZVVsJfCcwkSfyvBB6U9HNJZ0g6oNpBkhZKWixp8fbt20sK1czs9zWg4lUAN0i6Jy3n2nBlJfyIiEpE3BARZwLTga8AJ5B8GFQ7yCUOzawlRHRl3vp2VtNtoIT+uog4CjgROFvSMY1+DWWdtN3nIy8i9gDXANdIGlNSDGZmdav0Zu8fR8QiYNEQ+2xIv26RdDVwNHBrnhiHUlYP/13VHoiIZ0uKwcysbkWO4Usat3c4W9I44C3A0ga/hNJq2v6ujOcxM2uUgqdlHgJcLQmSPPz9iPh5kU8wEM/DNzPLoMjZNxGxEvijwhrMyAnfzCwDX2lrZtYhKi08vz4rJ3wzswzaYfE0lzhsQS5xaFasIkoc/u6UYzLnoBdfc2tLfjoMqx5+ESUOi2ijiFJ48za9perjK/naC89XZb+VU28YtExiFnnLJH5d1wPFlDhcv3ZX3W3MmDUayF8msYjfa2x5Sa42NGW5yyT2ceHumwqJowitvGRCVsMq4ZuZNYsTvplZh+htkRKHkiYCT0Qd4/HDf56RmVkJmrlapqSRkv5e0lPAZmBuev8/SPpI1nac8M3MMmjy8sh/C/wZcCbQ94TXPcAHszbihG9mlkEllHlrgNOBD0fEj4FKn/vvBw7L2ojH8M3MMmjySdsZwOoB7u+mhjzuHr6ZWQZNHtJ5EHj9APefBvw2ayNl1bR9FbAsIran69+fBxxF8iL+PiK2lRGHmVm9mry0wueA/5A0naSj/g5JhwHvB/40ayNl9fC/CTyT3r4EmABclN73rWoHucShmbWKZvbwI+KnwHuBU0iGcb4AvBw4NSJuyNpOWWP4XRHRk95ekJb1ArhN0pJqB/WtGjN//vzYuvWpBodpZjawZl94FRHXAdflaaOsHv5SSXunDt0raQGApBcDe0qKwcysbr2Vrsxbqyqrh/8h4BJJ/wfYCtwuaS2wNn3MzKylNbOHL+lJoOqVtRFxcJZ2yipxuA34QFrDcV76vOsiYnMZz29mlleTT9p+ot/3I4EjgVOBf8jaSKnz8CPiaeDeMp/TzKwIzezhR8SlA90vaTGQeXlVX3hlZpZBi1a8ugm4OOvOTvhmZhkELZnwTwMez7pz5oQv6SfAN4DrIqIy1P5mZu2kQatgdgOLgfURUbXSi6Tfsu9JWwFTgcnAR7M+Xy09/J3AD4Ftkv4D+FZEPFzD8WZmw1aDhnTOBZYB44fYr3/ZrgrwGPDfEfFA1ierqaatpPEkV3t9EFgA3EbS678iIp7N3FAdXNPWzOpVRE3bm159WuYcdPwdVwz5fJJmAt8muWr2rwfr4RelpjH8iNgOfBX4qqQ/JJlD/+/Av0r6AfCliFhWfJiJdqppO27zqwbZ44WEX22/nYfcWUgcRdSjXb16a91tzJkzCYA1j9a/dMbsQ5POUd66uEX8PJ/ePDlXGwcc8pjr4vbxszHXFhJHEWrp4UtaCCzsc9eidOWAvr4EfBI4IH902dR10jZdwOdtwMlAD3AlMAu4T9KnI+KfiwvRzKz5eiP7FbR9l4UZiKSTgS0RcY+kY6vsM+jFVv2er9gLrySNJEnyfwG8mWRJzn8ELo+IHek+/5PkRTrhm1lbKfik7euAUySdBOwHjJf0vYh4X599+l9slVstPfyNJGeGvw+cFxH3DbDPjcCTRQRmZtZKijxpGxGfBj4NkPbwP9Ev2Ve92CqPWhL+X5GcnH2u2g4R8SRpcV0zs3bS7NUyi5A54UfEdxsZiJlZK2vUxUcRcTNw82D7pEPq5wHvAWYDo/q1MWqg4/pr3XU8zcxaSJNLHH4OOAv4N5ICKOeTTInfRjKXP5OmLK0g6Y+Bo4GltVRrMTNrliavpfMu4MMRcb2kC4GrImKFpAeB40imyw+plB6+pLv63D4L+L8kc0//TtJ5ZcRgZpZHoMxbA0wF9l5RuwM4ML19HfDWrI2UNaQzss/thcCbI+IC4C0kV+4OyDVtzaxVVEKZtwZYC0xLb68gmRoPyUhJ1Yk0/ZWV8LskHSRpIslyDo8BRMROkgu3BhQRiyJiQUQsGD9+qKUmzMwapzeUeWuAa3ghyf8r8HlJD5MszfCtrI2UNYY/AbiHZB5/SJoaEZsk7Z/eZ2bW0poxhi/p+Ii4KSL+1977IuKHktYDrwV+FxE/ydpeWSUO51R5qAK8vYwYzMzyqGGdySLdKGk1cCnJCsUbkljiNpLFK2vS1GmZEfFMRKxqZgxmZllUUOatQH8IXAWcAzwq6T8lnZquo18zz8M3M8ugGfPwI2JZRHwCmEkyNTOAK4D1ki6SdFgt7Tnhm5ll0MxZOhHRExFXpWvmHwp8GXgH8KCkW7O244RvZpZB1LA1NI5kHP8rJEn/KZKVNzNxEXMzswyafKUtAJLeRLJE/akk8+8vJ1liIdvxtZQ4bCaXODSzehVR4vAbLz0rcw760INfL+zTQdJskrKyHyAZzrmVJMlfOdjqxQMZVj38IsoTrlq5Nlcbc+fNKqQU3q4thw6yxwsJv9p+o6c8Wkgcq1atq/v4uXNnArB61aa625gzd2rSRguUSSzi57llQ77/UlOm97hMYh/3T76ukDiK0FspfwRc0o0ka+VsIbnI6tKIeKTe9oZVwjcza5YmDTE8S3Jy9j8jojdvY074ZmYZNGMMPyJOKbI9J3wzswwaVQClTE74ZmYZdFSJQzOzTuYevplZh2iHHn7TrrSV9J1mPbeZWa0qkX0biqT9JN0l6V5JD0i6oPGvoKQevqRr+t8FHCfpQCj+TLSZWdEKnpa5C3hjROyQNBK4TdL1EXFHsU+zr7KGdGYCD5JcHRYkCX8B8C+DHSRpIUlJRCZNmtTgEM3MqityWmYkSxzsSL8dmW4Nn+pf1pDOApKKV+cD2yLiZuDZiLglIqpeBucSh2bWKmopcdi3Hne6LezfnqRuSUtIrqK9MSLubPRrKKviVQX4oqQr0q+by3puM7Mi1DJLJyIWAYuG2KcXOCId2r5a0ssiYmmeGIdSatKNiHXAaZL+BKh/4RMzs5I1ap3JiHhK0s3ACUBDE35TZulExH9GxP9uxnObmdWjyBKHkibvnbQiaQzwJmB5g1+Ch1XMzLLIMt2yBtOAb6e1abuAH0XEtYU+wwCc8M3MMihySCci7gOOLK7FbJzwzcwyyDJU0+qc8M3MMhgmxQEH5RKHLcglDs2KVUSJw/NnfyxzDvrCmi+35J8Dw6qHX0R5wjwl/SAp61dEKbwnN2W7kKzafgdN3c6m9bnCYOoMcr2W+fPnA7RNmcQifq/r1+7K1caMWaNdJrGPrdNuKSSOIvS2QZdzWCV8M7NmaUbFq6I54ZuZZTBMRr8H5YRvZpaBC6CYmXUI9/DNzDqEe/hmZh2i4KUVmqK0hC/paJJ1/++W9FKSleGWR8R1ZcVgZlavNsj3pZU4/DvgRGCEpBuBVwE3A+dJOjIivlBGHMPRk88sGfj+lQPePaCxo33xllle7uFn907gCGA0sAmYGRHbJf0TcCcwYMJ3iUMzaxXtcNK2rPXweyKiNyKeAVZExHaAiHiWQc6FuMShmbWKnsi+taqyevi7JY1NE/4r994paQLtcfK7UAeNPWLofYZYWuGZXTWM+ZjZkFo4j2dWVsI/JiJ2wfP1bfcaCZxRUgxmZnXzGH5Ge5P9APdvBepf8crMrCTtMIbvefhmZhm0w9izE76ZWQYe0jEz6xBtkO9Lm5ZpZjasVSL7NhRJsyT9t6Rlkh6QdG7jX4FLHLYtl0k0e0ERJQ5PP+jczDno+09eMujzSZoGTIuI30g6ALgHODUiHswZ5qCG1ZBOEeUJV63akLON6YWUwiuiPOG6Nc9m2rfafjNnj+HR1U/UHcOhcw4G2qdMYhG/V7exbxtFlCcsolxjEXoK7BxHxEZgY3r7aUnLgBlAQxO+h3TMzDKIGjZJCyUt7rMtrNaupDnAkSTLzDTUsOrhm5k1Sy2zdCJiEbBoqP0k7Q/8GPj43iVnGskJ38wsgyh4no6kkSTJ/rKIuKrQxqtwwjczy6DIefiSBFwKLIuIi4treXAewzczy6BSw5bB64A/B94oaUm6nVR81PtyD9/MLIMip7BHxG1A7qmitXLCNzPLoB3W0iltSEfSSyQdn56V7nv/CWXFYGZWr4jIvLWqUhK+pI8BPwXOAZZKelufh/++jBjMzPLoici8taqyhnTOAl4ZETvSiwyulDQnIi5hkHEs17Q1s1ZR9LTMZihrSKc7InYARMRq4FjgREkXM0jCd01bM2sVBc/SaYqyEv4mSc8Xak2T/8nAJODlJcXQUUaNmPb8Zmb5VYjMW6sqa0jn/UBP3zsiogd4v6R/LykGM7O6VVp4bD6rsmraVl0KMSJ+VUYMZmZ5tMMYvufhm5ll0MpDNVk54ZuZZeCEb2bWITykY2bWIdqhh++atmbW9oqoafvKcWdkzkH37Px26QujZTGsevhF1KNdvXpLrjbmzJmSqw4sJLVgs9ajrSZvPdq9ceT5ecyZMwXI93uZO3d62kb+mrZ528hTExeSurirV2/N18acSax5NF/ho9mHjmf92l252pgxazRbNuRLD1Om9xRSj7aIOIoQLX1JVTbDKuGbmTVLOwzpOOGbmWVQkXv4ZmYdoeIhHTOzzuCEb2bWIXzS1sysQ7TDGH5pJQ7NzIazSg3/hiLpm5K2SFpaQujPa3rCl/TBZsdgZjaUXvZk3jL4D6D0et5NT/jABdUekLRQ0mJJi7dvz3cxiplZHkX28CPiViDflZN1KGUMX9J91R4CDql2XEQsAhZBsrTC44/vaEB0ZmZDC3oz79u3HndqUZrPmqqsk7aHAG8Fnux3v4BflxSDmVndapmW2bez2krKSvjXAvtHxJL+D0i6uaQYzMzq5mmZGUXEmYM8dnoZMZiZ5VGpYUinVbXCSVszs5YXVDJvQ5F0OXA7cJikdZKqdoqL5AuvzMwyqERxPfyIeE9hjdXACd/MLAOP4ZuZdYhapmW2Kpc4NLO2V0SJw4PHHZk5Bz2x87cucZhXnhJ2kJSxa6cyiUXE0TLlCVeurb+NebMKaaOI95fLJL6gqDKJRcRRhChwDL9ZhlXCNzNrFq+Hb2bWISKc8M3MOkI7nLR1wjczy8A9fDOzDuF5+GZmHcKzdGog6SXA24AZQAAbgGsiYllZMZiZ1asdhnRKWTxN0qeAH5Csf38XcHd6+3JJ55URg5lZHpXoyby1qrJ6+GcCfxgR+xR7lHQx8ABw4UAH9a0aM2nSpEbHaGZWVTuM4Ze1PHIFmD7A/dPSxwYUEYsiYkFELBg/fnzDgjMzG0pEb+atVZXVw/84cJOkh4G917/PBv4A+GhJMZiZ5TD8e/hlVbz6uaQXA0eTnLQVsA64O1r549DMLNUOJ21Lm6UTyU/rjrKez8ysSO0whu95+GZmmTjhm5l1Bg/pmJl1Bg/pmJl1ijaYX+ISh2bW9ooocSiNzJyDIva0ZIlDIqJtNmCh22i/NlohBrfR3m10ylbWlbZlWeg22rKNVojBbbR3Gx2h3RK+mZlV4YRvZtYh2i3hL3IbbdlGK8TgNtq7jY4wbGbpmJlZPu3Wwzczsyqc8M3MOkRbJHxJJ0h6SNIj9ZZMlPRNSVskLa3z+FmS/lvSMkkPSDq3jjb2k3SXpHvTNi6oJ5a0rW5Jv5V0bZ3Hr5Z0v6QlkhbX2caBkq6UtDz9ubymxuMPS59/77Zd0sfriOOv0p/nUkmXS9qvjjbOTY9/IGsMA72nJB0s6UZJD6dfD6qjjdPSOCqSFtQZxz+lv5f7JF0t6cA62vh8evwSSTdIGqjI0aBt9HnsE5JC0qCl7arE8VlJ6/u8T04arI2O1uwLAQq46KIbWAHMA0YB9wIvraOdY4CjgKV1xjENOCq9fQDwu1rjIKkTsH96eyRwJ/DqOuP5a+D7wLV1Hr8amJTzd/Nt4EPp7VHAgTl/z5uAQ2s8bgawChiTfv8j4AM1tvEyYCkwlmQ5kl8AL6rnPQX8I3Beevs84KI62jgcOAy4GVhQZxxvAUakty+qM47xfW5/DPharW2k988C/gt4dKj3XJU4Pgt8Is97tVO2dujhHw08EhErI2I3SbH0t9XaSETcCjxRbxARsTEifpPefhpYRpJsamkjImJH+u3IdKv5rLqkmcCfAN+o9diiSBpP8p/zUoCI2B0RT+Vo8nhgRUQ8WsexI4AxkkaQJO0NNR5/OHBHRDwTET3ALcDbhzqoynvqbSQfhKRfT621jYhYFhEPZYy9Whs3pK8FkjoVM+toY3ufb8cxxHt1kP9jXwQ+OdTxQ7RhGbRDwp/BC2UTIamkVVOiLZqkOcCRJD30Wo/tlrQE2ALcGBE1twF8ieQ/UJ7l/QK4QdI9aTH5Ws0DHgO+lQ4tfUPSuBzxvBu4vNaDImI98M/AGmAjsC0ibqixmaXAMZImShoLnETSK63HIRGxMY1tIzClznaK9BfA9fUcKOkLktYC7wU+U8fxpwDrI+Leep6/j4+mw0vfHGqYrJO1Q8IfaJGips01lbQ/8GPg4/16QJlERG9EHEHS4zpa0stqfP6TgS0RcU+tz93P6yLiKOBE4GxJx9R4/AiSP72/GhFHAjtJhjBqJmkUcApwRR3HHkTSq54LTAfGSXpfLW1ExDKSYY8bgZ+TDBv2DHrQMCHpfJLXclk9x0fE+RExKz2+pvrU6Yfn+dTxQdHPV4H5wBEkH+r/krO9ttUOCb6NMGUAAAM1SURBVH8d+/a2ZlL7n+yFkDSSJNlfFhFX5WkrHf64GTihxkNfB5wiaTXJ8NYbJX2vjuffkH7dAlxNMnRWi3XAuj5/oVxJ8gFQjxOB30TE5jqOfROwKiIei4g9wFXAa2ttJCIujYijIuIYkiGFh+uIBWCzpGkA6dctdbaTm6QzgJOB90ZE3k7S94E/q/GY+SQfxPem79eZwG8kTa2lkYjYnHaUKsDXqf292jHaIeHfDbxI0ty0J/hu4Jqyg5AkkvHqZRFxcZ1tTN47W0LSGJJktbyWNiLi0xExMyLmkPws/l9E1NSjlTRO0gF7b5Oc4Ktp9lJEbALWSjosvet44MFa2ujjPdQxnJNaA7xa0tj0d3Q8yfmVmkiakn6dDbwjRzzXAGekt88AflpnO7lIOgH4FHBKRDxTZxsv6vPtKdT+Xr0/IqZExJz0/bqOZOLDphrjmNbn27dT43u1ozT7rHERG8mY6u9IZuucX2cbl5P8ObiH5I13Zo3H/zHJUNJ9wJJ0O6nGNl4B/DZtYynwmZw/l2OpY5YOyfj7ven2QI6f6RHA4vT1/AQ4qI42xgKPAxNy/BwuIElGS4HvAqPraOOXJB9Y9wLH1/ueAiYCN5H8hXATcHAdbbw9vb0L2Az8Vx1tPEJy7mvve3WoGTYDtfHj9Gd6H/AzYEatbfR7fDVDz9IZKI7vAvencVwDTMvz/6adNy+tYGbWIdphSMfMzDJwwjcz6xBO+GZmHcIJ38ysQzjhm5l1CCd8M7MO4YRvZtYhnPDNzDqEE74NS+kyFBslfabPfa+Q9JykdzYzNrNW5SttbdiS9FaSS/rfQLI8wGLgroj4YFMDM2tRTvg2rEn6EsnCXbcArweOiBeKyJhZH074NqxJGk2yoNmLgNdGfQVjzDqCx/BtuJtDUg8hSFb5NLMq3MO3YSstOHM7yVLDd5IUs35FRKxpZlxmrcoJ34YtSRcCp5PUEdhGUpd1DHBcJNWPzKwPD+nYsCTpDcDfAO+PiKci6bl8ADicpJKTmfXjHr6ZWYdwD9/MrEM44ZuZdQgnfDOzDuGEb2bWIZzwzcw6hBO+mVmHcMI3M+sQTvhmZh3i/wMU3aWSk1T+IAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 396x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "gridworld.display_best_path(values, gamma, (4,2), cmap=cm.get_cmap('inferno'), figsize=(5.5,4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}