Notebook_Report_P2_DeepLearning.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0d201b74",
   "metadata": {},
   "source": [
    "#  Pokémon RL - Deep Learning \n",
    "\n",
    "## **Authors:**\n",
    "\n",
    "* [Bruno César de Oliveira Souza](mailto:b234837@dac.unicamp.br) - [Github](https://github.com/brunocosouza)\n",
    "\n",
    "* [Leonardo de Lellis Rossi](mailto:l261900@dac.unicamp.br) - [Github](https://github.com/leolellisr) \n",
    "\n",
    "* [Maurício Pereira Lopes](mailto:m225242@g.unicamp.br)\n",
    "\n",
    "## Special Thanks:\n",
    "\n",
    "* To our colleague **Henrique Lima Cará de Oliveira** who collaborated in the development of the codes and execution of the experiments\n",
    "\n",
    "\n",
    "## Main info\n",
    "\n",
    "[Git Repository](https://github.com/leolellisr/poke_RL)\n",
    "\n",
    "Python files of implemented methods available [here](https://github.com/leolellisr/poke_RL/tree/master/py)\n",
    "\n",
    "Trained models available [here](https://drive.google.com/drive/folders/17_Gn1RWOCh-ekiRhhj40ehPz9nv_d-Fy?usp=sharing)\n",
    "\n",
    "Graphs\n",
    "- [Graphs DQN and Double-DQN](https://app.neptune.ai/leolellisr/rl-pokeenv)\n",
    "- [Graphs PPO](https://github.com/leolellisr/poke_RL/tree/master/images/report/ppo_results)\n",
    "- [Graphs REINFORCE - Keras (2018)](https://app.neptune.ai/henriqueoliveira/rl-pokeenv/e/RLPOK-4/charts)\n",
    "\n",
    "\n",
    "Images and results are showed with IFrame from IPython.display, but it needs the images stored in the [/images/report](https://github.com/leolellisr/poke_RL/tree/master/images/report) folder, available in our [git repository](https://github.com/leolellisr/poke_RL). \n",
    "\n",
    "Images and Results also are presented with links from imgur."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "92a90ea7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import IFrame"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ba1ebc99",
   "metadata": {},
   "source": [
    "# Goal and Motivation\n",
    "\n",
    "* This project aims to employ different deep reinforcement learning agents in a Pokémon battle simulator;\n",
    "\n",
    "* In this report, we aim to go a step further into Reinforcement Learning and to employ advanced Deep Reinforcement Learning Techniques. \n",
    "\n",
    "* Our hypothesis is that the deep reinforcement learning trainers will perform better than tabular methods from Project 1 by using deep learning techniques. We also believe that it will automatically learn the best policy or actions, by making decisions through the analysis of states and rewards related to their performance, improving how to win battles throughout the episodes, noticing:\n",
    "\n",
    "  * the different types between Pokémon;\n",
    "  * which moves cause more damage to the opponent's Pokémon;\n",
    "  * what are the possible strategies using no-damage moves;\n",
    "  * and the best times to switch Pokémon. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e3f86d31",
   "metadata": {},
   "source": [
    "## **The  problem addressed**\n",
    "* [Pokémon](https://www.pokemon.com) is a popular Japanese RPG (Role Playing Game) which stands a world championship every year; \n",
    "* One single [battle](https://bulbapedia.bulbagarden.net/wiki/Pokémon_battle) of Pokémon has two players. Each player has a 6-Pokémon team; \n",
    "* Each Pokémon has:\n",
    " * 6 [stats](https://bulbapedia.bulbagarden.net/wiki/Stat) (Health Points, Attack, Defense, Special Attack, Special Defense, Speed). The first 5 are used in the damage calculation. The speed defined which Pokémon moves first in the turn.\n",
    "  * The Health Points goes from 100% (healthy) to 0% (fainted);\n",
    " * 4 possible moves (each with a limited number of uses). Each move has an accuracy, a percentage of success or fail;\n",
    " * one [ability](https://bulbapedia.bulbagarden.net/wiki/Ability) that has special effects in the field;\n",
    " * one [nature](https://bulbapedia.bulbagarden.net/wiki/Nature) that specifies which stats are higher and which are lower;\n",
    " * one [item](https://bulbapedia.bulbagarden.net/wiki/Item), that can  restore Health Points or increase the Power of an Attack.\n",
    "* The winner of the battle is the player that makes all Pokémon of the oposing team to faint (all oposing Pokémon with health points equals zero, \"last man standing\" criteria);\n",
    "* Only one Pokémon of each team can be at the battle field at the same time;\n",
    "* Every turn, each players select one action: one of the 4 moves of their active Pokémon or [switching](https://bulbapedia.bulbagarden.net/wiki/Recall) for one of other non-fainted Pokémon of their team;\n",
    "\n",
    "* Pokémon can be summarized as an analyze state (turn) -> take action sequence game. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "415dbb2b",
   "metadata": {},
   "source": [
    "* By standard, Pokémon is a stochastic game:\n",
    " * One move can have an accuracy value less than 100%, then this move has a probability to be missed;\n",
    " * The damage moves (attacks) have the following [damage calculation](https://bulbapedia.bulbagarden.net/wiki/Damage):\n",
    "  ![Damage](https://wikimedia.org/api/rest_v1/media/math/render/svg/b8c51fed93bb9a80ae8febc13700a40b8a5da402)\n",
    "  \n",
    " where:\n",
    "  * **[Level](https://bulbapedia.bulbagarden.net/wiki/Level)** (the level of the attacking Pokémon);\n",
    "  * **A** is the effective Attack stat of the attacking Pokémon if the used move is a physical move, or the effective Special Attack stat of the attacking Pokémon if the used move is a special move;\n",
    "  * **D** is the effective Defense stat of the target if the used move is a physical move or a special move that uses the target's Defense stat, or the effective Special Defense of the target if the used move is an other special move;\n",
    "  * **[Power](https://bulbapedia.bulbagarden.net/wiki/Power)** is the effective power of the used move;\n",
    "  * **Weather** is 1.5 if a Water-type move is being used during rain or a Fire-type move during harsh sunlight, and 0.5 if a Water-type move is used during harsh sunlight or a Fire-type move during rain, and 1 otherwise.\n",
    "  * **[Critical](https://bulbapedia.bulbagarden.net/wiki/Critical_hit)** has 6.25% chance of occurs and multiplies the damage by 1.5;\n",
    "  * **random** is a random factor between 0.85 and 1.00 (inclusive):\n",
    "  * **[STAB](https://bulbapedia.bulbagarden.net/wiki/Same-type_attack_bonus)** is the same-type attack bonus. This is equal to 1.5 if the move's type matches any of the user's types, 2 if the user of the move additionally has the ability Adaptability, and 1 if otherwise;\n",
    "  * **[Type](https://bulbapedia.bulbagarden.net/wiki/Type)** is the type effectiveness. This can be 0 (ineffective); 0.25, 0.5 (not very effective); 1 (normally effective); 2, or 4 (super effective), depending on both the move's and target's types;\n",
    "  * **[Burn](https://bulbapedia.bulbagarden.net/wiki/Burn_(status_condition))** is 0.5 (from Generation III onward) if the attacker is burned, its Ability is not Guts, and the used move is a physical move (other than Facade from Generation VI onward), and 1 otherwise.\n",
    "  * **other** is 1 in most cases, and a different multiplier when specific interactions of moves, Abilities, or items take effect. In this work, this is applied just to Pokémon that has the item **Life Orb**, which multiplies the damage by 1.3.\n",
    "  \n",
    "  * **Not** used in this work (equals 1):\n",
    "   * Targets (for Battles with more than two active Pokémon in the field);\n",
    "   * Badge ( just applied in Generation II);\n",
    "   \n",
    "   "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8cb2a5b2",
   "metadata": {},
   "source": [
    "# **MDP formulation and discretization model** \n",
    "\n",
    "## Original (stochastic)\n",
    "\n",
    "We considered our original (stochastic) MDP as a tuple $M = (S, A, \\phi, R)$, where:\n",
    "* **S** is the whole set of possible states. One state **s $\\in$ S**  is defined at each turn with 12 battle elements concatenated, that correspond to:\n",
    " * [0] Our Active Pokémon index (0: Venusaur,  1: Pikachu, 2: Tauros, 3: Sirfetch'd, 4: Blastoise, 5: Charizard);\n",
    " * [1] Opponent Active Pokémon index (0: Eevee,  1: Vaporeon, 2: Leafeon, 3: Sylveon, 4: Jolteon, 5: Umbreon);\n",
    " * [2-5] Active Pokémon moves base power (if a move doesn't have base power, default to -1);\n",
    " * [6-9] Active Pokémon moves damage multipliers;\n",
    " * [10] Our remaining Pokémon;\n",
    " * [11] Opponent remaining Pokémon.\n",
    " \n",
    "* **A** is the whole set of possible actions. Our action space is a range [0, 8]. One action **a $\\in$ A** is one of the possible choices:\n",
    " * [0] 1st Active Pokémon move;\n",
    " * [1] 2nd Active Pokémon move;\n",
    " * [2] 3rd Active Pokémon move;\n",
    " * [3] 4th Active Pokémon move;\n",
    " * [4] Switch to 1st next Pokémon;\n",
    " * [5] Switch to 2nd next Pokémon;\n",
    " * [6] Switch to 3rd next Pokémon;\n",
    " * [7] Switch to 4th next Pokémon;\n",
    " * [8] Switch to 5th next Pokémon.\n",
    "\n",
    "When a selected action cannot be executed, we random select another possible action.\n",
    "\n",
    "* **$\\phi$** is a stochastic transition function that occurs from state **s** to state **s'**, by taking an action **a**. The following parameters are part of our  stochastic transition function:\n",
    " * Move's accuracy (chance of the move successfully occurs or fail);\n",
    " * Damage calculation: The **[Critical](https://bulbapedia.bulbagarden.net/wiki/Critical_hit)** parameter (6.25% chance of occurs) and the **random** parameter, ranging from 0.85 and 1.00 (inclusive).\n",
    " * Move's effects: Some moves have [additional effects](https://bulbapedia.bulbagarden.net/wiki/Additional_effect). e.g.: Iron Head have 30% chance of flinching the target (target cannot move in the turn).\n",
    "\n",
    "* **R** is a set of rewards. A reward **r $\\in$ R** is acquired in state **s** by taking the action **a**. The rewards are calculated at the end of the turn. The value of reward **r** is defined by the sum of elements:\n",
    " * +Our Active Pokémon current Health Points;\n",
    " * -2 if our Active Pokémon fainted;\n",
    " * -1 if our Active Pokémon have a [negative status condition](https://bulbapedia.bulbagarden.net/wiki/Status_condition);\n",
    " * +Number of remaining Pokémon in our team;\n",
    " * -Opponent Active Pokémon current Health Points;\n",
    " * +2 if opponent Active Pokémon fainted;\n",
    " * +1 if opponent Active Pokémon have a [negative status condition](https://bulbapedia.bulbagarden.net/wiki/Status_condition);\n",
    " * -Number of remaining Pokémon in opponent team;\n",
    " * +15 if we won the battle;\n",
    " * -15 if we lost the battle.\n",
    " \n",
    "### Stochastic Team\n",
    "\n",
    "Our stochastic team, with each Pokémon, their abilities, natures, items, moves (base power and accuracy) and possible switches are shown in [Team](https://imgur.com/KSXvlmO).\n",
    "\n",
    "The stochastic opponent team, with each Pokémon, their abilities, natures, items, moves (base power and accuracy) and possible switches are shown in [Opponent Team](https://imgur.com/rLF5Cli)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f46553a5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/Team stochastic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x207472e5850>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/Team stochastic.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d5cdcd6d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/Opponent stochastic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x207472e59d0>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/Opponent stochastic.png\", width=1000, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "2d854780",
   "metadata": {},
   "source": [
    "## Deterministic\n",
    "\n",
    "To adapt Pokémon to a deterministic environment, we use Pokémon that cannot receive a critical hit, moves with only 100% accuracy and no side effects, and edit the server code to ignore the random parameter in damage calculation, removing the stochastic transition function $\\phi$ from our MDP. Therefore, now our MDP is a tuple $M = (S, A, R)$, where:\n",
    "* **S** is the whole set of possible states. One state **s $\\in$ S**  is defined at each turn with 12 battle elements concatenated, that correspond to:\n",
    " * [0] Our Active Pokémon index ;\n",
    " * [1] Opponent Active Pokémon index ;\n",
    " * [2-5] Active Pokémon moves base power (if a move doesn't have base power, default to -1);\n",
    " * [6-9] Active Pokémon moves damage multipliers;\n",
    " * [10] Our remaining Pokémon;\n",
    " * [11] Opponent remaining Pokémon.\n",
    " \n",
    "* **A** is the whole set of possible actions. Our action space is a range [0, 8] (len: 9). One action **a $\\in$ A** is one of the possible choices:\n",
    " * [0] 1st Active Pokémon move;\n",
    " * [1] 2nd Active Pokémon move;\n",
    " * [2] 3rd Active Pokémon move;\n",
    " * [3] 4th Active Pokémon move;\n",
    " * [4] Switch to 1st next Pokémon;\n",
    " * [5] Switch to 2nd next Pokémon;\n",
    " * [6] Switch to 3rd next Pokémon;\n",
    " * [7] Switch to 4th next Pokémon;\n",
    " * [8] Switch to 5th next Pokémon.\n",
    "\n",
    "When a selected action cannot be executed, we random select another possible action.\n",
    "\n",
    "* **R** is a set of rewards. A reward **r $\\in$ R** is acquired in state **s** by taking the action **a**. The rewards are calculated at the end of each turn. The value of reward **r** is defined by the sum of elements:\n",
    " * +Our Active Pokémon current Health Points;\n",
    " * -2 if our Active Pokémon fainted;\n",
    " * -1 if our Active Pokémon have a [negative status condition](https://bulbapedia.bulbagarden.net/wiki/Status_condition);\n",
    " * +Number of remaining Pokémon in our team;\n",
    " * -Opponent Active Pokémon current Health Points;\n",
    " * +2 if opponent Active Pokémon fainted;\n",
    " * +1 if opponent Active Pokémon have a [negative status condition](https://bulbapedia.bulbagarden.net/wiki/Status_condition);\n",
    " * -Number of remaining Pokémon in opponent team;\n",
    " * +15 if we won the battle;\n",
    " * -15 if we lost the battle.\n",
    " \n",
    "### Deterministic Team\n",
    "\n",
    "Our deterministic team, with each Pokémon, their abilities, natures, items, moves (base power and accuracy) and possible switches are shown in [Team](https://imgur.com/DeRAEQb).\n",
    "\n",
    "The deterministic opponent team, with each Pokémon, their abilities, natures, items, moves (base power and accuracy) and possible switches are shown in [Opponent Team](https://imgur.com/Hltn5OO).\n",
    "\n",
    "We use on both teams only Pokémon with [Battle Armor](https://bulbapedia.bulbagarden.net/wiki/Battle_Armor_(Ability)) or [Shell Armor](https://bulbapedia.bulbagarden.net/wiki/Shell_Armor_(Ability)) abilities, which prevent critical hits from being performed. Also, we use in both teams only moves with 100% accuracy, with no chance of getting it missed, and the moves haven't additional effects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "61899c0e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/Team deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x207472e5ac0>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/Team deterministic.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "4b5b59ca",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/Opponent deterministic.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x207472e5b50>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/Opponent deterministic.png\", width=1000, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a5bc4248",
   "metadata": {},
   "source": [
    "## Search space\n",
    "\n",
    "The features that integrate our states are shown in [this figure](https://imgur.com/tREjWCG) and below. For a single battle between two players with 6 Pokémon each, we have $1.016.064$ possible states.\n",
    "\n",
    "Since we have 9 possible actions, we total $9.144.576$ possibilities for each battle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "dd39682e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"920\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/possible_states.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x207472e5430>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/possible_states.png\", width=920, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3dc733ba",
   "metadata": {},
   "source": [
    "# **The environments built**\n",
    "\n",
    "The environment used is [Pokémon Showdown](https://play.pokemonshowdown.com), an [open-source](https://github.com/smogon/pokemon-showdown.git) Pokémon battle simulator.\n",
    "\n",
    "[Example](https://imgur.com/hjHikuc) of one battle in Pokémon Showdown.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "a744570f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"1000\"\n",
       "            src=\"images/report/showdownEx.jpg\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x207472e5190>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/showdownEx.jpg\", width=1000, height=1000)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d1742305",
   "metadata": {},
   "source": [
    "To communicate our agents with Pokémon Showdown we used [poke-env](https://poke-env.readthedocs.io/en/latest/) a Python environment for interacting in Pokémon Showdown battles.\n",
    "\n",
    "We used separated Python classes (available in our [git repository](https://github.com/leolellisr/poke_RL/tree/master/src)) for define the Players that are trained with each method. These classes communicates with Pokémon Showdown and implements the poke-env methods to:\n",
    "* Create battles;\n",
    "* Accept battles;\n",
    "* Send orders (actions) to Pokémon Showdown.\n",
    "\n",
    "## Original (stochastic)\n",
    "\n",
    "To speed up the battles, we hosted our own server of Pokemon Showdown in localhost. It requires Node.js v10+."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5730fde",
   "metadata": {},
   "outputs": [],
   "source": [
    "git clone https://github.com/smogon/pokemon-showdown.git"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "1714588c",
   "metadata": {},
   "source": [
    "After clone the repository, it's needed to create a logs folder in root.\n",
    "\n",
    "To configure the server, we used the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "566d9869",
   "metadata": {},
   "outputs": [],
   "source": [
    "cd pokemon-showdown\n",
    "npm install\n",
    "cp config/config-example.js config/config.js"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d1f33ea6",
   "metadata": {},
   "source": [
    "To start the server, we use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8307d0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "node pokemon-showdown start --no-security"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6b4108c6",
   "metadata": {},
   "source": [
    "## Deterministic environment\n",
    "\n",
    "To adapt our environment to a deterministic setup, we had to establish the following parameters:\n",
    "\n",
    "* We removed the random component of sim/battle.ts from the Pokémon Showdown simulator;\n",
    "\n",
    "* We use on both teams only Pokémon with Battle Armor or Shell Armor abilities, which prevent critical hits from being performed;\n",
    "\n",
    "* We used in both teams only moves with 100% accuracy, with no chance of getting it missed;\n",
    "\n",
    "* We didn't use any move with additional effect. "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "07fb6789",
   "metadata": {},
   "source": [
    "# **Characteristics of  the problem**\n",
    "\n",
    "* Both of our environments (stochastic and deterministic) are episodic. One state occurs after another;\n",
    "\n",
    "* Our terminal states are:\n",
    " * When all our Pokémon are fainted (we lose);\n",
    " * When all opponent Pokémon are fainted (we won).\n",
    "\n",
    "* As specified before, a reward **r** is calculated at the end of a turn (beginning of next turn). The value of reward **r** is defined by the sum of:\n",
    " * +Our Active Pokémon current Health Points;\n",
    " * -2 if our Active Pokémon fainted;\n",
    " * -1 if our Active Pokémon have a [negative status condition](https://bulbapedia.bulbagarden.net/wiki/Status_condition);\n",
    " * +Number of remaining Pokémon in our team;\n",
    " * -Opponent Active Pokémon current Health Points;\n",
    " * +2 if opponent Active Pokémon fainted;\n",
    " * +1 if opponent Active Pokémon have a [negative status condition](https://bulbapedia.bulbagarden.net/wiki/Status_condition);\n",
    " * -Number of remaining Pokémon in opponent team;\n",
    " * +15 if we won the battle;\n",
    " * -15 if we lost the battle."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6f1f9a41",
   "metadata": {},
   "source": [
    "# gym wrapper\n",
    "\n",
    "To implement Deep Reinforcement Learning methods in our project, we used poke_env's [env_player class](https://github.com/hsahovic/poke-env/blob/master/src/poke_env/player/env_player.py) as base class. \n",
    "\n",
    "The Deep Reinforcement Learning method can be used with the poke_env class by defining our agent as a gym environment to be used in the DeepRL method. \n",
    "\n",
    "The class defines an env_algorithm_wrapper for communication between a Pokémon Showdown environment in gym, a Deep Reinforcement Learning method and our agent, that will be used as an environment for the Deep Reinforcement Learning method to states/rewards analysis and decision making.\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6ed9f567",
   "metadata": {},
   "source": [
    "# **Deep Q-Learning - Keras (2018)**\n",
    "\n",
    "The first method we have implemented was a **value-based** method: [Deep Q-Learning (DQN), from Keras-RL Agents (2018)](github.com/keras-rl/keras-rl/blob/master/rl/agents/dqn.py).\n",
    "\n",
    "* Agent performs actions in the environment to learn how to map the observed states to certain actions;\n",
    "* The agent chooses an action in a given state based on a \"Q value\" (weighted reward based on the highest expected long-term reward); \n",
    "* DQN agent learns to perform its task in such a way that the recommended action maximizes potential future rewards;\n",
    "* The method is considered an \"Off-Policy\" method because its Q values are updated assuming the best action was chosen, even if the best action was not chosen.\n",
    "*  Q-value is calculated with the reward added to the next state maximum Q-value. Every time the Q-value calculates a high number for a certain state, the value that is obtained from the output of the neural network for that specific state, will become higher every time. Each output neuron value will get higher and higher until the difference between each output value is high;\n",
    "* If action a in state s is a higher value than action b, then action a will get chosen every time for state s. If for some memory experience action b becomes the better action for state s, it is difficult to train the network to learn that action b is the better action in some conditions.\n",
    "\n",
    "* We used an [linear annealing $\\epsilon$-greedy exploration strategy,  from Keras-RL (2018)](https://github.com/keras-rl/keras-rl/blob/master/rl/policy.py), with $\\epsilon = 0,1$;\n",
    "\n",
    "* We used the following $\\epsilon$-greedy policy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17996947",
   "metadata": {},
   "outputs": [],
   "source": [
    "policy = LinearAnnealedPolicy(\n",
    "            EpsGreedyQPolicy(),\n",
    "            attr=\"eps\",\n",
    "            eps = 0.1,\n",
    "            value_max=1.0,\n",
    "            value_min=0.05,\n",
    "            value_test=0,\n",
    "            nb_steps=10.000 * epochs/10,\n",
    "        )"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "1d73ba87",
   "metadata": {},
   "source": [
    "We have applied the follow configurations and hyperparameters to the training of this method:\n",
    "* Optimizer: Adam (Learning Rate: $2.5e-4$)\n",
    "* Number of the steps in warmup: $10.000 * epochs/10$\n",
    "* Metric (training): Mean Absolute Error\n",
    "* gamma = $0.75$\n",
    "* target_model_update=$1$\n",
    "* delta_clip=$0.01$\n",
    "* epsilon (epsilon-greedy policy)=$0.1$  \n",
    "\n",
    "\n",
    "\n",
    "The internal model of our DQN agent is shown above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "baecc839",
   "metadata": {},
   "outputs": [],
   "source": [
    "N_HIDDEN = 128\n",
    "N_STATE_COMPONENTS = 12\n",
    "n_action = 9\n",
    "model = Sequential()\n",
    "model.add(Dense(N_HIDDEN, activation=\"elu\", input_shape=(1, N_STATE_COMPONENTS)))\n",
    "# Our embedding have shape (1, 12), which affects our hidden layer dimension and output dimension\n",
    "# Flattening resolve potential issues that would arise otherwise\n",
    "model.add(Flatten())\n",
    "model.add(Dense(int(N_HIDDEN/2), activation=\"elu\"))\n",
    "model.add(Dense(n_action, activation=\"linear\"))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "008236d8",
   "metadata": {},
   "source": [
    "* Since DQN trains by step and one battle has usually 30 steps, we defined $10.000$ steps per epoch (~$333$ battles per epoch) and defined the total number of battles with the number of epochs. We trained our agent against a MaxDamagePlayer, that always selects the move with the greatest base power. We trained for $300.000$ steps ($30$ epochs and ~$10.000$ battles) and  $900.000$ steps ($90$ epochs and ~$30.000$ battles). At the end of the $900.000$ steps training, we saved the Keras model for future use;\n",
    "\n",
    "* We defined one SequentialMemory with limit of the max number of steps ($10.000$ * epochs) and a window length of 1;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "930972e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "memory = SequentialMemory(limit=10.000 * epochs, window_length=1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7121f32e",
   "metadata": {},
   "source": [
    "* We defined and compiled our DQN as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7aee2ec7",
   "metadata": {},
   "outputs": [],
   "source": [
    "dqn = DQNAgent(\n",
    "        model=model,\n",
    "        nb_actions=len(env_player.action_space),\n",
    "        policy=policy,\n",
    "        memory=memory,\n",
    "        nb_steps_warmup=int(10.000 * epochs/10),\n",
    "        gamma=0.75,\n",
    "        target_model_update=1,\n",
    "        delta_clip=0.01,\n",
    "        enable_double_dqn=False\n",
    "    )\n",
    "dqn.compile(Adam(lr=2.5e-4), metrics=[\"mae\"])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "871cd8cb",
   "metadata": {},
   "source": [
    "* We defined the following algorithms to train and validate our DQN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25f32a74",
   "metadata": {},
   "outputs": [],
   "source": [
    "def dqn_training(player, dqn, nb_steps):\n",
    "    dqn.fit(player, nb_steps=nb_steps)\n",
    "    player.complete_current_battle()\n",
    "\n",
    "def dqn_evaluation(player, dqn, nb_episodes):\n",
    "    # Reset battle statistics\n",
    "    player.reset_battles()\n",
    "    dqn.test(player, nb_episodes=nb_episodes, visualize=False)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "038d065b",
   "metadata": {},
   "source": [
    "Finally, we call the following functions to train and test our agent:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4af68a7a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Training\n",
    "env_player.play_against(\n",
    "    env_algorithm=dqn_training,\n",
    "    opponent=opponent_player,\n",
    "    env_algorithm_kwargs={\"dqn\": dqn, \"nb_steps\": int(10.000 * epochs/10)},\n",
    ")\n",
    "model.save(\"model_%d\" % int(10.000 * epochs/10))\n",
    "\n",
    "# Evaluation\n",
    "env_player.num_battles=0\n",
    "env_player.play_against(\n",
    "    env_algorithm=dqn_evaluation,\n",
    "    opponent=opponent,\n",
    "    env_algorithm_kwargs={\"dqn\": dqn, \"nb_episodes\": int(number_of_battles/3)})"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0e581aff",
   "metadata": {},
   "source": [
    "* We validate our solution in both cases against a MaxDamagePlayer and against a RandomPlayer, that selects random actions at each turn. We validated for $3.333$ battles against each Player.  "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "b13e473a",
   "metadata": {},
   "source": [
    "## **Deep Q-Learning - Keras (2018)** 300k steps - Stochastic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-184/charts. \n",
    "\n",
    "The agent trained for $300k$ steps and $10.077$ battles, resulting in ~$29,77$ steps per battle.\n",
    " \n",
    "### **Deep Q-Learning - Keras (2018)** 300k steps - Stochastic - Rewards (per step) - Training \n",
    "\n",
    "[**Deep Q-Learning - Keras (2018)** 300k steps - Stochastic - Rewards (per step) - Training](https://imgur.com/ixM6ErF)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "9de3263d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/reward_computed_dqn_sto.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x207472e5940>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/reward_computed_dqn_sto.png\", width=800, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f4c3d50b",
   "metadata": {},
   "source": [
    "### **Deep Q-Learning - Keras (2018)** 300k steps - Stochastic - Win percentages (per battle) - Training\n",
    "[ **Deep Q-Learning - Keras (2018)** 300k steps - Stochastic - Win percentages (per step) - Training](https://imgur.com/IASqmaj)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "7e9b0e73",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/win_acc_dqn_sto.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x207472e5820>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/win_acc_dqn_sto.png\", width=1000, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "dcf4114f",
   "metadata": {},
   "source": [
    "### **Deep Q-Learning - Keras (2018)** 300k steps - Stochastic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player won $2319/3286$ battles [this is $69,57\\%$ and took $12.699$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player won $3286/3333$ battles [this is $98,59\\%$ and took $12.459$ seconds]\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "91beca0f",
   "metadata": {},
   "source": [
    "\n",
    "### **Deep Q-Learning - Keras (2018)** 300k steps - Stochastic - Memory and CPU usage (absolute time) - Training\n",
    "\n",
    "[**Deep Q-Learning - Keras (2018)** 300k steps - Stochastic - Memory and CPU usage (absolute time) - Training](https://imgur.com/WicjMaW)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "e6652aa7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/cpu and memory dqn keras sto.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x207472e5f70>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/cpu and memory dqn keras sto.png\", width=1000, height=600)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d509af2a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f415c97c",
   "metadata": {},
   "source": [
    "## **Deep Q-Learning - Keras (2018)** 300k steps - Deterministic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at  https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-189/charts.\n",
    "\n",
    "The agent trained for $300k$ steps and $13.021$ battles, resulting in ~$23,04$ steps per battle.\n",
    "\n",
    "### **Deep Q-Learning - Keras (2018)** 300k steps - Deterministic - Rewards (per step) - Training \n",
    "\n",
    "[**Deep Q-Learning - Keras (2018)** 300k steps - Deterministic - Rewards (per step) - Training](https://imgur.com/AqsupVL)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "32753076",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"800\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/reward_computed_dqn_det.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x207472e5040>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/reward_computed_dqn_det.png\", width=800, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5fec1fea",
   "metadata": {},
   "source": [
    "### **Deep Q-Learning - Keras (2018)** 300k steps - Deterministic - Win percentages (per battle) - Training\n",
    "[ **Deep Q-Learning - Keras (2018)** 300k steps - Deterministic - Win percentages (per step) - Training](https://imgur.com/rYU76ax)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "ddae0242",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/win_acc_dqn_det.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x207472e5df0>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/win_acc_dqn_det.png\", width=1000, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6a8fd52f",
   "metadata": {},
   "source": [
    "### **Deep Q-Learning - Keras (2018)** 300k steps - Deterministic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player won $2390/3333$ battles [this is $71,71\\%$ and took $11.016$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player won $3283/3333$ battles [this is $98,50\\%$ and took $12.889$ seconds]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "4da003b8",
   "metadata": {},
   "source": [
    "### **Deep Q-Learning - Keras (2018)** 300k steps - Deterministic - Memory and CPU usage  (absolute time) - Training\n",
    "\n",
    "[**Deep Q-Learning - Keras (2018)** 300k steps - Deterministic - Memory and CPU usage (absolute time) - Training](https://imgur.com/yJflda0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "e21d53ca",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"600\"\n",
       "            src=\"images/report/cpu and memory dqn keras det.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x207472e5e80>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/cpu and memory dqn keras det.png\", width=1000, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "22eedae0",
   "metadata": {},
   "source": [
    "## **Deep Q-Learning - Keras (2018)** 900k steps - Stochastic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-185/charts. \n",
    "\n",
    "The agent trained for $900k$ steps and $29.979$ battles, resulting in ~$30,02$ steps per battle.\n",
    "\n",
    "### **Deep Q-Learning - Keras (2018)** 900k steps - Stochastic - Rewards (per step) - Training \n",
    "\n",
    "[**Deep Q-Learning - Keras (2018)** 900k steps - Stochastic - Rewards (per step) - Training](https://imgur.com/bhxDe1S)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff943224",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/reward_computed_dqn_sto90.png\", width=800, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e8ceadc0",
   "metadata": {},
   "source": [
    "### **Deep Q-Learning - Keras (2018)** 900k steps - Stochastic - Win percentages (per battle) - Training\n",
    "[ **Deep Q-Learning - Keras (2018)** 900k steps - Stochastic - Win percentages (per step) - Training](https://imgur.com/NaWa56r)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c4d263d",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/win_acc_dqn_sto90.png\", width=1000, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "96c66647",
   "metadata": {},
   "source": [
    "### **Deep Q-Learning - Keras (2018)** 900k steps - Stochastic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player won $2032/3333$ battles [this is $60,57\\%$ and took $36.988$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player won $3309/3333$ battles [this is $99,28\\%$ and took $39.774$ seconds]\n",
    "\n",
    "\n",
    "### **Deep Q-Learning - Keras (2018)** 900k steps - Stochastic - Memory and CPU usage (absolute time) - Training\n",
    "\n",
    "[**Deep Q-Learning - Keras (2018)** 900k steps - Stochastic - Memory and CPU usage (absolute time) - Training](https://imgur.com/8kG2OlA)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0bb0e39",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/cpu and memory dqn keras sto90.png\", width=1000, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ce393563",
   "metadata": {},
   "source": [
    "## **Deep Q-Learning - Keras (2018)** 900k steps - Deterministic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-190/charts. \n",
    "\n",
    "The agent trained for $900k$ steps and $38.922$ battles, resulting in ~$23,12$ steps per battle.\n",
    "\n",
    "### **Deep Q-Learning - Keras (2018)** 900k steps - Deterministic - Rewards (per step) - Training \n",
    "\n",
    "[**Deep Q-Learning - Keras (2018)** 900k steps - Deterministic - Rewards (per step) - Training](https://imgur.com/f1sQsub)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "692d7fe7",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/reward_computed_dqn_det90.png\", width=800, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "182905b1",
   "metadata": {},
   "source": [
    "### **Deep Q-Learning - Keras (2018)** 900k steps - Deterministic - Win percentages (per battle) - Training\n",
    "[ **Deep Q-Learning - Keras (2018)** 900k steps - Deterministic - Win percentages (per step) - Training](https://imgur.com/Ulx7nYp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b357b37",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/win_acc_dqn_det90.png\", width=1000, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e9fdfc2f",
   "metadata": {},
   "source": [
    "### **Deep Q-Learning - Keras (2018)** 900k steps - Deterministic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player won $2487/3333$ battles [this is $74,62\\%$ and took $41.872$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player won $3296/3333$ battles [this is $98,89\\%$ and took $45.381$ seconds]\n",
    "\n",
    "### **Deep Q-Learning - Keras (2018)** 900k steps - Deterministic - Memory and CPU usage (absolute time) - Training\n",
    "\n",
    "[**Deep Q-Learning - Keras (2018)** 900k steps - Deterministic - Memory and CPU usage (absolute time) - Training](https://imgur.com/WPaX7kJ)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dbbb3623",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/cpu and memory dqn keras det90.png\", width=1000, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "f6f11514",
   "metadata": {},
   "source": [
    "# **Double Deep Q-Learning - Keras (2018)**\n",
    "\n",
    "The second method we have implemented also is a **value-based** method: [Double Deep Q-Learning (DQN), from Keras-RL Agents (2018)](github.com/keras-rl/keras-rl/blob/master/rl/agents/dqn.py). We used the same code from DQN.\n",
    "\n",
    "* Double DQN uses two identical neural network models. One learns during the experience replay, just like DQN does, and the other one is a copy of the last episode of the first model. The Q-value is actually computed with this second model;\n",
    "* As DQN, agent performs actions in the environment to learn how to map the observed states to certain actions, chooses an action in a given state based on a \"Q value\" (weighted reward based on the highest expected long-term reward) and learns to perform its task in such a way that the recommended action maximizes potential future rewards;\n",
    "* Differently from DQN, if action a in state s is a higher value than action b, we use a secondary model that is the copy of the main model from the last episode and, since the difference between values of the second model are lower than the main model, we use this second model to attain the Q-value;\n",
    "* Double DQN calculates Q-value finding the index of the highest Q-value from the main model and then using that index to obtain the action from the second model;\n",
    "* The method is considered an \"Off-Policy\" method because its Q values are updated assuming the best action was chosen, even if the best action was not chosen.\n",
    "\n",
    "*  As DQN, we used an [linear annealing $\\epsilon$-greedy exploration strategy,  from Keras-RL (2018)](https://github.com/keras-rl/keras-rl/blob/master/rl/policy.py), with $\\epsilon = 0,1$;\n",
    "\n",
    "* We also used the following $\\epsilon$-greedy policy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c141075",
   "metadata": {},
   "outputs": [],
   "source": [
    "policy = LinearAnnealedPolicy(\n",
    "            EpsGreedyQPolicy(),\n",
    "            attr=\"eps\",\n",
    "            eps = 0.1,\n",
    "            value_max=1.0,\n",
    "            value_min=0.05,\n",
    "            value_test=0,\n",
    "            nb_steps=10.000 * epochs/10,\n",
    "        )"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5a942186",
   "metadata": {},
   "source": [
    "We have applied the follow configurations and hyperparameters to the training of this method:\n",
    "* Optimizer: Adam (Learning Rate: $2.5e-4$)\n",
    "* Number of the steps in warmup: $10.000 * epochs/10$\n",
    "* Metric (training): Mean Absolute Error\n",
    "* gamma = $0.75$\n",
    "* target_model_update=$1$\n",
    "* delta_clip=$0.01$\n",
    "* epsilon (epsilon-greedy policy)=$0.1$  \n",
    "\n",
    "\n",
    "\n",
    "The internal model of our Double DQN agent is shown above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c84616a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "N_HIDDEN = 128\n",
    "N_STATE_COMPONENTS = 12\n",
    "n_action = 9\n",
    "model = Sequential()\n",
    "model.add(Dense(N_HIDDEN, activation=\"elu\", input_shape=(1, N_STATE_COMPONENTS)))\n",
    "# Our embedding have shape (1, 12), which affects our hidden layer dimension and output dimension\n",
    "# Flattening resolve potential issues that would arise otherwise\n",
    "model.add(Flatten())\n",
    "model.add(Dense(int(N_HIDDEN/2), activation=\"elu\"))\n",
    "model.add(Dense(n_action, activation=\"linear\"))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "12d29c37",
   "metadata": {},
   "source": [
    "* Since Double DQN trains by step and one battle has usually 30 steps, we defined $10.000$ steps per epoch (~$333$ battles per epoch) and defined the total number of battles with the number of epochs. We trained our agent against a MaxDamagePlayer, that always selects the move with the greatest base power. We trained for $300.000$ steps ($30$ epochs and ~$10.000$ battles) and $900.000$ steps ($90$ epochs and ~$30.000$ battles). At the end of the $900.000$ steps training, we saved the Keras model for future use;\n",
    "\n",
    "* We defined one SequentialMemory with limit of the max number of steps ($10.000$ * epochs) and a window length of 1;\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "292fea5a",
   "metadata": {},
   "outputs": [],
   "source": [
    "memory = SequentialMemory(limit=10.000 * epochs, window_length=1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3b044e0f",
   "metadata": {},
   "source": [
    "* We defined and compiled our Double DQN as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bce7f6e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "dqn = DQNAgent(\n",
    "        model=model,\n",
    "        nb_actions=len(env_player.action_space),\n",
    "        policy=policy,\n",
    "        memory=memory,\n",
    "        nb_steps_warmup=int(10.000 * epochs/10),\n",
    "        gamma=0.75,\n",
    "        target_model_update=1,\n",
    "        delta_clip=0.01,\n",
    "        enable_double_dqn=True\n",
    "    )\n",
    "dqn.compile(Adam(lr=2.5e-4), metrics=[\"mae\"])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "142703d4",
   "metadata": {},
   "source": [
    "* We defined the following algorithms to train and validate our Double DQN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "883cefa4",
   "metadata": {},
   "outputs": [],
   "source": [
    "def dqn_training(player, dqn, nb_steps):\n",
    "    dqn.fit(player, nb_steps=nb_steps)\n",
    "    player.complete_current_battle()\n",
    "\n",
    "def dqn_evaluation(player, dqn, nb_episodes):\n",
    "    # Reset battle statistics\n",
    "    player.reset_battles()\n",
    "    dqn.test(player, nb_episodes=nb_episodes, visualize=False)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9314a1dd",
   "metadata": {},
   "source": [
    "Finally, we call the following functions to train and test our agent:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "944a4690",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Training\n",
    "env_player.play_against(\n",
    "    env_algorithm=dqn_training,\n",
    "    opponent=opponent_player,\n",
    "    env_algorithm_kwargs={\"dqn\": dqn, \"nb_steps\": int(10.000 * epochs/10)},\n",
    ")\n",
    "model.save(\"model_%d\" % int(10.000 * epochs/10))\n",
    "\n",
    "# Evaluation\n",
    "env_player.num_battles=0\n",
    "env_player.play_against(\n",
    "    env_algorithm=dqn_evaluation,\n",
    "    opponent=opponent,\n",
    "    env_algorithm_kwargs={\"dqn\": dqn, \"nb_episodes\": int(number_of_battles/3)})"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "b3b39802",
   "metadata": {},
   "source": [
    "* We validate our solution in both cases against a MaxDamagePlayer and against a RandomPlayer, that selects random actions at each turn. We validated for $3.333$ battles against each Player.  \n",
    "\n",
    "## **Double Deep Q-Learning - Keras (2018)** 300k steps - Stochastic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-152/charts. \n",
    "\n",
    "The agent trained for $300k$ steps and $10.053$ battles, resulting in ~$29,84$ steps per battle.\n",
    "\n",
    "\n",
    "### **Double Deep Q-Learning - Keras (2018)** 300k steps - Stochastic - Rewards (per step) - Training \n",
    "\n",
    "[**Double Deep Q-Learning - Keras (2018)** 300k steps - Stochastic - Rewards (per step) - Training](https://imgur.com/uyBq9GJ)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38a28964",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/reward_computed_double_dqn_sto.png\", width=800, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "cab9422c",
   "metadata": {},
   "source": [
    "### **Double Deep Q-Learning - Keras (2018)** 300k steps - Stochastic - Win percentages (per battle) - Training\n",
    "[ **Double Deep Q-Learning - Keras (2018)** 300k steps - Stochastic - Win percentages (per step) - Training](https://imgur.com/gxOglQv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "baeb849d",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/win_acc_double_dqn_sto.png\", width=1000, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "fa95260a",
   "metadata": {},
   "source": [
    "### **Double Deep Q-Learning - Keras (2018)** 300k steps - Stochastic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player won $2233/3333$ battles [this is $67\\%$ and took $9.750$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player won $3266/3333$ battles [this is $98\\%$ and took $14.001$ seconds]\n",
    "\n",
    "### **Double Deep Q-Learning - Keras (2018)** 300k steps - Stochastic - Memory and CPU usage (absolute time) - Training\n",
    "\n",
    "[**Double Deep Q-Learning - Keras (2018)** 300k steps - Stochastic - Memory and CPU usage (absolute time) - Training](https://imgur.com/rKdi5DW)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9b82ddd",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/cpu and memory double dqn keras sto.png\", width=1000, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d2eef111",
   "metadata": {},
   "source": [
    "## **Double Deep Q-Learning - Keras (2018)** 300k steps - Deterministic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at  https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-192/charts.\n",
    "\n",
    "The agent trained for $300k$ steps and $13.006$ battles, resulting in ~$23,07$ steps per battle.\n",
    "\n",
    "### **Double Deep Q-Learning - Keras (2018)** 300k steps - Deterministic - Rewards (per step) - Training \n",
    "\n",
    "[**Double Deep Q-Learning - Keras (2018)** 300k steps - Deterministic - Rewards (per step) - Training](https://imgur.com/cviU2k5)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e035037a",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/reward_computed_double_dqn_det.png\", width=800, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6f6311ce",
   "metadata": {},
   "source": [
    "\n",
    "### **Double Deep Q-Learning - Keras (2018)** 300k steps - Deterministic - Win percentages (per battle) - Training\n",
    "[**Double Deep Q-Learning - Keras (2018)** 300k steps - Deterministic - Win percentages (per step) - Training](https://imgur.com/g1UqVF4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "411ae929",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/win_acc_double_dqn_det.png\", width=1000, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "163d9929",
   "metadata": {},
   "source": [
    "### **Double Deep Q-Learning - Keras (2018)** 300k steps - Deterministic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player won $2569/3333$ battles [this is $77,08\\%$ and took $11.508$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player won $3312/3333$ battles [this is $99,37\\%$ and took $13.261$ seconds]\n",
    "\n",
    "### **Double Deep Q-Learning - Keras (2018)** 300k steps - Deterministic - Memory and CPU usage  (absolute time) - Training\n",
    "\n",
    "[**Double Deep Q-Learning - Keras (2018)** 300k steps - Deterministic - Memory and CPU usage (absolute time) - Training](https://imgur.com/wtXIKni)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce77c8a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/cpu and memory double dqn keras det.png\", width=1000, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5879b91b",
   "metadata": {},
   "source": [
    "## **Double Deep Q-Learning - Keras (2018)** 900k steps - Stochastic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-168/charts. \n",
    "\n",
    "The agent trained for $900k$ steps and $30.049$ battles, resulting in ~$29,95$ steps per battle.\n",
    "\n",
    "\n",
    "### **Double Deep Q-Learning - Keras (2018)** 900k steps  - Stochastic - Rewards (per step) - Training \n",
    "\n",
    "[**Double Deep Q-Learning - Keras (2018)** 900k steps - Stochastic - Rewards (per step) - Training](https://imgur.com/sD2rnhb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eb968411",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/reward_computed_double_dqn_sto90.png\", width=800, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "17654ce0",
   "metadata": {},
   "source": [
    "\n",
    "### **Double Deep Q-Learning - Keras (2018)** 900k steps - Stochastic - Win percentages (per battle) - Training\n",
    "[ **Double Deep Q-Learning - Keras (2018)** 900k steps - Stochastic - Win percentages (per step) - Training](https://imgur.com/1HEDLLU)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a7994714",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/win_acc_double_dqn_sto90.png\", width=1000, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "af76d969",
   "metadata": {},
   "source": [
    "### **Double Deep Q-Learning - Keras (2018)** 900k steps - Stochastic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player won $2798/3333$ battles [this is $83,94\\%$ and took $26.557$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player won $3308/3333$ battles [this is $99,24\\%$ and took $28.491$ seconds]\n",
    "\n",
    "\n",
    "\n",
    "### **Double Deep Q-Learning - Keras (2018)** 900k steps - Stochastic - Memory and CPU usage (absolute time) - Training\n",
    "\n",
    "[**Double Deep Q-Learning - Keras (2018)** 900k steps - Stochastic - Memory and CPU usage (absolute time) - Training](https://imgur.com/cUSo9hB)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78c63620",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/cpu and memory double dqn keras sto90.png\", width=1000, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "695a8a07",
   "metadata": {},
   "source": [
    "## **Double Deep Q-Learning - Keras (2018)** 900k steps - Deterministic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at  https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-191/charts.\n",
    "\n",
    "The agent trained for $900k$ steps and $38.932$ battles, resulting in ~$23,12$ steps per battle.\n",
    "\n",
    "### **Double Deep Q-Learning - Keras (2018)** 900k steps - Deterministic - Rewards (per step) - Training \n",
    "\n",
    "[**Double Deep Q-Learning - Keras (2018)** 900k steps - Deterministic - Rewards (per step) - Training](https://imgur.com/IWHPKKl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67a4be36",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/reward_computed_double_dqn_det90.png\", width=800, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "29db57be",
   "metadata": {},
   "source": [
    "### **Double Deep Q-Learning - Keras (2018)** 900k steps - Deterministic - Win percentages (per battle) - Training\n",
    "[**Double Deep Q-Learning - Keras (2018)** 900k steps - Deterministic - Win percentages (per step) - Training](https://imgur.com/Nd1m6aw)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4fe9ceaf",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/win_acc_double_dqn_det90.png\", width=1000, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6d70337d",
   "metadata": {},
   "source": [
    "### **Double Deep Q-Learning - Keras (2018)** 900k steps - Deterministic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player won $2597/3333$ battles [this is $77,92\\%$ and took $46.701$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player won $3299/3333$ battles [this is $98,98\\%$ and took $61.444$ seconds]\n",
    "\n",
    "### **Deep Q-Learning - Keras (2018)** 900k steps - Deterministic - Memory and CPU usage  (absolute time) - Training\n",
    "\n",
    "[**Deep Q-Learning - Keras (2018)** 900k steps - Deterministic - Memory and CPU usage (absolute time) - Training](https://imgur.com/EzSWLC9)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0767df91",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/cpu and memory double dqn keras det90.png\", width=1000, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "936c244a",
   "metadata": {},
   "source": [
    "# ** Proximal Policy Optimization (PPO2) - Stable Baselines (2021)**\n",
    "\n",
    "The next method we have implemented was a **policy-based** method: [Proximal Policy Optimization (PPO2), from Stable Baselines (2021)](github.com/Stable-Baselines-Team/stable-baselines/blob/master/stable_baselines/ppo2/ppo2.py).\n",
    "\n",
    "* The Proximal Policy Optimization algorithm combines ideas from A2C (having multiple workers) and TRPO (it uses a trust region to improve the actor).\n",
    "* The main idea is that after an update, the new policy should be not too far from the old policy. For that, PPO uses clipping to avoid too large update;\n",
    "* Proximal Policy Optimization (PPO) is an Actor-Critic method. As the name suggests, the Actor-Critic system has two models: the Actor and the Critic. The Actor corresponds to the policy and is used to choose the action for the agent and update the policy network. The Critic corresponds to the value function. The Critic updates the parameters of the network for the value function used during the Actor update. The actor network receives observation (state) as the input and outputs a list of probabilities, with one probability per action. These probabilities form a distribution, and the action can then be chosen by sampling from this distribution. To represent the state value function, the critic network also receives the state as the input and outputs a single number representing the estimated state value of that state;\n",
    "* Actorcritic RL methods combine policy-based and value-based RL methods by predicting both policy and value for a given state, and then using the value prediction (the “critic”) as an estimate of expected return when  pdating the policy prediction (the “actor”);\n",
    "* We used an [MlpPolicy,  Stable Baselines (2021)](https://github.com/Stable-Baselines-Team/stable-baselines/blob/master/stable_baselines/deepq/policies.py), a policy object that implements DQN policy, using a MLP (2 layers of 64).\n",
    "\n",
    "We have applied the follow configurations and hyperparameters to the training of this method:\n",
    "* Optimizer: Adam (Learning Rate: $2.5e-4$)\n",
    "* gamma = $0.75$\n",
    "* clip_range=$0.2$\n",
    "\n",
    "Since PPO also trains by step and one battle has usually 30 steps, we defined $10.000$ steps per epoch (~$333$ battles per epoch) and defined the total number of battles with the number of epochs. We trained our agent against a MaxDamagePlayer, that always selects the move with the greatest base power. We trained for $10.000$ battles ($30$ epochs) and $30.000$ battles ($90$ epochs. At the end of the $30.000$ battles training, we saved the PPO model for future use;\n",
    "\n",
    "* We defined our PPO as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8825365",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = PPO2(\"MlpPolicy\", env_player, gamma=0.75, verbose=0)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6af677fc",
   "metadata": {},
   "source": [
    "* We defined the following algorithms to train and validate our PPO."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b42ee9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "NB_TRAINING_STEPS = args.epochs*10000\n",
    "NB_EVALUATION_EPISODES = int(10000/3)\n",
    "def ppo_training(player):\n",
    "        print (\"Training...\")\n",
    "        model.learn(total_timesteps=NB_TRAINING_STEPS)\n",
    "        print(\"Training complete.\")\n",
    "        \n",
    "def ppo_evaluating(player):\n",
    "        player.reset_battles()\n",
    "        for _ in range(NB_EVALUATION_EPISODES):\n",
    "            done = False\n",
    "            obs = player.reset()\n",
    "            while not done:\n",
    "                action = model.predict(obs)[0]\n",
    "                obs, _, done, _ = player.step(action)\n",
    "                # print (\"done:\" + str(done))\n",
    "        player.complete_current_battle()\n",
    "\n",
    "        print(\n",
    "            \"PPO Evaluation: %d victories out of %d episodes\"\n",
    "            % (player.n_won_battles, NB_EVALUATION_EPISODES)\n",
    "        )\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "844da17f",
   "metadata": {},
   "source": [
    "Finally, we call the following functions to train and test our agent:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb52ddbc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Training\n",
    "env_player.play_against(\n",
    "    env_algorithm=ppo_training,\n",
    "    opponent=opponent\n",
    ")\n",
    "model.save(\"model_%d\" % NB_TRAINING_STEPS)\n",
    "\n",
    "# Evaluation\n",
    "env_player.mode = \"val_max\"\n",
    "print(\"Results against max player:\")\n",
    "env_player.num_battles=0\n",
    "env_player.play_against(\n",
    "    env_algorithm=ppo_evaluating,\n",
    "    opponent=opponent)\n",
    "env_player.mode = \"val_rand\"\n",
    "\n",
    "print(\"\\nResults against random player:\")\n",
    "env_player.num_battles=0\n",
    "env_player.play_against(\n",
    "    env_algorithm=ppo_evaluating,\n",
    "    opponent=second_opponent)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "695fedc0",
   "metadata": {},
   "source": [
    "* We validate our solution in both cases against a MaxDamagePlayer and against a RandomPlayer, that selects random actions at each turn. We validated for $3.333$ battles against each Player.  \n",
    "\n",
    "## **Proximal Policy Optimization (PPO2) - Stable Baselines (2021)** 300k steps - Stochastic - Results \n",
    "\n",
    "The graphs of \"Rewards in Training\" and \"Wins in Training\" can be seen at https://github.com/leolellisr/poke_RL/tree/master/images/report/ppo_results. \n",
    "\n",
    "\n",
    "The agent trained for $300k$ steps and $13.800$ battles, resulting in ~$21,74$ steps per battle.\n",
    "\n",
    "\n",
    "### **Proximal Policy Optimization (PPO2) - Stable Baselines (2021)** 300k steps - Stochastic - Rewards (per step) - Training \n",
    "\n",
    "[**Proximal Policy Optimization (PPO2) - Stable Baselines (2021)** 300k steps - Stochastic - Rewards (per step) - Training](https://imgur.com/TD0yPcL)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ed22582",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/ppo_results/rewards_ppo_stochastic_10k_battles.pdf\", width=800, height=600)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "2ce2c72d",
   "metadata": {},
   "source": [
    "### **Proximal Policy Optimization (PPO2) - Stable Baselines (2021)** 300k steps - Stochastic - Win percentages (per step) - Training\n",
    "[ **Proximal Policy Optimization (PPO2) - Stable Baselines (2021)** 300k steps - Stochastic - Win percentages (per step) - Training](https://imgur.com/TnbGzzS)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "34c17f1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/ppo_results/win_ppo_stochastic_10k_battles.pdf\", width=1000, height=600)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "1f6ffe99",
   "metadata": {},
   "source": [
    "### **Proximal Policy Optimization (PPO2) - Stable Baselines (2021)** 300k steps - Stochastic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player won $2964/3333$ battles [this is $88,63\\%$ and took $2700$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player won $3331/3333$ battles [this is $99,67\\%$ and took $3200$ seconds]\n",
    "\n",
    "\n",
    "\n",
    "## **Proximal Policy Optimization (PPO2) - Stable Baselines (2021)** 300k steps - Deterministic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\" and \"Wins in Training\" can be seen at https://github.com/leolellisr/poke_RL/tree/master/images/report/ppo_results.\n",
    "\n",
    "The agent trained for $300k$ steps and $14.000$ battles, resulting in ~$21,43$ steps per battle.\n",
    "\n",
    "### **Proximal Policy Optimization (PPO2) - Stable Baselines (2021)** 300k steps - Deterministic - Rewards (per step) - Training \n",
    "\n",
    "[**Proximal Policy Optimization (PPO2) - Stable Baselines (2021)** 300k steps - Deterministic - Rewards (per step) - Training](https://imgur.com/38rgkAO)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b489061",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/ppo_results/rewards_ppo_deterministic_10k_battles.pdf\", width=800, height=600)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9f1a9e9b",
   "metadata": {},
   "source": [
    "### **Proximal Policy Optimization (PPO2) - Stable Baselines (2021)** 300k steps - Deterministic - Win percentages (per step) - Training\n",
    "[**Proximal Policy Optimization (PPO2) - Stable Baselines (2021)** 300k steps - Deterministic - Win percentages (per step) - Training](https://imgur.com/at7H6gL)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2ed456cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/ppo_results/win_ppo_deterministic_10k_battles.pdf\", width=800, height=600)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0447c6fd",
   "metadata": {},
   "source": [
    "###  **Proximal Policy Optimization (PPO2) - Stable Baselines (2021)** 300k steps - Deterministic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player won $2495/3333$ battles [this is $74,86\\%$ and took $2400$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player won $3324/3333$ battles [this is $99,73\\%$ and took $3600$ seconds]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "005307a7",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9ff08de7",
   "metadata": {},
   "source": [
    "## **Proximal Policy Optimization (PPO2) - Stable Baselines (2021)** 900k steps - Stochastic - Results \n",
    "\n",
    "The graphs of \"Rewards in Training\" and \"Wins in Training\" can be seen at https://github.com/leolellisr/poke_RL/tree/master/images/report/ppo_results. \n",
    "\n",
    "\n",
    "The agent trained for $900k$ steps and $40.000$ battles, resulting in ~$22,5$ steps per battle.\n",
    "\n",
    "\n",
    "### **Proximal Policy Optimization (PPO2) - Stable Baselines (2021)** 900k steps - Stochastic - Rewards (per step) - Training \n",
    "\n",
    "[**Proximal Policy Optimization (PPO2) - Stable Baselines (2021)** 900k steps - Stochastic - Rewards (per step) - Training](https://imgur.com/VBSGmXM)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b986450f",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/ppo_results/rewards_ppo_stochastic_30k_battles.pdf\", width=800, height=600)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c72ab388",
   "metadata": {},
   "source": [
    "### **Proximal Policy Optimization (PPO2) - Stable Baselines (2021)** 900k steps - Stochastic - Win percentages (per step) - Training\n",
    "[ **Proximal Policy Optimization (PPO2) - Stable Baselines (2021)** 900k steps - Stochastic - Win percentages (per step) - Training](https://imgur.com/JTx57ln)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10ad66ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/ppo_results/win_ppo_stochastic_30k_battles.pdf\", width=1000, height=600)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ac1991fd",
   "metadata": {},
   "source": [
    "### **Proximal Policy Optimization (PPO2) - Stable Baselines (2021)** 900k steps - Stochastic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player won $2940/3333$ battles [this is $88,21\\%$ and took $7200$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player won $3322/3333$ battles [this is $99,67\\%$ and took $7200$ seconds]\n",
    "\n",
    "\n",
    "\n",
    "## **Proximal Policy Optimization (PPO2) - Stable Baselines (2021)** 900k steps - Deterministic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\" and \"Wins in Training\" can be seen at https://github.com/leolellisr/poke_RL/tree/master/images/report/ppo_results.\n",
    "\n",
    "The agent trained for $900k$ steps and $45.000$ battles, resulting in ~$20$ steps per battle.\n",
    "\n",
    "### **Proximal Policy Optimization (PPO2) - Stable Baselines (2021)** 900k steps - Deterministic - Rewards (per step) - Training \n",
    "\n",
    "[**Proximal Policy Optimization (PPO2) - Stable Baselines (2021)** 900k steps - Deterministic - Rewards (per step) - Training](https://imgur.com/czHoaza)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17d08c05",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/ppo_results/rewards_ppo_deterministic_30k_battles.pdf\", width=800, height=600)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "eeb15dbd",
   "metadata": {},
   "source": [
    "### **Proximal Policy Optimization (PPO2) - Stable Baselines (2021)** 900k steps - Deterministic - Win percentages (per step) - Training\n",
    "[**Proximal Policy Optimization (PPO2) - Stable Baselines (2021)** 900k steps - Deterministic - Win percentages (per step) - Training](https://imgur.com/HC6dryN)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "062a969b",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/ppo_results/win_ppo_deterministic_30k_battles.pdf\", width=1000, height=600)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "bd19d0d2",
   "metadata": {},
   "source": [
    "###  **Proximal Policy Optimization (PPO2) - Stable Baselines (2021)** 900k steps - Deterministic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player won $2758/3333$ battles [this is $82,75\\%$ and took $7800$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player won $3316/3333$ battles [this is $99,49\\%$ and took $7200$ seconds]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "49b26095",
   "metadata": {},
   "source": [
    "# ** REINFORCE - Keras (2018) **\n",
    "\n",
    "The last method we implemented was the **policy-based** method REINFORCE.\n",
    "\n",
    "* REINFORCE updates parameters by stochastic gradient ascent.\n",
    "* The goal is to compute $\\nabla_{\\theta}J(\\theta)=\\nabla_{\\theta}\\mathbb{E}_{d}[v_{\\pi_{\\theta}}(S)]$\n",
    "* Applying the Policy Gradient Theorem on Monte Carlo Policy Gradient, computing above equal is equivalent to iteratively update $\\theta$ as following: $\\theta\\leftarrow\\theta+\\alpha\\nabla_{\\theta}log\\pi_{\\theta}(s_t, a_t)v_t$, where $v_t=R_{t+1}+{\\gamma}R_{t+2}+{\\gamma^2}R_{t+3} +\\ ...$\n",
    "\n",
    "Hence, we implemented the following algorithm:\n",
    "\n",
    "Initialize $\\theta$ arbitrarily\n",
    "$for$ each episode $(s_1, a_1, r_2, s_2, a_2, r_3,\\ ..., s_{T-1}, a_{T-1}, r_T)$ ~ $\\pi_\\theta$ $do$\n",
    "$\\ \\ \\ \\ for$ $t=1$ $to$ $T-1$ $do$\n",
    "$\\ \\ \\ \\ \\ \\ \\ \\ v_t \\leftarrow$ discounted return from step t\n",
    "$\\ \\ \\ \\ \\ \\ \\ \\ \\theta\\leftarrow\\theta+\\alpha\\nabla_{\\theta}log\\pi_{\\theta}(s_t, a_t)v_t$\n",
    "$return\\ \\theta$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90d78382",
   "metadata": {},
   "outputs": [],
   "source": [
    "class REINFORCEAgent(object):\n",
    "    def __init__(self, nb_states, nb_actions, nb_hidden=128, alpha=0.000025, gamma=0.99):\n",
    "        self.nb_actions = nb_actions\n",
    "        self.action_space = [i for i in range(nb_actions)]\n",
    "        self.gamma = gamma\n",
    "        self.list_of_s = []\n",
    "        self.list_of_a = []\n",
    "        self.list_of_r = []\n",
    "        # model construction\n",
    "        self.model = Sequential()\n",
    "        # 1st hidden layer\n",
    "        self.model.add(Dense(nb_hidden, activation=\"relu\", input_shape=(nb_states,)))\n",
    "        # fix\n",
    "        self.model.add(Flatten())\n",
    "        # 2nd hidden layer\n",
    "        self.model.add(Dense(nb_hidden, activation=\"relu\"))\n",
    "        # output layer\n",
    "        self.model.add(Dense(nb_actions, activation=\"softmax\"))\n",
    "        # set the learning rate\n",
    "        optimizer = tf.keras.optimizers.Adam(alpha)\n",
    "        # loss function\n",
    "        self.model.compile(optimizer=optimizer, loss=self.log_likelihood)\n",
    "\n",
    "    @staticmethod\n",
    "    def log_likelihood(y_true, y_pred):\n",
    "        pi = K.clip(y_pred, 1e-8, 1 - 1e-8)\n",
    "        return K.sum(-K.log(pi) * y_true)\n",
    "\n",
    "    def get_action(self, s):\n",
    "        p = self.model.predict(s[np.newaxis, :])[0]\n",
    "        return np.random.choice(self.action_space, p=p)\n",
    "\n",
    "    def add_sar(self, s, a, r):\n",
    "        self.list_of_s.append(s)\n",
    "        self.list_of_a.append(a)\n",
    "        self.list_of_r.append(r)\n",
    "\n",
    "    def fit(self):\n",
    "        s_array = np.array(self.list_of_s)\n",
    "        list_of_v = []\n",
    "        for r in reversed(self.list_of_r):\n",
    "            if not list_of_v:\n",
    "                list_of_v.append(r)\n",
    "            else:\n",
    "                list_of_v = [r + self.gamma * list_of_v[0]] + list_of_v\n",
    "        # adjust and normalize the list of returns\n",
    "        list_of_v = list_of_v[1:] + [0]\n",
    "        mean = np.mean(list_of_v)\n",
    "        std = np.std(list_of_v)\n",
    "        std = 1 if std == 0 else std\n",
    "        list_of_norm_v = (list_of_v - mean) / std\n",
    "        v_array = np.zeros([len(list_of_norm_v), self.nb_actions])\n",
    "        v_array[np.arange(len(list_of_norm_v)), self.list_of_a] = np.array(list_of_norm_v)\n",
    "        self.model.fit(x=s_array, y=v_array)\n",
    "        # reset\n",
    "        self.list_of_s = []\n",
    "        self.list_of_a = []\n",
    "        self.list_of_r = []"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "38decad6",
   "metadata": {},
   "source": [
    "In method `fit()`, we first normalize the returns $v_t$ then we pass the array of states and the array of normalized returns to `self.model.fit(x=s_array, y=v_array)`. When this latter method is called, operation $log\\pi_{\\theta}(s_t, a_t)v_t$ is executed in `log_likelihood()`.\n",
    "\n",
    "Our internal model has 2 hidden layers of size 128 each, using \"relu\" as the activation function; and the output layer has the same size of our action space (9 actions), and activation function \"softmax\"."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "4f007eca",
   "metadata": {},
   "source": [
    "We applied the follow configurations and hyperparameters for training:\n",
    "* Optimizer: Adam (Learning Rate: $2.5e-4$)\n",
    "* gamma = $0.99$\n",
    "* clip_range = $1e-8$\n",
    "\n",
    "During training, we run Monte Carlo player and store the triple (state, action, reward) at every step: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f06b759e",
   "metadata": {},
   "outputs": [],
   "source": [
    "class MCPlayer(Gen8EnvSinglePlayer):\n",
    "    ...\n",
    "    def choose_move(self, battle):\n",
    "        if self.state is not None:\n",
    "            # observe R, S'\n",
    "            self.reward = self.compute_reward(battle)\n",
    "            next_state = self.embed_battle(battle)\n",
    "            # S <- S'\n",
    "            self.state = next_state\n",
    "        else:\n",
    "            # S first initialization\n",
    "            self.state = self.embed_battle(battle)\n",
    "\n",
    "        # choose A from S using policy pi\n",
    "        self.action = self.agent.get_action(self.state)\n",
    "        # store the (state, action, reward) tuple into the agent\n",
    "        self.agent.add_sar(self.state, self.action, self.reward)\n",
    "        ..."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e524f777",
   "metadata": {},
   "source": [
    "Then we update our agent by a call to `fit()` at the end of each battle (episode)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a85bc5ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "    def _battle_finished_callback(self, battle):\n",
    "        ...\n",
    "        self.agent.fit()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d576b3f8",
   "metadata": {},
   "source": [
    "After training `MCPlayer` against a MaxDamagePlayer we validate our solution both against MaxDamagePlayer and RandomPlayer, which selects random actions at each turn. Our validation ran for $\\frac{1}{3}$ of the number of episodes used in training.\n",
    "\n",
    "## **REINFORCE - Keras (2018)** 10k episodes - Stochastic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at https://app.neptune.ai/henriqueoliveira/rl-pokeenv/e/RLPOK-4/charts. \n",
    "\n",
    "\n",
    "### **REINFORCE - Keras (2018)** 10k episodes - Stochastic - Rewards (per battle) - Training\n",
    "\n",
    "[**REINFORCE - Keras (2018)** 10k episodes - Stochastic - Rewards (per battle) - Training](https://imgur.com/a/AQGAtL1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36ec2c88",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/reward_reinforce_10000_sto.png\", width=800, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "78a79a70",
   "metadata": {},
   "source": [
    "### **REINFORCE - Keras (2018)** 10k episodes - Stochastic - Win percentages (per battle) - Training\n",
    "\n",
    "[**REINFORCE - Keras (2018)** 10k episodes - Stochastic - Win percentages (per battle) - Training](https://imgur.com/a/4w6xxwk)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0656d3e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/win_acc_reinforce_10000_sto.png\", width=1000, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9fb9cebd",
   "metadata": {},
   "source": [
    "### **REINFORCE - Keras (2018)** 10k episodes - Stochastic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player won $2917/3286$ battles [this is $87.52\\%$ and took $956$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player won $3314/3333$ battles [this is $99.43\\%$ and took $924$ seconds]\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "df95c0e5",
   "metadata": {},
   "source": [
    "### **REINFORCE - Keras (2018)** 10k episodes - Stochastic - Memory and CPU usage (absolute time) - Training\n",
    "\n",
    "[**REINFORCE - Keras (2018)** 10k episodes - Stochastic - Memory and CPU usage (absolute time) - Training](https://imgur.com/a/9bgjc8F)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8488b8e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/cpu_and_memory_reinforce_10000_sto.png\", width=1000, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d6388cc6",
   "metadata": {},
   "source": [
    "## **REINFORCE - Keras (2018)** 10k episodes - Deterministic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at https://app.neptune.ai/henriqueoliveira/rl-pokeenv/e/RLPOK-2/charts. \n",
    "\n",
    "\n",
    "### **REINFORCE - Keras (2018)** 10k episodes - Deterministic - Rewards (per battle) - Training\n",
    "\n",
    "[**REINFORCE - Keras (2018)** 10k episodes - Deterministic - Rewards (per battle) - Training](https://imgur.com/a/vqeIf5p)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e2c1847",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/reward_reinforce_10000_det.png\", width=800, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e4de331d",
   "metadata": {},
   "source": [
    "### **REINFORCE - Keras (2018)** 10k episodes - Deterministic - Win percentages (per battle) - Training\n",
    "\n",
    "[**REINFORCE - Keras (2018)** 10k episodes - Deterministic - Win percentages (per battle) - Training](https://imgur.com/a/Xd1jQxe)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17b059ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/win_acc_reinforce_10000_det.png\", width=1000, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "391a00b5",
   "metadata": {},
   "source": [
    "### **REINFORCE - Keras (2018)** 10k episodes - Deterministic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player won $2005/3333$ battles [this is $60,16\\%$ and took $862$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player won $3304/3333$ battles [this is $99,13\\%$ and took $777$ seconds]\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "850f821f",
   "metadata": {},
   "source": [
    "### **REINFORCE - Keras (2018)** 10k episodes - Deterministic - Memory and CPU usage (absolute time) - Training\n",
    "\n",
    "[**REINFORCE - Keras (2018)** 10k episodes - Deterministic - Memory and CPU usage (absolute time) - Training](https://imgur.com/a/eYeH563)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ed0e624",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/cpu_and_memory_reinforce_10000_det.png\", width=1000, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "397c8174",
   "metadata": {},
   "source": [
    "## **REINFORCE - Keras (2018)** 30k episodes - Stochastic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at https://app.neptune.ai/leolellisr/rl-pokeenv/e/RLPOK-199/charts. \n",
    "\n",
    "\n",
    "### **REINFORCE - Keras (2018)** 30k episodes - Stochastic - Rewards (per battle) - Training\n",
    "\n",
    "[**REINFORCE - Keras (2018)** 30k episodes - Stochastic - Rewards (per battle) - Training](https://imgur.com/a/YPm2v9f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6033fe2",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/reward_reinforce_30000_sto.png\", width=800, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "150968fd",
   "metadata": {},
   "source": [
    "### **REINFORCE - Keras (2018)** 30k episodes - Stochastic - Win percentages (per battle) - Training\n",
    "\n",
    "[**REINFORCE - Keras (2018)** 30k episodes - Stochastic - Win percentages (per battle) - Training](https://imgur.com/a/vBgc7Dc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00a45f52",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/win_acc_reinforce_30000_sto.png\", width=1000, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7d620f5b",
   "metadata": {},
   "source": [
    "### **REINFORCE - Keras (2018)** 30k episodes - Stochastic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player won $8507/10000$ battles [this is $85,07\\%$ and took $2424$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player won $9914/10000$ battles [this is $99,14\\%$ and took $2861$ seconds]\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "42e49db4",
   "metadata": {},
   "source": [
    "### **REINFORCE - Keras (2018)** 30k episodes - Stochastic - Memory and CPU usage (absolute time) - Training\n",
    "\n",
    "[**REINFORCE - Keras (2018)** 30k episodes - Stochastic - Memory and CPU usage (absolute time) - Training](https://imgur.com/a/PNK0urg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2043801f",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/cpu_and_memory_reinforce_30000_sto.png\", width=1000, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "60ca87fa",
   "metadata": {},
   "source": [
    "## **REINFORCE - Keras (2018)** 30k episodes - Deterministic - Results \n",
    "\n",
    "The instant values of the graphs of \"Rewards in Training\", \"Wins in Training\" and \"Memory/CPU usage in Traning\" can be seen at https://app.neptune.ai/henriqueoliveira/rl-pokeenv/e/RLPOK-1/charts. \n",
    "\n",
    "\n",
    "### **REINFORCE - Keras (2018)** 30k episodes - Deterministic - Rewards (per battle) - Training\n",
    "\n",
    "[**REINFORCE - Keras (2018)** 30k episodes - Deterministic - Rewards (per battle) - Training](https://imgur.com/a/yGTLQ0Z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad37fef7",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/reward_reinforce_30000_det.png\", width=800, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "fc811559",
   "metadata": {},
   "source": [
    "### **REINFORCE - Keras (2018)** 30k episodes - Deterministic - Win percentages (per battle) - Training\n",
    "\n",
    "[**REINFORCE - Keras (2018)** 30k episodes - Deterministic - Win percentages (per battle) - Training](https://imgur.com/a/DGW32Gl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2fbc4ec3",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "IFrame(\"images/report/win_acc_reinforce_30000_det.png\", width=1000, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "915cc10b",
   "metadata": {},
   "source": [
    "### **REINFORCE - Keras (2018)** 30k episodes - Deterministic - Win percentages - Validation\n",
    "\n",
    "### Versus MaxPlayer\n",
    "* Player won $7597/10000$ battles [this is $75,97\\%$ and took $2749$ seconds]\n",
    "\n",
    "### Versus RandomPlayer\n",
    "* Player won $9955/10000$ battles [this is $99,55\\%$ and took $3835$ seconds]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d6b13780",
   "metadata": {},
   "source": [
    "### **REINFORCE - Keras (2018)** 30k episodes - Deterministic - Memory and CPU usage (absolute time) - Training\n",
    "\n",
    "[**REINFORCE - Keras (2018)** 30k episodes - Deterministic - Memory and CPU usage (absolute time) - Training](https://imgur.com/a/NDdGJFu)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54ac8f62",
   "metadata": {},
   "outputs": [],
   "source": [
    "IFrame(\"images/report/cpu_and_memory_reinforce_30000_det.png\", width=1000, height=600)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c8c1651e",
   "metadata": {},
   "source": [
    "# Pytorch DQN and REINFORCE\n",
    "\n",
    "Our project extends the Keras version to be run also using Pytorch Framework. We intend with these versions to be more receptive and flexible to the PyTorch community who wish also to study deep reinforcement learning with the pokemon environment. In addition, we believe that coding the DQN and REINFORCE learning using another deep learning framework increased and consolidate our knowledge in the area. All discussions about the results yielded are in 'Discussion Section'\n",
    "\n",
    "### DQN - Pytorch\n",
    "The DQN in Pytorch can be found [here](https://github.com/leolellisr/poke_RL/tree/master/py/P2%20-%20Deep%20Reinforcement%20Learning/DQN-pytorch)\n",
    "\n",
    "The DQN agent code was inspired by [Mike github](https://github.com/xkiwilabs/DQN-using-PyTorch-and-ML-Agents)\n",
    "\n",
    "In order to train and test the project in a comparable way, we used the same paramereters and hyperparameters than the ones used in Keras DQN part.\n",
    "\n",
    "The folder has four files representing the DQN agent, the QNetwork by the agent, the replay memory function, and finally the main algorithm that trains our experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "679a7be1",
   "metadata": {},
   "outputs": [],
   "source": [
    "class DQNAgent():\n",
    "\n",
    "    def __init__(self, state_size, action_size, dqn_type='DQN', replay_memory_size=1e5, batch_size=32, gamma=0.99,\n",
    "    \tlearning_rate=1e-3, target_tau=2e-3, update_rate=4, seed=0):\n",
    "\n",
    "        self.dqn_type = dqn_type\n",
    "        self.state_size = state_size\n",
    "        self.action_size = action_size\n",
    "        self.buffer_size = int(replay_memory_size)\n",
    "        self.batch_size = batch_size\n",
    "        self.gamma = gamma\n",
    "        self.learn_rate = learning_rate\n",
    "        self.tau = target_tau\n",
    "        self.update_rate = update_rate\n",
    "        self.seed = random.seed(seed)\n",
    "\n",
    "        self.network = QNetwork(state_size, action_size, seed).to(device)\n",
    "        self.target_network = QNetwork(state_size, action_size, seed).to(device)\n",
    "        self.optimizer = optim.Adam(self.network.parameters(), lr=self.learn_rate)\n",
    "\n",
    "        # Replay memory\n",
    "        self.memory = ReplayBuffer(action_size, self.buffer_size, self.batch_size, seed)\n",
    "\n",
    "        # Initialize time step (for updating every UPDATE_EVERY steps)\n",
    "        self.t_step = 0\n",
    "\n",
    "    def step(self, state, action, reward, next_state, done):\n",
    "        # Save experience in replay memory\n",
    "        self.memory.add(state, action, reward, next_state, done)\n",
    "        \n",
    "        self.t_step = (self.t_step + 1) % self.update_rate\n",
    "        if self.t_step == 0:\n",
    "            if len(self.memory) > self.batch_size:\n",
    "                experiences = self.memory.sample()\n",
    "                self.learn(experiences, self.gamma)\n",
    "\n",
    "    def act(self, state, eps=0.0):\n",
    "        state = torch.from_numpy(state).float().unsqueeze(0).to(device)\n",
    "        self.network.eval()\n",
    "        with torch.no_grad():\n",
    "            action_values = self.network(state)\n",
    "        self.network.train()\n",
    "\n",
    "        # Epsilon-greedy action selection\n",
    "        if random.random() > eps:\n",
    "            return np.argmax(action_values.cpu().data.numpy())\n",
    "        else:\n",
    "            return random.choice(np.arange(self.action_size))\n",
    "        \n",
    "    def learn(self, experiences, gamma, DQN=True):\n",
    "        states, actions, rewards, next_states, dones = experiences\n",
    "        Qsa = self.network(states).gather(1, actions)\n",
    "\n",
    "\n",
    "        if (self.dqn_type == 'DDQN'):\n",
    "            Qsa_prime_actions = self.network(next_states).detach().max(1)[1].unsqueeze(1)\n",
    "            Qsa_prime_targets = self.target_network(next_states)[Qsa_prime_actions].unsqueeze(1)\n",
    "\n",
    "        else:\n",
    "            Qsa_prime_target_values = self.target_network(next_states).detach()\n",
    "            Qsa_prime_targets = Qsa_prime_target_values.max(1)[0].unsqueeze(1)        \n",
    "\n",
    "        \n",
    "        # Compute Q targets for current states \n",
    "        Qsa_targets = rewards + (gamma * Qsa_prime_targets * (1 - dones))\n",
    "        \n",
    "        # Compute loss (error)\n",
    "        loss = F.mse_loss(Qsa, Qsa_targets)\n",
    "\n",
    "        # Minimize the loss\n",
    "        self.optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        self.optimizer.step()\n",
    "\n",
    "        self.soft_update(self.network, self.target_network, self.tau)\n",
    "\n",
    "    def soft_update(self, local_model, target_model, tau):\n",
    "        for target_param, local_param in zip(target_model.parameters(), local_model.parameters()):\n",
    "            target_param.data.copy_(tau*local_param.data + (1.0-tau)*target_param.data)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "1023562f",
   "metadata": {},
   "source": [
    "### REINFORCE - Pytorch\n",
    "The REINFORCE in Pytorch can be found [here](https://github.com/leolellisr/poke_RL/tree/master/py/P2%20-%20Deep%20Reinforcement%20Learning/REINFORCE-torch)\n",
    "\n",
    "The Reinforce agent code was inspired by [Ching-Yao github](https://github.com/chingyaoc/pytorch-REINFORCE)\n",
    "\n",
    "In order to train and test the project in a comparable way, we used the same paramereters and hyperparameters than the ones used in Keras Reinforce part.\n",
    "\n",
    "The four has three files. A file to create our Policy class that uses a deep learning model to create our Reinforce agent, a normalized action function and the main algorithm. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "edcba119",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Policy(nn.Module):\n",
    "    def __init__(self, hidden_size, num_inputs, action_space):\n",
    "        super(Policy, self).__init__()\n",
    "        self.action_space = action_space\n",
    "        num_outputs = action_space\n",
    "\n",
    "        self.linear1 = nn.Linear(num_inputs, hidden_size)\n",
    "        self.linear2 = nn.Linear(hidden_size, num_outputs)\n",
    "\n",
    "    def forward(self, inputs):\n",
    "        x = inputs\n",
    "        x = F.relu(self.linear1(x))\n",
    "        action_scores = self.linear2(x)\n",
    "        return action_scores\n",
    "\n",
    "\n",
    "class REINFORCE:\n",
    "    def __init__(self, hidden_size, num_inputs, action_space, lr):\n",
    "        self.action_space = action_space\n",
    "        self.model = Policy(hidden_size, num_inputs, action_space).float()\n",
    "        self.model = self.model.cuda()\n",
    "        self.optimizer = optim.Adam(self.model.parameters(), lr=lr)\n",
    "        self.model.train()\n",
    "\n",
    "    def select_action(self, state):\n",
    "        state = torch.from_numpy(numpy.array(state)).float()\n",
    "        probs = self.model(Variable(state).float().cuda())  \n",
    "        #action = probs.multinomial(1).data\n",
    "        action = torch.argmax(probs).data\n",
    "        prob = probs[action].view(1, -1)\n",
    "        log_prob = prob.log()\n",
    "        entropy = - (probs*probs.log()).sum()\n",
    "\n",
    "        return action, log_prob, entropy\n",
    "\n",
    "    def update_parameters(self, rewards, log_probs, entropies, gamma):\n",
    "        R = torch.zeros(1, 1)\n",
    "        loss = 0\n",
    "        for i in reversed(range(len(rewards))):\n",
    "            R = gamma * R + rewards[i]\n",
    "            loss = loss - (log_probs[i]*(Variable(R).cuda())).sum() - (0.0001*entropies[i].cuda()).sum()\n",
    "        loss = loss / len(rewards)\n",
    "        \n",
    "        self.optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        utils.clip_grad_norm_(self.model.parameters(), 40)\n",
    "        self.optimizer.step()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "4dcc1f14",
   "metadata": {},
   "source": [
    "### Aditional Information about Pytorch Framework\n",
    "Besides allowing to use of the Pytorch Framework that increases its importance in the AI community, the PyTorch version allows the user to run the code using GPUs as a booster. It is important to point out, however, that the main bottleneck of our project is the Pokemon showdown. \n",
    "\n",
    "We highlight that in the Reinforce experiments carried out using PyTorch, we had results similar to those presented by Keras in the same configurations. However, the Keras framework performed slightly better for DQN.\n",
    "For Pytorch DQN model in stochastice environment with 5k battles, the model won around $30,00\\%$ against MaxDamagePlayer, while with the RandomDamagePlayer the agent won almost $95,00\\%$ of the battle.\n",
    "\n",
    "It is important to point out that a noticeable improvement over time was noticed. Using Using the Keras framework for the DQN took about 4h to train the model while using Pytorch with GPU took about 1h30 to train for 15000 battles.\n",
    "Using the Keras framework for the Reinforcement technique took about 4h to train the model while using Pytorch with GPU took about 2h to train for 30000 battles."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "136a9e4a",
   "metadata": {},
   "source": [
    "\n",
    "# Limitations\n",
    "\n",
    "As our previous project, the only limitations of this project with types of games or teams are in the use of the **deterministic** environment. Given the need to remove **randomness**, our deterministic solutions require the use of modified Pokémon Showdown, without the random parameter in damage calculation, and the Pokémon on both teams with:\n",
    "- Shell Armor or Battle Armor abilities, to prevent critical hits;\n",
    "- Moves with 100% accuracy and no side effects likely to occur.\n",
    "\n",
    "Our **stochastic** solutions can be applied to any case and with any team formation.\n",
    "\n",
    "We also found some libraries limitations in the use of PPO2, from Stable Baselines (2021). The requirements listed [here](https://github.com/leolellisr/poke_RL/blob/master/requirements_poke_env_ppo.yml) must be used. Neptune cannot be used.\n",
    "\n",
    "# Related work\n",
    "\n",
    "Given the stochasticity of the Pokémon game, the environment proves to be a good alternative to validate more robust reinforcement learning systems. However, in recent years, few reinforcement learning works have been willing to use Pokémon battles to validate their models.\n",
    "\n",
    "DQN or Deep-Q Networks were first proposed by DeepMind back in 2015 in an attempt to bring the advantages of deep learning to reinforcement learning (RL). Reinforcement learning focuses on training agents to take any action at a particular stage in an environment to maximise rewards. Reinforcement learning then tries to train the model to improve itself and its choices by observing rewards through interactions with the environment.\n",
    "The memory and computation required for the Q-value algorithm for every possible combination of state and action in a complex environment would be huge. Thus, a deep network Q-Learning function approximator is used instead. This learning algorithm is called Deep Q-Network (DQN). The key idea in this development was thus to use deep neural networks to represent the Q-network and train this network to predict total reward. This propose approach is called off-policy method.\n",
    "\n",
    "Amongst deep reinforcement learning we also find is the policy-Gradient methods which are subclass of Policy-Based methods that estimate an optimal policy’s weights through gradient ascent. \n",
    "REINFORCE algorithm is a vanilla version of the policy gradient method. This algorithm is the fundamental policy gradient algorithm on which nearly all the advanced policy gradient algorithms are based.\n",
    "\n",
    "Regarding deep reinforcement learning methods, the work of [Huang, D. & Lee, S. 2019](https://ieeexplore.ieee.org/abstract/document/8848014) can be mentioned. The authors proposes the use of PPO with an actor-critic neural network with $1.327.618$ parameters. Actor and critic are represented by a two-headed neural network $f_\\theta$ (parameterized by $\\theta$). To reduce the variance of policy gradient estimates, they use Generalized Advantage Estimation. The authors defined as inputs a list of features for each Pokémon in battle (species, item, ability, moveset, last move used, stats, boosts, HP, PP used, status, types etc). They used 128-dimensional entity embedding layers for each of the categorical variables. This enables capture similarities between different moves, species, items, and abilities without having to directly model their effects. The network has two outputs: a probability distribution over actions to take and an estimate of player strength in the current state. The author used 3 different teams for validation and reported an average of $92,9\\%$ wins against a MaxPlayer and an average of $99,5\\%$ wins against a RandomPlayer.\n",
    "\n",
    "\n",
    "# Comparisons\n",
    "\n",
    "One table with performance comparisons in Validation between our methods for P2, our best method from P1 and the method proposed by Huang & Lee are showed below and in [this figure](https://imgur.com/4A4X54z).\n",
    "\n",
    "The method with the best performance in the Stochastic environment against both Players (MaxDamagePlayer and RandomPlayer) was **Proximal Policy Optimization (PPO2) - Stable Baselines (2021)** trained by 300k steps.\n",
    "\n",
    "The method with the best performance in the Deterministic environment against both Players (MaxDamagePlayer and RandomPlayer) was **Proximal Policy Optimization (PPO2) - Stable Baselines (2021)** trained by 900k steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "56a5d9d1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"1000\"\n",
       "            height=\"1000\"\n",
       "            src=\"images/report/comparisons_valP2.png\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x207472e58e0>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame(\"images/report/comparisons_valP2.png\", width=1000, height=1000)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6157ee08",
   "metadata": {},
   "source": [
    "# Video\n",
    "\n",
    "https://youtu.be/pgIfTQnjdzA\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "9ebf7fe9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAUDBBAQDRAQEBAQDg0QEBAQDxAQDg8QDRAQEA4PEA4NEBAODRAODg0ODg0ODRUQEBERExMTDxAWGBYSGBASExIBBQUFCAcIDwkJDxcVEhUVFRUVFxUXFRUVFxcXFRUVFRUVFRUVFRUVFxUVFRUVFRUVFRUVFRUVFRUVFRUVFRUVFf/AABEIAWgB4AMBIgACEQEDEQH/xAAdAAABBAMBAQAAAAAAAAAAAAAABQYHCAIDBAEJ/8QAXxAAAgECBAIGBAgHCwgIBQMFAQIDBBEABRIhBjEHEyJBUWEUMnGBCCNCUpGhsdQVVGJywdHwGCQzU4KSk5SV1eEXNENEY6Ky8RZVdbS1wtLTc4OFs8RFpMMlNXSEo//EABsBAAEFAQEAAAAAAAAAAAAAAAABAgMEBQYH/8QANxEAAQQBAgQEBAUDBQEBAQAAAQACAxEEEiEFMUFREyJhcQaBkaEUMrHB8ELR4RUjUnLxYpIH/9oADAMBAAIRAxEAPwCmWDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDBgwIRgwYMCEYMGDAhGDE2/uZ6/wDjaT+kn+7YP3M9f/G0n9JP92xL4L+yo/6njf8AMKEsGJ2y/wCC3mDtpE1GDa+8k9u7wpT44wk+C7mIFy9KB4l57f8AdcN8N11Sf+PgrVqFKDMGJ0ovgv17uqiajBYgC8s9gSbC9qU8/wBOFup+BxmamxnoPdLU/c8Nc0t5qWHIjlFsNquGDFkKX4G2aMdp6D3y1Nv+546f3FWa/jGX/wBNU/csIplWbBiylT8DDNFFzPQW8pqn7ljmb4HuZjnPQj/5tT9zwloVdMGJ+zz4J2ZxRPJ1lJIEUsUjknaQgbtpU0q6mAu2kG5tYAkgGPn6KKqwYGJ0IuHR2YEc7gBNZ28FJ8sIXAKCXJiiID3AXytMLBiVeFOhGaqNoqygL/xbS1CSD+Q1IG+i+HfTfBIzFvVqKA25jrai49oNHce8YgbmQudoDhfbkfoVa8N2nVW3fmFXvBixEvwQsyH+nof6Wo+541fuScy/j6H+lqPumLQFqAytHMqvmDFgz8EnMv4+h/paj7pjwfBKzL+Pof6Wo+6YXSUnjs7qvuDFj8v+Bvmji4noAAbby1P6KM43VHwMM0UXNRQe6ap+5YadlICCLCrVgxZL9xpmn8fQf01T9zxrf4HeZA2NTlwPgZ6gH/ueGeI3unUVXHBixn7j7MfxrLf6xP8Ac8bf3G+ZfjWW/wBYqPueFDwUUq3YMWR/cb5l+NZb/WKj7ng/cb5l+NZb/WKj7nhdQSKt2DFkf3G+ZfjWW/1io+54P3G+ZfjWW/1io+54NQQq3YMWR/cb5l+NZb/WKj7ng/cb5l+NZb/WKj7ng1BCrdgxZH9xvmX41lv9YqPueD9xvmX41lv9YqPueDUEKt2DFkf3G+ZfjWW/1io+54P3G+ZfjWW/1io+54NQQq3YMWR/cb5l+NZb/WKj7ng/cb5l+NZb/WKj7ng1BCrdgxZH9xvmX41lv9YqPueD9xvmX41lv9YqPueDUEKt2DFkf3G+ZfjWW/1io+541t8DzMfxrLf6xP8Ac8IXgJaVcsGLGfuPsx/Gst/rE/3PHTF8C/ND/rGX/wBNU/csAe0ooqtWDFl5PgW5oP8AWMv/AKap+5Y1fuNM0/j6D+mqfueAvA5oAJVbcGLJH4Gmafx9B/TVP3PHifA1zT+PoB7Zqj7nhPEb3RRVbsGLIt8DXNPxig/pqn7njSfgfZn/AB9D/S1H3PCh4PJIdlXTBiyNJ8DTNGvaeg28Zqn7njKr+Bnmai5qMv8A6apuf/2WHJLVbMGLDfuRcy/j6H+lqPumPD8EbMv4+h/paj7pgTRI08iq9YMWE/cj5l/H0P8AS1H3TGyP4IWZH/T0P9LUfc8JYTrCrvgxYlvggZkP9PQ/0tR9zxq/cj5l/H0P9LUfdMFhFhV7wYsGfgk5lcDrqIk8gJai5/8A2mNlV8EDNFFzLRAf/GmNvbal2wWiwq8YMT7+5RzH+Oov6Wo+6Y2QfBKzJjZZaMnykqD/APiYLSagrQY9vjXqwXxsrzROXhGn9ZyOdgp8r9r3XA+jCrRUmkFb3QkkAjkDzXzF793fjm4ekvCvvH0Mcd5bGfI4lxXV4kTGxN9v1Tezeh6tldOWoEDnpYbgew2uPYR4Yk/KKxZI1deTC/v5Ee0EEe7Dby2EOxRuTAjzvzBHgwIBB7jjKHhqZBojqSsY5DQLi5udwRfc37sI5weADzCs4sToHl0bbaegrY/NOqOIAmwtfnjZjhymldFs8hlPiQAfZt3Y7sQLYabFkUjHHm1tPK5Ow9px2YxdAcCVI8tCdAIvq7x+3hivvS3wgKeQzxACmmftqOUUzHmByEU7ezTKe/rRps3bDP4pyuOZJYZAGjkDIwI7j+kHceYGGEUs/iGE3JiLD8vdVHz7huKbdl0vzDrs4I5G45/b5jGjIOkKsy+RVmZqin7iWvKqjYmN25270fntc2IOFLPchqqSeSJmEug9kP2S8Rv1ciSAm+oAqdYNmVh3YS67MY3UpURtED88XS/isi3UEeJtirk4kWQ3TIL/AFHqDzC5LBz8rh8ulrthzHP7FWY4J44hrIQ0bq1+RG3a+YyneOTusefdzGF6+KKJPPl9QJIH7JFwRvHKnzWAJVgOXip3HMXtN0R9JMdbCD6si7OpPaQ91/FT3N9OKcGTJguEeQ7Uwmmv6jsH/s76rtGuj4gzxIRTgN29/Vv7jopCc4wvjJseBcdGss3afVBEqqAvLu8/P388cGaSnVbuH1+eN+SVWtAe8bH2j9Y3xpzc9oez9OKj1tNI0ghZWxGnBPBFFU1eayVNHS1MgzAIHnpoZXCDLMvIQNIjMFDMzab2uxPecSSjXGGt0Tf5xm3/AGkP/C8txVg/MVO47Ji8XfgenqpqZcgFXJTwpPKaXLMvdVSTXp2kkjkZj1bdlUJ22vhy8O8NZDPTw1EdFlnVTojxl6OkRiJDpQaWjBDF7x2+cCBc4Ts36NZKnOa2aSSqp6WWlpYkemqBD1zIZutjfTeSyB1sez6xsT3clF0SomfU8i0qfg6lyhIKZ3CSCKrjrTJEVWRmk69I2aQSkc2J1aicWlGnhT9HmStI0S0OVtKgu8YpKQyKPFkEepRv3jCXlHCeSvTJUS5ZQUiOzKFqaSgRgwdkCkp1kRLaCyhZGNiL2NwI+4L4BqIRlqS0q0smW1M9TWZo01OY6iIpOJdMglNU/pgkR5evSMIEIJNlwz+AODaifLcqqYUlqIIkzaJo4Fy95VaevlKSqmZo9M0UiIYnZfjApFtSlwRCsHmfR9ksQUy0OVxBzZDJS0aBj3KuqMaj5DHDS8J5G1ZJRrQZcamKNJZE9Apuyrlgm5isWOhiQtyosTa4uwKTo/qaVqcvl7ZnAMnFEtO9TSyPTziVnMMjz+jRNBLHIkLVEKEgQAFCNN3hwVwjJDnT1DUSRxTZfSRrLCYXip5oRKJodTGOoI0GKJZFhs6qt9IWyqhIGYvlEdUInyGJac1iUAq2y6iWE1UltCLGwFQ0JYhOvEfV6uRI3xI0XRflBJAy7LiVNmAoqUlSRcAgRXBI33wxcyyvM5s5SeegWehp5gKIfhCOOOBSdEmYSU4ic1NV1bM8as6iJTpVQ5MmJT4cgAnqiKX0ctIhab4v99ERKBL2CX+LHxXxgB7O21jgQoSzbPcjiapLZCDT0dQ1PU1SZXl708Tpo1uQshnMarIjkiEkKeXMYkmTgHJAyIaHK1eUXiU0lGHkB5FFMepx7AcMnhroZ9IkzP01quOnqMymlSnjqjHTVEBSHRJIkLajrZWUgsrWRdhYHCb0i9H9S0maxLQCqbMDT+hVnWU6x0ipFHGFl6yRaiAUbo1Qno8cuvVtZrjAhPHKeHckkqKyD8GUMbUTwpK8lDRrEzTwrKnVtpJNlYA6gna2F+eOo8GZP6b6IMroDJ1PXM3odEFUatKppK9aztu/ZQqqi7MupA0bZv0Y1wzmavMbVUENZQSpTMYtFSFoVpp61Q0gHpVI562IS6ACr6TdgcZ5D0YVi5mGljqWVc2lrlqY5cuSm6ltbIWcwPmjyMhFG9KSsencSBFXAhSg/A+RgkGjykEIZCDTUVxGpIaQ9jZFIILcgQd9sZ1PAWSLEJWosqWE20ytS0YiOr1bOY9J1d1jviK+CuiGRRk5moo9cFdmM9WWWBiqStUGmdzqOu59GKhdRUqlwNG2fCXR9VUy5fJJQGqgpXzVGo0am1wiqqmkpqmJJpkp3HVDqivWKyLLcA9oBAUKVKvo7yVF1PQ5WilS4ZqSkVdAAJe5jA0gEEtyFx44Kno5ycQtMMty94whkulFSMGUKWup6vSbgbb2PjiNeC+iyoWfJjUUyGGnObSyRao5IqMVTI9JTbm0nVC6AxqyKy7GwVi9OhbhWemyI0skXVTD04LFdDZZamoeEAqxQBkkUgX2vvbfAeSEy8izrJJBSu+Qimp614o6eomyvL+od5v4FSYJJpE607BmQKO8gb4ePAXDWS1vpPVZVRr6LWT0UnWZfRjVLBpDumhXvEdY0ltLGxuo7474X6IqqmXJp2SorBTiEVeXz1jSRUsoQhK+lR5xAslKxK9WNa6WHVqrDWHV0UmtoZcxjfLaqZKnNqyqilikouq6md0EbESViSg2QuRovYjvuA5IlPpn6MstjynMJI8voY5Eoat0dKOnV0daaQq6ssQZWVgGDAgggEYlTKUtGn5q3/mjDX6dv/7Lmf8A2fWf91lwvUs10Tw0r9gxFI4N3KcBa31Mt/ZjSMBOPAMUnOJNqUbL04xmlAxpnqgOW5+rHEz3w9rL3PJMdJWwWyaa/wCrGsnHHUZlGuxYX8Nz9gOO/I5VkYWZTbc2O/lccxue/Cx5ULnaGPaT2BFqMsfzIKUskB322254Sc3n1OfAEgD2bH6ThzTSgAk7AYZ9ZPqYta1z+3vxdCr5B2ACxLYwvjAnGSD3DDHG01jNIWyJLnHRLJYbY1CYAfZ+s453e/PCKS6WbNhT4cp1fVcXtYeW97+/YYbtRU9w+n9WFrgzMFUMrEKSbgnYHa1vbt774UIad0pVkcUd7KQzDmD2gPIsdvdhIkUA7Xt3XFjhbz1R6211G3jcmwJ8huR54bxb9v0YlAUGQ7dc8iKAR3+O+3l4Y21k+mNEBsCCzW+USxAvbwA5ezwxk2E+SAjzw9jW2qkkr9NJljfzx5q93txwVHDiyDXC3f6rDkbb7mxH0HCK2ZywSBJNegsBy1Kq3W7G4YFba2BUC9gvM75MHxMzI3hAPoTTv0U8vwUY2+aQg+236p60uZOosrWHhYEc/MbY7q3MzJEBsG1doW5jmCO4bjcc7+XNo0mdXvdeRIVgNIYAkXKlmKggauffy2w4qCIst7ADusb93sGJx8R4F/7rtLhzB3+4sKnJ8McUhFNbqaRsQf2NFdVJnMgsPDltz9p53HiLYefDHFqnsSHSe5iez7Ce7yv7L8sMeSmPtxpJxp42TiZrbheD7Hf6KgTm8PcPFaR/25H5qR824yjQ6VBlPeQQFHkDY39wx7kHGCyMFZerY+rvdSfC9hY+Hj9AMbXx7fFn8IyqSf61Pr1bV2/ypwU49whcEZiZIFJ3Zbqx8SOR9pUi/nfC7jOc3SaK6qKQSMDxyIQcNvNEs589x78OTCbniqRuQG7v08sNKe4KLul7hzr6frFHx0N2W3Nk/wBLH57DWo+coG2o4gu+LU4q1mUWmR1+a7r/ADWI/RhoXGfEWO1r2yDmdj8k0uK8hZkPVWtfUY+6/e0fgWHNfVPO1wLsrgviB6OqWRbhb2de4r3gjxU38+fjiW74aXH+RK6GQDtDd7d6/O/OXnfwve9hiLIgZNGY3iwRRVXhfE348jTe4OxVqODM8WeFWU3BAI33t4e1Tt9Hjhd1Yq18HPjQxOaZzsDeMnlY7W9ncfC45acWgpn1KGHqn6Qe8HwI/bnijwXLcwnCmPnZ+Un+pvQ+45FdvxCASMGVEPK7mB/S7qP7Je4VlIcjuI3+vfHbmcl3PlthJyjMRGfVvfv7/wBWI5y7iGWas6lq6WB6iprY6ZUo6ZogtKxOhpJELCVowzAG+rS3hvsyt3UWKQWValWKS2GdllJmFPUVjU8VDNDVVIqFM1ZUQyr+9KaAoyR5fOuxpi1xIbhhsLYQ5cyYGpX8JVJalligYLl9Kxlmm7MccAWImU9ZeJjYKjI+ohUZh38MRVUsphetq6SpCdaIaijy7U8WoKZUanM8Eio7KrBZC0ZdNar1iaoC08wrbfVOD8MZv+K5b/aVX/dGD8MZv+K5b/aVX/dGEzOYpoqimp3zSXr6ppFiUUVGf4OGSZ2Y9VZV0xMATuWIABsxVOoswlaQq2Y1USaZ2SaXL6RIJFpm01DIxiuBGdwZFTrFBePWgLBtSdwn7JxVNZmkisjUeVujAqytmNUVYHYqVbJ7EEbEHGdLU5qihUo8rRFACquY1QVQOQAGTgADwGEbhWOqqCQK6siOhJF6/LqWPrInvokW8Rt6p1RvplS660TUt3GOFa3/AKzl/qlH/wCzhw19aSbLV+Ec3/FMt/tKr/ujB+Ec3/FMt/tKr/ujDLi4kqCpkNZXLTLM8DVJoMvMKvHUNTOxCFphGs6MpcxWAGo2W7Ydwyqp63qvwueu06uq9Hoet0/P0dXq0+drYXzeiFu/COb/AIplv9pVf90YPwjm/wCKZb/aVX/dGE+lpqgrEWzZonmClI5IMvEhZ1DCMBVZXcA8o2cHmCRvhPoMwldqgfhZ0jpnWKSaSny9IOsIuUViNV0uFJZVUk2UtZrHm9EicH4Rzf8AFMt/tKr/ALowfhHN/wAUy3+0qv8AujGmXKqgFAc3IaSwjBp6EFyQSAgMd2JCsRpvcAnuONMVJMUeQZyDHGSJHENAUjK+sHYJpQr3hiLYPMhdn4Rzf8Uy3+0qv+6MH4Qzf8Uy3+0qv+6MclRRzLGJGznTEwDK5goBGykqAwYx6SCXUXBtdlHeMbIssqCZAM3JMW0oFPQkxm17SWj7Btv2rbYPMlW/8I5v+KZb/aVX/dGMHzXNx/quW/2lV/3RjnFHMYutGc/EkEiXqKDqiFuWIfq9BChSTvtY+GOVKaYzpAM3ZpZYmnQCloiGiVlUuCIrEEuCLcwGPIHCHX0pGyUBm+b/AIrlv9o1f90Y9bNs3H+q5b/aVX/dGOLJsvqZZaiJMzm100iRS3oqMDU8EU66T1W46uZLnbe47sdEOSVDSNEM3JmQBnjFPQmRQeTMgj1KDcWJFjhKk9Euyz/C+b/iuW/2lV/3Rg/DGb/iuW/2lV/3RjmostmfXoznX1Z0yaYKBtDE2CvaM6WJ2sbHGMNDKertnQPW/wAFaGgPWbavi7R9vskN2b7G+EqT0R5Vw8dJm1TRVNMYMtjFRTzQF/whVto66Jo9en8EjVp1arXF7WuMPmJyFUXAAABbxsLHTfn7cM2bK5wmqXNnhXrHjAlgy9QSkrRAghSvxjLdQTq3AYK11G6q4bmVxG2blZWBZUaChDso5sFMeoqO8gWGAxl35kl9k6zVAbDfzONEkxPM/qwj/wDQWr/6yl/qlH/7OErgSpm11sU0vXmmrOpSQxxxsUNFRz2KxqFuHqHF7creGFEYCY4uS7nmaJDC8shIjjUs1tzYcgBcAsxIUAkbkbjFUekHpurS0jwy9QpssaKkbqovt/CxtqexJZyN7WAVQBiXvhQZkVolQMAHks6g9o2UFQQPk2Jbe24XyxT3i7MlICKdTg3su/ceZ5X8sVJwJD4ZFjr2St8rbHNWD6MemVZ45PS9EM0QVtS6tMwYkHQhLOJFYC6gsLMCLAGyPxH8JEU7nqYwJFPZ1MXe219Ua6UU2PqmU+/Fd8xztz2U+Kj8B67Hv1Nsb+z33whNAN/PGZD8O4jJvG09iADsPZT/AIt5bpJVi6j4YVYygGOBrd5hf6SEq1B+rEnfB26YKzM3cSUkawR311MbuiBiLpEIpOtLyNzIWUBF3J3UNUPgXh2KaeMSFup1hZVSWOKYq17MkkyGBbHf4wgWU7j1hf8A6Lsip6akSOlhNPDctpZld2ZgNUjSJLKkhawGpZGWygLZVUDoARyCqyNA3ITrwFsYF8Yu9ueBQ2sy2OKeovsOX2401FRf2YKVbkDxIH0m2GSO0tJSA2aW2elYcwR4XGxx7l/8Il+Wtb+zUL/VhRzHM2WRgD2b20nddtrWPs8sZ5fDHM1tJQgXOkjQQCL8/Vv5YpMyZObhspjG26BShnRZZHvurgW9gAt9DAn3+eExsKeYN1hupVgBsAe19BthOmjI5gj2jF2PKjO17qpPG6yRyXl8c9TU22HPG4nHLDSl3sN/sA8T5Yn8VjRqcdlVLXu2bzUe5BmCIvquX3vb1T4c2tfztfnhr9InF6oB1jLrU3WJCDLYkXYgkchvclQbbb7YiTMeI6qRbNLJp/Jsl/6MKSMN2am3uCNzvcWsfM+J8frx57g8MhheJNR1D5BdbP8AGGPOXNjFX1d+yn3h/iCOZQysLj5Q8fBlIBRvba/0YftCNroxZTzBJY+4sTY+WKhUtQ8TalJRh3j7DzBHkbjyxIHCXSayECW47taXt/KX/wBNx+SOeH8U4S3JGuGg7r2P+Vo8P4214DZTt35qwNLV2fTuQSQboV3HmFCm/u9px3VEIPtt7x5efjhp8McWxSKDqDflqbg/nAHY/tth35GDq+SQbnVt2gRyYcjYAdob2HhfHMQPyMKe2ktcP58wtLNx4cmGngOafp7j+WkyRSOf+GMNWN+fIV7aWNt3QcnXvttcOtrjv2tvtjlgkDrqQ3X6x7cepcC+KmZdR5FNfyB6E/sfReY8a+FZMVvjQW5nM92/3HqnRwdmMkYJUB0J7SXs1wPWBO17dx2Nhyw8sv4pjd9BDIxtbWLAnwBuRfu8+6+I6yCtsdG255997cvqt3fXhYq4Ay2PuPeD3EY35mDXuFSwcp7YhoN10P6KSg2EjOKFi2ob7AW7/r2ths8P5o4dWYsdRVHUm632QOo+STsxtsbnyIfoxVeytlu484mbY2TD4q69InMCI9RbsJK5jjJuLksqMdhcgWAJABZQSwp3xnnVbT1DpUU6RylixBksraiTrSQ2jdD4qedwbEEC+2ZUYcefcf27jhhcWcNQVUfVVESyKCbah2kb5yNzU7d2x5EEbYjA7qpxDCbOASLI78lTaPjgo+moheIkAgi+6m9nAYDUhts6lgbbXw6MtzKOVbowde+3MeTA7j2EDG3p06JjRx9bExkoy1mjc3aJmNgy8uyWIQstmBK7tqOiECGjbUhK25MpII8rjf8AQcJyXOz8MjJoeU/ZOPiqkammDpsL3jPdY80Pmpt9R78Wg6EOPo5Io1ZxqeNWIN+y1rEEkWOltSE/KChvG1S824nlliEcmlgGDarWbYEWNuz39wGLQfBPyxDRFmUMbKouL2vqLWvyJ2xi8Ux3OkifFs8HY+lGwfRdV8PyOZFJFNu2twO9gAjsVN0lOD5ft4YjDIuHKiTr5oE6x6eor5IgJETrKiLMaWeOm1OboKqOGWnLkaVV2uRcYlCJbADwAG/PYW+nDXn6P6Yu7/vlGkdpHEWY5hChdzd2EcNWkaljudKgY2w91DUnta1riQuKl4CqoiswjEs0ElHVFVeONaufqqxcwVdTEI7vXTTx9aVQy9WC4XU6u/KaaaozCKqeCSkhp6aeFFmaEzSyVMlM7tpglmRIoVpFUFn1O7t2QqK0jTzTgSJY2MfpkkluypzjNV39vpxFwNwCNyLbXxHXDdMnVjrXrzpYI00ma16Ru+ykDRXjSvWHRqAN9uW4xRzeJw4leJe/KhfotbE4dLkxl8dUNjvupSzngWu/CNPUrPTSp6eZpCaOQTQ060NbDFEZDmWmSOMVLRKscKWmqGqCpHWo/HxjwjV1zyK0HokrU9dSS1JmWWlmp6iCSKFIIhK00ZMvo1VKzxQMDA0YaUNqxspuBaVgCHrSDyP4WzW3Lzrgfq8uYxsPR9T/AD63+1s0+/YuNeHCwqBNGilbovyOVKh3NNJRRdRHHIklSk5qKlWJaqGiWUWC9nr5Sk8+v4xF6pLvfK6SVXcvN1qsbovVqnVi5OnUpu+xAufm378Rj/k+p/n1v9rZp9+x7/k+p/n1v9rZp9+w+03UEl0/RfKlMs2iaWpjzCqqXonrZWo6mF8xnljUQGq9CjmELx1URKqvXqvW2Jdl3ZVwDUrmF3FS6fhCWtEwbLEpQsjOUXUaR81adIXWiKaxG0SkCZI7RDvTgGm+dWH/AOrZr+ivxuXgOk7zXe7N81P/AOeMNL66J2oFN7IOjedcvkR6ZfSvwZldOl2hLdfSRt1iK+shTDLpYPqClrFSbXC3JwnLFU+kik9IRayrk6hDTLKwniiSKrj66SOEyRaZotMkkbBJ5SLnsv2J0f0fzq335tm33/G5OjakPJqw/wD1fNfv+IzMBzBTg3sU38l6MZ+pnjKpBJJl9VBTyKQRSNVVdXNHTRlSJFSkjlp4wYwqWhXR6oAw4l4TqJ442ioDQejyUjMkTZe1TUrTx1SrFCsgnoeppnqI54DVaGLI3Yp2WOQOT/JpS+Nb/a+a/f8AB/kzpfnVv9r5r9/wn4hvqnaCkDhXo9lE0EksLsghzYkVDUTyRy1ktD1YKUkMNNG86Q1LuIFkRWeTVKxkueDMOjCf0KKKKIQumX5bFIFNPqd6SrjnmpryCWJ3ZBKqtKskDO5Dlld7u5ujSk+dW/2vmv3/ABok6P6MfKrT7M3zX7/bDvHB5AppFJG4b4FnM9NLJDMV/CTVUwq3y8zBVyqopkneOghjplczmFAI2neyxyMy2Kp39HnCc1PVU7PTdgfhaPrFaAinSbM3qaO460P1UlOdKrCrmMsFdYxqIyk4Cpe41vvzfNfv+Na9H9N86tP/ANWzX9Ffh4PVMLglXKpKmmr69vQameKpqIJI5oZKER6VoaWBtSz10MwKyQuCOrNwARe4wz8i4ArEqd0nbqZ6+oSSSTLky+RqlKnR/AUv4VlaU1CiWKZkRSrOJJeqhDrP+T6n+dW/2tmn37B/k+p/nVv9rZp9+wto1hNSi4CrJNSvSSrC1JDTmOY5Uka9XWUztTxxUJ0+jCESlOuklYhZFPVkoJXvxNwO7vXulOheWbL2gb4oMy0zQuxBLAp1TiRwG0nVcrckX4/8n1P86t/tbNPv2D/J9T/Orf7WzT79gSagsck4Bl6yQzU6spjzpV1GJx+/c1lnhW2o266nZWNxYA6W0m64ZmeRdTHJTzQxz1M0uTnW09MZKZ0joYVpmRpvSWfroZp4TTxyxu073ZPjCXr/AJPqf51b/a2affsaZOjOkLhz6WZFBCuc0zMuoPMBvTbgG52Bwto1BTLiKuDrel5rfkMxF/7My39O2OX/ACf0/wA+t/tbNPv2GP0xoKKkFJRLManMJpLfHTTzySmFIQ3WVEskmrq1WxLBUManYAnDHuoIsFQT098V/hHNNMBvGfiogGAWXtEKb3sRIbsu+kh15EvhncK8MmeZIUiDyS7hWkWEK0SsZgxkZbAqmrSLtctsSgxlxHwc61yUafGTkpEdBsrSlpAxQnTaNCpGokWVSdhymODo9qTVUdXpCl5YpJpdLAFpFGuSWEjXD1wJgnB9SVXcAhiopPk0i1IyMk39VB9XwsoeRCGSWJmWWEleujZWIYG11kAI2IIBFiG3wz66HQ3ZIZeYPj9tiPDf2nEsfC9yEw5zM4Xq0eOnePtWLhYEiaRLAbh42VgDcEauTjERyyMRdu1f5RNz/KO5/nb+HnPFuNV80j20aCc2R08eoHcJINgpAIcDtRqTcLewZbhgpUjfkbddBHDfVQ64amSeF7GzaRGed7ILtFOh7Dkt2mB7CjTilOQVYVwGCtGSCdRsAQGswO2k3PP7OeLndCGY0rQfvbr1UgXEqgKSot2ZERYpnUWUtqMmkIGAAQBQ3z2U3IkGmgpWet+n27fSMccsxPPGnVgvixSo2Vs1YU+Hk+MBPJbufYu/22wk3w6OFst1xvuRqstxzsNza/jsPdipmOpunupoBbk36qa5JPMkk+/c47soqtMc1vWKqB4212b6mGF2bJdiEiQqPlSM+tj3kWtYedx5C1jhqVUWliPsYMLe0d3t+k4uQQ0AFWyZPDBK6KSpte+FCjzhgbE3B7m39nPe2E3KoNT256VLkHvCi9j5E292ErKpyyBibs1yx7yb2J+rlyAsBsMSy4cUt2FQZlSRgG09DNE/foP+7+r7Ma8wnWNNC8zYlvEH9tvI+eG2suBpcZw4P5xbjp7K0eJjSaG/dVfqzGTojjfVeysZTI7Hl6oQKQ3OygeR7sLOScAGYMZGMRVilgoLagAdzqsVFwLb9+4w7+E+GVgXVYNORzO4W49UeA+cRudxuNivUUOlQtybcyeZJ3Zj5sSTttvtjzLI4lpBER+a56LH3tygzP8Ag+WKETbNFyYg2KdvQt1bcq5tYrq572G+GxGF3uOfh3eYxJnSnmZnqYqKM2AdOsPMBm5XFxdYYiZCL9/cVwzOMOFnp9LX66nkUNDUKpEbhgLcydD7+qSbjcEjHccIxDNAJJti7cAdul+q18T/AGiKNE9O6RoWaNtUbEEcips3f+3/ADw9cp6VaiMAMoO57QLJfxuAdOrcXsB7MJvRhRiSrjDAEKGcgi4JUdnbyYhvdiU8/wAhbrTPE1pSi3Un4mYDfRIO/WraQ/NdIOMbiU0DJvBmZZ7n7LUxuPZONenvv2+h2XNwt0trIdLOI37hJp0nyD9kX8msfC9sLfDuZlLMN1OxHcRfa3dy5f44UMn4UhqIPUVBupjeMEIQe0jIRp0/KV0OkqVIG4wwOKcinoH62KItSkfHRqxZRblKgNyqhbXK9kW7QUbjHihjyS+KBpDhvV3yvl1/nNdxhcdgdG18zhR2cKo718jXp9FLzWIDr6p3Hl+x+g4ddFJdFJ5lQT9GI36NM/inhOg6gbkb2IO11I7mXnbvvfkQS+8krAzLGey2yg9xNrDu27tv+WO4+HeKy5Ubsef87O/Mt9fZcd8QcJiwsjxYPyP5dr57e4StTygMpOyhlJJ5AAgk+4C+JBikBAINwdwRyPniOquAqxVhuNiOY/xBGF/o+nsjxk30NdRfcRsAV9wbUvu9mOgeLFqjhylr9B6/qnScNfPR8YbeX2Yc0q3BGGrm9OVa53vvfz7/ANfsOICtKTkoy+EXmMUeVTiTnIFjiHyjKxutvzArSN+SrDvxTQnFofhDcE11WUeJYaiGHXog1PDLdxHqcnrSk5GggW6hlBIAe98VnzqeWnkKSUiQyD5MqTlhfkdMkxRr8xdCPbhKJHosPMjLpLSLXUttxy+z/DFtfgg1N6CQD5MifXEv/mVh7b4qDNOWNyfqAH0AAD3DFk/gYZ0LTwk7kqVHiRqP2Ej6BilkkNcxx/5V9dld4cD5m9dP6bqyuG3JnMksrR04UIhtJO24B71jX5T+bXHfaxUs4r4jPrJYKiRFa8XWNIpRrKNRPxbryZlFlIseQOx2XP43nfhWNc69JPmqtVel/tuug4RijIc4CtQHlvl/PfZPeLI++SWWQ+blU/mLZR7rY15xw1HJAsIvFEvqiMKLD5oJUlR5oVPna4x3ZPVmSNHIsWUHblvyI8iNx5HHXi7Dh47mBwbdi7Nk/U7qq7NyGP8AzUQelVY9BstWX0oRFRb6VUKLszGwG12YlifMknG/GODF4AAUFTc8uJJO5WV8F8Y3wXwqbazDYzEo7wD7Nj+r6sar48vhKSgrqVVPeR7Rf6xjL0M9xB9hxx3x6rWw0g9CnBw6hdo6weP1H9eBqtu8faDjmNS3zj9ONZbDQzuAlMnYlZu9/H3m+Mb4xvgviRMtZXwXxjfBfCpFlfBfGN8F8CFlfBfGN8F8CVZXx6MdEdHtdyEHn63uXnjcuYKm0ab/ADnO/wBA3+tcLSOXNbqHKSd27I8O/wDw/blhFqqOFa0VNhI6RCKEDlGWZjM99wWddCgi5ADi/bOOmprXYEM2x7gAB+v68JU8oGy+8/o9n24Wgmukd/Tt7qsmQ5Iw4qhB36kSu5PfeCd17rXPpCG3dfFpqBFsoPM3IHcRe5Phe5B8e/xwz/8AonqzGOs5BaaWFx3ljJE0TD2L1ysfzMZdKLOIVMZKsGAVlbSQSCrbjuMZdP5Vx44z8qL/AGXX03C0MFznzsaP6qH1TC6TcwyWqq2XMJSGhcLGI+vEh1KmoFoUKLAQqv2mDsTto0jrPM++DtlFahOXSpBKBuFlaeM+bxySGWMn5wIH5JOI141zCGOImYK17hUIBZj4L3ixtdtgPHEZ8KcbNE4EhYILaHTaSIgWuCCGZPYdQ7tWy4i4dkCRlFtDoVu8a4K3Hd5X6ieY6hSDm/wfK2nv+9GmVd+shcS38wqN13//ADBwdHua1FBMrRkCHWFnhlDPCursnrYi6tDKoOoN2SbLcsBYSFwP04VKKPjErYvy2vJbw61e2G8RKrsPLlhw8XVkNdD1wUSRMGWSN96ilYmxCTIRMsDmxXS4TkrKAQg19I5rkJsZzTbXH57rDMuOJZyaeLqKatN5KR2iJpqhBcNSykOOrlUsh61bAjS2lVaQIkdCXH81ZUvSzokdUC102j06AAyqskhYhbFmVTI4uWF03VrzZK0Zpm6yMJSyB1dwdTKuwjYCw1FdiwbtW9QcsRXx5xDI1QtZGTFLGw0yISH7LMUkv/GDWBf5uxuObH3zCsYzWuZon/N3H6+yvZlPD7daokF0Oo7agOztY3VSN7eFxyuMPaggA2UAAcgBYYg74NnTsK9BT1RVK4DsPsqVQAJLKAAEnUAloxYEdpNtSRzvTLii8+JK2uQVjwPBsLa+OWSgQsGKgsLWJG+17fRc47GOIh4p4gmkq7UsjuNK6VjvpBt2tQI07HmW2F7bWxpxsLuSpZWQ2JoJF78uqVOkni6xaCK17aZH77MLGNfPfc93LnuGZkteQpW2wvY+d/V9nfe+Ms/yholLVF/SJGJVdSna93lcqSOZ0qNtyT3Y46BbL7d8aMEba2XK5+RIX27bsOyX6eRjYm1ufnb2d2Oi+EtK2y27+WPaituLD34eYySoWztDeajTpP4yFJDcbzuCIlPIW5yN+Stxt8prDlciPK7pIqJIkjg+L0oiyTNbrHYKAzLsVjDEFtgzbg3XEeyxPKzMzF5SNTMxJZiCNrk+dgOVttsdHDWYKoKuSO1flva249ot9fljg8L4fx4IwHDU4GyT39vRaUkdRlzNyOaeWRUvVBpQWaYow1Me0We+p7kk3Zbx8/Vdr7nE5dF8d6COKRbhQ8TI63Vow7CO4OzKYimITymqD+r6gIUHxsAWO/dvp9oPPEycBzkoACugbKtrOosCpH5J3Fud+/uxq2eSxX5DtJa7nYN9R6fdNriToZbrC9HLoRgQ8TSOrAHmscgVtStt2ZLAd7keqs0mbp1QLHQyskJRjcq5cRhCRsSGIW/Iix7jaQKee2G1xTwdG3pM4vraMOE+SJIiHMg53ZwgXlca5N+3tkcWwBmNAedwbsdfRWo817hvv3WzhmUpMDa5ZSptzK7up8ygvt36m2uRZncccfS0NeY2tPQsA4jsOsUMzI/Vvf145Y5AFJtbs9mwZXdw3VK0kRUggk2I5aSDbf3nFeek+okNfUmW+rrpNIJuBGWJituQAYiht5773xz/AMMYXj5L/E5NbXYg3tR6ELdx3Ax1f86qb8r4biMkddQOOpka80YHYdTcMdH+imjub2AF9QIBJbEgcNZaKiZlBKnQWVrbBlK2uPA3PeDyxXHoG4maGq6osVjn7IHyRKPUax2GoAx7cyU+aLWQypDrSSI6GDDULkLa41G3zdNyRytfkRjoMbLjwOImPJ/M5tMfyBF/1DvtV/ZWcl8skDIhuxrrrr8j23uk/uNYRZG776T9F/qIP04QcjciojIbTc6G8GUgnQfPUAQfHbvsXbxLSa49t7b/AFc8MqWkvseXiOYtyPkQdwe7HWxmwoMtpbKHD0Kk0nDe4olFwO8G/s2I+k35eWM+HqovFpZ7vuNQ2J52OwA1AW94wlZjRsh7W9+R8f8AHELhS0vF1MsJu8dcSJSUstRJ6sa3t4k7BfecVwzvihM1iV501KGZQAEWSIg+oHQBrMpDhWdlN77kXxO/TPkDVWW1MCfwjpdNwO0jBwLnYA6dJPcCcUToK+WFm0lo29V1I7wfVZWBGpTfmLqeVjjPzcWSVvkeWnpX70pcTPix31LGHN62AfarXRxNlnUzPHfUFI0tyJUgFb/lWNj3XBttbDl6FOKfRK1HvZWsrey9x9BF/MgDvw0qucuC7ks5Iux79gB7AALfRjkViNxsRuPb44V0JlgMbzvVE+vcfNUWziHIEsY2uwPTsfkre8WcZ1TDZrQEXDxAhGX8+1x4FSzD246OD+Faqot1hanpr32usjjuCX7YG3rsbDmA3IRx0Z8a9TACSAzpZVLjQJLg6tLGx5i45m9u++HZk/Fzs95WLMTfVftDzU89vC/Lljz+WR7ZC7JaZNJ2s86+v2Xo8LBLHWOQyxfLfdT7CgAAGwAAA8ABYD3DGd8MzLeM1CapTdRaziw25dq5Avewvt58t21xL080ENwHMj+CAtv4HRqH0kY7bD41jZEYdHf/AFo2PTYLkMrhU8DyJKHqSAPupPzOoKxswAJAvZm0r7S1jaw3tbflte4iXP8Ap+p4BaRC0m/YRgW28SLqv8oj9UTcd9P08x0wxiNfktILkeaoDpBtyLM58u7ESNBfdiWPmd/27/M3Pfh7RkTy6zbGjptZ9+dfW1UmzMbGi0gB7j13oeg5X+injN/hRyH+CpVH/wARzf8A3bjCOvwma3+KiPv+zsD674iChpweQubkDa5/b2YetD0WV00euOmaRLXuCq/zS5QP/wDLLYu+CDzJ+p/usr/U5NVAD/8AI/sU+6T4T03y6df5LD9KjCzlvwn47jrKeVfNerYfRrQ/RiI+jfo8abM46OpWSDUHYq6MkhVAWYAOveoaxAKk2xaPKuhHK0WxgaU+LzPf/cKL/u4qu4ew7hzx7Pd/dXGcQnf/AEtPu0D9Ba5OG+nWgmAvIYif4xSv27fQxxIuVZrHKoaJ1kU8irAg+PI92GHP0JZaeUBT2SMf+PUMbMj6JaenbXTyVEDbX0yRlGt85DDZvC/MdxGJBFPGPK7V6O5/UV9wl8cuPmYB6tP7H+6kS+C+OakRgLMQx+cBpv49m5t9J/XuxbjcXNsivQodsaWd8F8YYMOSWs74L4wwYEWs74L4wwYEWsycepMRy29mx+nnjXjW8oHPAi1uGNc1QB7fD9uWOKas8Nh49/8AhjnQX8zhaTS7st81ST5DwxupabvP0frxoDBfAt9Q/Wcc1VVWF2P+P6MKm3XNKNVVdwNvE9wGG3xLP1qFF5DdSe9huCfAX+on3aqmsLezuF/2uca74HRhzdJSMyHRvD28wQfoq3dKHC/Wu7Lq1KTdO/Ym4UcgRuCvI92/OIaumI2O5Hf4jx/bv5nFrOkjLtLiUDZ9mt88d/8AKH1g+OIY4+yH/Sxi6ndgOW/yh+S30A78jtzUUjseUwv+S9Wlhj4hiNzIeZG/uOf0UY5XmjRPqQlTe/M2NvYQQfMHEpcLdMJQ/Gra4sWQBWNxY7iyuCNrELcHfEXZrTAAMPYf1kdx2sfPCcGxsxybbLkJoASb2PcKeM6ztahtcbaoyotvy8RY+qfH7SMNfiCNCoQWBJJ7Ntthv4XvYnyGGPwqDrYhtKBbtyPPlzFr7E3t3eeHCVIjYgEvyHe3Ox5ctLEiwsOyDiyDqCxxhOjlvVtz9U3Yq96acWYqQweNgfVZTdSPAggMAeR8RYm3nQ58KdHdYa8aFawSqAGkMeazqgChdV/jkVVFxqRAC+KiVlINFnU3a9r3Gm3yvEm52vtsbg3wn0rlDZuXj9jeV+8YikZo8zVoQPbMdEny9F9bUZXAIIZSNiDdSrWNxY2II5Hw9uOHKslhgDGNFjBtqI8r2uSeQucUS6C+n+oyxepdDV0XyYi+mSG536lyGGjmTEw03IIZO1qspm/S/T1tGpo3JEhZJgylZI7KpaJge9hIBqUspAYAnumgf4hoKlnxfhWmR4sDkUj8cZ76RUM/yB2E/MUmxPmxJbyuB3Xwn0lZYWI28cJ4ODVjZYNIoLgZnGVxc7ql+OS/LfA8lue2EelRu648+Q/xx1Q0m92N/wBvPE4JPRVS0Dqqu5AR2vHYfb9pwl59tISNtx7zbc/rxJdFksdRJl0I008707wTNo7DTRnVE5KkKzTFmIYtqKpY9oBcRRXSEsbjSbm4PMG+4P2Y5xjfMuphxHxzknlSk7g6SMLEJCFjuqyEbAXA7RNjYkEPqbbcXOJU4ZyqWGVHUpJTOuhyCQy2volC7qVNlDWsw1G4IQEV64VzmzdW9tLALfzA0rf2gBfcD43lnhfOp4ZE9V6eXVsAFs6i3VxqihS7MFQR2DFmUhgHuYHN0ndYeRjOjkc3qeV8jfbsVNML47aec/V9n7Ae7DQy7NhIDIhLRgKU0bs4+WjRtuGVhsezs3cVbHfmcswV+p0PK20YkYrGnZtqbQCzDVuQAD3X2GIXttZ7A9jqO3ukfpPo1paearhYwvpKdWovGZJiI1lVQQI5UL69ViCB6t9zWxkJNySSxuxJuxJNyxJ3Jbcknckm+LPcXU3XPDHIFaJA0silbpJJbq4wQxPYTXJJZr9rqzfsnELT8KF66aFVKopkZQO5dBeMAm/ZYtGnfs3li5iY7YY/EPNx+Z7LUiyvDZpHumZISGDKdLCxDDmGBurDzBA99sWx6MOJOtjimGxdQSO4SKSHX2agy+YIxVOWPe3l9uHX0d8XPTSbk9S/rgfIbkJVHcyncgesu25C2xfiHhxzomPhPnabb/b9/ktDC4h4dNdyV+6GYOgYciAf8P0YS83ykHcD6Of+IwhdCebNNRRO3rNFDIfIyRK7D2ai2HvKgIsdweYxq8PyDNC1559fcbH7hdCWtlaEz6eFo38v2sfLCtnb64H+cBcW57Hcjv5XG2G3xJS1MLaoyZIuQWxkK8zYggkL3XU+22ObLeOrC0kQO/NWtt32DX38tQHsxeO6qMIjJadguHrT4n6cVp+FHBl+slHtmS9WHjTUYzGb/wAINBjjkVSCFDo2kglSCDicOljjlIEPo1LNLLIp6kq8PVB9DE9ZG0nXqqMFJKI0VmF3XezG4I6AllpfSq8PV1FTaZ3WR10Le6gdWUY61AJYLp0kKmkLcsI6JrmE8t1WCXJJhCk5icQOWVJCp0MVNms1rGxBHuPhjkootTqoFyzAWBsTc2tfe3tI292LW9LlUZYVyqhjUyyrYnTppaaGHQzAuEZVexjjCoC0fWqTpLx6/cz6LD1VFBF1CRwKxmlZSZ2mdUV5VGj4y4VgFZ0VQRsQijFeXWGnSL7IiiYXDUaHVQfWZPociMO0apqcleyAL94UAgDc2vbfwONNFXMjKyu2gEXXUdFiedjsoHf4c+7E+U/AYhMwaR/Q0jMplmKs2ohhMwKBT/ArpaIqkQBSzOxkCx70ZcNwGmVqhQVl0qzG/wAX1p0J39nY7MN9RG+wI5XJD8cf7wuyB7rqsaMTyasZ1UNRvpXTZPHhbTU08kTbq6lWB5jWpU7eIIv5G/hhq0vwdaowK4mpxIyhurPWKBcA6S+g9oXt6ltue+3b0e00lNWzUkpvJCNIYXtIjBZIpBcfKQBvIkjnfE+0dReKEDvIuL/JU/oOn6MVOFZBwcmeE8iA4e96fvYVjjmMziEMM557tP0v9lXTo/6AKiWcms/e0ETWIUq0spIBvGw1RiOxA1nUb3Gm4NpS4g6BKB4tESNTuSt5RJNI4UEFwqyTGMO4GnUysFBJC3tiVb4L47ulxbYGAVSYXA3RBQ0h1IjTSbWechytvmqqJGN976Cb232GH/fGN8F8Cla1reQWqqokcozKrNG2qNiO0jFSpZG5qWVmQ2I1KzKbgkHffGN8F8CXZZXwXxjfBfAltZXwXxjfBfAi1lfBfGN8F8CLWV8F8dmT5f1h8FHM/YB5472rKQN1epNfK1+1fw1ePvw4NJ5JdgNyB7pFxrkmA5nHfmscCHQ0rB+dlUsQO6+lGt9WEOoMY3XU45AsStyALkrYNpJO247+dr4XSVGXtBqwspq7w+k45Ge/PGDN5W8t/wBJJ+vHl8FJhNrPVjYqNpuA2m9rgHTfwvy92FXhPJ+tJZvUU2I72Nr28hYgnx+nDa496TOpDmMiOCPYEKCzm+kWU7C7GyjzubdzgywT0G5KXq1oBLnGmgbkkrfU1IX29w7/APl54WeDMxjWKaRwGdCLbAkK2yqt+V2Bv7r4her6bJDJEERKoGxaOWJzKr/MiYHc2v8AIdbi9jyxKtPnSywoz+kIHFzGywgqQbEEJ2bgjmNjzGx3oszonOpu+9e/stfiXAszhzGyTt06gSASLHTcXtv8lvfiZpAEkSNkJs2lbNvtdTc2ZeYOG5URFWKnmpKn2g2P2YWUaAEEGUEbjsxncfyhjuzOtgl3e4bmXWJVYi2+q0pB9pG2LEmSOjSsCEF2z3gnomVmtEJI2RuTD6D3N7Qd8QZW16IXUglVZkbSpYM9yHVQNiAb6mNlv2dzfTOfFGe00SkpKZnAvZUsq7X7T6yLr3qtzfY254rpX0yuz2kOs32EhULf1QQpuBfv5nckk74wOITQzOFcxzXqPwdjZePFJ4gppotB6nrV90w+Iog7GyCMMTYDle/Pc3JB52Fh5YZ0sdvdh7ZyZwjCaMMQTok1dobnT2lN3Uf7Rb78xhIo8oM9WkIH8JJ2vJT2nb+Smo+62L2OKFLM4i65yaIvoRRHySjwnRfFgEWL9s/m7af5JBU+x74Vc2g1AjkCNvb/AM/pw5eJ409Kk0CyosaWA7IKoBZfILZSPFfLCLWrjSAoBc944M5HTkmVUM0d1IBF7i4uL+IPmB9A8sc9NSNJeyl3dgqKqksz8yFVQWNkubDuAGOvNJzI9l3Fyq+BPym9m3PwF+84fvQJlyyVTyc46aPRGbc5ZSdUgPjoR19jrywjAZX0FLkSNxITId+389VE9SCDa5sO7w8QfMcvLDr6LeMjST6jdonGmRRzKi5Ui+2tCSRfmCy7ariXukno1jqtUkdo6o7liTokIAADjfSbADWg9oba1fM8yqSCQxyoUkXmD4dzAjZlPcwJB38DhJIXwOv7puNnwZ8ZaeootP8APurf5XmSSoJI2Do17MNx4EeRBuCDuCO4jChl6XbfkN8VL4E4tlpZA6ljHqHWR6iEkFrG43GsDk1riy9wINneAuJIqlDJETb1SGFmVgASp7r2YHYkWONXFyWymjzXHcX4Y7DNt3aeR/unbqwasadeDXjS0rnbVP4M/ljHZcix1LubqwNwykEFSDvsefnhS4Tp1nBichSgGgqBqtyIt3hRbbw9gw1qg7HHtNKVOoEqwNwQbEewjHMAL1E7lSVnXRROIhLTBqqMjdVX49fHsAdu1uQAf8k88cfC/HUkGqKaNZoiR1kUoZXuBsbneOQDa5UkgkHuIcvRl0mS0zQK5LxuGUt3qQxtfuYG4G9iO44m3PcjoM1jHXKFmIslRGQsynw1WtIt+avf9OIjIHbFVcnAjmHmCiDIc2y7ReocXKlNKpOW5i7MUS2m2wFyDd7jYYcWXZ7RoFanadhc2bXNHGAdKt2GKlgum+llIvfTp1E4ibpS4AmoJtEhEsRPxUyghWHzWW50SAHkSQe4ncKr8MTBoFt3AqfcT9oIPvxGIhEAQsDjglvW4nfoPyj5Kf6DLlQtYuzGwLSSySsQt7AGRm0qLk2Ww3JxvSmXWXsA5AUtbtaQSQPMAkm2Mqdrqp+cit/OUH7cbAMdRHpcwEDboucJN7qNen7o4eknNREC9DMQ6ONxGX3aN/AaiWRjtYhea7x7neUtEQrC2qNJUNraklUOjb773KnwKlfk4vZ0eqJKJFcB17aEMAVKh2ABBFiLbWwyOnfojWqpUNMirU06hYlvYPCBb0e52uuxjLbKQRdQ7Ecnl4b2OJYeR2Hz/lLpJOGa4/Fj6gGv1SD8H6tqkywueSJFJGzdpZIWaVBGfAwrGospBA0g89pHyrpCQ7SoUPzl7S/R6w9gDY5Oj3IWbJaWE6opBTRhlK6WDhQWjdSLiz3Dcjt5kFkV9C8Zs6Mh7tSkA+w8j7icXoI2tZTRXX67rS1Pia2uwU10WaxS+pIrHwBGr3qdx4chho9IHDBJedLbC7pbc29Zwb89NiRbuO9ziOr47Y84lFrSyWBBA6xiBblsTa3la2J9FJHZTXtpw+i50kI3Fx4EbbjwPiMOjI+NZU2c6x4kBiP95CfaWPswizZ9I0JibSyltdyo1A3ubEWAufK+58ccQkW+6kCw2Vu+wu3aDEjmdNxz57YXSoGvLfylO6fiCllbVLAVktbrEtqt3XIZW87HUPbjjNIrN8RMsnhHLdJL/NBICuT7Rhn5lXJGjPIwSNRdnbZVFwNRPIAE7k7DmdgTiN67pAmcLFAIZakkFZKeWKopXQGzF0D9dDcG4Rh6y7O6hsQyytiFuU8WuY0Bf2Tv6UuJgQ9AEL1cyMrLsVhFv4WS4I2I1BPlWFyAQWVOEuE4lovR3OsSRmOTezaSuk6bWIYDfULENuLWGEngThz0cF2brKiQ6pZGsxZveCCByAtbwAAAEg5lnySqOsiUygi7qdBZR8k2Vtzyv3d1sUIsd08njSigPyjr/wBj+yvyZTIIjDE7c/mPQ/8AyPQde6gLK5ZTXLK+8sd6KpHKzxErHMP9nNGbr806V31C845NLsgNrBbAHmS12P0aQR7DiJOlKuSlzKGeJGWnqFPpN+0QYrAykjkkUZjcsw5Rm/IYljLoCX7K62VSbagLAfKFzbkbew447iGM+Li8La8rjV9wCHV8tl00OWybhjyPzCyfcir+aWEmI7z9ONy1reR936rYTnmYc43A8dJK/TyOMFzBPH6j+rHodLh9fqllcx8vrxsGYDwP7e/CL6Wvzh9ONizDxH0jCUna0sitXx+o4yFSviMI18F8GlHiJcEo8R9OMg2EG+PMJpS+IlxpwO8fTjA1i+P1H9WEe+C+DSl1pWNcvj9Rx2ieFdpHcNYEaELCxFxuqvv5be/DcvgBw4NHVNL3Vsl7MuIQYjHCjJf5bleXfsGLXPLcDbww3IKIBbHc3uTyN73BB5i2Nt8e6sTNk0igFTmx3ynU43+i9hjA5d/Mk3JPmTucbqaFnNlBY+QJt+oY5y2JP4bp1WFNNt1DE+JIuSf225d2GPfasYeJR5phR5PKWC6GBO+4sLDmbnbbCtkPDOo3dlMY+Y19R8LjkB323w9KqAMpU8iCDbbYixxrp4gihRsALD3Yhc8NFlaLccAr2CEKAqgBR3DEP9IuUxF5opIhPDMyM8Q2YDTfWjXBEquWcBCCFDG99KtLNVOACzEBRuSTYAe/EK1+cmSZ5O5m2B7gLaR7VAXl3jEDY5MqN4Z2+6jyM2LDmie/mHXt0HUpI4R4CpaYu8Ukt5LW60anjW28YvoYAnc3AbazE2Fk7pxzvqKRBDIyytIoDXAcqLtIVtyUHQDbuNiTqN1TMI3ZiQSAfyh7+RviLfhB0zR0scpN9MhXvLdpdVhfx6sj6MZEeLk7arAHpS2MvikGQ8uJ1OcKsm9k3si6R6z0gRmYsvVM51Kh5Mqj5F73a/PHfn2f1MoOqQMPmsCsfvVCAT5kYgjIal5J7l2UsrWKsQQLg6B+SLcvIYdNDw08psDLMb+qilj7DYMb9/uxLlQyOcBr2rlzWtwjiGFhQ26EF9nzbAge9JycQ5kxhZHkRdjYJKCW2Nl0GMMwv8nVv4jDFyzKg4uy2APK4Uf8LG30e3Eq8L9ElQeaLTqebyMGe3kqksT5MUHmMSnwh0dwU9mYdfL8+RRpH5iXZVPmSzDuYcsW8Th0tUBQ78vssfjXxRDK/UOddDf35KtkuQqtgsQ1sLjXr0Wtcsb/ACQDfYeG4uMKPRTUJH11VIbvvHEveSbNIx7lQdhdRsAA4G9gX/8ACGglEw0RkioXR1qgkKFA1qx7nZbWuQNIvuVOIiMaoOr6xXATUdDb6ie0uwIkC8+yxNuSnezYwY5C1/Q7f3WgWB+IyfH8xLRqJPInoO9eid0A7N76ixLFvnFt9X8oWOEDirMNK6Qe01x5hflN/wCUeZv3HHLQ1bxIrEE07sUVh6utQrMqNy1BZFcpfcMCO84SDLrkZ25cyPBRsqe0m1x4scXZHbbLAwMbzkv5Dcr2QdXH4Ow2/JT/AB5f8sS78HEj0SS1rmUsT37qoUH2BftxCOcVBb2sbewd9vYBiWfg6VVjNH4qrj2IbH/7oxNARG9rf5yS8SacnHldXIAj0oj9lMt8NDpM4JWtjXtFJYw/VnbSSwFlfsltGpQezbv54dd8bqZLn2Y1nRh40lcPFkOgd4jDRCqFxTlUtM3VTRtE/ibaSL+sjDssp5ah7OYsLKfBS4fK0LSzah6RLqjtserRdIYgrydtRHeQqncHDwqIEYqWVWZblSyglSdiVJBIJFr2/RhSOY2CWJBUWI+Qd9tr+BsQfC4OMzJ4dM0VA6jfPt/7yXQQ/EOPMD+LZe3Lod/25ruzWiVRcEnyIG4vbULHx2thN14682rwyLawsTdflC48e9dvs8sJl9r+7mL/AEc7efLF/hhmMI8Y+azdrC403HbkkYw8tAiva1TafkcZ0j2N+drn7cY1g5425Pu9vMfWxBxjk7LvOqc+V9pFPgwYewj9Y+vDvyLO5IWuh2717j7vHDJ4Rk7BXvRmT3A3H1G3uw4AcZuQKNq0PyqfOA8xjzGgqY1A6/qmUK+9nGoDYk2TWAdrghu7livHCRkimkppFKOhOpW2ZXWwKkHndbMD4C4uDcOXoczFvSzJCWikiTQyjdZG3Ac+CskPaHLUEF7m5lnOaSnrx1kkYizGFbagQnXRg7oGKsFbmAWVjGSdirb2PFbZZ1oFZXEcMzwmhfZLvD29NAb3+KjF+XyB9lsdgGGl0e1k4pow6l0QHrHa5dCeqIj2XQY4WeoQOSFZKdVQs5Cs8tGN7h7v9rT2XF5eG+B1OCefRzxQI7QvYISSr8tJO5DX+ST39x8uUhwZrExssiMfAOpP0A4gnTjJT3jYjcHvBHIjwIw+XFa42FexOLvhYGOFgforAqMcOfZcJYnQi9wbeTW7LDwIO+NPCmaCaFX77WYeDDZvp5jyIwqHGaRRpdU1wkbY5EKvUtwbN2WGxB2II5gjyO2C+J+NCna7K9r1uyN9rb7b7bb4RjwfTm94l92pf+Ei3uw8PVB2C7oVDtPEWYKouzEAAcyTyGH1lPR0xUmV9DfJCdq351wL+xfpw8aDhqCNg6RhXF7Hc2vztckXsbXwsYQv7KWLDA/Puogz/gmaJSwtKg56b6rd5Knu8bE4aVPTKtyqqpO50qBc+JsBc+ZxYthiCeKaHqqiRByDdnyVgGUe5WA92FFO5qHJh8PdqS6mfT3Ft7dkXI87X5X28r+FzjYG/wCW36LjGOEjNcwkSVAEZorEyFULMu9gBZgDcm5ABIAOxvhsjxGLPJMxsd2SfDZV0Tuauunv27qPundZGkhChuzHMeyT2oygE8bBbEpKgKML6dIOqwbaSeH5w0alR2SoIJFufMWIBHjuL8wQCLY5OrSSQSpudIjk1B0cRamZgoYKyOxuoY2A537OOySsijF1IEG4DkhQpQX6toyxeOy3I20WHPdQcHLaz8bDK47Auo9DYrn3XRxtk/AvgYw6qbfQgg3uPbtzSlG1jcGx8tvsxlLMTzJPtJP2403wY6NckbWYOC+MVGBh+wIP2bYKSLYoPdb6QP043pr7tXuN/sOOPBhKSgru1yflfRf9GPfSZPP+b/hjiWQjkSPfjYKpvnH7ftwUl1LqSufwv/JP6LY3R1zfMv7L/qxxDMH8fqH6sZjMm8j7v1HBScHeqVonJ5i3vv8AoxnfCP8AhNvBfoP/AKsZLmbeA+v9eG0neIEqs2NBmbuT6WA+y+OSPMTfcD6/14JsyIOwH14KS6wu1WbwUe8n9Aw/ejhZNDFm+LvZFtyPNjuTtuB7b4jBszbwA+k/pxI3Rjm+qEofXRifarG4PuJI+jxwyRwY2yrGI630nlJJhLq8yUSJGT2nuQNuQ7zfxPZHecdqrfc45RkaGQyMNTm1r8lA5AD23b2k8r4pMBmNu5LSk1ADTzTb6XqopSgDbXIFP5oVmP1qMRVQoWYKNyeXl538MTD0pzaabVoRyHUDWoYLfbUFIILfJ9/liN8k4eqZZAyR9V4syGOP22I91kW2w2G5xv4RayLsuT4xE+TKFWdhsAlOfJo4oSztqe2x5C/zVHM+0/ViOOkPIFrKVoGcxXZGVwuoqVbfs6lB1KWTc2sx8Bh4caU4ilEbSGWUAazbTGlxcIo3Oq1mJuBy2JJsgyPcjFhsQkZvvazp5zDIA0UW/Pf1UYcI9CkcVRHK03WqhJ6tqddLXUrY3mf5191O+Jgpo1UaUAVRyCiyj2AWA92E8PjaJMI3DjZyCbJxKaUU4/TZKEbYzdsc0DY2UtCanrYoKqCGpUALrAlZST2j1IkRm0rfe9gxW4IuMMmdoCmwoTO8NHzVdOljpKjeoYKkjGMGLS3YCsrHrAeZBLixsNwq73G0QZzxI7XtpU6tVwoLFzY3BttpHK3La1sWU48+DTTRmy180UzEnXWxj0aYm5YekxxgRSk3c6xIbHkxBxBfHPRnU0ThZ4hZgWidGWaKRAbdYjx3uveSwRhcXAvbHPGANeXuFkr0tnEJJIm47HBrWigBt/mz3UehWck7k95/x9uHO9MdIZXMgOnWSLHXbtAjUxsDqCufWFjZSdA50iPgdudt7e0C9vfjzqwfHwP6sL4+/JN/BWNjutUhu/kNgPbufqt9eJQ6DZLVIvtdJFH+61v90/RiPYAo3tc+fL9Z9mw9uJC6FiWqg17nRJ7LWAsBy5nDWyF8zPcKxJjMiwprO+h36Kbr430p5n3ft+3fha4b4IqagKyR2ia9pHYBLDmbXLkX2BCm58rkYZ5w89PKkc5CA2JZO2AhYgsBsSdjttfHTRvbqq15TNBKI9WnY9UnGTACSCdyBa57hflfwv3YUs4mpQpWFZXbl1krALbvYIgU3HIatu8g8sPbohyNalTJMA0cTBI4gAItWkM7soHbchl7TXJ3vews+SfQ3UQmQYRll8IOF+nL6pkZYmoWiikmltudOtUv3hFU79wZ2I79INrc2Y0EkdusjeO/LWjLf2XAviUukjjUU16enVUktdmCgLHq3ACgWMhG++wuOfLEVSZ/MQytK7q3rK7F1PnZ7gHvBFiO44ZC57/NW3rzUmZDDCfD1EkdgKH7qn+Zjc+zBlNw4PmV+skfYR78bc2X9X14zyiK6k+DX9wa5HvFxjn3mmr0Kt128Py/GzfnKfpLX/Rh1IcNCi2qpF8Qp94CH/zHDtTkDihkbkKePlS1dF9eYsyIPquXVvAAsGVj9Sj8/E9V2WqxvyPeR3/4+f24r5RL8fKyWuKfrLn/AGU8Ra35S9Vq7uyr4sHkVV1kMT8tcaMfaygn6zjP4jdteOymg5EJW4ZzOSGQBH1hg46t7FCEVSdxYoe0ovZvMG2zmqO0SbWub28L7kYRYcrvJDINrBlYb76l2IHK9+flbww42ixt8DedJJPZcz8QNstHutEKLyIPtB7Q9gJ0nxsQCfEY9zTLjG1jYgqGUjkysLhhfcew8iDjIph+8RUUMlCzIb9RE5R/lDRHfS1x36RcW8xjafNoNrFx8Lx2kDmOX7rV0ZZkir1JDLIxZxcdlthyPiFH24Tcy6TysrhYl0qzoC0hVmMbFSx0xsES4Nib7EHa9sIsHE0cbIw7bobhRyuVIILchzPiQbbHDHKSlpLummRnYWQ61EjFio7Wmy6iASG2AuDi5w7BE5L5W7dL2UnEOInFY2KF4JB3quXr05qdeF+OoZ5DENUcwF9Di1xYnUjAlWXY+B7LbdlrOktivmWyaJ4pl2kivZSeyykEFH2Jt4MNwb8wSDJfB/XVBE8kh6sMdCJdVJVip25lQwK9oktY8ltevncPEJ1NPl9ed9lc4bxc5DdJFu9OVdynBn/ECQadYazGwIFxy3PO+221r77X3xnl3EcMh0pIpY8gbqT7AwBPuxnn+TJMmlx+a3ylPiP1cjiFpBpY2O6tswPeDswPuuDinDE2QeqsZ2bLjPBoFp+vqp7viGekShkWpkdlIRiCrc1ICqOY2HLkbHD14d43RgiSXEhspNuwWJsO+4vtzFhfnh3soIsdweY7j5YhLSw7hWtUeUzylV3wY35ogWWRRyV3UewMQPqGOe+JVlHY0kbiPLpX9RlCkWYMW5d5sAQSPA2HPlcnHbQZeqxLFpUoAAeyDrNu0WvewJLdi5ABtvjs1YL4zJuE48sge8XRur2v2WuzjmUyHwWkAdwN/r+/Ne4MeXwXxprIXuDHl8e3wIRj1RjqyelDyBSbDv8A1e/EmwZFEqchb9vpPtw0upSxwl+6ilFubd+F9uG2EeokDblb9PjjOuhQSE3tpO24+v8AbuxpreImZbe4YS0BrW3qSCcGPceYeokYyje2McGBC9dr48GAY7MtoS5sLbC5JNlUDmzHuUeP2kgYjkkDBZStaXGgscvo2dtKi5+gADmxJ2Cgbknlh4cDVCipSNNwQ2t7buwQkAX9WMcwOZO57gFjJeHY1ivISI+bBuyXt6pex1BL2Kxbd2q7Gw6qXiiFCFRCsYvuFAHkQoN7d57/ACxSax0x1HktBrWwEanAFKOf58sTKtr33b8lf0nv93mMLMNQCAQQQQCN+YPI+zDD4phDzalYFWVSLeG4389v+Vt+WCpeN0bUWC325DT8pff3ee+LdN2A5pwnkDnFw8t7J85rRamjLEdXGS5U8iwA0MTe2lO01j8rSfk4Q+I+PqeGw1daT3RlWsPEnVYfbhD6ceIGjhWJDYyEiQjmEt6vlr+wHxxCqnFvHxg8anLO4hxMwvLIhv1KW+IszM07yH5ZuPIWsB7gLY5I3tjkVsbUbGq3YUFyzyXEkrpRsZT1QUamIVRzJIAHvO2ErO83SFNb+xVHNj4D9J7h7gYuz3iJ5XGo7X7Kg9lfZ4tbvO59m2KeZntg25nsr/D+EvyjZ2b3/sndxXxqWBSElV75OTt5JyKDz2Y/k98PZ3KHdma2mP1T8pSvadwwsVYG1iDfs4dBlxE/EOZuLwja1xJYbsdR+gHYnxJ8Nsc8+V877J/wu0xsWPHZpjH89U/aDp6r4oRAJ3qYioEkdYEqCGDNfRKQJGTTpskvWKpuLNpBPG3SgjqQ1O0BLam9GmPUux5u1NNeFZT/ABkXVvuRfc3jIR294x7iwHFI5jTupSnzaKe1nhmPJdQWGoU+QmuG9qsbnuwk1+RgXZgUHi0TAe26tv435YYDxg42ROyeq7Lf5rEf8JGENHmlYXM5H+fJOk5V331KPlC9vfcbbnvGHj0TMFqlCixYMCbki1rmy3sDsNx9GIpTMZd7SNv+Vf7cOLo54keGpR2OtblbHxYcgdtyBYd1yD43WMNa4FNm8V7XDVzBFK9XDvSy8caI0KuqIEBDlWJUWBPZIFwN7Dz8sNDPc0mraktpLSNZUjQFtKi9lHkLlixsLknYbBBjcEAjcEAg+IIuD7xiQOgYgVx2JvE4BAuF7SG58AdOm/iQO/GyA2MF4G9Lki+XJc2B7trpK/C3RGXj1Tu0bsOyiaTp8C5Nw3jpW353g9uEOGkoIZTreW/bchSTZRsEjXUS1jyF2Y2HgA4M/wA5ip4mlmdY4kF2Zjt5Ad5YnYKLkmwAJw0YuJpGZJZg8ELG8FP/AKxIoG9RU3IEEajtCK4ttrJdxEMnK4hpaTK4AevJdZh8Hjjp7G8tr6n0/nJQ/wAewzid5qiE04mYvGGZWIXYAOy9hZFW2pQTp8SN8NihrdY1AEJ8knbUPnAcwnzSfW5gW0ln/wBMsbSFJncSBi0Zit2IOwjqhDH13jkDMSoNrXsCFDCD41OGzOmha/p0vmfU9ly3HIoYch7Wjc0drpti6F7n3VZ80H6P+IY7+GIgV+kH9vYccmary9o+2/6MdfDGxt4qCPaNj+jGFN+RdwOa0xRFKzffUux8bKB7L3Q4dFVJaK47nQH2F1B+psIXFI0hJAL9W4v46TsfpNh78KdOS8M9t+0+m3kigEfylJxTfvpcnt2NJJzMdhWBsRNPG3mHJJHsKMynyJxPXRPX66NBftRloz7jqX/cZR7jiDxT3o2fvMplHgO2FPutc+/D56IM8WOR0a9pACLeK32587MT7AcQ5zNcRroVJE6ne6sRkhuE9o+oHC66Yb3CEoZUI5E/rFj54dbpi9wU1GfdYnGW6nj2SXVuFBLGwGI74o6TNCvHGx0NsyhrIe46iOdxtpS9xsTht9KXHfWsUjNoFJC2O8pGxc+EZ+T4jc89IY2V5bJPIFUFmPIAbD9v+eOsfNjYEXiT7vIsN7e65QtkmfpjNDlt1XdmPFM8p7LFR3abr9AB+0n3YW8i4mqY7apEkXvDqSwF+YdSDfuu2oe3Dgl4JSkiUzMGnk9SMHc25tbnoXlc7XsNziQOirKJ5qcGIIE7YmjY2iktI6aXUhhJrRQDqFgOVhbGHHxrLzXlzH6W9Nv0V1/CBEwamE2enNc0ERaOOa2qJtJEic0OodiQcwpYAahdTuDosC0pdDDHqphc6RNdQeS6o0LgeRfU5/KZj34S+EeAFjOqnkkipXZxNRzoJFDbqRG5YMlmG9zKrrfxBDw4E4ZFLGyBmfU2rtG+nYKEBsCQoUbtdjuSTiVmfPLCYsgDUCKcOo9ex/VaWLwxsMzZYj5aNj+3p78kvulxbEc5zwUyB9Hai0lhqtrRlF7A23DAFPeLjsg4kg4RuJxKY2WJQxYEG7WIB2OkWsTbxItiON5adley4GSMtwurquahfEx8Fgmlj1HXdTufDUbKb/NHZ92IiWnOrQey2rSdWwBvY3vyA78TFFNHTQIGYBFAXVubnxAFzvufIeQxbyzsAsXgjSHuJ5VShbOYAksifNdlFzvYMQPfbHJjo4keNp2MchkVmJLONJ1E3Y7C2i5uDYHy2ucc1yZ4yvqyB/VaJusVj80EC+rytv3XscVwp3N3NLlY4NWNBfHrSYVMpbtWM4xfHLrxkJcCFuJwasaNeDXgQlPKUYuNJsfHCtmuaTDslvo/bb7cN6krSvLnjKeuLbnc4RODqGyyJxhrxpabHmvCpq36sGrGjVj0PgSLdqx7fGgNhTyqh1XZjpjX1m5+xVHynPcvvNgCcRSyhgsp7GFxoLPK6Avc3Cou7ufVUefeWPIKN2PLvOHFwo6yVEcSi0IJYg21SFFJVntt61iE9UDbc3JbWa5nqsqjRGvqrf6XY/Kc957uQsNsZcMZp1NRG52AYarg+o2zG3P1TcedsVmROlOp/wBFZa9rCAPmVIXGFUZJurB7CWv+dbc+dr2Hvxxx0qgWtf27nHmk9dLfnqJ9xJIPsIION+HyuIOkclaxY2vuRwsknn0WiOnC3IuPLuxprZttsdMzYTKh8OiaXGyo8yRsTNLdklZ3RCQdonZgxJ35c7353F+eI4kIubbi5t7L7YfnF1Zoha3NuyPfzP0YjxTjZgvSuUmouXSjYReK+KFgFtmlI7KeA+c1uS+A5nu7yEzjHiwQ9hLNNbfvVL958WI5L7ztYNGNTOWYsxLMxuSeZP7fRyxRzuIiPyR8/wBFpcO4T4p8ST8vQd/8Lr4gz5nJeR7+fcB81QOXsH24QeHKwySO/ctlUeBPP2tYC/tthA4orG3v8klQvcPBvPVz91vaocA5VVzJopoS++p32CqWsd3chL23t2m8sY2hzhqPMrqmhrBQFBOXMM1WMXY91wO828B+nlhg5fQvPKwUXd2Lkdy3e5ufBb2+gd4GO3PsmnjlMcqsJj3HtXF7XBF9gbjy5bcsdvCWcCmZiRcMAD47G/7ewYkY3SonyJH4nyJoSobe4t9ZwiW/b3nDh464g9ImDAaVUaVHsZrt7SfqA88ICsLbg+0H9BxMkWOPGxnYdx+nb7dvrxiRgQtIwpZY+3mDf9R+rCe4wscJ5PJM4WNb3IF+7c295v3DCO5KR+7dlbP4NGSfhCGxfQkFlc/LKtdowo5eqCtzy0HY4sLkHBVPSzdZGX6zQQFaTs8xcnYbnYbnT32uLiJvgn9HU+X6pJmK9egUwkWsAdUbPfcSAmQAbWEhB35SdxHmS9aU1AMw5d5t6x8dKkgXO1yBzIxhfEPxHJwvGaQ3UTtXpXP+ckYXCYpZS4DfnfZNtaSWedair0tKp/e1Op1QUpOyv4TVZvYykaVJ0xj5Tc3HPGWiRKemZajM53WMG3WJTi9mlcL2WeMam08lszEWXtMrjbpSETkQFZJhdVYdqKHYgsDyknNyLjsoNgT29TO6O+kAURllWET1km3XSPaNEJuwCKtyWIUk6lGwGwXflOGulypRkZ7t+YbvpZ7Dq7sTy6L1DE+GMqWHxzHYAAYzYWe5vkwcz1d87Uw9MsMVPS09KNTygmUys3aJN+tlk+c8zszHkLg/NAxDRzQE6U7bd9vUHtbl9F8a88zipr5OtqnNrAKqoEXTckBV56QSSC+om/PCVS8d01JMQ0JqerW6RqwWNprnszudxGlgToWQuTY6QDq7NvxFLKRi4Td+rjvXrS5zM+A8PC1ZvF5NZ6Rs2BPYu5n1qgO6hvPU2H7eOMaOo0sjd3I+w2v9BA+vHRnA2HmbfTfCdUbMBzDX2+33nGtVilzLhRTuqYQylTyIIPv/AFY5eAp7WQ7EKQR5oxDfSWxlk1RqQeI2Pu7/AHjfHDkYtWEflH6JNDfbfFDTs5pUl7gpx1FJopXTuCvb6WKn33GE3h9CIBKPXVy6+aq+l1/ktpJ/JY+eFrM4T6GHY6VZIzqbvBCklRzfv9UEDvKjfGjhKANRBVFyHkK37zqYFSd9IdWaM2vYMdyRcxtd5DfVyeRuK7Kb+ijNrzIlyUkK6Lb2a97c9r7g2vuBh4dNXEvolGx3Esp6mMcmBYHW/loS+/cxTxwy/glgNM6ncJGJFuOeoBQSDuDodW8jjk+GijLU0rn+DaKQL4dYrr1h9rK0P83yxf4a1sJN+6yeLNLm232+qjXhjKXqZxGguxIv4KP2sAPMeOLOcP8ADUOXUjyBBJMsbOxJ7TFVLaAbHSu3MA+88278GLhpUpPSGWzyk6WPeASCw/JvdQfAE/Kwo8RZx1sVc9+wIGSMeC6JRf2uxLH3D5IxmubLxXJe5x8jd3fsFBA2LBYwnd7jTfTuVCL5/JPVGaZtTufYqix0og+Si8gPaSSSSbN/BoX95vvf41ja3IEA387kn6MVQqqNkVH2Cvq02O/YIBuO71hiy3waJSaNxq0akJ1m3Z0vIpffbs3B3223xqQ6QKby5LVlbe5WrpL+EdT0ld6HHTyVUqsiyskiKiFgGKLcMXdEIZgdKjlquGtLnBvECVUCzIHQEspSRNEsboxWSN1ubOjqVNiQbXBIIOKL8M9FE5q6mNJgssNzK0gYpUdZI5EgU/HmPq9ALOesaVpF1KN8W56GuIHMXV1CRQ1PWMCImukmmyiUDkvWBdQUH2hCdIewkvIHRROcGNBcAL5G+ffZSPfGLDHoOPcSIUZdJOUaZOsA7L8/DUP1ix+nCJk2aMhsLlfC/wCjliYqiAMCGAYHmCLg+0HFfeOpOqrJUidgisLAMezdVZl8wrEgeHLuxcjntukhYGZgFknisNWpGgWnm9eNNXmoB/nAD7caKngCBt1LIfJrj/eviM6XimdeUrD6D9oOOyDjeqvYSXPh1cZJ+iO+IyN9lNG/an7lOit6Nn5pKD+cN/pB/RhKn4BqRyVX/NcX+g2xpTpEql2Oi/g0dj9RU464elOcfIhP8lx//IcJRTqiPdJNRwxUrzgk9y6v+G+OKTLZRzikHtjf/wBOHWvSrJ3xIfYzD7Qcev0m6ucNvZJ+tMG6Qsi6OTMeJhzVh/JP6sazJh5Lx1H3pIPZoP8A5hjP/prD4SfzF/8Acwbpvht/5JkiUfsce9bh8w8X05O4e35g/Q2O5eLKLvDH/wCWf0kYLPZKIm/8lHIkx7rxIrcV0H8U39Ev/rxqfi+g/iHP/wAtP0y4L9EeC3/kFH/WY9EmHu3GNF+LN9CD/wA5xnR8U07MFSkJZiAo1KLkmwHI4a9+kWUohBNagmtlFJqJLHTGou7c7C+wA73Y7KvefAAkZ5nmwbsgAIotGtz2N7k7EBnf5TEb91gAA5M141iQlI4FZQdm6wgE8i1goJHMA3BtvtcjCdHx5ITZYYb+AWVm+gS7/RirGwyHW75KR2hg0g+6TcvzJ1RlWNDqFixjLPzuLEkgW5bAedyAcdWcZhPObuATtYCIAiwIABC67bnYsd/YLLcNTXyerGsY8THGv/3Az/QDhxZFw5UsVaae6qQSidkNbfSSoTbx2OLdgJrY3O2Frk4by2V4o+yQ6gKS1xZRsoN99k07eWFuvydkW5II7gAxYnu2tYe0mww7I47cseSQAkX7voxAWgndaDGFo25qN6qkktqKG3kCbe+1sJZbEq5jVaImcj1VZrewXt78RGk99/Hc+/8ARi1CLHJYfEWeG4AuslJvGbKKaQuyoqgHUzaVBuLXPmbADvJA78QRxRxkFGmEhnPN+ar7PnN7dh53w4PhMZo2uCDlHoMx8GYsyL/MCv8Az/IYh1cU8nOewmNv16qxhcNY8CV+/p0WTNckk3JuSTuSe8nvJJ7zjnrJwq3Pd+32kY3HCRxMbKCSFW9iTfvINhpDEnsju7jyxlsGo0tw7DZcnGNJddQ3uLHz71P1W+jFhfg71aJQxrIAiqCCB67N3kW2sdiWYjc27het8WcalCC2lQBqIJc2tuFvYXI+UbWx05bxVMqaVYoBcbc+X1W/xxdjBaKKic/qpM6UuMomq3AAW21vWaw5Frd5HyRe3vuYgzqqDMzfONwP2+k88cwkvdydydyedvH2nCtwjwfPWP2F0xg2aRriNB9rPbki7m4J0g3xI1pcaCgc4DdxoJtXJNhueQ9p7gO8k+GJw6E+jpUvPWRBmIHUxSKG033Mjo2wbkFVhde0SAbYdXBHAFPSWZR1k38a4GofmDlGPZdu4scSdw/wZUTHZerUFVLy9hQWAKgAjUzEMpAA31L440occR+Z/wBFj5We6YeFAD7/ANuygvpV4ZgEiOsMaBwwIRFUXUggnSALkPb+SPDEYZ1kQQakvp+Up7QHnvvbxO5GLffCR6P4IKLWkpMyPEwjZgXK2ZJWsBe3bWQ8gAh3NwMVsOMnLkLZiW8itnBY5sIbJuRtztRu8QuNWwuLkbi19/MG3txPfQfxRRx10IkdEiUXXlbUq/Fjblue/wDXiHc2oArafkndfZ832jCDWU9jbn+3fhzH2FaLQ07cleLpZ+EBAkDR0rq9Y91SxBCDvlPdqC3KK3rMBtpDHENZXxJUSRFOsdtZ7ZuTLJsLI8hJdkUcluF3NxvivsD6T4ftz9uJI4K4hO9jZ7W2tvcdpgLWXlz7u7GVxnGMsdjou4+BcyOPN8KQAh3Kx/UNwU+aqBIheRu13RqRf+UfkjHXllISBI4WJL3Graw/JU82/wBpJcj5K/Kw04GYOHvdgbi+4B8d73Pmfqwo1k4ZwzF5N91bst7iLgA+Vv045h0RAoHfqf7D9yvapMaeb8xIHpzPp2A+/qlHjLiPs6YpAm41Sb3Hgqnbckc99r+3EaVsag2VtXIlhe5JFzuRcWJse+4O5GJRzeaGSDsRWKerZLlCbarkQVBANhdjHYgc/CL81itqPWU7m5OlZola9+WlhGBbwsPIb43eANayOgK361v6ml498cgmYsDOg3Nk+3YfIX6rozmjI0/nD9OEDNj2l8j+jEyyccxzkLWwie9/3xHpirV5C+sARVFgT2ahGJ+evPDW414EPVmppnFVSLcvIilZINtlqoSS8HeA93ha20m9sbEc5Bp4XCvCbOT1QBB+S1tX2avcdj5ezDkzCQU6pMqoZ5HYKzoH0rCIT2Ve6As0ttenUNBCspBw2OGctdyIwupmfQq7DUXIAFzYC7HmSAMKfHNcrSQIjakhXqVfkJAGDtNbuWWRnlW+4Vwp3U4CAX7JByXRxlXM1LGzMzMyKWZmLM3xPNmYlieW5N8dfRlMfRrH57kfT2gfYSD/ACh4YQ+KX/e0A/2IJ90af447ujmX97kd4kYjy7KfrsfK4xDIweB81KD5/krIfBxiC1Ezi+8Md/D1yCR5dkbd1vAjDr+Etw8lVRwluUNTG7Wvco3Ykj25B1bn3EK3ccNL4NE+pqjyRAPLkT9ZOHZx/W9a4pwfVV3bw1BdIH8lnH+94YcHaY9+yhkbqsJVzPOwKBbWDOOqAGwFh2iLcho+gsMRTV5+P3xTq25gLMLXvoZb791tajz1H5uOipzvsKD6scRYj8s2J+kWGGFwUjyVFQ+5IjWO/dqmnTn/AEZb+V541sfHGDwVz/6nm/vt9lzIlOXxdo6M2H03WnPpOzCvgjny7UjfoA/YYXH6XvQqSGmg0mokWYTsyhxFA0j2spuplckMA4YBEa6nrFIT+knOaVJANQD730lmUm5LIFUEAgm53Xcm+9zhp8QcHwvTzVyzNHeVUjE0ZRmHVRxuNIJbYrI8YAJZVA5tc5GJONAG4v8AVdhLE5j7I5bkHt7KQTx2IaUTNEVqHJg66EqOskhWFxOVbeJ5I5GSVBqV+yQRqZVkTh3isVYZyEWVSusR3CMHQPFOoJuEmQhx+UHHNTirWccWGoRI0j6uGEsQCe08sgjEs0jd8jLFGoRVCxqoUFiWdpF+DlSyvVSyGT4qOnWEIX37UxdQF+YrPI1+4uRtqtjTxHeEQ55581l8Zghna8Y7TQNsHUAXt87/AEVl8lrjCplZ2AKm+55d3M89tj3AnzxInBGZGWmjkPNwx92trfUMQH0j5zb4ocltf2/tt9Pjh1dF3GaU1Mome0ZW8YvdjJc/FIO9nAJAuACpO1ycPsyOJUDNONGxh67fOlNBxAPSDwbNFNI6RMYC91ZTrtqPI76x2ja5HeNzzxMGV55I0hDxLGoB2EuuUG40agsfVDUAx7MrWsvPUdK1DUg+3zxC3JYDVhTyY/iN3UIf5KqnQrXj1EjUmrtICed7aWKjcgHu21YmPh/JIoECRoFFrEgDUx+cx5sx88KenHoGJi4lNjgazkmZ0r8PRy00khAEkSNIr237I1Mp2uQwBHtse7FfhJi2NREGBVgCpBBBFwQdiCORBG1sN3OeCaaWHqurVAPVKAKyHxUgfSDse++FY+lFPjazYVcOsx7rxK69Dyfx7/zVxvj6HIv4+U+xUH2g4k1hVPwsnZRD1mPdeJkHRDTjcyTH+UgH/wBvHNJwfl0fruv8up0/8Lrg1hH4Zw5qJNeDrcSlK+Vx8jC386X/ANeOccY0Seov8yEL/wAQTC6knggcyEwKeB29VHf81Gb7AcKVPwxUNyhYfnaU/wCMjDlqOk2P5MTn85lX7NeE6o6TJD6sSL+cWb7NGCyjRGOZXtHwBOfWMaD84sfoC2/3sOfJOCFjUkyFmZSoKqFKqdnIuW3YXS/cC3iDhs5fxZO93kcJCltehVBYm+mJSwY63sd79lQzfJsWXU51UTwtVT1tPTRyVdRTwQtBVzyydRM0fxUNPIXkACi/VxnSOZ78UZSZHab26q1FG2raN1LLZNRReuUv/tJLn+ZcA/zcYy8cUsQtEpbyjjCL/vafqBww+Duj6oq4RNTV1DLESVuKapUqy+sjo86vHIvejqrDvAwr/wCRjMPxmi/q8/3jFoaQKSeHJ0AC66/pKc+pGi/nEufq0D7cSX0ZZ401MryaQ2pkFjbXp3Jt3H1rgeF+WIq/yMZh+M0f9XqPvGFvhPo9zOnkVxUUThQ+lWgqdKtIFDMLVHMqlsDi2tk6JkrXW5TNgwyuqzX+My/+hqfvGDqs2/jMv/oan7xiNXE4+JoC8EqjmUYD26dh78Q7TPcYf5hzX+My/wDoar7xhtZpwPmLvrEmXxk8wsFTYnxt6RsfG3P24sQyhoIKyeJYL5yHM5jZQV8Jiib97SW7A6yMt3620sqnbvVHIse5thteEqiTSQfkmwPkfkn2N6p89Pni5nFHRXX1FPJA81DpkUDUIKjUrBgyut6i2pSPtHecRrL8FCsIINbSkEWINLNYg8x/nGM6eIvkLhyKv4jHRxBruYUDYSOJpBpA2ubn3Arc/WB7ziwj/BCrCuk5jAR//jPfblv1tzbzJPjfGr9xzVd+YQMbWuaeUkC4O3x3iB+xOI2YzgbKsO3GyqxLDvcbH6jjmOq5258wN+YxaqT4GFSf9fg/q8nn/tvPGCfAtqR/+oQ8rf5vJ/7uLAYVAGFQf0ScDCrLtI+mGJlDRr68hIJA1X7CbbsASeQ03DCwdBRqqqiKFUbKqgAAeA7h4395w96HoGq4lCQzUUUSgBUEFQeQALMxqLs7kF2Y7liccnEfQBmE0TR+mUqBtmK089yvem9R6rciO8XHInF+GSONvqsjLw8jIko7N9/uomy/jcS16U8OlovjC0t/W0RsxaLcDq1YKNZvqFyABZi48q4xjnkaOOoMjwlXsJGIBNwJIyTZraQDIl7XTftLjCX4GtUW1fhCAG1tqeTvBB/03eCQfInxOPYPgbVS3tX0+/O9NJv7fjr4RuU7rupX8IYfykjb+EpMznORLP1fWBiLl7sS8ptpK95OkNcltr7XvpvGFVDpZlPNWK/QbfoxYngz4MlVTF/3zSy6wBvDUIRbuBSoBtzO1jfv2FuSv+C3WO5b0ykW5vZaafSNu7VUs3nuSScYD8ed+S+R3IgVv+g6VyXXTZGL+Bhx4m0WXfrfMk9STv6cuiqxxxWaerA59o+7YfWfsw1J59RJxbzMvgeVMhBatprgWv6NLyuT3zkcyccD/AoqCf8AP4B7KeS3/wB3F+OMtACyjSqaw7sKPD9WUcEd2/lbvB8jiz5+BPUf9YQ/1eT/AN3GQ+BVU2t+EIQPKnk+v43fDnR6hRT45nxuD2GnAggjoQmlkeURvSpK0iBSgfWpCqAR2lYv8wgrqJHfe2GjPxLRgkCaQ2NriC49x6wXxPWX/BQqkhWH02naNTq3gmvq1MdQ+Osp7QGw30i/fjhzP4JlaSZGzGBiqm16Qmyi5C7vyFzzvjDh4JTnGRxIs0BQodOhXoUv/wDQsxsbBC4WANVjVZ69lBa8cQxsGiMkjDuKBFItuD8YxsduQPs5YanEPEss7OzEBWsCAkYNgbhWZUVmsSdzhOqMxL2LW8bBEH/CoJ9+MTASpZR2V06jtYajZbjnckW2xp4+DFCbaN+/Vcjxn4mzuJuAnfYHYaR9k6K+XtewD6yf0WxryfimanmWWGRopVBs6nex5qQbhkaw1IwKsNiCMaap7lj5/YAP0HHDl2XNLOsa2Bchbn1UFiWkbwjjW8jt8lFYnYYnLARus1x3U38L11JURGVxHl9ZOJI0dFf0KT1VlmMSq70hdXan1x64STMRHGUJWPukvhCaAp1iWDKTHIpV4ZUB2eOVC0cg3J7LEi4DAHbDbzDPlM147iCO0UIOxEKiwZgNhJIzPUSW5yySHvw9Mh43kgjZCFmpmOqWmmGunkt8rTcGOUc1miKSKQO1YWNIxOjfqbySgghNHOaktEoPyI9OOro+m2YflE/7qj9GHRxJw3FNSNVUeowKB18DsHnpCWsA7ADrqUkWSpCjayyBH3Zm8IgqzX8v0/qxYaWvYWpwPmCsR8HvNRE9Y1+0RGq9/bZRp+oat9rLh6cKLraSc37Z0JfvjQnt+13Lt7LYijoOgMhnAYqGmKMQN9KU8RFvPUx388TfSoqgIuwVVAHgu4X/AIT9GM/MloaB809jd7UL53UaVkU7Hrim/grEX9lkBv4W8cMKim1SMhkMKNKHllJayxxodyqdpzZnKxi5ZtKjc3w5OkybTUzoLjTI7b+MrdZf2aXBHkcN/hemWSWTUeyo1kX3sV06/NYnMbOLepqvbv6visgGDEzpQP2XH8Pa5mY9w52f1XNnVaHkAiDU0JGmJmY9YwuQs07IwBYtuVX4uJdlWTSWkdnGuVu2XwLGhZIqZnnbULIxMbg2ZgXfTE5NgxUMp5NYtfPH63qzYoZW0s1gUD3UNIu5uSGDFSOfeQwx1cdPPHEtN14lSZY2kYAK10RXMTqp7LCMxbjZ1C8+UfM6S7TW266vG4hI0ufJv6+6YOV1N5JDe+p2bw5km9vME/QMS/0FU8iSNOqM0fqSsCFChV1IQW2LansVs11Yi3aOGUnAbooIZWPeACBcAE29a5Uj6R3csSLwAJIadoZOxqlMhuy6ApjQa2YHsgCNyb/N7jz0cgNDPNsFDiZgDj4dE0Rv0Hceo6JxcRZw7zK4WKNercOxBfXKWjCMY2FkRU6wsQzMSRsCA2F3jPNVGVMI1sJ5aZV1WusYleckbEXJhVfIN5YZlHn4JeRQOoRCkdwLyu5tra4uAdLFVFrIrk31Wwi9KfHgjgSiWPU6w08nWdZsrsNRRkUAn4pgQdY3cdkgbpgSsGpx2AoD16qtxUvkaxgrmT07VdqQ+EOm2OlMqyNdFIF3uzHSu/I6mdd17IYmwBGwuuS9L0xmLavirdgKu99t9JbSUte1yx5eO1ZqyvNTTIDbWj2e1tyUbQ/lrewPiS3gMKvAmda4AhPajsvtX5B9wGnz0788Usp4cS9rQN9+vzU+LCQ0Mc4nt0V++jrixKyHrEWRQraCXTSHYKCzJuQyXJGx2IIIGHNio/wUeNnjrWpWf4iRpAQ7bCRI+sSVSflPGArX9Y271F554r42dJHjjC9mw1nfewJsOW17b33B2xpYxMw8oUOTOzGbbyn9hldJvSPTUCAzODI3qRKRrbnubmyoLHc+BsCRbEecUdLE9LCSSs0jgrEGUAq1v4Q6LB0XvWwJJUatzauvEGetLMJJXMsml5HZyD2yVALAWsEQMFGwAIsAFFqXEMvwH+GNzVn0/wArQ4ZCMtni8m3Qvr/hTJxJ09StTSSInVFStkUsJSrjbUzrdANyT1asLDuIvzZr0vyzUNFURs1PNJO9PNHrMiaVDFJwzKA4Cxm9156wSSgLVtzvPCXDRkhlL3YgWYMsQ0kb7Ex3333HLHsOcSTTRdfMI0BKjTEixJ1q6CdCBVJa4Qu3at37DDYpHSxBhvUf56JJnxxTEj8oPupzz7iuaQ2eQygdx7KH2Bbi/mQb+IwvdGWTpVVEcbtpjYMTpIu2kE6ASuzN37XAB5HEbcO5LI5C9cBYlWcdqPUpsSCzKtjzvfY3XmCMLWb5DVUtpFdGjuCKiJiUU3A7WmzKbkcgd+RJvijG7iOMfO0kX/N1pS4/DsptscAavt9lZrjfguD0GRY4QGijZotCgy6lGoDVYu+sizAklr+NjiAWoJALmOTSNyerewA5knTYAYkjou6WR6Kq1DSyzoNCmOCZzIqCwkMhUqZGA1sZGG53AxJ3R3mrz0UMkm8jKRJsANSsyNsCQN1O2OhjkNbhc7Nihztj9FWDVjdSKSQBuSbADmSeQHmcOXpb4aWlqtMe0Torot7leasu+9rrcE+PlhX6GuFWll64i0UZ2JGzP4D8wdq/c2nnvhZpA1qoshOvSpC4G4IRVRpFDaPUU7rrNtcxHJmYgKl/VjROTE4ivIlkp61K4xTzUyz5/Ss0EDVD0s0ucSyR1BhjDSsjrG0RaNGtYBrBheyMa4ZHQl/m0/8A2lmv/ilXivG2gtUADYKGssyWsq44VniqkgmzpmleOlbLp56T0CRXnqkpSssaTzKIzJKVZk6vcXUDszfhupWrmRIaz04ZjSNl1ShqPQocsjFOHieUSejrGsSVSSwSduZ3VtMpKsLJ4MSpVXfI2rBVU8DQ114s8rJpZTFN6N6JKlW1Meu/g5ISHjAAJWNlCuEJQM3fg2wzSjKJkSt1CnrDmc8rTNDPCwdaSNZJGaOV1m0MiRHVEquG03s1qscHD+TRU8KQwxrFDGLJGgsqgkmwHcLkn34OlIXD0fKgo4OrSeKPq10R1IkFSg7ll61mk6wd+sk4XcMjiHjOaOaoSOmWZKWGOaVmqRExWQSnTGrRFCyiBt3kjW5FyBchR/6eUosGl0MYUnZWV7xxOpZGlspEWoBrayuoq4FypsITmwYRM64spoSwlmSMqY1IJ7WqXV1KAAEs8uhgircsRYAkgHkl4+oxEknXApIZFQBJGlLRErMOqVDMpgYES6kHVH19OBCc2DCDn/FUUVJ6UD10TCMxdUVbrzOyJTpE2oIxnkljRGLBO2CWAuccOXcSzqzCsp46WNYnm6+OpM1MiRaesWeSSnp+olCt1gGl4yqyESdgjAhOzBhnZJ0hQPFJJIyx6JTGEHWPMyszCmbqupWbXUxr1qoiPtezPpYjfVdIlEsaSGdSkiyOuhXdtETaJnZY0Z40hk+LkZwojfssVO2BCdWDCNScU07z9QkqvMFVyqXbSjLqRmZQUQSLdk1MNYVtN9DWSs+6QaeKZIA4edp4qcoCQFeWzadekxmVI2ExhDdZ1fasF3wITuwYQsm4wpppTFFMryDXsNVm6pwkpjYgJMIZCI5DGWEbEK1iQMJ2b8UTmokgpKdKhoQhneapNPEpkBZYUKU9Q8k+gLIVZI0CyR/GXYgCE7sGGHlvSSknV6U6vVpVxM4jMcvp60EtM2lZAZo5yyLpJSR1UBgriQON+KqYRpKZo1ikR5VdmCoY401ySEmwVI0GpmawXvtgQlnBht0nGcEgBikVj1scbK+uJ1625Q6JIw51gEpdQsgBs22OWPpKoSius4dHv1eiOVzKFVWd4lSNmlijEia5IwyIWAYg7YEJ3YMNzifjCKGiNYCJYSsTRsrqI3EzokTmQ9hICZFdpT2Uj1OdlxzZbxVIiSPWxRUscao4mjqTUUrq7FQqyPBTydcGCjR1Nm6yPQzksqiE7MGEPKuLqaXRolUl5WgVSGSTrliMzQtG6rJHKIVM2h1U6LNaxBPLmPH1HGpd50VB1l3sxQdTI8UupgpVerkikRrkWKnAhObGjMf4N/zW+w4RRxtS9eIOuHWlhHbS+gSMgkWEyaeqWcxkOIiwcqQdO4wtZj/Bv+a32HAhfIugjGkFt9hZR6x/QPf7gcPSvprZe3ZEd9DWuLk603/QveRv34y4B4PVkjkY9ZcAhQOyPzieduRGw5jcYenGWXKKKYMRcoWH50d5FA/mYryTtYQOpVRzwHKMT4d+/wBuOuB+qp2kt26lnhjPzYUA9If2ysUplIBuiVSm1xjTR07O4VBqdyEQXtd3IVFueV2YD344+Ma9WkKxm8MIWGEjk0cRIMu1t6iRpKojezTEXNr4fzWgSkGlbdh5/t9mF6OqvC3iEYH+abH3/bhuxNZz5/8APG6pew9u3u/Vh5aCmBPLgriWWlOuN9MkbFb7FWBtqRlYaXidSQyMCrA7jD4q8mhqImrKRRGgt6VTAk+iuTZZYr9o0MxvpJJML/FkkFDiE/SuxpvuWufYFUD6/sw5+j7i2SllEsZXUAyFXGqJ0caWilQkCSGQHSyHmN7ggEVpIq8zVIHdFNXwXZjqqNWysomNxy1Od/EfFhQfzB4YdvRnxuKp+s3GtY0IO2/VKwNu7U+sD2nDJz60VJUVFB2aeWL0eohJLS0bSHQva2ZoCGYQz+J0SdtRqafRNmBjlRb2DoF9joLofcQyj87FF8YlDndVK11UE4PhARFK5zc/GpE4HgFXq9v5UR+k4YFFWNFIkqespB8jzup/JdSUPkTiU+n2n6+SjKjeWORfIFGUkfydbfRhg8TcMvEpZD1kQ53trUefcw8xuPCwvjfxGmbGaXCxVfTZc5M0R5Dq739V05lEl0eM3iftx35oQQXgb8qI6R5qVO/PCdmMxkmj1bnSqnzGpIlBtzOglb87KO/HBk2ZlLpsY2IJBtYMNg6k+q4BIuOYJBvtZQ4Mputrl07qJFc7bdXEO0SO7U7BR5kYz2wFkldOilebaSOykziklYowG0lWRmVeySHdY9OxBCXkYXHKyja4xs4i4TlqZb6iKeNAAitu7liXXmLdnRdm9298M3jWvD10dt9MiRry3IcAi5/2jn6PLDlzDjdqauIa7UpWMSgC5jY3PWrYXNlKal31La24AwzKL3Rks53sm4LY2yM19ja7M2oGjWKNlEa7m4uVvsoUkX7SKpawvsx9mI06RcrAeSZp1Z3ddMWghyltKsO0exGqKlyBfy2BsOZY5orqVkjcGxUhlO1tiDa4v4j3Yq1xplrw1Dq5LXYsGPyhqI3uBuCCpFhaw2AIGM7Cmc8aDtXRb2XG38wC0ZRmJjYkciLEePgfaDuP8cbqeqMT619UsVI7tPZdQPMq23gQcJMBBNibefcD3X8r8/Dnhw0uWaowrAhiD3i2pbiM3W/yH8dwoxddpbz6qCEE7J8cHV4Ssp5LjS0saE7abOdIJv3FZGF/zfDFkLYqPwFmIBQN/o5EYj8kOCfoKsPZbFtzjU4PsHtPQrnviMeaN/cFQt0t8SAVZCKGZFCEuNSLbeyqdtVyblgeew5HDXzYLWICOrgqkATd9Ecse5I5WVlbkN/Wty9Xh40V5KuqmjjdoWnZQ4BKsY7I5U27XLVtewI8sNSsBYk27O1vD2/T7xiKcRlhdp3JP/qs40krS1hd5QB7f+p2ZTwQqMfS5OqjuAohAllkZt+ztZVBNrkMSfk27WNfFeR0sSiJJJJ6jWGcvC8IjQdYAmhrg62CPrDNq0KQUBZWRDNICsbt1lyFjUFnNmsFVAFu2ogKFUEkkAX2GHlFmUdRTFZpFE0ZAilPM6iAobSDqUHZ7clAYi66jPiYkVB3XnulmynWQ0bcrT94SrFenjK2sFVSB8llUArbu8R5EHvwpsL7cwWBsfV1fJY87Wue1bYXO2+Ib4CzECWzPJEkikHTYhmHqjSVe7bmzKuoG1ja95PnrA5UKzKda7C12UOC229hpXckDZresSBuxStkZSzacw2pp6BOHbSVvWhZtEqIrML3cBmkaxuASGjB52It3YmOKFVFlAVRyAAAHedhtuTfFeMoZgqSIxRwosVNt+/lvvb3+eJu4SqpHp4zJfrCDquLHZiASLDcgA8hzxzcsIxWadV7n35raxeJ/jJCNNV9Ntkg9InBi1c0B9XTqEjj1imxWMed9RBN9N28bF45XQLGioihUUWUDkBjjz3OIqdA0jWBNhYEsT5Ab8vdjiyjjimlcIsnbbkGR138LlQt/fvhkcEjxrcFM7IhY/QXAE+u6cowxehL/Np/+0s1/wDFKvD5vhjdCX+bT/8AaWa/+KVeJVOn1gwYMCEYMGDAhRLx/wBHTz1s0xoMsrklghiRq2RllhMfXaygGXVF1brVPZljN17tiOtOjqf0SphadZZpqKkp1qJNZeSanidWmmW5bTJIwc2kdrFrm+5UH4vkTNKqApNNClLRSRpFEr6HlkrllZmFm7YhiABJA0GwF2u1aHpiaOCAOIpak0/pM3X1CUjGNppo40hAgdJaljBIojPVINI1SLrFxCXIOEKySq9JnFNGTPSSdXFNLKFSnjqkYdY9NF1jsZ0cHq4wLsvyA8nicHVcNR6TAKaWXXXr1U0ssSGKrngmRxMlNMySRtTgNGImV9frr1YJyk6U2MjlKbVSRVNFTyzNPolvXpRtA8cHUtrCNXRiUPJEVUXXrDdF5q7pQKqKiWNoaNJ65AY5UkklWghr/STLE9N8Wgai1RCGfW7FdRVQyOVuhLdRwK/4LjpBKvXxdRJHLoPVekU88dTGxjDahAZ4lBQPqEZKhr7448/yGvrYp4ZzBRwPTSxBIZpKgyTyaNEzuaelkjig0MAkbB5etJLRlFv5V9Ik0JK1NIIn/eZXq6kSoRWVsdIoLGCMiWFpNcihStrBXe5I84s6VFgkliEcfWx1a0qmeoEFOSaGKteV5RFK0aLHL1QVY5WaTTsFZmQpFpBpujOo6uZ2ipzO5pgEfNc2qXKQGZrrmEzCopHDVDNH1FOer+MDGXr/AIv3Muj7MWgiiMyyjq5kbVmFbC0Jkmd4SZqeJajM0hhZICs70/WmLW1jK2hdz7jtpMirK2D4maKmrSh7MirNTCZNaEqUmhMkXWRuVtJGUYqNRUOzivMjGIN2HWVEMZ0lN9ZOza437Btvp0t4MuEQkfou4Xlplfrer1PHSL8WzMAYKSKGQXaOMlesRipsLrYkKSVCTNwhViURp6MaP08V3WPJKKrtTddJTiMQmO4kZys3Wi6BYjGN5cJuX9K9TJDHKlAmmah/CEIausTAixmZZLUjCOe80fVIvWJICxd4Cuk9nEPS2IqmOPq4erd6RbNWoK5hVvFGk0dGkchaCN51V2llgPYlKq4VC6pOi5ejDowkpJoA5Dw0kbRwyNmGYzySXTq0b0SaQUVGRFcMIxMrE9gQgABxVWUVUFXPNSxwTx1eh5UmqJIGjnjiWESK6U1QHikhihVl0oYzGWHWdYVTZxrXzNWUtHFK1MJoqmolmRY2m0UzUyCGLrUkjVpHq0ZnZGKpGwUBnDpxZ3nc+X05MssNVqqNEU1XUQ0SpG0esLUypCVeQOrxp6PTFnDQhkFpJcCVIa9Fs4WP4yJn62GonbtorTHO4czqerXS+mPSskcQZifUDHdnxnU9FMjtXo0ka08sU8dBp6wyQemyek1xlsyXV6tYyqxOpEaaQym1lDhXpMkqfR+qplJkSoeYmpASNaWqNLMIm6n49mlBaLWIFdAS7Qmynf0Z9JXpkzRFacHquuVqatWsULrCPFMyQxxx1CFlusbzJ61nNgSJEkcM9HU6ydbIsUchmoyR+EMwr3MVM0zn4+u3F2qG0QpEgXtlpH6yyYJ0fViUWW06vGwpaQU9QiVlVRhpAlOFqI6iki9JkSPqpB6OTCsvWAswKLhTzvjWpaVOpiVaQV8dI8/XKZWZZhHOPR2hKrAXD04lE3W6xcRhD1g86PulpKyaNFWLqp0d6dkqetn0qNQ9Jg6lPRjJHd0AkmGxVzG2lSI6pQ4f4XqYMnp6SN4TVU8EERLgtTS9ToEkTXjLrHURq0RcIzR69QV9OlmvS9GMtpWigo8uOqjljpoJHkpJJ6WpSo62UCmgSJn0ejh4omfSRI2sokauR3nq6yqjSqmpIaNoobQJTmSWaSnjqGldqmCcGNEniRERVGpZS5kBVUa2W9IVQdHWubrMtLL1KxoJJUz6PLWnHWxylI5k7Txg3VXdUZXCSqdUJVqeDq1pVrLUvpi1gqBTieUU3VigmoRGan0QymXTOZ+s9FA2WLTZetKZBwRmKLDEUoqmnjnq6uSNquop1kqJswqKqAuBl9Trhp0ljkEZI+PAJLCFGZfqeleNYBK0TXWKslqEDC8LUMogmi1MFVnapYQxljGrDU9wqnHJkPSaZ3EZVElWalDNSVKVNMyVDSqE66SljOtDA3WxiJGCtEVkOvZEq4814ErpaxJWMJjSshqAfTqwL1SuhaAUSQJSdbGNWmokMjylBcQmTVFLFat0YDmVIH0Yi4dK8vodLUmClhWsUSQek5ksCaGjjdEY+ivIaqQyNohhimXQhZ5IyQhfPC3EC1VDDVIGRZ6dJlU21KJIw4Ukbalva48MKhVQ6N+jSb8D0Txr1kskWsIpAB61jIryyOQsSJGyrZVYm2wLE3My6CC1vT6/tt2lpKSF5jtsDGt1kcAmxdobC5uwG+H78Fbi1KrKoYCSJaeCON9OpDoF1QqytquEVbspU3Yd98S1QUKRAhFVATdrDdj85z6zt+UxJPjjzPM41kYmRIx2x1GqAsi9tzdD2HzVmPDik84/ndfOzLZeqhlnvZlXqouYPWzq6htuRjgWolU3GmRIj5FgU8l9vP8AVh38ez6IIIb76DUSb/LqAvVLblYU0UMo7wZ38cMmiO/7eIx6RGOqgcVuqdtJ8DbHtfJgzAbe8fZb9WOENiRIsoTvjc8pHs/VjQF78ezN9GBClzIOLJILSxkA6QCGAeKSN7CSGWNuzJDIp0sh5jcWYKwdlNkcc6CpoQfiz1k1GWLT05UhmaIkaqqkB5SKOsjUgSLsXMS0Uuqm81Wx91j9gt7QcKXAebvEdaMyMjKyspKup37SkG4PmMU5YNiWqYOU7cXITTxEb9VVIb+Ec0ckZ5dxfR5XI9uEHMCGjbv2P1fpBFvI4kHKqyKvpCsjLBPLGC04GmF3icMZJFQfFSK0ZbXGAp31KvrYjLOY6infRURlezuSRpIN7SK4vG4YbAq2lrbHni9wLNY1hheaNrF4pGfE1eijHMotJO21z7iO7HZwbNJG/WIWFk1uFa2pdQKqfEE2Ok8wMdLMra7+qyMRuBZghKm52BDC3vtuDjGhiZaKWQAgSSQxg/kIJCTfw1KEPn7sWpWC/RV45CY663S9p1MtREqk36xLn5QLOLsfMAasKWb14epmJKi8zKLsFGlSI0JJIUAqoJJsBvhG6P57VaXNh1ikk9wAuT9AP14SaklWYNswJDXPygbN9d8Ujj6m6QVNpp2/ZPbIYZaaUyRVNNGObIalGR/yHRGa/gCASN9JHLDo6T46arpmmWROviBfSHW57O4sbEhgq3YXFo7/ACcRPECD2ri4uL33+nu77476ZUJs4JUgg2vqF/lAd/gQeYJ5GxEI4ZqcH6tx6fYqyM7wxprZIkGTzNA9QIn9HQqry2tGGZtKqCbazqIU6NWkkarXF88kzdkYAm8fIg/J35jwte58d/K005Lx5Tpl8uX1zEgxaIHUFg8TgiNwQDpMTKGuQbFb25qsKcXZWkMxSOUTxWBSUAgOLDVa4F9D6kuNja45jFKKSSSSSGaPSQaHUEbbg9/RXxpAa9h5hK0CaZ2ZbFDYn8173YeIDKD7D78WRpukGnioIqiaRQTHbqw6GeR42Mb6ELgtd0Y3NgBuSLXxUpa1wAASLXAI2Nj58/L2AY0tc99+f18/ebC/jYYv47nwuJHav8qpnYjMpoDuhv8AwpB6QekeSqqDIo6uILojRrMQvMliBuzm7EC4Gwubajhk/SZOkokZYpnXa7xJrt5SFTICBsCDyuOROGTEn04dnB+QGXU7FYoIwDNMyAqgN9KgCzSTSEERwqwZyCSVRZJER76tx5qaLGYxoaByUn0fTDG6XkpmFrXKhGUXNgSWsFueV+ZwlV9YaxlSCCWRNWooobUzC1wvUayraRbWBcbWtbDel49eIWoy9JApHYVh1ktww66pcD46Qm/YN4YtSrGq2ZnXs345qnhiLzzFDp1L1jLGQy73RSENyfDFZz5DyVpkLT0UhcK5WY1RZcs9HUWDavR4hpB3JepaN9xzLajfc3xKFDQ5PKyhPRoZL9k09VCTuN0AhlbVF8rQU6sEagARqxWKvzxqiOGPRGiQrpBjTSzbntSG5DP2m7QC3ub32t05ZRskgUudQAeN9wLA+AJ9Ru7wZe82xFrmB2NIMDN7CtPmVL6LpjX1gg+NPrk2sxUXIi7QJsCWF9mA2wrcLceLFThHRzKmyhRs/fcsTYG53J9ove2ObNFaWgikbeRANRBuCCdBPnqAR7/rw0NWOgxYGSR27c9SuKz8iXDyD4ZoEbdt115/XNUSGSTYnZVB2RRyUePiT3n3YSp6S2632+n2gjvGOrVjzXjTB0CuixHapXEnc87UodGnGPXJ1crASraxJA6weO/yh329vfYHQgf3tUf9pZr/AOKVWIrlgU8xh8fBbS1BNb/rDMP++SDGblwtYdTevRdVwbPfMPDf0HPupXwYMGKS3kYMGDAhJVFksa1M1QCeumigik7V10QNO0Vha4N6iS5vvt4YbzdGkAVAklRAViMDPDMY5JYS7P1UjBfkPI5SRAksWt9Dpra/HlOZRR5xX65I4yabL/XdVJGqu+cRcYaeW8Y1bVzXmAIr5qYUbTUoQwx6xHpiEHpomkgRa8O0pUq526srYQpEqOA4GEw7dp6ilqX7f+kpBSiGxIJ0/vKHUDct2txq2xfo9pjGkTKzxJLVzaHIKu1b6T6Qji3ajYVkwC7WBXc23iSs6QahKEzxV/pFRJlNZVyx9XTkUdRDGjKUVIQ0SxTu9IYKoysSgBOuKYuvcUcRVFNNVUzVk7bZY0UvV0QqA9XNWJNCryJDSRRutENLzK7IZHCB3aFAWik4V6MAZpBJLNNA9NBErSTs1RHJT1LTQsjaQAYWEbrIdTs63kLkXKinRzCNR62oFQ84qvSOuHXiYUy0zOOx1ZRoFCGIxmLlZBpTSyOA+JqmqkhiFY+gVNajyRmklkkSnWmeONpVpupJRpmRmijW6ixLHtlvJxbMVp6sVfpFcctzCoajaOFhSzqkJeJY4I0qAtPLqpmhmZ5HMQAKuJC6IU21XB8TUMtGzStDNHNHIzys8zCfX1rdY+o6iZGI20rsFVVUKNMXCe69bU1E+mSORBKYAA8ZJFuqp4yb33BJ2G1t8InRznRarlhStOYwCmp5+tPo7GOSVpRp6ykiihZJ40WZU0lk3N9Ekaqx6zPGlq6F3qusqVzSvUZf8QBGKeizOOIBBEKkN1XUuzySMr+kBwArwhVQpQyzgGCOKKJTJohomoEuwv1DCIEnsi8loU7VgOe2+E6foxp2LWknVDLTzvEsoETT0y04hmcaNRYLSwXQt1Z0BtGrtYa3QnxbUzzQ9ZULOs1G08qGSmZ45Q8IUxR08KPBCDJNC0dS7uDHGAS6Tszj4AzKNKzNlaREY5hHZWdVY3yrLQLAkE3Ow88HVCcvFvDaz6G6ySnmhLNFPCyCWPUulxaWOSF43X1kljdCVVrakRlQYejmAWYT1HpXXNUNVdbH6TJKYBTsXAi6jR6OEiEawqiAKVVT2sRZV8X1UsM/WTqwmgzNZ6VpaUiFYaefsxRRQCqSSnlSKFzPKynrXLdowgdRzaSKWs6qWKnb9+uJZFTTG6ZZk2iVnaGVo0TUSx0snJnSQIFwDqhSpkPRxTQoUHWuhjqYmEkrOWSrqGqJwzn4xmaR2s5YtY7knfHvCnCEUM4lWeaeVYDTjrZldRFrRlUIqKoKFLawNTA9svZCvB0L548qVCSSSyyQzBSZHpZdIaGOQIk1GqRzJ29YLxxyqHAZbaWaJ+Gsk00+Uu8NJBA9RAxrYoL1iSLJrhikfsdUtay+iPNeQHreqKHrw6CFNFVwBC0wl1zqBOtV1KykU/pC2+N0Wv27XaO/VliZNIkPWY3cM8ERU7qY5J+rjV1hgaZjTwq5BZY0AF1WwVBIZBCnYj6tTpxHFJxlPqicVoeeaevimpGSEx0iU8NUyv1cUYqwaWSCnSR5JGEvXkWBlgCJOV8c1Jhkj9MJmU0DmR6igaCSOpM66aWshpOqRqloDpSrpI2B7KFeujaJEKXM+4QEk5miqKijndFSVqdofjY0LaA6VEE8YZC7ASoiS2IXXYKBzf5NaULGih0WLqSoDkktDWx1od2cM8kklTEHkdmLPqck6m1Yb3RHxM1RLESzv+9qhWaaOmE5eCveBtUlLeJwDGV1QsIpLawqk6VlPChCabdH1MZKx2Vm9PRUqFMj6NKoyWiCkGAtqLs0ZVi9nvqAOMqPgdAQzzVE7iSGQNLNqt1BcxIFCrGqgyMWKoHkuNbNpWzqwYEJlxdHEKx0qRyTwmkpzSxPHKFkNOwh1xOxQ+uaeJtaBJFKXVlubreRZJHTUiU8d+pghEUeo6mCImlQTteygC/M2ws40Zj/AAb/AJrfYcCF82uijpB/BksEyP1imJOuiXfWjAa473sGVgSCSCrLYixN7s8N8XJURxzRaZaaVC6yK1jfUAE0EE3BEga7AoyaSCSdPzYyeC9j3AA+/uGJU4I6Zqmgpmp4ljdS5dTIGbqtQ7aqqlR227faJAYubHVtl5fCsbLszMBPK+v1T3XFtGTfP0UZcf5mJaiVxsrOdI+agOmNNtrJGqoLdy4bsL2ONlQ18aMaTW0KTSbSrVL2ThKwp073X3W+rCdENx7cOQUoLB2bftfHDIuHLltJrDWVmI+arEDbyFgfbjPhPNFp5jKfWUWTsqxu3MgMDZhawO1rnffAgrtyfJZRAHKFUdWQk7b7lWtz3VgB78cPAdNI5eykqVuT5r3C/M2JO2FWbjB6hWhUaEsGXe7F0Nxc22BHcPO5OELh3iJ4mCrYLqN9twG2a3hsTzG2EpKp36HajWrQkkW1i49YLLE4JA5khtR94xycK8a1KosbhK+jPaAcrG2/N4WADU8h5kWdTydX3wndDldor4QeUjdWfa1tP+8AvvxGVRxI8E0mnf4yQlTfSTrIJPeD7N8UDjtc9wI7FJKzVVqWuPsiPUPPEkb0rC/WdUBU07nlBUhJiqSE7LMivFJa4I3RWVw20qKSiNUK12lgRXbSoVl61tCSKEs9ix9hHqnCpwD0rKj3ZSt1KSIwEkEsbbPDKhA1xuNipHgQQwDCbuDK2AN6VRxrLFLeIhI0E8UhKuaaR4ohIzXCtGXv1qWNtQbS9kjofL07qt4DK/L9FV+mgLzMEGgesVdto0HrNI2lSI1va9tRuFAZiAXak89RVFIiVDMzKUbq4tG7STykEaPlyuWvpuVF7KDbPMaiF1dauFDGqkukj00mplFwmhnaQkN2QOrv3i2rDcy98qiv1EEMRewYrE2shWV1UnqxpUOquAthqVSblFsoynE01SlkWjfmoZ4d4Mjd3RYqiQ8mmeMtFI3e4QRrPTqGBKnVPIwN2WM3UY5t0UTo3ZiqAp5MsRmiPhuvbS/PRIFcd4GLMUOYRFBZiOzeNQjXZR3qum+gdxAsbNb1GsZbxJTg6nkKr5I5LeRCqTp8dt8ReM4Gw6ioDitd1VReN+A6mOC7o1lPYJikSxYgEXKEKrbEgta+/PEtUvRbHFTxwTRtU0pSMzfGqk0MyKxergLaFji7QgaHrNUo7Vrqzu4+nnpJozAKaF5BJM66nSFyEVHRyuiZ4bmQ2W6k6VvtewLMiWRKHrWeSeAWncKTpZTUyRm0UoieNCqausYMjkuFIVCTmcWyMpwaWOG3PoT2oit1cxY2xjTuf2/wmvnfwc63+EpBHUwuNSASLHIFO6gdey6xo0nUzKx3BUEHDa6TuCoaIRU6v19XoL1bD+DjY6dEEWw2Uai7HtE29UdnF0OFuI1kj1KCqFiqbqVdQLq8ZRmDI62IGzAgggEYiXirhRcwqJZ441DSlEF4YWmRgBGZGLyqsih0a6xuDpGpXe2nGfw3i+RPJol208/U9L+Xb5q/NjNaLBVZuGMkMrMSwigiUPNM4JWFCwUXUWMkkjdiOJTqkba6qHdFnPc+VwIo1MVPET1cRIL3YANPMVsslTMANTjsqoWNLIgGOvjymmjVY1ppIaSG7WNpEeSzJLUzTRL1TTkho7XtThWiAWzhmrlmVSyt8QkkzD5MSPI4B+QUjDOVPMbbezHUgh25VRcNC3aseR7J9jWN/wCSbN7Vw+Kn/MQd7rGp5fNt+rCZH0b1oF3p2iBG/pEkNMfaRVSxEe+2HtBw0WpdLz0iXja59Mhm3IYnajaoawN+SnltfBJI0UnMNJqdH1TfUD+2+HzXxmyuo7SG6j5wPZaP2ODbybSfk4a/AVLRxy2NW8xIN1hpGVL7G3W1UsLj3U7X+rEjtmsAQ6afVY3vNO78iN/iBTW8bG/tOIJHkHkkc5PLou4z0p6O5Po8osjHfRr5G3MDkRbkdiLEFHLPAysVIsVJB9o/RiH3zncgLGik7BY1sL2JsW1Nza/rE7nEocKZo0sQZ2Lub6iTuSGI387WxpcNmkst+a5f4iibobJ1ul2EYTZ6i0yjxH68OQ0R0atJ0fO0m3hse/3YZ2Z29MiW9gdI3NgLsRuTyHn4Y1ZJNt+6wsKIl52/pKXb4fPwXv8AMJv+0cx/75LjEcCIsZeWcRnutYqD4XJ7R8hbGXwY5U9DqVjcSKuZZjYjfY1kpQk2tdks9h3MD34hy5WvGy2OCYssLyZOoHXfqpVwYMGKC6ZGDBgwITSz3OKT06GkliVqieN3jZoUZCI9RMWtt+sKrJIqW3WOQ/JODJeL6SV6pwVj9ElNNNNIEjTUttSrIx3jVyYzcga1Yd2EHpP4fqXnNRTRiSaCGGSnBdFEk0VSzSU92YdWaimeWn6wjSolJubWw2argerhTSsTT6aimcyoKJqtimXNFLV0y1jLSJUNVsdRnA+LeYopJTCIUwUvUMCV6plmUuSugiVdgXNtpF7QGrcdoeOOXNsypwO11UjSKCI9UReVF7QsHZQ6KH187AG/fvFHA3CtbSmnkamedlGbwuvXUiuvp+Yw1VPUSFWig6rRE3WiBNSsw0QkbDLhTo+qEoGR4AKn8HZVTjtwluspois8YYOQBGzML3Ctc6S2BClWrzKnjMgXq2mhjaVoUaET6Qg+SzoF1KiIGcouygsANuhKmnE+gGEVLDWUugnK2tr031kWFtVrbYh7iTg6pakrKUUCSzOM4kjrTLAFc1qVfo6RapPSBUsKiGmkEyRwoiPaQhI1KvVcJTmYxejDUczjrxmGuGwiSZJDERrFWKj0dWywKIzEYCLyWJjwqE8864kjpp46aOnlllljlnCU6QgaI3iSR2MksS31zxjYkm/kcZ5RxNSSRrUao4tcjQ3mCxSieGSSF4D1lm62ORZI7Am++klWBKVxTwrJNmlPMGljgjoquJpYpFRhJJUUTxxkbuVZIZWuF0goLkEgFo8ddHMkckXoqVElOKeeFkhfL5Jusnm66aSZs4hmEkdWzEzPG/W6kW6TAjq0QpZWqgSQrqiSV9Tst0WRgoBZyLhmChlJY8gwvzwnmalkMcqpDOJXKiZOodQUjd9Zctcheq0djWwJGwUMyxpV9GExo6uILaZ48tRZWkillnjo4qfroWllg0ydbongLT04jcyszRFHZTpy7giqLpKY6i5rGlkFTJlom0jJ6+lWZo8uiiplZpZ4INpJ5GRYy3VqhCiFKk+cUa6naWmXWxhZzJENbgWMLMW7TAGxQkkeGNuf5hDThHdP4SaKBSqKTrqHSJb8uyToDHwA52GIn4n4MqRT0sMNKwC5eKZmpxlwdJCkayQTvXCRVo3CJqalilkJQ/Nju6OIMpqPwXQqsEktRA+XSSwK8Al/e8kLzIHknSBnUIw/htLEbMbg4UIT5qqmGnjuzRU8QNrsUijBY7C5KqCx7u8411tZToFjdoUVyqojNGAxYkoqqTZixRioA30m3I4Y+edfLPTVZy6oZadaqE0kklAZy04pjHWR2rnpSI1impzrmSQLPJYEEh49puF5UlqKU0MdVNJk9LAAskPU0gnrM3KQlpjG5o4QUjLQI72p49MTbaRCn01UCz6NUS1Mi303QTui9+m+tlW3PcC2NVBJTP1kUZgfS1po0MbaWPMSIt7MbfKFzbER1fR9Ventf0qWOSspKrro5MtSntTpTgtM0tLJmgmX0dkEcJaORGRDJEskuhY6LeGKiGtJaBoaZIZ4/jmo5tDPPG6JRVECrWtSuFeSVa5VbUsGkEhrCFKsVOotZQLDSLACyjkotyXy5Y24MGBCMGDBgQjGM0dwQeRBH04ywYEL5B0lSAosO4XuSb7b+HM3Nu7ztckk9xbYDwAsMOXM+DjFSwTE9mWNGFmvbUoNiCv6ThutS998MKQG90gnHmDBh6VdVLJ2SPI41Uo7QxgrY9hksb4EKU8k46WGkEaJ8cFK7gFdxbVzG55m+9/HvjnMzffx3P7e/Gn00+X1/rxrkqCRbbAlWdDUFHDDmp+nxHvG2NzSgyk/JLkju2LfqOOKR748U4Eil7hydfi5A2l1KupK3UOhBFyDqtqXuU4b3HfCkjz1M8AE8BllkJhYO8SM7sDNEPj4VUfLkjVPysNmi4hdF0gKRvzB7zf5w78amzyTrBKrGORTdXQsrKQbgqwbUpHiCMQtjIdaeXWFopNicPDgjPZImkCStAZI9GtXdBuw2cowOgprS5DaS4baxYdnDXTFUQFz1NJLJIAJnlgYtMFJKGQJIkbOupvjNId9XbZyAQ38z4vDzCX0SljF7tFEKiOF9iLFVqboOR+KMdyBe+93ObaYDuntRZTUKocSGGFTpkeWRo4UcDVo21NIzKQQkKySFSGClTfHXn3ESxKskBaduRlluI0IuNUdOSQxN/Xqta7LeBGwxq7pCmkUrKkUqiwiBRkECj/RwrC8axxf7OxW92tqZmbgp+KSoYCGHtCxJErj26ZJmjJHddTiBsLgb/lJxITgzvO5Sxfr5pCbMWeVyxY7i51Dcbey3kMSx0C9LMLzCOvio9IDNJWTiQTsFuQDo1RzTtcAEojOFu7u4GuudTmTNe9t/AW8AAALAAAWsBjmimI5YkfAHNootXdq+m7JpgmqkjcxMHj65YlCFT2XF1uDv6uk35b40v0jZfVz9WpbVNIhERdGgLBREnZCqQgJvvub2J0gRilhqz5Y6Muzd42DLba+xvp3FuQIxF+CjIAIuu6LN819DpOhyndTYqoa5bqrgElSpbSrKgfSSNfrDxw5ujjg0UMAi6wzKEWMExWOhQdIfQzqSATdgFXfkL4oDk3TZmMP8HOQL3s15B7AJS9h7LYXZfhJZoRYyR+fxfP3FtO/LYDD3Y7HEEt5ckhJHIqavhMZ7l/YekrOprkcdumaRhfVoMcrwmyhLM5uWdAlgjagjQ3mubVLJpfMacRWI6uOWVYDfn8RRUvV6vPq9/HxjPO+KpZmZpNJZ3LswBBLEkk7GwuSdgLYTfwg3l9f68OEW26dqpPynoaVSL1Oo33FNSO497VclE1z5I1vPDziyQNCstNIZ41VjLGyCKqiQOeslaISSI8Ca+1JDJJpDapFiGIUpszIYEqrAfJOoA+0o6sPcwwsx8d1CtG0bCAxNqi6q66HvfWO0Tq3tc3uLg3ubsfATyTg+loyqrMcgbvG4/b6sSzRZpeMkciNveMQ/nnEJlmeXqootbFzHErLEpO7BFLsVUtdtIOlb2UKoVR2UHGsqLpCxkeatf6nGHOY7skDgpBzXOWcxqhtKzhVBHZLEFFubbbvfn3ezFoeG1otLxwTnrY2csj30sRYFEZlUhiVuvafV2h4FaKf9KJNavZNSMHGzWuDcfK5XwrZf0kVEY7Ijva17Pf23Eg3xLDqjNhVMzHbkRlhCu3n+YmOItzAIFieV7+3v7hhgV9bqroPyhEdvMYgzOfhAVsqaGjpgLg3VJQdvbORvfwwiS9LdSZEcpBdAqgaZLWW9r/G37/EYvSTNcQsLD4VLEw6qsgjn9FbbpJz5YKcEte9xpB8eSfklzufyRv4YjzoZ6XJ8sgq9NNDUrJPLUsTUyRMoILkaRSyD1Rz1d3hviCOJ+lWoqFCukK6TcFFkB5cu1Kwt37AfULJq8dzCKSPTHaUWZrPqtYCw+MsALX5XJO9xYBJHsc/0r7q7w/Cfjxf/V7+3/itbJ8MqUf/AKdF/XX+547eHvhbVNRMkMWWRvLIbIorWux52H7zsLC7EmwABJIAxSr8LN4D6/14WeCeO56OpjqYdIkjOwIbQynZo3CupKONiAQfMYqjnuth1VtzV9+LemiupmVXo6JpCpYpHmFQzKoFyW//AKYABseRPInkL4wz/psroE1vR0WkAayMwntGzC4icnLQvX7j4lS0n5NgSKgfuha/rxPppxJcFrRv2yLeset1CwAChGUR/I0XN2pxx0mVdbL1tRJ1jC4RdxHGp+RGgYKi7DkLta7FjviSVzQBoFn1KZiMLn/77qb6CyvoDTdJmYNEsopcuETIrhjmVQF0sAQSfwVYc/HniMuIPhazwuVbL4W8CtbJpO9ttVCp+kDFSMm6R6mKPqwweIEsqOXKIT6xQdYAuo7m3fc8yb6c644klHaji27wrg/XIcR/0+qeKbKerenQq0ifDXkJt+DU/rrfdMb/AN2dL/1bH/XW+54pw2Ztcmy3PPY/Tz5nHv4VbwX6/wBeEF9UslX5eSupw/8AC9lmmSL0Gni1lhrlr3WJdKM/aYULEA6dINjuRyFyJS4X6Rs0qU1x5fSdWT2HevqEDi9tahssDabg+sFJ5gWIOPn90bdI81BUrUxRwSSJqsJkdk7SspNllQ3AY2II3xL4+Gbmn4vQf0NT98w6xSiIdfNXD/DWbfiuW/2lV/oyfGLZ7m34plx9mY1f9z4qAPhoZp+L5f8A0NT99wN8NDNP4ig/oan75hE5XDTN83P+qZd/aVX/AHRjP8J5v+KZb/aVX/c+KYyfDHzU/wCjoh7Ip/vWOWo+Fzmjc0pf5lR96wiN1dj8J5v+KZb/AGlV/wBz48bNs2H+q5b/AGlV/wBz4o3J8KbMjzjpf5k/3rHNJ8JnMO6OlHsjl/TUE4VIS7sr0nO81/Fss/tOr/ujGK5pmt9QpMsuQAW/CNVcgEkC/wCB7kAs1h3XPicUNf4RdefkwfRUH7ak292FDKPhR5nELKKci97MkzD6DUWHLutgTQX9ld6v4hzdFLeh5e9vkrmVTqO/dqylRtz54aGddL2aQ+vlUdvnLVVTL9Iyy2K1L8MPM/4ig/oaj75gPww8z/iKD+iqPvmBKQ7optn+E9UL61FSr+dmEo+2gGNDfCqlH+qUf9oyfcMVx4t+EPVVQIkpaEajdikdQpJ8T++yL+dr4jabiWQsSAi37lB0j2AscB9EN1dVc6o+FrKB/mVO3ktdMT/4fb6cJ5+GPP8A9Vr/AFx/ueKgf9JZPBfob/1YyHEz/NT6G/8AXhhLuykACt8vwxJ/+q099aR9tJjyP4Y85NvwYl72t6a2/svSb4qHDxM4IJSNx3qwex8jpkVrewg4eXDHTI9P/B0OX6r3DtFUM4Pdv6X3YUauqQ+idWWdCudVUEZ9Gbq41WNFkmiiNgoGoJJIGt3XIHkNsKH7mLN++OAf/wCyDb3KrYypfhg5mosIKC3/AMKp++Y3H4ZOafxFB/Q1P3zC0m7qt2DBgwqcjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBCMGDBgQjBgwYEIwYMGBC/9k=",
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"960\"\n",
       "            height=\"540\"\n",
       "            src=\"https://www.youtube.com/embed/rEVboOny7rY\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.YouTubeVideo at 0x207472e5760>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import YouTubeVideo\n",
    "\n",
    "YouTubeVideo('pgIfTQnjdzA', width=960, height=540)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "0ba933b9",
   "metadata": {},
   "source": [
    "# Discussion\n",
    "\n",
    "Differently from the works found in the literature, which train and validate their models only against a RandomPlayer, our methods were trained against a MaxDamagePlayer and validated against a MaxDamagePlayer and against a RandomPlayer, showing significant improvements. All of our implemented methods showed better results than most of our methods from P1 with the same battle configurations.\n",
    "\n",
    "Our DQN agent, in a stochastic environment, with 300k training steps, averaged $69,57\\%$ wins against a MaxDamagePlayer and $98,59\\%$ wins against a RandomPlayer. With 900k training steps, in a stochastic environment, our DQN agent averaged $60,97\\%$ wins against a MaxDamagePlayer and $99,28\\%$ wins against a RandomPlayer. In a deterministic environment, with 300k training steps, the agent averaged $71,71\\%$ wins against a MaxDamagePlayer and $98,5\\%$ wins against a RandomPlayer. With 900k training steps, in a deterministic environment, our DQN agent averaged $74,62\\%$ wins against a MaxDamagePlayer and $98,89\\%$ wins against a RandomPlayer. In a visual analysis of the battles, we noticed that the agent promoted switches to Pokémon with super-effective attacks against the opponent. In the stochastic environment, the agent also made use of a combo with Sleep Powder (cause status to the opponent) and Leech Seed (removes opponent's Health Points in each turn and adds them to ours). Compared to the other implemented methods, the agent with DQN trained by 900k steps was the one with the worst performance against MaxDamagePlayer.\n",
    "\n",
    "Our Double-DQN agent, in a stochastic environment, with 300k training steps, averaged $67\\%$ wins against a MaxDamagePlayer and $98\\%$ wins against a RandomPlayer. With 900k training steps, in a stochastic environment, our DQN agent averaged $83,94\\%$ wins against a MaxDamagePlayer and $99,24\\%$ wins against a RandomPlayer. In a deterministic environment, with 300k training steps, the agent averaged $77,08\\%$ wins against a MaxDamagePlayer and $99,37\\%$ wins against a RandomPlayer. With 900k training steps, in a deterministic environment, our DQN agent averaged $77,92\\%$ wins against a MaxDamagePlayer and $98,98\\%$ wins against a RandomPlayer. In a visual analysis of the battles, we noticed that the agent promoted switches to better type Pokémon, to not suffer super-effective attacks, and to Pokémon with super-effective attacks against the opponent. \n",
    "\n",
    "Our PPO agent, in a stochastic environment, with 300k training steps, averaged $88,93\\%$ wins against a MaxDamagePlayer and $99,94\\%$ wins against a RandomPlayer. With 900k training steps, in a stochastic environment, our PPO agent averaged $88,21\\%$ wins against a MaxDamagePlayer and $99,67\\%$ wins against a RandomPlayer. In a deterministic environment, with 300k training steps, the agent averaged $74,86\\%$ wins against a MaxDamagePlayer and $99,73\\%$ wins against a RandomPlayer. With 900k training steps, in a deterministic environment, our PPO agent averaged $82,75\\%$ wins against a MaxDamagePlayer and $99,49\\%$ wins against a RandomPlayer. In a visual analysis of the battles, we noticed that the agent also promoted switches to better type Pokémon, to not suffer super-effective attacks, and to Pokémon with super-effective attacks against the opponent.  In addition, the agent made a lot of use of the Giga Drain damage move (promotes damage and recovers HP), even when this move wasn't very effective. We hypothesize that this occurs because the Player realized that, even if the move was not very effective, it promoted a decrease in the opponent's Health Points and an increase in ours, promoting greater rewards.\n",
    "\n",
    "Our REINFORCE Keras agent, in a stochastic environment, with 10k training steps, averaged $87,52\\%$ wins against a MaxDamagePlayer and $99,43\\%$ wins against a RandomPlayer. With 30k training steps, in a stochastic environment, our REINFORCE Keras agent averaged $85,07\\%$ wins against a MaxDamagePlayer and $99,14\\%$ wins against a RandomPlayer. In a deterministic environment, with 10k training steps, the agent averaged $60,16\\%$ wins against a MaxDamagePlayer and $99,13\\%$ wins against a RandomPlayer. With 30k training steps, in a deterministic environment, our REINFORCE Keras agent averaged $75,97\\%$ wins against a MaxDamagePlayer and $99,55\\%$ wins against a RandomPlayer. In a visual analysis of the battles, we noticed behaviors similar to the ones learned by the PPO agent, mainly the agent made many uses of Giga Drain damage move (causes damage and recovers HP), even when this move wasn't very effective. Again, we hypothesize that the agent learned to have a preference for this move because causing a decrease in the opponent's Pokémon Health Points and an increase in ours is a way to promote a greater accumulation of rewards in the long run of the episode.\n",
    "\n",
    "In Pytorch Framework, the models perfomed as good as the Keras using our REINFORCE Keras agent, and slighty worst when running out the DQN model. For Pytorch DQN model in stochastice environment with 5k battles, the model won around $30,00\\%$ against MaxDamagePlayer, while with the RandomDamagePlayer the agent won almost $95,00\\%$ of the battle. The main contribution, however, is the available GPU in the Pytorch Framework. It decreased sharply the time training (around $50\\%$ faster) and allow us, from now on, to try new architectures of the models and new hyperparameters in order to improve the results. As a negative point, and probably the explanation about the lower accuracy results acquired, is that Pytorch does not provide an easy API to apply Reinforcement Learning so it is easier prone to errors. In addition, in order to use the Pytorch correctly, several data transformation into tensor was necessary. Therefore, a thorough and detailed analysis of the code is more necessary than when using Keras Framework. For future improvements, we intend to analyse the DQNAgent Keras algorithm itself to compared how the models were build and try to improve the results of the Pytorch Framework."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7efc47cb",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "\n",
    "Given the complexity of a Pokémon battle (as shown, $1,016,064$ states and $9,144,576$ possibilities), the implemented methods were able to generalize the states and explore the best reward acquisitions, with all methods performing better than a RandomPlayer, a MaxPlayer and most of our tabular methods with the same battle configurations. The methods showed increasing learning, in view of the increase in rewards and victory rate as battles went on.\n",
    "\n",
    "Particularly, PPO found it easier to generalize the states, given the influence of the Actor-Critic network.\n",
    "\n",
    "In addition to the difference in training and validating using a MaxDamagePlayer (getting good results), our PPO method showed better results than related work against a RandomPlayer.\n",
    "\n",
    "As future work, we propose to consider as inputs for our networks the types of Pokémon, their abilities and possible side effects of moves.\n",
    "\n",
    "In the professional scope, it is important to mention that this project allowed us to know theoretically and practically the models and techniques of Reinforcement Learning. Since RL is an area increasingly demanded by current technologies, this project is of particular importance, as it brought us relatively in-depth knowledge of RL techniques, how to apply and improve them. Finally, it is important to emphasize, for everyone on the team, that after this project, we are able to \"walk on our own two legs\" to go deeper into the RL area and contribute to this growing community."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d21626ae",
   "metadata": {},
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9837eaab",
   "metadata": {},
   "source": [
    "# Contributions\n",
    "\n",
    "## Methods\n",
    "\n",
    "### Deep Q-Learning - Keras\n",
    "* **Code development**: Leonardo;\n",
    "* **Code review**: Bruno, Henrique, Maurício;\n",
    "* **Experiments**: Leonardo.\n",
    "\n",
    "### DQN - Pytorch\n",
    "* **Code development**: Bruno;\n",
    "* **Code review**: Henrique, Leonardo, Maurício;\n",
    "* **Experiments**: Bruno.\n",
    "\n",
    "### Double Deep Q-Learning - Keras\n",
    "* **Code development**: Leonardo;\n",
    "* **Code review**: Bruno, Henrique, Maurício;\n",
    "* **Experiments**: Leonardo.\n",
    "\n",
    "### PPO\n",
    "* **Code development**: Leonardo;\n",
    "* **Code review**: Bruno, Henrique, Maurício;\n",
    "* **Experiments**: Leonardo.\n",
    "\n",
    "### REINFORCE - Keras\n",
    "* **Code development**: Henrique;\n",
    "* **Code review**: Bruno, Leonardo, Maurício;\n",
    "* **Experiments**: Henrique.\n",
    "\n",
    "### REINFORCE - Pytorch\n",
    "* **Code development**: Bruno;\n",
    "* **Code review**: Henrique, Leonardo, Maurício;\n",
    "* **Experiments**: Bruno.\n",
    "\n",
    "## Video\n",
    "* **Editing**: Leonardo;\n",
    "* **Review**: Bruno, Henrique, Maurício.\n",
    "\n",
    "## Report\n",
    "* **Writting**: Leonardo, Henrique, Bruno, Mauricio."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6249f56",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}