Q-learning-cart.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Q-learning\n",
    "\n",
    "In this notebook, we'll build a neural network that can learn to play games through reinforcement learning. More specifically, we'll use Q-learning to train an agent to play a game called [Cart-Pole](https://gym.openai.com/envs/CartPole-v0). In this game, a freely swinging pole is attached to a cart. The cart can move to the left and right, and the goal is to keep the pole upright as long as possible.\n",
    "\n",
    "![Cart-Pole](assets/cart-pole.jpg)\n",
    "\n",
    "We can simulate this game using [OpenAI Gym](https://gym.openai.com/). First, let's check out how OpenAI Gym works. Then, we'll get into training an agent to play the Cart-Pole game."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import gym\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">**Note:** Make sure you have OpenAI Gym cloned into the same directory with this notebook. I've included `gym` as a submodule, so you can run `git submodule --init --recursive` to pull the contents into the `gym` repo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-09-07 19:39:24,589] Making new env: CartPole-v0\n"
     ]
    }
   ],
   "source": [
    "# Create the Cart-Pole game environment\n",
    "env = gym.make('CartPole-v0')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We interact with the simulation through `env`. To show the simulation running, you can use `env.render()` to render one frame. Passing in an action as an integer to `env.step` will generate the next step in the simulation.  You can see how many actions are possible from `env.action_space` and to get a random action you can use `env.action_space.sample()`. This is general to all Gym games. In the Cart-Pole game, there are two possible actions, moving the cart left or right. So there are two actions we can take, encoded as 0 and 1.\n",
    "\n",
    "Run the code below to watch the simulation run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] True\n",
      "[1.0] False\n",
      "[1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n",
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] False\n"
     ]
    }
   ],
   "source": [
    "env.reset()\n",
    "rewards = []\n",
    "tic = time.time()\n",
    "for _ in range(1000):\n",
    "    env.render()\n",
    "    state, reward, done, info = env.step(env.action_space.sample()) # take a random action\n",
    "    rewards.append(reward)\n",
    "    if done:\n",
    "        rewards = []\n",
    "        env.reset()\n",
    "toc = time.time()\n",
    "# Shut simulation window after 10s\n",
    "if toc-tic > 10:\n",
    "    env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To shut the window showing the simulation, use `env.close()`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you ran the simulation above, we can look at the rewards:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]\n"
     ]
    }
   ],
   "source": [
    "print(rewards)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The game resets after the pole has fallen past a certain angle. For each frame while the simulation is running, it returns a reward of 1.0. The longer the game runs, the more reward we get. Then, our network's goal is to maximize the reward by keeping the pole vertical. It will do this by moving the cart to the left and the right."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q-Network\n",
    "\n",
    "We train our Q-learning agent using the Bellman Equation:\n",
    "\n",
    "$$\n",
    "Q(s, a) = r + \\gamma \\max{Q(s', a')}\n",
    "$$\n",
    "\n",
    "where $s$ is a state, $a$ is an action, and $s'$ is the next state from state $s$ and action $a$.\n",
    "\n",
    "Before we used this equation to learn values for a Q-_table_. However, for this game there are a huge number of states available. The state has four values: the position and velocity of the cart, and the position and velocity of the pole. These are all real-valued numbers, so ignoring floating point precisions, you practically have infinite states. Instead of using a table then, we'll replace it with a neural network that will approximate the Q-table lookup function.\n",
    "\n",
    "<img src=\"assets/deep-q-learning.png\" width=450px>\n",
    "\n",
    "Now, our Q value, $Q(s, a)$ is calculated by passing in a state to the network. The output will be Q-values for each available action, with fully connected hidden layers.\n",
    "\n",
    "<img src=\"assets/q-network.png\" width=550px>\n",
    "\n",
    "\n",
    "As I showed before, we can define our targets for training as $\\hat{Q}(s,a) = r + \\gamma \\max{Q(s', a')}$. Then we update the weights by minimizing $(\\hat{Q}(s,a) - Q(s,a))^2$. \n",
    "\n",
    "For this Cart-Pole game, we have four inputs, one for each value in the state, and two outputs, one for each action. To get $\\hat{Q}$, we'll first choose an action, then simulate the game using that action. This will get us the next state, $s'$, and the reward. With that, we can calculate $\\hat{Q}$ then pass it back into the $Q$ network to run the optimizer and update the weights.\n",
    "\n",
    "Below is my implementation of the Q-network. I used two fully connected layers with ReLU activations. Two seems to be good enough, three might be better. Feel free to try it out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class QNetwork:\n",
    "    def __init__(self, learning_rate=0.01, state_size=4, \n",
    "                 action_size=2, hidden_size=10, \n",
    "                 name='QNetwork'):\n",
    "        # state inputs to the Q-network\n",
    "        with tf.variable_scope(name):\n",
    "            self.inputs_ = tf.placeholder(tf.float32, [None, state_size], name='inputs')\n",
    "            \n",
    "            # One hot encode the actions to later choose the Q-value for the action\n",
    "            self.actions_ = tf.placeholder(tf.int32, [None], name='actions')\n",
    "            one_hot_actions = tf.one_hot(self.actions_, action_size)\n",
    "            \n",
    "            # Target Q values for training\n",
    "            self.targetQs_ = tf.placeholder(tf.float32, [None], name='target')\n",
    "            \n",
    "            # ReLU hidden layers\n",
    "            self.fc1 = tf.contrib.layers.fully_connected(self.inputs_, hidden_size)\n",
    "            self.fc2 = tf.contrib.layers.fully_connected(self.fc1, hidden_size)\n",
    "\n",
    "            # Linear output layer\n",
    "            self.output = tf.contrib.layers.fully_connected(self.fc2, action_size, \n",
    "                                                            activation_fn=None)\n",
    "                        \n",
    "            ### Train with loss (targetQ - Q)^2\n",
    "            # output has length 2, for two actions. This next line chooses\n",
    "            # one value from output (per row) according to the one-hot encoded actions.\n",
    "            self.Q = tf.reduce_sum(tf.multiply(self.output, one_hot_actions), axis=1)\n",
    "            \n",
    "            self.loss = tf.reduce_mean(tf.square(self.targetQs_ - self.Q))\n",
    "            self.opt = tf.train.AdamOptimizer(learning_rate).minimize(self.loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experience replay\n",
    "\n",
    "Reinforcement learning algorithms can have stability issues due to correlations between states. To reduce correlations when training, we can store the agent's experiences and later draw a random mini-batch of those experiences to train on. \n",
    "\n",
    "Here, we'll create a `Memory` object that will store our experiences, our transitions $<s, a, r, s'>$. This memory will have a maxmium capacity, so we can keep newer experiences in memory while getting rid of older experiences. Then, we'll sample a random mini-batch of transitions $<s, a, r, s'>$ and train on those.\n",
    "\n",
    "Below, I've implemented a `Memory` object. If you're unfamiliar with `deque`, this is a double-ended queue. You can think of it like a tube open on both sides. You can put objects in either side of the tube. But if it's full, adding anything more will push an object out the other side. This is a great data structure to use for the memory buffer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from collections import deque\n",
    "class Memory():\n",
    "    def __init__(self, max_size = 1000):\n",
    "        self.buffer = deque(maxlen=max_size)\n",
    "    \n",
    "    def add(self, experience):\n",
    "        self.buffer.append(experience)\n",
    "            \n",
    "    def sample(self, batch_size):\n",
    "        idx = np.random.choice(np.arange(len(self.buffer)), \n",
    "                               size=batch_size, \n",
    "                               replace=False)\n",
    "        return [self.buffer[ii] for ii in idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploration - Exploitation\n",
    "\n",
    "To learn about the environment and rules of the game, the agent needs to explore by taking random actions. We'll do this by choosing a random action with some probability $\\epsilon$ (epsilon).  That is, with some probability $\\epsilon$ the agent will make a random action and with probability $1 - \\epsilon$, the agent will choose an action from $Q(s,a)$. This is called an **$\\epsilon$-greedy policy**.\n",
    "\n",
    "\n",
    "At first, the agent needs to do a lot of exploring. Later when it has learned more, the agent can favor choosing actions based on what it has learned. This is called _exploitation_. We'll set it up so the agent is more likely to explore early in training, then more likely to exploit later in training."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q-Learning training algorithm\n",
    "\n",
    "Putting all this together, we can list out the algorithm we'll use to train the network. We'll train the network in _episodes_. One *episode* is one simulation of the game. For this game, the goal is to keep the pole upright for 195 frames. So we can start a new episode once meeting that goal. The game ends if the pole tilts over too far, or if the cart moves too far the left or right. When a game ends, we'll start a new episode. Now, to train the agent:\n",
    "\n",
    "* Initialize the memory $D$\n",
    "* Initialize the action-value network $Q$ with random weights\n",
    "* **For** episode = 1, $M$ **do**\n",
    "  * **For** $t$, $T$ **do**\n",
    "     * With probability $\\epsilon$ select a random action $a_t$, otherwise select $a_t = \\mathrm{argmax}_a Q(s,a)$\n",
    "     * Execute action $a_t$ in simulator and observe reward $r_{t+1}$ and new state $s_{t+1}$\n",
    "     * Store transition $<s_t, a_t, r_{t+1}, s_{t+1}>$ in memory $D$\n",
    "     * Sample random mini-batch from $D$: $<s_j, a_j, r_j, s'_j>$\n",
    "     * Set $\\hat{Q}_j = r_j$ if the episode ends at $j+1$, otherwise set $\\hat{Q}_j = r_j + \\gamma \\max_{a'}{Q(s'_j, a')}$\n",
    "     * Make a gradient descent step with loss $(\\hat{Q}_j - Q(s_j, a_j))^2$\n",
    "  * **endfor**\n",
    "* **endfor**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyperparameters\n",
    "\n",
    "One of the more difficult aspects of reinforcememt learning are the large number of hyperparameters. Not only are we tuning the network, but we're tuning the simulation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_episodes = 1000          # max number of episodes to learn from\n",
    "max_steps = 200                # max steps in an episode\n",
    "gamma = 0.99                   # future reward discount\n",
    "\n",
    "# Exploration parameters\n",
    "explore_start = 1.0            # exploration probability at start\n",
    "explore_stop = 0.01            # minimum exploration probability \n",
    "decay_rate = 0.0001            # exponential decay rate for exploration prob\n",
    "\n",
    "# Network parameters\n",
    "hidden_size = 64               # number of units in each Q-network hidden layer\n",
    "learning_rate = 0.0001         # Q-network learning rate\n",
    "\n",
    "# Memory parameters\n",
    "memory_size = 10000            # memory capacity\n",
    "batch_size = 20                # experience mini-batch size\n",
    "pretrain_length = batch_size   # number experiences to pretrain the memory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tf.reset_default_graph()\n",
    "mainQN = QNetwork(name='main', hidden_size=hidden_size, learning_rate=learning_rate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Populate the experience memory\n",
    "\n",
    "Here I'm re-initializing the simulation and pre-populating the memory. The agent is taking random actions and storing the transitions in memory. This will help the agent with exploring the game."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Initialize the simulation\n",
    "env.reset()\n",
    "# Take one random step to get the pole and cart moving\n",
    "state, reward, done, _ = env.step(env.action_space.sample())\n",
    "\n",
    "memory = Memory(max_size=memory_size)\n",
    "\n",
    "# Make a bunch of random actions and store the experiences\n",
    "for ii in range(pretrain_length):\n",
    "    # Uncomment the line below to watch the simulation\n",
    "    # env.render()\n",
    "\n",
    "    # Make a random action\n",
    "    action = env.action_space.sample()\n",
    "    next_state, reward, done, _ = env.step(action)\n",
    "\n",
    "    if done:\n",
    "        # The simulation fails so no next state\n",
    "        next_state = np.zeros(state.shape)\n",
    "        # Add experience to memory\n",
    "        memory.add((state, action, reward, next_state))\n",
    "        \n",
    "        # Start new episode\n",
    "        env.reset()\n",
    "        # Take one random step to get the pole and cart moving\n",
    "        state, reward, done, _ = env.step(env.action_space.sample())\n",
    "    else:\n",
    "        # Add experience to memory\n",
    "        memory.add((state, action, reward, next_state))\n",
    "        state = next_state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "\n",
    "Below we'll train our agent. If you want to watch it train, uncomment the `env.render()` line. This is slow because it's rendering the frames slower than the network can train. But, it's cool to watch the agent get better at the game."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Episode: 1 Total reward: 4.0 Training loss: 1.2308 Explore P: 0.9996\n",
      "Episode: 2 Total reward: 51.0 Training loss: 1.2220 Explore P: 0.9946\n",
      "Episode: 3 Total reward: 11.0 Training loss: 1.2811 Explore P: 0.9935\n",
      "Episode: 4 Total reward: 7.0 Training loss: 1.4345 Explore P: 0.9928\n",
      "Episode: 5 Total reward: 10.0 Training loss: 1.2150 Explore P: 0.9918\n",
      "Episode: 6 Total reward: 82.0 Training loss: 1.6457 Explore P: 0.9838\n",
      "Episode: 7 Total reward: 15.0 Training loss: 1.2781 Explore P: 0.9823\n",
      "Episode: 8 Total reward: 17.0 Training loss: 1.9587 Explore P: 0.9807\n",
      "Episode: 9 Total reward: 42.0 Training loss: 1.7015 Explore P: 0.9766\n",
      "Episode: 10 Total reward: 15.0 Training loss: 1.9394 Explore P: 0.9752\n",
      "Episode: 11 Total reward: 22.0 Training loss: 1.9603 Explore P: 0.9730\n",
      "Episode: 12 Total reward: 25.0 Training loss: 1.9164 Explore P: 0.9706\n",
      "Episode: 13 Total reward: 22.0 Training loss: 2.0326 Explore P: 0.9685\n",
      "Episode: 14 Total reward: 9.0 Training loss: 1.9740 Explore P: 0.9677\n",
      "Episode: 15 Total reward: 16.0 Training loss: 1.9114 Explore P: 0.9661\n",
      "Episode: 16 Total reward: 20.0 Training loss: 2.0516 Explore P: 0.9642\n",
      "Episode: 17 Total reward: 19.0 Training loss: 1.7359 Explore P: 0.9624\n",
      "Episode: 18 Total reward: 19.0 Training loss: 2.4000 Explore P: 0.9606\n",
      "Episode: 19 Total reward: 14.0 Training loss: 2.3364 Explore P: 0.9593\n",
      "Episode: 20 Total reward: 12.0 Training loss: 2.2636 Explore P: 0.9581\n",
      "Episode: 21 Total reward: 16.0 Training loss: 2.8310 Explore P: 0.9566\n",
      "Episode: 22 Total reward: 10.0 Training loss: 2.9749 Explore P: 0.9557\n",
      "Episode: 23 Total reward: 19.0 Training loss: 2.3288 Explore P: 0.9539\n",
      "Episode: 24 Total reward: 16.0 Training loss: 2.9721 Explore P: 0.9524\n",
      "Episode: 25 Total reward: 11.0 Training loss: 4.3025 Explore P: 0.9513\n",
      "Episode: 26 Total reward: 29.0 Training loss: 2.2420 Explore P: 0.9486\n",
      "Episode: 27 Total reward: 25.0 Training loss: 3.8777 Explore P: 0.9463\n",
      "Episode: 28 Total reward: 54.0 Training loss: 3.3829 Explore P: 0.9412\n",
      "Episode: 29 Total reward: 21.0 Training loss: 2.6760 Explore P: 0.9393\n",
      "Episode: 30 Total reward: 41.0 Training loss: 7.3668 Explore P: 0.9355\n",
      "Episode: 31 Total reward: 11.0 Training loss: 3.1044 Explore P: 0.9345\n",
      "Episode: 32 Total reward: 12.0 Training loss: 5.8160 Explore P: 0.9333\n",
      "Episode: 33 Total reward: 27.0 Training loss: 12.7592 Explore P: 0.9309\n",
      "Episode: 34 Total reward: 35.0 Training loss: 4.4698 Explore P: 0.9276\n",
      "Episode: 35 Total reward: 15.0 Training loss: 9.3394 Explore P: 0.9263\n",
      "Episode: 36 Total reward: 14.0 Training loss: 4.8045 Explore P: 0.9250\n",
      "Episode: 37 Total reward: 19.0 Training loss: 13.5382 Explore P: 0.9232\n",
      "Episode: 38 Total reward: 10.0 Training loss: 9.4512 Explore P: 0.9223\n",
      "Episode: 39 Total reward: 8.0 Training loss: 47.9321 Explore P: 0.9216\n",
      "Episode: 40 Total reward: 11.0 Training loss: 8.1713 Explore P: 0.9206\n",
      "Episode: 41 Total reward: 10.0 Training loss: 11.4322 Explore P: 0.9197\n",
      "Episode: 42 Total reward: 15.0 Training loss: 17.6473 Explore P: 0.9183\n",
      "Episode: 43 Total reward: 16.0 Training loss: 9.2386 Explore P: 0.9169\n",
      "Episode: 44 Total reward: 15.0 Training loss: 12.2307 Explore P: 0.9155\n",
      "Episode: 45 Total reward: 19.0 Training loss: 89.1912 Explore P: 0.9138\n",
      "Episode: 46 Total reward: 17.0 Training loss: 12.3282 Explore P: 0.9123\n",
      "Episode: 47 Total reward: 11.0 Training loss: 45.3992 Explore P: 0.9113\n",
      "Episode: 48 Total reward: 25.0 Training loss: 12.3309 Explore P: 0.9090\n",
      "Episode: 49 Total reward: 19.0 Training loss: 14.7068 Explore P: 0.9073\n",
      "Episode: 50 Total reward: 53.0 Training loss: 20.6310 Explore P: 0.9026\n",
      "Episode: 51 Total reward: 24.0 Training loss: 48.8468 Explore P: 0.9004\n",
      "Episode: 52 Total reward: 23.0 Training loss: 23.6429 Explore P: 0.8984\n",
      "Episode: 53 Total reward: 39.0 Training loss: 61.0569 Explore P: 0.8949\n",
      "Episode: 54 Total reward: 11.0 Training loss: 94.3398 Explore P: 0.8940\n",
      "Episode: 55 Total reward: 30.0 Training loss: 28.2704 Explore P: 0.8913\n",
      "Episode: 56 Total reward: 12.0 Training loss: 138.2957 Explore P: 0.8902\n",
      "Episode: 57 Total reward: 15.0 Training loss: 25.2394 Explore P: 0.8889\n",
      "Episode: 58 Total reward: 12.0 Training loss: 38.1609 Explore P: 0.8879\n",
      "Episode: 59 Total reward: 16.0 Training loss: 16.8602 Explore P: 0.8865\n",
      "Episode: 60 Total reward: 24.0 Training loss: 24.5123 Explore P: 0.8844\n",
      "Episode: 61 Total reward: 28.0 Training loss: 147.8650 Explore P: 0.8819\n",
      "Episode: 62 Total reward: 35.0 Training loss: 30.3783 Explore P: 0.8789\n",
      "Episode: 63 Total reward: 27.0 Training loss: 28.3287 Explore P: 0.8765\n",
      "Episode: 64 Total reward: 38.0 Training loss: 25.1105 Explore P: 0.8733\n",
      "Episode: 65 Total reward: 12.0 Training loss: 29.0901 Explore P: 0.8722\n",
      "Episode: 66 Total reward: 12.0 Training loss: 51.4811 Explore P: 0.8712\n",
      "Episode: 67 Total reward: 9.0 Training loss: 241.2308 Explore P: 0.8704\n",
      "Episode: 68 Total reward: 8.0 Training loss: 97.3283 Explore P: 0.8697\n",
      "Episode: 69 Total reward: 13.0 Training loss: 209.9574 Explore P: 0.8686\n",
      "Episode: 70 Total reward: 14.0 Training loss: 65.9656 Explore P: 0.8674\n",
      "Episode: 71 Total reward: 15.0 Training loss: 238.1972 Explore P: 0.8661\n",
      "Episode: 72 Total reward: 19.0 Training loss: 60.4650 Explore P: 0.8645\n",
      "Episode: 73 Total reward: 22.0 Training loss: 591.2729 Explore P: 0.8626\n",
      "Episode: 74 Total reward: 21.0 Training loss: 32.7592 Explore P: 0.8608\n",
      "Episode: 75 Total reward: 14.0 Training loss: 87.3277 Explore P: 0.8596\n",
      "Episode: 76 Total reward: 21.0 Training loss: 53.0499 Explore P: 0.8579\n",
      "Episode: 77 Total reward: 16.0 Training loss: 41.8400 Explore P: 0.8565\n",
      "Episode: 78 Total reward: 12.0 Training loss: 79.1500 Explore P: 0.8555\n",
      "Episode: 79 Total reward: 17.0 Training loss: 467.7582 Explore P: 0.8540\n",
      "Episode: 80 Total reward: 85.0 Training loss: 80.2190 Explore P: 0.8469\n",
      "Episode: 81 Total reward: 11.0 Training loss: 45.4370 Explore P: 0.8460\n",
      "Episode: 82 Total reward: 12.0 Training loss: 47.9917 Explore P: 0.8450\n",
      "Episode: 83 Total reward: 16.0 Training loss: 735.5448 Explore P: 0.8436\n",
      "Episode: 84 Total reward: 34.0 Training loss: 66.1805 Explore P: 0.8408\n",
      "Episode: 85 Total reward: 19.0 Training loss: 839.9806 Explore P: 0.8392\n",
      "Episode: 86 Total reward: 15.0 Training loss: 44.7343 Explore P: 0.8380\n",
      "Episode: 87 Total reward: 19.0 Training loss: 109.4999 Explore P: 0.8364\n",
      "Episode: 88 Total reward: 15.0 Training loss: 51.1253 Explore P: 0.8352\n",
      "Episode: 89 Total reward: 14.0 Training loss: 413.9741 Explore P: 0.8340\n",
      "Episode: 90 Total reward: 14.0 Training loss: 231.5308 Explore P: 0.8329\n",
      "Episode: 91 Total reward: 17.0 Training loss: 420.3179 Explore P: 0.8315\n",
      "Episode: 92 Total reward: 13.0 Training loss: 42.5103 Explore P: 0.8304\n",
      "Episode: 93 Total reward: 35.0 Training loss: 117.7457 Explore P: 0.8275\n",
      "Episode: 94 Total reward: 20.0 Training loss: 448.5615 Explore P: 0.8259\n",
      "Episode: 95 Total reward: 8.0 Training loss: 42.1321 Explore P: 0.8253\n",
      "Episode: 96 Total reward: 15.0 Training loss: 481.2888 Explore P: 0.8240\n",
      "Episode: 97 Total reward: 17.0 Training loss: 44.0118 Explore P: 0.8227\n",
      "Episode: 98 Total reward: 9.0 Training loss: 184.3181 Explore P: 0.8219\n",
      "Episode: 99 Total reward: 26.0 Training loss: 107.4002 Explore P: 0.8198\n",
      "Episode: 100 Total reward: 26.0 Training loss: 999.7604 Explore P: 0.8177\n",
      "Episode: 101 Total reward: 40.0 Training loss: 145.2919 Explore P: 0.8145\n",
      "Episode: 102 Total reward: 22.0 Training loss: 36.5785 Explore P: 0.8127\n",
      "Episode: 103 Total reward: 8.0 Training loss: 308.2684 Explore P: 0.8121\n",
      "Episode: 104 Total reward: 20.0 Training loss: 62.3047 Explore P: 0.8105\n",
      "Episode: 105 Total reward: 11.0 Training loss: 410.9060 Explore P: 0.8096\n",
      "Episode: 106 Total reward: 11.0 Training loss: 49.0384 Explore P: 0.8087\n",
      "Episode: 107 Total reward: 17.0 Training loss: 1154.2255 Explore P: 0.8074\n",
      "Episode: 108 Total reward: 27.0 Training loss: 644.4222 Explore P: 0.8052\n",
      "Episode: 109 Total reward: 14.0 Training loss: 44.2177 Explore P: 0.8041\n",
      "Episode: 110 Total reward: 18.0 Training loss: 42.6806 Explore P: 0.8027\n",
      "Episode: 111 Total reward: 14.0 Training loss: 305.4220 Explore P: 0.8016\n",
      "Episode: 112 Total reward: 63.0 Training loss: 34.1939 Explore P: 0.7966\n",
      "Episode: 113 Total reward: 17.0 Training loss: 486.1391 Explore P: 0.7953\n",
      "Episode: 114 Total reward: 23.0 Training loss: 200.4733 Explore P: 0.7934\n",
      "Episode: 115 Total reward: 14.0 Training loss: 568.3517 Explore P: 0.7924\n",
      "Episode: 116 Total reward: 12.0 Training loss: 541.7924 Explore P: 0.7914\n",
      "Episode: 117 Total reward: 15.0 Training loss: 421.3456 Explore P: 0.7902\n",
      "Episode: 118 Total reward: 10.0 Training loss: 178.3528 Explore P: 0.7895\n",
      "Episode: 119 Total reward: 30.0 Training loss: 41.1061 Explore P: 0.7871\n",
      "Episode: 120 Total reward: 11.0 Training loss: 263.4268 Explore P: 0.7863\n",
      "Episode: 121 Total reward: 16.0 Training loss: 704.4373 Explore P: 0.7850\n",
      "Episode: 122 Total reward: 24.0 Training loss: 363.1497 Explore P: 0.7832\n",
      "Episode: 123 Total reward: 10.0 Training loss: 46.9377 Explore P: 0.7824\n",
      "Episode: 124 Total reward: 10.0 Training loss: 517.1523 Explore P: 0.7816\n",
      "Episode: 125 Total reward: 16.0 Training loss: 538.1329 Explore P: 0.7804\n",
      "Episode: 126 Total reward: 27.0 Training loss: 575.7062 Explore P: 0.7783\n",
      "Episode: 127 Total reward: 8.0 Training loss: 40.2588 Explore P: 0.7777\n",
      "Episode: 128 Total reward: 11.0 Training loss: 580.7977 Explore P: 0.7769\n",
      "Episode: 129 Total reward: 18.0 Training loss: 1132.8999 Explore P: 0.7755\n",
      "Episode: 130 Total reward: 16.0 Training loss: 30.3627 Explore P: 0.7743\n",
      "Episode: 131 Total reward: 62.0 Training loss: 37.7993 Explore P: 0.7695\n",
      "Episode: 132 Total reward: 24.0 Training loss: 480.1477 Explore P: 0.7677\n",
      "Episode: 133 Total reward: 15.0 Training loss: 239.6924 Explore P: 0.7666\n",
      "Episode: 134 Total reward: 10.0 Training loss: 30.5045 Explore P: 0.7658\n",
      "Episode: 135 Total reward: 10.0 Training loss: 27.5342 Explore P: 0.7651\n",
      "Episode: 136 Total reward: 10.0 Training loss: 859.9683 Explore P: 0.7643\n",
      "Episode: 137 Total reward: 15.0 Training loss: 27.4503 Explore P: 0.7632\n",
      "Episode: 138 Total reward: 18.0 Training loss: 465.4585 Explore P: 0.7618\n",
      "Episode: 139 Total reward: 23.0 Training loss: 34.6523 Explore P: 0.7601\n",
      "Episode: 140 Total reward: 15.0 Training loss: 482.5653 Explore P: 0.7590\n",
      "Episode: 141 Total reward: 11.0 Training loss: 32.1338 Explore P: 0.7582\n",
      "Episode: 142 Total reward: 11.0 Training loss: 241.3595 Explore P: 0.7573\n",
      "Episode: 143 Total reward: 8.0 Training loss: 225.3929 Explore P: 0.7567\n",
      "Episode: 144 Total reward: 30.0 Training loss: 940.6722 Explore P: 0.7545\n",
      "Episode: 145 Total reward: 20.0 Training loss: 259.6427 Explore P: 0.7530\n",
      "Episode: 146 Total reward: 16.0 Training loss: 628.3023 Explore P: 0.7518\n",
      "Episode: 147 Total reward: 23.0 Training loss: 24.7664 Explore P: 0.7501\n",
      "Episode: 148 Total reward: 43.0 Training loss: 1225.3132 Explore P: 0.7469\n",
      "Episode: 149 Total reward: 8.0 Training loss: 315.0865 Explore P: 0.7463\n",
      "Episode: 150 Total reward: 14.0 Training loss: 1585.9747 Explore P: 0.7453\n",
      "Episode: 151 Total reward: 16.0 Training loss: 16.9710 Explore P: 0.7441\n",
      "Episode: 152 Total reward: 12.0 Training loss: 1088.4243 Explore P: 0.7433\n",
      "Episode: 153 Total reward: 20.0 Training loss: 16.1115 Explore P: 0.7418\n",
      "Episode: 154 Total reward: 21.0 Training loss: 838.3549 Explore P: 0.7403\n",
      "Episode: 155 Total reward: 14.0 Training loss: 15.0271 Explore P: 0.7392\n",
      "Episode: 156 Total reward: 11.0 Training loss: 278.8210 Explore P: 0.7384\n",
      "Episode: 157 Total reward: 12.0 Training loss: 408.5822 Explore P: 0.7376\n",
      "Episode: 158 Total reward: 13.0 Training loss: 315.4813 Explore P: 0.7366\n",
      "Episode: 159 Total reward: 38.0 Training loss: 315.4493 Explore P: 0.7339\n",
      "Episode: 160 Total reward: 20.0 Training loss: 683.3125 Explore P: 0.7324\n",
      "Episode: 161 Total reward: 14.0 Training loss: 10.4538 Explore P: 0.7314\n",
      "Episode: 162 Total reward: 21.0 Training loss: 14.5787 Explore P: 0.7299\n",
      "Episode: 163 Total reward: 27.0 Training loss: 11.6875 Explore P: 0.7280\n",
      "Episode: 164 Total reward: 11.0 Training loss: 11.1261 Explore P: 0.7272\n",
      "Episode: 165 Total reward: 42.0 Training loss: 533.9030 Explore P: 0.7242\n",
      "Episode: 166 Total reward: 11.0 Training loss: 569.9063 Explore P: 0.7234\n",
      "Episode: 167 Total reward: 37.0 Training loss: 561.9527 Explore P: 0.7207\n",
      "Episode: 168 Total reward: 26.0 Training loss: 304.4156 Explore P: 0.7189\n",
      "Episode: 169 Total reward: 13.0 Training loss: 6.1163 Explore P: 0.7180\n",
      "Episode: 170 Total reward: 11.0 Training loss: 1190.3947 Explore P: 0.7172\n",
      "Episode: 171 Total reward: 24.0 Training loss: 4.9351 Explore P: 0.7155\n",
      "Episode: 172 Total reward: 14.0 Training loss: 3.2709 Explore P: 0.7145\n",
      "Episode: 173 Total reward: 17.0 Training loss: 4.2177 Explore P: 0.7133\n",
      "Episode: 174 Total reward: 8.0 Training loss: 267.3999 Explore P: 0.7128\n",
      "Episode: 175 Total reward: 15.0 Training loss: 4.0816 Explore P: 0.7117\n",
      "Episode: 176 Total reward: 7.0 Training loss: 433.4531 Explore P: 0.7112\n",
      "Episode: 177 Total reward: 10.0 Training loss: 522.3325 Explore P: 0.7105\n",
      "Episode: 178 Total reward: 17.0 Training loss: 3.3731 Explore P: 0.7093\n",
      "Episode: 179 Total reward: 12.0 Training loss: 302.9075 Explore P: 0.7085\n",
      "Episode: 180 Total reward: 9.0 Training loss: 523.8109 Explore P: 0.7079\n",
      "Episode: 181 Total reward: 40.0 Training loss: 572.0511 Explore P: 0.7051\n",
      "Episode: 182 Total reward: 17.0 Training loss: 900.3434 Explore P: 0.7039\n",
      "Episode: 183 Total reward: 11.0 Training loss: 405.0062 Explore P: 0.7031\n",
      "Episode: 184 Total reward: 10.0 Training loss: 749.5383 Explore P: 0.7024\n",
      "Episode: 185 Total reward: 8.0 Training loss: 427.7527 Explore P: 0.7019\n",
      "Episode: 186 Total reward: 34.0 Training loss: 581.1356 Explore P: 0.6995\n",
      "Episode: 187 Total reward: 13.0 Training loss: 2.7497 Explore P: 0.6986\n",
      "Episode: 188 Total reward: 19.0 Training loss: 487.9606 Explore P: 0.6973\n",
      "Episode: 189 Total reward: 17.0 Training loss: 195.0292 Explore P: 0.6962\n",
      "Episode: 190 Total reward: 32.0 Training loss: 699.1814 Explore P: 0.6940\n",
      "Episode: 191 Total reward: 16.0 Training loss: 271.9941 Explore P: 0.6929\n",
      "Episode: 192 Total reward: 15.0 Training loss: 187.5189 Explore P: 0.6918\n",
      "Episode: 193 Total reward: 14.0 Training loss: 154.1180 Explore P: 0.6909\n",
      "Episode: 194 Total reward: 9.0 Training loss: 1.1168 Explore P: 0.6903\n",
      "Episode: 195 Total reward: 31.0 Training loss: 1.2763 Explore P: 0.6882\n",
      "Episode: 196 Total reward: 10.0 Training loss: 259.6856 Explore P: 0.6875\n",
      "Episode: 197 Total reward: 10.0 Training loss: 1.8168 Explore P: 0.6868\n",
      "Episode: 198 Total reward: 19.0 Training loss: 160.3634 Explore P: 0.6855\n",
      "Episode: 199 Total reward: 36.0 Training loss: 218.2130 Explore P: 0.6831\n",
      "Episode: 200 Total reward: 10.0 Training loss: 137.6283 Explore P: 0.6824\n",
      "Episode: 201 Total reward: 13.0 Training loss: 283.0449 Explore P: 0.6816\n",
      "Episode: 202 Total reward: 10.0 Training loss: 1.1330 Explore P: 0.6809\n",
      "Episode: 203 Total reward: 23.0 Training loss: 0.6072 Explore P: 0.6793\n",
      "Episode: 204 Total reward: 13.0 Training loss: 364.5831 Explore P: 0.6785\n",
      "Episode: 205 Total reward: 9.0 Training loss: 455.3069 Explore P: 0.6779\n",
      "Episode: 206 Total reward: 13.0 Training loss: 273.5376 Explore P: 0.6770\n",
      "Episode: 207 Total reward: 22.0 Training loss: 1.5652 Explore P: 0.6755\n",
      "Episode: 208 Total reward: 15.0 Training loss: 417.8747 Explore P: 0.6745\n",
      "Episode: 209 Total reward: 12.0 Training loss: 2.5200 Explore P: 0.6737\n",
      "Episode: 210 Total reward: 13.0 Training loss: 110.9124 Explore P: 0.6729\n",
      "Episode: 211 Total reward: 20.0 Training loss: 121.8840 Explore P: 0.6716\n",
      "Episode: 212 Total reward: 20.0 Training loss: 237.7953 Explore P: 0.6702\n",
      "Episode: 213 Total reward: 12.0 Training loss: 132.3007 Explore P: 0.6694\n",
      "Episode: 214 Total reward: 32.0 Training loss: 140.0244 Explore P: 0.6673\n",
      "Episode: 215 Total reward: 35.0 Training loss: 1.7330 Explore P: 0.6650\n",
      "Episode: 216 Total reward: 12.0 Training loss: 3.3954 Explore P: 0.6643\n",
      "Episode: 217 Total reward: 14.0 Training loss: 1.5465 Explore P: 0.6633\n",
      "Episode: 218 Total reward: 18.0 Training loss: 2.2789 Explore P: 0.6622\n",
      "Episode: 219 Total reward: 22.0 Training loss: 331.6791 Explore P: 0.6607\n",
      "Episode: 220 Total reward: 21.0 Training loss: 1.0279 Explore P: 0.6594\n",
      "Episode: 221 Total reward: 30.0 Training loss: 95.3715 Explore P: 0.6574\n",
      "Episode: 222 Total reward: 23.0 Training loss: 3.6763 Explore P: 0.6559\n",
      "Episode: 223 Total reward: 16.0 Training loss: 464.6882 Explore P: 0.6549\n",
      "Episode: 224 Total reward: 15.0 Training loss: 96.4195 Explore P: 0.6539\n",
      "Episode: 225 Total reward: 16.0 Training loss: 268.1480 Explore P: 0.6529\n",
      "Episode: 226 Total reward: 23.0 Training loss: 84.9793 Explore P: 0.6514\n",
      "Episode: 227 Total reward: 16.0 Training loss: 2.6304 Explore P: 0.6504\n",
      "Episode: 228 Total reward: 13.0 Training loss: 389.1419 Explore P: 0.6496\n",
      "Episode: 229 Total reward: 14.0 Training loss: 2.9917 Explore P: 0.6487\n",
      "Episode: 230 Total reward: 11.0 Training loss: 207.6588 Explore P: 0.6480\n",
      "Episode: 231 Total reward: 9.0 Training loss: 2.6556 Explore P: 0.6474\n",
      "Episode: 232 Total reward: 19.0 Training loss: 2.9240 Explore P: 0.6462\n",
      "Episode: 233 Total reward: 19.0 Training loss: 262.9503 Explore P: 0.6450\n",
      "Episode: 234 Total reward: 13.0 Training loss: 183.9673 Explore P: 0.6442\n",
      "Episode: 235 Total reward: 20.0 Training loss: 81.9151 Explore P: 0.6429\n",
      "Episode: 236 Total reward: 14.0 Training loss: 1.8160 Explore P: 0.6420\n",
      "Episode: 237 Total reward: 19.0 Training loss: 95.3797 Explore P: 0.6408\n",
      "Episode: 238 Total reward: 10.0 Training loss: 486.4511 Explore P: 0.6402\n",
      "Episode: 239 Total reward: 23.0 Training loss: 3.3972 Explore P: 0.6387\n",
      "Episode: 240 Total reward: 10.0 Training loss: 2.0964 Explore P: 0.6381\n",
      "Episode: 241 Total reward: 11.0 Training loss: 345.2578 Explore P: 0.6374\n",
      "Episode: 242 Total reward: 9.0 Training loss: 72.4309 Explore P: 0.6368\n",
      "Episode: 243 Total reward: 11.0 Training loss: 280.3143 Explore P: 0.6362\n",
      "Episode: 244 Total reward: 14.0 Training loss: 2.7181 Explore P: 0.6353\n",
      "Episode: 245 Total reward: 16.0 Training loss: 83.5908 Explore P: 0.6343\n",
      "Episode: 246 Total reward: 16.0 Training loss: 68.1096 Explore P: 0.6333\n",
      "Episode: 247 Total reward: 36.0 Training loss: 61.5836 Explore P: 0.6310\n",
      "Episode: 248 Total reward: 28.0 Training loss: 412.8983 Explore P: 0.6293\n",
      "Episode: 249 Total reward: 12.0 Training loss: 4.1262 Explore P: 0.6286\n",
      "Episode: 250 Total reward: 11.0 Training loss: 167.4614 Explore P: 0.6279\n",
      "Episode: 251 Total reward: 11.0 Training loss: 4.2562 Explore P: 0.6272\n",
      "Episode: 252 Total reward: 15.0 Training loss: 2.6961 Explore P: 0.6263\n",
      "Episode: 253 Total reward: 18.0 Training loss: 358.6351 Explore P: 0.6252\n",
      "Episode: 254 Total reward: 17.0 Training loss: 237.5923 Explore P: 0.6241\n",
      "Episode: 255 Total reward: 14.0 Training loss: 2.3208 Explore P: 0.6233\n",
      "Episode: 256 Total reward: 15.0 Training loss: 67.2018 Explore P: 0.6224\n",
      "Episode: 257 Total reward: 8.0 Training loss: 2.6910 Explore P: 0.6219\n",
      "Episode: 258 Total reward: 15.0 Training loss: 58.2869 Explore P: 0.6209\n",
      "Episode: 259 Total reward: 14.0 Training loss: 57.2243 Explore P: 0.6201\n",
      "Episode: 260 Total reward: 24.0 Training loss: 78.7014 Explore P: 0.6186\n",
      "Episode: 261 Total reward: 20.0 Training loss: 56.8823 Explore P: 0.6174\n",
      "Episode: 262 Total reward: 24.0 Training loss: 177.0263 Explore P: 0.6160\n",
      "Episode: 263 Total reward: 12.0 Training loss: 282.9177 Explore P: 0.6152\n",
      "Episode: 264 Total reward: 9.0 Training loss: 2.4736 Explore P: 0.6147\n",
      "Episode: 265 Total reward: 19.0 Training loss: 142.7500 Explore P: 0.6135\n",
      "Episode: 266 Total reward: 12.0 Training loss: 3.6883 Explore P: 0.6128\n",
      "Episode: 267 Total reward: 11.0 Training loss: 167.4109 Explore P: 0.6121\n",
      "Episode: 268 Total reward: 12.0 Training loss: 1.6738 Explore P: 0.6114\n",
      "Episode: 269 Total reward: 17.0 Training loss: 113.1308 Explore P: 0.6104\n",
      "Episode: 270 Total reward: 7.0 Training loss: 2.5818 Explore P: 0.6100\n",
      "Episode: 271 Total reward: 11.0 Training loss: 4.2090 Explore P: 0.6093\n",
      "Episode: 272 Total reward: 40.0 Training loss: 378.4833 Explore P: 0.6069\n",
      "Episode: 273 Total reward: 8.0 Training loss: 250.6063 Explore P: 0.6065\n",
      "Episode: 274 Total reward: 39.0 Training loss: 138.9474 Explore P: 0.6041\n",
      "Episode: 275 Total reward: 18.0 Training loss: 53.3104 Explore P: 0.6031\n",
      "Episode: 276 Total reward: 12.0 Training loss: 48.9397 Explore P: 0.6024\n",
      "Episode: 277 Total reward: 8.0 Training loss: 70.3168 Explore P: 0.6019\n",
      "Episode: 278 Total reward: 19.0 Training loss: 186.6997 Explore P: 0.6008\n",
      "Episode: 279 Total reward: 22.0 Training loss: 1.9437 Explore P: 0.5995\n",
      "Episode: 280 Total reward: 24.0 Training loss: 54.8813 Explore P: 0.5980\n",
      "Episode: 281 Total reward: 19.0 Training loss: 50.6372 Explore P: 0.5969\n",
      "Episode: 282 Total reward: 17.0 Training loss: 3.8286 Explore P: 0.5959\n",
      "Episode: 283 Total reward: 15.0 Training loss: 267.3663 Explore P: 0.5951\n",
      "Episode: 284 Total reward: 12.0 Training loss: 89.5654 Explore P: 0.5944\n",
      "Episode: 285 Total reward: 11.0 Training loss: 114.0610 Explore P: 0.5937\n",
      "Episode: 286 Total reward: 9.0 Training loss: 3.5915 Explore P: 0.5932\n",
      "Episode: 287 Total reward: 8.0 Training loss: 313.0368 Explore P: 0.5927\n",
      "Episode: 288 Total reward: 13.0 Training loss: 50.3348 Explore P: 0.5920\n",
      "Episode: 289 Total reward: 11.0 Training loss: 3.3593 Explore P: 0.5913\n",
      "Episode: 290 Total reward: 36.0 Training loss: 106.5861 Explore P: 0.5892\n",
      "Episode: 291 Total reward: 21.0 Training loss: 4.1860 Explore P: 0.5880\n",
      "Episode: 292 Total reward: 19.0 Training loss: 2.4395 Explore P: 0.5869\n",
      "Episode: 293 Total reward: 11.0 Training loss: 45.6613 Explore P: 0.5863\n",
      "Episode: 294 Total reward: 10.0 Training loss: 179.1921 Explore P: 0.5857\n",
      "Episode: 295 Total reward: 12.0 Training loss: 336.6937 Explore P: 0.5850\n",
      "Episode: 296 Total reward: 14.0 Training loss: 221.2842 Explore P: 0.5842\n",
      "Episode: 297 Total reward: 8.0 Training loss: 1.9554 Explore P: 0.5838\n",
      "Episode: 298 Total reward: 30.0 Training loss: 278.5043 Explore P: 0.5820\n",
      "Episode: 299 Total reward: 12.0 Training loss: 399.7641 Explore P: 0.5814\n",
      "Episode: 300 Total reward: 16.0 Training loss: 111.3053 Explore P: 0.5804\n",
      "Episode: 301 Total reward: 11.0 Training loss: 162.5264 Explore P: 0.5798\n",
      "Episode: 302 Total reward: 14.0 Training loss: 53.8997 Explore P: 0.5790\n",
      "Episode: 303 Total reward: 54.0 Training loss: 108.8789 Explore P: 0.5759\n",
      "Episode: 304 Total reward: 11.0 Training loss: 103.5848 Explore P: 0.5753\n",
      "Episode: 305 Total reward: 26.0 Training loss: 2.1672 Explore P: 0.5739\n",
      "Episode: 306 Total reward: 11.0 Training loss: 106.1764 Explore P: 0.5732\n",
      "Episode: 307 Total reward: 11.0 Training loss: 1.7677 Explore P: 0.5726\n",
      "Episode: 308 Total reward: 15.0 Training loss: 85.6900 Explore P: 0.5718\n",
      "Episode: 309 Total reward: 21.0 Training loss: 41.1935 Explore P: 0.5706\n",
      "Episode: 310 Total reward: 9.0 Training loss: 105.1112 Explore P: 0.5701\n",
      "Episode: 311 Total reward: 14.0 Training loss: 157.8550 Explore P: 0.5693\n",
      "Episode: 312 Total reward: 21.0 Training loss: 2.9083 Explore P: 0.5681\n",
      "Episode: 313 Total reward: 8.0 Training loss: 38.5931 Explore P: 0.5677\n",
      "Episode: 314 Total reward: 20.0 Training loss: 44.8664 Explore P: 0.5666\n",
      "Episode: 315 Total reward: 14.0 Training loss: 1.2447 Explore P: 0.5658\n",
      "Episode: 316 Total reward: 15.0 Training loss: 1.5927 Explore P: 0.5650\n",
      "Episode: 317 Total reward: 9.0 Training loss: 1.6893 Explore P: 0.5645\n",
      "Episode: 318 Total reward: 29.0 Training loss: 88.0043 Explore P: 0.5629\n",
      "Episode: 319 Total reward: 8.0 Training loss: 104.8135 Explore P: 0.5624\n",
      "Episode: 320 Total reward: 18.0 Training loss: 192.6552 Explore P: 0.5614\n",
      "Episode: 321 Total reward: 7.0 Training loss: 35.6935 Explore P: 0.5610\n",
      "Episode: 322 Total reward: 19.0 Training loss: 44.7621 Explore P: 0.5600\n",
      "Episode: 323 Total reward: 11.0 Training loss: 262.6119 Explore P: 0.5594\n",
      "Episode: 324 Total reward: 7.0 Training loss: 82.2801 Explore P: 0.5590\n",
      "Episode: 325 Total reward: 16.0 Training loss: 2.0420 Explore P: 0.5581\n",
      "Episode: 326 Total reward: 12.0 Training loss: 37.0463 Explore P: 0.5575\n",
      "Episode: 327 Total reward: 13.0 Training loss: 203.1851 Explore P: 0.5568\n",
      "Episode: 328 Total reward: 18.0 Training loss: 0.8738 Explore P: 0.5558\n",
      "Episode: 329 Total reward: 27.0 Training loss: 85.9069 Explore P: 0.5543\n",
      "Episode: 330 Total reward: 29.0 Training loss: 1.3546 Explore P: 0.5527\n",
      "Episode: 331 Total reward: 11.0 Training loss: 128.9667 Explore P: 0.5521\n",
      "Episode: 332 Total reward: 10.0 Training loss: 104.3462 Explore P: 0.5516\n",
      "Episode: 333 Total reward: 9.0 Training loss: 36.3565 Explore P: 0.5511\n",
      "Episode: 334 Total reward: 18.0 Training loss: 0.7923 Explore P: 0.5501\n",
      "Episode: 335 Total reward: 13.0 Training loss: 32.0873 Explore P: 0.5494\n",
      "Episode: 336 Total reward: 29.0 Training loss: 73.1142 Explore P: 0.5479\n",
      "Episode: 337 Total reward: 12.0 Training loss: 0.7434 Explore P: 0.5472\n",
      "Episode: 338 Total reward: 18.0 Training loss: 62.9670 Explore P: 0.5463\n",
      "Episode: 339 Total reward: 26.0 Training loss: 80.4303 Explore P: 0.5449\n",
      "Episode: 340 Total reward: 16.0 Training loss: 1.0483 Explore P: 0.5440\n",
      "Episode: 341 Total reward: 19.0 Training loss: 147.6502 Explore P: 0.5430\n",
      "Episode: 342 Total reward: 10.0 Training loss: 106.8434 Explore P: 0.5425\n",
      "Episode: 343 Total reward: 16.0 Training loss: 78.3666 Explore P: 0.5416\n",
      "Episode: 344 Total reward: 11.0 Training loss: 32.9769 Explore P: 0.5410\n",
      "Episode: 345 Total reward: 16.0 Training loss: 66.5822 Explore P: 0.5402\n",
      "Episode: 346 Total reward: 50.0 Training loss: 145.6688 Explore P: 0.5375\n",
      "Episode: 347 Total reward: 21.0 Training loss: 1.0879 Explore P: 0.5364\n",
      "Episode: 348 Total reward: 18.0 Training loss: 27.0650 Explore P: 0.5355\n",
      "Episode: 349 Total reward: 18.0 Training loss: 0.7867 Explore P: 0.5345\n",
      "Episode: 350 Total reward: 9.0 Training loss: 96.3209 Explore P: 0.5341\n",
      "Episode: 351 Total reward: 9.0 Training loss: 1.2901 Explore P: 0.5336\n",
      "Episode: 352 Total reward: 10.0 Training loss: 127.5168 Explore P: 0.5331\n",
      "Episode: 353 Total reward: 10.0 Training loss: 38.0660 Explore P: 0.5325\n",
      "Episode: 354 Total reward: 13.0 Training loss: 102.1831 Explore P: 0.5319\n",
      "Episode: 355 Total reward: 20.0 Training loss: 0.9833 Explore P: 0.5308\n",
      "Episode: 356 Total reward: 9.0 Training loss: 59.6338 Explore P: 0.5304\n",
      "Episode: 357 Total reward: 11.0 Training loss: 1.4511 Explore P: 0.5298\n",
      "Episode: 358 Total reward: 12.0 Training loss: 0.6920 Explore P: 0.5292\n",
      "Episode: 359 Total reward: 48.0 Training loss: 81.7271 Explore P: 0.5267\n",
      "Episode: 360 Total reward: 38.0 Training loss: 33.9836 Explore P: 0.5247\n",
      "Episode: 361 Total reward: 37.0 Training loss: 83.7257 Explore P: 0.5228\n",
      "Episode: 362 Total reward: 38.0 Training loss: 84.9528 Explore P: 0.5209\n",
      "Episode: 363 Total reward: 30.0 Training loss: 1.1789 Explore P: 0.5193\n",
      "Episode: 364 Total reward: 18.0 Training loss: 1.0339 Explore P: 0.5184\n",
      "Episode: 365 Total reward: 29.0 Training loss: 0.8092 Explore P: 0.5169\n",
      "Episode: 366 Total reward: 34.0 Training loss: 25.5196 Explore P: 0.5152\n",
      "Episode: 367 Total reward: 8.0 Training loss: 42.9606 Explore P: 0.5148\n",
      "Episode: 368 Total reward: 47.0 Training loss: 1.0119 Explore P: 0.5125\n",
      "Episode: 369 Total reward: 42.0 Training loss: 70.5263 Explore P: 0.5103\n",
      "Episode: 370 Total reward: 27.0 Training loss: 1.2157 Explore P: 0.5090\n",
      "Episode: 371 Total reward: 45.0 Training loss: 0.8964 Explore P: 0.5068\n",
      "Episode: 372 Total reward: 14.0 Training loss: 0.9347 Explore P: 0.5061\n",
      "Episode: 373 Total reward: 35.0 Training loss: 0.9991 Explore P: 0.5043\n",
      "Episode: 374 Total reward: 13.0 Training loss: 44.3948 Explore P: 0.5037\n",
      "Episode: 375 Total reward: 39.0 Training loss: 22.9768 Explore P: 0.5018\n",
      "Episode: 376 Total reward: 8.0 Training loss: 22.4249 Explore P: 0.5014\n",
      "Episode: 377 Total reward: 35.0 Training loss: 0.6364 Explore P: 0.4997\n",
      "Episode: 378 Total reward: 40.0 Training loss: 82.1884 Explore P: 0.4977\n",
      "Episode: 379 Total reward: 15.0 Training loss: 65.5622 Explore P: 0.4970\n",
      "Episode: 380 Total reward: 51.0 Training loss: 57.6388 Explore P: 0.4945\n",
      "Episode: 381 Total reward: 54.0 Training loss: 1.4940 Explore P: 0.4919\n",
      "Episode: 382 Total reward: 22.0 Training loss: 1.3177 Explore P: 0.4908\n",
      "Episode: 383 Total reward: 38.0 Training loss: 55.9053 Explore P: 0.4890\n",
      "Episode: 384 Total reward: 42.0 Training loss: 38.6219 Explore P: 0.4870\n",
      "Episode: 385 Total reward: 82.0 Training loss: 57.0660 Explore P: 0.4831\n",
      "Episode: 386 Total reward: 33.0 Training loss: 19.8700 Explore P: 0.4815\n",
      "Episode: 387 Total reward: 26.0 Training loss: 1.0353 Explore P: 0.4803\n",
      "Episode: 388 Total reward: 42.0 Training loss: 36.2395 Explore P: 0.4783\n",
      "Episode: 389 Total reward: 25.0 Training loss: 83.8595 Explore P: 0.4772\n",
      "Episode: 390 Total reward: 37.0 Training loss: 0.9979 Explore P: 0.4755\n",
      "Episode: 391 Total reward: 34.0 Training loss: 0.7327 Explore P: 0.4739\n",
      "Episode: 392 Total reward: 52.0 Training loss: 29.9125 Explore P: 0.4715\n",
      "Episode: 393 Total reward: 53.0 Training loss: 1.0168 Explore P: 0.4690\n",
      "Episode: 394 Total reward: 61.0 Training loss: 1.3195 Explore P: 0.4662\n",
      "Episode: 395 Total reward: 87.0 Training loss: 26.8216 Explore P: 0.4623\n",
      "Episode: 396 Total reward: 17.0 Training loss: 22.8861 Explore P: 0.4615\n",
      "Episode: 397 Total reward: 51.0 Training loss: 35.5281 Explore P: 0.4592\n",
      "Episode: 398 Total reward: 16.0 Training loss: 1.7109 Explore P: 0.4585\n",
      "Episode: 399 Total reward: 13.0 Training loss: 28.9884 Explore P: 0.4579\n",
      "Episode: 400 Total reward: 25.0 Training loss: 0.8127 Explore P: 0.4568\n",
      "Episode: 401 Total reward: 20.0 Training loss: 1.5500 Explore P: 0.4559\n",
      "Episode: 402 Total reward: 41.0 Training loss: 15.7173 Explore P: 0.4541\n",
      "Episode: 403 Total reward: 20.0 Training loss: 49.9284 Explore P: 0.4532\n",
      "Episode: 404 Total reward: 26.0 Training loss: 1.2352 Explore P: 0.4520\n",
      "Episode: 405 Total reward: 11.0 Training loss: 1.1612 Explore P: 0.4516\n",
      "Episode: 406 Total reward: 17.0 Training loss: 31.5828 Explore P: 0.4508\n",
      "Episode: 407 Total reward: 25.0 Training loss: 20.5275 Explore P: 0.4497\n",
      "Episode: 408 Total reward: 37.0 Training loss: 18.1349 Explore P: 0.4481\n",
      "Episode: 409 Total reward: 23.0 Training loss: 1.3305 Explore P: 0.4471\n",
      "Episode: 410 Total reward: 29.0 Training loss: 18.9927 Explore P: 0.4458\n",
      "Episode: 411 Total reward: 27.0 Training loss: 0.8984 Explore P: 0.4446\n",
      "Episode: 412 Total reward: 25.0 Training loss: 21.6949 Explore P: 0.4435\n",
      "Episode: 413 Total reward: 29.0 Training loss: 1.2728 Explore P: 0.4423\n",
      "Episode: 414 Total reward: 32.0 Training loss: 62.2934 Explore P: 0.4409\n",
      "Episode: 415 Total reward: 8.0 Training loss: 16.7112 Explore P: 0.4406\n",
      "Episode: 416 Total reward: 25.0 Training loss: 24.3713 Explore P: 0.4395\n",
      "Episode: 417 Total reward: 35.0 Training loss: 47.3013 Explore P: 0.4380\n",
      "Episode: 418 Total reward: 55.0 Training loss: 1.1424 Explore P: 0.4356\n",
      "Episode: 419 Total reward: 29.0 Training loss: 1.2561 Explore P: 0.4344\n",
      "Episode: 420 Total reward: 42.0 Training loss: 1.2293 Explore P: 0.4326\n",
      "Episode: 421 Total reward: 26.0 Training loss: 0.9028 Explore P: 0.4315\n",
      "Episode: 422 Total reward: 43.0 Training loss: 23.4430 Explore P: 0.4297\n",
      "Episode: 423 Total reward: 29.0 Training loss: 22.9777 Explore P: 0.4285\n",
      "Episode: 424 Total reward: 18.0 Training loss: 25.1310 Explore P: 0.4278\n",
      "Episode: 425 Total reward: 21.0 Training loss: 0.9499 Explore P: 0.4269\n",
      "Episode: 426 Total reward: 21.0 Training loss: 19.8288 Explore P: 0.4260\n",
      "Episode: 427 Total reward: 20.0 Training loss: 1.1400 Explore P: 0.4252\n",
      "Episode: 428 Total reward: 19.0 Training loss: 23.5838 Explore P: 0.4244\n",
      "Episode: 429 Total reward: 27.0 Training loss: 89.5189 Explore P: 0.4233\n",
      "Episode: 430 Total reward: 27.0 Training loss: 1.4634 Explore P: 0.4222\n",
      "Episode: 431 Total reward: 27.0 Training loss: 1.4292 Explore P: 0.4210\n",
      "Episode: 432 Total reward: 18.0 Training loss: 1.6106 Explore P: 0.4203\n",
      "Episode: 433 Total reward: 18.0 Training loss: 1.3220 Explore P: 0.4196\n",
      "Episode: 434 Total reward: 27.0 Training loss: 23.9467 Explore P: 0.4185\n",
      "Episode: 435 Total reward: 32.0 Training loss: 25.7374 Explore P: 0.4172\n",
      "Episode: 436 Total reward: 23.0 Training loss: 1.5011 Explore P: 0.4162\n",
      "Episode: 437 Total reward: 42.0 Training loss: 15.6091 Explore P: 0.4145\n",
      "Episode: 438 Total reward: 27.0 Training loss: 1.1469 Explore P: 0.4134\n",
      "Episode: 439 Total reward: 59.0 Training loss: 22.8234 Explore P: 0.4111\n",
      "Episode: 440 Total reward: 56.0 Training loss: 0.9005 Explore P: 0.4088\n",
      "Episode: 441 Total reward: 64.0 Training loss: 16.1679 Explore P: 0.4063\n",
      "Episode: 442 Total reward: 64.0 Training loss: 22.0376 Explore P: 0.4037\n",
      "Episode: 443 Total reward: 41.0 Training loss: 1.2990 Explore P: 0.4021\n",
      "Episode: 444 Total reward: 68.0 Training loss: 0.8202 Explore P: 0.3995\n",
      "Episode: 445 Total reward: 34.0 Training loss: 18.5038 Explore P: 0.3982\n",
      "Episode: 446 Total reward: 30.0 Training loss: 1.2250 Explore P: 0.3970\n",
      "Episode: 447 Total reward: 19.0 Training loss: 27.5777 Explore P: 0.3963\n",
      "Episode: 448 Total reward: 22.0 Training loss: 1.3147 Explore P: 0.3954\n",
      "Episode: 449 Total reward: 25.0 Training loss: 50.9282 Explore P: 0.3944\n",
      "Episode: 450 Total reward: 31.0 Training loss: 53.9935 Explore P: 0.3933\n",
      "Episode: 451 Total reward: 29.0 Training loss: 1.2416 Explore P: 0.3921\n",
      "Episode: 452 Total reward: 28.0 Training loss: 39.1863 Explore P: 0.3911\n",
      "Episode: 453 Total reward: 58.0 Training loss: 23.7471 Explore P: 0.3889\n",
      "Episode: 454 Total reward: 50.0 Training loss: 20.4076 Explore P: 0.3870\n",
      "Episode: 455 Total reward: 32.0 Training loss: 1.4278 Explore P: 0.3858\n",
      "Episode: 456 Total reward: 65.0 Training loss: 34.3291 Explore P: 0.3833\n",
      "Episode: 457 Total reward: 17.0 Training loss: 0.5701 Explore P: 0.3827\n",
      "Episode: 458 Total reward: 36.0 Training loss: 31.9603 Explore P: 0.3814\n",
      "Episode: 459 Total reward: 33.0 Training loss: 19.4942 Explore P: 0.3801\n",
      "Episode: 460 Total reward: 22.0 Training loss: 23.0968 Explore P: 0.3793\n",
      "Episode: 461 Total reward: 62.0 Training loss: 23.4517 Explore P: 0.3771\n",
      "Episode: 462 Total reward: 28.0 Training loss: 18.2170 Explore P: 0.3760\n",
      "Episode: 463 Total reward: 30.0 Training loss: 25.0702 Explore P: 0.3749\n",
      "Episode: 464 Total reward: 24.0 Training loss: 1.9379 Explore P: 0.3741\n",
      "Episode: 465 Total reward: 26.0 Training loss: 1.0605 Explore P: 0.3731\n",
      "Episode: 466 Total reward: 37.0 Training loss: 1.3814 Explore P: 0.3718\n",
      "Episode: 467 Total reward: 55.0 Training loss: 19.9304 Explore P: 0.3698\n",
      "Episode: 468 Total reward: 27.0 Training loss: 0.7960 Explore P: 0.3688\n",
      "Episode: 469 Total reward: 29.0 Training loss: 27.2494 Explore P: 0.3678\n",
      "Episode: 470 Total reward: 35.0 Training loss: 19.0630 Explore P: 0.3665\n",
      "Episode: 471 Total reward: 41.0 Training loss: 22.8252 Explore P: 0.3651\n",
      "Episode: 472 Total reward: 37.0 Training loss: 1.5749 Explore P: 0.3638\n",
      "Episode: 473 Total reward: 28.0 Training loss: 0.9512 Explore P: 0.3628\n",
      "Episode: 474 Total reward: 35.0 Training loss: 20.6571 Explore P: 0.3615\n",
      "Episode: 475 Total reward: 47.0 Training loss: 16.9415 Explore P: 0.3599\n",
      "Episode: 476 Total reward: 19.0 Training loss: 2.1100 Explore P: 0.3592\n",
      "Episode: 477 Total reward: 30.0 Training loss: 31.7205 Explore P: 0.3582\n",
      "Episode: 478 Total reward: 33.0 Training loss: 33.5464 Explore P: 0.3570\n",
      "Episode: 479 Total reward: 39.0 Training loss: 1.0887 Explore P: 0.3557\n",
      "Episode: 480 Total reward: 35.0 Training loss: 18.4948 Explore P: 0.3545\n",
      "Episode: 481 Total reward: 29.0 Training loss: 1.0467 Explore P: 0.3535\n",
      "Episode: 482 Total reward: 40.0 Training loss: 42.6965 Explore P: 0.3521\n",
      "Episode: 483 Total reward: 52.0 Training loss: 17.9966 Explore P: 0.3503\n",
      "Episode: 484 Total reward: 35.0 Training loss: 21.0122 Explore P: 0.3491\n",
      "Episode: 485 Total reward: 32.0 Training loss: 1.4691 Explore P: 0.3481\n",
      "Episode: 486 Total reward: 48.0 Training loss: 1.3654 Explore P: 0.3464\n",
      "Episode: 487 Total reward: 37.0 Training loss: 1.5789 Explore P: 0.3452\n",
      "Episode: 488 Total reward: 57.0 Training loss: 21.3630 Explore P: 0.3433\n",
      "Episode: 489 Total reward: 33.0 Training loss: 40.3593 Explore P: 0.3422\n",
      "Episode: 490 Total reward: 34.0 Training loss: 18.8797 Explore P: 0.3411\n",
      "Episode: 491 Total reward: 36.0 Training loss: 0.9968 Explore P: 0.3399\n",
      "Episode: 492 Total reward: 42.0 Training loss: 1.2780 Explore P: 0.3385\n",
      "Episode: 493 Total reward: 135.0 Training loss: 42.4456 Explore P: 0.3341\n",
      "Episode: 494 Total reward: 19.0 Training loss: 1.0760 Explore P: 0.3335\n",
      "Episode: 495 Total reward: 36.0 Training loss: 73.5273 Explore P: 0.3323\n",
      "Episode: 496 Total reward: 22.0 Training loss: 1.5610 Explore P: 0.3316\n",
      "Episode: 497 Total reward: 29.0 Training loss: 1.3910 Explore P: 0.3307\n",
      "Episode: 498 Total reward: 18.0 Training loss: 16.2554 Explore P: 0.3301\n",
      "Episode: 499 Total reward: 15.0 Training loss: 67.9745 Explore P: 0.3296\n",
      "Episode: 500 Total reward: 58.0 Training loss: 1.6888 Explore P: 0.3278\n",
      "Episode: 501 Total reward: 44.0 Training loss: 21.3107 Explore P: 0.3264\n",
      "Episode: 502 Total reward: 24.0 Training loss: 18.3489 Explore P: 0.3256\n",
      "Episode: 503 Total reward: 22.0 Training loss: 35.4163 Explore P: 0.3249\n",
      "Episode: 504 Total reward: 44.0 Training loss: 1.0608 Explore P: 0.3235\n",
      "Episode: 505 Total reward: 43.0 Training loss: 50.9477 Explore P: 0.3222\n",
      "Episode: 506 Total reward: 58.0 Training loss: 20.2964 Explore P: 0.3204\n",
      "Episode: 507 Total reward: 33.0 Training loss: 24.9407 Explore P: 0.3194\n",
      "Episode: 508 Total reward: 26.0 Training loss: 1.8895 Explore P: 0.3186\n",
      "Episode: 509 Total reward: 36.0 Training loss: 25.2277 Explore P: 0.3174\n",
      "Episode: 510 Total reward: 28.0 Training loss: 44.9352 Explore P: 0.3166\n",
      "Episode: 511 Total reward: 28.0 Training loss: 62.2346 Explore P: 0.3157\n",
      "Episode: 512 Total reward: 18.0 Training loss: 18.4215 Explore P: 0.3152\n",
      "Episode: 513 Total reward: 37.0 Training loss: 31.5702 Explore P: 0.3141\n",
      "Episode: 514 Total reward: 40.0 Training loss: 0.9515 Explore P: 0.3128\n",
      "Episode: 515 Total reward: 34.0 Training loss: 25.3898 Explore P: 0.3118\n",
      "Episode: 516 Total reward: 33.0 Training loss: 1.7531 Explore P: 0.3108\n",
      "Episode: 517 Total reward: 25.0 Training loss: 35.2445 Explore P: 0.3101\n",
      "Episode: 518 Total reward: 18.0 Training loss: 1.3253 Explore P: 0.3095\n",
      "Episode: 519 Total reward: 35.0 Training loss: 27.5180 Explore P: 0.3085\n",
      "Episode: 520 Total reward: 46.0 Training loss: 25.6032 Explore P: 0.3071\n",
      "Episode: 521 Total reward: 42.0 Training loss: 23.0536 Explore P: 0.3059\n",
      "Episode: 522 Total reward: 22.0 Training loss: 1.0892 Explore P: 0.3052\n",
      "Episode: 523 Total reward: 25.0 Training loss: 24.2875 Explore P: 0.3045\n",
      "Episode: 524 Total reward: 45.0 Training loss: 39.1339 Explore P: 0.3032\n",
      "Episode: 525 Total reward: 52.0 Training loss: 19.4110 Explore P: 0.3016\n",
      "Episode: 526 Total reward: 45.0 Training loss: 18.8509 Explore P: 0.3003\n",
      "Episode: 527 Total reward: 30.0 Training loss: 1.4485 Explore P: 0.2995\n",
      "Episode: 528 Total reward: 18.0 Training loss: 16.8004 Explore P: 0.2989\n",
      "Episode: 529 Total reward: 29.0 Training loss: 46.7495 Explore P: 0.2981\n",
      "Episode: 530 Total reward: 31.0 Training loss: 0.7873 Explore P: 0.2972\n",
      "Episode: 531 Total reward: 101.0 Training loss: 1.6255 Explore P: 0.2943\n",
      "Episode: 532 Total reward: 35.0 Training loss: 1.2068 Explore P: 0.2933\n",
      "Episode: 533 Total reward: 57.0 Training loss: 1.6491 Explore P: 0.2917\n",
      "Episode: 534 Total reward: 34.0 Training loss: 35.3905 Explore P: 0.2908\n",
      "Episode: 535 Total reward: 30.0 Training loss: 46.9522 Explore P: 0.2899\n",
      "Episode: 536 Total reward: 25.0 Training loss: 0.8124 Explore P: 0.2892\n",
      "Episode: 537 Total reward: 44.0 Training loss: 23.6348 Explore P: 0.2880\n",
      "Episode: 538 Total reward: 38.0 Training loss: 1.3445 Explore P: 0.2869\n",
      "Episode: 539 Total reward: 26.0 Training loss: 21.0853 Explore P: 0.2862\n",
      "Episode: 540 Total reward: 21.0 Training loss: 2.1627 Explore P: 0.2856\n",
      "Episode: 541 Total reward: 28.0 Training loss: 18.7570 Explore P: 0.2849\n",
      "Episode: 542 Total reward: 29.0 Training loss: 0.8870 Explore P: 0.2841\n",
      "Episode: 543 Total reward: 38.0 Training loss: 42.4302 Explore P: 0.2830\n",
      "Episode: 544 Total reward: 46.0 Training loss: 1.3761 Explore P: 0.2818\n",
      "Episode: 545 Total reward: 54.0 Training loss: 2.1377 Explore P: 0.2803\n",
      "Episode: 546 Total reward: 19.0 Training loss: 1.5087 Explore P: 0.2798\n",
      "Episode: 547 Total reward: 46.0 Training loss: 36.5652 Explore P: 0.2786\n",
      "Episode: 548 Total reward: 24.0 Training loss: 0.8248 Explore P: 0.2779\n",
      "Episode: 549 Total reward: 37.0 Training loss: 0.9270 Explore P: 0.2769\n",
      "Episode: 550 Total reward: 46.0 Training loss: 1.3072 Explore P: 0.2757\n",
      "Episode: 551 Total reward: 26.0 Training loss: 72.6987 Explore P: 0.2750\n",
      "Episode: 552 Total reward: 28.0 Training loss: 26.7740 Explore P: 0.2743\n",
      "Episode: 553 Total reward: 20.0 Training loss: 43.4947 Explore P: 0.2738\n",
      "Episode: 554 Total reward: 46.0 Training loss: 52.1265 Explore P: 0.2725\n",
      "Episode: 555 Total reward: 30.0 Training loss: 20.9509 Explore P: 0.2718\n",
      "Episode: 556 Total reward: 46.0 Training loss: 22.8541 Explore P: 0.2706\n",
      "Episode: 557 Total reward: 25.0 Training loss: 1.0911 Explore P: 0.2699\n",
      "Episode: 558 Total reward: 26.0 Training loss: 23.5392 Explore P: 0.2692\n",
      "Episode: 559 Total reward: 41.0 Training loss: 19.3796 Explore P: 0.2682\n",
      "Episode: 560 Total reward: 40.0 Training loss: 39.5729 Explore P: 0.2671\n",
      "Episode: 561 Total reward: 32.0 Training loss: 49.0634 Explore P: 0.2663\n",
      "Episode: 562 Total reward: 40.0 Training loss: 76.9064 Explore P: 0.2653\n",
      "Episode: 563 Total reward: 22.0 Training loss: 19.7384 Explore P: 0.2647\n",
      "Episode: 564 Total reward: 28.0 Training loss: 91.1711 Explore P: 0.2640\n",
      "Episode: 565 Total reward: 36.0 Training loss: 39.2367 Explore P: 0.2631\n",
      "Episode: 566 Total reward: 40.0 Training loss: 0.9988 Explore P: 0.2621\n",
      "Episode: 567 Total reward: 18.0 Training loss: 76.0809 Explore P: 0.2616\n",
      "Episode: 568 Total reward: 26.0 Training loss: 16.0721 Explore P: 0.2610\n",
      "Episode: 569 Total reward: 24.0 Training loss: 1.2279 Explore P: 0.2604\n",
      "Episode: 570 Total reward: 18.0 Training loss: 0.9556 Explore P: 0.2599\n",
      "Episode: 571 Total reward: 18.0 Training loss: 1.1712 Explore P: 0.2595\n",
      "Episode: 572 Total reward: 24.0 Training loss: 1.8211 Explore P: 0.2589\n",
      "Episode: 573 Total reward: 18.0 Training loss: 2.0726 Explore P: 0.2584\n",
      "Episode: 574 Total reward: 23.0 Training loss: 1.4410 Explore P: 0.2579\n",
      "Episode: 575 Total reward: 18.0 Training loss: 1.7357 Explore P: 0.2574\n",
      "Episode: 576 Total reward: 19.0 Training loss: 1.6698 Explore P: 0.2570\n",
      "Episode: 577 Total reward: 11.0 Training loss: 1.8265 Explore P: 0.2567\n",
      "Episode: 578 Total reward: 21.0 Training loss: 66.4207 Explore P: 0.2562\n",
      "Episode: 579 Total reward: 15.0 Training loss: 25.1615 Explore P: 0.2558\n",
      "Episode: 580 Total reward: 23.0 Training loss: 1.1918 Explore P: 0.2552\n",
      "Episode: 581 Total reward: 19.0 Training loss: 1.4638 Explore P: 0.2548\n",
      "Episode: 582 Total reward: 18.0 Training loss: 0.7955 Explore P: 0.2543\n",
      "Episode: 583 Total reward: 19.0 Training loss: 40.7752 Explore P: 0.2539\n",
      "Episode: 584 Total reward: 23.0 Training loss: 26.7561 Explore P: 0.2533\n",
      "Episode: 585 Total reward: 14.0 Training loss: 0.6219 Explore P: 0.2530\n",
      "Episode: 586 Total reward: 21.0 Training loss: 52.1995 Explore P: 0.2525\n",
      "Episode: 587 Total reward: 28.0 Training loss: 0.9996 Explore P: 0.2518\n",
      "Episode: 588 Total reward: 33.0 Training loss: 1.0518 Explore P: 0.2510\n",
      "Episode: 589 Total reward: 27.0 Training loss: 1.4760 Explore P: 0.2503\n",
      "Episode: 590 Total reward: 25.0 Training loss: 33.1751 Explore P: 0.2497\n",
      "Episode: 591 Total reward: 31.0 Training loss: 27.3014 Explore P: 0.2490\n",
      "Episode: 592 Total reward: 19.0 Training loss: 1.5002 Explore P: 0.2485\n",
      "Episode: 593 Total reward: 19.0 Training loss: 1.4766 Explore P: 0.2481\n",
      "Episode: 594 Total reward: 23.0 Training loss: 31.7838 Explore P: 0.2475\n",
      "Episode: 595 Total reward: 20.0 Training loss: 25.8651 Explore P: 0.2471\n",
      "Episode: 596 Total reward: 21.0 Training loss: 1.4600 Explore P: 0.2466\n",
      "Episode: 597 Total reward: 18.0 Training loss: 0.6337 Explore P: 0.2461\n",
      "Episode: 598 Total reward: 25.0 Training loss: 1.4827 Explore P: 0.2455\n",
      "Episode: 599 Total reward: 24.0 Training loss: 2.1649 Explore P: 0.2450\n",
      "Episode: 600 Total reward: 18.0 Training loss: 23.8040 Explore P: 0.2446\n",
      "Episode: 601 Total reward: 19.0 Training loss: 49.9423 Explore P: 0.2441\n",
      "Episode: 602 Total reward: 14.0 Training loss: 78.8876 Explore P: 0.2438\n",
      "Episode: 603 Total reward: 16.0 Training loss: 79.6696 Explore P: 0.2434\n",
      "Episode: 604 Total reward: 13.0 Training loss: 2.2016 Explore P: 0.2431\n",
      "Episode: 605 Total reward: 20.0 Training loss: 1.3412 Explore P: 0.2426\n",
      "Episode: 606 Total reward: 16.0 Training loss: 93.5501 Explore P: 0.2423\n",
      "Episode: 607 Total reward: 20.0 Training loss: 44.7606 Explore P: 0.2418\n",
      "Episode: 608 Total reward: 28.0 Training loss: 38.3460 Explore P: 0.2412\n",
      "Episode: 609 Total reward: 16.0 Training loss: 40.4234 Explore P: 0.2408\n",
      "Episode: 610 Total reward: 27.0 Training loss: 61.5312 Explore P: 0.2402\n",
      "Episode: 611 Total reward: 31.0 Training loss: 0.6938 Explore P: 0.2395\n",
      "Episode: 612 Total reward: 16.0 Training loss: 0.8128 Explore P: 0.2391\n",
      "Episode: 613 Total reward: 25.0 Training loss: 42.8754 Explore P: 0.2385\n",
      "Episode: 614 Total reward: 35.0 Training loss: 0.6934 Explore P: 0.2377\n",
      "Episode: 615 Total reward: 31.0 Training loss: 21.3010 Explore P: 0.2370\n",
      "Episode: 616 Total reward: 26.0 Training loss: 85.7812 Explore P: 0.2364\n",
      "Episode: 617 Total reward: 46.0 Training loss: 0.6208 Explore P: 0.2354\n",
      "Episode: 618 Total reward: 32.0 Training loss: 25.3474 Explore P: 0.2347\n",
      "Episode: 619 Total reward: 28.0 Training loss: 0.3254 Explore P: 0.2340\n",
      "Episode: 620 Total reward: 45.0 Training loss: 17.8128 Explore P: 0.2330\n",
      "Episode: 621 Total reward: 20.0 Training loss: 38.3392 Explore P: 0.2326\n",
      "Episode: 622 Total reward: 26.0 Training loss: 0.7935 Explore P: 0.2320\n",
      "Episode: 623 Total reward: 22.0 Training loss: 0.8867 Explore P: 0.2315\n",
      "Episode: 624 Total reward: 24.0 Training loss: 39.1497 Explore P: 0.2310\n",
      "Episode: 625 Total reward: 28.0 Training loss: 44.1385 Explore P: 0.2304\n",
      "Episode: 626 Total reward: 34.0 Training loss: 47.3142 Explore P: 0.2296\n",
      "Episode: 627 Total reward: 40.0 Training loss: 56.5513 Explore P: 0.2287\n",
      "Episode: 628 Total reward: 50.0 Training loss: 22.2728 Explore P: 0.2277\n",
      "Episode: 629 Total reward: 43.0 Training loss: 1.2227 Explore P: 0.2267\n",
      "Episode: 630 Total reward: 45.0 Training loss: 17.9078 Explore P: 0.2257\n",
      "Episode: 631 Total reward: 23.0 Training loss: 38.2566 Explore P: 0.2253\n",
      "Episode: 632 Total reward: 31.0 Training loss: 15.1991 Explore P: 0.2246\n",
      "Episode: 633 Total reward: 24.0 Training loss: 15.0953 Explore P: 0.2241\n",
      "Episode: 634 Total reward: 29.0 Training loss: 15.4342 Explore P: 0.2235\n",
      "Episode: 635 Total reward: 35.0 Training loss: 0.5127 Explore P: 0.2227\n",
      "Episode: 636 Total reward: 26.0 Training loss: 14.6345 Explore P: 0.2222\n",
      "Episode: 637 Total reward: 35.0 Training loss: 13.9279 Explore P: 0.2214\n",
      "Episode: 638 Total reward: 11.0 Training loss: 1.2687 Explore P: 0.2212\n",
      "Episode: 639 Total reward: 22.0 Training loss: 0.6229 Explore P: 0.2207\n",
      "Episode: 640 Total reward: 26.0 Training loss: 1.1103 Explore P: 0.2202\n",
      "Episode: 641 Total reward: 38.0 Training loss: 13.2008 Explore P: 0.2194\n",
      "Episode: 642 Total reward: 41.0 Training loss: 40.0415 Explore P: 0.2185\n",
      "Episode: 643 Total reward: 58.0 Training loss: 1.1507 Explore P: 0.2173\n",
      "Episode: 644 Total reward: 57.0 Training loss: 1.9824 Explore P: 0.2161\n",
      "Episode: 645 Total reward: 44.0 Training loss: 49.2798 Explore P: 0.2152\n",
      "Episode: 646 Total reward: 33.0 Training loss: 1.0778 Explore P: 0.2145\n",
      "Episode: 647 Total reward: 32.0 Training loss: 0.7314 Explore P: 0.2139\n",
      "Episode: 648 Total reward: 37.0 Training loss: 32.9267 Explore P: 0.2131\n",
      "Episode: 649 Total reward: 34.0 Training loss: 23.5427 Explore P: 0.2125\n",
      "Episode: 650 Total reward: 43.0 Training loss: 0.8740 Explore P: 0.2116\n",
      "Episode: 651 Total reward: 35.0 Training loss: 27.8446 Explore P: 0.2109\n",
      "Episode: 652 Total reward: 50.0 Training loss: 14.5814 Explore P: 0.2099\n",
      "Episode: 653 Total reward: 56.0 Training loss: 0.5734 Explore P: 0.2088\n",
      "Episode: 654 Total reward: 49.0 Training loss: 29.6347 Explore P: 0.2078\n",
      "Episode: 655 Total reward: 72.0 Training loss: 11.9251 Explore P: 0.2064\n",
      "Episode: 656 Total reward: 48.0 Training loss: 10.8668 Explore P: 0.2054\n",
      "Episode: 657 Total reward: 79.0 Training loss: 21.9118 Explore P: 0.2039\n",
      "Episode: 658 Total reward: 84.0 Training loss: 30.9236 Explore P: 0.2023\n",
      "Episode: 659 Total reward: 50.0 Training loss: 0.9119 Explore P: 0.2013\n",
      "Episode: 660 Total reward: 92.0 Training loss: 16.6190 Explore P: 0.1996\n",
      "Episode: 661 Total reward: 115.0 Training loss: 0.4198 Explore P: 0.1974\n",
      "Episode: 662 Total reward: 183.0 Training loss: 0.4887 Explore P: 0.1940\n",
      "Episode: 663 Total reward: 199.0 Training loss: 15.8044 Explore P: 0.1904\n",
      "Episode: 664 Total reward: 199.0 Training loss: 19.8614 Explore P: 0.1868\n",
      "Episode: 665 Total reward: 199.0 Training loss: 20.1617 Explore P: 0.1833\n",
      "Episode: 666 Total reward: 135.0 Training loss: 0.4613 Explore P: 0.1810\n",
      "Episode: 667 Total reward: 115.0 Training loss: 0.3214 Explore P: 0.1791\n",
      "Episode: 668 Total reward: 141.0 Training loss: 0.5214 Explore P: 0.1767\n",
      "Episode: 669 Total reward: 42.0 Training loss: 0.9824 Explore P: 0.1760\n",
      "Episode: 670 Total reward: 46.0 Training loss: 1.0302 Explore P: 0.1752\n",
      "Episode: 671 Total reward: 18.0 Training loss: 18.2530 Explore P: 0.1749\n",
      "Episode: 672 Total reward: 29.0 Training loss: 0.9339 Explore P: 0.1744\n",
      "Episode: 673 Total reward: 53.0 Training loss: 0.9333 Explore P: 0.1736\n",
      "Episode: 674 Total reward: 44.0 Training loss: 32.0637 Explore P: 0.1729\n",
      "Episode: 675 Total reward: 28.0 Training loss: 0.8321 Explore P: 0.1724\n",
      "Episode: 676 Total reward: 17.0 Training loss: 14.6811 Explore P: 0.1721\n",
      "Episode: 677 Total reward: 19.0 Training loss: 0.9758 Explore P: 0.1718\n",
      "Episode: 678 Total reward: 17.0 Training loss: 0.7562 Explore P: 0.1715\n",
      "Episode: 679 Total reward: 15.0 Training loss: 10.3573 Explore P: 0.1713\n",
      "Episode: 680 Total reward: 25.0 Training loss: 12.7584 Explore P: 0.1709\n",
      "Episode: 681 Total reward: 22.0 Training loss: 19.8228 Explore P: 0.1705\n",
      "Episode: 682 Total reward: 25.0 Training loss: 16.0152 Explore P: 0.1701\n",
      "Episode: 683 Total reward: 23.0 Training loss: 0.5854 Explore P: 0.1698\n",
      "Episode: 684 Total reward: 19.0 Training loss: 0.8901 Explore P: 0.1695\n",
      "Episode: 685 Total reward: 17.0 Training loss: 71.5846 Explore P: 0.1692\n",
      "Episode: 686 Total reward: 18.0 Training loss: 1.0155 Explore P: 0.1689\n",
      "Episode: 687 Total reward: 17.0 Training loss: 60.3700 Explore P: 0.1687\n",
      "Episode: 688 Total reward: 16.0 Training loss: 0.5466 Explore P: 0.1684\n",
      "Episode: 689 Total reward: 22.0 Training loss: 0.7459 Explore P: 0.1680\n",
      "Episode: 690 Total reward: 17.0 Training loss: 16.4315 Explore P: 0.1678\n",
      "Episode: 691 Total reward: 18.0 Training loss: 25.2215 Explore P: 0.1675\n",
      "Episode: 692 Total reward: 21.0 Training loss: 1.3556 Explore P: 0.1672\n",
      "Episode: 693 Total reward: 17.0 Training loss: 14.3043 Explore P: 0.1669\n",
      "Episode: 694 Total reward: 14.0 Training loss: 53.4511 Explore P: 0.1667\n",
      "Episode: 695 Total reward: 14.0 Training loss: 1.2496 Explore P: 0.1665\n",
      "Episode: 696 Total reward: 20.0 Training loss: 0.8535 Explore P: 0.1661\n",
      "Episode: 697 Total reward: 14.0 Training loss: 28.2897 Explore P: 0.1659\n",
      "Episode: 698 Total reward: 10.0 Training loss: 0.4379 Explore P: 0.1658\n",
      "Episode: 699 Total reward: 21.0 Training loss: 26.1294 Explore P: 0.1654\n",
      "Episode: 700 Total reward: 21.0 Training loss: 0.9142 Explore P: 0.1651\n",
      "Episode: 701 Total reward: 20.0 Training loss: 16.6795 Explore P: 0.1648\n",
      "Episode: 702 Total reward: 17.0 Training loss: 0.5950 Explore P: 0.1645\n",
      "Episode: 703 Total reward: 19.0 Training loss: 0.7282 Explore P: 0.1643\n",
      "Episode: 704 Total reward: 19.0 Training loss: 0.7450 Explore P: 0.1640\n",
      "Episode: 705 Total reward: 17.0 Training loss: 19.3063 Explore P: 0.1637\n",
      "Episode: 706 Total reward: 13.0 Training loss: 1.7498 Explore P: 0.1635\n",
      "Episode: 707 Total reward: 11.0 Training loss: 42.9421 Explore P: 0.1633\n",
      "Episode: 708 Total reward: 16.0 Training loss: 1.6005 Explore P: 0.1631\n",
      "Episode: 709 Total reward: 13.0 Training loss: 18.3835 Explore P: 0.1629\n",
      "Episode: 710 Total reward: 13.0 Training loss: 0.8406 Explore P: 0.1627\n",
      "Episode: 711 Total reward: 15.0 Training loss: 1.4787 Explore P: 0.1625\n",
      "Episode: 712 Total reward: 15.0 Training loss: 54.5316 Explore P: 0.1622\n",
      "Episode: 713 Total reward: 21.0 Training loss: 29.0448 Explore P: 0.1619\n",
      "Episode: 714 Total reward: 22.0 Training loss: 38.4474 Explore P: 0.1616\n",
      "Episode: 715 Total reward: 17.0 Training loss: 1.0946 Explore P: 0.1613\n",
      "Episode: 716 Total reward: 15.0 Training loss: 42.5911 Explore P: 0.1611\n",
      "Episode: 717 Total reward: 15.0 Training loss: 114.6681 Explore P: 0.1609\n",
      "Episode: 718 Total reward: 14.0 Training loss: 2.4593 Explore P: 0.1607\n",
      "Episode: 719 Total reward: 15.0 Training loss: 3.3141 Explore P: 0.1604\n",
      "Episode: 720 Total reward: 25.0 Training loss: 241.2822 Explore P: 0.1601\n",
      "Episode: 721 Total reward: 12.0 Training loss: 24.2723 Explore P: 0.1599\n",
      "Episode: 722 Total reward: 13.0 Training loss: 24.2225 Explore P: 0.1597\n",
      "Episode: 723 Total reward: 18.0 Training loss: 29.7402 Explore P: 0.1594\n",
      "Episode: 724 Total reward: 14.0 Training loss: 1.0003 Explore P: 0.1592\n",
      "Episode: 725 Total reward: 16.0 Training loss: 228.5476 Explore P: 0.1590\n",
      "Episode: 726 Total reward: 17.0 Training loss: 2.8136 Explore P: 0.1587\n",
      "Episode: 727 Total reward: 15.0 Training loss: 197.4700 Explore P: 0.1585\n",
      "Episode: 728 Total reward: 11.0 Training loss: 177.5511 Explore P: 0.1583\n",
      "Episode: 729 Total reward: 15.0 Training loss: 10.5528 Explore P: 0.1581\n",
      "Episode: 730 Total reward: 15.0 Training loss: 24.3793 Explore P: 0.1579\n",
      "Episode: 731 Total reward: 19.0 Training loss: 17.3759 Explore P: 0.1576\n",
      "Episode: 732 Total reward: 19.0 Training loss: 3.9532 Explore P: 0.1573\n",
      "Episode: 733 Total reward: 23.0 Training loss: 29.1545 Explore P: 0.1570\n",
      "Episode: 734 Total reward: 12.0 Training loss: 29.7964 Explore P: 0.1568\n",
      "Episode: 735 Total reward: 14.0 Training loss: 1.3759 Explore P: 0.1566\n",
      "Episode: 736 Total reward: 22.0 Training loss: 26.4411 Explore P: 0.1563\n",
      "Episode: 737 Total reward: 25.0 Training loss: 259.2937 Explore P: 0.1559\n",
      "Episode: 738 Total reward: 30.0 Training loss: 18.9500 Explore P: 0.1555\n",
      "Episode: 739 Total reward: 29.0 Training loss: 1.5472 Explore P: 0.1551\n",
      "Episode: 740 Total reward: 30.0 Training loss: 0.4419 Explore P: 0.1546\n",
      "Episode: 741 Total reward: 26.0 Training loss: 2.2502 Explore P: 0.1542\n",
      "Episode: 742 Total reward: 17.0 Training loss: 1.3902 Explore P: 0.1540\n",
      "Episode: 743 Total reward: 20.0 Training loss: 1.4723 Explore P: 0.1537\n",
      "Episode: 744 Total reward: 17.0 Training loss: 1.0833 Explore P: 0.1535\n",
      "Episode: 745 Total reward: 20.0 Training loss: 0.3909 Explore P: 0.1532\n",
      "Episode: 746 Total reward: 30.0 Training loss: 38.3354 Explore P: 0.1528\n",
      "Episode: 747 Total reward: 19.0 Training loss: 175.9782 Explore P: 0.1525\n",
      "Episode: 748 Total reward: 24.0 Training loss: 3.3926 Explore P: 0.1521\n",
      "Episode: 749 Total reward: 16.0 Training loss: 28.9747 Explore P: 0.1519\n",
      "Episode: 750 Total reward: 17.0 Training loss: 0.5379 Explore P: 0.1517\n",
      "Episode: 751 Total reward: 20.0 Training loss: 1.6322 Explore P: 0.1514\n",
      "Episode: 752 Total reward: 23.0 Training loss: 24.0555 Explore P: 0.1511\n",
      "Episode: 753 Total reward: 18.0 Training loss: 14.2390 Explore P: 0.1508\n",
      "Episode: 754 Total reward: 23.0 Training loss: 145.8319 Explore P: 0.1505\n",
      "Episode: 755 Total reward: 19.0 Training loss: 2.2094 Explore P: 0.1502\n",
      "Episode: 756 Total reward: 20.0 Training loss: 0.3719 Explore P: 0.1499\n",
      "Episode: 757 Total reward: 19.0 Training loss: 202.4566 Explore P: 0.1497\n",
      "Episode: 758 Total reward: 27.0 Training loss: 1.3490 Explore P: 0.1493\n",
      "Episode: 759 Total reward: 28.0 Training loss: 5.1101 Explore P: 0.1489\n",
      "Episode: 760 Total reward: 29.0 Training loss: 2.0159 Explore P: 0.1485\n",
      "Episode: 761 Total reward: 17.0 Training loss: 157.2063 Explore P: 0.1483\n",
      "Episode: 762 Total reward: 20.0 Training loss: 0.6353 Explore P: 0.1480\n",
      "Episode: 763 Total reward: 29.0 Training loss: 1.4448 Explore P: 0.1476\n",
      "Episode: 764 Total reward: 34.0 Training loss: 138.8495 Explore P: 0.1471\n",
      "Episode: 765 Total reward: 62.0 Training loss: 13.8135 Explore P: 0.1463\n",
      "Episode: 766 Total reward: 46.0 Training loss: 1.7427 Explore P: 0.1457\n",
      "Episode: 767 Total reward: 45.0 Training loss: 3.0071 Explore P: 0.1450\n",
      "Episode: 768 Total reward: 75.0 Training loss: 9.2709 Explore P: 0.1440\n",
      "Episode: 769 Total reward: 93.0 Training loss: 143.8115 Explore P: 0.1428\n",
      "Episode: 770 Total reward: 53.0 Training loss: 0.3759 Explore P: 0.1421\n",
      "Episode: 771 Total reward: 75.0 Training loss: 4.1389 Explore P: 0.1411\n",
      "Episode: 772 Total reward: 108.0 Training loss: 2.4484 Explore P: 0.1397\n",
      "Episode: 773 Total reward: 175.0 Training loss: 2.7009 Explore P: 0.1374\n",
      "Episode: 774 Total reward: 177.0 Training loss: 0.9763 Explore P: 0.1352\n",
      "Episode: 775 Total reward: 75.0 Training loss: 2.1757 Explore P: 0.1343\n",
      "Episode: 776 Total reward: 65.0 Training loss: 2.6825 Explore P: 0.1335\n",
      "Episode: 777 Total reward: 76.0 Training loss: 2.8936 Explore P: 0.1325\n",
      "Episode: 778 Total reward: 52.0 Training loss: 289.3327 Explore P: 0.1319\n",
      "Episode: 779 Total reward: 40.0 Training loss: 115.7544 Explore P: 0.1314\n",
      "Episode: 780 Total reward: 32.0 Training loss: 1.8383 Explore P: 0.1310\n",
      "Episode: 781 Total reward: 37.0 Training loss: 10.8902 Explore P: 0.1306\n",
      "Episode: 782 Total reward: 28.0 Training loss: 1.1046 Explore P: 0.1302\n",
      "Episode: 783 Total reward: 54.0 Training loss: 1.2954 Explore P: 0.1296\n",
      "Episode: 784 Total reward: 79.0 Training loss: 11.6392 Explore P: 0.1287\n",
      "Episode: 785 Total reward: 99.0 Training loss: 2.3639 Explore P: 0.1275\n",
      "Episode: 786 Total reward: 199.0 Training loss: 90.7695 Explore P: 0.1252\n",
      "Episode: 787 Total reward: 104.0 Training loss: 1.6586 Explore P: 0.1240\n",
      "Episode: 788 Total reward: 106.0 Training loss: 2.1569 Explore P: 0.1228\n",
      "Episode: 789 Total reward: 91.0 Training loss: 8.9578 Explore P: 0.1218\n",
      "Episode: 790 Total reward: 71.0 Training loss: 8.6505 Explore P: 0.1210\n",
      "Episode: 791 Total reward: 75.0 Training loss: 1.7379 Explore P: 0.1201\n",
      "Episode: 792 Total reward: 130.0 Training loss: 76.4648 Explore P: 0.1187\n",
      "Episode: 793 Total reward: 199.0 Training loss: 2.1482 Explore P: 0.1166\n",
      "Episode: 794 Total reward: 107.0 Training loss: 426.9602 Explore P: 0.1154\n",
      "Episode: 795 Total reward: 110.0 Training loss: 3.2026 Explore P: 0.1143\n",
      "Episode: 796 Total reward: 154.0 Training loss: 83.0835 Explore P: 0.1127\n",
      "Episode: 797 Total reward: 165.0 Training loss: 1.6075 Explore P: 0.1110\n",
      "Episode: 798 Total reward: 199.0 Training loss: 0.6734 Explore P: 0.1090\n",
      "Episode: 799 Total reward: 173.0 Training loss: 1.8273 Explore P: 0.1073\n",
      "Episode: 800 Total reward: 149.0 Training loss: 1.2950 Explore P: 0.1059\n",
      "Episode: 801 Total reward: 185.0 Training loss: 2.0001 Explore P: 0.1041\n",
      "Episode: 802 Total reward: 127.0 Training loss: 1.7598 Explore P: 0.1029\n",
      "Episode: 803 Total reward: 110.0 Training loss: 107.9991 Explore P: 0.1019\n",
      "Episode: 804 Total reward: 108.0 Training loss: 101.6628 Explore P: 0.1009\n",
      "Episode: 805 Total reward: 199.0 Training loss: 1.7500 Explore P: 0.0991\n",
      "Episode: 806 Total reward: 113.0 Training loss: 0.7946 Explore P: 0.0981\n",
      "Episode: 807 Total reward: 199.0 Training loss: 1.3275 Explore P: 0.0964\n",
      "Episode: 808 Total reward: 136.0 Training loss: 1.7107 Explore P: 0.0952\n",
      "Episode: 809 Total reward: 73.0 Training loss: 2.2424 Explore P: 0.0946\n",
      "Episode: 810 Total reward: 185.0 Training loss: 1.2792 Explore P: 0.0931\n",
      "Episode: 811 Total reward: 138.0 Training loss: 317.9551 Explore P: 0.0919\n",
      "Episode: 812 Total reward: 199.0 Training loss: 1.7319 Explore P: 0.0903\n",
      "Episode: 813 Total reward: 161.0 Training loss: 91.0721 Explore P: 0.0890\n",
      "Episode: 814 Total reward: 78.0 Training loss: 2.4586 Explore P: 0.0884\n",
      "Episode: 815 Total reward: 61.0 Training loss: 134.8788 Explore P: 0.0879\n",
      "Episode: 816 Total reward: 86.0 Training loss: 1.4940 Explore P: 0.0873\n",
      "Episode: 817 Total reward: 55.0 Training loss: 1.4453 Explore P: 0.0868\n",
      "Episode: 818 Total reward: 62.0 Training loss: 1.4897 Explore P: 0.0864\n",
      "Episode: 819 Total reward: 38.0 Training loss: 1.9849 Explore P: 0.0861\n",
      "Episode: 820 Total reward: 56.0 Training loss: 86.0620 Explore P: 0.0857\n",
      "Episode: 821 Total reward: 84.0 Training loss: 3.4532 Explore P: 0.0850\n",
      "Episode: 822 Total reward: 32.0 Training loss: 2.2645 Explore P: 0.0848\n",
      "Episode: 823 Total reward: 30.0 Training loss: 2.0489 Explore P: 0.0846\n",
      "Episode: 824 Total reward: 29.0 Training loss: 126.5050 Explore P: 0.0843\n",
      "Episode: 825 Total reward: 38.0 Training loss: 102.5296 Explore P: 0.0841\n",
      "Episode: 826 Total reward: 31.0 Training loss: 1.7673 Explore P: 0.0838\n",
      "Episode: 827 Total reward: 32.0 Training loss: 2.0744 Explore P: 0.0836\n",
      "Episode: 828 Total reward: 39.0 Training loss: 2.9873 Explore P: 0.0833\n",
      "Episode: 829 Total reward: 47.0 Training loss: 2.8640 Explore P: 0.0830\n",
      "Episode: 830 Total reward: 26.0 Training loss: 1.0232 Explore P: 0.0828\n",
      "Episode: 831 Total reward: 43.0 Training loss: 1.7200 Explore P: 0.0825\n",
      "Episode: 832 Total reward: 47.0 Training loss: 2.1147 Explore P: 0.0821\n",
      "Episode: 833 Total reward: 36.0 Training loss: 118.2283 Explore P: 0.0819\n",
      "Episode: 834 Total reward: 31.0 Training loss: 99.3558 Explore P: 0.0816\n",
      "Episode: 835 Total reward: 35.0 Training loss: 2.3564 Explore P: 0.0814\n",
      "Episode: 836 Total reward: 51.0 Training loss: 3.2684 Explore P: 0.0810\n",
      "Episode: 837 Total reward: 35.0 Training loss: 3.0176 Explore P: 0.0808\n",
      "Episode: 838 Total reward: 32.0 Training loss: 50.8539 Explore P: 0.0806\n",
      "Episode: 839 Total reward: 31.0 Training loss: 109.3862 Explore P: 0.0803\n",
      "Episode: 840 Total reward: 23.0 Training loss: 116.9963 Explore P: 0.0802\n",
      "Episode: 841 Total reward: 46.0 Training loss: 1.1246 Explore P: 0.0799\n",
      "Episode: 842 Total reward: 31.0 Training loss: 96.5281 Explore P: 0.0796\n",
      "Episode: 843 Total reward: 25.0 Training loss: 3.2992 Explore P: 0.0795\n",
      "Episode: 844 Total reward: 42.0 Training loss: 2.4680 Explore P: 0.0792\n",
      "Episode: 845 Total reward: 28.0 Training loss: 2.7376 Explore P: 0.0790\n",
      "Episode: 846 Total reward: 29.0 Training loss: 82.8647 Explore P: 0.0788\n",
      "Episode: 847 Total reward: 22.0 Training loss: 1.9221 Explore P: 0.0786\n",
      "Episode: 848 Total reward: 27.0 Training loss: 198.2552 Explore P: 0.0784\n",
      "Episode: 849 Total reward: 39.0 Training loss: 2.5600 Explore P: 0.0782\n",
      "Episode: 850 Total reward: 23.0 Training loss: 2.9519 Explore P: 0.0780\n",
      "Episode: 851 Total reward: 27.0 Training loss: 1.7459 Explore P: 0.0778\n",
      "Episode: 852 Total reward: 23.0 Training loss: 2.8528 Explore P: 0.0777\n",
      "Episode: 853 Total reward: 20.0 Training loss: 2.3178 Explore P: 0.0775\n",
      "Episode: 854 Total reward: 25.0 Training loss: 0.9997 Explore P: 0.0774\n",
      "Episode: 855 Total reward: 22.0 Training loss: 3.6184 Explore P: 0.0772\n",
      "Episode: 856 Total reward: 23.0 Training loss: 141.1686 Explore P: 0.0771\n",
      "Episode: 857 Total reward: 25.0 Training loss: 2.2411 Explore P: 0.0769\n",
      "Episode: 858 Total reward: 25.0 Training loss: 280.0510 Explore P: 0.0767\n",
      "Episode: 859 Total reward: 28.0 Training loss: 1.7384 Explore P: 0.0766\n",
      "Episode: 860 Total reward: 31.0 Training loss: 2.6077 Explore P: 0.0763\n",
      "Episode: 861 Total reward: 26.0 Training loss: 2.3632 Explore P: 0.0762\n",
      "Episode: 862 Total reward: 32.0 Training loss: 1.8722 Explore P: 0.0760\n",
      "Episode: 863 Total reward: 38.0 Training loss: 101.2295 Explore P: 0.0757\n",
      "Episode: 864 Total reward: 35.0 Training loss: 1.4889 Explore P: 0.0755\n",
      "Episode: 865 Total reward: 32.0 Training loss: 114.0449 Explore P: 0.0753\n",
      "Episode: 866 Total reward: 26.0 Training loss: 2.8172 Explore P: 0.0751\n",
      "Episode: 867 Total reward: 26.0 Training loss: 120.6029 Explore P: 0.0749\n",
      "Episode: 868 Total reward: 31.0 Training loss: 1.5438 Explore P: 0.0747\n",
      "Episode: 869 Total reward: 31.0 Training loss: 132.1862 Explore P: 0.0745\n",
      "Episode: 870 Total reward: 27.0 Training loss: 2.5176 Explore P: 0.0744\n",
      "Episode: 871 Total reward: 30.0 Training loss: 3.5129 Explore P: 0.0742\n",
      "Episode: 872 Total reward: 36.0 Training loss: 3.2646 Explore P: 0.0739\n",
      "Episode: 873 Total reward: 23.0 Training loss: 124.7652 Explore P: 0.0738\n",
      "Episode: 874 Total reward: 22.0 Training loss: 131.3420 Explore P: 0.0737\n",
      "Episode: 875 Total reward: 30.0 Training loss: 78.8095 Explore P: 0.0735\n",
      "Episode: 876 Total reward: 23.0 Training loss: 3.3635 Explore P: 0.0733\n",
      "Episode: 877 Total reward: 28.0 Training loss: 2.8845 Explore P: 0.0731\n",
      "Episode: 878 Total reward: 16.0 Training loss: 101.6937 Explore P: 0.0730\n",
      "Episode: 879 Total reward: 22.0 Training loss: 109.8820 Explore P: 0.0729\n",
      "Episode: 880 Total reward: 24.0 Training loss: 71.9479 Explore P: 0.0727\n",
      "Episode: 881 Total reward: 19.0 Training loss: 71.0419 Explore P: 0.0726\n",
      "Episode: 882 Total reward: 21.0 Training loss: 2.8485 Explore P: 0.0725\n",
      "Episode: 883 Total reward: 19.0 Training loss: 1.7249 Explore P: 0.0724\n",
      "Episode: 884 Total reward: 47.0 Training loss: 101.8414 Explore P: 0.0721\n",
      "Episode: 885 Total reward: 23.0 Training loss: 87.9634 Explore P: 0.0719\n",
      "Episode: 886 Total reward: 21.0 Training loss: 347.4538 Explore P: 0.0718\n",
      "Episode: 887 Total reward: 25.0 Training loss: 107.2089 Explore P: 0.0717\n",
      "Episode: 888 Total reward: 24.0 Training loss: 4.6952 Explore P: 0.0715\n",
      "Episode: 889 Total reward: 32.0 Training loss: 4.3574 Explore P: 0.0713\n",
      "Episode: 890 Total reward: 32.0 Training loss: 4.1155 Explore P: 0.0711\n",
      "Episode: 891 Total reward: 25.0 Training loss: 3.0341 Explore P: 0.0710\n",
      "Episode: 892 Total reward: 24.0 Training loss: 2.9099 Explore P: 0.0708\n",
      "Episode: 893 Total reward: 32.0 Training loss: 5.4084 Explore P: 0.0706\n",
      "Episode: 894 Total reward: 31.0 Training loss: 3.1034 Explore P: 0.0704\n",
      "Episode: 895 Total reward: 37.0 Training loss: 101.0471 Explore P: 0.0702\n",
      "Episode: 896 Total reward: 31.0 Training loss: 2.5097 Explore P: 0.0700\n",
      "Episode: 897 Total reward: 27.0 Training loss: 1.8024 Explore P: 0.0699\n",
      "Episode: 898 Total reward: 22.0 Training loss: 98.0944 Explore P: 0.0697\n",
      "Episode: 899 Total reward: 37.0 Training loss: 68.2138 Explore P: 0.0695\n",
      "Episode: 900 Total reward: 23.0 Training loss: 3.0429 Explore P: 0.0694\n",
      "Episode: 901 Total reward: 24.0 Training loss: 1.9488 Explore P: 0.0692\n",
      "Episode: 902 Total reward: 35.0 Training loss: 1.9037 Explore P: 0.0690\n",
      "Episode: 903 Total reward: 36.0 Training loss: 110.8456 Explore P: 0.0688\n",
      "Episode: 904 Total reward: 47.0 Training loss: 1.7248 Explore P: 0.0685\n",
      "Episode: 905 Total reward: 42.0 Training loss: 2.0307 Explore P: 0.0683\n",
      "Episode: 906 Total reward: 28.0 Training loss: 1.9134 Explore P: 0.0681\n",
      "Episode: 907 Total reward: 29.0 Training loss: 2.3500 Explore P: 0.0680\n",
      "Episode: 908 Total reward: 41.0 Training loss: 2.0067 Explore P: 0.0677\n",
      "Episode: 909 Total reward: 52.0 Training loss: 71.2297 Explore P: 0.0674\n",
      "Episode: 910 Total reward: 89.0 Training loss: 2.5105 Explore P: 0.0669\n",
      "Episode: 911 Total reward: 109.0 Training loss: 1.5073 Explore P: 0.0663\n",
      "Episode: 912 Total reward: 129.0 Training loss: 96.4386 Explore P: 0.0656\n",
      "Episode: 913 Total reward: 175.0 Training loss: 1.5922 Explore P: 0.0646\n",
      "Episode: 914 Total reward: 154.0 Training loss: 172.1991 Explore P: 0.0638\n",
      "Episode: 915 Total reward: 199.0 Training loss: 91.2019 Explore P: 0.0627\n",
      "Episode: 916 Total reward: 199.0 Training loss: 1.6086 Explore P: 0.0617\n",
      "Episode: 917 Total reward: 199.0 Training loss: 1.5095 Explore P: 0.0607\n",
      "Episode: 918 Total reward: 110.0 Training loss: 84.8018 Explore P: 0.0601\n",
      "Episode: 919 Total reward: 199.0 Training loss: 1.1679 Explore P: 0.0591\n",
      "Episode: 920 Total reward: 199.0 Training loss: 68.1259 Explore P: 0.0582\n",
      "Episode: 921 Total reward: 124.0 Training loss: 43.9295 Explore P: 0.0576\n",
      "Episode: 922 Total reward: 114.0 Training loss: 0.9801 Explore P: 0.0570\n",
      "Episode: 923 Total reward: 118.0 Training loss: 0.8024 Explore P: 0.0565\n",
      "Episode: 924 Total reward: 184.0 Training loss: 49.6087 Explore P: 0.0556\n",
      "Episode: 925 Total reward: 154.0 Training loss: 0.9127 Explore P: 0.0549\n",
      "Episode: 926 Total reward: 123.0 Training loss: 1.1584 Explore P: 0.0544\n",
      "Episode: 927 Total reward: 130.0 Training loss: 1.0117 Explore P: 0.0538\n",
      "Episode: 928 Total reward: 106.0 Training loss: 1.1279 Explore P: 0.0533\n",
      "Episode: 929 Total reward: 109.0 Training loss: 0.6439 Explore P: 0.0529\n",
      "Episode: 930 Total reward: 104.0 Training loss: 1.3743 Explore P: 0.0524\n",
      "Episode: 931 Total reward: 123.0 Training loss: 68.0549 Explore P: 0.0519\n",
      "Episode: 932 Total reward: 128.0 Training loss: 0.8787 Explore P: 0.0514\n",
      "Episode: 933 Total reward: 74.0 Training loss: 0.8805 Explore P: 0.0511\n",
      "Episode: 934 Total reward: 104.0 Training loss: 103.7298 Explore P: 0.0506\n",
      "Episode: 935 Total reward: 103.0 Training loss: 1.1678 Explore P: 0.0502\n",
      "Episode: 936 Total reward: 97.0 Training loss: 0.9351 Explore P: 0.0498\n",
      "Episode: 937 Total reward: 83.0 Training loss: 0.8506 Explore P: 0.0495\n",
      "Episode: 938 Total reward: 96.0 Training loss: 1.3029 Explore P: 0.0491\n",
      "Episode: 939 Total reward: 83.0 Training loss: 0.6682 Explore P: 0.0488\n",
      "Episode: 940 Total reward: 76.0 Training loss: 1.2871 Explore P: 0.0485\n",
      "Episode: 941 Total reward: 89.0 Training loss: 0.7438 Explore P: 0.0482\n",
      "Episode: 942 Total reward: 96.0 Training loss: 61.9405 Explore P: 0.0478\n",
      "Episode: 943 Total reward: 95.0 Training loss: 43.5713 Explore P: 0.0475\n",
      "Episode: 944 Total reward: 69.0 Training loss: 183.2538 Explore P: 0.0472\n",
      "Episode: 945 Total reward: 47.0 Training loss: 1.6047 Explore P: 0.0470\n",
      "Episode: 946 Total reward: 53.0 Training loss: 1.1273 Explore P: 0.0468\n",
      "Episode: 947 Total reward: 57.0 Training loss: 1.4529 Explore P: 0.0466\n",
      "Episode: 948 Total reward: 68.0 Training loss: 53.5436 Explore P: 0.0464\n",
      "Episode: 949 Total reward: 53.0 Training loss: 46.9386 Explore P: 0.0462\n",
      "Episode: 950 Total reward: 72.0 Training loss: 0.7298 Explore P: 0.0459\n",
      "Episode: 951 Total reward: 66.0 Training loss: 1.9359 Explore P: 0.0457\n",
      "Episode: 952 Total reward: 76.0 Training loss: 0.7813 Explore P: 0.0454\n",
      "Episode: 953 Total reward: 25.0 Training loss: 1.6697 Explore P: 0.0453\n",
      "Episode: 954 Total reward: 24.0 Training loss: 1.7904 Explore P: 0.0452\n",
      "Episode: 955 Total reward: 17.0 Training loss: 1.4421 Explore P: 0.0452\n",
      "Episode: 956 Total reward: 26.0 Training loss: 1.4990 Explore P: 0.0451\n",
      "Episode: 957 Total reward: 29.0 Training loss: 1.0424 Explore P: 0.0450\n",
      "Episode: 958 Total reward: 30.0 Training loss: 1.7971 Explore P: 0.0449\n",
      "Episode: 959 Total reward: 31.0 Training loss: 1.1870 Explore P: 0.0448\n",
      "Episode: 960 Total reward: 31.0 Training loss: 1.3981 Explore P: 0.0447\n",
      "Episode: 961 Total reward: 24.0 Training loss: 0.9970 Explore P: 0.0446\n",
      "Episode: 962 Total reward: 44.0 Training loss: 1.6205 Explore P: 0.0444\n",
      "Episode: 963 Total reward: 29.0 Training loss: 1.0329 Explore P: 0.0443\n",
      "Episode: 964 Total reward: 60.0 Training loss: 1.1124 Explore P: 0.0441\n",
      "Episode: 965 Total reward: 53.0 Training loss: 1.2141 Explore P: 0.0439\n",
      "Episode: 966 Total reward: 30.0 Training loss: 221.6600 Explore P: 0.0438\n",
      "Episode: 967 Total reward: 30.0 Training loss: 1.7581 Explore P: 0.0437\n",
      "Episode: 968 Total reward: 27.0 Training loss: 115.3489 Explore P: 0.0436\n",
      "Episode: 969 Total reward: 21.0 Training loss: 63.0709 Explore P: 0.0436\n",
      "Episode: 970 Total reward: 28.0 Training loss: 232.3880 Explore P: 0.0435\n",
      "Episode: 971 Total reward: 38.0 Training loss: 1.1753 Explore P: 0.0434\n",
      "Episode: 972 Total reward: 44.0 Training loss: 0.9916 Explore P: 0.0432\n",
      "Episode: 973 Total reward: 34.0 Training loss: 1.3207 Explore P: 0.0431\n",
      "Episode: 974 Total reward: 23.0 Training loss: 39.2687 Explore P: 0.0430\n",
      "Episode: 975 Total reward: 20.0 Training loss: 51.7968 Explore P: 0.0430\n",
      "Episode: 976 Total reward: 15.0 Training loss: 62.3697 Explore P: 0.0429\n",
      "Episode: 977 Total reward: 21.0 Training loss: 222.9603 Explore P: 0.0428\n",
      "Episode: 978 Total reward: 24.0 Training loss: 58.0918 Explore P: 0.0428\n",
      "Episode: 979 Total reward: 28.0 Training loss: 57.2448 Explore P: 0.0427\n",
      "Episode: 980 Total reward: 24.0 Training loss: 58.1396 Explore P: 0.0426\n",
      "Episode: 981 Total reward: 15.0 Training loss: 2.3291 Explore P: 0.0425\n",
      "Episode: 982 Total reward: 17.0 Training loss: 2.0210 Explore P: 0.0425\n",
      "Episode: 983 Total reward: 20.0 Training loss: 53.8194 Explore P: 0.0424\n",
      "Episode: 984 Total reward: 22.0 Training loss: 289.5934 Explore P: 0.0423\n",
      "Episode: 985 Total reward: 18.0 Training loss: 1.1958 Explore P: 0.0423\n",
      "Episode: 986 Total reward: 21.0 Training loss: 2.2203 Explore P: 0.0422\n",
      "Episode: 987 Total reward: 17.0 Training loss: 2.3086 Explore P: 0.0422\n",
      "Episode: 988 Total reward: 22.0 Training loss: 0.9925 Explore P: 0.0421\n",
      "Episode: 989 Total reward: 23.0 Training loss: 118.6779 Explore P: 0.0420\n",
      "Episode: 990 Total reward: 27.0 Training loss: 1.3074 Explore P: 0.0419\n",
      "Episode: 991 Total reward: 30.0 Training loss: 1.4410 Explore P: 0.0418\n",
      "Episode: 992 Total reward: 24.0 Training loss: 1.8827 Explore P: 0.0418\n",
      "Episode: 993 Total reward: 34.0 Training loss: 1.3251 Explore P: 0.0417\n",
      "Episode: 994 Total reward: 70.0 Training loss: 1.7231 Explore P: 0.0414\n",
      "Episode: 995 Total reward: 46.0 Training loss: 43.2006 Explore P: 0.0413\n",
      "Episode: 996 Total reward: 79.0 Training loss: 1.0231 Explore P: 0.0410\n",
      "Episode: 997 Total reward: 58.0 Training loss: 1.2557 Explore P: 0.0409\n",
      "Episode: 998 Total reward: 65.0 Training loss: 154.2917 Explore P: 0.0407\n",
      "Episode: 999 Total reward: 33.0 Training loss: 1.2997 Explore P: 0.0406\n"
     ]
    }
   ],
   "source": [
    "# Now train with experiences\n",
    "saver = tf.train.Saver()\n",
    "rewards_list = []\n",
    "with tf.Session() as sess:\n",
    "    # Initialize variables\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    \n",
    "    step = 0\n",
    "    for ep in range(1, train_episodes):\n",
    "        total_reward = 0\n",
    "        t = 0\n",
    "        while t < max_steps:\n",
    "            step += 1\n",
    "            # Uncomment this next line to watch the training\n",
    "            # env.render() \n",
    "            \n",
    "            # Explore or Exploit\n",
    "            explore_p = explore_stop + (explore_start - explore_stop)*np.exp(-decay_rate*step) \n",
    "            if explore_p > np.random.rand():\n",
    "                # Make a random action\n",
    "                action = env.action_space.sample()\n",
    "            else:\n",
    "                # Get action from Q-network\n",
    "                feed = {mainQN.inputs_: state.reshape((1, *state.shape))}\n",
    "                Qs = sess.run(mainQN.output, feed_dict=feed)\n",
    "                action = np.argmax(Qs)\n",
    "            \n",
    "            # Take action, get new state and reward\n",
    "            next_state, reward, done, _ = env.step(action)\n",
    "    \n",
    "            total_reward += reward\n",
    "            \n",
    "            if done:\n",
    "                # the episode ends so no next state\n",
    "                next_state = np.zeros(state.shape)\n",
    "                t = max_steps\n",
    "                \n",
    "                print('Episode: {}'.format(ep),\n",
    "                      'Total reward: {}'.format(total_reward),\n",
    "                      'Training loss: {:.4f}'.format(loss),\n",
    "                      'Explore P: {:.4f}'.format(explore_p))\n",
    "                rewards_list.append((ep, total_reward))\n",
    "                \n",
    "                # Add experience to memory\n",
    "                memory.add((state, action, reward, next_state))\n",
    "                \n",
    "                # Start new episode\n",
    "                env.reset()\n",
    "                # Take one random step to get the pole and cart moving\n",
    "                state, reward, done, _ = env.step(env.action_space.sample())\n",
    "\n",
    "            else:\n",
    "                # Add experience to memory\n",
    "                memory.add((state, action, reward, next_state))\n",
    "                state = next_state\n",
    "                t += 1\n",
    "            \n",
    "            # Sample mini-batch from memory\n",
    "            batch = memory.sample(batch_size)\n",
    "            states = np.array([each[0] for each in batch])\n",
    "            actions = np.array([each[1] for each in batch])\n",
    "            rewards = np.array([each[2] for each in batch])\n",
    "            next_states = np.array([each[3] for each in batch])\n",
    "            \n",
    "            # Train network\n",
    "            target_Qs = sess.run(mainQN.output, feed_dict={mainQN.inputs_: next_states})\n",
    "            \n",
    "            # Set target_Qs to 0 for states where episode ends\n",
    "            episode_ends = (next_states == np.zeros(states[0].shape)).all(axis=1)\n",
    "            target_Qs[episode_ends] = (0, 0)\n",
    "            \n",
    "            targets = rewards + gamma * np.max(target_Qs, axis=1)\n",
    "\n",
    "            loss, _ = sess.run([mainQN.loss, mainQN.opt],\n",
    "                                feed_dict={mainQN.inputs_: states,\n",
    "                                           mainQN.targetQs_: targets,\n",
    "                                           mainQN.actions_: actions})\n",
    "        \n",
    "    saver.save(sess, \"checkpoints/cartpole.ckpt\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing training\n",
    "\n",
    "Below I'll plot the total rewards for each episode. I'm plotting the rolling average too, in blue."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def running_mean(x, N):\n",
    "    cumsum = np.cumsum(np.insert(x, 0, 0)) \n",
    "    return (cumsum[N:] - cumsum[:-N]) / N "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x1e2ea0276a0>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsvXeYJFd57/99qzpMh8kzuzu7M7Ozu9pdobiSVogkgRAg\nDCbZWBbXxrJ/2EK2zDXgaxsw5nKNZWMuweZngy2CkTEIZCQsggAhAQoWymG1kjZMDjs7OfV0rKr3\n/lF1qqu7q8OEru5Znc/zzDPd1V1dp6urznveTMwMiUQikUjyUWo9AIlEIpHUJ1JASCQSicQVKSAk\nEolE4ooUEBKJRCJxRQoIiUQikbgiBYREIpFIXJECQiKRSCSuSAEhkUgkElekgJBIJBKJK75aD2Aj\ndHR0cF9fX62HIZFIJFuKJ554YpaZO8u9b0sLiL6+Pjz++OO1HoZEIpFsKYhopJL3SROTRCKRSFyR\nAkIikUgkrkgBIZFIJBJXpICQSCQSiStVExBE1ENEPyei54noOSL6E2t7GxH9lIhOWv9bHft8mIj6\nieg4EV1drbFJJBKJpDzV1CA0AH/KzOcAeBmAG4noHAAfAnAvM+8HcK/1HNZr1wI4F8AbAXyBiNQq\njk8ikUgkJaiagGDmSWZ+0nq8AuAFALsAvA3ALdbbbgHwduvx2wB8i5lTzDwEoB/AS6s1PolEIpGU\nxpM8CCLqA3ARgEcAbGfmSeul0wC2W493AXjYsdu4tS3/s64HcD0A9Pb2VmfAEokEhmFgYWEBoVAI\n4XC41sOxSSQSUBRzbWsYBjKZDKLRKGKxGHRdRzAYhM/nQyAQ8HxsmqZhcXERmUwG7e3tNRnDZlJ1\nAUFEUQC3A3g/My8Tkf0aMzMRrakpNjPfDOBmADh8+LBsqC2RVIl4PI7Z2Vn4fD7s27ev1sOxGR0d\nLdjW3NyMpaWlnG0HDx70akg2y8vLmJubAwDouo7u7m7Px7CZVDWKiYj8MIXDN5j5DmvzFBF1Wa93\nAZi2tk8A6HHs3m1tk0gkNYDZXH/pul7jkZQnk8nUeggAsucMMLWJrU41o5gIwFcAvMDMn3W89D0A\n11mPrwNwp2P7tUQUJKI9APYDeLRa45NIJKURk51T65e8uKimiemVAN4N4Fkietra9hEAnwRwGxG9\nB8AIgGsAgJmfI6LbADwPMwLqRmau/6WLRHKGIgWEpGoCgpkfBFDsyrqqyD43AbipWmOSSCSV4zSX\nSF6cyExqiURSEqlBVM6ZJlSlgJBIJK6caZOdZO1IASGRSFzZSj4IKcyqgxQQEonEFTnpSqSAkEgk\nJdkKGoSkOkgBIZFIXNlKGkS9jLVexrFZSAEhkUhcOdMmO8nakQJCIpG4IgWERAoIiUSy5Ukmk7Ue\nAoAzT6hKASGRSFwRk92ZNul5ha7rW75gnxQQEonEFSkYNoamaRgYGKj1MDaEFBASiUQicUUKCIlE\n4orUINbOmXbOpICQSCQSiStSQEgkkpKcaatiSeVIASGRSCQSV6rZcvSrRDRNREcd275NRE9bf8Oi\n0xwR9RFRwvHav1RrXBKJZGvBzFuiLzZw5mlb1Ww5+jUA/wTg38UGZv5N8ZiIPgNgyfH+AWY+VMXx\nSCSSLcjU1BSWlpZw4MABWTjQY6rZcvR+Iupze43MX/kaAK+t1vElEsnGqJfV8NKSuY5k5roXEIZh\n1HoIm0qtfBCXA5hi5pOObXss89J9RHR5jcYlkUgk60LXdaysrNR6GJtKNU1MpXgXgFsdzycB9DLz\nHBFdAuC/iOhcZl7O35GIrgdwPQD09vZ6MliJ5MVMvWgS9c5W8ZOsBc81CCLyAfg1AN8W25g5xcxz\n1uMnAAwAOOC2PzPfzMyHmflwZ2enF0OWSCSSstS7+Ws91MLE9DoAx5h5XGwgok4iUq3HewHsBzBY\ng7FJJBKJxKKaYa63AvglgINENE5E77Feuha55iUAuALAESvs9TsAbmDm+WqNTSKRlEealiTVjGJ6\nV5Htv+uy7XYAt1drLBKJZOtT7wJLmpgkEolE4opTgD06NI87n5qo4Wg2h1pFMUkkki1Cva/c65Gb\n7zddqG++YGeNR7IxpAYhkUgkVWJ6pT5aoa4XKSAkEomkSiQyWzs3QgoIiUQi2QTcTHHPjC/WYCSb\nhxQQEonEFel7WB+G47zddeR0DUeycaSAkEgkJZGCYm3oxplzvqSAkEgkkk1EkwJCIpFIvKXeNRkx\nvnwNQtO3bglwKSAkEolkE9F1U0DsaA4CAFaSmVoOZ0NIASGRSCSbiGY1DYoE/QCA1fTWDXWVAkIi\nkbgiTCb1btoB6muMmmVRigRUAEA8tXU1CFlqQyKRbGlmVlL42PeOIqMx/uSq/Ti/u7km48j6IEwJ\nEQ6Y02siLX0QEolEUhPmYilkNHNy/sd7T9bc5i+imMIBc3qNp7euBiEFhEQi2dJk9Fzz0nJCq9FI\nTHRbQJgaRHwLl9uQAkIikZSlnmz8+aTzwkhrnYdgaxBB0weRlE7qQojoq0Q0TURHHds+TkQTRPS0\n9fcmx2sfJqJ+IjpORFdXa1wSiaQy6lkoOMnkCYiUVhubf4EPwm9pECkpINz4GoA3umz/HDMfsv7u\nAgAiOgdmK9JzrX2+IHpUSyQSCVBcYGW0fAFR2wlZDCcsopgytTV5bYSqCQhmvh9ApX2l3wbgW8yc\nYuYhAP0AXlqtsUkkkjOHTJ5J6fvPnMLH7jxa5N3VR9dFFJMpIFIyimlNvI+IjlgmqFZr2y4AY473\njFvbJBKJpCT5JqbBmVWcWkzWrMSF8EGEhJM6LTWISvkigL0ADgGYBPCZtX4AEV1PRI8T0eMzMzOb\nPT6JRLLFSBfxOczG0p6OI78WU8CnwKeSNDFVCjNPMbPOzAaALyFrRpoA0ON4a7e1ze0zbmbmw8x8\nuLOzs7oDlkgkAOrbYa0VGVqt2n0KDcKnEAI+BSkZxVQZRNTlePoOAMJQ+D0A1xJRkIj2ANgP4FEv\nxyaRSHKpZ6HgJM2mKeei3pac7as1ih4SPghVIQR9isyDcIOIbgXwSwAHiWiciN4D4FNE9CwRHQFw\nJYAPAAAzPwfgNgDPA/gxgBuZeeueVYlE4hkZ3UBjgw9/9Jp9AGW3i6J5XuPUIIKqsqX7UletFhMz\nv8tl81dKvP8mADdVazwSieTMJKUZ8KsEIoJfJbvsRr7zutrk+yB8ioJogx/Pji+CmUFEpXavS2Qm\ntUQi2RIUzYPQGQHVnMrIoULkl+DwCqFBqArwkq4mnFpM1mwsG0UKCIlEUpJjp5fxxMhcrYdRlJRu\nwGcJCGdEk9cahEBkUquKgpBVsK/WyXvrRQoIiURSkk//5ASu+deH69ZpndEM+NXCqaxWAkLIKFUB\n/IoCgJHMbM1kOSkgJBKJK06BEKZM3XZGS+mmDyIfr806zlpMqiJ8IgoIUoOQSCRnMG2UwHKiPvsa\nZDRGwFc4laVrNClrBkNVTIHl95n/a1VAcKNIASGRSCpiqU4FRFrTbROTT8lqElqNHMO6wfY4TBMT\nkNyioa5SQEgkkqI4J9l6FRAZnW0B8TfvOM+xvXZ5ED5VaBDCSS01CIlEcoaRdJhp6tXEJPIgAKAj\nGsQXf/tiALX3QQDI+iCkk1oikZxpOMtVJOt0FZzWDTsPAoA9OddMg9DhEBAEgHME7VZCCgiJZJNg\nZszNzUHXt+ZkkA8zI5bKag2pGtvRi4XZpvPCXBUiENXSxGRkfRDWuKQGIZG8yFldXcXs7Cymp6dr\nPZRNw2lWqlc7elo3bFu/wKdQQa9qrzCd1OZ4bAEhNQiJ5MWNWOHWa0LZeoilsr0M6kFA5J9b3WAY\nBsOv5OZBRIM+zNewH4QwMQVUsvIgan/u1oMUEBKJpChOR289roKFGSk/D6IlHMDkUqIWQ8rNg7BN\nTPV37iqhaDVXInoKQNGlEDNfXJURSSSSusFZMrse7ejCjJRfaqM1HMBzy6laDAmano1i8vmULa1B\nlCr3/U7r/w0AVABft57/FoCtKQ4lEsmaEJVJgfqc5ISGky8gAj6lZhqPbjCCPtUch6IgQFpdnrtK\nKCogmHkAAIjoqjxt4SkiehLAX1R7cBKJpHYwM0RAVrCGE24pMroBBtklLQSqAui1yoNghpBXqgI0\nKakzOpNaJaKXiSdEdBlMjUIikZzhaAZDUQhBv1KXq+CMNSZR0kLgU5Ua5kGwXX6ciOAjwt1PDcKo\nUYe7jVCJgHgPgC8TUT8R9QP4MoDfL7cTEX2ViKaJ6Khj2/8lomNEdISIvktELdb2PiJKENHT1t+/\nrPcLSSSSzUPnbNnquvZB5DmpVQIyRm00CM2RSW0+Z8SX5vGjJ4c8Hc9mUFJAEJEKYDcznwfgMgCX\nMfP5zPxYBZ/9NQBvzNv2UwDnMfMFAE4A+LDjtQFmPmT93VDxN5BIJFVDs2L6FYVq1uO5FMV8EKqi\nQKtRwqJu5BYNFNQqL2MjlBQQzKwD+Ij1eI6ZK24rxcz3A5jP23Y3M4vA6ocBdK9tuBKJxEs0qzKp\nqtSuOqrALb8kY0cx5fsgqC5qMQHAe161BwCQqEMNrByVmJjuJqL3E1EXETWJv0049v8H4EeO53ss\n89J9RHR5sZ2I6HoiepyIHp+ZmdmEYUgkEjeY2bSnKwSfUjubfv6YnAgfREAtzKTO1Ejj0QzYPggA\nuLCnBQCwktp6jupSYa6C37b+/6ljGwPoXe9BiegvAWgAvmFtmgTQy8xzRHQJgP8ionOZeTl/X2a+\nGcDNAHD48OEzJ2VVIqlDNIOhqmStyGsvIPIRQqDAB6EQmHOzmqtNjg/CcciQXwERsJLUiuxZv5QV\nEMzcs5kHJKLfBfCrAK5i64wycwpAynr8BBENADgA4PHNPLZEIlkbWRMT5eREeEW5siUZjcFw80GY\nJS4yugFV8Sbo0jYx6dlaTIAZydTgU7GaPgMFBAAQ0dkAzgHQILYx8zfXejAieiOAPwfwamaOO7Z3\nAphnZp2I9gLYD2BwrZ8vkUg2F+GkVhVCqh41iCI+CJ+j5HeD39uofN3SupwE/QoSddrTuxRlBQQR\nfRTAGwCcDeAnAK4G8CCAkgKCiG4F8BoAHUQ0DuB/w4xaCgL4KREBwMNWxNIVAP6aiDIADAA3MPO8\n6wdLJBLPEHWFVIVq7qR2w67FlKdBKJaA8HLMzGz6bYxcDQIAGnwK4lswWa4SDeI3ARwC8CQzv5uI\numCGsJaEmd/lsvkrRd57O4DbKxiLRCLxEKeJyeu8gnIwM+58+hQAH4IF5b7N5177TcTR8hQIBPwq\nEqmtZ2KqJIopYYW7akTUCOA0gN3VHZZEIqk1IopJVQg+haDVmYlpaHbVzu62LBI2QqHwUqgxM3Tr\neL48jSboUxA/E01MMGsvtQD4Kkyn8TKAR6s6KolEUhdouoGwT4FKtQ9zzXdYjy8UL+etWhqE10JN\nHC4/ciroU7B0JpqYmPm91sN/JqKfAGhi5ierOyyJRFIPaFZWcK0S5fKFgvP5QjyT/3Yb4bT2sn4U\nM0OcovxM6ga/injiDBQQRPRvAO4H8AAz91d/SBKJpF4w6wqpUBWlZolnxViIF+/3ELD6MHgZOeQ0\nMeVrEAGfgsQWDHOtxAfxTQB7AHyJiAaI6NtEdGOVxyWRSOqAlEF1U2ojn9USmclByweQ8NisU8wH\nccZGMTHzT4noHgCXALgKwI3W43+u8tgkEkkNYWakDUD1N0DxETI1Kn5XDN2KsPrX/3EJiLQc85No\nQeq1BqHZGoS5LRQKIZFIIOhXkUzXlwZWCWU1CMvv8EsA1wEYAvAyZj6r2gOTSCS1R9PN1bBfQd1V\nc9UMRk9bGLvbIwWv2QLCaw3CElI+RcH+/fsRDAYBmBpERjeQrsOeGqWoxMR0AmbdpP0wy1+cRUSB\nqo5KIpHUHGZGRjfg85Hpg6izyc0wGD6VQEQFYa6i5aeXoaWmD8J8rCoERVHscQWs8Wy1bOpKTEzv\nAwAiagbwOzB7U28DEKru0CQSSa3JiFIbxDVJlCtVi0lnhkruhfhqpUFoRZzUDdZ4YmkNzWG/p2Pa\nCJVEMd0A4HIAlwI4BeDfATxQ5XFJJJIaw8xIa4ZZ7pu8zykoh6YzAkFz4lUUJaelpxAQcQ+zl0UU\nU5J98OcJiEjQB4CxsJrGrpats7auJFGuBcAXADzGzOkqj0ci2bKUqzy6lciWrjb7K6swYLBp1lE8\nKp9dbEwC3WD4iODz+eDz+aBpWWHQ4FPgV0vnSlRjfLoBzHAEO3r7AADJZBIA0BY1rfKnFhM4b1ez\nZ2PaKGV9EMz8SQA6gGsBgIjaiGjdvSAkEsnWwGCGwabD1a6OWmNHtVNI6JawUlUVTU25PcyICG3h\nIOZixXMlqoFpYiIEfObaO5Uyj98WzgqIrUQlUUwfhVmJ9aPWphDKVHKVSCRbG2EuYQA+q2EQAM/b\neJZCN0zzV0NDA1pbW7F3796c19ujAcytemf0cIa5iv4UQqCZJiZgMeGdRrMZVBLF9E4AbwKwCgDM\nPAFgM1qOSiSSOiZjT3ZZAeG1H6KU2U5jBjU02qGkfn+u87epwYcljydkkSgnfCDRaBSA6bSOBH1Y\nTmytbOpKBETK6vzGAEBE4eoOSSKR1Bpmhq4bYFCuiamuNAiG6ivuRg34FE8FmrPUhtAgurq67Neb\ngornAmujVCIg7iCifwbQTES/B+BuAP9W3WFJJJJaI9IenCamekqW0y0HejH8qoK0xw2DMnnF+pz5\nGdv9KSwnzzABwcx/D+AHAL4H4EIANzHz58rtR0RfJaJpIjrq2NZGRD8lopPW/1bHax8mon4iOk5E\nV6/v60gkks3AXA2bwsDUIET5bG81iJImJoMLwkmd+FXvS5SLcxZwNDDq6ekBADQ1+HFqPubpeDZK\nJRoEmPlHzPwBZn4/gB8T0W9WsNvXALwxb9uHANzLzPsB3Gs9BxGdAzNK6lxrny8QkbeNZCWSTSI/\nq7cYzJwTu19vaJaT2q8SVOturHVPCCe6zlDV4tOEXyVPxysaLJnHzk6twjeyf3sUJ04vY2ULaRFF\nBQQRRYnoz4joH4jotWRyA4ABmBnVJWHm+wHk95V+G4BbrMe3AHi7Y/u3mDnFzEMA+gG8dI3fRSKp\nCyrNh5iYmMDJkyerPJr1YU52lgahmg2DAO99EM5zWZAHwQxfiWWkX/W+PIg4Pf78nqMAtjc1gMA4\ntZj0dEwboZQG8R8wTUonYVZwvQfAbwO4hpnfvM7jbWfmSevxaQDbrce7AIw53jdubZNIzlhWV1dr\nPYSSmPIhW+4bqB8NQqzWA0qd+SBcNAhBezQABYzJpa2TC1Eqk3ofM58PAET0LzAn9F5m3pRvx8xM\nRGv+9YjoegDXA0Bvr8zXk9QflZqY6hkzpt8As6lBsC6c1LWNYnJmeANAwF9chQjUwAfhZmIStIQD\nIDAml84MDcI2lDGzDmBsE4TDFBF1AYD1f9raPgGgx/G+bmtbAcx8MzMfZubDnZ2dGxyORCIphj3Z\nKVSzHs9CICQyOkbn49mxWQIi6Cs+hflUbzUeZobGDIUKi/UBQEvIDx8xJrdQNnUpAXEhEc1bfwsA\nLhCPiSjft1Ap34PZVwLW/zsd268loiAR7YFZWvzRdR5DIpFskJxMal/t8yA+f+9JvP2f/9t26ouJ\n322lLvA6iklkUuePSWiUqkJojwa2VLmNUiamDfV8IKJbAbwGQAcRjcMs1/FJALcR0XsAjAC4BgCY\n+Tkiug3A8zB7T9xoaS0SiaRGaHaYK0FTcrd5zcmpGAgRzFq1lYR2U0qDMAUEg5k9M/tpemmh1RPK\nIDYzhljsLDvLup4pKiA2OkEz87uKvHRVkfffBOCmjRxTIpFsDtmQTTKjmBQRxVTbUhvjC3G0IKvJ\n+MsICFjvDfiqLyCE38YtgknQ3ODHfDyD+fn5LSEgKsqDkEgkLz5EHkSgjor1LVrlu0V/7GCJOFd/\nDXI33ExMTppCfixvoXIbUkBIJJICnHWFnLWYvM6kHhkZyZngY0mz2F1+1VQ3AtZrSY+6ygmtq9SY\nGkNmAcFUnbVvLYYUEBKJxBXNMArKfXvpgxDmpdOOsNCY1SFOaDKBEiamSNDMYF5NeefONDWIXBOT\n0//R3GCO6cN3POvZmDZCUR+EFbnktlwgmGkMbVUblUQiqTl2sb4aV3MdnM0mFK4k0wB8drhtoMRq\nPRI0bUxeFchjNvt2l9IghC9neG5rRDKVimLq8GwUEomkrhDlvgHTjFOLfhBCg4g5JvjVlAbAZ5uN\nQoHiPohGq0lPzKO+1FkTU/ExNTWYY6qfoumlqTiKiYjaADQ4Np2q1qAkEkntEe0zczvKeW87F5pM\nRzSAmVgaQIPti2gJF4/Gj1qTsXivF2gGl4ysuqi3BeGAitZwyLMxbYRKWo6+mYhOwKyP9Ij1/2fV\nHphEIqktdrnvGkUxCQ1CZwOqQuhtC2NszjQ3LVtaQWsJAREJmAJiJeWtiSlQwgdBRLh0T5unQmsj\nVOKkvgnAKwEcZ+YeAFcDeKCqo5JINgAz4/jx41haWqr1ULYsZky/aQpRqbZOasMAFAXY2RKy6xit\nJjUEfEpJE1PAb05vaQ8jhjTdsHtnFCPkVxFPnzkCQmPmGQAKEREz/xSyFLekjtGtGPnZ2dkaj2Rr\noxkGAqoCRVEgrCa1cFKLSbct4sdiPA1mRiyt2U7oYsxOmuXcvBizEGaZMiYmwPSbaDp7Fn67EUo5\nqQVLRBQF8CCAfyeiaQBbwwVfI3Rdx8DAAHbt2oVIJFLr4Ugk60LXGT7LXCL6QXiZB2GbmAyGqhCa\nQwGkNQNpzUAirdsmpGLUwrGu6YUmpnzCfhUAYzmZQUOJarT1QCUaxNthCoT3A/gFzCqrv1rFMW15\n0mlzlTM3N1frobwoERPLVim7XWmDIS8ZHx+3ykaYUwQRoFBtTEw6M1QitIbNHIKVlIZ4WitpXgIA\nRSF0K4tIJb1bz2q6UTLMFQDCluazsgX8EJUIiA8zs87MGWb+CjN/FsAHqz0wiURSWzTD7AUh8KkK\n0rWIYtIZPoUQDfpAAOJpHfG0gXBALbkI8BGBACRj1fdFZftU5J4zoHChEvKrIGBLlNyoREDk95UG\ngPV2lKsrZmZmcOqUjNY9EzAMAwMDA4jH4+XfLKkI4YMQ+BWqjYmJTRNT0K8AYGR0A/G0hrC/MhOT\nlzItrZcu1gcAoYAPoK2hQZTKpH4vgBsAHCCiJx0vNQJ4otoD84L5+fW2tZDUG+l0GpqmYWZmBl1d\nXQC2jomp3nDa/n2Oyc7vUzxvGCTGoaqEBqswn/BBhMs4qUXPHt1Ds5ieJ1TdCFsaxJYWEABuA3Av\ngL8D8CHH9hVmnnbfRSKRbHVEU578yqQ+RUHGw5ajzvaiqqKgwZpYU5qBeEa3nL3FISs814s2qXYU\nU5l+EIDI/mbPSoBshKLfhJkXmLmfmX8DZgb1660/2edTIjmDsQVEXmVSv0o1KbWh6wZ8CiwTk2W7\nZyBURoMATDOT7qFQ03QjR+tyIxwQTur1CYiZmRnPQrgryaS+EcB/Aui1/m4joj9a7wGJ6CARPe34\nWyai9xPRx4lowrH9Tes9hkSylai3KKasicnIc1J764MQ6Gwm64V8KoiARcu5G/b7ypoRvdIgBGmd\nC0xM+WMMWi1c51fXJyDm5+c9i5CsJA/ivQBeyswxACCivwXwEIAvrOeAzHwcwCHrs1SYYbPfBfB7\nAD7HzJ9ez+dKJIKtFuZabzhNOwGV7PPoV7yNYrLHoZulNgJ+AsBYEgKiTJgr4L2AKNcwCDCvy7ZI\nAFPLyZLvqwcqiWIiAGnH84y1bTO4CsAAM49s0udJJJJNwnRS106DyEYxmf6P5XnTrLIYN6ejSgSE\nTyHoHmVSG9ZfOQEBAO1hPyYXVzE2NoZMpn59EUW/CREJ7eLrAB4hoo8S0Udhag+3bNLxrwVwq+P5\n+4joCBF9lYhaN+kYEolkHTh9EGxNfF4mygkMw4CimGG2BNgaRKhMJjVgmqY0j0x4utWitZQPYseO\nHVAUBW2RABaXVhCPx+u6JEwpUfcoADDzp2CameLW3w2bYQYiogCAt8L0bwDAFwHshWl+mgTwmSL7\nXU9EjxPR4zMzMxsdhuQMpN5s+lsNp4nJn5coV4tqrhkr+UwhM6N72WFiUsoUxvPSxCSOUyrMtbm5\nGZFIBG2RAKZXkmDmujaFlhLB9qiZ+VFYAmMT+RUATzLzlHWMKfvARF8C8AO3nZj5ZgA3A8Dhw4fl\nTCApSj3feFsB3TByCs/5FaqZBqEqpi/ErxCWE2b+QDigQi3RnAcAVNWbKCZnD2+3RLmOjg67LhsR\noS3st/I5DLTW8XVaSkB0ElHRkhpWyY2N8C44zEtE1MXMk9bTdwA4usHPl7yIcGoNW02DqLfxLi4u\nAhAmpuzk5VOpNv0gDLMWE2D2oNbS5vYGf6GA2LFjBzRNs802KnmnQZi+DnKt5tre3p7zvDUSQCOl\nsBBPY9cWFRAqgCg2zyFtQ0QRmDkV73Vs/hQRHYJZgn447zWJpO6pt4l+vYg+GrqR2z7Tryqete90\nImoxAUBQJRgAmkJ+W6tw0tzcjHQ6nRUQHjmpgWwhQ38ZsxcAtEX88MHAfDxd15puKQExycx/XY2D\nMvMqgPa8be+uxrEkLz5kmOvmkLF8EHaYq6rUKA+C7bpKfpWQAhBo6kBHR4fr+52/u8/DTGrNAJgB\nv6/0dWeamIIAgOHZVbxmDdep1xFPpUSdvLskkhcxZrG+3MnWy57U+f0ggKwDuLUpWmC2cUPx0G8i\naj5VEubaFvGjtz2MZyeW1rSQmZqaKv+mTaTUN7nKs1FIJJsEEW3I1MPMmJ6eruvYdK/IaAaC/lwT\nk5dJZwLNyJqYhE8kWKIOU74G4ZVM02wndflEOV3XsbstjJmV1JoEhNdmzFK1mGSpU8mLjlQqhYWF\nBUxOTpZ/8xkMMyOjc07HM1+tajEZDNWadAMqgVF6EnZOuKpHGoSIYjLHVtmEv62xAStJDatr6E/t\ntdm0kkxqiWRLspGbycuVWj06t0W0kujaxsxmNdda+CAMhphzxeRbiRkHsASER62f16JBAMC2JtMP\ncWqxfkuQIyPaAAAgAElEQVRuSAEhOePYqIkJyN7E6XT6Rdk6VtRcanCEbAZ83voggOzKXLUig4SA\nCJRwBOdoEKRA5/rzQQBAR9QUEKeXU+s7nl59yScFhETiIF8YTExMYHZ2FppW/81dNpO0Zk52znIW\nPsVbH4RZ38g6tkNzYBB8JUJJc01M8CzyyjxlVLEGEbHKla83dHh6uvpteaSAqCG6rmN2drYuTQxb\nmfWGuRqGgdXV1YJtaz3umYCtQeT5ILzWIER2crCxDYCZzQ1Uvkr3SqgxM3RdaBCVXXfi3MbXKSC8\nuN6kgKghU1NTmJubK+ijnEql7GQlSWVsxs1S6jOWl5eRTNavrXizERpEQ34Uk0ercWY2NTdrcvfl\nO6krNTGp8KxhkGY7qSvTIIKW+S6eXp+pqFwdqs2gkn4QkipRbHU6PDwMwMwKlayd9QqLUtqCKAx5\n8ODBdX32ViPj0CDEhObzMKdgeXkZuq7bxwuouWGu5fo+C1TFuwq0eoVOaoFfNRsHrSWKyYkXEU1S\ng5Ccsaz1BqqViageTVNiVdvY4PBBWNVcvRivXcnV0mQCliYjrDelfBBOVBK+gepiZlIXL9bnxHld\nNgRUxC0BkU6ny1oOnPtKASGRVIhz0hKawFpvoEQisalj2sqISauxwW9vE6t4r1t4AkDQZwoIs6cc\nlS1n0dpqtpNRFcX2DVQbTTfAyI61Ehr8ChIpUxiPjo7i9OnTFe8rBYREsg7W66TOvzl1XV9X9BIR\nYWlpCZlMBgsLC2tydNcCTdOwuLiYI2TjaXPMzWFTQDBnu8t54Yeww4yt5b+w11s+6rImJvFdVIWQ\n8ej0C7Nc0KWaq5McDcKnImYJCBG2Wun14oWAkD4IyRnHejWIfNYSRphMJu0IKMMwcoSNpmno7Ows\num86nYbf7y/6erWZnZ3F0tIS/H6/3bNAaBBNDX7ErDB9Ue4iYxgIofJV8kYQ/oOg3wdAg+miLm/n\nFwIi6FOQ0Q1oupHTPnWzEZnn5lgrP04ooCJhnWtFUWAYRsUmvHK9MDYDqUFUgXq0KZ/puPWD2KiA\nWMvKf2RkBLFYbF2fMz4+vqZxbTYiGiaVStmr2NWkBp9KOathv4cahED4IBocJiWGWYSvFOIaEJng\nK8nq57GYGgSVNTE5r8ugT0E8o+dsryeNUwoIyRlHuRtsZWXF1XTk8+Uq1G4CRkTXnEk4v6fQmsYX\nE+hqDuWEUopkNS/qMdkmJmG2sZzUSct5Hi5RrA/ICoiwlei3nKx+8cWMbsCnkl15thJCftXOg5AC\nYgsRi8VkRc9NpJoTKzPnRH+U0uAMw8CpU6dcV+2hUCjnuZuAmJycrFohv/zv4TXMbAvOmZUUupob\ncl4XGkTaw2S5rNnGFAiLiRQYZNcxKkZLSwsAsy0pALtNabUQJqZABQ7qHB+EPxvFJIRxKQHh3Fcm\nytWQiYkJjIyMFGwXN7E0I1VOKpXC5OTkmiI01sL09HSOeafUDSZ+NzcNIv83LWaiKue4rvRz8pmd\nncXp06exsrJS0fs3C7fxxdOaXQpClFDn9CoA9tbEZNeEMsdy/i4zN+jg9saS+4XDYQDmCp1BWEp4\no0EE1+jniAR9WE5oYGb7d6inuaUmTmoiGgawAkAHoDHzYSJqA/BtAH0wW45ew8wLtRifwG3Fu7i4\niOnpaTCzvUqpFs6LZisjJuxq1TNKpXKLnQnNbyM3GlFhO0vnawJmXvOErut6QfY8kD0/tczHEH/x\ntG6bZwSZlQVEKO1puY1kxkAaKkIBFck48Cvn7sAbztuFbU0N5XdGNgprarn6WfBp3UCgTAQTkHv9\nNIV8yBgGlhOaNDHlcSUzH2Lmw9bzDwG4l5n3A7jXel413G7QShA3cSlzST2tAF4M5P8WoiRGfl2l\ntVKJcF5ZWVmzyWlqagqnTp0q2F6rVqn5Dv6kZoA5a54R+BSCCkbKg8wzcQ7iaQ1TRiOaQgFrOxBt\nqDziqz0SABEwtmDe75lMZt33fimYTc2qVCMj53sBoKmpCS3W95peWbsAe7GZmN4G4Bbr8S0A3l7N\ng42NjRV9rdSJPxNW9LViredudXW1Iq2j2IpL07SC/Uv9tvmvVVLrZj1+lWK+La8ERLHz6tQeABcB\noSogsKc+iHhaR8Cn2FVl1zop+lUFLSE/Ti+ZE/DIyEjJe3+9mD6IyjQI8R0URUFzyA8CML2yvpLf\n1aZWAoIB3ENETxDR9da27cwslmKnAWx325GIrieix4nocVEfZ6tT7KJ/sWsi4+Pj67qZ3UJeiz3f\nbAZmYgXF4Sqd8L0SEPnnVRxX/Bdx+fkmJlFCIu2BBmGPJaOjqcG/oXMSCfqwkiqv+W8EZlOzagiU\n1yDEgkZRFNsE5tQg1rKIqTa1EhCvYuZDAH4FwI1EdIXzRTbPguuZYOabmfkwMx8ulXwkOTNIp9P2\n43g87kmI6YMnZ/GR7z6LYvdisUiS8YU4/u6uY/jKA4PrOq4XAkIcw3lenQJCVVWsWpm93bv7cvb1\nqwoI3ggIQTyloanBt6EaRJGAiliV8yCYGWnNQIOvvFs3R0CELAHhaBpUqRA4Y01MzDxh/Z8G8F0A\nLwUwRURdAGD9r343jOLj25TPkeao9ZP/GxiGgbGxMUxMTICZiyallfvtSk3CzIxkRsfXHhrG8dMr\nmI+nC94jyGQyBc5xMXHe9WyuTyL/WLXUGN3McU4BQUR22GVrODeUVBTI81JAJDI6GkP+DQmIcFDF\najKVE0ywnnOdTCZdTXOpVArpdNoUEGvUIBp85t/0SmpN3yuW0s5MAUFEESJqFI8BvAHAUQDfA3Cd\n9bbrANzp9djqjRe7icmNZDKJxcVFTExMYHl5uej7MrqBU4trL743v5oVChMLxfcfHBy0y7ILnOGf\nIjs2n2QyWVQLEr/3ysoKEokEFhcXNyXyi5lth3058wUzI8U+jBktaC4QEOSZD0KMM57WCzSItRL2\nq/DFZnICAxKJxJq10ZGREQwNDRVsHx4exvz8PNK6joZAeQ0iGo2a4wqHQURoiwZyfBCV3PefufsE\nPvGD59cw+vVRCw1iO4AHiegZAI8C+CEz/xjAJwG8nohOAnid9bwmbHRilhN79TA7d5k3tltSmTj3\nH7vzOXzszuewsJp2fb3YZ2dNEYwHTs5WNB5B2uF7WIi5ax8jIyNFJ33xWYuLixgdHcXU1BRGR0fL\njqEcs7OzGB8fx/z8vOv3d2oQzIyYpUEI84cQLr5amJjSGppCG/NBhAM+JDK553xsbMw1kqwcpUJQ\n05pRNsMbACKRCA4ePIhg0BTA7ZEApisMw1VVFUuJDMbm4zhQJhdkM/A8D4KZBwFc6LJ9DsBVXo9H\nUp+UmshFdFE8Hi8w8wBA/3QMM9aK7PRyEttaIhUf19m85dRiHEBhrkuxySqjWStSAvpnYtjVGnJ9\nn5O7np3E2TsasWdP2nXy2YxsfuFvmJmZQUNDQ872QCBgPxfnfHQugfZIAE0hX84YfKqlQXjopI6n\nDTQ1+HMiytYqLNoifiwlNBjMUBz7bnaHwLRm5PTwrpT2SBBH51J2uXlmRiKRQDAYLIikIyIsJXUw\ngN628tfXRpHVXOuYgvj0ZLKgHMSLBee5cN40+WYCZsbx01nT06nFBC7oLfwcEQKbX39JNJDf1RLC\nQgkfhBtO08vIXDbWvtiEphuMO56cAADs7YwWjKUalPNBJBIJjC4kcHZXmz1ucY5FP+iUh2Gug6kI\nXrVBE9OOphB0g7EQz6A9Eii/wzpJrUNAEBHaIn4sDM/b2zRNw9TUFJqamtDV1VWwz2pKA4PQGKz+\n9VJPeRA1Yy2x8rViZmYGo6OjBSvmRCJRl+PdKJVoEG6k02mMO3wHxyZzs5ydnzswMJDzWiKRwPxq\nBiBgT3ukwDxVbpx2H2efgsV4+YxjZ4XRLz8wWLVIm3L1e8S21dVVPDu+hPHpBXQ1Fy5ERLlsr0xM\nGd1ASjPQZJm6xPdYq7DYYdWUmlzK1RjWUlq7HMyMtM45PbwrpTUSQDyVsc+rmI/ctGMAiIlufyEp\nIDwhf6KoR4Q67FwBrq6uYnR0FAsLNa1I4jnFJogXJpfx199/Ho8NL+DCnhb0tIXxxNB0Rc5IEeEy\nu5JCa9iPjmgAC6sZ1wmkmNlHFJbriAZwZHwJ/+f7zwEozDMQOOsDPTw4j58dc69VtZmlF9xyRMT/\nkbk4/vHekwByE7eEU9Wvkmc+CGZGwnL0N1ltTytJXHRju1XY7x9+eqLgtc3KpUrrBhhUkFxYCe1h\nU6sR10O5SLtVS8uNBqvfQ0QKCBecN1ExKb5eNE3btCqxYqWRH9O+2bbVUui6nnP8alFspUdE9nn4\nzN0nMDpvmnZ2tjTg4PYoJk5NoH9wqOzniO+wkEijLRxAS7QBGcNAwuqsdu8LU/jFcXMycfv9Hhue\nxzcfMR3Kok3n6aVUjkkrf7/88grDs+6lQQYGBpBOpzclB6SUBnFiKqtt/c/XnmU/3rZtG4BswyCv\nNIh42rS1b1SD2NZYvPLrZt0r4pyE1iEg2qIBAIRFS0CUa3i1mtTAgF1MsZpIAVGGYhEnpQRHKbV1\nYGAAg4OVJVKtR/1dXFzEyMhIVerNuDE2NoahoaFNLzBW6XefnZ11fX9LyI8D2xuR0Rj9pysvoR1P\n6Qg3BNAaMSeVpWQazIxbHx3DfzxcWN1XXAf/ep/5m17a15rT7SyW0pDRDQwMDBSESA7P5f5GySKh\nsYZhYGhoyLW6cCU4Jxrn7+QUXgAwOhdHU8iPL193GIf72lw/R1WAtAfJisyMRFoHQGhqyBUQa0VV\nCC/f1w7ADEUenY/b313X9Q0t2MTnpDSzH/V6TExtll9kKV6ZgIhnDPhVpWzb1c1ACogyFPuRVlZW\n7NVHOu0egVJNih1TTFhe9bIQx6tGfZtiOIWBEISr6TxnNYADO8wwwONT5ROk7OJwGQ3hoB+tYbNG\nzlIiY9cmAoBU3iQuEvZExM9vvWx3zor16bFFvPEf7rc1GycryUyOSaKcz2IzflO3a0ZoJv0zMZzV\nGS143XkP+FXFUw0CABotE9N6NYhMJoN91vf69mOjthkSMO8j54KNmddkNbAz04UGsUYBQURojwTA\nABYTphZbrhBoPK2ty5S1HqSAcKHS1atYeQ0NDa0rpnot5NuOh4aG7O5f1SqjvRaqbdZyfv/5+fmC\n10X00RUHzPIr+7c1Ihr0oS3ix+nFJNLptB3nX4p4SkckoKLVqpGznNDwecsuDwDz1ipvIZ62E/EW\n4xmsJDW87dBORIM+/M4rdmNni+kYfWhgDgRgYKbQfJTKGGgO+/EP1x7CBd3NSGWqP/E6Jx9xPnRd\nh8GMudU0ulpKl9H2q+SZDyKe1lxNTJUi3p9IJGwh83PLTHhk3F2rXFxcxPDwsB1y6hyPE8MwcrLp\nRQTbeibuxqAPPiXbs6KsBpE2EA6oZ2Ym9ZmEaKQCbLy0tBtuF4DzmIJKjq1p2rpt2OvxMWQy7g7e\nSim1r9v3EI67i3e34ubfuQS7282GMU0hP5ZTGQwNDZV1SDIz4hkdkaAPLZbjcDGRzpncRWTTh+94\nFh+703RCPzo0B2bggm4zZyKoAG+5cCcAU0sIkIb8wk6xlIbBmRgafCqiQR+CPsU23fzHwyP4xA+e\nx3IVmtzkaxDiPD82PA/DYLSGS4eB+lXFs2quibRwUpsCQkzGla7wzzrrLDt0ON9e//DgHJ4aLQzu\nENd6KpWCYRhFF1/j4+MYHBy0ExnTmfVpEIDZX7sjGsBSPIOT0zE8OjiLWx8dxU+ecw9aMDUIbzIU\npIBwwTk5lZvoxGRVbnVTyjzgxUrAzfehaVpZ01gikcDQ0NCaIqV0Xcfg4KCt4Wz293OLZhECIhJQ\nc5Khmhr8drvJeDxeciyxlAawKVQiARV+FfjhEbOu0qFec/L/zyfG8Z0nxu2yGoMzMdz2uNm+tCNq\nTq6ZTMZ26J5eSiEIHY8Oz9vmqYxumFne8Yyt+QT9ClKagXhaxy+Oz2BkLo6HBuYKKsOuB+e15xSu\n3/zlEO4+Yk5w4nvu6ShMKsw1MZEn/SCArImpKS+cs1KNWVEU+1qJOnIGfusyMzHm3x4aLjDrOZv2\njI6O2hGO+ddNvoYhzkkltZjcaI8GMDgbw9//6Bj+6Wcnce8L0/jiLwqjK5kZq2kd4aBPahDVYi0n\n9vTp0yUnx1KOQzEBp1IpDA4OFphG1qIyu4UnlkLX9YLJP//5wMAAJiYmSn6OmFzyb4hyxwY2V6vK\nF9r5iAqk0bzkoaaQDyuOhvWlzp1Q8ZvDAfu3EZ973ct3AwDG5uP48dHsyu5v7zpmP3aaF3x5DsST\nUzH8/U+OAwBG5+O2diAyvgOqgoXVDP7nrU/Z+3zniXF84Rf9+NAdR3Dro+sruRGLxXJ+O3ENjMzF\n8bV7nsJNt/03FuJpTCwm8bZDO7G7PWyHtbrhqYkpo8GnKAWr8vWEu4oV997OCK48exu6WhoQT+n4\nxiO596+zL/RafBGiY113BdnzTsR11h4J4PSSeTwxK2xvdo++SqS1dUVLrYcXpYBYK8UmR2eIpVsy\nUjqdxujoqD2hrTWyaCMrhJmZGfT39xd9XUwU5ca0nsiRzahiW65mUj5CCEQa8gREgx8ryWzly1JO\naqFptIQCYM7tvdzY4Ldt4W68fF977kpbcZwD6+HoXBzxlI47nhy3X2q3tI5i0S/PjC1hdiWNe1+Y\nLkj0EpTSTvMnOcMwixj+08+yfpVHBs2Fi5jcdu7cmbNP/vfyykmdSBtodMmiXs/Cqi0SwHUv3433\nWeG7F/WYGuELViJl/vXhlqFfionFBJoa/OiMFg+pLYXTtPfrF3ejJezHctxdU1pN64hIE1PtyL8Y\nKik1XGx7KlW+jO9aBUGpWPb8bcX8DkKwlVuNbUYj9c1WhaeXU/jYnUdzqrUuxDMI+hSE8jp6NTb4\noRvZLmliMn14cA43/fAFPH/KLMuh6zqWLSHTFg3CMAzk/2rhQPazfWr21XO6mvCeV+3Jea9Tg/jI\nr5xtC5fP3nMcx0+bkU+fePt5+Ms3v8T8jJ1NOftf2tda8L0/99OTBSaRRCKBwcFB18KFQOF1qes6\nvvXYGBbiGfz2y0ytaNxqx9kSDqC9vb3k9epXyRMfhK7rWIhnsK2x0Gm+luvJqTVffqDTzlF526Fd\n2L89CiPPhFdJ//R8oTu+EMcDJ2exqzW0rsURM9tBEQDw6oOduPLsbUhqekHoc9bEJJ3UdUM99XUo\ndlEU297f3+8qJMQ2VS2tqm5EQGzmBez8rOOnl3FqMYmP3fmc7XuYXEqgJVxY9VPYr5eTZsTJ3Nwc\nAODLDwxhaHYVb/r8AwCAubk52+zT1hiCruv4v79+LppCPrzj4l0AgIB1rv7qV8/BF3/rYvzRa/bh\ns9dciA++4QAAYM+erJBwCpDd7RG894q9AIDhWXMyPrgjiq7mBtsBu6c9a/v/x2sP4XXnmA0VD/W2\n4H+94QDO2dmEmdU0fjkwl/P9hFPV2evAiZjwJpcS+IvvHMGTw3MYnY/jZXvbcfFucxU9vpBAnANo\nDvntCqNO8n0QXmgQhmFgNp5xjarajOtKVQjn7WrGQjyDnx+bhm4JPbGASKfTWIhn0D8dKzhefpn3\nL91v5rdc2F1Y2LFSmoPZcxzyK2aJc5iRZU6SaQ1pndHU2on29vZ1H69SXpTF+spdYKU0iGLvq+Si\nLefsLjZZl9IEKvlsN1OTM52/1LEr+fxy+60nR6TU8ZzF4j50x7M4sL0RRyeWceXBwg6DwvbszGV4\ndiJ3tS1KLS8lNfhVQmODH0tLcXQ3qvjsNYfs9/3BFXvwwyOT6LZWihfvzq7y9+3bl1Nsz+cwMakK\noas5O9Gdt6sJN16ZzVQGgKBfxe72MC7Y1YxI0Id9nVF8+brD9uu7WkN437efxYmpFTuU18nq6iri\n8TjC4bC9LZlM2gLxlodGMLeaxj/ecwIGCH0dYdtMMb6YQJwjOO/gWWiMlLahh/wq5pIbj65iZrtY\nomEYUBSlIKFvLpbB4T3Z8fT19WF4eHhN16KqqkXvnX2dplD+xiOjuPDcg7igK2L7zb7z2DC+97QZ\nuv7al56fk/zoJKMbOL2cxCv2tePNFxQW1iuHuP9C+mrOtqYGMw9nLpbCrpbsOVhcTYBBaGuOugrz\nzUZqEC5UKiDGx7O25FIXbTkNJJFIoL+/v6BLmvjMsbGxHJW3lIlprf2PNU1Df3+/a26B2/vX8tmA\n6cR3yxHJFxyicFolBdRufyJ73hNpHc+MLQIArj5vR8F7g9aNnXGsep8aNd//GkugDFolLpbiGTSF\n/EWrqnY1h/D7l++FqhSe4/x9hIlJaDBNIb+dG/HSPe32hOP3Z00Lf/Wr5+BtF+1yPbbwgRw77a4p\nAIWmD2duinN+M0DobglBVQjRoAqwaV4qJxwAM1KrVCOlSllYWMDg4CCGhobQ39+Pqamp7PgMA/Fk\nGotJPWdyLLeIccN5fgFg+/btdonzA9sb0WuFQ/dPZ535952YsYUDADzwxHMFn6vpjE/ffRx/+B9P\nQjcYl/a1rdvSEIvFEM3znTVaGsRsLPubrqysYGpxFQHoOQuOaiIFhAv5kT2V/PAbUXvFhVnMYex0\nkq/VxFQM8X4xSZdr4bne71csEmRychInT2YdpSdPnsTY2BhOnjzpmq/gHEdGdx9Lh4uDMOA3L3Gh\ndegG4/GReVyyu9USEIw56yZcTmbMldsmmBSFbXt7U/ZG/oPL9+Lle9txQXezvc3Zn6EcPW2RnHpJ\n+eSPWywqdIMxsZDAy/e149pLe/COi7px0Moy32aNz+3cuX1uZzSYE567XsS1Lkw6Th9Kf38/Ts0v\nw+BczWs95s6mplzfTiCQjVBTiPBXb34JmkI+/PzYlL04uevIJPZ0hPHxt54DALj/5Ey21wdMreHD\n3z1iVwr2qYTzdjWt+7ph5oLS3R3RIAhmGLUz12oxnoECzrmuqkktWo72ENHPieh5InqOiP7E2v5x\nIpogoqetvzd5NaaBmVhOvLlzZZvSdGhrNJFUMokXy+qtxBxTqZN6LazV7FbJe0vt47SZi+8sBOHy\n8nLRfUVJjXdcvCvHBFOMQF6J6qnlJOIpHRf1tqAx6AeBMbNkCsflRAbNYf+awihbWlqwb9++gu1d\nzQ14x0U7ccOrs6/1tIXxnsv35ITirqUHRG+7KSCKXavOCUrXddu89NToImIpHRf1tOB152zHu17W\nZ79XTDR7OitrqiRyPYRje72IVXz+99d1HcyM5YSZRe10Uosxr6UnSlNTE3bv3m3vQ0QFQSdXHtyG\n+07MYGopgYnFBOZW07hifyd2tYTQ3RrCfcdncMvPnrX3eXx4AQurpmD7w9fswwded8A0C+UJo7XQ\nbtX+etnZ3ea4Q360hQO45UcP42dPmuHRzGyXiC8l0DeTWmgQGoA/ZeZzALwMwI1EdI712ueY+ZD1\nd1e1BuCcfPqnY/i7u47hsr+9B0fGF3Pel9YM3PiNp/AP9xQPFxWstdnLxMREQZ6Arus5q+r1mnU2\n8/3VcjQ7WUs/DhFv3mVNbB94vekg3r/dPXY/4MsVENlqryFEG3zoUOI48kI/mBkL8TSawg05E4hb\nwxYgO0n5/e4mKSLCmy/YabftLMZazCY9LQ1IZoycuk7FBIQz9FVUjT3XipRyHlP4Sl6yM6vVlEII\niN/4l19WPG43xLjz/QNikbCayoBBaHFE9yiKgr6+vqK/STHytbT8lf45O5vQpiTQP71iZ82/ZKep\nEfzZ1QcBAM+PZk1gDw1kW9FesrsVB3c0ore3d0NO4+awH5/89fPx4V8939523k5Ty/v47U8AsExv\nVvmRctfVZuG5gGDmSWZ+0nq8AuAFAO6GVw8QoZKzsRQ+8t1nc14bsW5EZ8q7mLzmV9M46YhwcDoH\n8xHvcZqK3JLIKskQLaZ5VFuD2IzPcqvX5ObENgwDi4uLBe8DgAdPzsKnkG0iOXdnEz7zGxfiA687\n4HpMISCWEhl88b4BfPmBIfhVQk+raYO/bHczHhtewMmpGFZTOi4+Z3/OBOJsySnYt2+fPels1By1\nloVFV6MPAWg5HfPyz7XbuZ9fTSMcVBG0ci2cxxThta/YV9nktqulAQBjJZkpWn02H7drVgiGfN+Z\nnWRplfp2CggArm04K0H8XqqqFvxm3S0hNJCGT/7wKJ6dWELQp9id5yJBH15xVjtemFzBcjKD6eUU\nXphcwQXdzfi7X8tO5uvt9OgcS0c0mPPbXHu4G2GrREhaM81f8bQOv0II+ryZumvqgyCiPgAXAXjE\n2vQ+IjpCRF8losJA8CogmpJ0KqvY7Qg11A3Gw4PZkMKjVuSLCIf88+8cwd//6Bh+aSUZ5d8AI3Nx\nLMQzeGRozt63GBux8y/E0/j8vSexEFtb34qNaBCVCqlKBZmbWa1Y2ObRiSVcvLsVEYeZpjnstwUB\nkHvTiRvpjicn8IRVwfOKA51obTUvr4utEhrffMysRnvR7vLORp/PVzCxFUMcp9RnVUp3awidDQa+\n8uBQgQ8pmdHxv//rGXzpB/+NxcXFnPM8PLeKlvZt9nOnBnFpXxs+/66LcF53YXlvN/wK4WOXt6JH\nWcLppcqc1SdOnMDk5GTONudv/sjQHGYtLUeMO2611WwpUxuqUjo7O9Hb25vjgxAE/Spe/5LtCEDH\n46PL2Nm7G93d3bamctXZ25HM6Pjgt5/BR777LIiAd7+8D50l+kysF6fwCwdVvOtSsyzI+IJZJiaR\nMXMgvAq9r5mAIKIogNsBvJ+ZlwF8EcBeAIcATAL4TJH9rieix4no8c3oBnX1uTvwlgu60AANmuXI\nbGxsxEMDs7jvePbzxxxq/cNDWcHx8OAc/v2XwznmqblYCp/4wfP4s/98Bl+6fwjXffXRipx65SpI\num2//YkJHBlfwg8ffhaZTKbqF04ymcSJEydw4kRhd64TJ05gaWlpTVnQxYSNGynNTJ7aVaKcQV9f\nH/8s5dMAAB3ASURBVPbu3Ws/d6uZ/5uX9tgrvldZK+fx+TiIgO7WcMkWnWIlWumYFUUpGY7o8/nQ\n29tb9PW+vj77sV9VcM35bTg2MonTy9kOg6PzcfzxN5/CL144ja//cgSTjoig5WQGw7NxvHJ/NsIr\nfwUeDqhrWpWLrN+J+cpLqeQLfE0zs9vHF+L40v1D+OSPjuXUBptbTcOvKohsUkkJIsrxQwC5wvmd\nl3RjZ0sDlowG3Pjag4hGo/Y52d0eRk9b1kKwf1s0J7FtI+Sbv/JNjtusbnjDc6swDAOrKQ3+gDf+\nB6BGAoKI/DCFwzeY+Q4AYOYpZtaZ2QDwJQAvdduXmW9m5sPMfLizszAevBLyb+63XbQLezsjdqw8\nM5u9iQE723TRUVlTlGR4xb52PH9qGfefmMVHv3vU3je/CicBJStzFquWmslkcPz48ZJjj6XMzz0y\nvlRR1VXxec7PiaU0PDs6l3Os8fFxHD9+vOB45cp6F1v5Fxt/sW1u75mLmd9P2MHzISIEg7lqOlG2\nDeRVL9mGT73zAigORyWzYZsTLtvThoAvG4+ff7OGw2F0d3cXHLMU5YS2c+JyI1+49LWH0UJJPPXC\nAI4fP46lpSXcd8JcyFy+25zEHh6YsyfaIcumftm+rIYgzGbOca1JQFjn6/TC2n9rwLwG0+k07np2\nEh//3vMAzEz4h59+HjMzM3hiZAH3vjCN87tbqrLgEec0EslaDFSF8Jdvegm+/4GrcNlec9HgPHZv\nW/Y3+r0rz8sR6hvJRxDXWEdHB3p7ewvCckVvkeHZODRNw+hiGm3b1p5vsV5qEcVEAL4C4AVm/qxj\nu/NbvwPAUS/H1eBTc1b5K8kMIkEVL9/XZjeOEYzOxbGzpSHPMcoYPL1grqJXsiur3zjcDYDtao8n\npmL49N3H7ZIJD/bP4n23PoUbvv6EHW4pKBYi6rzxRL2gp0YXc5xnpTh+/Ljt79B0xvu/9TQ+9eNj\nORmywkeSb/4qN5GoqlqwTynWkki3aPViaA25C4hiN2qXFUt/xaGz7e5dOQ5dKyroV176kpzX8gVE\nNBq1t1VqYqo0U71Stjeb32V0yjSXZXQDD/XP4sKeFtzw6n3obgvj+8+css+p6C3tNJ+Gw2H09PTk\nZH6vBbF6npiazbmW8n/H4eHhgkZSzvfcfczUxF9t5aOcnI7BYLarmF7s0tVuM2hvb0d3d3dBUcKg\nX8W+7dlIJOfq/vVWZvvH33oOLtyzLUeol9IAK0VRFIRChaU6okEfQn4VwwMnMTG/imMLBl66t2PD\nx6t4XJ4dKcsrAbwbwGvzQlo/RUTPEtERAFcC+ICXg5qNpfDU6CLueX4KzIyRuVV0NgYRUBX41Ozk\nlNJ0HJtawQXdLbZ9VEyDf/1fz4CZsbhsOhH/1xsOYHd7BIRsOeBP330cxyZX8IMjk/jM3cdxy0PD\nSKR1aAbjL243neSpjI6vPDCErzwwgGOnl3ME1/GhMRwbMFP7l+KZnMSpu47kJqTpBhctiyA0gecn\ns/6RUxXYlEtNaJ+/9yT+45HSVUf7+/vLakXiOHOxFI6eyjpkF+KmBiFWsPnkr+4FV1s393k92QnH\n+T1uuGIvLulrxWWWGcaOk1eUnLG1tGRLKVQi/Do6OtDR0VFwznbu3GkL2koEhNPM1BIye0dMWr/V\nyFwcKR141VntUBXClQc7MbOSwsD0ivX6KsJBFc0NeVpVOLwuZy9gFhYMB1SMTi/g0aF59Pf3Ix6P\n4+TJk3blY8MwkE6nC3J77LyblIaFhIZ3XtKN376sF6pC+OqDQ7j+382InfN2NeGawxufeN0gIkQi\nEUSjUezatcsWBDt25CZaqqpqT/7drWF8+brD6G7Nnre9e/eir69v3edRjMVJ/mcRETobg/j58Rn8\n5XefhcYK3nSedxqE56U2mPlBoKAOGgBULay1Es7vbsbplRk82D+LfRc2Ymg2jrdc0AUiQkc0iFHL\n3jo2n4BhMPZvi2JPRwQHtkdxsKsZgWADvvzwKUwutSBtJdU4zRU/PjqJ3vawnUAl6u+78f0jk/jl\n4BwSvIy7KIMLupvxqrM68AVrZdUeDeDvf/0CDM6a8fv/47JeDM7E8IPnpvC+K3bbn/OlBwbxuOWY\nfcuFO/GGc7fbpZNjsRiOnV7G95/JCpW/+cELaO/ajRY9W968Ulv7QjyDI+NLeGgsgRuuOrdg39PL\nSXzlgUH8xiU9UFWyW0CW8kH8/z/rx/hCArt7e9GoLWLREhD5kS2CYqv1i3e34svXHUZbNIiFqcLX\nD+xoxIEdjQj4zP3FJOb8vHA47DqZl5rgI5GI6+QRjUYLNJC+PjM3Ib9vNZCrGRER9m2L4qmxRbzl\nwp0Yml0FA9jbYZ7PC7pbEPSN4Yt3PYF3XdaDhwfn8cqz2u1xNDdXFs5ajj2dETwyOI9HBufR2HAA\nRKamsLCwgNbW1qLlLcT2sfk4dCb0tpnndW9nBCenssma1xzusaOuqkk0GkUsFkMymXTVQPPNf5FI\nxDZN5ZuDNgMiQk9Pj+mfsSo1vPbsbfjaQ8MAzCz4nrb1RUytB5lJbfGbl/agPeLD4tQEZuaXoYPQ\nbGkI57b78NzoLEZGRmxTU3s0gGjQhz9/49l45+FeXL6/AwoYL0wu29pC0Kci6FPQrCRxdGIZ33zY\nfXX9ibefZ4cbpjXDPoZC2dXW/SezDvO5WBozKyn8889NgXFBdzMO9bQCnM0TAGALBwD4/jOn8PE7\nsyUDDGZ8+icnMDQbx0W92ZXx1+99qmxPXIGzAZEQoCoYd79QOAvf/sQ4hmbj+NRPjuPv7jpmlxAY\nGBhwNaUlUhmMWyUdfviEeZyFuGn2K1YXZy24mbXyV/XlnMtAZXkMpcpVO48lfAOBQAC7d+/OcbY7\nufJgJxZWM/jAt5/Btx8bQ2ODH82W0GwN+/Gm83fg+cll/NV/mb/3Wy/ciWAwiF27dmHbtmw000bs\n+++8uBsv3WNqZF//5YhtMs1kMpiamipqNhThy2MLcbBjsvvDV+/DH7/2LPzuK/pw/RV7sbMltK7S\nGuth27ZtOZpEKUr1ytgIzvsqHA7nhFe/an8HvvQ7l+CGV+/Dt977Ck+Lh74oBYTbilUhQldzCCcm\nFzATS0EBo8lSyzuiAaxYjVdiKQ0ZKDlhloqiYFtjEDubAjgxHbPNOgGfgoBPgQ/m88eGzZDYC3ta\n7JaUALCjKYhLrcJvKykN8bSGHc1BNFgVQTWdEQ2aE0B7JACQ2fJS0BzyozVivi6iWzSXchRzq2mz\ncqXBtskMAN58QRfecqGZ1HV0dCbHLMXMGJ5bxYnJBbz1b27DR+94yj5/IhnrvhMz+Nf7zElcJSPH\nl5LWDBw9tWzXPxKIjFAg19/xzUdG8K1Hx/C5e7JRUj9+ehgfvO0ZjMzFy7bELIXbxOycFMS2SCSC\nnTt3oqOjo6ivobOzEzt37izIf3GzR7utNN0cxQDQ09ODnp4eNDQ05OznfN+FPS3odkTVXNDTgl27\nsqlEr9rfmRNl025l3Tojc9yOvRZ62sK4/oq9+ODrD2B6JYX/8/3ncHo5iQf7Z/HR2x7BY4NZf9jQ\n7CrmVtOYn5+3BcfYfAJtEb9dfrsp5Mehnha8an+HLXg2YrpZC4qiVDTxd3R0bJoGJij2G7iZmg73\nteJQb3X8MsV4UVZzLcabzt+BT//kBO58egKEbLP05pAfmgErmUpDin055RLEj3lgWxjDU/O2EzW/\n65MokfDOS3ahqzlkm3eICE3WDf0X3zkCADh7RyM+8baD+MqDgzg5HcPATAwHtkfxZ1cfxOfuOWn3\nMXj/6w/AryroaglhVyCBh54fxTmX78G3H3fXVr7xyCjueHICv/vKPgDAB19/AHs6ouhrj6CnNYQv\n/GIAE4sJu/WkYRj4mx+8YO8/e2ICtwc0TC+n8Iev2QuGuYIEgNee3Yn7ToiYdnP/Lz0waAuHtx/a\nCVVRcPuT43bJDHGMeEbHkbFF/OxYbujy71++B19+YAjLCQPLiQzO2rM5duloNIquri40NjYimUwi\nk8nk3JSNjWYintiWn69ARPZ7nIRCIfj9/pzopW3btiESiSAUCtnaUk9PD5LJZMEEUSzhcs+ePVhd\nXTXLUC8s4ONvMYsPTC4lsLOtEdFoFGeddZZZdHF+Hn/7a+djYCa2ab2Le3t7sbCwUBClds7OJrxs\nbzseHpyzI/kA4D23PIZvvOssaAbjph+a18/H36qhuzWME1MxPDo0j7N7t7t+z5mZGcRisTVXJ6g2\n1SivXUwTLSYcvRKagvr6BWrMwe3mDS+6TPVZkR/t0SAaKINP/eQ4rj5vBwKqkmPmICIwMy7qbsJT\nA6dxzwtTePm+dluIXHNpN3rbIvi01XJyh1UmIuhTbHPUOV25dVwMZhBZ5ZWt8M53XLQrJ2zz3S/f\njfMs01TYr2JPZwTTK2atoaMTyzjU24I/vvIsMDOOnlrGP95jlvFIZHQ7UqSzMQi/3490Oo0dVmG0\nm374Al61vx0zKym89iVZh1h3Wxjjjpabf/D/2jv34Liq84D/vt29d1+S9qHXrlbSSiYmWOAH2BBD\nwmNKSkMgcYaSQIBCZ8IwzLSTvhsofcBkMhkymZZh0iZNCW3aUJgOSShNQkJIAmmahuAQh4cxDxdj\nMFIsbGwL62np9I/70NU+9LIsybvfb2Zn7567997z3b17vnPO953vcw2KAOevbeGjW7rYPXiUR555\nk2KshdGJyRkjhzWtDaQTFl9/+g3u+r4zQrjtsnX0tiT59H/t9NNvetx84Sls6cmQTtj+vWudZ5Cy\njo6OilFkg3ixc+LxeFVX03g8Tj6fX9DUQjgcZmJiwu8th0IhX5kEG4Sgm+VcWJblG8mDKXDzqbhv\nOwmHw2SzWQ4ePIgVDnFabvGxgUqJx+NEo1FfQXR1dfkeSp94Xw9r2xu4/8m9rOvMMjExDv1D3Pnd\nXQwcmf5Nb394J80Ntv88X3RGkebmNLZt09/fTywWw7Ztcrkcw8PDFVex1xrpdJpwOFzW2SjtOKTT\naZLJ5LLnpqlLBVHNKCoiXPTuVh5/cZBtmzr8pC8bOlNszCXYNTDEs28cKkti4imIjYXpP+SHA1NI\nl/Q53hF3X30mwxPH/B/5zis3+HO3VjhEX76Jnf3OyMBbpRl3l9pHQjLDVgCULbfPJmx+uvsAn3zA\nyWvsKQ8RYX0hxac/cgZ/9dBM7+Fs0iYejzM+Pk5bY4yEHWZ4fJKfvOy4IO4aeBkBrj+3yJrWpO+3\nXsoZhRRRK8KVmzu543t7+NIT5QnXk9FIWarEz3z7Ba59T7evHCIh4TNXrCcaCfkKtjswndJeIcNY\nJSr1xBf751poELZoNMro6OiShi+ZL8EepmdQraaIFno/QqEQsVjMN+j29PQ4C/X27uXCU1u58NRW\ncrkcAwMD3PXYSzy37wi5VJSLT2snbod55NkB9rmhbT52dicf2lQgmZhWzF49KzWYK0mxWJz7S4tk\nvkH+EonZc4WfKOpSQQTxXP68efDrtha5+uzuGRnBQiJcvjHProEh3jw0ytbTyv2QJyYmSEQMa1qT\ntDXGKi7DT0TDfmwVYMY0FcCNF/QyfmyK1w8Oc0bBmevsTDsN3fDU9KjFC1TnZSPzaC+JER8rmeLK\np2J8/qMb+bMHf4UxjmdTOCSk02kikQgHDhzgjm2n89Seg/xg536a4hFedTOgbepOz0ge/9kr1jMx\nOcVfu4bvnpYkkUiEdfkmzu1N8+ye/fR1NHFOb5Z/+Z89ACSjYRrj5Y/cfa5r7LXv6ebcU5rL8jPH\nrelGry0VA8oNoKUG3UoGzuXqfbW1OX7ys8XnWgry+Tz9/f1VbRXpdHrJG5XOzk6OHj1KOByueI+9\nshvPX8OLA0Ns6kr7+TO2rmnmS0/sprclySV9OazI9PHHEwn1RLOQkOxLSVdXF6FQiLGxsRVTmHWv\nIDKZDJZlzQieF1QOHh2p6Z7OqRUihx475owMvnDjxTOmNnrcLFjVSKVSfix8r8FvaYj6oxLPu6kr\nM93YXL6xg56WJOvyMx+ac3qyfPPpfbQ2RrnmnG7e1e4YJYMeJemExZ/85qkYYJ07rWVZFs3NzRw4\ncIBMwuaSvhyX9OUYGp3gHx53/tBe3c5f28KkMbQ2Rv0e8uaeDM1J2++93nB2nidSES7fkCccEorZ\nBI+/NEg2aRMS4Z+u38z45BTRSJjHdv6aB9w4SJuLmTLlUFp/J/xzeajpE+FyuFhCodCSGzMr0djY\niDGmTAnYtj2vVfWLIRwOlzXmhULBz6HiKYiGaISLzuimpaVlhutuMPz5arMxrDa8DsZKKShQBeE3\nxHPRFLf47BXrOTQywcbedoaPzkwR6FGq6W3b5pRTTmH37vIpl1wuh2VZFRPORyIRJiYmfFfajuZG\nvCV54ZCwsas8/21rY5S7rzmTY5NTNMUsYrGYv2ApyGkl9g7vTx2NRhkbG/PvSWPM4lMfOM1vcESE\nG87rmSH3F687i1BA/nA4TDoB2zZNT7F1ZRP8ztbijOOibu/x/X3tnF5oYsfewzTGyh9HT0Fs6k6z\nY+8hR0GMOwqi2n316OzsJBQKsXfv7Iv3TlZEpKIiKhaLHDlyZEE2juMhqKCCirqlpQXbtsuUvMdy\nG1yVhaO/UIBKw9xgL6e1McratgasQFmxWCzzlw+6OopI1Z5SKBSqOg0RPObU9gbSsfLhfKXzJqyw\n39tvbGysOq2Sy+Vobm6e4R6ZzToudKU9Uu+zp0iC57TCIX8KwRizqN5OPhXnY+c5YbZt257xO3iN\nyE3nr+GmC9awIeDmN1cPNJlM+vWJRCLLbuA7UcwlRygUIp0+MXGM5iL4m3hG5u7ubgqFgq88uru7\nq656V1YXdTmCqBaWOvhw9/T0MDIyQkNDg99LtSyLTCZDKpUiHo+TTCb90M9e9i5wPD683utseMqh\n0ijGq0s8HmdkZMSPdxMKhWhvb/fj4g8ODpJMJhkYGKAUY0zVOlTqeXrXLM1Lkc06IbDD4TD79+/3\nRzeVrheLxSrmushms4yNjVXcB869GBoaIhwOk0gkOHLkCLZt+z1POxLinN7sghdPiQj5fH7R8fpX\nIz09PVXjdC2Ujo6O4wo2V+2cQQ+kaDTqv4aHh2vqt6h16lJBBAmmIAyFQn7D7j3QQTo7O/0HP9jA\nVurJzjW8j8VifmPX09PD6Ogok5OT7N+/H3BsI5FIhFQqNcOGUer10NHhTOUcOHCgrNEeHx/HsixG\nRkawLAvbtjl69GjVnrdXLiK+Nwo4I4eWlha/ca+mIKampvz7E4/HyWQyvj3Gi7xbGofJo6GhwV+B\n29TUxOTkJOl0esZq7cWymg2gi8G27SVzAV0q42exWPQ7FtXOaVnWsthmlKVDFYQ4XjxTU1N+T7ka\n1f6U8zG2eaMEzxgc7NkH//Cegqjmmz/bdJVHJpPh7bffxhjjjzYymQwiwuHDh6v24Gzbpq2tjcbG\nRiKRCJZlVUxpGYlEyOVyTE1N+fUFR5F45zbGVGwoWlpaeOut8qiz3jljMSflpzfdlcvlGBsbw7Ks\nJe/pKkvHShpSlRNH3SsILwXhbKski8XirF4hIkKhUJhVuRSLRUZHR0mlUoRCoaq92s7OzrLz5PN5\nbNtmZGSkau+sUCj4ve22tjYikQhNTU3+4imPuXpwwQxopfaRRCJBa2srqVSKcDjMO+9MB1dra2uj\noaEBy7Job2/3j83n8zMa9ubmZj+0QenooFLdGhoaymwiwSmWQqGwoGknTwkpijI3shKLeZaKLVu2\nmO3bty/4uNHRUV577TWi0SjFYrFmjJdDQ0NEIpFlm+M1xnDw4EF/NehCGR0d5dChQxWVgKIoJw4R\n+YUxZstc36vrEUSlWP0nM8u9mGaukddcxGKxshj8iqKsHtTNVVEURanIqlMQIvIBEXlRRF4RkVtO\nxDW84Gm6klNRFKU6q0pBiEgY+HvgUqAP+LiI9C31dWzbpqOjQ42ViqIos7CqFARwDvCKMeb/jDHj\nwAPAthWuk6IoSl2y2hREAXg98PkNt0xRFEVZZlabgpgTEblJRLaLyPbBwcG5D1AURVEWxWpTEPuA\nrsDnTrfMxxjzZWPMFmPMFi98g6IoirL0rDYF8RSwVkR6RcQGrgYeXuE6KYqi1CWrys/TGHNMRH4f\n+B4QBu41xjy/wtVSFEWpS1aVggAwxnwH+M5K10NRFKXeWW1TTIqiKMoq4aQO1icig8Brizy8BSiP\nO13bqMz1gcpcHxyPzEVjzJxePie1gjgeRGT7fKIZ1hIqc32gMtcHyyGzTjEpiqIoFVEFoSiKolSk\nnhXEl1e6AiuAylwfqMz1wQmXuW5tEIqiKMrs1PMIQlEURZmFulQQy5GUaLkRkS4R+ZGI7BSR50Xk\nD9zyrIh8X0Redt8zgWNude/BiyLyWytX++NDRMIi8ksR+Zb7uaZlFpG0iDwoIrtE5AURObcOZP4j\n97l+TkTuF5FYLcosIveKyH4ReS5QtmA5RWSziDzr7rtbFptb2RhTVy+cEB67gTWADfwK6Fvpei2B\nXHngLHe7EXgJJ+nS54Bb3PJbgDvd7T5X9ijQ696T8ErLsUjZ/xj4d+Bb7uealhn4KnCju20D6VqW\nGSfk/6tA3P38H8Dv1qLMwAXAWcBzgbIFywn8HNgKCPAIcOli6lOPI4iaTEpkjOk3xjztbg8BL+D8\nsbbhNCi47x9xt7cBDxhjxowxrwKv4NybkwoR6QQuA+4JFNeszCKSwmlEvgJgjBk3xhyihmV2iQBx\nEYkACeBNalBmY8yPgYMlxQuSU0TyQJMx5mfG0Rb/GjhmQdSjgqj5pEQi0gOcCTwJtBtj+t1dA0C7\nu10r9+Eu4M+BqUBZLcvcCwwC/+xOq90jIklqWGZjzD7g88BeoB84bIx5lBqWuYSFyllwt0vLF0w9\nKoiaRkQagK8Df2iMORLc5/YmasZtTUQuB/YbY35R7Tu1JjNOT/os4IvGmDOBozjTDj61JrM7574N\nRzl2AEkRuS74nVqTuRrLLWc9Kog5kxKdrIiIhaMc7jPGfMMt/rU75MR93++W18J9eC/wYRHZgzNV\n+Bsi8jVqW+Y3gDeMMU+6nx/EURi1LPP7gVeNMYPGmAngG8B51LbMQRYq5z53u7R8wdSjgqjJpESu\nl8JXgBeMMX8b2PUwcIO7fQPwn4Hyq0UkKiK9wFocw9ZJgzHmVmNMpzGmB+d3/KEx5jpqW+YB4HUR\nebdbdDGwkxqWGWdqaauIJNzn/GIcG1styxxkQXK601FHRGSre7+uDxyzMFbaar8SL+CDOF4+u4Hb\nVro+SyTT+3CGns8AO9zXB4Fm4AfAy8BjQDZwzG3uPXiRRXo5rJYXcBHTXkw1LTOwCdju/tYPAZk6\nkPkOYBfwHPBvOJ47NSczcD+OnWUCZ7T4icXICWxx79Vu4Au4i6IX+tKV1IqiKEpF6nGKSVEURZkH\nqiAURVGUiqiCUBRFUSqiCkJRFEWpiCoIRVEUpSKqIBQlgIhMisiOwGvWaL8icrOIXL8E190jIi3H\nex5FWUrUzVVRAojIO8aYhhW47h5gizHmreW+tqJUQ0cQijIP3B7+59wY+z8XkXe55beLyJ+6258U\nJx/HMyLygFuWFZGH3LKficgGt7xZRB51cxzcgxOW2bvWde41dojIP4pIeAVEVhRVEIpSQrxkiumq\nwL7Dxpj1OCtT76pw7C3AmcaYDcDNbtkdwC/dsr/ACb0M8DfAT4wxpwPfBLoBRGQdcBXwXmPMJmAS\nuHZpRVSU+RFZ6QooyipjxG2YK3F/4P3vKux/BrhPRB7CCYEBTgiU3wYwxvzQHTk04eR0uMIt/7aI\nvO1+/2JgM/CUmwQsznRwNkVZVlRBKMr8MVW2PS7Dafg/BNwmIusXcQ0BvmqMuXURxyrKkqJTTIoy\nf64KvP9vcIeIhIAuY8yPgE8BKaAB+G/cKSIRuQh4yzh5On4MXOOWX4oTcA+coGxXikibuy8rIsUT\nKJOiVEVHEIoyk7iI7Ah8/q4xxnN1zYjIM8AY8PGS48LA19yUoALcbYw5JCK3A/e6xw0zHbb5DuB+\nEXke+ClOSGuMMTtF5C+BR12lMwH8HvDaUguqKHOhbq6KMg/UDVWpR3SKSVEURamIjiAURVGUiugI\nQlEURamIKghFURSlIqogFEVRlIqoglAURVEqogpCURRFqYgqCEVRFKUi/w9DyWWWZK2XfQAAAABJ\nRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1e2e9fced30>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "eps, rews = np.array(rewards_list).T\n",
    "smoothed_rews = running_mean(rews, 10)\n",
    "plt.plot(eps[-len(smoothed_rews):], smoothed_rews)\n",
    "plt.plot(eps, rews, color='grey', alpha=0.3)\n",
    "plt.xlabel('Episode')\n",
    "plt.ylabel('Total Reward')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing\n",
    "\n",
    "Let's checkout how our trained agent plays the game."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from checkpoints\\cartpole.ckpt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[2017-09-07 21:17:38,803] Restoring parameters from checkpoints\\cartpole.ckpt\n"
     ]
    }
   ],
   "source": [
    "test_episodes = 10\n",
    "test_max_steps = 400\n",
    "env.reset()\n",
    "with tf.Session() as sess:\n",
    "    saver.restore(sess, tf.train.latest_checkpoint('checkpoints'))\n",
    "    \n",
    "    for ep in range(1, test_episodes):\n",
    "        t = 0\n",
    "        while t < test_max_steps:\n",
    "            env.render() \n",
    "            \n",
    "            # Get action from Q-network\n",
    "            feed = {mainQN.inputs_: state.reshape((1, *state.shape))}\n",
    "            Qs = sess.run(mainQN.output, feed_dict=feed)\n",
    "            action = np.argmax(Qs)\n",
    "            \n",
    "            # Take action, get new state and reward\n",
    "            next_state, reward, done, _ = env.step(action)\n",
    "            \n",
    "            if done:\n",
    "                t = test_max_steps\n",
    "                env.reset()\n",
    "                # Take one random step to get the pole and cart moving\n",
    "                state, reward, done, _ = env.step(env.action_space.sample())\n",
    "\n",
    "            else:\n",
    "                state = next_state\n",
    "                t += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "env.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extending this\n",
    "\n",
    "So, Cart-Pole is a pretty simple game. However, the same model can be used to train an agent to play something much more complicated like Pong or Space Invaders. Instead of a state like we're using here though, you'd want to use convolutional layers to get the state from the screen images.\n",
    "\n",
    "![Deep Q-Learning Atari](assets/atari-network.png)\n",
    "\n",
    "I'll leave it as a challenge for you to use deep Q-learning to train an agent to play Atari games. Here's the original paper which will get you started: http://www.davidqiu.com:8888/research/nature14236.pdf."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}