actor_critic_second_agent.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "actor-critic-second-agent.ipynb",
      "version": "0.3.2",
      "views": {},
      "default_view": {},
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "metadata": {
        "id": "MSTjbSrh_NBJ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Actor-Critic (AC) \n",
        "Below we will implement the actor-critic method with eligibility traces for the second agent. The implementation has $3$ main parts as follows:\n",
        "\n",
        "\n",
        "*   **Helper Function:** Here we will import the libraries we need and implement our helper functions\n",
        "*   **Implementation of (AC):** In this part we will implement the (AC) method used for the learning of the second agent\n",
        "*   **Results**: Using the (AC) method, we will train the agent and produce our plots here\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "9gInkSfeI2Hq",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Helper Function"
      ]
    },
    {
      "metadata": {
        "id": "U3yPZDws_Vvh",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Uploading The Environment\n",
        "Below we will upload the environment from our repository and import the libraries we need as well."
      ]
    },
    {
      "metadata": {
        "id": "4DQgreZoy-og",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "3408bca7-4afd-4f75-85cf-656f4f4329ef",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524408794990,
          "user_tz": 240,
          "elapsed": 1932,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "!rm -rf comp-767-Final-Project/; git clone https://github.com/navin-mordani/comp-767-Final-Project.git; cp comp-767-Final-Project/environments.py ."
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'comp-767-Final-Project'...\n",
            "remote: Counting objects: 59, done.\u001b[K\n",
            "remote: Compressing objects: 100% (31/31), done.\u001b[K\n",
            "remote: Total 59 (delta 12), reused 39 (delta 10), pack-reused 17\u001b[K\n",
            "Unpacking objects: 100% (59/59), done.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "EZW2-0ur_wWy",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "from copy import deepcopy\n",
        "from matplotlib import pyplot as plt\n",
        "from environments import EnvSetTarget\n",
        "from environments import EnvChaseTarget\n",
        "import random\n",
        "import sys\n",
        "import os\n",
        "from scipy.stats import bernoulli"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "my2Wj1lfFLOA",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Random Step Taker\n",
        "Below is a simple function that allows us to take a random action in the environment."
      ]
    },
    {
      "metadata": {
        "id": "eDLBVglMFNwe",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "def random_step(max_step):\n",
        "  return random.randint(0,max_step)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "I-JURDYDGUFH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Random Target Producer\n",
        "Below is a simple function that generates a random target for the agent to chase."
      ]
    },
    {
      "metadata": {
        "id": "Q7ZzCF9oGYiy",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "def random_target(max_target):\n",
        "  return random.randint(0,max_target)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "t7ddMQ9RjZd7",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Softmax Function:\n",
        "Below is a Softmax function, specifically designed for our purposes in the learning of the second agent. It takes in a feature vector $x$ and paramter vector $\\theta$ then it returns the softmax of actions 0, 1, 2, 3 in an array."
      ]
    },
    {
      "metadata": {
        "id": "-HWvHw8mBeC7",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "def action4_softmax(x,theta):\n",
        "  act0=np.exp(np.dot(np.append(x,np.array([1,0,0,0])),theta))\n",
        "  act1=np.exp(np.dot(np.append(x,np.array([0,1,0,0])),theta))\n",
        "  act2=np.exp(np.dot(np.append(x,np.array([0,0,1,0])),theta))\n",
        "  act3=np.exp(np.dot(np.append(x,np.array([0,0,0,1])),theta))\n",
        "  act_sum=act0+act1+act2+act3\n",
        "  softmax_return=np.array([act0/act_sum,act1/act_sum,act2/act_sum,act3/act_sum,])\n",
        "  return softmax_return"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "eaYXCls6y2l5",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Picking an Action\n",
        "The function below takes as input a vector of probabilities of actions, then based on those probabilities it picks an action and returns it."
      ]
    },
    {
      "metadata": {
        "id": "Z119cxWjGfBb",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "def choos_action(x):\n",
        "  y=np.zeros(x.size)\n",
        "  a=random.random()\n",
        "  for i in range(x.size):\n",
        "    y[i]=np.sum(x[:i+1])\n",
        "    if a<=y[i]:\n",
        "      return i"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "89H_fikSBlkT",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Actor-Critic with Eligibility Traces (episodic)\n",
        "Below function is based on the algorithm in Sattun and Barto page $274$. It runs the algorithm for one episode of the problem and then returns the parameters $\\theta$ and $w$ at the end of that episod along with whether the episod was a success or failur (i.e. the reward of $+1$ or $-1$)\n",
        "\n",
        "The actor is defined as:$$\\pi(a|s,\\theta)=softmax(h(s,a,\\theta))$$\n",
        "Where $h$ is the parameterized numerical preferences for each state-action pair. We define $h$ to be:\n",
        "$$\\theta^Tx(s,a)$$\n",
        "This means that we define our preferences to be linear in features. Where the feature vector $x$ is simply given by:\n",
        "$$[1,s,1,0,0,0]$$ $$[1,s,0,1,0,0]$$ $$[1,s,0,0,1,0]$$ $$[1,s,0,0,0,1]$$\n",
        "For action 1, 2, 3, 4 respectively. The vetor $s$ here indicates the state of the agent. Note that under this definition we have:\n",
        "$$\\nabla_\\theta\\mathrm{ln}\\pi(a|s,\\theta)=x(s,a)-\\sum_b\\pi(b|s,\\theta)x(s,b)$$\n",
        "The critic function $\\hat{v}(s,w)$ is given by $w^Tx(s)$ where $x(s)$ is the degree 1 polynomial feature of state $s$, therfore we have: $$x(s)=[1,s]$$. Note that in this definition we have: $$\\nabla_w\\hat{v}(s,a)=x(s)$$"
      ]
    },
    {
      "metadata": {
        "id": "TwT5EedEriNf",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "def actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w,max_target=36):\n",
        "  target=random_target(max_target)\n",
        "  env = EnvChaseTarget(6,2,target)\n",
        "  observation_old = env.reset(target)\n",
        "  z_theta=np.zeros(theta.size)\n",
        "  z_w=np.zeros(w.size)\n",
        "  I=1\n",
        "  done=False\n",
        "  while not done:\n",
        "    x_old=np.append(1,observation_old)    \n",
        "    action=choos_action(action4_softmax(x_old,theta))\n",
        "    if action==0:\n",
        "      action_f=np.array([1,0,0,0])\n",
        "    if action==1:\n",
        "      action_f=np.array([0,1,0,0])\n",
        "    if action==2:\n",
        "      action_f=np.array([0,0,1,0])\n",
        "    if action==3:\n",
        "      action_f=np.array([0,0,0,1])      \n",
        "    observation_new, reward, done, info = env.step(action)\n",
        "    if observation_new[2]<0:\n",
        "      observation_new[2]=0\n",
        "    x_new=np.append(1,observation_new)\n",
        "    if done:\n",
        "      delta=reward-np.dot(x_old,w)\n",
        "    else:\n",
        "      delta=reward+gamma*(np.dot(x_new,w))-np.dot(x_old,w)\n",
        "    z_w=gamma*lambda_w*z_w+I*x_old\n",
        "    z_theta=gamma*lambda_theta*z_theta+I*(np.append(x_old,action_f)-action4_softmax(x_old,theta)[0]*np.append(x_old,np.array([1,0,0,0]))\n",
        "                                          -action4_softmax(x_old,theta)[1]*np.append(x_old,np.array([0,1,0,0]))\n",
        "                                         -action4_softmax(x_old,theta)[2]*np.append(x_old,np.array([0,0,1,0]))\n",
        "                                         -action4_softmax(x_old,theta)[3]*np.append(x_old,np.array([0,0,0,1])))\n",
        "    w=w+alpha_w*delta*z_w\n",
        "    theta=theta+alpha_theta*delta*z_theta\n",
        "    I=gamma*I\n",
        "    observation_old=observation_new\n",
        "    if done:\n",
        "      break\n",
        "  return theta, w, reward"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "FY3pRvC8K93F",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Results\n",
        "In this section we will simulate the environment for our (AC) method to investiage the quality of its learning. This section has 3 parts as follows:\n",
        "\n",
        "\n",
        "1.   **Test on Random Policy**: In this section we investigate the return of the environment of ranodm policy\n",
        "2.   **AC Performance:** Here we will investigate the performance of the (AC) on the environment\n",
        "3.   **Plots**: Produce a few plots here\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "FF5vedEcAFBp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Test On Random Policy\n",
        "Note that in this environment the maximum score any agent can achieve is $6\\times 6=36$ with the probability of $(0.6)^6=0.046656$ or $4.6656\\%$. Below we will compute the average return of the second player over $1,000,000$ episodes given that he plays randomly and the target he needs to achieve is randomly chosen."
      ]
    },
    {
      "metadata": {
        "id": "bAmvRKbBAPKJ",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "c693c25f-f5d0-4307-c8b0-6155aaf538ab",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524408866215,
          "user_tz": 240,
          "elapsed": 65765,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "epochs=1000000\n",
        "max_target=36\n",
        "avg_reward=0\n",
        "for i in range(epochs):\n",
        "  done=False\n",
        "  target=random_target(max_target)\n",
        "#  target=25\n",
        "  env=EnvChaseTarget(6,2,target)\n",
        "  initial_state=np.array([0,0,target])\n",
        "  while not done:  \n",
        "    action = random_step(3)\n",
        "    observation, reward, done, info = env.step(action)\n",
        "  avg_reward=avg_reward+reward\n",
        "avg_reward=avg_reward/epochs\n",
        "print(avg_reward)"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "-0.244094\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "ypI3wBlW7014",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Actor-Critic Performance\n",
        "Below we will investigate the perfomance of the (AC) method over a total of a little less than $5\\mathrm{M}$ episodes averaged 10 times. Note that since this code was ran in google colab, we had to break the loops into smaller pieaces so that the server does not disconnect us. "
      ]
    },
    {
      "metadata": {
        "id": "ahCWCxSdp8u0",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "f2cf0e15-7ddd-4463-b14a-9cc01022621f",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524409006169,
          "user_tz": 240,
          "elapsed": 137377,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=10000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=np.zeros(8)\n",
        "  w=np.zeros(4)\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "w31xLqg8z_Eo",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "fe858909-5f07-4e95-d957-95a11b609308",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524409006851,
          "user_tz": 240,
          "elapsed": 393,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.array([avg_reward])\n",
        "improvement_method=np.array([0])\n",
        "\n",
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424  ]\n",
            "[0.         0.69399494]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "4h-B-FmV4ooc",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "3f400d38-2a5c-4278-a0c6-744ac393b522",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524409143841,
          "user_tz": 240,
          "elapsed": 136591,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "oYcZAOCY4tr8",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "2e535e2d-c5be-476a-83b1-25ab3544aaeb",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524409144428,
          "user_tz": 240,
          "elapsed": 367,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904 ]\n",
            "[0.         0.69399494 2.07051382]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "aWn5lwPm5D5K",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "1cceba57-d4a0-4055-dfe3-f7d15ee8df19",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524409279964,
          "user_tz": 240,
          "elapsed": 135080,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "B0U8Erb95GMQ",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "c2c05642-f23d-4a5e-b4b3-9459f9fbb0e3",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524409280451,
          "user_tz": 240,
          "elapsed": 311,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024 ]\n",
            "[0.         0.69399494 2.07051382 9.77246471]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "YtAiiVYz620H",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "012218a3-9204-426d-e912-53991efe0e3a",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524409412545,
          "user_tz": 240,
          "elapsed": 131644,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "-wWm09RO64kX",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "f15bf9ce-c2e4-49c6-ad50-0f4428fd5622",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524409413225,
          "user_tz": 240,
          "elapsed": 459,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "9_yp7j9o67UV",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "337f3ea8-a633-4f04-dde4-1240fdf1701c",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524409545713,
          "user_tz": 240,
          "elapsed": 131987,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "gb8UUV1969Li",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "e020b31f-9aa7-42d2-c278-fc66976aeadc",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524409546297,
          "user_tz": 240,
          "elapsed": 367,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896 ]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "ZM7ZWGv87ckz",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "609aaffd-0897-4fc7-d14b-4a35e76ff8bb",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524409677964,
          "user_tz": 240,
          "elapsed": 131160,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "GiuGae0R8Dwr",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "outputId": "57b7e556-820d-41a9-8809-075bdfd4e930",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524409678543,
          "user_tz": 240,
          "elapsed": 416,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "ZSe6S1FN8G-0",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "78cffd9e-9974-4e97-8bbd-bc96f9b366d7",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524409808900,
          "user_tz": 240,
          "elapsed": 129952,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "2H5q0biv8JNI",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 85
        },
        "outputId": "f8e7117f-3dbf-4709-c0b7-0e98562091fe",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524409809505,
          "user_tz": 240,
          "elapsed": 381,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169   ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "LneUrEsb8f7t",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "4a40b93a-1174-4202-ff38-4583a4430703",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524409937508,
          "user_tz": 240,
          "elapsed": 127539,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "eqZJEzKO8hao",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 85
        },
        "outputId": "f122b4c4-01d1-4c8d-f8fd-10e77dcb6835",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524409938018,
          "user_tz": 240,
          "elapsed": 340,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "KiRd9RpS8-C1",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "8331a105-49c1-480c-9c26-36e9ef62065f",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524410065331,
          "user_tz": 240,
          "elapsed": 126839,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "0_Gt1Viy9AP_",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 85
        },
        "outputId": "97f81304-96bf-4b58-f566-35978eaa7432",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524410065835,
          "user_tz": 240,
          "elapsed": 340,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "O-aWlfy79hlo",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "3d53b0e2-edfd-4a82-8519-749c4aaf90f5",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524410191360,
          "user_tz": 240,
          "elapsed": 125048,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "zTWfcDMh9jw-",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 85
        },
        "outputId": "c388c9ac-b6b9-4d99-e1db-96e0728764c8",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524410191981,
          "user_tz": 240,
          "elapsed": 408,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "Q0f5dDgj95Sl",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "abe901e3-95b2-4edc-c767-069d7b033ec9",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524410317445,
          "user_tz": 240,
          "elapsed": 125010,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "7aHzsESw96-n",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 85
        },
        "outputId": "941a084e-bbce-4b2c-f5d5-c8af1817df7e",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524410318002,
          "user_tz": 240,
          "elapsed": 380,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "4b7wNU7D-iU3",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "d0b2b2e5-726c-4b52-b010-9af73c8617a1",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524410442329,
          "user_tz": 240,
          "elapsed": 123898,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "0jOCxmf8-j9m",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "a71f20c7-2834-4f23-b1f6-e0ef31aa05f2",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524410442913,
          "user_tz": 240,
          "elapsed": 346,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "WWfHbjda-6gL",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "ed813eae-da28-4af2-ad8c-8cdcdca43552",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524410568309,
          "user_tz": 240,
          "elapsed": 124765,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "eQ3IPilg_AqT",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 102
        },
        "outputId": "d8e30ae1-00eb-4eab-bb25-06909ea194e3",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524410568868,
          "user_tz": 240,
          "elapsed": 387,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "faxWunQw_XK9",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "469d8bb0-fa8d-4ed6-bef6-5f12ec8206b5",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524410692017,
          "user_tz": 240,
          "elapsed": 122692,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "HheNhcVi_ZZL",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 119
        },
        "outputId": "91b8fbf1-6585-49c0-fc28-5f4c07fffd8b",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524410692568,
          "user_tz": 240,
          "elapsed": 331,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "9zbfeWkQ_0by",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "f2ee2a68-0432-4cdc-c444-3b9203504bb0",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524410815518,
          "user_tz": 240,
          "elapsed": 122328,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 37,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "YMHUsVd1_2Va",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 119
        },
        "outputId": "05f80d97-1c16-4e01-b464-aa853435d880",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524410816045,
          "user_tz": 240,
          "elapsed": 355,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 38,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "c8gtQy4wAWrN",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "3794984d-74b5-46d8-bae8-1e5b068c48e3",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524410938168,
          "user_tz": 240,
          "elapsed": 121654,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 39,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "v5sEx1blAYn0",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 119
        },
        "outputId": "8e53ce45-5805-4962-de63-3b511091bee8",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524410938798,
          "user_tz": 240,
          "elapsed": 470,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 40,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "PX_k8PscAvv6",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "ee948f7f-8f2a-4c57-c8fd-725129ffae46",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524411060286,
          "user_tz": 240,
          "elapsed": 120894,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "yA12-WgeAyFy",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 119
        },
        "outputId": "e3e77a6e-9e62-41c7-e4f4-fbed18b04758",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524411060981,
          "user_tz": 240,
          "elapsed": 439,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "iE5fE41zBNqk",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "868a159e-8e15-4815-8229-537488bbcce8",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524411182498,
          "user_tz": 240,
          "elapsed": 121065,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "wXYkDUqEBRdQ",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 136
        },
        "outputId": "28af9dd0-dacb-4b5a-ac6b-07ec3998ee11",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524411183127,
          "user_tz": 240,
          "elapsed": 457,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "WeOrijjxB5Og",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "a4c0224e-5d55-4d6e-c8a9-5c0585f858eb",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524411304580,
          "user_tz": 240,
          "elapsed": 121031,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "1fsREIKIB7NC",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 136
        },
        "outputId": "6a5d8361-bd87-43f9-9c0b-a3609b3689a8",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524411305140,
          "user_tz": 240,
          "elapsed": 397,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 46,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "p7m9a0njCMw8",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "7d3a0a7e-1394-41ec-80e1-7a300384e498",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524411426572,
          "user_tz": 240,
          "elapsed": 120971,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 47,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "7LmzmWbUCPCy",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 136
        },
        "outputId": "77df2436-25a1-4152-968a-4e6ace3d7f61",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524411427166,
          "user_tz": 240,
          "elapsed": 378,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 48,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764  -0.10262 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439 57.95881914]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "zXhF2BJUCjSx",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "d50b8d47-27d0-40c4-c9fc-d5c8db15165c",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524411548021,
          "user_tz": 240,
          "elapsed": 120411,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "ENu0YEOlClzd",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 153
        },
        "outputId": "59490482-20d8-46cf-c42c-160da4c26596",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524411548579,
          "user_tz": 240,
          "elapsed": 384,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 50,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764  -0.10262\n",
            " -0.09454 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439 57.95881914 61.26901931]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "ZEuzAXo3C-z0",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "bc47336c-4e85-41c6-9610-aa88e0fabf38",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524411668801,
          "user_tz": 240,
          "elapsed": 119782,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 51,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "jbIowOOODCQE",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 153
        },
        "outputId": "7dd2987e-ce98-4289-b249-33eb571c36fe",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524411669322,
          "user_tz": 240,
          "elapsed": 363,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764  -0.10262\n",
            " -0.09454  -0.09288 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439 57.95881914 61.26901931 61.94908519]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "75OMPFKcDbMl",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "37a81e25-05b4-49f7-97bd-ff13cf6919b4",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524411788719,
          "user_tz": 240,
          "elapsed": 119012,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 53,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "EuCOb-gkDeW8",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 153
        },
        "outputId": "35eb55ba-d382-4ea1-bd0d-130a8bbc8e2b",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524411789287,
          "user_tz": 240,
          "elapsed": 389,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 54,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764  -0.10262\n",
            " -0.09454  -0.09288  -0.09244 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439 57.95881914 61.26901931 61.94908519 62.12934361]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "b3XwJ7pSDruP",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "42b9379a-da1a-4232-8d75-1d535b41802b",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524411907918,
          "user_tz": 240,
          "elapsed": 118179,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 55,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "RvoZA1YrDuQI",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 170
        },
        "outputId": "fa7fb26f-6461-459f-fa99-9da64b60f7bc",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524411908406,
          "user_tz": 240,
          "elapsed": 333,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 56,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764  -0.10262\n",
            " -0.09454  -0.09288  -0.09244  -0.10032 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439 57.95881914 61.26901931 61.94908519 62.12934361\n",
            " 58.90107909]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "kDcwnKnGEBqc",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "6d4f3112-a78c-4ee3-d53c-3433e569782f",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524412025643,
          "user_tz": 240,
          "elapsed": 116854,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 57,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "PtNL_IUMEEyi",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 170
        },
        "outputId": "45e7fbdd-853f-43d6-f69d-1e25f747f8f5",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524412026197,
          "user_tz": 240,
          "elapsed": 396,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 58,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764  -0.10262\n",
            " -0.09454  -0.09288  -0.09244  -0.10032  -0.08844 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439 57.95881914 61.26901931 61.94908519 62.12934361\n",
            " 58.90107909 63.76805657]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "QXww2NfJEyrv",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "73b4f2a4-606e-4cae-b25e-8eb75040aa5d",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524412142178,
          "user_tz": 240,
          "elapsed": 115575,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 59,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "uyFK2IoFE0j8",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 170
        },
        "outputId": "2512f376-0f50-4248-c9dc-5280e5fef6c4",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524412142725,
          "user_tz": 240,
          "elapsed": 385,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 60,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764  -0.10262\n",
            " -0.09454  -0.09288  -0.09244  -0.10032  -0.08844  -0.08994 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439 57.95881914 61.26901931 61.94908519 62.12934361\n",
            " 58.90107909 63.76805657 63.15353921]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "lCbJ1NcUFNC6",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "094bc0a4-a891-45ed-8a6a-14240decb092",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524412259704,
          "user_tz": 240,
          "elapsed": 116526,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 61,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "NSPPbOXoFPm1",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 170
        },
        "outputId": "405ac041-918f-4318-9409-99650f850380",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524412260276,
          "user_tz": 240,
          "elapsed": 372,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 62,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764  -0.10262\n",
            " -0.09454  -0.09288  -0.09244  -0.10032  -0.08844  -0.08994  -0.09318 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439 57.95881914 61.26901931 61.94908519 62.12934361\n",
            " 58.90107909 63.76805657 63.15353921 61.82618172]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "feUafnKRFSlD",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "bd50f7d1-1a62-453b-bc29-defbcd439b04",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524412377521,
          "user_tz": 240,
          "elapsed": 116803,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 63,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "xIgR5gySFUjm",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "outputId": "b2223eb3-8f78-4823-ab17-e52082594efc",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524412378098,
          "user_tz": 240,
          "elapsed": 414,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 64,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764  -0.10262\n",
            " -0.09454  -0.09288  -0.09244  -0.10032  -0.08844  -0.08994  -0.09318\n",
            " -0.09078 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439 57.95881914 61.26901931 61.94908519 62.12934361\n",
            " 58.90107909 63.76805657 63.15353921 61.82618172 62.80940949]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "xvzZ8yeCGDww",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "5362c01c-bc89-46d4-967a-ded9f1e0b487",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524412495211,
          "user_tz": 240,
          "elapsed": 116663,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 65,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "WcfNpqVnGF5w",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "outputId": "4fa90e47-2ad6-450f-b90f-3e80be11f3e9",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524412495721,
          "user_tz": 240,
          "elapsed": 359,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 66,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764  -0.10262\n",
            " -0.09454  -0.09288  -0.09244  -0.10032  -0.08844  -0.08994  -0.09318\n",
            " -0.09078  -0.08884 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439 57.95881914 61.26901931 61.94908519 62.12934361\n",
            " 58.90107909 63.76805657 63.15353921 61.82618172 62.80940949 63.60418527]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "JE3y9hB-GH1y",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "7dcf4c2c-018b-4762-d778-36084fecc9c2",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524412612995,
          "user_tz": 240,
          "elapsed": 116781,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 67,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "lywcbFFzGMLo",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "c9311db6-a3d9-4f92-bf4f-548b30061251",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524412613628,
          "user_tz": 240,
          "elapsed": 467,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 68,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764  -0.10262\n",
            " -0.09454  -0.09288  -0.09244  -0.10032  -0.08844  -0.08994  -0.09318\n",
            " -0.09078  -0.08884  -0.08376 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439 57.95881914 61.26901931 61.94908519 62.12934361\n",
            " 58.90107909 63.76805657 63.15353921 61.82618172 62.80940949 63.60418527\n",
            " 65.68535073]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "I26085uEGhdc",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "7495a6fa-c203-4671-b504-0468e91420bb",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524412730307,
          "user_tz": 240,
          "elapsed": 116233,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 69,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "7vQvxdZPGjiB",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "09eb5503-9bda-44b9-cb52-0901be34b310",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524412731081,
          "user_tz": 240,
          "elapsed": 623,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 70,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764  -0.10262\n",
            " -0.09454  -0.09288  -0.09244  -0.10032  -0.08844  -0.08994  -0.09318\n",
            " -0.09078  -0.08884  -0.08376  -0.0845  ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439 57.95881914 61.26901931 61.94908519 62.12934361\n",
            " 58.90107909 63.76805657 63.15353921 61.82618172 62.80940949 63.60418527\n",
            " 65.68535073 65.38218883]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "OyRqhsXSG55W",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "b4c89e0a-67f2-409a-fe8b-b37b307d0895",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524412847943,
          "user_tz": 240,
          "elapsed": 116285,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 71,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "3TTgE1jZG8z2",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "76d609ad-9569-49a8-ac60-3fe750c014c0",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524412848426,
          "user_tz": 240,
          "elapsed": 327,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 72,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764  -0.10262\n",
            " -0.09454  -0.09288  -0.09244  -0.10032  -0.08844  -0.08994  -0.09318\n",
            " -0.09078  -0.08884  -0.08376  -0.0845   -0.0833  ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439 57.95881914 61.26901931 61.94908519 62.12934361\n",
            " 58.90107909 63.76805657 63.15353921 61.82618172 62.80940949 63.60418527\n",
            " 65.68535073 65.38218883 65.87380272]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "VBfsAvTmG-9Z",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "64dbd7b9-fab4-45e9-ea58-bb67610aabce",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524412965338,
          "user_tz": 240,
          "elapsed": 116458,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 73,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "ZFVmTYBPHBMA",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "5414a30c-e5ce-4488-d23a-336b4b8fa96f",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524412965978,
          "user_tz": 240,
          "elapsed": 456,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 74,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764  -0.10262\n",
            " -0.09454  -0.09288  -0.09244  -0.10032  -0.08844  -0.08994  -0.09318\n",
            " -0.09078  -0.08884  -0.08376  -0.0845   -0.0833   -0.085   ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439 57.95881914 61.26901931 61.94908519 62.12934361\n",
            " 58.90107909 63.76805657 63.15353921 61.82618172 62.80940949 63.60418527\n",
            " 65.68535073 65.38218883 65.87380272 65.17734971]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "Fwl8PtLXHvhO",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "f1a61d4f-9b06-4e48-8346-373f508546ad",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524413082724,
          "user_tz": 240,
          "elapsed": 116236,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 75,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "daHWo37DHyto",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "934cd513-ffbf-4d82-8be1-c0aa82bca3e8",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524413083285,
          "user_tz": 240,
          "elapsed": 415,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 76,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764  -0.10262\n",
            " -0.09454  -0.09288  -0.09244  -0.10032  -0.08844  -0.08994  -0.09318\n",
            " -0.09078  -0.08884  -0.08376  -0.0845   -0.0833   -0.085    -0.08166 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439 57.95881914 61.26901931 61.94908519 62.12934361\n",
            " 58.90107909 63.76805657 63.15353921 61.82618172 62.80940949 63.60418527\n",
            " 65.68535073 65.38218883 65.87380272 65.17734971 66.54567503]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "r8h17iUrH0cr",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "331fb503-afee-4fd2-bc32-084b1f6c234b",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524413199626,
          "user_tz": 240,
          "elapsed": 115971,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 77,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "VVFgH5ABHyBg",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 221
        },
        "outputId": "6190a6d0-edfa-4cd9-a592-5752c689aec9",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524413200173,
          "user_tz": 240,
          "elapsed": 391,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 78,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764  -0.10262\n",
            " -0.09454  -0.09288  -0.09244  -0.10032  -0.08844  -0.08994  -0.09318\n",
            " -0.09078  -0.08884  -0.08376  -0.0845   -0.0833   -0.085    -0.08166\n",
            " -0.08284 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439 57.95881914 61.26901931 61.94908519 62.12934361\n",
            " 58.90107909 63.76805657 63.15353921 61.82618172 62.80940949 63.60418527\n",
            " 65.68535073 65.38218883 65.87380272 65.17734971 66.54567503 66.06225471]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "APUbkj82IlEK",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "1537dee8-b11e-484d-90bc-85577461ef88",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524413316584,
          "user_tz": 240,
          "elapsed": 115979,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 79,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "u5AZU-eqIoEL",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 238
        },
        "outputId": "7876366f-5009-46fa-b1ff-57677c002f48",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524413317155,
          "user_tz": 240,
          "elapsed": 356,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 80,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764  -0.10262\n",
            " -0.09454  -0.09288  -0.09244  -0.10032  -0.08844  -0.08994  -0.09318\n",
            " -0.09078  -0.08884  -0.08376  -0.0845   -0.0833   -0.085    -0.08166\n",
            " -0.08284  -0.0849  ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439 57.95881914 61.26901931 61.94908519 62.12934361\n",
            " 58.90107909 63.76805657 63.15353921 61.82618172 62.80940949 63.60418527\n",
            " 65.68535073 65.38218883 65.87380272 65.17734971 66.54567503 66.06225471\n",
            " 65.21831753]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "srdTzA5XIp8x",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "c9bd0e2d-5a11-4d3f-b0ce-897c4e28927b",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524413433892,
          "user_tz": 240,
          "elapsed": 116310,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 81,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "cvIWlC-SItLF",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 238
        },
        "outputId": "8b5a25e6-fae2-4800-93df-b75a9ba59ab9",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524413434366,
          "user_tz": 240,
          "elapsed": 322,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 82,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764  -0.10262\n",
            " -0.09454  -0.09288  -0.09244  -0.10032  -0.08844  -0.08994  -0.09318\n",
            " -0.09078  -0.08884  -0.08376  -0.0845   -0.0833   -0.085    -0.08166\n",
            " -0.08284  -0.0849   -0.08636 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439 57.95881914 61.26901931 61.94908519 62.12934361\n",
            " 58.90107909 63.76805657 63.15353921 61.82618172 62.80940949 63.60418527\n",
            " 65.68535073 65.38218883 65.87380272 65.17734971 66.54567503 66.06225471\n",
            " 65.21831753 64.6201873 ]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "HvJV32B3JZ0b",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "802b8e41-6532-4ee1-f5bb-94ebee5523b9",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524413551098,
          "user_tz": 240,
          "elapsed": 116341,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 83,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "pSUF9XUyJcsC",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 238
        },
        "outputId": "34c7a048-21b3-4d1a-db38-d4f3fcdc5641",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524413551585,
          "user_tz": 240,
          "elapsed": 326,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 84,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764  -0.10262\n",
            " -0.09454  -0.09288  -0.09244  -0.10032  -0.08844  -0.08994  -0.09318\n",
            " -0.09078  -0.08884  -0.08376  -0.0845   -0.0833   -0.085    -0.08166\n",
            " -0.08284  -0.0849   -0.08636  -0.07952 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439 57.95881914 61.26901931 61.94908519 62.12934361\n",
            " 58.90107909 63.76805657 63.15353921 61.82618172 62.80940949 63.60418527\n",
            " 65.68535073 65.38218883 65.87380272 65.17734971 66.54567503 66.06225471\n",
            " 65.21831753 64.6201873  67.42238646]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "hGK0b7T6JfIC",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "00bb3647-3ba6-49c6-a2dd-9167ab300f0d",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524413668854,
          "user_tz": 240,
          "elapsed": 116822,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 85,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "8xZARR7NJhcG",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 238
        },
        "outputId": "4db8f28b-2188-429a-d5bf-273bf5bd729c",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524413669401,
          "user_tz": 240,
          "elapsed": 402,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 86,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764  -0.10262\n",
            " -0.09454  -0.09288  -0.09244  -0.10032  -0.08844  -0.08994  -0.09318\n",
            " -0.09078  -0.08884  -0.08376  -0.0845   -0.0833   -0.085    -0.08166\n",
            " -0.08284  -0.0849   -0.08636  -0.07952  -0.07866 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439 57.95881914 61.26901931 61.94908519 62.12934361\n",
            " 58.90107909 63.76805657 63.15353921 61.82618172 62.80940949 63.60418527\n",
            " 65.68535073 65.38218883 65.87380272 65.17734971 66.54567503 66.06225471\n",
            " 65.21831753 64.6201873  67.42238646 67.77470974]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "EZxDV84dKfOw",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "3132fb9d-7280-489a-ef63-1f3fcb3c0736",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524413786435,
          "user_tz": 240,
          "elapsed": 116598,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 87,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "zljYpPkWKkzk",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 238
        },
        "outputId": "b3456cc0-eda6-46e7-c30a-fa5893e361bf",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524413786877,
          "user_tz": 240,
          "elapsed": 391,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 88,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764  -0.10262\n",
            " -0.09454  -0.09288  -0.09244  -0.10032  -0.08844  -0.08994  -0.09318\n",
            " -0.09078  -0.08884  -0.08376  -0.0845   -0.0833   -0.085    -0.08166\n",
            " -0.08284  -0.0849   -0.08636  -0.07952  -0.07866  -0.0798  ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439 57.95881914 61.26901931 61.94908519 62.12934361\n",
            " 58.90107909 63.76805657 63.15353921 61.82618172 62.80940949 63.60418527\n",
            " 65.68535073 65.38218883 65.87380272 65.17734971 66.54567503 66.06225471\n",
            " 65.21831753 64.6201873  67.42238646 67.77470974 67.30767655]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "KXWYqgHzKw5a",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "00bbdbed-33a7-4062-fc3a-4129fbeb3d95",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524413904114,
          "user_tz": 240,
          "elapsed": 116775,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 89,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "1cH1M8UJKzG6",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 238
        },
        "outputId": "7d49ba0c-61ee-446c-e233-4b7743b991cf",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524413904693,
          "user_tz": 240,
          "elapsed": 374,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 90,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764  -0.10262\n",
            " -0.09454  -0.09288  -0.09244  -0.10032  -0.08844  -0.08994  -0.09318\n",
            " -0.09078  -0.08884  -0.08376  -0.0845   -0.0833   -0.085    -0.08166\n",
            " -0.08284  -0.0849   -0.08636  -0.07952  -0.07866  -0.0798   -0.07478 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439 57.95881914 61.26901931 61.94908519 62.12934361\n",
            " 58.90107909 63.76805657 63.15353921 61.82618172 62.80940949 63.60418527\n",
            " 65.68535073 65.38218883 65.87380272 65.17734971 66.54567503 66.06225471\n",
            " 65.21831753 64.6201873  67.42238646 67.77470974 67.30767655 69.36426131]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "Gje7RHKGLhUN",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "09769c23-2168-4de8-aacf-ec4ced95914e",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524414020974,
          "user_tz": 240,
          "elapsed": 115843,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 91,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "dASpH5o9LjeK",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 272
        },
        "outputId": "eba05618-9958-4e63-adfb-563473b41877",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524414021522,
          "user_tz": 240,
          "elapsed": 383,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 92,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764  -0.10262\n",
            " -0.09454  -0.09288  -0.09244  -0.10032  -0.08844  -0.08994  -0.09318\n",
            " -0.09078  -0.08884  -0.08376  -0.0845   -0.0833   -0.085    -0.08166\n",
            " -0.08284  -0.0849   -0.08636  -0.07952  -0.07866  -0.0798   -0.07478\n",
            " -0.08168 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439 57.95881914 61.26901931 61.94908519 62.12934361\n",
            " 58.90107909 63.76805657 63.15353921 61.82618172 62.80940949 63.60418527\n",
            " 65.68535073 65.38218883 65.87380272 65.17734971 66.54567503 66.06225471\n",
            " 65.21831753 64.6201873  67.42238646 67.77470974 67.30767655 69.36426131\n",
            " 66.53748146]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "6kIs5qs3LmZc",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "9d0eb60b-c21c-4766-ccdb-abe8b9e368d3",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524414137699,
          "user_tz": 240,
          "elapsed": 115721,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 93,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "qCnavgHRLot-",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 272
        },
        "outputId": "70238cb3-94cb-4648-c6b7-97488bc5e7c1",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524414138312,
          "user_tz": 240,
          "elapsed": 456,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 94,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764  -0.10262\n",
            " -0.09454  -0.09288  -0.09244  -0.10032  -0.08844  -0.08994  -0.09318\n",
            " -0.09078  -0.08884  -0.08376  -0.0845   -0.0833   -0.085    -0.08166\n",
            " -0.08284  -0.0849   -0.08636  -0.07952  -0.07866  -0.0798   -0.07478\n",
            " -0.08168  -0.07942 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439 57.95881914 61.26901931 61.94908519 62.12934361\n",
            " 58.90107909 63.76805657 63.15353921 61.82618172 62.80940949 63.60418527\n",
            " 65.68535073 65.38218883 65.87380272 65.17734971 66.54567503 66.06225471\n",
            " 65.21831753 64.6201873  67.42238646 67.77470974 67.30767655 69.36426131\n",
            " 66.53748146 67.46335428]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "RThlZ4x8MAXq",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "d492f1a0-a1d7-4224-eb8a-9c35732f817c",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524414253747,
          "user_tz": 240,
          "elapsed": 114996,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 95,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "5ENuJKbfMEDz",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 272
        },
        "outputId": "7f580208-b9f2-43fa-d8b9-688bd2ee94fd",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524414254311,
          "user_tz": 240,
          "elapsed": 417,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 96,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764  -0.10262\n",
            " -0.09454  -0.09288  -0.09244  -0.10032  -0.08844  -0.08994  -0.09318\n",
            " -0.09078  -0.08884  -0.08376  -0.0845   -0.0833   -0.085    -0.08166\n",
            " -0.08284  -0.0849   -0.08636  -0.07952  -0.07866  -0.0798   -0.07478\n",
            " -0.08168  -0.07942  -0.08554 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439 57.95881914 61.26901931 61.94908519 62.12934361\n",
            " 58.90107909 63.76805657 63.15353921 61.82618172 62.80940949 63.60418527\n",
            " 65.68535073 65.38218883 65.87380272 65.17734971 66.54567503 66.06225471\n",
            " 65.21831753 64.6201873  67.42238646 67.77470974 67.30767655 69.36426131\n",
            " 66.53748146 67.46335428 64.95612346]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "4CxFTn2HMyXX",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "b9148884-e45f-40bb-b683-65f5a92ba6a0",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524414369852,
          "user_tz": 240,
          "elapsed": 115119,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 97,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "nBmqIpG_M0az",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 272
        },
        "outputId": "151eee45-ac50-4ec7-f63e-6c25e9171ad2",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524414370368,
          "user_tz": 240,
          "elapsed": 302,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 98,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764  -0.10262\n",
            " -0.09454  -0.09288  -0.09244  -0.10032  -0.08844  -0.08994  -0.09318\n",
            " -0.09078  -0.08884  -0.08376  -0.0845   -0.0833   -0.085    -0.08166\n",
            " -0.08284  -0.0849   -0.08636  -0.07952  -0.07866  -0.0798   -0.07478\n",
            " -0.08168  -0.07942  -0.08554  -0.07876 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439 57.95881914 61.26901931 61.94908519 62.12934361\n",
            " 58.90107909 63.76805657 63.15353921 61.82618172 62.80940949 63.60418527\n",
            " 65.68535073 65.38218883 65.87380272 65.17734971 66.54567503 66.06225471\n",
            " 65.21831753 64.6201873  67.42238646 67.77470974 67.30767655 69.36426131\n",
            " 66.53748146 67.46335428 64.95612346 67.73374192]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "uFwLxl_sM3Xu",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "9f7d9151-f63f-4e27-c306-1d061f978fb7",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524414485876,
          "user_tz": 240,
          "elapsed": 115061,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 99,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "hBB9zYstM6D2",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 272
        },
        "outputId": "07c2c5aa-71b6-4684-9826-2e370a3ae550",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524414486489,
          "user_tz": 240,
          "elapsed": 457,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 100,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764  -0.10262\n",
            " -0.09454  -0.09288  -0.09244  -0.10032  -0.08844  -0.08994  -0.09318\n",
            " -0.09078  -0.08884  -0.08376  -0.0845   -0.0833   -0.085    -0.08166\n",
            " -0.08284  -0.0849   -0.08636  -0.07952  -0.07866  -0.0798   -0.07478\n",
            " -0.08168  -0.07942  -0.08554  -0.07876  -0.07716 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439 57.95881914 61.26901931 61.94908519 62.12934361\n",
            " 58.90107909 63.76805657 63.15353921 61.82618172 62.80940949 63.60418527\n",
            " 65.68535073 65.38218883 65.87380272 65.17734971 66.54567503 66.06225471\n",
            " 65.21831753 64.6201873  67.42238646 67.77470974 67.30767655 69.36426131\n",
            " 66.53748146 67.46335428 64.95612346 67.73374192 68.3892271 ]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "msF1seRiN2Rx",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "d6f326e4-f4e2-4a8c-83d7-aa3ad6f0b1d8",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524414601722,
          "user_tz": 240,
          "elapsed": 114762,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 101,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "YaOd8sNyN53W",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 272
        },
        "outputId": "967f31dc-c276-49a8-a8c1-eb2587c4139f",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524414602231,
          "user_tz": 240,
          "elapsed": 364,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 102,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764  -0.10262\n",
            " -0.09454  -0.09288  -0.09244  -0.10032  -0.08844  -0.08994  -0.09318\n",
            " -0.09078  -0.08884  -0.08376  -0.0845   -0.0833   -0.085    -0.08166\n",
            " -0.08284  -0.0849   -0.08636  -0.07952  -0.07866  -0.0798   -0.07478\n",
            " -0.08168  -0.07942  -0.08554  -0.07876  -0.07716  -0.08078 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439 57.95881914 61.26901931 61.94908519 62.12934361\n",
            " 58.90107909 63.76805657 63.15353921 61.82618172 62.80940949 63.60418527\n",
            " 65.68535073 65.38218883 65.87380272 65.17734971 66.54567503 66.06225471\n",
            " 65.21831753 64.6201873  67.42238646 67.77470974 67.30767655 69.36426131\n",
            " 66.53748146 67.46335428 64.95612346 67.73374192 68.3892271  66.90619188]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "9o2jvwMBOE6u",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "0d26dc30-63e4-4cd8-aa94-a217fe94c832",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524414717754,
          "user_tz": 240,
          "elapsed": 115076,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 103,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "91DVBV5QOHYW",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 289
        },
        "outputId": "ffb83950-e47f-4db0-b580-64ee3dc8a43a",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524414718344,
          "user_tz": 240,
          "elapsed": 373,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 104,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764  -0.10262\n",
            " -0.09454  -0.09288  -0.09244  -0.10032  -0.08844  -0.08994  -0.09318\n",
            " -0.09078  -0.08884  -0.08376  -0.0845   -0.0833   -0.085    -0.08166\n",
            " -0.08284  -0.0849   -0.08636  -0.07952  -0.07866  -0.0798   -0.07478\n",
            " -0.08168  -0.07942  -0.08554  -0.07876  -0.07716  -0.08078  -0.0763  ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439 57.95881914 61.26901931 61.94908519 62.12934361\n",
            " 58.90107909 63.76805657 63.15353921 61.82618172 62.80940949 63.60418527\n",
            " 65.68535073 65.38218883 65.87380272 65.17734971 66.54567503 66.06225471\n",
            " 65.21831753 64.6201873  67.42238646 67.77470974 67.30767655 69.36426131\n",
            " 66.53748146 67.46335428 64.95612346 67.73374192 68.3892271  66.90619188\n",
            " 68.74155039]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "xpj21XizOI5O",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "collapsed": true,
        "outputId": "65b049f0-8b2a-4039-fd1f-8b81a88facd3",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524414833402,
          "user_tz": 240,
          "elapsed": 114676,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "gamma=0.1\n",
        "lambda_theta=0.5\n",
        "lambda_w=lambda_theta\n",
        "alpha_theta=0.0001\n",
        "alpha_w=0.001\n",
        "number_of_episods=100000\n",
        "number_of_runs_for_average=10\n",
        "average_success_v0=np.zeros(number_of_runs_for_average)\n",
        "theta_saved=theta\n",
        "w_saved=w\n",
        "\n",
        "for n in range(number_of_runs_for_average):\n",
        "  print(\"processing: \",((n+1)*100)/number_of_runs_for_average,\"% of the work\")\n",
        "  theta=theta_saved\n",
        "  w=w_saved\n",
        "  r_t=0\n",
        "  for i in range(number_of_episods):\n",
        "    theta,w,temp=actor_critic_F(gamma,lambda_theta,lambda_w,alpha_theta,alpha_w,theta,w)\n",
        "    r_t+=temp\n",
        "  r_t=r_t/number_of_episods\n",
        "  average_success_v0[n]=r_t"
      ],
      "execution_count": 105,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "processing:  10.0 % of the work\n",
            "processing:  20.0 % of the work\n",
            "processing:  30.0 % of the work\n",
            "processing:  40.0 % of the work\n",
            "processing:  50.0 % of the work\n",
            "processing:  60.0 % of the work\n",
            "processing:  70.0 % of the work\n",
            "processing:  80.0 % of the work\n",
            "processing:  90.0 % of the work\n",
            "processing:  100.0 % of the work\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "N8v_j-2DOMdZ",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 306
        },
        "outputId": "f3127e36-8a82-499f-cdc2-d0c5e6a348ac",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524414833998,
          "user_tz": 240,
          "elapsed": 443,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "avg_method=np.append(avg_method,np.average(average_success_v0))\n",
        "temp=((np.average(average_success_v0)-avg_reward)/(-avg_reward))*100\n",
        "improvement_method=np.append(improvement_method,temp)\n",
        "\n",
        "print(avg_method)\n",
        "print(improvement_method)"
      ],
      "execution_count": 106,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[-0.244094 -0.2424   -0.23904  -0.22024  -0.20586  -0.18986  -0.18036\n",
            " -0.169    -0.15884  -0.14996  -0.14094  -0.13526  -0.12486  -0.12716\n",
            " -0.11836  -0.11438  -0.10854  -0.11348  -0.09984  -0.10764  -0.10262\n",
            " -0.09454  -0.09288  -0.09244  -0.10032  -0.08844  -0.08994  -0.09318\n",
            " -0.09078  -0.08884  -0.08376  -0.0845   -0.0833   -0.085    -0.08166\n",
            " -0.08284  -0.0849   -0.08636  -0.07952  -0.07866  -0.0798   -0.07478\n",
            " -0.08168  -0.07942  -0.08554  -0.07876  -0.07716  -0.08078  -0.0763\n",
            " -0.08224 ]\n",
            "[ 0.          0.69399494  2.07051382  9.77246471 15.66363778 22.2184896\n",
            " 26.11043287 30.76437766 34.92670856 38.56465132 42.25994904 44.58692143\n",
            " 48.84757511 47.90531517 51.51048367 53.14100306 55.53352397 53.50971347\n",
            " 59.09772465 55.90223439 57.95881914 61.26901931 61.94908519 62.12934361\n",
            " 58.90107909 63.76805657 63.15353921 61.82618172 62.80940949 63.60418527\n",
            " 65.68535073 65.38218883 65.87380272 65.17734971 66.54567503 66.06225471\n",
            " 65.21831753 64.6201873  67.42238646 67.77470974 67.30767655 69.36426131\n",
            " 66.53748146 67.46335428 64.95612346 67.73374192 68.3892271  66.90619188\n",
            " 68.74155039 66.30806165]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "5cHxPUOZNuaD",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "### Plots\n",
        "Below we will plot the performance of the model in comparision to the random policy."
      ]
    },
    {
      "metadata": {
        "id": "m9kP2SnHPasA",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          }
        }
      },
      "cell_type": "code",
      "source": [
        "itteration=np.zeros(avg_method.size)\n",
        "for i in range(avg_method.size-1):\n",
        "  itteration[i+1]=i*100000+10000"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "pRqKs_FrQol3",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 376
        },
        "outputId": "eb0d870f-ba54-4f31-e64b-f7f00b9a558f",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524414835616,
          "user_tz": 240,
          "elapsed": 736,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "ranom_play=np.zeros(itteration.size)+avg_reward\n",
        "plt.plot(itteration,avg_method,color=(0,0,0),label='Average Policy Gradient Method Return')\n",
        "plt.plot(itteration,ranom_play,'r--',label='Average Random Retun')\n",
        "plt.legend()\n",
        "plt.xlabel('Number of Iteration')\n",
        "plt.ylabel('Average Return')\n",
        "plt.title('Average Return vs. Policy for Second Agent')\n",
        "plt.show()\n"
      ],
      "execution_count": 108,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhAAAAFnCAYAAAD3z3BtAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3XVYVNn/wPH30BIqKGGja+taqNgK\ngoBgCxbYLViIyRordmOiay+CvbYYYCfi6hqIuWKANAhI3t8ffJ2fLKEoYJ3X8+yzzsy55577mTvM\nZ8499xyZJEkSgiAIgiAIeaDwtRsgCIIgCML3RyQQgiAIgiDkmUggBEEQBEHIM5FACIIgCIKQZyKB\nEARBEAQhz0QCIQiCIAhCnokEQvim9ezZk44dO37tZuSZqakpJiYmWFpaYmlpibm5OaNHjyYsLOyj\n2966dYvAwMBCaOWXmzx5Mk2aNMHS0hILCwssLS1ZtmwZaWlpuW539epVzM3NAViyZAleXl5f3Jbg\n4GDMzc3p1KnTF9Xz/Plzhg8fjoWFBRYWFnTu3JlTp059cfu+hL+/P6amprmWKazPyq5duwp8H8L3\nQSQQwjcrKCgILS0tSpcuzc2bN792c/Js0aJFHD9+XP6frq4u8+fP/+h2e/fu5cGDB4XQwvzRt29f\njh8/jo+PD3v27OHSpUt5+pJxdnamV69eX9yOGzduoKury4EDB76ongkTJtCyZUt8fHzw8fFh1qxZ\nTJgwgdevX39xGwtKYX1W0tLSWLhwYYHVL3xfRAIhfLP279+PpaUlNjY2/PXXX/Lnu3fvjo+Pj/zx\nqVOnsLOzk/+7Q4cOtG3bloEDBxIZGQnAypUrcXV1pXv37mzZsoX09HRmzZqFhYUFpqamuLi4kJKS\nAsCLFy/o3LkzpqamTJ8+nWHDhrFv3z4g40uqW7dumJubY2dnR3Bw8Ccdi6KiIm3atJH3LEiSxKpV\nq7CwsMDExAQ3NzfS0tLw8vLiwIEDLFq0iM2bN7Ny5UqmTZsmr+fDxw4ODixbtgwrKysCAgKYPHky\n7u7uDBgwABMTEwYMGEBiYmKmdjx69IjGjRuTmpoqf27kyJF4eXkRFBREjx49sLa2pl27dvz555+f\n9kZ9QFNTk86dO3Px4kUAXr16xaBBg7CwsMjyPr43efJk1qxZA8CdO3fo2rUrFhYW2NvbExwczIIF\nC/j999/l5WNiYqhbt678vQW4efMmixcv5t69e/Jf4ceOHcPGxgZLS0v69u3L8+fP5TH88Fz4r6Cg\nIOrWrSt/XLduXXx8fDAwMAByPsfevXvHxIkTMTU1xcrKSp7IJCUlMX36dCwsLLCysmL+/PnyHhpT\nU1O8vb3p3r07LVq0yJRgrlmzhtatW9O5c2cuXbqUa9xz+qwArFu3jqZNm9KtWzc8PT3lPRnJycm4\nubnJPwPr1q2Tb5NTuwYMGEBcXByWlpaffO4LPzBJEL5BqampUtu2baW4uDgpISFBatOmjZSUlCRJ\nkiStX79emjhxorzsxIkTpU2bNknPnz+X6tevLz148ECSJElat26d5OTkJEmSJLm7u0stWrSQIiIi\nJEmSpOPHj0s2NjZScnKy9O7dO8nKykr666+/JEmSJCcnJ2nhwoWSJEnSyZMnpdq1a0t79+6V4uLi\npEaNGkkXLlyQJEmSDh06JHXp0iXb9puYmEjXr1+XP05MTJTGjBkjTZs2TZIkSdq/f79kbW0txcbG\nSikpKdLQoUOl7du3S5IkSfb29vK2uLu7S1OnTpXX8+Fje3t7aeDAgVJaWpokSZI0adIkycrKSoqK\nipJSUlKkjh07SgcOHMjSNisrK+ny5cuSJElSQkKCVL9+fSkiIkJycnKS9u3bJ0mSJEVEREgjRoyQ\nxzwnkyZNklavXp3puS1btkjjxo2TJEmSBg4cKK1bt06SJEl68eKFZGRkJAUHB0tXrlyRzMzMstRh\nbm4unTlzRpIkSdq8ebM0ZMgQ6c6dO1LTpk2llJQUeewGDhyYpS179+6V+vXrJ0mSJL18+VIyMjKS\nnj17JkmSJG3cuFH+2n/Phf9ycnKSTExMpK1bt0qPHj3K9Fpu59jq1aulsWPHSpIkSa9fv5aMjIyk\nkJAQycPDQxoyZIiUkpIiJSYmSt26dZO/vyYmJtL48eOl1NRUKSQkRKpVq5b0+vVr6eHDh1KjRo2k\nsLAwKTU1VRo5cqRkYmKSbXtz+6wEBQVJRkZGUmhoqPTu3TvJ3t5eXs+qVaukfv36SUlJSVJ8fLzU\nuXNnydfXN9d2BQcHSzVq1Mi2HcLPR/RACN+kCxcu8Ouvv6KpqUmRIkVo3Lgxfn5+AFhaWnL27FnS\n0tJITU3lzJkzWFpacu7cORo3bkzVqlWBjGvCvr6+8l97devWRUdHBwALCwv27t2LsrIyqqqq/Prr\nr/JfVP7+/tjY2ABgZmaGnp4ekNH7oK+vT/PmzQGwsbHh+fPnvHr1KttjcHFxwdLSknbt2tG4cWP0\n9PSYOnUqAH5+fnTr1g0tLS2UlJSwtbXlxIkTeY5T69atUVBQyPS4ePHiKCkpUbVq1Wy73S0sLPD1\n9QXg/Pnz1KlTBx0dHUqUKIGPjw93795FW1ubNWvWoKKikqf2REREsHfvXszNzUlJSeHSpUv07t0b\ngDJlymBsbMyVK1ey3fbp06dERUXRunVrAOzt7Vm5ciW1atVCS0uLy5cvAxk9AO3bt8+1HRcvXsTY\n2JgKFSoAYGtry9WrV+U9Lx+eC/+1aNEi+vTpw6FDh7CxscHU1FQ+RiO3c+zcuXNYW1sDYGBgwNmz\nZ9HX1+fMmTPY2dmhpKSEmpoaHTp0kPfQAHTo0AFFRUX09fUpUaIEr1+/5vr16zRq1IiSJUuiqKiY\n69iG3D4r169fl597qqqqdOvWTb6dn58fvXv3RkVFBXV1dTp16pTpHMyuXYLwIaWv3QBByM6+ffs4\nd+4cDRs2BDKuvcbExGBhYUG5cuUoVaoUN2/eJCUlhYoVK1KqVCni4uLw9/fH0tJSXo+mpibR0dEA\nFCtWTP58ZGQks2fP5t69e8hkMsLDw+nXrx8AsbGxmcrq6+vLnw8ODs5Uv4qKCpGRkZQuXTrLMSxa\ntIiGDRuSnJyMpaUlJiYmqKurAxAXF8fGjRvZuXOn/Phy+kLLzYftBNDS0pL/W1FRMdvBjBYWFjg6\nOjJ16tRMX8YTJkzAw8ODsWPHkpSUxLBhw+jTp89H27Bt2zYOHjwIQJEiRbC1tcXKyoqwsDAkScrU\npqJFixIZGUm5cuWy1BMVFZWprJKSEkpKGX+ibGxsOHz4MI0aNeLatWvMnTs31zZFRUVRtGjRTHGR\nJImoqCgga9w+pKqqyqBBgxg0aBCxsbEcP36cuXPnUrZs2VzPsf+2X0NDA8g41z7cX7FixYiIiMi0\n/Xvv37OYmJgscctJbp+VnM5lyDgH582bx9KlS4GMSxp16tTJtV2C8CGRQAjfnJiYGK5du8bVq1fl\nv4BTU1Np3bo1kZGR6OjoYGFhwenTp0lJScHKygoAPT09mjVrhru7+0f3sWzZMpSUlDh06BAqKio4\nOzvLX9PQ0CAhIUH++P2dE3p6elSqVEk+HuJTqaio4OjoyMKFC9m7dy8KCgro6elhamqKvb19rtsq\nKCiQnp4ufxwTE5OnfWenevXqKCoqEhgYyIULF5gyZQqQcdzjx49n/Pjx3L59myFDhtCsWTMqVqyY\na319+/Zl5MiRWZ7X1tZGQUGBmJgY+ZdYdHQ0JUqUyLYebW1toqOjSU9PR0FBgZSUFEJDQylbtizW\n1tbY2dnRqlUrGjRokOsXKkCJEiUyDSaMiYlBQUEBbW3tXLeLjIzk/v378l6mokWLYmdnx/nz5wkK\nCsr1HNPW1pYnKAAhISEUK1aMkiVLypPY9zEoWbJkru0oWrQocXFx8scf1vuhj31WNDU1M53Lb968\nkf9bT0+PgQMHYmJikmtbBCEn4hKG8M05cuQITZo0ydR9rqSkRIsWLTh8+DCQ8Sv68uXL+Pn5yX8N\ntmjRAn9/f/mliNu3b+Pm5pbtPiIiIqhatSoqKioEBgZy8+ZN+R/aOnXqcOzYMSCjm/f9H926desS\nFhbGrVu3gIzbBl1cXJA+YUHbTp06kZSUJB9Y17ZtWw4cOCAf5Ojt7c3+/fvlx/r+y0NPT4+goCDS\n09OJjIzk3LlznxrGXFlYWLBy5Upq1Kgh/1IdPnw4Dx8+BKBq1apoamoik8k+ex/v37P3vSzPnz/H\n39+fZs2aZVve0NAQAwMDeTf6nj17mD59OgCVKlWifPnyLFmyRJ4w5qZ58+aZzgVvb2+aN28u79HI\nybt37xg9ejTnz5+XP/fvv/9y69YtGjZsmOs5Zmpqyl9//YUkSYSFhdG5c2eioqJo06YNe/bsIS0t\njYSEBA4cOCC/TJOT+vXrc+PGDSIjI0lLS5P38PzXxz4rderU4erVq0RGRpKcnJxpgGXbtm3ZvXs3\naWlpSJLEmjVrPnp+KSsrk56eztu3b3MtJ/wcRA+E8M3566+/5JcTPmRubs6aNWvo27cvFStWJD09\nHX19fXm3rJ6eHrNnz2bUqFGkpKSgoaEhH3PwXwMHDmTSpEns27ePhg0bMmnSJKZNm0adOnVwcXHB\n2dmZI0eO0KpVK+rVq4dMJkNNTQ13d3dmz55NfHw8ysrKjBkz5pO+ZBUVFRkzZgxz587FysoKMzMz\nHj58SJcuXQAoX748c+bMATLGXSxatIjg4GAcHR05ePAgZmZmVKpUCUtLy0zd35/LwsKCrl27Zkqw\n7O3tcXZ2lt+N0rt3bwwNDbl9+zYrVqxg48aNed7PrFmzcHV1Zd++fSgrK+Pm5kapUqXkd0R8SCaT\nsWLFClxcXFi6dCm6urrMmzdP/rq1tTUrVqygbdu2H92vgYEBbm5ujBw5kpSUFMqWLcvs2bM/ul3p\n0qVZu3Yt7u7uuLm5IUkSmpqaTJkyRX5nRk7nWP/+/fn3338xMTFBTU2NSZMmUbp0aRwcHAgODsba\n2hqZTIalpeVHk6AaNWrQs2dPunTpQvHixbG2tiYoKChLuU/5rHTp0oUuXbpQqlQp2rdvL7/zpHfv\n3rx48QJra2skSaJ27drZ1vUhXV1djIyMMDExwcPDgwYNGnw0psKPSyZ9ys8nQfjJSJIkTwy6devG\niBEjMDMz+8qt+rkdPXoUHx8fVqxY8bWb8l358Fw+c+YMy5cvz/Z2WkHIK3EJQxD+Y8GCBcyaNQuA\nx48f8+TJE2rXrv2VW/VzS0xM5I8//sDBweFrN+W7EhkZSZMmTXj58iWSJHHs2DHq1av3tZsl/CBE\nD4Qg/MebN2+YOHEiL1++REFBgeHDh8svNQiFz8/Pj1mzZtGtWzecnJy+dnO+O15eXmzatAmZTEal\nSpWYM2dOjgNZBSEvRAIhCIIgCEKeiUsYgiAIgiDkmUggBEEQBEHIs5/qNs6wsLiPF8ojbW11oqIS\nPl5Q+CQinvlLxDP/iZjmLxHP/JVdPHV1tXIo/WVED8QXUlJS/NpN+KGIeOYvEc/8J2Kav0Q881dh\nxlMkEIIgCIIg5JlIIARBEARByDORQAiCIAiCkGcigRAEQRAEIc9EAiEIgiAIQp6JBEIQBEEQhDwT\nCYQgCIIgCHkmEohvxMmTx2nd2pjo6Oiv3ZRMAgL8sbExw9FxKI6OQxkxYiC3bv2da3lX14kATJ48\n/rP3m5qayrp1qxg40J5Ro4YwdGh/rl278ll1PXnyCEfHoZ/dppCQEO7du5PleUfHoSxaNDfTc3v3\n7qRFi4a51vf33wFERUUC0L17BxIS8j6JzqpVyzl69FCm544ePUTXrtby92rw4L54eXnlWs+jRw95\n/vzfPO9fEAShUGeiTElJYfLkybx69QpFRUXmzZtHuXLlMpU5ePAgW7duRUFBATs7O2xtbVm7di2X\nLl0CID09nfDwcHx8fDA1NcXAwABFxYyJMxYvXoy+vn5hHlK+OXnShzJlynLmzCk6d+7+tZuTSb16\nDXBzWwjAy5cvcHEZw44dez+63fz5Sz97nzt2bCMhIZ6NG7cjk8l4/vwZY8eOYsuWHRQtWuyz6/2c\nNgUEXCcxMYGaNbMu6R0U9IDU1FSUlDI+ShcunKNEiZK51nfkyEF69bJHW1snz235GFNTcxwdxwKQ\nnJzMkCEO1KrVgFKlSmdb/uxZX6pXr0n58hXyvS2CIPzYCjWBOHz4MEWLFmXJkiVcuHCBJUuWsHz5\ncvnrCQkJrF69mj179qCsrEz37t0xNzdnxIgRjBgxAoD9+/cTEREh32bDhg1oaGgU5mHku9jYGO7f\nv8uUKdPZsWMbnTt35+HDIFauXIq7+zoANm1aj5ZWURo2bMyyZQuRyWSoq6szdepM3r6N4/fff6NI\nEXW6dbMjPv4te/bsRFFRAUPDX5g0aRpv377F1XUiSUlJNG3anEOH/mL37oPcunUTD4/VKCkpoaen\nz6RJrigrK+fY1jJlyhIfH09aWhrPnj1l6dIF/2uLBq6uMzOVtbZuy5EjpwkKCmTJkgUoKMioXbsu\n1tYdWbhwDmvW/AHA1q0bUVfXwNa2p3zbgwf3s3WrFzKZDIDy5Q3ZtesASkpKBAT44+39JwkJCTg6\njuPmzRucOXOa9PR0mjZtzsCBQ3nzJpTffpuMsrIylStXzdKmp0+fZBvHOXNmUrp0GR49ekjVqtUY\nNsyRTZvWo6SkhL6+AS1atM50jDVr1uLatSs0a9aC0NAQlJSU5PFLSIhn7txZxMXFkZaWxtixLkRF\nRXD+/BmePn0iT8r27t3FlSsXSUtLY+nSlaioqLJw4RxevXpJcnIygwcPp3HjJvj4HMXTcyu6uvqo\nqqpSqdIvuZ5XKioqVK1alVevXqKnpy+vMzU1lcGDh1O8uDYHDuzj7FlftLW1mT59Ctu27URdXZ1V\nq5bL679y5RLh4WEMH+7I+vVrMsVn8uTfcm2DIAg/rkJNIC5fvkznzp0BaNasGVOnTs30+q1bt/j1\n11/R0sqYt7tBgwYEBARgamoKZHRre3l5sW3btgJp38yZrhw69FeetlFQkJGenvOK6B06dGbmTLdc\n6/D1PUWzZi0wNm7KggVuhIW9oUqVqoSHhxEXF4eWlhYXLpxjwYKluLnNwMVlKuXKlWffvt3s27eL\ndu2sePjwAXv3HqZYseIcOLCPJUtWoqWlxahRQ3j8+BE3b/pjaFiJsWMnsG/fbt6v4r58+SJWrFhL\n0aLFWLNmBX5+p2jXzirHtt67dwc9PX0UFRVZsWIxI0eOoVat2uzYsZ3du72pX98oyzbLly/GxWUq\nlStXYfbs6aipqZGSksybN6Ho6elz6dIF5s1bLC//9u1bVFRU0NDQzFTP+1/5AI8fP8LLax8qKirc\nvHmDNWv++F+vVSd69OjNnj3etG3bDju7Xvz55xYePQr6T5sWZRvHBw/uM2vWXLS1dejSpT2jRo3F\nysqG4sWLZ0keANq0acuhQ3/RrFkLTp8+SatWJjx9+gSAXbu8MDZuRocOnXn69AkrVixm+fI1VK5c\nlfHjJ2JgYABApUq/4ODQnxkzpuLvf534+IzjX7VqPeHhYTg6DsPLay8eHqvZuHE7WlpFGTTIPtdz\nCiAyMoLbt28zcuQ4Tp48TokSJZkyZTrR0dGMGTOcrVu9MTZuSps2bbPtXXkvNDSEdes2ERLyOkt8\n3p+fgvC5fH19ARV+/bXu126KkEeFmkCEh4ejo5PRbaugoIBMJiM5ORkVFZUsrwPo6OgQFhYmf3zi\nxAlatGiBmpqa/LkZM2bw8uVLjIyMcHZ2lv9i/Z6cOuVDv36DUFRUxMSkLadPn6BnT3uaN2/F1auX\nqF27LqqqKujq6nHv3l0WLMhISFJSUqhRoyaQ0TNQrFhxAIoWLcqUKc4A/PvvU2Jionn27Jn8y71F\ni1bs2LGNyMgIXrwIZupUFwDevXsnr+NDf/8dIB9DoKGhgavrLACePXtKrVoZXzwNGjRk8+b12SYQ\nz5//S+XKVQD47bffAWjXrj2+vicxM7NAQ0MTHZ0S8vIyWcalqvf279/D6dMnePv2LT169EZf34DK\nlavIzxs1NTUcHYeiqKhIdHQ0sbGxPHv2FBMTMwDq12/IlSuXMrUp5ziWk1+CKFlSl/j4t7m8c1C3\nbn0WLHAjKekdZ8/6smDBMrZu3QjAP//cJjo6Ch+fowAkJb3Lto46deoBoKurR3z8Wx48uC+PY8mS\nuqioKBMdHYW6uob8skdOf2x9fU8SGHiP5ORkIiMjmDFjOtraOty5c5tbt25y+/bf/2tLEikpKbke\n23s1atSUf66yi49IIITPtX79GlxdJ6OursHJk2epUqXqxzcSvhkFlkDs3r2b3bt3Z3ru1q1bmR6/\n/xWck/++vnfvXmbNmiV/PHr0aFq2bEmxYsUYNWoUPj4+WFpa5liftrZ6rguNrF69gtWrV+Tapvz2\nfoDeunXueHjIePfuHVpaWjg5jaBTJ2v+/PNP0tLeYW3dHl1dLdTVi+DtvSNTovTixQvU1FTR1dUi\nOTmZ5csXceDAAXR1dRk2bBjFi6ujpqZE8eLq6OpqkZ6egKKiAgYG2ujr67NzZ84D7YoXV8fY2Bh3\nd/csrykoyOSrvL14oYyamgrFi6ujqqqMrq4WMlnG60pKillWg+vZsxtOTk7o6mrTtWunTK8bGpYC\nJBQUkilRogRDhw5g6NABrFy5EgWFNIoXV0dTM+NYXr58yZ49Xuzfvx8NDQ1sbGzQ0dFAWVkRHR1N\ndHW1KFZMDRUVpUxtyimO79sOoKSkgLa2OhoaqmhqqmU5BhUVJfT1i9GqVUt8fA5StKgmVaqUQ1FR\nAV1dLTQ01Bg3bgb169fPsp22tga6ulooKiqgr18MDQ0N1NVV0NRUpUgRFbS0/n9/6elplCypJT8G\nAFVVpUxlALS01LCxsWbSpEkkJibStWtXatasia6uFlpa6jg6jsLGxiZTW9TUlClWrIi8LSVLaqKh\noYGysgwtrYxEvWjRjLYmJWlkG5+CWunvW/YzHnN+mzt3Lq6u0yhevDjR0dEMHdqPq1evfveXpHPz\n/jutoH/oFtb5WWAJhK2tLba2tpmemzx5MmFhYVSvXp2UlBQkSZL/igTQ09MjPDxc/vjNmzfUq5fx\n6ywhIYGQkBDKli0rf/395RCAVq1aERQUlGsCURBLxurqan3RMuE7d+6lSxdbnJzGARknWM+eXfj7\n7/uUKfMLDx4EERYWiYvLVMLC4qhUqTKHDvnQtGlzTp3yoXhxbcqUKUtqajphYXHExEQjkykAaty5\n85Dbt/8hPDwWHR19rl0LwMioOUeOnCAtLZ3kZAXS0tK5du0WFStWYs8eb+rVM5L3FgBERyeQlJSS\n7TFWqFAJP7+L1K5dBz+/C1SsWCVTeUmSCAuLo3x5Q86cuUytWrWZN+93evVywNCwIkWKaLBnzz6W\nLHGX1/8+np07d2f69Fm4us5CSUmJhIR4rl8PoE0b00z7ePLkJUWLFiMhIZ2bN6/z4sVLQkOjMTAo\nw+XL/ujrV8DX9xzJyamZ2vSxOAKkpqYTGRlPYmIKCgrxWWLwvs4mTVrh6jqJIUNGEBYWR1paRh2V\nK1fn4MGjlC1bmadPn3D16iV69rQnNTWd8PBYtLUzyoaHvyUhIZ2EhGTi4t5haFiFM2fO07hxK0JD\nQ5AkSElRJDo6hidPXlGkSBGuXbvOL79Uz9SmuLh3JCQky59zcBjI3Llz+e23OVSsWI2jR49jbNya\nqKhIdu3yYtiwUSQlpRIZ+ZawsDjU1NR58OAZpUuXwd8/gHLlKgHI64yMjM82Pqqqn3/+f4++9DP/\ns5Mkiblzf2fFiiWULVsOPz9f5s9fxMaN6xk4cAgrV677LnuSPyY4+DmDBjmgoqKKt/deNDUL5ks+\nu/Pzh1jOu3nz5hw/fhwAPz8/jI2NM71et25d/vnnH2JjY4mPjycgIICGDTNuiQsMDKRSpUrysnFx\ncQwaNIjk5GQArl+/TpUqVfjenDrlg7V1B/ljmUyGlZUNp075IJNlDDqMj38rv14+ZswEtm/fjKPj\nUI4ePUzVqtUy1VesWHEaNTJm8OC+bN68gd69HXB3X4qVlQ23b9/E0XEokZERKChkvPWTJ09n7txZ\njBw5mNu3b+VpNP7YsRPw8FjN6NHDCQy8m2kQ5IfGjJnAqlXLGDFiEFpaRTE0rAhkjB/Q1dVDXT3r\nL44ePfpQvXoNhgzpy6hRQxg9egRGRo2wtu6YqVyVKlUpUkSdESMGcvr0CTp16sqSJQuwte3FkSMH\nGT/ekbi4rH/sPxbHD9Wu/Suents4ceJYtq/Xq9cAFRUVWrc2yfR89+49ePkymJEjB7NggRv16jWQ\nl3d1ncSTJ4+zra9t23akp6fj5DSMmTOn4uIyFQUFBQYOzLg909V14kcHUAKYm1vy5s0brl27gqmp\nGUWKqDN8+EAmThwnv2xSt259li9fhL//Nbp1s2PSpHFMm+ZCxYqVPlK7IOSdJEm4uk5ixYolVKxY\niYMHj1O5cmVmzpxD/foN2LXLC0/Pghnj9rkkSeKff26xePF8VqxY8lm3Xd+4cR0LCxP+/vsm165d\nYeBAh0++hPgtk0kfu46Qj9LS0nB1deXZs2eoqKgwf/58SpUqxfr162nUqBH169fn+PHjbNy4EZlM\nhr29PR07Znxh+Pj4cOnSpUyXMLZu3cpff/2FqqoqNWvW5Lfffss1cy2IXw3fy6+RkJDX/PvvM4yN\nm3Lnzm02bvRg2bLVX7VNbm4zaN++Aw0a/P+8Cd9LPL8XIp75T8T086SlpeHiMpY//9xK9eo12L37\nAPr6BvJ4Bgc/p23bFiQmJnL06Gl+/bXOV2trUlISFy+e4/jxo5w4cZxXr17KX/vll8qsXr0+09+t\n3Bw69BejRg0lOTmZ2bPnceaMLydP+tCjR2/c3dfme29LYfZAFGoC8bX9zAlEXFwcM2ZMITExAUnK\n6D2oXr3mV2lLUlISTk7DqFGjJuPGTcz02vcSz++FiGf+EzHNkJKSwtOnT5DJZPzyS2V5r2ZOZZ2c\nhrNv327q1KnHzp37KVEiY+D0h/E8efI4ffrYYWhYkVOnzn3RnC95FRkZwcmTPvj4HMPP77R8AHXx\n4sUxM7PAwsIKf//reHisRlHBEo4wAAAgAElEQVRRkTFjnHF2npTjbe+SJLFy5TLc3GaioaHJhg2b\nMTOzID4+nq5drbl5M4Dx413y/VZokUAUkJ85gfheiHjmLxHP/PezxTQpKYnHjx8RFBTIgweBBAU9\nICgokMePH5GamgqAllZR6tc3omHDhjRo0JAGDRpRsmRJ+fbDhg3k6NFDNGpkjJfXnkyJwX/j6eY2\nE3f3pVhbd2TTpu2FMh7C3/8a3bt3IiEhHgBDw4pYWlpjadmexo2bZLqF/MKFc4wePYIXL4KpU6ce\nq1evp1q16pnqS05OxsVlLF5ef1KmTFn+/HOX/I41gLCwMNq3b8u//z5j8eIV9O07IN+ORSQQBUQk\nEN8+Ec/8JeKZ/36WmD558phhwwbyzz+3Mt1WDRkJQ9Wq1ahWrTqpqakEBPjz6NHDTGUqVDDEyKgR\nISGvuXTpAi1btmbrVi80NTPP7/LfeKamptK9e0cuXbrA7NnzGDZsVI5tfPEimM2b/8DLaztmZhYs\nX746156Q7ISHh2Nm1pKQkNe4uEyhQ4fOVKlSNdfEJTY2BlfXyXh7e6Kqqsq0aTMYOnQkCgoKREVF\nMnCgAxcvnqdevfps374TfX2DLHU8efIIa2tzoqKi2LbNK9f5d/JCJBAFRCQQ3z4Rz/wl4pn/foaY\nJiQk0L69Gffu3aFhw8bUqFGLatWqUbVqdapVq46BQaksX7DR0VEEBNzgxo3rBAT4c+PGdfnaPubm\nFvzxxzaKFCmSZV/ZxTM0NART0xZERUVy4MAxGjX6/wH3kiRx5colNmxYx9Gjh0hPT0cmkyFJEuPG\nTWDKlOmffJxpaWn06NGVc+f8mDZtBmPGOOclTBw9epgJE0YTHh5O8+YtcXaehIvLWB4/foS1dUdW\nr16Purp6jtvfuHGdrl1tkMlk7Nt3+JPHVeRGJBAFRCQQ3z4Rz/wl4pn/foaYjhkzEi+vP+nbdyCL\nFy//+AbZkCSJp08f8/r1axo3bpLjWIGc4nnhwjm6d++IgUEpTp++gLq6Ovv372HDhnXcvfsPkDGh\n2pAhw2nVqg2dO7fn2bOnrFixhl69Pj5TK8C8eb+zbNliLCys2LrVK8+9F5BxOcLZeTTHjx+RP+fo\nOBZX15mfVN/x40fp3783Ojo6HD588pPusMqNSCAKiEggvn0invlLxDP/FXRMJUni3LkzJCYm0rBh\nY/lYgsLi6bmNceMcqVu3PocO+WSa+bcg5BbPZcsWMW/ebGrUqEVo6GsiIyNRVFTE2rojgwcPx9i4\nibwn5NGjh7Rv35a3b9+yc+d+WrbMOvX8h3x8juHg0IMKFQw5depctrPwfipJkti5cwerV69g2LBR\n2Nv3y9P2W7ZsZOLEcVSsWIkjR0590XsuEogC8i0nECdPHsfNbQYHDvhQvPjnn8j5LSDAn+nTJ2No\nmDEvQFLSO4yNmzF48PAvqvf9olb/9TnxbN3aWD61c1paGiVKlGTq1OnZzi/x3pkzp2nTpm3eGv0d\nEglE/ivImAYHP2fixHGcPn1S/lzFipVo2LAxjRoZ/+9yQk35CsT57Z9/btG+vRlFihTh1KnzhbJK\na27xTE9Pp08fW06fPomOjg4ODgPo338QZcqUzbb8pUsXsLXthLq6BkeOnMxxfpdnz55iZtaK5OQk\njhw59VVvGX1vzpxZrFixBCOjhuzff/SzE7fCTCAKdS0MIWffy3Le6enpjBs3ilu3blK3bv2PbFk4\nNDU1WbVqvfzxxo0e7NrlRf/+g7Mt//r1K06d8vkpEgjh+5CWlsaGDWuZP9+NhIQEWrUyoUmTpvj7\nX+PGDX927/Zm925vADQ1tahf34imTZvRrZtdvk36FRMTzcCBDiQlJbFp0/ZvYol3BQUFNm7czuXL\nF2jWrGW2Yyg+1KxZC5YtW4Wj4zB697bl2LHT6OrqZiqTmJjIwIEOxMbG4O6+9ptIHgCmTp3Oq1cv\n2b3bmwcP7n8zf19zIxKIb8D3tJy3goIC1arVIDj4ObVq/cqcOTMJC3vzvw/lUJo3b4mj41AaNTIm\nIMCf6OhoFixYRsmSJZk1y5U3b0LlC1dBxqqaHy4JvmzZYgICbrJ7tzeKiooEBQXSt+9Arl69zMOH\nDxg5cgytWrXJNZ41a9bm1CkfAM6e9cXb+08UFZWoVq0GTk7jWLp0Affv32Xz5g2kp6dTvHhxunXr\nwZMnj1i6dCGrVq2nR4/OtGzZhn/+uYWmphaLFi3/rOujwvcnLS0tz7/wExMT+fvvAK5fv4q//zVU\nVFSxsrKmXTtLtLSK5rrtP//cxtnZib//vomOjg4LFizFzq6XvGs+PT2dhw+D8Pe/Jq///PkznD9/\nhoUL59KyZRvs7ftiZWXz2b9aJUnCyWkE//77jLFjJ2BunvOSAIVNXV2dtm3bfXJ5O7tePHv2lMWL\n59O3b0/27TssTzwkSWLSpPHcuXMbB4cB9OzZp6CanWcymYyVK9cxduyETMsJfMtEAvEfOkbZL2uc\nMHI07wZlrEipNXIIylcvZ7ygIEPnf8t5pxg1JG79FgDUtm9BffliIm/c+eg+v6flvBMSErh27Qrm\n5hbExcXSuHETrKxsePnyBb/9NpnmzVsCGat2rlixlrVrV3LunC/lypUnNTUVD4/N3L17hz17dgJk\nWRJ827ZtVKv2K48eBeHpuYdbtwKYNes3du8+yN27/7B3785cEwhJkjh71pdq1aqTkJDA1q0bWbdu\nMyoqKvz222Ru3/6bXr0c2LdvFwMGDGHjRo9s63n16iWWltY4Oo5l6ND+PH78kCpVcp7uWvj+vX0b\nx+DB/Th71o8yZcpRsWJFKlasRMWKv/zv/5WoUMEQ0CI0NIRr165w7dpVrl+/wu3bt+RzIrx38OB+\nVFRUaNWqDTY2nbC0bJ9p1dmEhAQWL57P2rUrSUtLo3v3Hvz++7ws178zkvaMux/69OkLQFRUJCdP\n+uDpuU2eTGhra2Nr25M+ffplStI/xapVKzh+/AgtW7Zm0qRpnxW/b4mLyxSePXvKnj07cXIazvr1\nm1FQUMDTcxve3p7UrVufOXMWfO1mZqGgoPBdrUgqEohvwPeynHd6ejovXgQzbNgoqlSpRmpqKvfv\n3+XgwX3IZArExsbIt3nf/aanp0dMTAxPnz6VdxXWqlUbVVVVIOuS4J6em6hW7Vf5ct0lSpSkXLny\nFClSBB0dHd6+zbq89tu3b+XLjT979pR27Szp2tWOBw8CCQ0NYfx4RwDi498SEhLySQOUNDQ05L8C\n9PT0st2v8OOIjY2hZ89u+Ptfo1KlX4iPj+fsWT/OnvXLVE4mk6Gjo0NERIT8OWVlZerWrUfDhsY0\nbtyERo0aExMTw+HDBzh8+CCnTp3g1KkTKCoq0qxZS6ytO2BgUIoZM6by77/PKF++AgsXLsPU1OyT\n26utrYOdXS/s7Hrx+PFDPD234+3tyfr1a1m/fi1GRo2wt++HjU3Hjw4OvHTpAnPnzsLAoBTr1m0q\nsPEVhUkmk7Fs2SpevAjm4MH9GBpWpEOHTkyZMgFtbW02btxW4INDfwYigfiPT+kxiFuzQf5vXV0t\nIrMZAPTOoT/vHPp/tK43b0K5d+8Oq1YtRyZ7v5y3Jj172tO6tQl79+4iJiaa1q1NAVBTU2PlSo9M\n92C/fv0KJaWMyw4pKSksXbqQLVt2UKJESSZOHPu/UhIKChnbvN9WSUmZkiV1M40fyM77MRCSJDFs\n2AB++SXji/XkyePExsayevUfxMbGMniwg3ybD/8IZfR2SP9bJfTD5zJLTU2RXyb4cPusdWX24RiI\nVauWo6uri5KSEsrKGZctli5dlal8QIC//N8fxvHDX5D//SP6E401/ulERUXSo0cX/v77Jl272rJq\nlQdKSkq8ffuWf/99xtOnT3j69AnPnmX8PyTkFUZGjWjcuAmNGzehbt36Wa7NGxiUolq16jg7Zyya\nduTIIY4cOSDvLYCMX5sjR47GxWXKFy1h/csvVZg+/XemTPkNH59jeHpuxdf3FDduXGfChDE0aNAQ\nE5O2mJi0pV69BpnO7dDQEIYOzZgFccOGrVnGC3zPVFVV2bLFk/btzXB3X8q2bZtITk5m69Yd38T4\njh+BSCC+slOnfLJdzvvlyxfUqvUrS5bMJzY2FheXqQBUrlyFK1cuZVmG+r2EhHgUFRUpUaIkoaEh\nBAbeJzU1ldKlyxIYeB8TEzOuXLkEZPRUADx9+iTH5bw/JJPJ5GMI1q3bRHR0NKVKlUZBQYGzZ31z\nXV2ufPkKnDyZMS7hn39uyVdRrVjxF+7cuU3t2nW4eTOA2rWzv4T0qfr3H8ygQfa0bWtB+fKGPHv2\nlKioSLS1ddi40YOOHbugoKBAWloakNHT8H4J+du3//6ifQvfn7CwMGxtO3Hv3h1693ZgyRJ3+Res\npqYmtWrVzjQFMeT9LoxKlX7ByWksTk5jefnyBUePHiIw8D79+g2Ur4qaH5SVlbGx6YiNTUdevAhm\n925vTp704caN61y/fpWFC+eira1Nq1YmmJqa0bJlaxwdh/HmTSizZs3F2LhJvrXlW6GjU4IdO/bQ\nvn1bIiMjcXGZgqmp+ddu1g9DJBBf2alTPri6/v8Kox8u592v3yBq167Lw4cPMi3nvXDhHDw9t6Ki\nosrMmW7Ex8fLt/9wOe/KlavIl/NeudKDqVOd5QMc/7uct7JyRm9Ex45dc23vr7/WpXTpMhw69Bdt\n2pgyefJ47t27g7V1R/T09Ni8eUO22zVp0pwjRw7i6DiUypWroKurB2Qs6vV+EKWWlhZLly7m4sXr\nnx1PTU1Nevfuy6pVy5g5cw5jxjgzYcIYVFSUqVKlGiVL6qKkpMyDB4G4uy/Bzq43Li5juH//rny5\nbeHnEBLymu7dOxIU9ID+/Qcxf/6SAh8oW6ZMWYYMGVGg+wAoW7Yc48a5MG6cC9HRUZw/fxY/v9P4\n+Z3mwIF9HDiwT17WxqYTw4fnPF30965SpV/Yv/8oV69eztc1JwQxD8QX+17us/8Wl/POzvcSz++F\niGf2XrwIplu3Djx9+oThwx2ZNWvOJy/a9D3HVJIkHj4Mws/vFH5+p0lLS2PTpu0fvVOkIH3P8fwW\niXkghHynoaHJzp2ebNmyQb6ctyD8jJ49e0q3bh0IDn7OuHETmDz5t0JZ8fFbIJPJqFq1GlWrVst1\nkSpB+BQigfhJZFweWPXxgoJQAI4fP8rr168wMWmLoWHFr9aOR48e0q1bB16/fsXkya6MHz/xq7VF\nEL53IoEQBKFAvX79ioED7eV3ufzyS2XMzNphampO06bNC+V2usePH7Jrlxfbtm0mIiKCGTPcGDVq\ndIHvVxB+ZCKBEAShQG3cuJ7U1FR69uxDVFQk58+fw8NjDR4ea1BXV6d585aYmppjbm6Rr7fXZSwF\nvZ+dO3dw40bGwFxNTS0WLlxG//6D8m0/gvCzEoMov5AYAJS/RDzz19eOZ3x8PPXr10BJSYmAgHuo\nqamRlJTElSuXOH36JL6+JwkKeiAv7+AwAFfXGWhr63zW/lJSUvD1PcWuXV74+BwlOTkZBQUFWrc2\nwc6uF1ZWNqirq3/RMX3tmP5oRDzzlxhEKQjCD2HXLi+io6Nxdp4kv1ShqqpK69YmtG5twu+/zyU4\n+DmnT59k06b1bN++mWPHDjFz5hxsbXt+8uDG0NAQ/vjDA0/PrfJ5PapVq46dXW+6d7ejVKnSBXaM\ngvCzEj0QX0hkz/lLxDN/fc14pqen07x5Q4KDnxMQcA89Pb1cy6ekpLBu3WoWL55HYmIizZu3ZOHC\nZbmuDRAU9IA1a9zZs2cnycnJ6Ojo0LWrLT169KZOnXoFcneFOEfzl4hn/irMHgixvKAgCDlKT08n\nMjKCxMTEPG976pQPjx8/ols3u48mD5Axk6KT01jOn79Gu3aWXLx4njZtmjJ//uxM+5ckicuXL2Jv\nb0eLFo3YsWM7ZcuWY9Gi5dy8eZ+5cxdRt279n+bWTEH4WsQlDEH4iYWEvMbP7zSvXr0kIiKc8PAw\nIiIiCA8PIzw8nMjICNLS0ihXrjy+vhc+ujDThzw81gDkeb6B8uUrsH37To4dO8K0aRNZunQRe/fu\nZt68RSQmJrJ69QoCAm4A0LBhY0aNGoOlZfsfYhEoQfieiARCEH4yjx8/5MiRwxw7dogbN/yzLVO0\naDFKliyJoWFF0tPTuXHjOtOnT2XFijWftI87d/7h/PmztGzZhpo1a+W5jTKZjPbtbWjVqg2LF8/H\nw2M1vXvbyl+zsrJh1KgxNG5snOe6BUHIHyKBEIQfnCRJ3Lp1k6NHD3Ps2GEePAgEMlYcbdmyNZaW\n7alSpRolSpREV1cXHZ0SqKioyLdPSUnByqotXl5/0qFDJ8zMLD66z/XrMxKN4cNHflHbNTU1mTnT\nDVvbnixbtghtbR2GDRuZ44JvgiAUHjGI8guJAUD5S8Qzf/n5HWP8eGdevnwBZCwH36ZNW9q3t6Fd\nO0t0dEp8Uj13796hXbvWlChRkvPnr+Z6KSM0NBQjo1qUL1+BCxeuF/gCVYVNnKP5S8Qzf4lBlIIg\nfLFnz54yYMAAoqIi6d69B5s2/cn9+0/Zts2Lnj37fHLyAFCrVm3Gj59ISMhrfvttSq5lN2/eQHJy\nMkOHjvzhkgdBEP6fuIQhCD8gSZJwdh5NQkICa9f+Qbdudl9c5+jR4zl27Aje3p506NAJc3PLLGUS\nExPZunUj2tra2Nn1+uJ9CoLw7RI/DwThB+TpuY3z589ibW1N1662+VKnsrIyK1asQVlZGWfnMcTE\nRGcps3fvLiIiIujXb9AXz/goCMK3TSQQgvCDef36FTNmTENLqyjr1q3L1/kQcruUIUkSHh6rUVZW\nZuDAIfm2T0EQvk0igRCEH4gkSUyaNJ64uFhmzJhN2bJl830fo0ePp06denh7e3Ly5HH5835+p3nw\nIJBOnbpiYFAq3/crCMK3pVATiJSUFJydnenVqxf29vYEBwdnKRMTE8OgQYMYPXr0R7cLDAykZ8+e\n9OzZkxkzZhTacQjCt+rAgX0cP36U5s1bYm/fr0D2oaysjLv7WvmljOjoKAA8PFYDMHx43iaOEgTh\n+1SoCcThw4cpWrQoXl5eDB8+nCVLlmQpM2PGDIyMjD5puzlz5jB16lS8vb15+/YtZ8+eLZTjEIRv\nUUREBFOnulCkSBGWLHEv0DsgatashbPzJEJCXjN9+lQCA+/j53eaZs1aUKdOvQLbryAI345CTSAu\nX76Mubk5AM2aNSMgICBLGTc3tywJRHbbJScn8/LlS+rUqQOAiYkJly9fLuAjEIT8ExoaQlxcbL7V\n5+o6ifDwcCZNcqVSpV/yrd6cODmNk1/KGDVqKJD3aasFQfh+FWoCER4ejo6OTsaOFRSQyWQkJydn\nKqOpqflJ24WHh1O0aFF5mRIlShAWFlaArReE/PP2bRytWhnToYMlSUlJX1zfyZPH2bt3Fw0aGDFs\n2JfN/vipPryU8c8/tzA0rEi7dllv7RQE4cdUYPNA7N69m927d2d67tatW5kef+4kmNlt9yl1aWur\no6SU/wvuFNQsXz+rnyGeBw/uIioqiqioKNasWcrcuXM/u67Y2FgmTRqPsrIyW7duwcAg8yyRBRnP\n1q2bMGPGDFxdXZk40SXLvn9UP8M5WphEPPNXYcWzwBIIW1tbbG0z338+efJkwsLCqF69OikpKUiS\nlGnO/Zzo6ell2U5XV5fo6P+/Dz00NPSjSwZHRSV83sHkQkzDmr9+lnh6eGxAJpNhYFCKBQsW0Lq1\nOQ0aNPysulxcxvPixQtcXKagr18hU/wKI55DhjhhbNySOnXq/RTv3c9yjhYWEc/89cNOZd28eXOO\nH8+47cvPzw9j409bSS+77ZSVlalUqRL+/hmrCZ44cYKWLVsWTMMFIR8FBT3g+vWrtG5twpo1G0hP\nT8fJaTiJiYl5ruvixfNs3bqRGjVqMmaMcwG09uNkMhl169bP1/kmBEH49hVqAtG+fXvS09Pp1asX\nnp6eODtn/MFbv349N2/eJC0tDQcHB+bOncu1a9dwcHDg8uXLOW43depUli5dSs+ePSlfvjzNmjUr\nzMMRhM+yY8d2AHr3dqB585YMGTKchw+DWLBgTp7qiYiIYNw4RxQUFFi2bNUn9eYJgiDkF7Ea5xcS\n3W/560ePZ0pKCnXrVictLZXbt4NQVVUlPj4eU9PmPHv2lIMHfTA2bvLRekJDQ7C17URg4H3GjHFm\n2rTs50H50eP5NYiY5i8Rz/z1w17CEISf3cmTPoSHh9G9ew9UVVUB0NDQwN19HQCjRw8nISH3sTov\nXgTTsaMlgYH3GTp0BFOnTi/wdguCIPyXSCAEoRDt2LENgF69HDI9b2zchOHDHXn69Alz5szMcfsn\nTx7TsaMlT58+Ydy4CcyePV+MPRAE4asQCYQgFJKQkNecOnWCunXrU7v2r1lenzzZlSpVqrJhwzou\nXbqQ5fXAwPt07GjJixfBTJs2gylTpovkQRCEr0YkEIJQSHbt8iI9PZ3evR2yfb1IkSK4u69FQUGB\n0aNH8vbtW/lrt2//TefOVrx5E8qcOQu+2h0XgiAI74kEQhAKgSRJ7NixHTU1Nbp27Z5jOSOjRjg6\njuX582f8/vtvAFy/fpWuXTsQFRXFsmWrGDJkRGE1WxAEIUcFNpGUIAj/7+rVyzx58phu3ewoViz3\n2RpdXKZw4sQxtmzZiL6+AStXLicp6R1r1/5B1662uW4rCIJQWEQPhCAUAk/PjMGTOV2++JCqqior\nV65DUVGRBQvmkJqawqZNf4rkQRCEb4pIIAThM4SFhREaGvpJZePiYjl06C/KlzekefNPmy21bt36\nuLrOomTJkmzb5o2VlfWXNFcQBCHfiQRCEPLo0aOHtGzZiGbNjLhy5dJHy//11z4SEhLo3dseBYVP\n/8iNGjWau3cfY2pq9iXNFQRBKBAigRCEPAgJeU2PHl2IjIwkISGeHj264Ot7MtdtduzYhkwmo0eP\n3nnen7hNUxCEb5VIIAThE8XGxtCzZzeCg58zebIr27d7I0kSDg49OXhwf7bbBAbe58YNf0xM2lKm\nTNlCbrEgCELBEQmEIHyCpKQk+vXrzb17dxgwYDDjxrlgZmbBzp37UVVVY+jQAfJFsj704cJZgiAI\nPxKRQAjCR6SlpTFq1FAuXjyPtXVH5s5dJL+00LRpc/btO0Tx4sUZO3YUHh6r5dslJyezZ483Ojo6\nWFi0/1rNFwRBKBAigRCEXEiShKvrJA4e3E/Tps1Zu/YPFBUVM5WpV68BBw4cR1/fgN9+m8LChXOR\nJIkTJ44THh6OrW1P+cJZgiAIPwoxkZQg5GLlymVs3LieGjVqsm2bF2pqatmWq1atOocO+WBr24nF\ni+cTFxfLo0cPgawLZwmCIPwIRAIhCDnw9vbEzW0mZcqUxdt730dnkDQ0rChPIjw81gBQv34Datas\nVfCNFQRBKGTiEoYgZOPUKR/GjXNEW1ubnTv3U6pU6U/azsCgFAcOHKNevfoA2Nv3L8BWCoIgfD2i\nB0IQ/uPYsSOMGDEIFRUV/vxzF1WrVsvT9jo6Jdi37wjXrl3BxKRtAbVSEATh6xI9EILwPykpKcyY\nMY1+/XohSRIbNmyhUSPjz6pLU1MTU1MzMRGUIAg/LNEDIQjAy5cvGDKkP/7+16hcuQp//LFNjF0Q\nBEHIheiBEH56vr4nadu2Bf7+1+jatTsnTpwRyYMgCMJHiB4I4aeVmprKokVzWbZsMSoqKixatJy+\nfQeIyw6CIAifQCQQwk8pNDSE4cMHcfHieSpUMGTjxm3UqVPvazdLEAThuyEuYQg/nStXLmNq2oKL\nF8/Tvn0HTp8+L5IHQRCEPBI9EMJPJS4ulsGD+xIVFcns2fMYOnSkuGQhCILwGUQPhPBTWbhwHm/e\nhDJhwmSGDRslkgdBEITPJBII4adx//49/vhjHYaGFRk5cvTXbo4gCMJ3TSQQwk9BkiSmTJlAWloa\nc+cuzHFRLEEQBOHTiARC+Cns37+HS5cuYGnZHjMzi6/dHEEQhO+eSCCEH97bt3HMnOmKmpoas2fP\n/9rNEQRB+CGIBEL44S1evICQkNc4OY2jQgXDr90cQRCEH0Kh3saZkpLC5MmTefXqFYqKisybN49y\n5cplKhMTE8P48ePR0NDA3d0dyJgxcNq0aTx//py0tDQmTpxIw4YNcXBwICEhAXV1dQAmTZpE7dq1\nC/OQhG/cgweBrF+/hvLlDXF0HPu1myMIgvDDKNQE4vDhwxQtWpQlS5Zw4cIFlixZwvLlyzOVmTFj\nBkZGRgQGBsqfO3DgAEWKFMHLy4uHDx8yZcoU9uzZA8C8efOoWrVqYR6G8J2QJImpU11ITU3FzW0+\nRYoU+dpNEgRB+GEU6iWMy5cvY25uDkCzZs0ICAjIUsbNzQ0jI6NMz3Xs2JEpU6YAoKOjQ3R0dME3\nVvgmSZJEenr6J5U9eHA/58+fxcysHRYWVgXcMkEQhJ9LoSYQ4eHh6OjoZOxYQQGZTEZycnKmMpqa\nmlm2U1ZWRlVVFYCtW7diY2Mjf83d3Z0+ffowffp03r17V4CtF74FCxa4UaGCPmPGjOTu3Ts5lnv7\n9i3Tp09FRUUFN7cFYsIoQRCEfFZglzB2797N7t27Mz1369atTI8lScpTnZ6enty9e5d169YB0Ldv\nX6pVq0b58uWZMWMGnp6eDBo0KMfttbXVUVJSzNM+P4Wurla+1/kzyymeL1++ZPVqd5KSkvDy+hMv\nrz8xNTVl7NixWFtbo6Dw//nw0qVzef36Fa6urhgb/9zrXIjzM/+JmOYvEc/8VVjxLLAEwtbWFltb\n20zPTZ48mbCwMKpXr05KSgqSJKGiovJJ9e3evRtfX1/WrFmDsrIygPxyCICpqSlHjx7NtY6oqIQ8\nHsXH6epqERYWl+/1/qxyi+eMGb+TlJTE4sUrMDAwwMNjLb6+vvj6+lKxYiWGDBlOz559CAkJYcmS\nJZQrV57Bgx1/6vdHnF7KOyoAACAASURBVJ/5T8Q0f4l45q/s4llQCUWhXsJo3rw5x48fB8DPzw9j\nY+NP2i44OBhvb29WrVolv5QhSRL9+/cnNjYWgKtXr1KlSpWCabjw1b169ZLt27dQvrwhvXrZ066d\nFXv3HuTMmcv06dOXV69eMnXqROrVq4mDQw9SUlL4/fd58jt0BEEQhPxVqHdhtG/fnkuXLtGrVy9U\nVFSYPz9jUp/169fTqFEj6tSpI08KQkNDcXBwYOTIkVy+fJno6GiGDh0qr2vjxo3Y2dnRv39/ihQp\ngr6+Pk5OToV5OEIhWr58McnJyTg7T5T3QAHUrFmLZctWMW3aTLZt28SmTRt4/PgRJiZtad/eJpca\nBUEQhC8hk/I6EOE7VhDdZKL7LX9lF88XL4IxNq5HmTJluXTpBkpKOee9ycnJXLhwloYNG1O0aLGC\nbu43T5yf+U/ENH+JeOavwryEUag9EILwOZYvX0JKSgrOzpNyTR4AVFRUMDU1z7WMIAiC8OXEVNbC\nN+3583/ZsWMblSr9Qrdudl+7OYIgCML/iARC+KYtX76Y1NTUT+p9EARBEAqPSCCEb9azZ0/x9vak\ncuUqdO1q+/ENBEEQhEIjEgjhm7Vs2SJS/6+9e4/vuf7/P3577/DGbIutEcVHKYeKFc357JNYplKT\nFdJH9XHqQA4zsglhrEQkorL8LHMo9mEoLIchYR+fyrF8MKfNDMPOr98fvt61z8Zs3nu/95779XJx\nufR+vZ/v1+v+ftA8PF+HZ3Y2w4YF4+xs/QeAiYhI8amBkFLp99+PsGTJYurWrcczz3S3dxwREfkf\naiCkVProo6nk5ORo9kFEpJRSAyGlzpEjh4iOjqJ+/YcJCHjW3nFERKQAaiCk1ImICCc3N5dhw0bl\nWSBLRERKD/10llJl//79LF8ezcMPP8rTTwfYO46IiNyAGggpVd5//31yc3MZMSJEsw8iIqWYfkJL\nqfHNN/+PxYsX06CBL126PG3vOCIichNqIKRUiI1dzTvvDKJy5crMmjUXk8lk70giInIThT4b+NCh\nQ0RHR3PhwgX+unBneHh4iQaTO8fWrZt5/fVXKFeuPKtXr6Z27fr2jiQiIoUotIF455136NKlC/Xr\n64e6WF9Cwh569+5Jbm4ukZGLaNasmZb2FRFxAIU2EHfffTeDBw+2RRa5wxw+fIiePbtz+XIa8+Z9\nSbt2HewdSUREblGh10C0adOGLVu2kJmZSW5uruWXyO1ITDxBYOAznDt3jmnTPqZbt+fsHUlERIqg\n0BmITz/9lLS0NMtFbYZhYDKZ+O2330o8nJRN586do0ePZ0lMPMGYMWH07t3X3pFERKSICm0gdu7c\nqfvxxWrS0i4RFNSdQ4cOMnDgW7z55hB7RxIRkWIotDN45ZVXbJFD7gDp6en06RPE3r17CArqRWjo\neN2uKSLioAqdgahfvz4ff/wxjz/+OK6urpbtzZs3L9FgUvZ88MH7bNnyI126dCUiYoaaBxERB1Zo\nA3H9Woddu3ZZtplMJjUQUiQHDx7g88/n8Le/1WLOnPm4uBT6R09EREqxQn+KR0ZG2iKHlGGGYTBm\nzEiys7N5//1JVKhQwd6RRETkNhXaQLz00ksFTjUvWrSoRAJJ2bN27Ro2bdpA27bt6dzZ395xRETE\nCm7pSZTXZWVlsX37dtzc3Eo0lJQdGRkZjB07CmdnZyZMmKLrHkREyohCG4gmTZrked2yZUtef/31\nEgskZctnn83i6NE/eOONAdStW8/ecURExEoKbSCOHz+e5/WpU6f4448/SiyQlB2nTp3kww+n4u3t\nzfDho+wdR0RErKjQBuKvz4EwmUx4eHhobQy5JePHh3LlymXGj5/EXXdVsnccERGxokIbiHnz5lG7\ndu082/bu3VtigaRs+OmnHSxd+g0NGvjy0ku97R1HRESs7IZPorx48SLHjh0jJCSE48ePW379/vvv\njBw50pYZxcHk5uYyevQIACZODMfZ2dnOiURExNpuOAOxZ88evvrqK3777bc8pzGcnJxo1aqVTcKJ\nY4qKWsTevXvo3v0FmjXTA8dERMqiGzYQbdu2pW3btixevJigoCBbZhIHdvHiBSZMCMPNzY2xY8fb\nO46IiJSQQhfT6tKlC1OmTGH48OEAbNiwgZSUlGIdLCsri3fffZegoCB69eqV7w4PgAsXLtCvXz/e\neusty7bly5fTtm1bevfuTe/evfn0008B2L9/Pz179qRnz56EhoYWK5NY17RpU0hOTuLtt9+levV7\n7R1HRERKSKENxHvvvUe1atUsf9lnZmYW+xqImJgYPD09Wbx4Mf379yciIiLfmNDQUBo3bpxvu7+/\nP5GRkURGRjJgwAAAJk6cSEhICFFRUaSlpREXF1esXGIdhw4d5PPP51CzZi0GDHjT3nFERKQEFdpA\npKSk0KdPH8tKnJ07dyY9Pb1YB4uPj+fJJ58EoEWLFuzevTvfmAkTJhTYQPyvzMxMEhMTadiwIQDt\n27cnPj6+WLnk9hmGwXvvBZOdnc24cRMpX768vSOJiEgJuqUlEbOysiyPIE5OTubKlSvFOlhycjJe\nXl7AtYsxTSYTmZmZmM1myxh3d/cCP7tz50769etHdnY2I0eOxNvbG09PT8v73t7eJCUl3fT4lSu7\n4eJi/TsCfHw8rL5PR7Ny5Uo2bPiejh078sorQbf1yGrV07pUT+tTTa1L9bQuW9Wz0Abi5Zdf5oUX\nXiApKYn+/fuzb98+Ro8eXeiOo6OjiY6OzrMtISEhz2vDMG4ppK+vL15eXrRr1449e/YwcuRIPv/8\n8yLv6/z54jU+N+Pj40FS0iWr79eRXL16lTfffBsXFxfCwiaRnJxW7H2pntalelqfampdqqd1FVTP\nkmooCm0g/P39adSoEXv27MFsNvP+++9TpUqVQnccGBhIYGBgnm3BwcEkJSVRr149srKyMAwjz+zD\njdSuXdvyMKvHH3+clJQUKleuTGpqqmXMmTNnbimXWN/s2TM4duwo/fsP1noXIiJ3iJteA3HkyBHW\nrl1Lbm4uXbp0oWPHjlSpUoU1a9YU62AtW7YkNjYWgI0bN9K0adNb+ty8efOIiYkB4ODBg3h5eWE2\nm3nggQfYtWsXAOvWraN169bFyiXFd/z4MT7+OAIfnyoMHx5s7zgiImIjN5yBWLx4MV988QV16tRh\n/PjxTJkyhfr16xMWFsbp06fp0qVLkQ/m7+/Ptm3bCAoKwmw2M3nyZADmzp2Ln58fDRs2pG/fvly8\neJEzZ87Qu3dvBg4cSEBAAMOHDycqKors7GwmTpwIQEhICGPHjiU3NxdfX19atGhRzDJIcYWGjiY9\nPZ2pU6fj4eFZ+AdERKRMMBk3uHigR48eLFy4kPLly3P8+HFee+01srOzeeWVV+jdu/dtXSRnLyVx\nnu1OPn8XF7eRwMBn8PNryqpVa3FyKvSmnkLdyfUsCaqn9amm1qV6WlepuAaiXLlyllvxatSoQYUK\nFZg9ezbVq1cvkSDiWLKyshg9egQmk4lJk6ZapXkQERHHccOf+v87w+Dp6anmQSw+//wzDh48QJ8+\n/6Bhw8fsHUdERGzshjMQGRkZeR41/b+va9SoUbLJpNQ6c+Y0U6dOonLlyowaNcbecURExA5u2EAk\nJSXRt2/fPM9XuL4qp8lk4ocffij5dFIqjR8fSlraJcLDP8LLy9vecURExA5u2EBs2LDBljnEQezc\nuYMlSxbToIEvvXv3tXccERGxE135JrcsJyeHUaOGAfDBB1Nxdrb+Y8FFRMQxqIGQWxYZ+SX79iUQ\nGNiTpk2b2TuOiIjYkRoIuSUpKeeYNOl93N09GDv2fXvHEREROyu0gbhw4QJTpkxh2LBrU9cbNmwg\nJSWlxINJ6fLxxx9y/vx53n13JFWr3mPvOCIiYmeFNhBjxoyhWrVqnDhxAoDMzExGjhxZ4sGk9Dh9\n+hRffDGPe++9j9de+6e944iISClQaAORkpJCnz59cHV1BaBz586kp6eXeDApPT76aCrp6ekMHTqC\ncuXK2TuOiIiUArd0DURWVpblyZTJyclcuXKlRENJ6XHs2H/5+uuvqFXrfnr2fNnecUREpJS44XMg\nrnv55Zd54YUXSEpKon///uzbt4/Ro0fbIpuUAh9+GE5WVhbDh4+yzEKJiIgU2kD4+/vTqFEj9uzZ\ng9ls5v3336dKlSq2yCZ2duTIIb755v9Rt249uncPtHccEREpRQptIJYuXWr578uXL/Pjjz/i4uLC\n/fffj6+vb4mGE/uaOnUSOTk5jBgxWg+NEhGRPAptILZu3crWrVtp1KgRzs7O/Pzzz/j5+XH8+HHa\ntm3LkCFDbJFTbOzXX39hxYplNGjgS9eu3ewdR0RESplCG4icnBxWr17N3XffDcC5c+eYNGkSK1as\noGfPniUeUOxjypSJGIbBqFFj8i3tLiIiUuhdGGfOnLE0DwDe3t6cOHECk8lEbm5uiYYT+9i7dzdr\n1sTg59eUjh072TuOiIiUQoXOQFSvXp233nqLJk2aYDKZ2LNnDxUrViQ2NpZq1arZIqPY2KRJ4wEY\nNeo9zT6IiEiBCm0gpkyZwnfffcf+/fvJzc3F19eX7t27k5aWRtu2bW2RUWxo+/ZtbNz4A61bt6NV\nqzb2jiMiIqVUoQ2E2WwmMPDPW/gyMzMZNmwYM2bMKNFgYnuGYVhmH4KD9awPERG5sUIbiG+//ZbJ\nkydz4cIFAJycnGjWTEs5l0VxcRuJj9/Kk08+hZ9fU3vHERGRUqzQBiIyMpJVq1YxdOhQPvvsM1at\nWoWHh4ctsokNGYbB5MnXZx/G2DmNiIiUdoXeheHh4YGPjw85OTm4ubnx4osvsmzZMltkExtau3YN\nu3f/TEDAszRooAeEiYjIzRU6A+Hs7MzGjRupVq0aM2fO5MEHHyQxMdEW2cRGDMNg6tRJODk5MXKk\nrn0QEZHCFToDER4ezj333ENISAhnz55l5cqVvPfee7bIJjayadMG9u1LICDgWerUqWvvOCIi4gAK\nnYHYtGkTzz//PADjx48v8UBie5988jEAb775jp2TiIiIoyh0BmL9+vVcunTJFlnEDhIS9rB58yba\ntGlPw4aP2TuOiIg4iEJnINLT0+nQoQP3338/rq6ulu2LFi0q0WBiG9dnHwYPftvOSURExJEU2kAM\nHDjQFjnEDv7443dWrfqWBg18adu2vb3jiIiIAyn0FEaTJk24cuUKBw8epEmTJtxzzz34+fnZIpuU\nsE8/nUlubi6DB7+tNS9ERKRICp2BmDp1Kv/97385efIkvXr1YtWqVaSkpBTrToysrCyCg4M5efIk\nzs7OTJo0iRo1auQZc+HCBYYOHUrFihUtj8v+9NNP2bZtGwC5ubkkJyezdu1aOnTowD333IOzszMA\n06ZNo2rVqkXOdSdKSkoiKmoRNWvWIiDgWXvHERERB1NoA/HTTz+xZMkSevfuDcCgQYPo2bNnsQ4W\nExODp6cnERERbNmyhYiICKZPn55nTGhoKI0bN2b//v2WbQMGDGDAgAEArFixgnPnzlnemzdvHhUr\nVixWnjvZ/PlzSE9PZ8CAwbi4FPrHQEREJI9CT2GUK1cOwDLFnZOTQ05OTrEOFh8fz5NPPglAixYt\n2L17d74xEyZMoHHjxgV+Pjs7m8WLF9OrV69iHV+uSUtLY8GCeXh7exMUpFqKiEjRFfpPz0aNGhEc\nHMzZs2f54osvWLduHU2aNCnWwZKTk/Hy8gKuLcplMpnIzMzEbDZbxri7u9/w8+vWraNVq1aUL1/e\nsi00NJTExEQaN27Mu+++e9Nz+ZUru+Hi4lys7Dfj4+NYa4MsWjSf1NRUxo0bx9/+VvpO+ThaPUs7\n1dP6VFPrUj2ty1b1LLSBGDJkCLGxsVSoUIHTp0/z6quv0qlTp0J3HB0dTXR0dJ5tCQkJeV4bhlGk\nsMuWLWPcuHGW12+99RatW7fmrrvuYtCgQaxdu5bOnTvf8PPnz18p0vFuhY+PB0lJjvOcjKysLKZN\ni/i/dU36lLrsjlbP0k71tD7V1LpUT+sqqJ4l1VAU2kAMHTqUZ555hvfeew8np0LPeFgEBgYSGBiY\nZ1twcDBJSUnUq1ePrKwsDMPIM/twM1euXOH06dPcd999lm3PPvvnxX9t2rTh4MGDN20gBFasWEpi\n4glef70/Xl7e9o4jIiIOqtCOoF27dixevJgOHTowYcIE9u3bV+yDtWzZktjYWAA2btxI06ZNb/mz\n+/fv54EHHrC8vnTpEv369SMzMxO4drHnQw89VOxsdwLDMJg162OcnZ355z8H2TuOiIg4sEJnILp1\n60a3bt24dOkS69ev59NPP+XYsWPExMQU+WD+/v5s27aNoKAgzGYzkydPBmDu3Ln4+fnRsGFD+vbt\ny8WLFzlz5gy9e/dm4MCBNG/enKSkJMv1E3BtmfE2bdrw4osvUq5cOR5++GHNPhTihx/W8dtvv/L8\n8z2oWfNv9o4jIiIOzGTcwoUIhmGwc+dOYmNj2b59O4888gjTpk2zRT6rKonzbI50/u7ZZ/3Ztm0L\nGzZs5dFHG9g7ToEcqZ6OQPW0PtXUulRP6ypV10CMHTuWuLg46tevz9NPP82IESOoUKFCiYSRkvPz\nzz+xbdsW2rfvWGqbBxERcRyFNhB169blnXfeyXP64OTJk1SvXr1Eg4l1/blk9xA7JxERkbKg0Abi\n5ZdfBiAjI4O1a9eybNkyjhw5wpYtW0o8nFjHwYMHWL16FY899jgtW7a2dxwRESkDCm0g9u7dy7Jl\ny1izZg25ubm8//77PPXUU7bIJlYyYUIohmEwZMgILZolIiJWccPbOOfNm4e/vz9DhgzB29ubZcuW\nUbNmTbp27Yqrq6stM8pt2Lp1M7Gxq2nWrAWdO/vbO46IiJQRN5yBmD59Og8++CBjx46lWbNmAPrX\nq4PJzc0lLGwMAGFhE/T7JyIiVnPDBmLTpk2sWLGC0NBQcnNzee6558jKyrJlNrlNK1YsJSFhD927\nv0CjRk/YO46IiJQhNzyF4ePjwxtvvMHatWv54IMPOHbsGImJifTv35+4uDhbZpRiSE9PZ+LEcZjN\nZkJCQu0dR0REyphbWtzCz8+PyZMns3nzZtq1a8esWbNKOpfcps8//4wTJ47z2mv99dRJERGxult6\nEmVZcac8ifLcuXM0bfoYzs5O7Nixl0qVKts70i0rjfV0ZKqn9amm1qV6Wpctn0R568trisP48MMp\nXLx4gaFDRzhU8yAiIo5DDUQZ8/vvh/nii8+pVet+Xn31dXvHERGRMkoNRBkzYcI4srOzGTMmDLPZ\nbO84IiJSRqmBKEN27NhOTMx3NG7sR0DAs/aOIyIiZZgaiDLCMAzCwkYDMG7cB3polIiIlCg1EGXE\nqlXf8vPPP9G16zM0adLU3nFERKSMUwNRBmRkZDB+fCguLi6MGRNm7zgiInIHUANRBnz11Xz++9+j\n/OMfr/PAA7XtHUdERO4AaiDKgMjILylfvjxDhoywdxQREblDqIFwcAcO7OfAgf106PAk3t7e9o4j\nIiJ3CDUQDi4m5jsAAgKesXMSERG5k6iBcHCrVn2H2WymU6fO9o4iIiJ3EDUQDuzIkUP8+ut/aN++\nIx4envaOIyIidxA1EA4sJmYlAF276vSFiIjYlhoIB7Zq1Xe4urry1FNd7B1FRETuMGogHNTRo3/w\n73/vpU2bdlqyW0REbE4NhINater63RdaNEtERGxPDYSDion5FmdnZzp39rd3FBERuQOpgXBAx48f\nY8+e3bRq1QYvLz08SkREbE8NhAO6fveFTl+IiIi9qIFwQKtWfYuTkxNdunS1dxQREblDudjyYFlZ\nWQQHB3Py5EmcnZ2ZNGkSNWrUyDNm9erVLFiwACcnJ5o3b86QIUNu+Ln9+/cTFhYGQN26dRk3bpwt\nv45dnDyZyK5dO2nVqg0+Pj72jiMiIncom85AxMTE4OnpyeLFi+nfvz8RERF53r969SrTpk3jyy+/\n5JtvvmHbtm0cPnz4hp+bOHEiISEhREVFkZaWRlxcnC2/jl386196eJSIiNifTRuI+Ph4nnzySQBa\ntGjB7t2787xfoUIFVq5cibu7OyaTiUqVKpGamlrg5zIzM0lMTKRhw4YAtG/fnvj4eFt+HbtYteo7\nTCYTTz/dzd5RRETkDmbTUxjJycl4eXkB4OTkhMlkIjMzE7PZbBnj7u4OwIEDB0hMTMTX15fZs2fn\n+1xycjKenn+u/+Dt7U1SUtJNj1+5shsuLs7W/lr4+HhYfZ8FOXXqFDt2xNO6dWseffRBmxzTHmxV\nzzuF6ml9qql1qZ7WZat6llgDER0dTXR0dJ5tCQkJeV4bhlHgZ48ePcqwYcOIiIjA1dU13/sFfe5G\n+/qr8+evFDqmqHx8PEhKumT1/RZk4cLFGIZB585dbXZMW7NlPe8Eqqf1qabWpXpaV0H1LKmGosQa\niMDAQAIDA/NsCw4OJikpiXr16pGVlYVhGHlmHwBOnz7NoEGDCA8Pp379+gBUqVIl3+d8fHxITU21\nfO7MmTNUqVKlpL5OqRATc+3pkzp9ISIi9mbTayBatmxJbGwsABs3bqRp06b5xowePZqwsDAeeeSR\nm37O1dWVBx54gF27dgGwbt06WrdubYNvYR9JSUls27YFP7+mVKtW3d5xRETkDmfTayD8/f3Ztm0b\nQUFBmM1mJk+eDMDcuXPx8/OjUqVK7Nq1ixkzZlg+07dv3xt+LiQkhLFjx5Kbm4uvry8tWrSw5dex\nqTVrYsjNzSUgQHdfiIiI/ZmMW7l4oIwoifNstjp/Fxj4DHFxG9m9+xfuu69G4R9wUDofal2qp/Wp\nptalelqXLa+B0JMoHUBKyjm2bPmRRo0al+nmQUREHIcaCAcQG7uanJwcunbV2hciIlI6qIFwACtX\nrgCga1fdfSEiIqWDGohS7vz5FH78cRMNGz5GrVr32zuOiIgIoAai1PvuuxVkZ2fzzDPd7R1FRETE\nQg1EKRcdHYXJZOKFF3rYO4qIiIiFGohS7Pffj/DTTzto06adHh4lIiKlihqIUiw6OgqAwMCedk4i\nIiKSlxqIUsowDKKjv8HNrSL+/gH2jiMiIpKHGohSaseO7Rw7dpSuXbtZljgXEREpLdRAlFI6fSEi\nIqWZGohSKD09ne++W061atVp1aqNveOIiIjkowaiFFq3bg0XL17g+ed74OzsbO84IiIi+aiBKIV0\n+kJEREo7NRClTHJyMj/8sJ4GDXypX/9he8cREREpkBqIUubbb5eSnZ1Njx6afRARkdJLDUQps2TJ\nYpydnXnuuUB7RxEREbkhNRClyMGDB9i7dw/t23ekSpUq9o4jIiJyQ2ogSpHrF0/26BFk5yQiIiI3\npwailMjNzWXp0m/w8PDkqaf87R1HRETkptRAlBLbtm0hMfEE3bo9S4UKFewdR0RE5KbUQJQSS5Ys\nBvTsBxERcQxqIEqBK1eusGrVd9SoUZNmzVrYO46IiEih1ECUAmvWxHD5chqBgS/i5KTfEhERKf30\nt1UpoNMXIiLiaNRA2NmZM6eJi9tI48ZPULv2Q/aOIyIickvUQNjZ8uVLyc3N5YUXNPsgIiKOQw2E\nna1fHwtAQMCzdk4iIiJy69RA2FFaWho7dsTj6/u4Hl0tIiIORQ2EHW3dupmsrCzat+9o7ygiIiJF\nogbCjjZu/B6ADh3+buckIiIiReNiy4NlZWURHBzMyZMncXZ2ZtKkSdSoUSPPmNWrV7NgwQKcnJxo\n3rw5Q4YMITs7m9GjR3Ps2DFycnIYMWIETzzxBL179+bKlSu4ubkBMHLkSB599FFbfqXbsmHD97i7\ne9C4sZ+9o4iIiBSJTRuImJgYPD09iYiIYMuWLURERDB9+nTL+1evXmXatGmsXLmSihUr0qNHDwIC\nAkhISKBChQosXryYQ4cOMWrUKJYuXQrApEmTqFOnji2/hlX8/vsRjh79A3//AFxdXe0dR0REpEhs\negojPj6eJ598EoAWLVqwe/fuPO9XqFCBlStX4u7ujslkolKlSqSmptKtWzdGjRoFgJeXF6mpqbaM\nXSI2bvwBQNc/iIiIQ7LpDERycjJeXl4AODk5YTKZyMzMxGw2W8a4u7sDcODAARITE/H19c3zL/Sv\nvvqKrl27Wl7PmDGD8+fPU7t2bUJCQihfvryNvs3tuX79gxoIERFxRCXWQERHRxMdHZ1nW0JCQp7X\nhmEU+NmjR48ybNgwIiIi8jQPixYt4pdffmHOnDkA9OnTh7p161KzZk1CQ0NZtGgR/fr1u2GmypXd\ncHFxLu5XuiEfH48ijc/IyGDr1s3Uq1ePxo0d55oNWylqPeXmVE/rU02tS/W0LlvVs8QaiMDAQAID\nA/NsCw4OJikpiXr16pGVlYVhGHlmHwBOnz7NoEGDCA8Pp379+pbt0dHRbNiwgdmzZ1uaiuunQwA6\ndOjA6tWrb5rp/Pkrt/u18vHx8SAp6VKRPrN5cxyXL1+mTZv2Rf5sWVecesqNqZ7Wp5pal+ppXQXV\ns6QaCpteA9GyZUtiY689eXHjxo00bdo035jRo0cTFhbGI488Ytl2/PhxoqKi+OSTTyhXrhxwbfai\nb9++XLx4EYAdO3bw0EOOsZbE9esfdPumiIg4KpteA+Hv78+2bdsICgrCbDYzefJkAObOnYufnx+V\nKlVi165dzJgxw/KZvn37kpCQQGpqKm+88YZl+/z58+nRowd9+/alQoUKVK1alTfffNOWX6fYNmz4\nnnLlytGsWUt7RxERESkWk3GjCxHKoJKYJivq9Nvp06do2LAu7dp1YMmSb62ex9FpOtO6VE/rU02t\nS/W0rjJ7CkNg06YNALRvr9MXIiLiuNRA2NiGDesBXf8gIiKOTQ2EDeXk5BAXt5F7772POnXq2juO\niIhIsamBsKG9e3dz/vx52rfviMlksnccERGRYlMDYUN/Pr5apy9ERMSxqYGwoQ0bvsfZ2Zk2bdra\nO4qIiMhtUQNhYGU9ngAAEnhJREFUI6mp59m9exeNG/tx112V7B1HRETktqiBsJEff9xEbm6u7r4Q\nEZEyQQ2EjWj5bhERKUvUQNiAYRhs2PA93t7e+Po+bu84IiIit00NhA3s3/8bp06dpG3b9jg5qeQi\nIuL49LeZDej2TRERKWvUQNjAhg3fA9Cuna5/EBGRskENRAm7fPky27dv5dFHG1K1alV7xxEREbEK\nNRAlLD5+C5mZmbp9U0REyhQ1ECVMt2+KiEhZpAaiBJ08mciaNf+iYkV3/Pya2juOiIiI1aiBKAG5\nubksWDCPVq2acOLEcV58MQiz2WzvWCIiIlbjYu8AZc2BA/t599232LlzO3fdVYmPPvqEl17qbe9Y\nIiIiVqUGwkoyMjKYMeNDPv44gszMTLp1e46JE8N154WIiJRJaiCsYOfOHbz77pscOLCfatWqM3ly\nBF26PG3vWCIiIiVG10DchoyMDAYPHkxAQCcOHNjPq6++xpYtO9U8iIhImacZiNswf/5cZs2aRZ06\ndYmImEnTps3sHUlERMQm1EDchpiY73BycuK772Lx9va2dxwRERGbUQNRTGfPnuXnn38i0WymSqe2\n+d6/MvAt0vu9AYDHwNdx3RGfb0xW4ye4NPdLAMpHfonb9GkFHislfjeYzTgfOshdPbsXOObShzPJ\natsegEpPtcMpOTnfmPQeQVwZORqAiqGjKRfzXb4xOTX/xoUV/wLAvOZfuI8ZWeDxUletJbf6vZhS\nz1O5Y+sCx1wOGUvG8z0A8Hw5EJf9v+Ubk9n+76RNmw5AhZnTYeF8vHKNPGMMNzfOb94JgMuunXj+\n8x8FHu/igkiy/2+59MpNH8OUnZ1vzNU3BnD1n4MAcH9nEObNcfnGZDfw5eKXiwAoF7WIilMnFXi8\nlLjt4O6O09E/qPR8QIFj0sI/JLNjJwAqde2E06mT+cZkPPcCl8eEAVBxQhjlVizNNya3WnVSY9YB\nYP5hHe4jhhZ4vNRlq8itdT+kpeHVthk4mfLV8/LwUWT0fBkAz74v47IvId9+Mlu3JW36LAAqfDaL\nCnM/zTfGcHHh/I69ALgk7MHzHwXfbXTxswVkP9EEgMqtm2C6ciXfmKt9X+Pqm+8A4D7sHcwbv883\nJrtefS4uigag3LIlVPzg/QKPd/6HzRiVKuN0MpFKAU8VOCZtwhQy/+9U413PPY3zsf/mG5PR9Rku\nj5sIgNuUiZRfsvjaG3+pae7dd5O6dhMArnEb8Rj6ZoHHuxC1nJyH6kBmJl7NGxU45so7w0jv3RcA\njzf64vrzrnxjspo259LseQCUnz8Xt9kzCtxXys//AcD511+4q/eLBY659MlnZDVvCUDl9i0xXbyQ\nb0z6y324MnQEABVDhlNu7Zp8Y3JqP8iFJd8CYF71Le5hYwo83vk1GzCqVMF09iyVu3T4842/1DMt\nbAKZAc8CcFePZ3E+cjjffjKe6sLlD6YC4PZhOOUXLcw3xvC8i/MbtwLgGr8Vj8H/LDDThchvyHn4\nEQC8Gj9a4Bhb/yy//nvnCNRAFNPVq1coV64cHhUr2juKiIiIzZkMwzAKH1Y2JCVdsur+srOzqVat\nstX3eyfz8fFQPa1I9bQ+1dS6VE/rKqiePj4eJXIs3YVxG1xcNIEjIiJ3JjUQIiIiUmRqIERERKTI\n1ECIiIhIkdn0JH5WVhbBwcGcPHkSZ2dnJk2aRI0aNfKMWb16NQsWLMDJyYnmzZszZMgQli9fzscf\nf0zNmjUBaNGiBQMGDGD//v2EhYUBULduXcaNG2fLryMiInLHsukMRExMDJ6enixevJj+/fsTERGR\n5/2rV68ybdo0vvzyS7755hu2bdvG4cPX7gP29/cnMjKSyMhIBgwYAMDEiRMJCQkhKiqKtLQ04uLy\n39MvIiIi1mfTBiI+Pp4nn3wSuDaLsHv37jzvV6hQgZUrV+Lu7o7JZKJSpUqkpqYWuK/MzEwSExNp\n2LAhAO3btyc+Pv8DPkRERMT6bNpAJCcn4+Xlde3ATk6YTCYyMzPzjHF3dwfgwIEDJCYm4uvrC8DO\nnTvp168fr7zyCr/++ivnz5/H09PT8jlvb2+SkpJs9E1ERETubCV2DUR0dDTR0dF5tiUk5H1k7o2e\nYXX06FGGDRtGREQErq6u+Pr64uXlRbt27dizZw8jR47k888/v6V9/VXlym64uDgX8ZsUrqQe0nGn\nUj2tS/W0PtXUulRP67JVPUusgQgMDCQwMDDPtuDgYJKSkqhXrx5ZWVkYhoHZbM4z5vTp0wwaNIjw\n8HDq168PQO3atalduzYAjz/+OCkpKVSuXDnP6Y0zZ85QpUqVm2Y6fz7/M/hvl56iZl2qp3Wpntan\nmlqX6mldZfZJlC1btiQ2NhaAjRs30rRp03xjRo8eTVhYGI888ohl27x584iJiQHg4MGDeHl5YTab\neeCBB9i169qCM+vWraN164IXdRIRERHrsultnP7+/mzbto2goCDMZjOTJ08GYO7cufj5+VGpUiV2\n7drFjBl/rjDXt29fAgICGD58OFFRUWRnZzNx4rUV8kJCQhg7diy5ubn4+vrSokULW34dERGRO9Yd\ntZiWiIiIWIeeRCkiIiJFpgZCREREikwNhIiIiBSZGggREREpMjUQIiIiUmRqIERERKTIbPociLLk\ngw8+ICEhAZPJREhIiGVRrzvVwYMHGThwIH379qVXr16cOnWKESNGkJOTg4+PD1OnTsVsNrNy5Uq+\n+uornJyc6NGjB4GBgTdc5v1Gy7V//vnnxMbGYjKZGDx4MG3btuXSpUu8++67XLp0CTc3NyIiIqhU\nqZIdK3J7wsPD+fnnn8nOzuaf//wnDRo0UD2L6erVqwQHB3Pu3DkyMjIYOHAg9erVUz2tID09na5d\nuzJw4ECaN2+umhbTjh07ePvtt3nooYcAqFOnDq+99lrpr6chRbZjxw7jjTfeMAzDMA4fPmz06NHD\nzons6/Lly0avXr2MMWPGGJGRkYZhGEZwcLCxevVqwzAMIyIiwli0aJFx+fJlo1OnTsbFixeNq1ev\nGk8//bRx/vx5Y/ny5UZYWJhhGIaxefNm4+233zYMwzB69eplJCQkGIZhGEOHDjU2bdpkHDt2zHju\nueeMjIwM49y5c8ZTTz1lZGdnGzNnzjTmzZtnGIZhREVFGeHh4bYug9XEx8cbr732mmEYhpGSkmK0\nbdtW9bwN//rXv4y5c+cahmEYJ06cMDp16qR6WsmHH35odO/e3Vi2bJlqehu2b99uvPnmm3m2OUI9\ndQqjGOLj4/n73/8OXFun48KFC6Slpdk5lf2YzWbmzZuXZy2SHTt20LFjR+DPpdYTEhJo0KABHh4e\nlC9fnkaNGrF79+4Cl3m/0XLtO3bsoHXr1pjNZry8vLj33ns5fPhwnn04+tLufn5+fPzxxwB4enpy\n9epV1fM2+Pv78/rrrwNw6tQpqlatqnpawZEjRzh8+DDt2rUD9P+8tTlCPdVAFENycjKVK1e2vPby\n8rqjlxJ3cXGhfPnyebZdvXrVslDa9aXW/7qcO/xZt4KWeU9OTi5wufZb2Ye3tzdnz54tse9b0pyd\nnXFzcwNg6dKltGnTRvW0gp49ezJs2DBCQkJUTyuYMmUKwcHBlteq6e05fPgw/fv3JygoiK1btzpE\nPXUNhBUYehr4Td2oPkXZbo2xjub7779n6dKlLFiwgE6dOlm2q57FExUVxW+//cbw4cPzfCfVs+i+\n/fZbHnvsMWrUqFHg+6pp0dSqVYvBgwfTpUsXjh8/Tp8+fcjJybG8X1rrqRmIYqhSpQrJycmW12fP\nnsXHx8eOiUofNzc30tPTgT+XWi+obte3X5/Bub7Mu4+PT4HLtf/vPv66/fo+bmVp99Ju8+bNzJkz\nh3nz5uHh4aF63ob//Oc/nDp1CoD69euTk5NDxYoVVc/bsGnTJn744Qd69OhBdHQ0s2fP1p/R21C1\nalX8/f0xmUzUrFmTu+++mwsXLpT6eqqBKIaWLVuydu1aAH755ReqVKmCu7u7nVOVLi1atLDU6PpS\n676+vuzbt4+LFy9y+fJldu/ezRNPPFHgMu+urq4FLtferFkzNm3aRGZmJmfOnOHs2bM8+OCDefbh\n6Eu7X7p0ifDwcD777DPLVdCqZ/Ht2rWLBQsWANdOP165ckX1vE3Tp09n2bJlLFmyhMDAQAYOHKia\n3oaVK1cyf/58AJKSkjh37hzdu3cv9fXUapzFNG3aNHbt2oXJZCI0NJR69erZO5Ld/Oc//2HKlCkk\nJibi4uJC1apVmTZtGsHBwWRkZFC9enUmTZqEq6srsbGxzJ8/H5PJRK9evejWrRs5OTmMGTOGo0eP\nWpZ5r1atGocPH86zXPuoUaMAiIyMZNWqVZhMJt555x2aN2/O5cuXGT58OKmpqXh6ejJ16lQ8PDzs\nXJni+eabb5g5cyb333+/ZdvkyZMZM2aM6lkM6enpjB49mlOnTpGens7gwYN59NFHGTlypOppBTNn\nzuTee++lVatWqmkxpaWlMWzYMC5evEhWVhaDBw+mfv36pb6eaiBERESkyHQKQ0RERIpMDYSIiIgU\nmRoIERERKTI1ECIiIlJkaiBERESkyNRAiDiIEydOULduXVauXJlne4cOHayy/7p165KdnW2Vfd3I\n2rVr6dixI9HR0Xm2z5w5k48++gi49kjfX375xSrHO3PmjOWZ/suXL893XBEpPjUQIg6kVq1azJo1\ny2EXb4uLi6Nfv34EBgbecMz69ev59ddfrXK8HTt2sH37dgC6d+9+0+OKSNFoLQwRB1KlShVatWrF\n7NmzGTFiRJ73li9fzrZt25g2bRoAvXv3ZsCAATg7OzNnzhzuuece9u3bh6+vL3Xr1mX9+vWkpqYy\nb9487rnnHgDmzJnD9u3buXz5MlOmTKFOnTrs37+fKVOmkJ2dTVZWFmPHjuXhhx+md+/e1KtXj99+\n+42vvvoKZ2dnS5ZNmzYxa9YsypcvT4UKFRg/fjx79uwhLi6On3/+GWdnZ1588cV832/Pnj18/fXX\nuLu7U758edq0aUNoaCgpKSmkpaXx6quvEhAQwMyZMzlx4gQnT55k5MiRpKenM23aNMxmM+np6YSG\nhuLp6cn06dMxDINKlSqRlpZGdnY2Q4YMKTBf1apV6dChA3369OHHH3/kxIkTjBs3jubNm5fg76iI\n49IMhIiDefXVV4mLi+P333+/5c/8+9//ZuTIkSxbtoxVq1bh6elJZGQkjzzyiOXxtXBtefqvv/6a\nl156iU8++QSA4cOHM27cOCIjIwkLC2PMmDGW8W5ubnz99dd5moerV68yZswYZs6cSWRkJG3atGH6\n9Ol07tyZ1q1b89prrxXYPAA8/vjjljEBAQFMnz6d1q1bs3DhQr7++mtmzJhBSkoKcO2UzsKFC3n0\n0UdJTU0lLCyMhQsX0qdPHz777DNq1KjBc889R7du3Xj11VcLzXdduXLlWLBgAQMGDGDhwoW3XGOR\nO41mIEQcjNlsZsSIEUycONHy/PzC1K5d27KuRqVKlXj88ceBa4v4/PV0SMuWLQFo1KgRCxYs4Ny5\nc/zxxx+MHj3aMiYtLY3c3FzLuP919OhRvL29LbMaTZo0ISoqqhjf9NopiH379vHtt98C15aOP3Hi\nBAC+vr6YTCYA7r77bsLDw8nIyODSpUvcddddN9xnYfmaNGkCQPXq1blw4UKxcovcCdRAiDigtm3b\nsnjxYtavX2/Zdv0v0+uysrIs//3XGYL/ff3Xp9k7OTlZtplMJsxmM66urkRGRhaYw9XVNd+2/81x\nfV/FYTabCQ0NpUGDBnm2x8XF5Tn2iBEjLKcbNm7caFk8qyCF5XNxccnznogUTKcwRBxUSEgIERER\nZGZmAuDu7s7p06cBOHfuHIcOHSryPq/fsbB7927q1KmDh4cH9913H3FxcQD88ccfllMbN1KrVi3O\nnTvHyZMnLfv09fW95Qwmk8nS/DRu3Jg1a9YA1xbFCgsLK/BOkeTkZB566CFycnKIjY211MRkMuUb\nf7v5ROQazUCIOKiaNWvy1FNPMWfOHODa6Yf58+fTo0cPateubTlNcaucnZ05dOgQUVFRnD9/nqlT\npwIwZcoUJkyYwNy5c8nOziY4OPim+ylfvjwTJ05kyJAhmM1m3NzcmDhx4i3naNasGeHh4RiGweDB\ngxkzZgxBQUFkZmby4osv5pkhuO7111/nlVdeoXr16vTr148RI0bw5Zdf8sQTTzBkyBBcXV0tsy63\nm09ErtFqnCIiIlJkOoUhIiIiRaYGQkRERIpMDYSIiIgUmRoIERERKTI1ECIiIlJkaiBERESkyNRA\niIiISJGpgRAREZEi+//BZGpqUe1ZTgAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<matplotlib.figure.Figure at 0x7f41274fd358>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "HQ-Lt3-8Luq9",
        "colab_type": "code",
        "colab": {
          "autoexec": {
            "startup": false,
            "wait_interval": 0
          },
          "base_uri": "https://localhost:8080/",
          "height": 119
        },
        "outputId": "fe89d68d-2fc7-4f1c-855c-1e09110414c5",
        "executionInfo": {
          "status": "ok",
          "timestamp": 1524414837483,
          "user_tz": 240,
          "elapsed": 382,
          "user": {
            "displayName": "Reza Davari",
            "photoUrl": "//lh5.googleusercontent.com/-QVkbnQn2wK4/AAAAAAAAAAI/AAAAAAAAAFA/kgVC69JIYw4/s50-c-k-no/photo.jpg",
            "userId": "106412143772601057521"
          }
        }
      },
      "cell_type": "code",
      "source": [
        "print('W:\\n',w,'\\n')\n",
        "print('Theta:\\n',theta)"
      ],
      "execution_count": 110,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "W:\n",
            " [ 0.44449872 -0.06789564 -0.28582969 -0.01802017] \n",
            "\n",
            "Theta:\n",
            " [-5.60971188e-18  3.23035180e-19  1.44906461e-19 -9.42286369e-17\n",
            " -1.62954016e+00 -1.25595806e+00  1.22414383e+00  1.66135438e+00]\n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}