Actor-Critic算法.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "49C35CCB207841D0BED69F6A9B026293",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# Actor-Critic算法\n",
    "## 简介\n",
    "在之前的内容中，我们学习了基于值函数的方法（DQN）和基于策略的方法（REINFORCE），其中基于值函数的方法只学习一个价值函数，而基于策略的方法只学习一个策略函数。那么一个很自然的问题，有没有什么方法既学习价值函数，又学习策略函数呢？答案就是Actor-Critic。Actor-Critic是一系列算法，目前前沿的很多高效算法都属于Actor-Critic算法，今天我们将会介绍一种最简单的Actor-Critic算法。需要明确的是，Actor-Critic算法本质上是基于策略的算法，因为这系列算法都是去优化一个带参数的策略，只是其中会额外学习价值函数来帮助策略函数的学习。\n",
    "\n",
    "## Actor-Critic算法\n",
    "我们回顾一下在REINFORCE算法中，目标函数的梯度中有一项轨迹回报，来指导策略的更新。而值函数的概念正是基于期望回报，我们能不能考虑拟合一个值函数来指导策略进行学习呢？这正是Actor-Critic算法所做的。让我们先回顾一下策略梯度的形式，在策略梯度中，我们可以把梯度写成下面这个形式：\n",
    "$$\n",
    "g = \\mathbb{E}[\\sum^{\\infty}_{t=0}\\psi_{t}\\nabla_{\\theta}\\log \\pi_{\\theta}(a_{t}|s_{t})]\n",
    "$$\n",
    "其中$\\psi_{t}$可以有很多种形式：\n",
    "$$\n",
    "\\begin{align}\n",
    "&1.\\sum_{t‘=0}^{\\infty} \\gamma^t r_{t’} : 轨迹的总回报  &&4.Q^{\\pi_\\theta}(s_{t},a_{t}) : 动作价值函数\\\\\n",
    "&2.\\sum_{t'=t}^{\\infty}  \\gamma^{t'-t} r_{t'} : 动作 a_{t}之后的回报 &&5.A^{\\pi_\\theta}(s_{t},a_{t}): 优势函数\\\\\n",
    "&3.\\sum_{t'=t}^{\\infty}r_{t'}-b(s_{t}) : 基准线版本的改进 \\quad\\quad \\quad\\quad &&6.r_{t} + \\gamma V^{\\pi_\\theta}(s_{t+1}) - V^{\\pi_\\theta}(s_t) : 时序差分残差\n",
    "\\end{align}\n",
    "$$\n",
    "在REINFORCE的最后部分，我们提到了REINFORCE通过蒙特卡洛采样的方法对梯度的估计是无偏的，但是方差非常大，我们可以用第三种形式引入基线（baseline）$b(s_t)$来减小方差。此外我们也可以采用Actor-Critic算法，估计一个动作价值函数$Q$来代替蒙特卡洛采样得到的回报，这便是第4种形式。这个时候，我们也可以把状态价值函数$V$作为基线，从$Q$函数减去一个$V$函数则得到了$A$函数，我们称之为优势函数（advantage function）。这就是第五种形式。进一步的，我们可以利用$Q=r+\\gamma V$等式得到第6种形式。\n",
    "今天我们将着重介绍的便是第六种形式：$\\psi_{t}=r_{t} + V^{\\pi}(s_{t+1}) - V^{\\pi}(s_t)$ 来指导策略梯度进行学习。事实上，用$Q$值或者$V$值本质上也是用奖励来进行指导，但是用神经网络进行估计的方法可以减小方差、提高鲁棒性。除此之外，REINFORCE算法基于蒙特卡洛采样，只能在序列结束后进行更新，而Actor-Critic的方法则可以在每一步之后都进行更新。\n",
    "\n",
    "我们将Actor-Critic分为两个部分：分别是Actor（策略网络）和Critic（价值网络）：\n",
    "- Critic要做的是通过Actor与环境交互收集的数据学习一个价值函数，这个价值函数会用于帮助Actor进行更新策略。\n",
    "- Actor要做的则是与环境交互，并利用Ctitic价值函数来用策略梯度学习一个更好的策略。\n",
    "\n",
    "Actor的更新我们采用策略梯度的原则，那Critic如何更新呢。我们将Critic价值网络表示为$V_\\omega$，参数为$\\omega$。于是，我们可以采取时序差分的学习方式，对于单个数据定义如下价值函数的损失函数：\n",
    "$$\n",
    "\\mathcal{L}(\\omega) = \\frac{1}{2} (r + \\gamma V_\\omega(s_{t+1}) - V_\\omega(s_t))^2\n",
    "$$\t\n",
    "与DQN中一样，我们采取类似于目标网络的方法，上式中$r + \\gamma V_\\omega(s_{t+1})$作为时序差分目标，不会产生梯度来更新价值函数。所以价值函数的梯度为\n",
    "$$\n",
    "\\nabla_\\omega \\mathcal{L}(\\omega) = -  (r + \\gamma V_\\omega(s_{t+1}) - V_\\omega(s_t)) \\nabla_\\omega V_\\omega(s_t)\n",
    "$$\n",
    "\n",
    "然后使用梯度下降方法即可。接下来让我们总体看看Actor-Critic算法的流程吧！\n",
    "\n",
    "- 初始化策略网络参数$\\theta$，价值网络参数$\\omega$ \n",
    "- 不断进行如下循环（每个循环是一条序列）：\n",
    "\t- 用当前策略$\\pi_\\theta$采样轨迹$\\{s_{1},a_{1},r_{1},s_{2},a_{2},r_{2} ... \\}$\n",
    "\t- 为每一步数据计算: $\\delta_{t} = r_{t} + \\gamma V_{\\omega}(s_{t+1}) - V_{\\omega}(s)$\n",
    "\t- 更新价值参数$w = w + \\alpha_{\\omega} \\sum_t \\delta_{t} \\nabla_{\\omega}V_{\\omega}(s)$\n",
    "\t- 更新策略参数$\\theta = \\theta + \\alpha_{\\theta} \\sum_t \\delta_t \\nabla_{\\theta} \\log \\pi_{\\theta}(a|s)$\n",
    "\n",
    "好了！这就是Actor-Critic算法的流程啦，让我们来用代码实现它看看效果如何吧！\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "B6524090185C41FE9746A0F50C49A708",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "\n",
    "## Actor-Critic代码实践\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6F8740D490A64DB68C4A245DE981F1F1",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "我们仍然在Cartpole环境上进行Actor-Critic算法的实验。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "A170A48DD67340E6BAE74281AA4E25C2",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "import gym\n",
    "import torch\n",
    "import torch.nn.functional as F\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import rl_utils"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ECC7C7256E244F6980792CEB89A61272",
    "jupyter": {},
    "mdEditEnable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "定义我们的策略网络PolicyNet，与REINFORCE算法中一样。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "D98CF87765F84E2F9AC8370BFDABE328",
    "jupyter": {},
    "notebookId": "5f8f327c46ba5e00307827a6",
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class PolicyNet(torch.nn.Module):\n",
    "    def __init__(self, state_dim, hidden_dim, action_dim):\n",
    "        super(PolicyNet, self).__init__()\n",
    "        self.fc1 = torch.nn.Linear(state_dim, hidden_dim)\n",
    "        self.fc2 = torch.nn.Linear(hidden_dim, action_dim)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = F.relu(self.fc1(x))\n",
    "        return  F.softmax(self.fc2(x),dim=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "F436AB9FA91E41EC8BC52C3C29B42DF4",
    "jupyter": {},
    "mdEditEnable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "Actor-Critic算法中额外引入一个价值网络，接下来的代码定义我们的价值网络ValueNet，输入是状态，输出状态的价值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "B8C6D0141CC747D7854DCFBF1CB50F90",
    "jupyter": {},
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class ValueNet(torch.nn.Module):\n",
    "    def __init__(self, state_dim, hidden_dim):\n",
    "        super(ValueNet, self).__init__()\n",
    "        self.fc1 = torch.nn.Linear(state_dim, hidden_dim)\n",
    "        self.fc2 = torch.nn.Linear(hidden_dim, 1)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = F.relu(self.fc1(x))\n",
    "        return self.fc2(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4D0257A23F5E4F97AA6EBAD7FFAA80A5",
    "jupyter": {},
    "mdEditEnable": true,
    "notebookId": "5f8f327c46ba5e00307827a6",
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "再定义我们的ActorCritic算法。主要包含采取动作和更新网络参数两个函数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "954BDF09E2964F2D87BF449429BE9CF9",
    "jupyter": {},
    "notebookId": "5f8f327c46ba5e00307827a6",
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "class ActorCritic:\n",
    "    def __init__(self, state_dim, hidden_dim, action_dim, actor_lr, critic_lr, gamma, device):\n",
    "        self.actor = PolicyNet(state_dim, hidden_dim, action_dim).to(device)\n",
    "        self.critic = ValueNet(state_dim, hidden_dim).to(device) # 价值网络\n",
    "        self.actor_optimizer = torch.optim.Adam(self.actor.parameters(), lr=actor_lr)\n",
    "        self.critic_optimizer = torch.optim.Adam(self.critic.parameters(), lr=critic_lr) # 价值网络优化器\n",
    "        self.gamma = gamma\n",
    "    \n",
    "    def take_action(self, state):\n",
    "        state = torch.tensor([state], dtype=torch.float)\n",
    "        probs = self.actor(state)\n",
    "        action_dist = torch.distributions.Categorical(probs)\n",
    "        action = action_dist.sample()\n",
    "        return action.item()\n",
    "    \n",
    "    def update(self, transition_dict):\n",
    "        states = torch.tensor(transition_dict['states'], dtype=torch.float)\n",
    "        actions = torch.tensor(transition_dict['actions']).view(-1, 1)\n",
    "        rewards = torch.tensor(transition_dict['rewards'], dtype=torch.float).view(-1, 1)\n",
    "        next_states = torch.tensor(transition_dict['next_states'], dtype=torch.float)\n",
    "        dones = torch.tensor(transition_dict['dones'], dtype=torch.float).view(-1, 1)\n",
    "\n",
    "        td_target = rewards + self.gamma * self.critic(next_states) * (1 - dones) # 时序差分目标\n",
    "        td_delta = td_target - self.critic(states) # 时序差分误差\n",
    "        log_probs = torch.log(self.actor(states).gather(1, actions))\n",
    "        actor_loss = torch.mean(-log_probs * td_delta.detach())\n",
    "        critic_loss = torch.mean(F.mse_loss(self.critic(states), td_target.detach())) # 均方误差损失函数\n",
    "        self.actor_optimizer.zero_grad()\n",
    "        self.critic_optimizer.zero_grad()\n",
    "        actor_loss.backward() # 计算策略网络的梯度\n",
    "        critic_loss.backward() # 计算价值网络的梯度\n",
    "        self.actor_optimizer.step() # 更新策略网络参数\n",
    "        self.critic_optimizer.step() # 更新价值网络参数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "142F4BCD606F439C8488EB6A3AA0B1E6",
    "jupyter": {},
    "mdEditEnable": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "定义好Actor和Critic，我们就可以开始实验了，看看Actor-Critic在Cartpole环境上表现如何吧！"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "BEC67C8575254DA9815D72796FE69270",
    "jupyter": {},
    "notebookId": "5f8f327c46ba5e00307827a6",
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Iteration 0: 100%|██████████| 100/100 [00:00<00:00, 218.65it/s, episode=100, return=21.100]\n",
      "Iteration 1: 100%|██████████| 100/100 [00:01<00:00, 95.81it/s, episode=200, return=72.800]\n",
      "Iteration 2: 100%|██████████| 100/100 [00:02<00:00, 45.96it/s, episode=300, return=109.300]\n",
      "Iteration 3: 100%|██████████| 100/100 [00:05<00:00, 12.55it/s, episode=400, return=163.000]\n",
      "Iteration 4: 100%|██████████| 100/100 [00:08<00:00, 11.24it/s, episode=500, return=193.600]\n",
      "Iteration 5: 100%|██████████| 100/100 [00:08<00:00, 11.11it/s, episode=600, return=195.900]\n",
      "Iteration 6: 100%|██████████| 100/100 [00:08<00:00, 11.88it/s, episode=700, return=199.100]\n",
      "Iteration 7: 100%|██████████| 100/100 [00:08<00:00, 11.77it/s, episode=800, return=186.900]\n",
      "Iteration 8: 100%|██████████| 100/100 [00:08<00:00, 11.23it/s, episode=900, return=200.000]\n",
      "Iteration 9: 100%|██████████| 100/100 [00:08<00:00, 11.22it/s, episode=1000, return=200.000]\n"
     ]
    }
   ],
   "source": [
    "actor_lr = 1e-3\n",
    "critic_lr = 1e-2\n",
    "num_episodes = 1000\n",
    "hidden_dim = 128\n",
    "gamma = 0.98\n",
    "device = torch.device(\"cuda\") if torch.cuda.is_available() else torch.device(\"cpu\")\n",
    "\n",
    "env_name = 'CartPole-v0'\n",
    "env = gym.make(env_name)\n",
    "env.seed(0)\n",
    "torch.manual_seed(0)\n",
    "state_dim = env.observation_space.shape[0]\n",
    "action_dim = env.action_space.n\n",
    "agent = ActorCritic(state_dim, hidden_dim, action_dim, actor_lr, critic_lr, gamma, device)\n",
    "\n",
    "return_list = rl_utils.train_on_policy_agent(env, agent, num_episodes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "20D3926A2B0B4EFE819068138C0F9329",
    "jupyter": {},
    "mdEditEnable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "在CartPole-v0环境中，满分就是200分，让我们来看看每个序列得分如何吧！"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "768090415A3B4FD788C1A02BB7861332",
    "jupyter": {},
    "notebookId": "5f8f327c46ba5e00307827a6",
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://cdn.kesci.com/upload/rt/768090415A3B4FD788C1A02BB7861332/qwal1ik1oi.png\">"
      ],
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<img src=\"https://cdn.kesci.com/upload/rt/768090415A3B4FD788C1A02BB7861332/qwal1jjhna.png\">"
      ],
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "episodes_list = list(range(len(return_list)))\n",
    "plt.plot(episodes_list,return_list)\n",
    "plt.xlabel('Episodes')   \n",
    "plt.ylabel('Returns')  \n",
    "plt.title('Actor-Critic on {}'.format(env_name))  \n",
    "plt.show()\n",
    "\n",
    "mv_return = rl_utils.moving_average(return_list, 9)\n",
    "plt.plot(episodes_list, mv_return)\n",
    "plt.xlabel('Episodes')   \n",
    "plt.ylabel('Returns')  \n",
    "plt.title('Actor-Critic on {}'.format(env_name))  \n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "214D1941DDD64823BC9F0ADAE6A5D15A",
    "jupyter": {},
    "mdEditEnable": true,
    "notebookId": "5f8f327c46ba5e00307827a6",
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "根据实验结果我们发现，Actor-Critic算法很快便能收敛到最优策略，并且训练过程非常稳定，抖动情况相比REINFORCE算法有了明显的改进，这多亏了价值函数的引入减小了方差。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9984D8D76C51490C8706CC9DA04A6371",
    "jupyter": {},
    "mdEditEnable": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 总结\n",
    "我们在本章中学习了Actor-Critic算法，它是基于策略和基于价值的方法的叠加。Actor-Critic算法非常实用，往后像DDPG、TRPO、PPO、SAC这样的算法都是在Actor-Critic框架下进行发展的，深入了解Actor-Critic算法对读懂目前深度强化学习的研究热点大有裨益。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}