batch size vs n_env*n_steps

[geoffreyvd]

I have a question regarding this exmaple PPO hyperparameter configuration:

MountainCarContinuous-v0:
normalize: true
n_envs: 1
n_timesteps: !!float 20000
policy: 'MlpPolicy'
batch_size: 256
n_steps: 8
gamma: 0.9999
learning_rate: !!float 7.77e-05
ent_coef: 0.00429
clip_range: 0.1
n_epochs: 10
gae_lambda: 0.9
max_grad_norm: 5
vf_coef: 0.19
use_sde: True
policy_kwargs: "dict(log_std_init=-3.29, ortho_init=False)"

How can the batch size be beigger than the collected rollout buffer size?
The rollout buffer size is n_envs*n_steps, which is 8.
And the batch size is 256. how is this possible?

And on top of that the n_epochs is 10.

[araffin]

Hello,
you are correct, PPO will actually print a warning:

stable_baselines3/ppo/ppo.py:155: UserWarning: You have specified a mini-batch size of 256, but because the `RolloutBuffer` is of size `n_steps * n_envs = 8`, after every 0 untruncated mini-batches, there will be a truncated mini-batch of size 8
We recommend using a `batch_size` that is a factor of `n_steps * n_envs`.

Each minibatch will be truncated to 8.

In fact, it will give you the same result as python train.py --algo ppo --env MountainCarContinuous-v0 -param batch_size:8

I would appreciate a PR that updates this parameter ;)