Version 0.16.0 Release

release-note-build

• Remove setup.py and add pyproject.toml
• Support black

release-note-bugfix

• Fix test error due to mypy's bug
• Fix github actions error

release-note-algorithm

• Feature/20240911 enable squashed gaussian on ppo
• Add IQL
• Support creating variables with tuple states
• Add support for info elements with tuple values
• Add option critic architecture

release-note-docs

• Update readthedocs yaml
• Add isort badge

Install the latest nnablaRL by:

pip install nnabla-rl

Version 0.15.0 Release

special notes

We now support Gymnasium environments in this version of nnabla-rl. You can utilize the Gymnasium environments by importing Gymnasium2GymWrapper from nnabla_rl.environments.wrappers.gymnasium. With this wrapper, all nnabla-rl algorithms should be compatible.

Additionally, we would like to inform you that we have discontinued support for Python 3.7 and have introduced support for Python 3.10, in line with the changes in supported Python versions by nnabla, the deep learning framework we use.

release-note-support

• Remove python 3.7 support and add 3.10 support
• Support gymnasium environment

release-note-bugfix

• Avoid raising exception when info has inconsistent values
• Fix ppo tuple support

release-note-algorithm

• Add HyAR
• Add SRSAC
• Add amp

release-note-utility

• Add format specifier to file writer

Install the latest nnablaRL by:

pip install nnabla-rl

Version 0.14.0 Release

special notes

This version does NOT support the version v0.26.0 and greater of openai gym.
We're going to support openai gym version v0.26.0 and greater in the next release of nnablaRL. nnablaRL will stop officially supporting version less than v0.26.0 of openai gym from the next release.

release-note-bugfix

• Fixing testing code errors
• Fix Deprecation error when using PendulumEnv
• Pass id as positional arg to avoid unexpected error on old gym
• fix evaluation script

release-note-algorithm

• Add xql
• Support mixed (discrete and continuous) state env
• Support tuple action
• Add begin_of_episode to warmup_action

release-note-utility

• improve plot script

release-note-docs

• Apply docformatter

release-note-samples

• Add sample rl project template

Install the latest nnablaRL by:

pip install nnabla-rl

Version 0.13.0 Release

special notes

• This version does NOT support the version v0.26.0 and greater of openai gym.
• We're going to support openai gym version v0.26.0 and greater in the next release of nnablaRL. nnablaRL will stop officially supporting version less than v0.26.0 of openai gym from the next release.

release-note-bugfix

• Fix colab notebook
• Add pygame installation for colab demos
• Fix float max/min value range of gaussian explorer

release-note-algorithm

• Implement SACD
• Add decision transformer

release-note-utility

• Add progress bar hook
• Add accepted shape decorator

release-note-docs

• Add docformatter

Install the latest nnablaRL by:

pip install nnabla-rl

Version 0.12.0 Release

special notes

• This version does NOT support the version v0.26.0 and greater of openai gym.
• We're going to support openai gym version v0.26.0 and greater in the next release of nnablaRL. nnablaRL will stop officially supporting version less than v0.26.0 of openai gym from the next release.
• Only support python 3.7 or greater
  • Python 3.6 is not supported from this new release

release-note-bugfix

• Fix algos. Properly apply grad clip and weight decay
• Correct variable to use during rnn training
• Check np_random instance and use correct randint alternative
• Fix pendulum-env render
• Fix ScreenRenderEnv to support gym 0.25.0

release-note-algorithm

• Run PPO on single process when actor num is 1
• Add qrsac algorithm
• Add REDQ algorithm
• Update to support discrete tuple
• Add icra2018 qtopt
• Add goal_env module
• Add PPO tuple state support
• Add iLQR and LQR
• Add mppi
• Add ddp

release-note-distributions

• Add gmm and Update gaussian

release-note-utility

• Support nnabla-browser

release-note-docs

• Fix module path of sac
• Improve README with graph visulization feature with nnabla-browser

release-note-build

• Extend github build timelimit to 5 minutes

Install the latest nnablaRL by:

pip install nnabla-rl

Version 0.11.0 Release

release-note-bugfix

• Fix readme of reproduction
• Fix cem test
• Fix README samples and add prerequisites for Atari reproduction codes
• Fix tutorial-model
• Fix add workaround to avoid gym error

release-note-algorithm

• Add ATRPO
• Add implementation for RNN support and DRQN algorithm,
  Support RNN models on DQN and DQN inherited algorithms,
  Follow DRQN author's implementation and update results
• Expand RNN support to dist rl algorithms
• Add rnn support to actor critic algorithms
• Support n-step q learning in ddpg, td3, her, sac and ICML2018SAC
• Stop back propagating to target v function
• Add MME-SAC algorithm and Sparse/Delayed mujoco environment and
  Add Disentangled version of MME-SAC

release-note-functions

• Add stop gradient function
• Add random shooting
• Update cem function interface

release-note-distributions

• Add Bernoulli distribution
• Enable sampling from multidimensional logits
• Add one hot softmax

release-note-utility

• Support batched states for evaluation
• Add convenient episode result env
• Add profile function

release-note-docs

• Update version in algorithm catalog
• Add readthedocs yaml and Fixed yaml file
• Add HER and IQN to algorithm catalog

Install the latest nnablaRL by:

pip install nnabla-rl

Version 0.10.0 Release

release-note-bugfix

• Fix interactive-demos used in colab and Fix interactive-demos used in colab about gpu id

release-note-algorithm

• Add HER
• Add Rainbow
• Fix algorithm reproduction directory path
• Add rank-based prioritized replay
• Add Double Dqn
• Move algorithms reproduction dir to reproductions/algorithms
• Enable injecting explorer to algorithm
• Support multi-step Q learning
• Add Categorical Double Dqn
• Add c51 all atari game results
• Support Tuple State and Update compute_v_target_and_advantage to support tuple state

release-note-parametric_functions

• Add spatial_softmax function and Add spatial softmax docs
• Add noisy net

release-note-functions

• Add batch_flatten function
• Add triangular_matrix function

release-note-utility

• Fix load_snapshot

release-note-docs

• Fix docs typo
• Fix typo in readme
• Display correct version
• Fix numpy array typing to np.ndarray
• Add function docs
• Fix docstring of algorithms
• Update NNablaRL to nnablaRL
• Fix typo seemless -> seamless
• Fix build badge URL

Install the latest nnablaRL by:

pip install nnabla-rl

Version 0.9.0 Release

We are happy to announce the release of nnablaRL, a deep reinforcement learning (RL) library built on top of nnabla.
Reinforcement learning is one of the cutting edge machine learning technology that achieves super human performance in the field of gaming, robotics, etc..
We hope that this new library, nnablaRL, helps RL experts and also non-RL experts using reinforcement learning algorithms easily among our nnabla ecosystem.

Features of nnablaRL is the following.

Friendly API

nnablaRL has friendly Python APIs which enables to start training with only 3 lines of python code.

import nnabla_rl
import nnabla_rl.algorithms as A
from nnabla_rl.utils.reproductions import build_atari_env

env = build_atari_env("BreakoutNoFrameskip-v4") # 1
dqn = A.DQN(env)  # 2
dqn.train(env)  # 3

You can also customize the algorithm's hyper parameters easily. For example, you can change the batch size of training data as follows.

import nnabla_rl
import nnabla_rl.algorithms as A
from nnabla_rl.utils.reproductions import build_atari_env

env = build_atari_env("BreakoutNoFrameskip-v4")
config = A.DQNConfig(batch_size=100)
dqn = A.DQN(env, config=config)
dqn.train(env)

In addition to algorithm hyper parameters, you can also flexibly change the training component such as neural network models and model solvers. For details, see sample codes and API documents.

Many builtin algorithms

Most of famous/SoTA deep reinforcement learning algorithms, such as DQN, SAC, BCQ, GAIL, etc., is already implemented in nnablaRL. Implemented algorithms are carefully tested and evaluated. You can easily start training your agent using these verified implementations.
Please check the sample codes and document for detail usage of each algorithm.
You can find the list of implemented algorithms here.

Seemless switching of online and offline training

In reinforcement learning, there are two main training procedures, online and offline, to train the agent. Online training is a training procedure that executes both data collection and network update alternately. Conversely, offline training is a training procedure that updates the network using only existing data. With nnablaRL, you can switch these two training procedures seemlessly. For example, as shown below, you can easily train a robot's controller online using simulated environment and finetune it offline with real robot dataset.

import nnabla_rl
import nnabla_rl.algorithms as A


simulator = get_simulator() # This is just an example. Assuming that simulator exists
dqn = A.DQN(simulator, config=config)
dqn.train_online(simulator)

real_data = get_real_data() # This is also an example. Assuming that you have real robot data
dqn.train_offline(real_data)

Getting started

You can find both notebook style interactive demos and raw python scripts as a sample code to get started. If you are unfamiliar with reinforcement learning, we recommend trying the notebook as a starting point. You can immediately launch and start training through google colaboratory! Check the list of notebooks here.

Development of nnablaRL has just started. We will continue adding new reinforcement learning algorithms and SoTA techniques to nnablaRL. Feedbacks, feature requests and contributions are welcome! Check the contribution guide for details.