Example of doing Reinforcement Learning in Gazebo

[arjo129]

Desired behavior

Currently, doing Reinforcement Learning in Gazebo is really hard. It is not impossible (as we already have reset methods), but as RL gains traction in the Robotics Community we are seeing real world use cases where walking models are being developed on other simulators. It would be nice to provide users with a way of doing RL. In particular, perhaps an example which exposes a gym api for one of the major RL libraries (stable baselines, rllib, etc). This work would also necessitate some improvements to the python API and some work on the python packaging side. In particular my proposal is we do the following:

• Start with replicating the simplest possible RL scenario: The inverted pendulum or cart-pole. Adds an initial StableBaselines3 RL environment as an example #2667
• Improve the reset API story (currently we support resetting over the network, but a direct API call would be nice).
  • Allow resetting a test fixture Adds support for ISystemReset in test fixture #2647
    OR
  • Expose an API that exposes the ECM outside of PostUpdate, Update and PreUpdate.
• Expose a gym API for one of the RL libraries.
• Run multiple instances with randomized environment.
• Write measured sensor values to ECM for deterministic closed-loop control #2391
• Figure out a way of starting the gui client in a cross platform manner.

Stretch goals

• Improve collision APIs. - Often we may want to use collisions as a proxy for whether a task should terminate or not.
• Improve story of imports by removing package number from names.
• Set up real world test rig to compare results
• Extend to multi-agent support
• More examples including grasping/object manipulation examples.
• Make it easier to load python ML frameworks with gazebo.
• Have a ROS Integration story
• Environment randomization API story
• Reset individual models or support vectorization.

Some relevant projects

• Early attempt at RL with gazebo: https://github.com/robotology/gym-ignition
• Gymnasium (which provides example environments): https://gymnasium.farama.org/

Few Other Design Considerations

The work done on the python bindings actually exposes a really nice API. It is also nice that we have a separation between client and server API.