FTR-Bench: Benchmarking Deep Reinforcement Learning for Flipper-Track Robot Control

Introduce

This project uses the NuBot rescue robot as the platform and obstacles crossing as the task. It builds a reinforcement learning training system in Isaac Sim and implements commonly used reinforcement learning algorithms.

The core objective of FTR-Bench is to establish a learning framework that enables the articulated tracked robot to navigate obstacles in various terrains. Specifically, FTR-Bench consists of three primary components: the simulation, tasks, and learning algorithms.

The flipper tracked robots have the capability to learn in various terrains. Considering real-world rescue robot applications, we have created four types of terrains, including flat ground, obstacles, bollards, and stairs. The task defines the conditions and objectives that the tracked robots need to meet in each scenario.

Eventually, our experiments demonstrate that RL can facilitate the robots to achieve some remarkable performance on such challenging tasks, enabling the agent to learn the optimal obstacle-navigation strategies within the environment. And there is still some room for improvement and more difficult tasks for future work.

Tasks Design

Task Name	Description	Demo
Up the Steps	Simulate the tracked robot ascending steps. The height of the steps ranges from 0.2$m$ to 0.4$m$.
Down the Steps	Simulate the tracked robot descending steps. The height of the steps ranges from 0.2$m$ to 0.6$m$.
Cross the Uplifts	Simulate the tracked robot crossing over uplifts. The height of the uplifts ranges from 0.2$m$ to 0.4$m$, and the width ranges from 0.2$m$ to 0.5$m$.
Through Single-sided Steps	Simulate a tracked robot going through a single-side step. The height of the single-side step ranges from 0.1$m$ to 0.3$m$.
Drive on Plum Piles	Simulate the tracked vehicle navigating on plum piles. The size of each individual plum pile ranges from 0.2$m$ to 0.5$m$, and the height ranges from 0.2$m$ to 0.5$m$.
Drive on Wave Terrains	Simulate the tracked vehicle navigating through waveform terrains. The size of each individual peak in the waveform terrain ranges from 0.25$m$ to 1$m$, and the height ranges from 0.1$m$ to 0.4$m$.
Cross the Rails	Simulate the tracked robot crossing over rails. The height of the railing ranges from 0.2$m$ to 0.3$m$, and the angle with the vehicle ranges from 10 to 45 degrees.
Up the Stairs	Simulate the tracked vehicle climbing stairs. The width of each step of the stairs ranges from 0.2$m$ to 0.3$m$, and the slope of the stairs ranges from 20 to 45 degrees.
Down the Stairs	Simulate the tracked vehicle descending stairs. The width of each step of the stairs ranges from 0.2$m$ to 0.3$m$, and the slope of the stairs ranges from 20 to 45 degrees.
Mixed Terrains	Simulate the tracked vehicle navigating through various complex terrains, with the successful completion of the task defined when the vehicle reaches the specified endpoint.

Install

Setup

When using FTR, you need to configure some environment variables. Please execute the following code in the root directory of the project.

source shell/setup/ptask.sh

Or you can add the following code to .bashrc

export PTASK_HOME=<your FTR directory>
source ${PTASK_HOME}/shell/setup/ptask.sh

Isaac Sim

The simulation platform used in this project is Isaac Sim, and version 2023.1.1 is recommended. For detailed installation instructions, please refer to the official website installation address.

download and install Isaac Sim

If there are multiple Isaac Sims on your computer, please modify the environment variable ISAACSIM_HOME configured in the shell/setup/ptask.sh file.

export ISAACSIM_HOME=`realpath <Specify the path of Isaac Sim>`

Conda

If you want to run the project in Conda, please configure the environment variable PTASK_CONDA_NAME to indicate the conda environment name you want to use.

export PTASK_CONDA_NAME=<your conda environment name>

Install dependencies

If your configuration is successful, you can use the isaac-python and isaac-pip commands in bash. Use isaac-pip to install dependencies

isaac-pip install -r ptask_requirements.txt

Train

To start training, you can execute the following code

isaac-python -m ptask_ftr task=benchmark/StepsUp train=FTR/SAC headless=True num_envs=2 experiment='runs/StepUp_SAC' rl_device='cuda:0' sim_device=cpu

If you want to test the training results, you can use the following command

isaac-python -m ptask_ftr task=benchmark/StepsUp train=FTR/SAC test=True checkpoint=<checkpoint file>

Supported RL algorithms are listed below:

• Proximal Policy Optimization (PPO) (Paper, Config)
• Trust Region Policy Optimization (TRPO) (Paper, Config)
• Twin Delayed DDPG (TD3) (Paper, Config)
• Soft Actor-Critic (SAC) (Paper, Config)
• Deep Deterministic Policy Gradient (DDPG) (Paper, Config)

Enviroment Performance

Up the Steps		Down the Steps
Cross the Uplifts		Through Single-sided Steps
Drive on Plum Piles		Drive on Wave Terrains
Cross the Rails		Up the Stairs
Down the Stairs		Mixed Terrains

Research papers published using FTR-Bench

(Click to Collapse)

• Research paper
  • Geometry-Based Flipper Motion Planning for Articulated Tracked Robots Traversing Rough Terrain in Real-time: Journal of Field Robotics, 2023, https://doi.org/10.1002/rob.22236.
  • Deep Reinforcement Learning for Flipper Control of Tracked Robots: ArXiv 2023, https://doi.org/10.48550/arXiv.2306.10352.