BIRDy: Benchmark for Identification of Robot Dynamics

Without a suitable framework, it is rather difficult for students, engineers or researchers to evaluate the relevance of a parameter identificaiton method for a given scenario.

We here propose a unifying benchmark dedicated to robot identification. So far the following algorithms are implemented:

• Inverse Dynamic Identification Model with Ordinary Least Square (IDIM-OLS)
• Inverse Dynamic Identification Model with Weighted Least Square (IDIM-WLS)
• Inverse Dynamic Identification Model with Iteratively Reweighted Least Square (IDIM-IRLS)
• Inverse Dynamic Identification Model with Total Least Square (IDIM-TLS)
• Inverse Dynamic Identification Model with Maximum Likelihood (IDIM-ML)
• Inverse Dynamic Identification Model with Instrumental Variables (IDIM-IV)
• Direct and Inverse Dynamic Identification Model (DIDIM)
• Closed-Loop Output-Error (CLOE)
• Closed-Loop Input-Error (CLIE)
• Direct Dynamic Identification Model with Extended Kalman Filter (DDIM-EKF)
• Direct Dynamic Identification Model with Square-Root Extended Kalman Filter (DDIM-SREKF)
• Direct Dynamic Identification Model with Unscented Kalman Filter (DDIM-UKF)
• Direct Dynamic Identification Model with Square-Root Unscented Kalman Filter (DDIM-SRUKF)
• Direct Dynamic Identification Model with Central Difference Kalman Filter (DDIM-CDKF)
• Direct Dynamic Identification Model with Square-Root Central Difference Kalman Filter (DDIM-SRCDKF)
• Adaline Neural Network (AdaNN)
• Hopfield-Tank Recurrent Neural Network (HTRNN)
• Physically Consistent Inverse Dynamic Identification Model with Ordinary Least Square (PC-IDIM-OLS)
• Physically Consistent Inverse Dynamic Identification Model with Weighted Least Square (PC-IDIM-WLS)
• Physically Consistent Inverse Dynamic Identification Model with Iteratively Reweighted Least Square (PC-IDIM-IRLS)
• Physically Consistent Direct and Inverse Dynamic Identification Model (PC-DIDIM)
• Physically Consistent Inverse Dynamic Identification Model with Instrumental Variables (PC-IDIM-IV)

Getting started

Prerequisites

So far the benhmark was successfully tested on the following platforms:

• Ubuntu 14.04/16.04 + (Matlab R2017a/b, Matlab R2018a/b, Matlab R2019a/b, Matlab R2020a/b, Matlab R2021a)
• Windows 10 + (Matlab R2017a/b, Matlab R2018a/b, Matlab R2019a/b)

It should work on Mac platforms as well. Please let us know if you find any bug.

Important: in case you want to test the physically consistent identification methods based on Semi-Definite Programming (SDP), you must install the CVX constrained optimization framework. Note that the results presented in the paper were obtained with the MOSEK solver. You should install these softwares and add them to the Matlab path to be able to use them.

Benchmark structure

• The "Utils" folder contains the initialization routines called by the main matlab script "run_identification_benchmark.m".
• The "Benchmark" folder contains 4 subfolders:
  • Robot_Data_Generation contains the model generation routines, as well as trajectory and experimental data generation tools. These tools are divided into three folders, namely:
    • Dynamic_Model_Generation (symbolic model generation routines)
    • Experiment_Data_Generation (experimental data generation routines)
    • Trajectory_Data_Generation (trajectory generation routines)
  • Robot_Generated_Data contains all the data generated by the previous tools and distributed for each robot in five sub-folders:
    • <robot_name>/Homogeneous_Transforms
    • <robot_name>/Jacobian_Matrices
    • <robot_name>/Dynamic_Model
    • <robot_name>/Trajectory_Data
    • <robot_name>/Experiment_Data
  • Robot_Identification_Algorithms contains the different parametric identification algorithms
  • Robot_Identification_Results contains the results of the different algorithms and a set of post-processing functions

How to run the benchmark

To run the parameter identification benchmark, execute the matlab script "run_identification_benchmark.m".

• Once started, this script will call the function "Utils/initBenchmark.m", which will in turn:
  1. build the benchmark data structures,
  2. check that the robot identification model has already been computed and recompute it automatically if necessary,
  3. check that the robot trajectory has already been generated and reoptimize it automatically if necessary,
  4. check that the experiment data exist and correspond to the trajectory data. Regenetate it automatically if not,
  5. check .mex files compatibility and recompile them if necessary.
• Then the desired identification algorithm will be called sequentially on the generated data.
• Once the identification process is over, the results are stored automatically in the folder "Benchmark/Robot_Identification_Results" as a set of .mat files.
• If desired, these files can eventually be parsed by the post-processing script "Benchmark/Robot_Identification_Results/postProcessing.m" in order to generate curves.

You can choose which algorithm to execute and which result to send to the post-processing script, by modifying the data structure "identificationMethods", within the main matlab script "run_identification_benchmark.m".

How to (re)generate an excitation trajectory

• If you wish to generate a new robot trajectory for the experiments, set the "regenerateTrajectory" flag to "true", in the script "run_identification_benchmark.m".
• At the next execution of the benchmarrk, an optimization script will be called and generate a suitable excitation trajectory by parametrising Fourier series.
• You can modify the optimization algorithm by changing the value of the "alg" variable in the script "Benchmark/Robot_Data_Generation/Trajectory_Data_Generation/generateExcitationTrajectory.m". So far, you have the choice between 'fmincon' and a generic genetic algorithm.
• The cost function of the optimizer can also be adjusted by modifying the variable "J" in the script "Benchmark/Robot_Data_Generation/Trajectory_Data_Generation/trajectory_criterion.m"
• It is possible to add constraints to the optimizer in order to avoid inconsistant trajectories (e.g. self-collisions, and so on). This can be done in the he script "Benchmark/Robot_Data_Generation/Trajectory_Data_Generation/traj_nl_constraint.m".
• Note that the experiment data is automatically re-generated when an inconststancy is detected between the actual data and the generated trajectory. For more details have a look at the trajectory check function in "Utils/trajectoryCheck.m"

How to (re)generate experiment data

• If you wish to generate a new bunch of experiment data, set the "regenerateData" flag to "true", in the script "run_identification_benchmark.m".
• At the next execution of the benchmarrk, the robot model will track the excitation trajectory using a noisy PID controller.
• PID gains and noise levels are defined for each robots in the file "Utils/SymbolicModelData/Robot_Description_Files/<robot_name>.m" as:

  robot.controlParameters.Kp          % Proportional gains
  robot.controlParameters.Ki          % Integral Gains
  robot.controlParameters.Kd          % Derivative Gains
  robot.numericalParameters.sd_q      % Encoders noise standard deviation (we assume zero mean gaussian noise)
  robot.numericalParameters.sd_tau    % Joint torque sensors noise standard deviation (we assume zero mean gaussian noise)

• Details about the experiment data generation procedure are given in "Benchmark/Robot_Data_Generation/Experiment_Data_Generation/generateData.m"

How to (re)generate the indentification model of a robot

• If the symbolic model of your robot already exists but got somehow corrupted, it is possible to re-generate it by setting the "regenerateModel" flag to "true", in the script "run_identification_benchmark.m".
• At the next execution of the benchmark, the whole robot kinematic, dynamic and identification models will be recomputed.
• If you want only one part of the model to be recomputed (e.g. only kinematics or only dynamics), modify the value of the flag "optionsSymbolicComputation" in the script "Utils/initBenchmark.m".
• Once the model has been regenerated, it is strongly advised to re-run the trajectory generation and experiment data generation routines.

How to recompiling the .mex files of the benchmark

• When possible, the Matlab C/C++ code generation toolbox was used in order to enhance the execution speed of concatenated for-loops, such as the ones found in the UKF, CDKF or PF sample propagation.
• If the Matlab C/C++ code generation toolbox is not detected on the host machine, the benchmark will proceed using normal matlab functions.
• Otherwise, the existence of the mex files will be checked automatically. The. mex files will be recompiled if they are not detected on the host machine or if they are incompatible with its operating system or the version of matlab used.
• It is possible to bypass this procedure and force recompilation of mex files by setting the "recompileMexFiles" flag to "true", in the script "run_identification_benchmark.m".
• It is also possible to specifically select which file(s) have to be recompiled by modifying the "buildList" variable in the script "run_identification_benchmark.m".

Contributing

Adding a new robot to the benchmark

This is actually pretty simple, provided that you know the Denavit Hartenberg parameters of your robot (classic or modified) and have a rough idea of its dynamic parameters. Robot models are defined within dedicated files, contained in the folder "Utils/SymbolicModelData/Robot_Description_Files". The function "Benchmark/Robot_Data_Generation/Dynamic_Model_Generation/loadRobotModelParameters.m" will parse the desired description file. This functon takes the name of your robot as an argument (e.g. mySuperRobot), and returns a "robot" data structure which can in turn be used in the rest of the benchmark.

To add a new robot, just create a new file in the folder "Utils/SymbolicModelData/Robot_Description_Files". The name of the file should be that of the robot and should follow the structure of the other robot description files of the folder:

        robot.name = 'mySuperRobot';                            % Name of the robot
        robot.nbDOF = 4;                                        % Number of Degrees Of Freedom (DOF)
        robot.jointType = ['revol';'revol';'revol';'prism'];    % Type of the robot joints: either 'revol' (revolute) or 'prism' (prismatic)
        ...

Then add a reference to this file in the "switch" condition of Benchmark/Robot_Data_Generation/Dynamic_Model_Generation/loadRobotModelParameters.m. Take a look at the examples already available in "Utils/SymbolicModelData/Robot_Description_Files".

Adding a new identification algorithm to the benchmark

• Create a new sub-folder in Robot_Identification_Algorithms with the name of your algorithm (e.g. Deep_NN).
• This folder should contain at least two matlab functions:
  1. run_Deep_NN.m: defines the data structure which contains the identification results and executes the parametric identification script for each starting point.
  2. Deep_NN_identification.m: contains the identification algorithm.