Benchmark

Introduction

This page explains how to reproduce results (like Precision/Recall curve) shown in the paper "Appearance-Based Loop Closure Detection for Online Large-Scale and Long-Term Operation".

To to that, we use the tool called "rtabmap-console" created when building RTAB-Map from source. The datasets used can be downloaded following the links on RTAB-Map's page on IntRoLab as well as the ground truths used. For datasets with right/left images like NewCollege and CityCentre, the tool "rtabmap-imagesJoiner" can be used to merge left and right images into single image. RTAB-Map should be installed with NON-FREE support (SURF and SIFT)!

Processing a dataset with rtabmap-console

Let's take for example the Lip6Indoor dataset. For other datasets, look at this section below. Download the indoor dataset on the ENSTA page. Since RTAB-Map's default parameters changed over time for RGB-D mapping/scanning, we must specify some loop closure parameters manually. A exhaustive list of all parameters can be found in "Parameters.h".

Here a mapping between parameters in the paper and those in the code:

Paper	C++ Code
`T_STM=30`	`Mem/STMSize=30`
`T_similarity=20%`	`Mem/RehearsalSimilarity=0.2`
`T_recent=20%`	`Mem/RecentWmRatio=0.2`
SURF dimension = 64	`SURF/Extended=false`
SURF `T_maxFeatures=400`	`Kp/WordsPerImage=400`
SURF `T_bad=0.25`	`Kp/BadSignRatio=false`

Variable thresholds depending on the datasets:

Paper	C++ Code
`T_time` (s)	`Rtabmap/TimeThr` (ms)
SURF `T_response`	`SURF/HessianThreshold`

The other parameters set:

Rtabmap/StatisticLogged=true: we need the logged files LogF.txt and LogI.txt for MATLAB. These files are written in the working memory of RTAB-Map (default "~/Documents/RTAB-Map").
Rtabmap/StatisticLoggedHeaders=false: Don't print column headers in log files.
Rtabmap/PublishStats=false: it is only useful in GUI mode.
Mem/RehearsalIdUpdatedToNewOne=true: When merging images, keep the latest ID. The ground truths are based on the latest ID for locations merged together during Weight Update.
Mem/BadSignaturesIgnored=true: In the paper, we ignore bad signatures.
Rtabmap/LoopRatio=0.0: The loop ratio is used to rejected dropping hypotheses, which means that we are moving to another area. In the paper it was not used, loop closures were accepted in the area of the ground truth even if the images are very different.
Bayes/FullPredictionUpdate=true: In the papers, the Bayes prediction was regenerated at each frame.
Kp/TfIdfLikelihoodUsed=false: In the papers, tf-idf weighting wasn't used.
Mem/TransferSortingByWeightId=true: (Version>=0.8.6) In the papers, locations selected for Transfer were sorted by weight/ID, not by weight/Age/ID.

The command to process the Lip6Indoor dataset: $ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 10 -Rtabmap/TimeThr 700 Lip6IndoorDataSet/Images

Recision/Recall results with MATLAB

Open MATLAB and add to path the directory of the file ShowLog.m from RTAB-Map: rtabmap-source/Matlab/ShowLogs. Then execute the showlogs method. The first path is where the LogF.txt dans LogI.txt files were saved and the second path is the ground truth. The ground truth is simply a matrix represented by a image file for convenience where white pixels are loop closures between the corresponding indexes of images and the black pixels are no loop closures. For this example, the ground truth can be downloaded here:

MATLAB:

>> showlogs('~/.', '~/Lip6Indoor.png');

Octave 3.8 (copy files from rtabmap/Matlab/ShowLogs in the current directory where LogF.txt and LogI.txt are):

$ octave octave:1> showlogs('.', 'Lip6Indoor.png');


Figure 9 is the Precision/Recall curve and you should see an output like this:

--- getPrecisionRecall --- Loading GroundTruth '~/Documents/RTAB-Map/Lip6Indoor.png' ...

GT_total_positives =

226

False positive! id=149 with old=86 (p=0.11314) False positive accepted! id=149 with old=86 (p=0.11314) False positive! id=150 with old=87 (p=0.11256) False positive accepted! id=150 with old=87 (p=0.11256) False positive! id=148 with old=85 (p=0.10664) False positive! id=140 with old=102 (p=0.1035) False positive! id=93 with old=22 (p=0.10234) False positive! id=55 with old=9 (p=0.09511) [...] Recall max (Precision=100%) = 98.2301% (p=0.11422), accepted=222 Recall max accepted (Precision=100%) = 98.2301% (p=0.11422), accepted=222 ignored = 0

Note that the lines beginning with "False positive!" may contain true positives if the ground truth is too narrow on the accepted loop closures. Sometimes, we have to manually add white pixels to the ground truth file if a loop closure is found with an image just next to the defined ground truth one. 

[[doc/Tutorials/Benchmark/recall_lip6indoor.png]]

## Commands for all datasets

#### New College 
 * [Dataset link](http://www.robots.ox.ac.uk/~mobile/IJRR_2008_Dataset)
 * [Ground truth](https://introlab.3it.usherbrooke.ca/mediawiki-introlab/images/5/5a/NewCollege.png)
 * Note: You can use rtabmap-imagesJoiner tool to merge left and right images.

`
$ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 1000 -Rtabmap/TimeThr 1400  NewCollege
`

 * [Example rtabmap-console and MATLAB output](https://github.com/introlab/rtabmap/wiki/Example-NewCollege)

#### City Centre
 * [Dataset link](http://www.robots.ox.ac.uk/~mobile/IJRR_2008_Dataset)
 * [Ground truth](https://introlab.3it.usherbrooke.ca/mediawiki-introlab/images/d/d5/CityCentre.png)
 * Note: You can use rtabmap-imagesJoiner tool to merge left and right images.

`
$ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 1000 -Rtabmap/TimeThr 1400  CityCentre
`

 * [Example rtabmap-console and MATLAB output](https://github.com/introlab/rtabmap/wiki/Example-CityCentre)

#### Lip6 Indoor
 * [Dataset link](http://cogrob.ensta.fr/loopclosure.html)
 * [Ground truth](https://introlab.3it.usherbrooke.ca/mediawiki-introlab/images/a/a5/Lip6Indoor.png)

`
$ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 10 -Rtabmap/TimeThr 700  Lip6Indoor
`

#### Lip6 Outdoor
 * [Dataset link](http://cogrob.ensta.fr/loopclosure.html)
 * [Ground truth](https://introlab.3it.usherbrooke.ca/mediawiki-introlab/images/0/08/Lip6Outdoor.png)
 * Note: Keeping only even images and remove the last one.

`
./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 10 -Rtabmap/TimeThr 1400  Lip6Outdoor
`

#### 70 km
 * [Dataset link](http://www.robots.ox.ac.uk/~mobile)
 * [Ground truth](https://introlab.3it.usherbrooke.ca/mediawiki-introlab/images/0/03/Eynsham70km.png)

`
./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 1000 -Rtabmap/TimeThr 700  70km
`

#### New College Omni
 * [Dataset link](http://www.robots.ox.ac.uk/NewCollegeData/)
 * [Ground truth](https://introlab.3it.usherbrooke.ca/mediawiki-introlab/images/3/33/NewCollegeOmni.png)
 
`
./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.35 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 1000 -Rtabmap/TimeThr 700  NewCollegeOmni
`

#### Crowded Canteen
 * [Dataset link](http://haselab.info/pirf.html)
 * [Ground truth](https://introlab.3it.usherbrooke.ca/mediawiki-introlab/images/c/c2/CrowdedCanteen.png)
 
`
./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.35 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 75 -Rtabmap/TimeThr 350  CrowdedCanteen
`

#### Bicocca Indoor 2009-02-25b
 * [Dataset link](http://www.rawseeds.org/home/category/benchmarking-toolkit/datasets/)
 * [Ground truth](https://introlab.3it.usherbrooke.ca/mediawiki-introlab/images/4/43/BicoccaIndoor-2009-02-25b.png)
 
`
./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 50 -Rtabmap/TimeThr 700  BicoccaIndoor-2009-02-25b
`

#### Bovisa Outdoor 2008-10-04
 * [Dataset link](http://www.rawseeds.org/home/category/benchmarking-toolkit/datasets/)
 * [Ground truth](https://introlab.3it.usherbrooke.ca/mediawiki-introlab/images/0/0d/BovisaOutdoor-2008-10-04.png)
 
`
./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 50 -Rtabmap/TimeThr 700  BovisaOutdoor-2008-10-04
`

#### Bovisa Mixed 2008-10-06
 * [Dataset link](http://www.rawseeds.org/home/category/benchmarking-toolkit/datasets/)
 * [Ground truth](https://introlab.3it.usherbrooke.ca/mediawiki-introlab/images/e/e1/BovisaMixed-2008-10-06.png)
 
`
./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 50 -Rtabmap/TimeThr 700  BovisaMixed-2008-10-06
`

#### UdeS 1Hz
 * [Dataset link](https://introlab.3it.usherbrooke.ca/mediawiki-introlab/index.php/RTAB-Map#Data_sets)
 * [Ground truth](https://introlab.3it.usherbrooke.ca/mediawiki-introlab/images/1/16/UdeS_1Hz.png)
 
`
./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 150 -Rtabmap/TimeThr 700 UdeS_1Hz
`

## Videos for all datasets
Visit [RTAB-Map on IntRoLab website](https://introlab.3it.usherbrooke.ca/mediawiki-introlab/index.php?title=RTAB-Map&setlang=en).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Benchmark

Introduction

Processing a dataset with rtabmap-console

Recision/Recall results with MATLAB

Clone this wiki locally