Benchmark

• Introduction
• Processing a dataset with rtabmap-console
• Recision/Recall results with MATLAB
• Commands for all datasets
• New College
• City Centre
• Lip6 Indoor
• Lip6 Outdoor
• 70 km
• New College Omni
• Crowded Canteen
• Bicocca Indoor 2009-02-25b
• Bovisa Outdoor 2008-10-04
• Bovisa Mixed 2008-10-06
• Malaga2009 Campus 2L
• Malaga2009 Parking 6L
• UdeS 1Hz
• Videos for all datasets

Introduction

This page explains how to reproduce results (like Precision/Recall curve) shown in the paper "Appearance-Based Loop Closure Detection for Online Large-Scale and Long-Term Operation".

To to so, we use the tool called "rtabmap-console". The datasets used can be downloaded following the links on RTAB-Map's page on IntRoLab as well as the ground truths used. For datasets with right/left images like NewCollege and CityCentre, the tool "rtabmap-imagesJoiner" can be used to merge left and right images into single image. RTAB-Map should be installed with NON-FREE support (SURF and SIFT)!

Processing a dataset with rtabmap-console

Let's take for example the Lip6Indoor dataset. For other datasets, look at this section below. Download the indoor dataset on the ENSTA page. Since RTAB-Map's default parameters changed over time for RGB-D mapping/scanning, we must specify some loop closure parameters manually. A exhaustive list of all parameters can be found in "Parameters.h".

Here a mapping between parameters in the paper and those in the code:

Paper	C++ Code
T_STM=30	Mem/STMSize=30
T_similarity=20%	Mem/RehearsalSimilarity=0.2
T_recent=20%	Mem/RecentWmRatio=0.2
SURF dimension = 64	SURF/Extended=false
SURF T_maxFeatures=400	Kp/WordsPerImage=400
SURF T_bad=0.25	Kp/BadSignRatio=false

Variable thresholds depending on the datasets:

Paper	C++ Code
T_time (s)	Rtabmap/TimeThr (ms)
SURF T_response	SURF/HessianThreshold

The other parameters set:

• Rtabmap/StatisticLogged=true: we need the logged files LogF.txt and LogI.txt for MATLAB. These files are written in the working memory of RTAB-Map (default "~/Documents/RTAB-Map").
• Rtabmap/StatisticLoggedHeaders=false: Don't print column headers in log files.
• Rtabmap/PublishStats=false: it is only useful in GUI mode.
• Mem/RehearsalIdUpdatedToNewOne=true: When merging images, keep the latest ID. The ground truths are based on the latest ID for locations merged together during Weight Update.
• Mem/BadSignaturesIgnored=true: In the paper, we ignore bad signatures.
• Rtabmap/LoopRatio=0.0: The loop ratio is used to reject dropping hypotheses, which means that we are moving to another area. In the paper it was not used, loop closures were accepted in the area of the ground truth even if the images are very different.
• Bayes/FullPredictionUpdate=true: In the papers, the Bayes prediction was regenerated at each frame.
• Kp/TfIdfLikelihoodUsed=false: In the papers, tf-idf weighting wasn't used.
• Mem/TransferSortingByWeightId=true: (Version>=0.8.6) In the papers, locations selected for Transfer were sorted by weight/ID, not by weight/Age/ID.
• Kp/IncrementalFlann=false: In the original paper, the boost performance of incremental FLANN was not implemented.

The command to process the Lip6Indoor dataset: $ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 10 -Kp/IncrementalFlann false -Rtabmap/TimeThr 700 Lip6IndoorDataSet/Images

Recision/Recall results with MATLAB

Open MATLAB and add to path the directory of the file ShowLog.m from RTAB-Map: rtabmap-source/Matlab/ShowLogs. Then execute the showlogs method. The first path is where the LogF.txt dans LogI.txt files were saved and the second path is the ground truth. The ground truth is simply a matrix represented by a image file for convenience where white pixels are loop closures between the corresponding indexes of images and the black pixels are no loop closures. For this example, the ground truth can be downloaded here:

• MATLAB:

>> showlogs('~/.', '~/Lip6Indoor.png');

• Octave 3.8 (copy files from rtabmap/Matlab/ShowLogs in the current directory where LogF.txt and LogI.txt are):

$ octave octave:1> showlogs('.', 'Lip6Indoor.png');

Figure 9 is the Precision/Recall curve and you should see an output like this:

--- getPrecisionRecall --- Loading GroundTruth '~/Documents/RTAB-Map/Lip6Indoor.png' ...

GT_total_positives =

226

False positive! id=149 with old=86 (p=0.11314) False positive accepted! id=149 with old=86 (p=0.11314) False positive! id=150 with old=87 (p=0.11256) False positive accepted! id=150 with old=87 (p=0.11256) False positive! id=148 with old=85 (p=0.10664) False positive! id=140 with old=102 (p=0.1035) False positive! id=93 with old=22 (p=0.10234) False positive! id=55 with old=9 (p=0.09511) [...] Recall max (Precision=100%) = 98.2301% (p=0.11422), accepted=222 Recall max accepted (Precision=100%) = 98.2301% (p=0.11422), accepted=222 ignored = 0

Note that the lines beginning with "False positive!" may contain true positives if the ground truth is too narrow on the accepted loop closures. Sometimes, we have to manually add white pixels to the ground truth file if a loop closure is found with an image just next to the defined ground truth one. 

[[doc/Tutorials/Benchmark/recall_lip6indoor.png]]

## Commands for all datasets

#### New College 
 * [Dataset link](http://www.robots.ox.ac.uk/~mobile/IJRR_2008_Dataset)
 * [Ground truth](https://introlab.3it.usherbrooke.ca/mediawiki-introlab/images/5/5a/NewCollege.png)
 * Note: You can use rtabmap-imagesJoiner tool to merge left and right images.

`
$ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 1000 -Kp/IncrementalFlann false -Rtabmap/TimeThr 1400  NewCollege
`

 * [Example rtabmap-console and MATLAB output](https://github.com/introlab/rtabmap/wiki/Example-NewCollege)

#### City Centre
 * [Dataset link](http://www.robots.ox.ac.uk/~mobile/IJRR_2008_Dataset)
 * [Ground truth](https://introlab.3it.usherbrooke.ca/mediawiki-introlab/images/d/d5/CityCentre.png)
 * Note: You can use rtabmap-imagesJoiner tool to merge left and right images.

`
$ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 1000 -Kp/IncrementalFlann false -Rtabmap/TimeThr 1400  CityCentre
`

 * [Example rtabmap-console and MATLAB output](https://github.com/introlab/rtabmap/wiki/Example-CityCentre)

#### Lip6 Indoor
 * [Dataset link](http://cogrob.ensta.fr/loopclosure.html)
 * [Ground truth](https://introlab.3it.usherbrooke.ca/mediawiki-introlab/images/a/a5/Lip6Indoor.png)

`
$ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 10 -Kp/IncrementalFlann false -Rtabmap/TimeThr 700  Lip6Indoor
`

#### Lip6 Outdoor
 * [Dataset link](http://cogrob.ensta.fr/loopclosure.html)
 * [Ground truth](https://introlab.3it.usherbrooke.ca/mediawiki-introlab/images/0/08/Lip6Outdoor.png)
 * Note: Keeping only even images and remove the last one.

`
$ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 10 -Kp/IncrementalFlann false -Rtabmap/TimeThr 1400  Lip6Outdoor
`

#### 70 km
 * [Dataset link](http://www.robots.ox.ac.uk/~mobile)
 * [Ground truth](https://introlab.3it.usherbrooke.ca/mediawiki-introlab/images/0/03/Eynsham70km.png)

`
$ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 1000 -Kp/IncrementalFlann false -Rtabmap/TimeThr 700  70km
`

#### New College Omni
 * [Dataset link](http://www.robots.ox.ac.uk/NewCollegeData/)
 * [Ground truth](https://introlab.3it.usherbrooke.ca/mediawiki-introlab/images/3/33/NewCollegeOmni.png)
 
`
$ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.35 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 1000 -Kp/IncrementalFlann false -Rtabmap/TimeThr 700  NewCollegeOmni
`

#### Crowded Canteen
 * [Dataset link](http://haselab.info/pirf.html)
 * [Ground truth](https://introlab.3it.usherbrooke.ca/mediawiki-introlab/images/c/c2/CrowdedCanteen.png)
 
`
$ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.35 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 75 -Kp/IncrementalFlann false -Rtabmap/TimeThr 350  CrowdedCanteen
`

#### Bicocca Indoor 2009-02-25b
 * [Dataset link](http://www.rawseeds.org/home/category/benchmarking-toolkit/datasets/)
 * [Ground truth](https://introlab.3it.usherbrooke.ca/mediawiki-introlab/images/4/43/BicoccaIndoor-2009-02-25b.png)
 
`
$ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 50 -Kp/IncrementalFlann false -Rtabmap/TimeThr 700  BicoccaIndoor-2009-02-25b
`

#### Bovisa Outdoor 2008-10-04
 * [Dataset link](http://www.rawseeds.org/home/category/benchmarking-toolkit/datasets/)
 * [Ground truth](https://introlab.3it.usherbrooke.ca/mediawiki-introlab/images/0/0d/BovisaOutdoor-2008-10-04.png)
 
`
$ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 50 -Kp/IncrementalFlann false -Rtabmap/TimeThr 700  BovisaOutdoor-2008-10-04
`

#### Bovisa Mixed 2008-10-06
 * [Dataset link](http://www.rawseeds.org/home/category/benchmarking-toolkit/datasets/)
 * [Ground truth](https://introlab.3it.usherbrooke.ca/mediawiki-introlab/images/e/e1/BovisaMixed-2008-10-06.png)
 
`
$ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 50 -Kp/IncrementalFlann false -Rtabmap/TimeThr 700  BovisaMixed-2008-10-06
`

#### Malaga2009 Campus 2L
 * [Dataset link](http://www.mrpt.org/malaga_dataset_2009)
 * [Ground truth](https://introlab.3it.usherbrooke.ca/mediawiki-introlab/images/d/d1/Malaga2009_campus_2L.png)
 * Note: Left camera images are used. There are 5221 images at 7.5Hz in this dataset. I used 1 image of 8 to have ~1Hz, so 653 images are used here.

`
$ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 1000 -Kp/IncrementalFlann false -Rtabmap/TimeThr 700 malaga2009_campus_2L
`

 * [Example rtabmap-console and MATLAB output](https://github.com/introlab/rtabmap/wiki/Example-Malaga2009-Campus-2L)

#### Malaga2009 Parking 6L
 * [Dataset link](http://www.mrpt.org/malaga_dataset_2009)
 * [Ground truth](https://introlab.3it.usherbrooke.ca/mediawiki-introlab/images/8/8c/Malaga2009_parking_6L.png)
 * Note: Left camera images are used. There are 3475 images at 7.5Hz in this dataset. I used 1 image of 8 to have ~1Hz, so 435 images are used here.

`
$ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 1000 -Kp/IncrementalFlann false -Rtabmap/TimeThr 700 malaga2009_parking_6L
`

 * [Example rtabmap-console and MATLAB output](https://github.com/introlab/rtabmap/wiki/Example-Malaga2009-Parking-6L)

#### UdeS 1Hz
 * [Dataset link](https://introlab.3it.usherbrooke.ca/mediawiki-introlab/index.php/RTAB-Map#Data_sets)
 * [Ground truth](https://introlab.3it.usherbrooke.ca/mediawiki-introlab/images/1/16/UdeS_1Hz.png)
 
`
$ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 150 -Kp/IncrementalFlann false -Rtabmap/TimeThr 700 UdeS_1Hz
`

## Videos for all datasets
Visit [RTAB-Map on IntRoLab website](https://introlab.3it.usherbrooke.ca/mediawiki-introlab/index.php?title=RTAB-Map&setlang=en).