Benchmark

Introduction

This page explains how to reproduce results (like Precision/Recall curve) shown in the paper "Appearance-Based Loop Closure Detection for Online Large-Scale and Long-Term Operation".

To to so, we use the tool called "rtabmap-console". The datasets used can be downloaded following the links on RTAB-Map's page on IntRoLab as well as the ground truths used. For datasets with right/left images like NewCollege and CityCentre, the tool "rtabmap-imagesJoiner" can be used to merge left and right images into single image. RTAB-Map should be installed with NON-FREE support (SURF and SIFT)!

Processing a dataset with rtabmap-console

Let's take for example the Lip6Indoor dataset. For other datasets, look at this section below. Download the indoor dataset on the ENSTA page. Since RTAB-Map's default parameters changed over time for RGB-D mapping/scanning, we must specify some loop closure parameters manually. A exhaustive list of all parameters can be found in "Parameters.h".

Here is a mapping between parameters in the paper and those in the code:

Paper	C++ Code
`T_STM=30`	`Mem/STMSize=30`
`T_similarity=20%`	`Mem/RehearsalSimilarity=0.2`
`T_recent=20%`	`Mem/RecentWmRatio=0.2`
SURF dimension = 64	`SURF/Extended=false`
SURF `T_maxFeatures=400`	`Kp/WordsPerImage=400`
SURF `T_bad=0.25`	`Kp/BadSignRatio=false`

Variable thresholds depending on the datasets:

Paper	C++ Code
`T_time` (s)	`Rtabmap/TimeThr` (ms)
SURF `T_response`	`SURF/HessianThreshold`

The other parameters set:

Rtabmap/StatisticLogged=true: we need the logged files LogF.txt and LogI.txt for MATLAB. These files are written in the working memory of RTAB-Map (default "~/Documents/RTAB-Map").
Rtabmap/StatisticLoggedHeaders=false: Don't print column headers in log files.
Rtabmap/PublishStats=false: it is only useful in GUI mode.
Mem/RehearsalIdUpdatedToNewOne=true: When merging images, keep the latest ID. The ground truths are based on the latest ID for locations merged together during Weight Update.
Mem/BadSignaturesIgnored=true: In the paper, we ignore bad signatures.
Rtabmap/LoopRatio=0.0: The loop ratio is used to reject decreasing hypotheses, which means that we are moving to another area. In the paper it was not used, loop closures were accepted in the area of the ground truth even if the images are very different.
Bayes/FullPredictionUpdate=true: In the papers, the Bayes prediction was regenerated at each frame.
Kp/TfIdfLikelihoodUsed=false: In the papers, tf-idf weighting wasn't used.
Mem/TransferSortingByWeightId=true: (Version>=0.8.6) In the papers, locations selected for Transfer were sorted by weight/ID, not by weight/Age/ID.
Kp/IncrementalFlann=false: In the original paper, the boost performance of incremental FLANN was not implemented.

The command to process the Lip6Indoor dataset: $ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 10 -Kp/IncrementalFlann false -Rtabmap/TimeThr 700 -Kp/DetectorStrategy 0 Lip6IndoorDataSet/Images

Recision/Recall results with MATLAB

Open MATLAB and add to path the directory of the file ShowLog.m from RTAB-Map: https://github.com/introlab/rtabmap/tree/master/archive/2010-LoopClosure/ShowLogs. Then execute the showlogs method. The first path is where the LogF.txt dans LogI.txt files were saved and the second path is the ground truth. The ground truth is simply a matrix represented by a image file for convenience where white pixels are loop closures between the corresponding indexes of images and the black pixels are no loop closures. For this example, the ground truth can be downloaded here:

MATLAB:

>> showlogs('~/.', '~/Lip6Indoor.png');

Octave 3.8 (copy files from https://github.com/introlab/rtabmap/tree/master/archive/2010-LoopClosure/ShowLogs in the current directory where LogF.txt and LogI.txt are):
```
 $ octave
 octave:1> showlogs('.', 'Lip6Indoor.png');
```

Figure 9 is the Precision/Recall curve and you should see an output like this:

```
--- getPrecisionRecall ---
Loading GroundTruth '~/Documents/RTAB-Map/Lip6Indoor.png' ...

GT_total_positives =

   226

False positive! id=149 with old=86 (p=0.11314)
False positive accepted! id=149 with old=86 (p=0.11314)
False positive! id=150 with old=87 (p=0.11256)
False positive accepted! id=150 with old=87 (p=0.11256)
False positive! id=148 with old=85 (p=0.10664)
False positive! id=140 with old=102 (p=0.1035)
False positive! id=93 with old=22 (p=0.10234)
False positive! id=55 with old=9 (p=0.09511)
[...]
Recall max (Precision=100%) = 98.2301% (p=0.11422), accepted=222
Recall max accepted (Precision=100%) = 98.2301% (p=0.11422), accepted=222
ignored = 0
```

Note that the lines beginning with "False positive!" may contain true positives if the ground truth is too narrow on the accepted loop closures. Sometimes, we have to manually add white pixels to the ground truth file if a loop closure is found with an image just next to the defined ground truth one.

[[doc/Tutorials/Benchmark/recall_lip6indoor.png]]

Using RTAB-Map's GUI

From 0.11.12, rtabmap can be also used with arguments for convenience. Just add these parameters: $ rtabmap [...parameters below...] -RGBD/Enabled false -Rtabmap/DetectionRate 0 -Rtabmap/ImageBufferSize 0 -Rtabmap/PublishStats true then open Preferences->Source, select "RGB" as source, set the input source rate, select Images under RGB group box and select the corresponding directory of images.

Commands for all datasets

New College

Dataset link
Ground truth
Note: You can use rtabmap-imagesJoiner tool to merge left and right images.

$ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 1000 -Kp/IncrementalFlann false -Rtabmap/TimeThr 1400 -Kp/DetectorStrategy 0 NewCollege

Example rtabmap-console and MATLAB output

City Centre

Dataset link
Ground truth
Note: You can use rtabmap-imagesJoiner tool to merge left and right images.

$ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 1000 -Kp/IncrementalFlann false -Rtabmap/TimeThr 1400 -Kp/DetectorStrategy 0 CityCentre

Example rtabmap-console and MATLAB output

Lip6 Indoor

$ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 10 -Kp/IncrementalFlann false -Rtabmap/TimeThr 700 -Kp/DetectorStrategy 0 Lip6Indoor

Lip6 Outdoor

Dataset link
Ground truth
Note: Keeping only even images and remove the last one.

$ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 10 -Kp/IncrementalFlann false -Rtabmap/TimeThr 1400 -Kp/DetectorStrategy 0 Lip6Outdoor

70 km

$ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 1000 -Kp/IncrementalFlann false -Rtabmap/TimeThr 700 -Kp/DetectorStrategy 0 70km

New College Omni

$ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.35 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 1000 -Kp/IncrementalFlann false -Rtabmap/TimeThr 700 -Kp/DetectorStrategy 0 NewCollegeOmni

Crowded Canteen

$ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.35 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 75 -Kp/IncrementalFlann false -Rtabmap/TimeThr 350 -Kp/DetectorStrategy 0 CrowdedCanteen

Bicocca Indoor 2009-02-25b

$ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 50 -Kp/IncrementalFlann false -Rtabmap/TimeThr 700 -Kp/DetectorStrategy 0 BicoccaIndoor-2009-02-25b

Bovisa Outdoor 2008-10-04

$ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 50 -Kp/IncrementalFlann false -Rtabmap/TimeThr 700 -Kp/DetectorStrategy 0 BovisaOutdoor-2008-10-04

Bovisa Mixed 2008-10-06

$ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 50 -Kp/IncrementalFlann false -Rtabmap/TimeThr 700 -Kp/DetectorStrategy 0 BovisaMixed-2008-10-06

Malaga2009 Campus 2L

Dataset link
Ground truth
Note: Left camera images are used. There are 5221 images at 7.5Hz in this dataset. I used 1 image of 8 to have ~1Hz, so 653 images are used here.

$ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 1000 -Kp/IncrementalFlann false -Rtabmap/TimeThr 700 -Kp/DetectorStrategy 0 malaga2009_campus_2L

Example rtabmap-console and MATLAB output

Malaga2009 Parking 6L

Dataset link
Ground truth
Note: Left camera images are used. There are 3475 images at 7.5Hz in this dataset. I used 1 image of 8 to have ~1Hz, so 435 images are used here.

$ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 1000 -Kp/IncrementalFlann false -Rtabmap/TimeThr 700 -Kp/DetectorStrategy 0 malaga2009_parking_6L

Example rtabmap-console and MATLAB output

UdeS 1Hz

$ ./rtabmap-console -Rtabmap/StatisticLogged true -Rtabmap/StatisticLoggedHeaders false -Rtabmap/PublishStats false -Mem/RehearsalIdUpdatedToNewOne true -Mem/BadSignaturesIgnored true -Rtabmap/LoopRatio 0.0 -Bayes/FullPredictionUpdate true -Kp/TfIdfLikelihoodUsed false -Mem/STMSize 30 -Mem/RehearsalSimilarity 0.2 -Mem/RecentWmRatio 0.2 -SURF/Extended false -Kp/WordsPerImage 400 -Kp/BadSignRatio 0.25 -SURF/HessianThreshold 150 -Kp/IncrementalFlann false -Rtabmap/TimeThr 700 -Kp/DetectorStrategy 0 UdeS_1Hz

Videos for all datasets

Visit RTAB-Map on IntRoLab website.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly