ROS package for semantic point cloud segmentation

ROS package for the semantic segmentation of stereo point clouds. Modified and forked from: https://github.com/brian-h-wang/ldls_ros. Currently, this code only supports offline segmentation of point clouds. It reads time synchronized point clouds and Mask-RCNN segmentation masks of reference images from an input rosbag, and then writes segmented point clouds to an output rosbag.

Subscribes to the topics:

1. /object_detection/results : contains Mask-RCNN person instance segmentation masks, and bounding boxes in pixels.
2. /object_detection/pc2 : contains the original colored point cloud, published in the color optical frame.

Additional repositories needed (for details follow the individual READMEs included):

1. https://github.com/timeamagyar/maskrcnn_finetuning -> performs transfer learning based on a custom 2D data set, annotated with the coco-annotator tool
2. https://github.com/timeamagyar/maskrcnn_ros -> performs 2D object detection and semantic segmentation
3. https://github.com/timeamagyar/PCDTrimmer -> Manipulates .pcd ground truth data collected with the Hitachi segmentation tool. Needed for the performance evaluation of the adapted LDLS algorithm.
4. https://github.com/timeamagyar/realsense-ros -> necessary and project specific modifications are added to the rs_from_file.launch file

External repositories used to annote 2D/3D data:

1. https://github.com/jsbroks/coco-annotator
2. https://github.com/Hitachi-Automotive-And-Industry-Lab/semantic-segmentation-editor

Installation

Requires/tested on:

1. Ubuntu 18.4
2. ROS Melodic
3. Nvidia GPU
4. Python 2.7

Installing dependencies with pip is recommended. A requirements.txt file is included in the project folder. To install the required dependencies inside a virtual environment run:

python -m virtualenv <name_of_the_virtualenv>
source <name_of_the_virtualenv>/bin/activate
pip install -r requirements.txt

Tested on an Nvidia GeForce GTX 1660 Ti.

Usage

System set up

# source python virtual env 
source <name_of_the_virtualenv>/bin/activate

# make sure to have the following on your python path
<path_to_catkin_ws>/catkin_ws/devel/lib/python2.7/dist-packages:/opt/ros/melodic/lib/python2.7/dist-packages

# finally
export NUMBAPRO_NVVM=/usr/local/cuda-10.0/nvvm/lib64/libnvvm.so -> tested for cuda 10
export NUMBAPRO_LIBDEVICE=/usr/local/cuda-10.0/nvvm/libdevice/ -> tested for cuda 10

LDLS Result Generation

To generate LDLS results run:

python write_ldls_results.py -intrinsics_bag <intrinsics_bag>  -data_bag <input_bag> -gt_path <path_of_gt_annotations>

Note: intrinsics bags and input bags are generated by the ros object detection package. See the project README for reference. Ground truth data is generated by the Hitachi segmentation tool and parsed by the pcd trimmer tool. For details visit: https://github.com/timeamagyar/PCDTrimmer.

Example:

python write_ldls_bag.py -intrinsics_bag /mnt/nvme/bags/20191213_112223/rs_20191213_112223_intrinsics.bag -data_bag /mnt/nvme/bags/20191213_112223/rs_20191213_112223.bag -gt_path /mnt/nvme/bags/20191213_112223/gt

Rosbag with LDLS results will be written to <name_of_input_bag>_ldls.bag

Please see the msg/Segmentation.msg file for specification of the output Segmentation message type, which includes labeled points for detected object instances.

LDLS Result Evaluation

To evaluate LDLS results run:

python evaluate_ldls_results.py -predictions <segmentation_results_bag> -gt_path <path_to_gt_annotations> 

Example:

python evaluate_ldls_results.py -predictions /mnt/nvme/bags/20191213_112223/rs_20191213_112223_segmentation.bag -gt_path /mnt/nvme/bags/20191213_112223/gt/

To visualize a specific file with the ROS pcl-viewer console tool:

python evaluate_ldls_results.py -predictions /mnt/nvme/bags/20191213_112223/rs_20191213_112223_segmentation.bag -gt_path /mnt/nvme/bags/20191213_112223/gt/ -pcd_path /mnt/nvme/bags/20191213_112223/pcd/340_1576232556.412253618.pcd -launch_trimmer True

The above command will generate a fn_tp_fp.pcd file in the project root folder. To visualize true positives, true negatives and false positives for a specific frame run:

pcl_viewer -multiview 1 fn_tp_fp.pcd

Note that segmentation result bags can be extracted from the <name_of_input_bag>_ldls.bag files as follows:

rosbag play <name_of_input_bag>_ldls.bag
rosbag record --buffsize=0 /ldls/segmentation --output-name=<file_name>_segmentation.bag

Example:

rosbag play rs_20191213_112223_ldls.bag
rosbag record --buffsize=0 /ldls/segmentation --output-name=rs_20191213_112223_segmentation.bag