RFNet: Real-time Fusion Network for RGB-D Semantic Segmentation Incorporating Unexpected Obstacle Detection of Road-driving Images

This repository is a Pytorch implementation for

Sun, Lei, et al. "Real-time Fusion Network for RGB-D Semantic Segmentation Incorporating Unexpected Obstacle Detection for Road-driving Images." arXiv preprint arXiv:2002.10570 (2020).

If you want to use this code in your research, please cite the paper.

Requirement

Python 3.6  
Pytorch 1.1  
Torchvision 0.3  
Opencv 3.3.1

Datasets

Get dataset from Cityscapes, and from Lost and Found.

If you want to use multi-dataset training, mix two datasets and the directory structure should be like this:

├─disparity
│  ├─test
│  │  ├─berlin
│  │  ├─bielefeld
│  │  ├─bonn
│  │  ├─...
│  │  └─munich
│  ├─train
│  │  ├─01_Hanns_Klemm_Str_45
│  │  ├─03_Hanns_Klemm_Str_19
│  │  ├─...
│  │  └─zurich
│  └─val
│      ├─02_Hanns_Klemm_Str_44
│      ├─04_Maurener_Weg_8
│      ├─05_Schafgasse_1
│      ├─...
│      └─munster
├─gtFine
│  ├─train
│  │  ├─01_Hanns_Klemm_Str_45
│  │  ├─03_Hanns_Klemm_Str_19
│  │  ├─...
│  │  └─zurich
│  └─val
│      ├─02_Hanns_Klemm_Str_44
│      ├─04_Maurener_Weg_8
│      ├─05_Schafgasse_1
│      ├─...
│      └─munster
└─leftImg8bit
    ├─test
    │  ├─berlin
    │  ├─bielefeld
    │  ├─bonn
    │  ├─...
    │  └─munich
    ├─train
    │  ├─01_Hanns_Klemm_Str_45
    │  ├─03_Hanns_Klemm_Str_19
    │  ├─...
    │  └─zurich
    └─val
        ├─02_Hanns_Klemm_Str_44
        ├─04_Maurener_Weg_8
        ├─05_Schafgasse_1
        ├─...
        └─munster

Pretrained weights

BaiduNetdisk

Download weights for Cityscapes, password: 4lts
Download weights for Multi-dataset: Cityscapes and Lost and Found, password: t1mh

Google Drive

Download weights for Cityscapes
Download weights for Multi-dataset: Cityscapes and Lost and Found

Training

Edit path to your dataset in mypath.py.

python train.py --depth --lr 1e-4 --weight-decay 2.5e-5 --workers 4 --epochs 200 --batch-size 8 --val-batch-size 3 --gpu-ids 0 --checkname test --eval-interval 2 --dataset citylostfound --loss-type ce --use-balanced-weights

Evaluation

python eval.py --dataset citylostfound --weight-path your/path/to/weight/ --depth

Note: The code is partially based Swiftnet and DeepLab v3+.