README.md

[3DV24] Cas6D: Learning to Estimate 6DoF Pose from Limited Data: A Few-Shot,
Generalizable Approach using RGB Images

[Paper: Arxiv]

Panwang Pan^1,*, Zhiwen Fan^2,*, Brandon Y. Feng^3,*, Peihao Wang², Chenxin Li⁴, Zhangyang Wang²

¹ByteDance   ²The University of Texas at Austin   ³MIT   ⁴The Chinese University of Hong Kong   ^*denotes equal contribution

We present a new cascade framework named Cas6D for few-shot 6DoF pose estimation that is generalizable and uses only RGB images.

Training

1. Download processed co3d data (co3d.tar.gz), google scanned objects data (google_scanned_objects.tar.gz) and ShapeNet renderings (shapenet.tar.gz) at here.
2. Download COCO 2017 training set.
3. Organize files like

Gen6D
|-- data
    |-- GenMOP
        |-- chair 
            ...
    |-- LINEMOD
        |-- cat 
            ...
    |-- shapenet
        |-- shapenet_cache
        |-- shapenet_render
        |-- shapenet_render_v1.pkl
    |-- co3d_256_512
        |-- apple
            ...
    |-- google_scanned_objects
        |-- 06K3jXvzqIM
            ...
    |-- coco
        |-- train2017
4. Train the detector
```shell
python3 train_model.py --cfg configs/detector/detector_train.yaml

5. Train the selector

python3 train_model.py --cfg configs/selector/selector_train.yaml

6. Prepare the validation data for training refiner

python3 prepare.py --action gen_val_set \
                  --estimator_cfg configs/gen6d_train.yaml \
                  --que_database linemod/cat \
                  --que_split linemod_val \
                  --ref_database linemod/cat \
                  --ref_split linemod_val

python3 prepare.py --action gen_val_set \
                  --estimator_cfg configs/gen6d_train.yaml \
                  --que_database genmop/tformer-test \
                  --que_split all \
                  --ref_database genmop/tformer-ref \
                  --ref_split all 

This command will generate the information in the data/val, which will be used in producing validation data for the refiner. 7. Train the refiner

python3 train_model.py --cfg configs/refiner/refiner_train.yaml

Evaluate all components together.

# Evaluate on the object TFormer from the GenMOP/LINEMOD dataset
python3 eval.py --cfg configs/cas6d_train.yaml 

Acknowledgement

We would like to thank Gen6D authors for open-sourcing their implementations.

Citation

If you find this repo is helpful, please consider citing:

@inproceedings{pan2024learning,
  title={Learning to estimate 6dof pose from limited data: A few-shot, generalizable approach using rgb images},
  author={Pan, Panwang and Fan, Zhiwen and Feng, Brandon Y and Wang, Peihao and Li, Chenxin and Wang, Zhangyang},
  booktitle={2024 International Conference on 3D Vision (3DV)},
  pages={1059--1071},
  year={2024},
  organization={IEEE}
}