LION: Linear Group RNN for 3D Object Detection in Point Clouds

Zhe Liu ^1,* , Jinghua Hou ^1,* , Xinyu Wang ^1,* , Xiaoqing Ye ³, Jingdong Wang ³, Hengshuang Zhao ², Xiang Bai ^1,✉
1 Huazhong University of Science and Technology, ² The University of Hong Kong, ³ Baidu Inc.
* Equal contribution, ✉ Corresponding author.

Project Page | NeurIPS 2024

🔥 Highlights

• Strong performance. LION achieves state-of-the-art performance on Waymo, nuScenes, Argoverse V2, and ONCE datasets. 💪

• Strong generalization. LION can support almost all linear RNN operators including Mamba, RWKV, RetNet, xLSTM, and TTT. Anyone is welcome to verify more linear RNN operators. 😀

• More friendly. LION can train all models on less 24G GPU memory (i.e., RTX 3090, RTX4090, V100 and A100 are enough to train our LION). 😀

News

• 2024.09.26: LION has been accepted by NeurIPS 2024. 🎉
• 2024.07.25: LION paper released. 🔥
• 2024.07.02: Our new works OPEN and SEED have been accepted by ECCV 2024. 🎉

Results

• Waymo Val Set~(100%)

Model	mAP/mAPH_L1	mAP/mAPH_L2	Vec_L1	Vec_L2	Ped_L1	Ped_L2	Cyc_L1	Cyc_L2	Config
LION-RetNet	80.9/78.8	74.6/72.7	79.0/78.5	70.6/70.2	84.6/80.0	77.2/72.8	79.0/78.0	76.1/75.1	config
LION-RWKV	81.0/79.0	74.7/72.8	79.7/79.3	71.3/71.0	84.6/80.0	77.1/72.7	78.7/77.7	75.8/74.8	config
LION-Mamba	81.4/79.4	75.1/73.2	79.5/79.1	71.1/70.7	84.9/80.4	77.5/73.2	79.7/78.7	76.7/75.8	config
LION-Mamba-L	82.1/80.1	75.9/74.0	80.3/79.9	72.0/71.6	85.8/81.4	78.5/74.3	80.1/79.0	77.2/76.2	config

Note: You could reduce the training epochs from 24 to 12~(the performance gap is within 1 mAP/mAPH) or reduce the 100% training to 20% training sets.

• nuScenes

Model	Split	Epoch	CBGS	NDS	mAP	Config	Download (Baidu Pan)	Download (Google Drive)
LION-RetNet	Val	36	False	71.9	67.3	config	nus_retnet.pth (ksmp)	nus_retnet.pth
LION-RWKV	Val	36	False	71.7	66.8	config
LION-Mamba	Val	36	False	72.1	68.0	config	nus_mamba.pth (2tvc)	nus_mamba.pth
LION-Mamba	Val	48	False	72.3	68.2	config
LION-Mamba	Test	36	False	73.9	69.8

Note: Our model on nuScenes does not use CBGS for training more time and without any test-time augmentation or model ensembling! For obtaining more stable and better performance, you could try to train more time~(e.g., 48 epochs)

• Argoverse V2 Val Set

Model	mAP	Config	Download (Baidu Pan)	Download (Google Drive)
LION-RetNet	40.7	config	argov2_retnet.pth (yghm)	argov2_retnet.pth
LION-RWKV	41.1	config	argov2_rwkv.pth (cr4e)	argov2_rwkv.pth
LION-Mamba	41.5	config	argov2_mamba.pth (k63i)	argov2_mamba.pth

• ONCE Val Set

Model	Vehicle	Pedestrian	Cyclist	mAP	Config	Download
LION-RetNet	78.1	52.4	68.3	66.3	config
LION-RWKV	78.3	50.6	68.4	65.8	config
LION-Mamba	78.2	53.2	68.5	66.6	config

Quick Validation

• We provide some examples of LION models on KITTI dataset for quick validation of any Linear RNN operators.
• Here, we provide the results of moderate difficulty for LION with RetNet, RWKV, Mamba, xLSTM, and TTT.
• Anyone is welcome to verify more linear RNN operators. 😀

Model	Car	Pedestrian	Cyclist	Config	Download
LION-TTT	78.0	58.6	69.6	config
LION-xLSTM	77.9	59.3	67.4	config
LION-RetNet	77.9	60.2	69.6	config
LION-Mamba	78.3	60.2	68.6	config
LION-RWKV	78.3	62.2	71.2	config

Installation

Please refer to INSTALL.md for the installation of LION codebase.

Getting Started

We provide all training&evaluation scripts for training our LION, please refer to tools/

• Train all models of LION on nuScenes

bash run_train_lion_for_nus.sh

• Train all models of LION on Waymo

bash run_train_lion_for_waymo.sh

• Train all models of LION on Argoverse V2

bash run_train_lion_for_argov2.sh

• Train all models of LION on ONCE

bash run_train_lion_for_once.sh

• Train all models of LION on KITTI

bash run_train_lion_for_kitti.sh

For more details about LION, please refer to GETTING_STARTED.md to learn more usage about LION.

TODO

• Release the paper.
• Release the code of LION on the Waymo.
• Release the code of LION on the nuScenes.
• Release the code of LION on the Argoverse V2.
• Release the code of LION on the ONCE.
• Release the code of LION on the KITTI.
• Release some important checkpoints of LION (nuScenes and Argoverse v2).
• Support more linear RNNs.

Citation

@article{liu2024lion,
  title={LION: Linear Group RNN for 3D Object Detection in Point Clouds},
  author={Zhe Liu, Jinghua Hou, Xingyu Wang, Xiaoqing Ye, Jingdong Wang, Hengshuang Zhao, Xiang Bai},
  journal={Advances in Neural Information Processing Systems},
  year={2024}
  }

Acknowledgements

We thank these great works and open-source repositories: OpenPCDet, DSVT, FlatFormer, HEDNet, Mamba, RWKV, Vision-RWKV, RMT, xLSTM, TTT, and flash-linear-attention.