Error in nuscenes test data

[Morizb]

Hi, when I use the following command for evaluation:

Evaluation

sh . /tools/dist_test.sh . /configs/MSMDFusion_nusc_voxel_LC.py $ckpt_path$ 2 --eval bbox
The results obtained, are consistent with epoch_6 on the validation set. However, I noticed that the results you gave for the test set have a big improvement compared to the validation set.

When I modify the read test set directly and run it again, the following two errors are reported:
KeyError: Caught KeyError in DataLoader worker process 0.
KeyError: 'num_lidar_pts'

I tried searching for "How to evaluate on Nuscenes test set" but did not find a suitable solution, can you help me with this?

[Morizb]

Hi, when I use the following command for evaluation:

Evaluation

sh . /tools/dist_test.sh . /configs/MSMDFusion_nusc_voxel_LC.py ckptpath 2 --eval bbox The results obtained, are consistent with epoch_6 on the validation set. However, I noticed that the results you gave for the test set have a big improvement compared to the validation set.

When I modify the read test set directly and run it again, the following two errors are reported: KeyError: Caught KeyError in DataLoader worker process 0. KeyError: 'num_lidar_pts'

I tried searching for "How to evaluate on Nuscenes test set" but did not find a suitable solution, can you help me with this?

My problem seems to be in line with this one: #9 (comment)
The error reported is:
FileNotFoundError: [Errno 2] No such file or directory: '...... /MSMDFusion/data/nuscenes/samples/FOREGROUND_MIXED_6NN_WITH_DEPTH/n015-2018-07-25-16-15-50+0800__LIDAR_TOP__1532506680948053.pcd. bin.pkl.npy'

[SxJyJay]

However, I noticed that the results you gave for the test set have a big improvement compared to the validation set.

Generally, results submitted to the leaderboard to get test set predictions are trained on train+val sets, therefore the results on the test set are higher than val set.

Hi, when I use the following command for evaluation:

Evaluation

sh . /tools/dist_test.sh . /configs/MSMDFusion_nusc_voxel_LC.py ckptpath 2 --eval bbox The results obtained, are consistent with epoch_6 on the validation set. However, I noticed that the results you gave for the test set have a big improvement compared to the validation set.
When I modify the read test set directly and run it again, the following two errors are reported: KeyError: Caught KeyError in DataLoader worker process 0. KeyError: 'num_lidar_pts'
I tried searching for "How to evaluate on Nuscenes test set" but did not find a suitable solution, can you help me with this?

My problem seems to be in line with this one: #9 (comment) The error reported is: FileNotFoundError: [Errno 2] No such file or directory: '...... /MSMDFusion/data/nuscenes/samples/FOREGROUND_MIXED_6NN_WITH_DEPTH/n015-2018-07-25-16-15-50+0800__LIDAR_TOP__1532506680948053.pcd. bin.pkl.npy'

It seems that we don't successfully upload the test set virtual points in the given link. I will check them.

[Morizb]

Thank you very much for your fast response!

[sun-yue2002]

Hi, when I use the following command for evaluation:

Evaluation

sh . /tools/dist_test.sh . /configs/MSMDFusion_nusc_voxel_LC.py ckptpath 2 --eval bbox The results obtained, are consistent with epoch_6 on the validation set. However, I noticed that the results you gave for the test set have a big improvement compared to the validation set.

When I modify the read test set directly and run it again, the following two errors are reported: KeyError: Caught KeyError in DataLoader worker process 0. KeyError: 'num_lidar_pts'

I tried searching for "How to evaluate on Nuscenes test set" but did not find a suitable solution, can you help me with this?

Hi! I met the same error. Have you find out the reason of this issue and resolution on how to deal with it?

[SxJyJay]

https://github.com/SxJyJay/MDU_preprocess

You can refer to this repo and generate the required virtual points for nuScenes test set using the very first command

python virtual_depth_mapping_multi_proj.py --info_path data/nuScenes/infos_train_10sweeps_withvelo_filter_True.pkl  MODEL.WEIGHTS centernet2_checkpoint.pth

[sun-yue2002]

https://github.com/SxJyJay/MDU_preprocess

You can refer to this repo and generate the required virtual points for nuScenes test set using the very first command

python virtual_depth_mapping_multi_proj.py --info_path data/nuScenes/infos_train_10sweeps_withvelo_filter_True.pkl  MODEL.WEIGHTS centernet2_checkpoint.pth

Thanks a lot!!!!

[Morizb]

@SxJyJay Hi, I am testing on the test set （nuscenes_infos_test.pkl ）and have followed the above code to generate virtual points on the test set, which solves the previous problem. But it reports a new error:
AssertionError: Error: You are trying to evaluate on the test set but you do not have the annotations!

The command I run when testing is:
./tools/dist_test.sh
sh ./tools/dist_test.sh ./configs/MSMDFusion_nusc_voxel_LC.py /work_dirs/20240429/epoch_6.pth 2 --eval bbox --cfg-options'test_evaluator.jsonfile_prefix=/work_dirs/20240429/results_eval'

Do you know what it is, please? How can I test it on a test set?