Having trouble reproducing the results in the paper

[HITKJ]

Thank you very much for your excellent work. I have encountered some issues while trying to reproduce your work.

• Did your final model use a single-frame pre-trained model? If so, could you please provide details about the pre-training and fine-tuning processes?

• I have conducted multiple tests on the same model that you provided using the same configuration and random seed, but the results are inconsistent each time. In other words, under the same model and configuration, there is significant fluctuation in the test results. Is this phenomenon normal, and how can I resolve it? Is it correct that what you report in your paper is the average of three tests conducted with different random seeds?

• Below are the results of three tests with the same configuration.

[anthonyhu]

Thanks for your kind words! To answer your questions:

1. Yes we pretrain the image encoder using the config mile/configs/one_frame.yml.
2. The stochasticity of evaluation runs is a known issue of CARLA. Depending on the random seed, the set of dynamic agents spawned is different, explaining the differences in driving score. The reported metrics in the paper use 3 different random seeds.

[HITKJ]

Thanks for your kind words! To answer your questions:

1. Yes we pretrain the image encoder using the config mile/configs/one_frame.yml.
2. The stochasticity of evaluation runs is a known issue of CARLA. Depending on the random seed, the set of dynamic agents spawned is different, explaining the differences in driving score. The reported metrics in the paper use 3 different random seeds.

Thanks for your response. I'm still a little confused.

• Is the pre-trained part referring to the network that responsible for mapping the image, speed, and route map to a 512-dimensional observation vector?
• Do we need to adjust the learning rate separately for the pretrained part when training the temporal model?
• Is it correct that a batch size of 64 was used with 50k iterations for single-frame pretraining, followed by an additional 50k iterations for the model with sequences transition enabled?
• I got different driving score with the same random seeds. Is it exactly the known issue of CARLA? Could you share the seeds used for the results that reported in your paper. I wil really apreciate it .

[anthonyhu]

• Yes that's correct.
• There is no need to adjust the learning rate, the same learning rate was used for the whole network (including finetuning the encoder).
• Yes.
• Different driving scores with the same random seed is also a known issue of CARLA, which is not exactly deterministic. The beginning of the run can look exactly the same, but some random process (appearing with some probability) triggers a change in the simulator which changes the outcome of the run. I think I used seed 0, 1, 2.