| ⚡ Complete replication package coming soon! ⚡ |
|---|
Replication Package for the paper: "Two is Better Than One: Digital Siblings to Improve Autonomous Driving Testing"
The repository contains the source code to run the experiments and analyze the results. The README only provides instruction on how to replicate the experiments by using pretrained models. The source code of the simulators (i.e., DonkeyCar and Udacity) as well as the datasets to train the models (both cyclegan and dave2) will be provided on request.
As for the BeamNG simulator we used the version 0.23.5.1.1288. A free version of the BeamNG simulator for research purposes can be obtained by registering at beamng.tech and following the instructions provided by BeamNG. If the above version is not available you can have a look at the code from the SBST CPS Tool competition which uses a recent version of BeamNG. The code in the package code_pipeline and envs/beamng have to be updated to reflect the changes of the beamngpy library that communicates with the simulator. Such library has to be updated in the requirements.txt file.
The link to download the pretrained models as well as the results of the experiment can be downloaded here. This needs to be done to execute the commands after step 0.
-
Install anaconda for your system;
-
Create the environment:
conda create -n myenv python=3.8 -
Activate the environment:
conda activate myenv -
Install the requirements:
pip install -r requirements.txt
Copy the logs folder in the archive downloaded before and place it at the root of the folder containing this repository.
Type:
./compute_correlations_probabilities.sh mapelites
to compute the correlations using the model trained with simulation data and type:
./compute_correlations_probabilities.sh cyclegan
to compute the correlations using the model trained using the cyclegan translation.
Type:
./compute_correlations_quality_metrics.sh mapelites
to compute the correlations using the model trained with simulation data and type:
./compute_correlations_quality_metrics.sh cyclegan
to compute the correlations using the model trained using the cyclegan translation.
The command to type for Udacity is:
python run_mapelites.py --env-name udacity \
--udacity-exe-path <path/to/udacity-simulator/executable> \
--agent-type supervised \
--model-path <path/to/mixed-dave2.h5> \
--min-angle 100 \
--max-angle 300 \
--population-size 20 \
--iteration-runtime 150 \
--mutation-extent 6 \
--feature-combination turns_count-curvature \
--num-runs 5
The simulator executable as well as the dave2 model (i.e., mixed-dave2.h5)
are provided in the previously downloaded archive (respectively under simulators and models).
The command above runs the MapElites algorithm in the Udacity simulator using the
dave2 model trained with simulated data for 5 runs, each of 150 iterations.
The command creates a folder for each run in the logs folder named mapelites_udacity_<date_str>_i where i is the index of the run.
Moreover it creates a folder with all the individuals executed in each run, namely mapelites_udacity_<date_str>_all.
The MapElites algorithm needs to be executed for the same number of runs and iterations using the BeamNG simulator. The command is the following:
python run_mapelites.py --env-name beamng \
--beamng-home <path/to/beamng-simulator/BeamNG.drive-0.23.5.1.12888> \
--beamng-user <path/to/beamng-simulator> \
--agent-type supervised \
--model-path <path/to/mixed-dave2.h5> \
--min-angle 100 \
--max-angle 300 \
--population-size 20 \
--iteration-runtime 150 \
--mutation-extent 6 \
--feature-combination turns_count-curvature \
--num-runs 5
In this case we need to use the pretrained cyclegan model to translate the images coming from the simulator to pseudo-real images. The command is similar to the one above except fo the cyclegan related parameters:
python run_mapelites.py --env-name udacity \
--udacity-exe-path <path/to/udacity-simulator/executable> \
--agent-type supervised \
--model-path <path/to/mixed-fake-dave2.h5> \
--min-angle 100 \
--max-angle 300 \
--population-size 20 \
--iteration-runtime 150 \
--mutation-extent 6 \
--feature-combination turns_count-curvature \
--num-runs 5 \
--cyclegan-experiment-name udacity \
--gpu-ids 0 \
--cyclegan-checkpoints-dir cyclegan/checkpoints \
--cyclegan-epoch 35
This is the command for Udacity. Similarly for BeamNG the
--cyclegan-experiment-name parameter is beamng and the
--cyclegan-epoch parameter is 35. For Donkey the
--cyclegan-experiment-name parameter is donkey and the
--cyclegan-epoch parameter is 40. In the commands below we only provide examples for the model trained using simulated images;
cyclegan related parameters need to be provided when the model under test is mixed-fake-dave2.h5 as shown above.
The command above needs to be executed on a machine with GPU as the inference time needs to match the
frame rate of the simulator. The checkpoints folder of the cyclegan should have already been downloaded;
the archive contains a folder named cyclegan with a sub-folder named checkpoints. Such folder needs to be placed
in the cyclegan folder at the root of the folder containing this repository.
python run_individual_migration.py --env-name udacity \
--udacity-exe-path <path/to/udacity-simulator/executable> \
--agent-type supervised \
--model-path <path/to/mixed-dave2.h5> \
--feature-combination turns_count-curvature \
--filepath mapelites_beamng_<date_str>_all
This command will create the folder mapelites_migration_udacity_<date_str>.
It is better to rename such folder in mapelites_migration_udacity_beamng_search.
It should be read in this way: migration on Udacity of individuals obtained by running the search (i.e., MapElites) on BeamNG. The command is for simulated images; for pseudo-real images the three options showed in Section 1.2. should be added.
python run_individual_migration.py --env-name beamng \
--beamng-home <path/to/beamng-simulator/BeamNG.drive-0.23.5.1.12888> \
--beamng-user <path/to/beamng-simulator> \
--agent-type supervised \
--model-path <path/to/mixed-dave2.h5> \
--feature-combination turns_count-curvature \
--filepath mapelites_udacity_<date_str>_all
This command will create the folder mapelites_migration_beamng_<date_str>.
It is better to rename such folder in mapelites_migration_beamng_udacity_search.
It should be read in this way: migration on BeamNG of individuals obtained by running the search (i.e., MapElites) on Udacity. The command is for simulated images; for pseudo-real images the three options showed in Section 1.2. should be added.
python run_individual_migration.py --env-name donkey \
--donkey-exe-path <path/to/donkey-simulator/executable> \
--agent-type supervised \
--model-path <path/to/mixed-dave2.h5> \
--feature-combination turns_count-curvature \
--filepath mapelites_beamng_<date_str>_all
This command will create the folder mapelites_migration_donkey_<date_str>.
It is better to rename such folder in mapelites_migration_donkey_beamng_search.
It should be read in this way: migration on Donkey of individuals obtained by running the search (i.e., MapElites) on BeamNG. The command is for simulated images; for pseudo-real images the three options showed in Section 1.2. should be added.
python run_individual_migration.py --env-name donkey \
--donkey-exe-path <path/to/donkey-simulator/executable> \
--agent-type supervised \
--model-path <path/to/mixed-dave2.h5> \
--feature-combination turns_count-curvature \
--filepath mapelites_udacity_<date_str>_all
This command will create the folder mapelites_migration_donkey_<date_str>.
It is better to rename such folder in mapelites_migration_donkey_udacity_search.
It should be read in this way: migration on Donkey of individuals obtained by running the search (i.e., MapElites) on Udacity. The command is for simulated images; for pseudo-real images the three options showed in Section 1.2. should be added.
In this step we merge the maps coming from the same simulators. This step is called Union in the paper.
First, copy the mapelites_run/mapelites_beamng_<date_str>_all folder and rename it as mapelites_beamng_search;
in the same way copy the mapelites_run/mapelites_udacity_<date_str>_all folder and rename it as mapelites_udacity_search.
Then, create a folder under logs grouping the _all folders of the MapElites search and migrations on Udacity, BeamNG and Donkey.
For the sake of the example let us call the folder mapelites_run.
At this point the mapelites_run folder under logs should contain the following folders:
mapelites_beamng_<date_str>_0
mapelites_beamng_<date_str>_1
mapelites_beamng_<date_str>_2
mapelites_beamng_<date_str>_3
mapelites_beamng_<date_str>_4
mapelites_beamng_<date_str>_all
mapelites_beamng_search
mapelites_migration_beamng_udacity_search
mapelites_migration_donkey_beamng_search
mapelites_migration_donkey_udacity_search
mapelites_migration_udacity_beamng_search
mapelites_udacity_<date_str>_0
mapelites_udacity_<date_str>_1
mapelites_udacity_<date_str>_2
mapelites_udacity_<date_str>_3
mapelites_udacity_<date_str>_4
mapelites_udacity_<date_str>_all
mapelites_udacity_search
while the logs folder contains:
mapelites_run
Run the following command:
./convert_probability_maps.sh mapelites_run
to convert the maps from success probability (generated by the search/migration) to failure probability required by the following analyses.
Run the following command:
./merge_individual_maps_simulators_probabilities.sh mapelites_run
The command creates the following folders in mapelites_run:
merged_mapelites_beamng_search_mapelites_migration_beamng_udacity_search
merged_mapelites_migration_donkey_udacity_search_mapelites_migration_donkey_beamng_search
merged_mapelites_udacity_search_mapelites_migration_udacity_beamng_search
At this point all maps contain the same individuals but executed on different simulators, i.e.,
respectively BeamNG, Donkey and Udacity.
Run the following command:
python plot_quality_metrics_map.py --folder logs \
--filepath mapelites_run \
--datetime-str-beamng <date_str_beamng> \
--datetime-str-udacity <date_str_udacity>
where <date_str_beamng> and <date_str_udacity> are the datetime strings
corresponding to the folders created by the MapElites search respectively for BeamNG and Udacity.
Move the quality metrics maps from mapelites_udacity_<date_str>_all to mapelites_run/mapelites_udacity_search as well as
move the quality metrics maps from mapelites_beamng_<date_str>_all to mapelites_run/mapelites_beamng_search.
The maps are the following:
raw_heatmap_max_lateral_positions_turns_count_curvature_iterations_0.json
raw_heatmap_std_lateral_positions_turns_count_curvature_iterations_0.json
raw_heatmap_std_speeds_turns_count_curvature_iterations_0.json
raw_heatmap_std_steering_angles_turns_count_curvature_iterations_0.json
The .png files can be copied as well but it is not mandatory.
The previous command also prints on the console the bounds for each quality metric. For instance:
INFO:plot_quality_metrics_map:Bounds for the quality metrics:
{
'std_steering_angles': (0.060142596293090676, 0.5921246225056953),
'std_lateral_positions': (0.05692786404029756, 0.8862904614683409),
'std_speeds': (4.6374119298227985, 12.133422056281063),
'max_lateral_positions': (0.20460000000000012, 2.1)
}
The first number in each tuple is the lower bound for the respective metric and the second is the upper bound.
Then run the following command:
./merge_individual_maps_simulators_quality_metrics.sh mapelites_run \
0.060142596293090676 0.5921246225056953 \
0.05692786404029756 0.8862904614683409 \
4.6374119298227985 12.133422056281063 \
0.20460000000000012 2.1
The bounds come from the previous step and need to be provided in the order specified in the previous step.
In this step we merge the maps coming from different simulators.
Run the following command:
./merge_maps_simulators_probabilities.sh mapelites_run
The command merges the failure probability maps of BeamNG and Udacity using the product operator.
The command should produce the folder merged_merged_beamng_udacity in mapelites_run.
Run the following command:
./merge_maps_simulators_quality_metrics.sh mapelites_run min \
0.060142596293090676 0.5921246225056953 \
0.05692786404029756 0.8862904614683409 \
4.6374119298227985 12.133422056281063 \
0.20460000000000012 2.1
The command merges the quality metrics maps of BeamNG and Udacity using the minimum operator.
The command should produce the quality metrics maps inside mapelites_run/merged_merged_beamng_udacity.
At the end of such step the content of the folder mapelites_run/merged_merged_beamng_udacity should be the following:
heatmap_failure_probability_multiply_turns_count_curvature_iterations_0.png
heatmap_max_lateral_positions_min_turns_count_curvature_iterations_0.png
heatmap_std_lateral_positions_min_turns_count_curvature_iterations_0.png
heatmap_std_speeds_min_turns_count_curvature_iterations_0.png
heatmap_std_steering_angles_min_turns_count_curvature_iterations_0.png
raw_heatmap_failure_probability_multiply_turns_count_curvature_iterations_0.json
raw_heatmap_max_lateral_positions_min_turns_count_curvature_iterations_0.json
raw_heatmap_std_lateral_positions_min_turns_count_curvature_iterations_0.json
raw_heatmap_std_speeds_min_turns_count_curvature_iterations_0.json
raw_heatmap_std_steering_angles_min_turns_count_curvature_iterations_0.json
In this step we compute the correlations between the digital siblings (i.e., Udacity and BeamNG) and the digital twin (i.e., Donkey).
Run the following command:
./compute_correlations_probabilities.sh mapelites_run
The correlations and all the other metrics are printed on console.
./compute_correlations_quality_metrics.sh mapelites_run
The correlations and all the other metrics are printed on console.
The command only computes the correlations for the max_lateral_position quality metric.
Download the offline evaluation datasets from here and copy them within the logs folder of this repository. Let us carry out the offline evaluation for the dave2 model.
From the root of the project type:
python compute_offline_metrics.py --archive-path logs \
--archive-names offline-evaluation-beamng-dave2 offline-evaluation-donkey-dave2.npz \
--show-plot
to compute the correlation and the distance between the prediction error distributions of BeamNG and Donkey. Save the plot where desired.
Type:
python compute_offline_metrics.py --archive-path logs \
--archive-names offline-evaluation-udacity-dave2 offline-evaluation-donkey-dave2.npz \
--show-plot
to compute the correlation and the distance between the prediction error distributions of Udacity and Donkey. Save the plot where desired.
Type:
python compute_offline_metrics.py --archive-path logs \
--archive-names offline-evaluation-beamng-dave2 offline-evaluation-udacity-dave2 offline-evaluation-donkey-dave2.npz \
--show-plot
to merge the prediction error distributions of BeamNG and Udacity and compute its correlation and distance w.r.t. the prediction error distribution of Donkey. Save the plot where desired.
From the root of the project type:
python compute_offline_metrics.py --archive-path logs \
--archive-names offline-evaluation-fake-beamng-dave2 offline-evaluation-fake-donkey-dave2.npz \
--show-plot
to compute the correlation and the distance between the prediction error distributions of BeamNG and Donkey. Save the plot where desired.
Type:
python compute_offline_metrics.py --archive-path logs \
--archive-names offline-evaluation-fake-udacity-dave2 offline-evaluation-fake-donkey-dave2.npz \
--show-plot
to compute the correlation and the distance between the prediction error distributions of Udacity and Donkey. Save the plot where desired.
Type:
python compute_offline_metrics.py --archive-path logs \
--archive-names offline-evaluation-fake-beamng-dave2 offline-evaluation-fake-udacity-dave2 offline-evaluation-fake-donkey-dave2.npz \
--show-plot
to merge the prediction error distributions of BeamNG and Udacity and compute its correlation and distance w.r.t. the prediction error distribution of Donkey. Save the plot where desired.