DACBench is a benchmark library for Dynamic Algorithm Configuration. Its focus is on reproducibility and comparability of different DAC methods as well as easy analysis of the optimization process.
You can try out the basics of DACBench in Colab here without any installation. Our examples in the repository should give you an impression of what you can do with DACBench and our documentation should answer any questions you might have.
We recommend installing DACBench in a virtual environment, here with uv:
pip install uv
uv venv --python 3.10
source .venv/bin/activate
uv pip install dacbench
Instead of using pip, you can also use the GitHub repo directly:
pip install uv
git clone https://github.com/automl/DACBench.git
cd DACBench
uv venv --python 3.10
source .venv/bin/activate
git submodule update --init --recursive
make install
This command installs the base version of DACBench including the three small surrogate benchmarks. For any other benchmark, you may use a singularity container as provided by us (see next section) or install it as an additional dependency. As an example, to install the SGDBenchmark, run:
uv pip install dacbench[sgd]
You can also install all dependencies like so:
make install-dev
DACBench can run containerized versions of Benchmarks using Singularity containers to isolate their dependencies and make reproducible Singularity images.
For writing your own recipe to build a Container, you can refer to dacbench/container/singularity_recipes/recipe_template
Install Singularity and run the following to build the (in this case) cma container
cd dacbench/container/singularity_recipes
sudo singularity build cma cma.def
If you use DACBench in your research or application, please cite us:
@inproceedings{eimer-ijcai21,
author = {T. Eimer and A. Biedenkapp and M. Reimer and S. Adriaensen and F. Hutter and M. Lindauer},
title = {DACBench: A Benchmark Library for Dynamic Algorithm Configuration},
booktitle = {Proceedings of the Thirtieth International Joint Conference on
Artificial Intelligence ({IJCAI}'21)},
year = {2021},
month = aug,
publisher = {ijcai.org},