DACBench: A Benchmark Library for Dynamic Algorithm Configuration

Theresa Eimer, André Biedenkapp, Maximilian Reimer, Steven Adriaensen, Frank Hutter, Marius Lindauer

Abstract:

Dynamic Algorithm Configuration (DAC) aims to
dynamically control a target algorithm’s hyperpa-
rameters in order to improve its performance. Sev-
eral theoretical and empirical results have demon-
strated the benefits of dynamically controlling hy-
perparameters in domains like evolutionary com-
putation, AI Planning or deep learning. Replicat-
ing these results, as well as studying new methods
for DAC, however, is difficult since existing bench-
marks are often specialized and incompatible with
the same interfaces. To facilitate benchmarking and
thus research on DAC, we propose DACBench, a
benchmark library that seeks to collect and stan-
dardize existing DAC benchmarks from different
AI domains, as well as provide a template for new
ones. For the design of DACBench, we focused on
important desiderata, such as (i) flexibility, (ii) re-
producibility, (iii) extensibility and (iv) automatic
documentation and visualization. To show the po-
tential, broad applicability and challenges of DAC,
we explore how a set of six initial benchmarks com-
pare in several dimensions of difficulty.