DensBO: Dynamic Ensembling of Surrogate Models for Hyperparameter Optimisation

Hadar Shavit, Anja Jankovic, Elena Raponi, Thomas Bäck, Holger Hoos

Keywords: hyperparameter optimisation, Bayesian optimisation, surrogate models

Abstract: Hyperparameter optimisation (HPO) of machine learning models is crucial for achieving optimal performance for different tasks. Surrogate-based optimisation techniques, such as Bayesian optimisation (BO), have been successfully applied to tackle this problem. BO is subject to different design choices of its components. In particular, depending on the nature and the size of the search space, the choice of the surrogate model has a substantial impact on the overall performance of BO. Surrogate models in BO approximate the function to optimise and guide the search towards promising regions by predicting the function value for different solution candidates. Combining different machine learning (ML) models is known to lead to performance gains, e.g., in different prediction tasks. To this end, we propose a novel dynamic approach to ensemble surrogate models in the BO pipeline, leveraging the complementary powers of different surrogate models at different stages of the optimisation process. We empirically evaluate our method on numerous benchmarks and demonstrate its advantage compared to state-of-the-art single-surrogate BO baselines. We highlight the usefulness of our approach in finding good hyperparameter configurations in mixed (numerical and categorical) search spaces for a wide range of problems.

Primary Area: optimization

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6625