Incremental Successive Halving for Hyperparameter Optimization with Budget Constraints

Jasmin Brandt, Marcel Wever, Dimitrios Iliadis, Viktor Bengs, Eyke Hüllermeier

Primary Area: general machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: hyperparameter optimization, sustainability, multi-fidelity

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We theoretically analyze incremental extensions of successive halving and propose a novel extension that is provenly sound and efficient.

Abstract: Hyperparameter optimization (HPO) is indispensable for achieving optimal performance in machine learning tasks. While some approaches focus on sampling more promising hyperparameter configurations, methods based on the successive halving algorithm (SHA) focus on efficiently evaluating hyperparameter configurations through the adaptive allocation of evaluation resources and stopping unpromising candidates early. Yet, SHA comes with several hyperparameters itself, one of which is the maximum budget that can be allocated to evaluate a single hyperparameter configuration. Asynchronous extensions of SHA (ASHA) devise a strategy of autonomously increasing the maximum budget and simultaneously allowing for better parallelization. However, while working well in practice with many considered hyperparameter configurations, there are limitations to the soundness of these adaptations when the overall budget for HPO is limited. This paper provides a theoretical analysis of ASHA in applications with budget constraints. We propose incremental SHA (iSHA), a synchronous extension of SHA, allowing to increment the maximum budget. A theoretical and empirical analysis of iSHA shows that soundness is maintained while guaranteeing to be more resource-efficient than SHA. In an extensive set of experiments, we also demonstrate that, in general, iSHA performs superior to ASHA and progressive ASHA.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5195