A SYSTEMATIC STUDY ON EARLY STOPPING CRITERIA IN HPO AND THE IMPLICATIONS OF UNCERTAINTY

Jiawei Guan, Feng Zhang, Jiesong Liu, Xiaoyong Du, Xipeng Shen

Primary Area: general machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: HPO, early stop, criteria, uncertainty

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: An empirical study of the reliability of early stopping criteria in Hyperparameter Optimization, and the implication of uncertainty integration.

Abstract:

The development of hyperparameter optimization (HPO) algorithms constitutes a key concern within the machine learning domain. While numerous strategies employing early stopping mechanisms have been proposed to bolster HPO efficiency, there remains a notable deficiency in understanding how the selection of early stopping criteria influences the reliability of early stopping decisions and, by extension, the broader outcomes of HPO endeavors. This paper undertakes a systematic exploration of the impact of criterion selection on the effectiveness of early stopping-based HPO. Specifically, we introduce a set of criteria that incorporate uncertainty and highlight their practical significance in enhancing the reliability of early stopping decisions. Through a series of empirical experiments conducted on HPO and NAS benchmarks, we substantiate the critical role of criterion selection, while shedding light on the potential implications of integrating uncertainty as a criterion. This research furnishes empirical insights that serve as a compass for the selection and formulation of criteria, thereby contributing to a more profound comprehension of mechanisms underpinning early stopping-based HPO.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 895