HPO: We won’t get fooled againDownload PDF

21 Apr 2022, 13:31 (modified: 23 Jun 2022, 08:45)AutoML 2022 (Late-Breaking Workshop)Readers: Everyone
Abstract: Hyperparameter optimization (HPO) is a well-studied research field. However, the effects and interactions of the components in an HPO pipeline are not yet well investigated. Then, we ask ourselves: Can the landscape of HPO be biased by the pipeline used to evaluate individual configurations? To address this question, we proposed to analyze the effect of the HPO pipeline on HPO problems using fitness landscape analysis. Particularly, we studied the DS-2019 HPO benchmark data set, looking for patterns that could indicate evaluation pipeline malfunction, and relate them to HPO performance. Our main findings are: (i) In most instances, large groups of diverse hyperparameters (i.e., multiple configurations) yield the same \emph{ill} performance, most likely associated with majority class prediction models; (ii) in these cases, a worsened correlation between the observed fitness and average fitness in the neighborhood is observed, potentially making harder the deployment of local-search-based HPO strategies. Finally, we concluded that the HPO pipeline definition might negatively affect the HPO landscape.
Keywords: AutoML, Hyperparameter Optimization, Landscape Analysis
One-sentence Summary: In this paper, we analyze Hyperparameter Optimization (HPO) problems using fitness landscape analysis, and look for patterns that could indicate evaluation pipeline malfunction, and relate them to HPO performance.
Reproducibility Checklist: Yes
Broader Impact Statement: Yes
Paper Availability And License: Yes
Code Of Conduct: Yes
Reviewers: René Traoré
Main Paper And Supplementary Material: pdf
Code And Dataset Supplement: https://github.com/anonymous-for-open-review/late-breaking-automlConf-2022
1 Reply

Loading