In-Context Freeze-Thaw Bayesian Optimization for Hyperparameter Optimization

Herilalaina Rakotoarison, Steven Adriaensen, Neeratyoy Mallik, Samir Garibov, Edward Bergman, Frank Hutter

Keywords: Bayesian Optimization, Learning Curve Extrapolation, Prior-fitted Networks, Hyperparameter Optimization, in-context learning

TL;DR: We propose FT-PFN an in-context learning surrogate for freeze-thaw Bayesian optimization, improving efficiency, reliability and accuracy of predictions, achieving state-of-the-art HPO performance in the low budget regime of 20 full training runs.

Abstract: With the growing computational costs in deep learning, traditional black-box Bayesian optimization (BO) methods for hyperparameter optimization face significant challenges. We introduce a novel surrogate leveraging transformers' in-context learning for freeze-thaw BO, which strategically allocates resources incrementally and performs Bayesian learning curve extrapolation efficiently in a single forward pass. Our method shows superior accuracy and speed compared to existing surrogates and achieves state-of-the-art performance on three deep learning benchmark suites.

Submission Checklist: Yes

Broader Impact Statement: Yes

Paper Availability And License: Yes

Code Of Conduct: Yes

Optional Meta-Data For Green-AutoML: All questions below on environmental impact are optional.

Submission Number: 26