Keywords: Bayesian Optimization, Learning Curve Extrapolation, Prior-fitted Networks, Hyperparameter Optimization, in-context learning
TL;DR: We propose FT-PFN an in-context learning surrogate for freeze-thaw Bayesian optimization, improving efficiency, reliability and accuracy of predictions, achieving state-of-the-art HPO performance in the low budget regime of 20 full training runs.
Abstract: With the growing computational costs in deep learning, traditional black-box Bayesian optimization (BO) methods for hyperparameter optimization face significant challenges. We introduce a novel surrogate leveraging transformers' in-context learning for freeze-thaw BO, which strategically allocates resources incrementally and performs Bayesian learning curve extrapolation efficiently in a single forward pass. Our method shows superior accuracy and speed compared to existing surrogates and achieves state-of-the-art performance on three deep learning benchmark suites.
Submission Checklist: Yes
Broader Impact Statement: Yes
Paper Availability And License: Yes
Code Of Conduct: Yes
Optional Meta-Data For Green-AutoML: All questions below on environmental impact are optional.
Submission Number: 26
Loading