Multi-objective Tree-structured Parzen Estimator Meets Meta-learning

Shuhei Watanabe, Noor Awad, Masaki Onishi, Frank Hutter

Keywords: Hyperparameter optimization, Black-box optimization, AutoML, Meta-learning, Bayesian optimization

TL;DR: Speedup multi-objective optimization by tree-structured Parzen estimator with a meta-learning using a new task similarity measure

Abstract: Hyperparameter optimization (HPO) is essential for the better performance of deep learning, and practitioners often need to consider the trade-off between multiple metrics, such as error rate, latency, memory requirements, robustness, and algorithmic fairness. Due to this demand and the heavy computation of deep learning, the acceleration of multi-objective (MO) optimization becomes ever more important. Although meta-learning has been extensively studied to speedup HPO, existing methods are not applicable to the MO tree-structured parzen estimator (MO-TPE), a simple yet powerful MO HPO algorithm. In this paper, we extend TPE’s acquisition function to the meta-learning setting, using a task similarity defined by the overlap in promising regions of each task. In a comprehensive set of experiments, we demonstrate that our method accelerates MO-TPE on tabular HPO benchmarks and yields state-of-the-art performance. Our method was also validated externally by winning the AutoML 2022 competition on "Multiobjective Hyperparameter Optimization for Transformers".

Community Implementations: [ 5 code implementations](https://www.catalyzex.com/paper/arxiv:2212.06751/code)