A Copula approach for hyperparameter transfer learning

David Salinas, Huibin Shen, Valerio Perrone

TL;DR: We show how using semi-parametric prior estimations can speed up HPO significantly across datasets and metrics.

Abstract: Bayesian optimization (BO) is a popular methodology to tune the hyperparameters of expensive black-box functions. Despite its success, standard BO focuses on a single task at a time and is not designed to leverage information from related functions, such as tuning performance metrics of the same algorithm across multiple datasets. In this work, we introduce a novel approach to achieve transfer learning across different datasets as well as different metrics. The main idea is to regress the mapping from hyperparameter to metric quantiles with a semi-parametric Gaussian Copula distribution, which provides robustness against different scales or outliers that can occur in different tasks. We introduce two methods to leverage this estimation: a Thompson sampling strategy as well as a Gaussian Copula process using such quantile estimate as a prior. We show that these strategies can combine the estimation of multiple metrics such as runtime and accuracy, steering the optimization toward cheaper hyperparameters for the same level of accuracy. Experiments on an extensive set of hyperparameter tuning tasks demonstrate significant improvements over state-of-the-art methods.

Keywords: Hyperparameter optimization, Bayesian Optimization, Gaussian Process, Copula, Transfer-learning

Community Implementations: [ 3 code implementations](https://www.catalyzex.com/paper/arxiv:1909.13595/code)