Ensemble Mashups: A Simple Recipe For Better Bayesian Optimization

Anand Ravishankar, Fernando Llorente, Yuanqing Song, Petar Djuric

Keywords: Bayesian Optimization, Ensembe Models, Gaussian Processes, Hedging Algorithm, Bayesian Model Averaging

TL;DR: This paper proposes a novel Ensemble Bayesian optimization approach, which combines ensembles of both kernel functions and acquisition functions, demonstrating improved robustness compared to single-ensemble methods

Abstract:

Bayesian optimization (BO) is a popular approach to optimizing costly, black-box functions that rely on a statistical surrogate model of the function, typically a Gaussian process (GP), and the so-called acquisition function (AF). Although the choices of the GP kernel and the AF can strongly affect the results, there does not exist an automatic way of selecting them.
Ensembling, namely, using several, different kernels (Multi-Model) or AFs (Multi-AF) is one possibility for deriving a BO algorithm that is robust and safer. These ideas have been considered separately in the past. In this work, we consider ensembles of both kernels and AFs (Multi-Model Multi-AF) and perform an empirical comparison to show their superiority with respect to single-ensemble algorithms.

Submission Number: 56