Re-Examining Linear Embeddings for High-dimensional Bayesian Optimization

Benjamin Letham, Roberto Calandra, Akshara Rai, Eytan Bakshy

Keywords: Bayesian optimization, high-dimensional, Gaussian process

TL;DR: We study the use of linear embeddings for high-dimensional Bayesian optimization, identify issues that have caused poor performance, and develop new techniques for improving optimization performance in the embedding.

Abstract:

Bayesian optimization (BO) is a popular approach to optimize resource-intensive black-box functions.
A significant challenge in BO is to scale to high-dimensional parameter spaces while retaining sample efficiency.
A solution considered in previous literature is to embed the high-dimensional parameter space into a lower-dimensional manifold, often a random linear embedding. In this paper, we identify several crucial issues and misconceptions about the use of linear embeddings for BO. We thoroughly study and analyze the consequences of using linear embeddings and show that some of the design choices in current approaches adversely impact their performance. Based on this new theoretical understanding we propose ALEBO, a new algorithm for high-dimensional BO via linear embeddings that outperforms state-of-the-art methods on a range of problems.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2001.11659/code)