Bayesian Optimization over High-Dimensional Combinatorial Spaces via Dictionary-based Embeddings

Aryan Deshwal, Sebastian Ament, Maximilian Balandat, Eytan Bakshy, Jana Doppa, David Eriksson

Abstract:

We consider the problem of optimizing expensive black-box functions over high-dimensional
combinatorial spaces which arises in many science, engineering, and ML applications. We use
Bayesian Optimization (BO) and propose a novel
surrogate modeling approach for efficiently handling a large number of binary and categorical
parameters. The key idea is to select a number of discrete structures from the input space
(the dictionary) and use them to define an ordinal embedding for high-dimensional combinatorial structures. This allows us to use existing Gaussian process models for continuous
spaces. We develop a principled approach based
on binary wavelets to construct dictionaries for
binary spaces, and propose a randomized construction method that generalizes to categorical
spaces. We provide theoretical justification to
support the effectiveness of the dictionary-based
embeddings. Our experiments on diverse realworld benchmarks demonstrate the effectiveness
of our proposed surrogate modeling approach
over state-of-the-art BO methods