Lightspeed Black-box Bayesian Optimization via Local Score Matching

Yakun Wang, Sherman Khoo, Song Liu

Keywords: Bayesian optimization, score matching, Probability Improvement

TL;DR: We maximize the acquisition function by gradient ascent where the gradient is learned by local score matching.

Abstract: Bayesian Optimization (BO) is a powerful tool for tackling optimization problems involving limited black-box function evaluations. However, it suffers from high computational complexity and struggles to scale efficiently on high-dimensional problems when fitting a Gaussian process surrogate model. We address these issues by proposing a fast acquisition function maximization procedure. We leverage the fact that Probability Improvement (PI) acquisition function is a likelihood function whose score can be estimated through a simple linear regression problem called local score matching. This enables fast gradient-based optimization of the acquisition function, and a competitive BO procedure which performs similarly to that of computationally expensive neural networks.

Submission Number: 121