Taking the GP Out of the Loop

ICLR 2026 Conference Submission21469 Authors

Keywords: bayesian optimization, high-dimensional, many observations

TL;DR: Fast, scalable Bayesian optimization: Replace GP w/KNN, skip hyperparameter fitting, and use non-dominated sorting of (μ, σ)

Abstract: Bayesian optimization (BO) has traditionally solved black-box problems where evaluation is expensive and, therefore, observations are few. Recently, however, there has been growing interest in applying BO to problems where evaluation is cheaper and observations are more plentiful. Scaling BO to many observations, $N$, is impeded by the $\mathcal{O}(N^3)$ cost of a na\"{\i}ve query (or $\mathcal{O}(N^2)$ in optimized implementations) of the Gaussian process (GP) surrogate. Many methods improve scaling at acquisition time, but hyperparameter fitting still scales poorly. Because a GP is refit at every iteration of BO, fitting remains the bottleneck. We propose Epistemic Nearest Neighbors (ENN), a lightweight alternative to GPs that estimates function values and epistemic uncertainty from $K$-nearest-neighbor observations. ENN has $\mathcal{O}(N)$ acquisition cost and, crucially, omits hyperparameter fitting, making ENN-based BO also $\mathcal{O}(N)$. Because ENN omits hyperparameter fitting, its uncertainty scale is arbitrary, making it incompatible with standard acquisition methods. We resolve this by applying a non-dominated sort (NDS) to candidate points, treating predicted values ($\mu$) and uncertainties ($\sigma$) as two independent metrics. Our method, TuRBO-ENN, replaces the GP surrogate in TuRBO with ENN and its Thompson-sampling acquisition with this NDS-based alternative. We show empirically that TuRBO-ENN reduces proposal generation time by one to two orders of magnitude compared to TuRBO and scales to thousands of observations.

Primary Area: optimization

Submission Number: 21469