Are Language Model Embeddings Sufficient for Bayesian Optimization?

TMLR Paper5346 Authors

Abstract: Bayesian Optimization is ubiquitous in experimental design and black-box optimization for improving search efficiency. However, most existing approaches rely on regression models which are limited to fixed search spaces and structured, tabular input features. This paper explores the use of LLM embeddings over string inputs for in-context regression in Bayesian Optimization. Our results show that representing inputs as strings enables general-purpose regression across diverse domains, including synthetic, combinatorial, and hyperparameter optimization. Furthermore, our approach achieves optimization performance comparable to state-of-the-art Gaussian Process-based methods such as Google Vizier, and demonstrates potential for broader and more flexible applications.

Submission Length: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Arto_Klami1

Submission Number: 5346