FunBO: Discovering Acquisition Functions for Bayesian Optimization with FunSearch

Virginia Aglietti, Ira Ktena, Jessica Schrouff, Eleni Sgouritsa, Francisco Ruiz, Alan Malek, Alexis Bellot, Silvia Chiappa

TL;DR: We propose FunBO, an LLM-based method that can be used to learn new acquisition functions for Bayesian Optimization that perform well across a variety of experimental settings.

Abstract: The sample efficiency of Bayesian optimization algorithms depends on carefully crafted acquisition functions (AFs) guiding the sequential collection of function evaluations. The best-performing AFs can vary significantly across optimization problems, often requiring ad-hoc and problem-specific choices. This work tackles the challenge of designing novel AFs that perform well across a variety of experimental settings. Based on FunSearch, a recent work using Large Language Models (LLMs) for discovery in mathematical sciences, we propose FunBO, an LLM-based method that can be used to learn new AFs written in computer code by leveraging access to a number of evaluations for a limited set of objective functions. We provide the analytic expression of all discovered AFs and evaluate them on various global optimization benchmarks and hyperparameter optimization tasks. We show how FunBO identifies AFs that generalize well both in and out of the training distribution of functions, thus outperforming established general-purpose AFs and achieving competitive performance against AFs that are customized to specific function types and are learned via transfer-learning algorithms.

Lay Summary: When tackling complex optimization problems where each experiment is expensive—like finding the best recipe for a new material or tuning a sophisticated AI model—we often use a method called Bayesian Optimization. This method smartly decides what to try next using a "guide" called an acquisition function. However, the best guide can differ greatly from one problem to another, making it tricky to choose the right one. We've developed FunBO, a new approach that employs Large Language Models (LLMs) to automatically discover novel and effective acquisition functions. Instead of relying on pre-defined guides, FunBO gets a LLM to write new ones as small pieces of computer code. This makes the discovered guides transparent, easy to understand, and simple to integrate into existing systems. Our experiments show FunBO can find guides that accelerate the discovery of optimal solutions for a wide variety of challenges, often outperforming standard approaches and helping to reduce the cost of searching for optimal solutions.

Primary Area: Deep Learning->Large Language Models

Keywords: Bayesian Optimization, LLM, Acquisition Functions

Submission Number: 10603