When to Use Which? An Investigation of Search Methods on Expensive Black-box Optimisation Problems

ICLR 2026 Conference Submission9351 Authors

Keywords: Black-Box optimisation, Search methods, Bayesian optimisation, Evolutionary algorithms, Local search

Abstract:

Many real-world optimisation problems are black-box in the sense that the structure of their objective function is not accessible or exploitable. Some of such Black-Box Optimisation (BBO) problems are also expensive, thanks to the use of simulations, experiments or costly computations to evaluate a solution (i.e., calculate its objective function value). Despite the prevalence of expensive BBO, different practical scenarios may require different computational resources and search budgets. In some scenarios, evaluating a solution may take hours or days (e.g., in drug design), allowing generating only a few hundred solutions at most, while in some other scenarios, the budget is more generous in which evaluating a solution takes a couple of minutes (e.g., in software configuration tuning), hence allowing a few thousand solutions to be generated. Consequently, a relevant question is that among various popular search methods for BBO (e.g., Bayesian optimisation and evolutionary algorithms), which one is the first choice for practitioners to use under different levels of tightness of their budget, and also what if some domain knowledge of the problem (e.g., ruggedness level of the search space) is available.

In this paper, we aim to answer these questions. Through an extensive experimental study on a suite of test functions with various features, we observe that some methods which were believed unsuitable for expensive BBO are actually competitive under certain circumstances; for example, Nelder Mead on small-size problems with simple landscapes under fairly tight budgets (e.g., 200--800 evaluations) and CMA-ES on medium-sized problems under fairly generous budgets (e.g., $\geq$800). On the other hand, Bayesian optimisation methods perform consistently well under very tight budgets (e.g., $\leq$200) regardless of problem attributes and characteristics.

Primary Area: optimization

Submission Number: 9351