LLMs for Bayesian Optimization in Scientific Domains: Are We There Yet?

ACL ARR 2025 May Submission7083 Authors

Abstract: Large language models (LLMs) have recently been proposed as general-purpose agents for experimental design, with claims that they can perform in-context experimental design. We evaluate this hypothesis using open-source instruction-tuned LLMs applied to genetic perturbation and molecular property discovery tasks. We find that LLM-based agents show no sensitivity to experimental feedback: replacing true outcomes with randomly permuted labels has no impact on performance. Across benchmarks, classical methods such as linear bandits and Gaussian process optimization consistently outperform LLM agents. We further propose a simple hybrid method, LLM-guided Nearest Neighbour (LLMNN) sampling, that combines LLM prior knowledge with nearest-neighbor sampling to guide the design of experiments. LLMNN achieves competitive or superior performance across domains without requiring significant in-context adaptation. These results suggest that current open-source LLMs do not perform in-context experimental design in practice and highlight the need for hybrid frameworks that decouple prior-based reasoning from batch acquisition with updated posteriors.

Paper Type: Long

Research Area: Resources and Evaluation

Research Area Keywords: Evaluation, Healthcare Applications, Reproducibility

Contribution Types: Model analysis & interpretability, Reproduction study, Approaches to low-resource settings, Data analysis

Languages Studied: English

Submission Number: 7083