Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings

Rong-Xi Tan, Ming Chen, Ke Xue, Yao Wang, Yaoyuan Wang, Fu Sheng, Chao Qian

Abstract: The pursuit of universal black-box optimization (BBO) algorithms is a longstanding goal. However, unlike domains such as language or vision, where scaling structured data has driven generalization, progress in offline BBO remains hindered by the lack of unified representations for heterogeneous numerical spaces. Thus, existing offline BBO approaches are constrained to single-task and fixed-dimensional settings, failing to achieve cross-domain universal optimization. Recent advances in language models (LMs) offer a promising path forward: their embeddings capture latent relationships in a unifying way, enabling universal optimization across different data types possible. In this paper, we discuss multiple potential approaches, including an end-to-end learning framework in the form of next-token prediction, as well as prioritizing the learning of latent spaces with strong representational capabilities. To validate the effectiveness of these methods, we collect offline BBO tasks and data from open-source academic works for training. Experiments demonstrate the universality and effectiveness of our proposed methods. Our findings suggest that unifying language model priors and learning string embedding space can overcome traditional barriers in universal BBO, paving the way for general-purpose BBO algorithms. The code is provided at https://github.com/lamda-bbo/universal-offline-bbo.

Lay Summary:

Most computer programs that try to find optimal solutions can only work on one specific type of problem at a time - like finding the best recipe ingredients or the most efficient robot movements. This limitation means we need different optimization programs for different problems, which is inefficient.

We discovered that modern AI language models, which are great at understanding relationships between words and concepts, could help solve this challenge. By converting different types of optimization problems into a format that language models can understand, we create a universal system that can tackle many different kinds of optimization tasks.

We test our approach on a variety of problems collected from academic research and find it work well across different domains. This breakthrough means that instead of needing separate specialized programs for each type of optimization problem, we can have a single, versatile system that handles them all - similar to how human experts can apply their problem-solving skills across different situations.

Link To Code: https://github.com/lamda-bbo/universal-offline-bbo

Primary Area: Optimization->Zero-order and Black-box Optimization

Keywords: Universal optimization; Offline optimization

Submission Number: 15805