Towards Optimizing with Large Language Models

Pei-Fu Guo, Ying-Hsuan Chen, Yun-Da Tsai, Shou-De Lin

Keywords: LLM Problem Solving, LLM Optimization

Abstract: In this study, we evaluate the optimization capabilities of Large Language Models (LLMs) across diverse mathematical and combinatorial optimization tasks, where each task is described in natural language. These tasks require LLM to iteratively generate and evaluate solutions through interactive prompting, where each optimization step involves generating new solutions based on past results and then pass to subsequent iterations. We demonstrate that LLMs can perform various optimization algorithms and act as effective black-box optimizer, capable of intelligently optimizing unknown functions. We also introduce three simple yet informative metrics to evaluate optimization performance, applicable across diverse tasks and less sensitive to test sample variations. Our findings reveal that LLMs excel at optimizing small-scale problems with limited data and their performance is significantly affected by dimension of problem and values, highlighting the need for further research in LLM optimization.

Submission Number: 3