Submodular Evaluation Subset Selection in Automatic Prompt Optimization

ACL ARR 2026 January Submission9375 Authors

Keywords: Automatic Prompt Optimization, Evaluation Subset Selection, Large Language Models

Abstract: Automatic prompt optimization reduces manual prompt engineering, but relies on task performance measured on a small, often randomly sampled evaluation subset as its main source of feedback signal. Despite this, how to select that evaluation subset is usually treated as an implementation detail. We study evaluation subset selection for prompt optimization from a principled perspective and propose SESS, a submodular evaluation subset selection method. We frame selection as maximizing an objective set function and show that, under mild conditions, it is monotone and submodular, enabling greedy selection with theoretical guarantees. Across GSM8K, MATH, and GPQA-Diamond, submodularly selected evaluation subsets can yield better optimized prompts than random or heuristic baselines.

Paper Type: Short

Research Area: Language Models

Research Area Keywords: Language Model, LLM Efficiency

Contribution Types: NLP engineering experiment, Theory

Languages Studied: English

Submission Number: 9375