Automatic Prompt Selection for Large Language Models

Viet-Tung Do, Xuan-Quang Nguyen, Van-Khanh Hoang, Duy-Hung Nguyen, Shahab Sabahi, Jeff Yang, Hajime Hotta, Minh-Tien Nguyen, Hung Le

Abstract: This paper proposes an effective approach to automatically select the optimal prompt for a given input from a finite set of synthetic candidate prompts. Different from existing methods that are usually costly and lack flexibility, the approach is designed to balance prompt generality-specificity and eliminate the need for resource-intensive training and inference. The approach consists of three steps: (1) clustering the training data and generating candidate prompts for each cluster using an LLM-based prompt generator; (2) synthesizing a dataset of input-prompt-output tuples for training a prompt evaluator to rank the prompts based on their relevance to the input; (3) using the prompt evaluator to select the best prompt for new input at test time. The approach demonstrates competitive performance on 5 question-answering datasets, including GSM8K, AQuA, MMLU, and real insurance data.

External IDs:dblp:conf/pakdd/DoNHNSYHNL25