SOA: Strategic Operator Adaptation for Accelerating Joint In-Context Prompt Optimization
Abstract: Strategic prompt-tuning in Large Language Models (LLMs) presents a formidable challenge that requires substantial resources and expert human input. Prior research has treated the tuning of prompt instructions and few-shot examples as distinct and separate problems, resulting in sub-optimal performance. This work overcomes this limitation by introducing a joint prompt-tuning approach that optimizes both the instruction and examples simultaneously. However, formulating such an optimization in the discrete and high-dimensional space of natural language poses significant challenges in terms of convergence and computational efficiency. To address these challenges, we propose, \frameNamenospace, a novel Strategic Operator Adaptation framework, designed to accelerate the optimization process by strategically employing a variety of operators to traverse the prompt space effectively for both zero-shot and few-shot scenarios. \frameName features a quad-phased design that fully exploits the potential of each phase, alternating between global traversal and local optimization to strike a balance between exploration and exploitation in this complex space. By adaptively selecting the best operators for traversal and actively pruning less desirable candidates, \frameName is able to identify the best combination of instructions and examples while minimizing inference costs. We have conducted a comprehensive evaluation across 35 benchmark tasks, and the results show that \frameName significantly outperforms state-of-the-art baseline methods by a large margin, achieving an average task performance improvement of {\bf 35.47}\% while significantly reducing computational costs by {\bf 58.67}\% in the BIG-Bench-Hard tasks. \footnote{The source code and datasets are ready to be publicly available for research purposes.}
Submission Length: Regular submission (no more than 12 pages of main content)
Changes Since Last Submission: NA
Assigned Action Editor: ~Maxime_Gasse2
Submission Number: 3278
Loading