Demostrations Aren’t All You Need For Long-form Generation! Learning Task-Inherent Attribute Guidelines For Large Language Models
Abstract: We study the sufficiency of demonstrations in enabling pre-trained large language models (LLMs) to implicitly learn the underlying task distribution for long-form generation. We prove the answer is no. For any long-form generation task, we show that if an LLM fails to initially grasp the task’s language distribution, demonstrations alone are insufficient. This gap is caused by a lack of explicit task-language distribution characterization exposed to the model. Addressing this by capturing these distributions explicitly through task guidelines enhances model performance. We then present LongGuide, the first efficient algorithm that generates two types of guidelines as additional instructions for LLMs: (i) Metric Guideline (MG) that instructs models to optimize for selected metrics; and (ii) Output Constraint Guideline (OCG) that constrains generation at both the token and sentence levels. LongGuide automatically selects the most useful combination of guidelines, improving strong open- and closed-source LLMs by 5.39% and 6.58% under zero- and few-shot settings across seven tasks. Furthermore, LongGuide enhances LLMs beyond demonstrations, is learnable by weaker models to enhance stronger ones, and synergistically combines with prompt optimizers.
Paper Type: Long
Research Area: Special Theme (conference specific)
Research Area Keywords: Large Language Models, In-context Learning, Automatic Prompt Design and Optimization
Contribution Types: Model analysis & interpretability, NLP engineering experiment, Approaches to low-resource settings, Approaches low compute settings-efficiency, Theory
Languages Studied: English
Submission Number: 5083
Loading