Can Few-shot Work in Long-Context? Recycling the Context to Generate Demonstrations
Abstract: Despite recent advancements in Large Language Models (LLMs), their performance on tasks involving long contexts remains sub-optimal. In-Context Learning (ICL) with few-shot examples may be an appealing solution to enhance LLM performance in this scenario too. However, na\"ively adding ICL examples with long context faces challenges, including domain shifts between demonstrations and the target query and substantial token overhead added for each example.
In this work, we propose to automatically generate few-shot examples for long context QA tasks by recycling contexts. Specifically, given a long input context (1-3k tokens) and a query, we generate additional query-output pairs from the given context as few-shot examples. This ensures that the demonstrations come from the same domain as the target query and only add a small number of tokens to the prompt. Furthermore, we enhance each demonstration example by instructing the model to \textit{explicitly} identify the relevant paragraphs before the answer. This approach acts as a structured Chain of Thought and provides fine-grained attribution to the answer.
We apply our method on multiple models and obtain a substantial improvement on various QA datasets with long context, especially when the answer lies within the middle of the text. Surprisingly, despite introducing only single-passage ICL examples, LLMs successfully generalize to multi-hop long-context QA using our approach.
Paper Type: Long
Research Area: Question Answering
Research Area Keywords: In Context Learning, Few-shots, Multi-hop QA
Contribution Types: NLP engineering experiment
Languages Studied: English
Submission Number: 4557
Loading