Unstructured Evidence Attribution for Long Context Query Focused Summarization

ACL ARR 2025 May Submission3254 Authors

Abstract: Large language models (LLMs) are capable of generating coherent summaries from very long contexts given a user query, and extracting and citing evidence spans helps improve the trustworthiness of these summaries. Whereas previous work has focused on evidence citation with fixed levels of granularity (e.g. sentence, paragraph, document, etc.), we propose to extract _unstructured_ (i.e., spans of any length) evidence in order to acquire more relevant and consistent evidence than in the fixed granularity case. We show how existing systems struggle to copy and properly cite unstructured evidence, which also tends to be "lost-in-the-middle". To help models perform this task, we create the **S**ummaries with **Uns**tructured **E**vidence **T**ext dataset (SUnsET), a synthetic dataset generated using a novel pipeline, which can be used as training supervision for unstructured evidence summarization. We demonstrate across 5 LLMs and 4 datasets spanning human written, synthetic, single, and multi-document settings that LLMs adapted with SUnsET generate more relevant and factually consistent evidence with their summaries, extract evidence from more diverse locations in their context, and can generate more relevant and consistent summaries than baselines with no fine-tuning and fixed granularity evidence. We release SUnsET and our generation code to the public (https://anonymous.4open.science/r/sunset-BD72/README.md).

Paper Type: Long

Research Area: Summarization

Research Area Keywords: query-focused summarization,long-form summarization,data augmentation,attribution

Contribution Types: Approaches to low-resource settings, Publicly available software and/or pre-trained models, Data resources

Languages Studied: English

Submission Number: 3254