Quality of Rejections Matters: Preference Data Construction for Faithful Summarization

Jihwan Oh, Taewon Yun, Minkyu Che, Hwanjun Song

Keywords: LLM, Data Quality, Alignment, Summarization, Factuality

Abstract: Factual reliability is a central challenge in abstractive summarization, as LLMs continue to generate hallucinations. A widely adopted solution is preference optimization, training models to prefer faithful over unfaithful summaries, but prior work emphasized the quantity of rejections over their alignment quality. In this paper, we show that effective alignment in summarization arises when rejected summaries achieve high alignment potential, characterized by a small preference margin that keeps rejections non-trivial and a large factuality margin that enforces clear factual contrast. Through both theoretical analysis and controlled prompting, we show that three factors, hallucination level, summary length, and prompt complexity, critically shape alignment potential. Building on these insights, we propose SPICE, a simple prompting strategy that produces rejected summaries with strong alignment value. Across diverse model scales and alignment algorithms, SPICE consistently achieves superior factuality without sacrificing coherence, relevance, or abstractiveness, outperforming existing rejection strategies.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 11445