ACUEval: Fine-grained Hallucination Evaluation and Correction for Abstractive Summarization

Anonymous

Abstract: The impressive generation capabilities of large language models (LLMs) have made it even harder to detect the subtle hallucinations they make in abstractive summarization, where generated summaries consist of a blend of correct and incorrect information w.r.t. a given document. Recently-proposed LLM-based evaluation metrics attempt to capture this, but still face challenges: (1) they are biased towards summaries generated from the same underlying LLM, and (2) they lack interpretability, offering only a single score. In this work, we present ACUEval, a metric that leverages the power of LLMs to perform two sub-tasks: decomposing summaries into atomic content units (ACUs), and validating them against the source document. Compared to current strong LLM-based metrics, our two-step evaluation strategy improves correlation with human judgments of faithfulness on three summarization evaluation benchmarks by 3% in balanced accuracy compared to the next-best metric, and also shows reduced preference bias towards LLMgenerated summary (by operating with fine-grained units). Further, we show that errors detected by ACUEVAL can be used to generate actionable feedback for refining the summary, successfully improving the faithfulness scores by more than 10%.

Paper Type: long

Research Area: Summarization

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: English