SummaCoz: A Dataset for Improving the Interpretability of Factual Consistency Detection for Summarization
Abstract: Summarization is an important application of Large Language Models (LLMs).
When judging the quality of a summary, factual consistency holds a significant weight.
Despite numerous efforts dedicated to building factual inconsistency detectors, the exploration of explanability remains limited among existing effort.
In this study, we incorporate both human-annotated and model-generated natural language explanations elucidating how a summary deviates and thus becomes inconsistent with its source article. We build our explanation-augmented dataset on top of the widely used SummaC summarization consistency benchmark.
Additionally, we develop an inconsistency detector that is jointly trained with the collected explanations.
Our findings demonstrate that integrating explanations during training not only enables the model to provide rationales for its judgments but also enhances its accuracy significantly.
Paper Type: Short
Research Area: Summarization
Research Area Keywords: evaluation,factuality
Contribution Types: Model analysis & interpretability, NLP engineering experiment, Data resources
Languages Studied: English
Submission Number: 3692
Loading