Identifying Factual Inconsistencies in Summaries: Grounding Model Inference via Task Taxonomy
Abstract: Factual inconsistencies pose a significant hurdle for the faithful summarization by generative models.
While a major direction to enhance inconsistency detection is to derive stronger Natural Language Inference (NLI) models, we propose an orthogonal aspect that underscores the importance of incorporating task-specific taxonomy into the inference.
To this end, we consolidate key error types of inconsistent facts in summaries, and incorporate them to facilitate both the zero-shot and supervised paradigms of LLMs.
Extensive experiments on ten datasets of five distinct domains suggest that, zero-shot LLM inference could benefit from the explicit solution space depicted by the error type taxonomy, and achieves state-of-the-art performance overall, surpassing specialized non-LLM baselines, as well as recent LLM baselines.
We further distill models that fuse the taxonomy into parameters through our designed prompt completions and supervised training strategies, efficiently substituting state-of-the-art zero-shot inference with much larger LLMs.
Paper Type: Long
Research Area: Summarization
Research Area Keywords: factuality
Contribution Types: NLP engineering experiment
Languages Studied: English
Submission Number: 3371
Loading