Identifying Factual Inconsistency in Summaries: Large Language Model is Ready to Help

Anonymous

TL;DR: LLM identifies factual errors in summaries directly under the right zero-shot design, and can be distilled smaller keeping high efficiency and efficacy.

Abstract: Factual inconsistency poses a significant hurdle for the commercial deployment of abstractive summarizers. Under this new era of Large Language Model (LLM), this work focuses around two important questions: what is the best way to leverage LLM for factual inconsistency detection, and how could we distill a smaller LLM with both high efficiency and efficacy? Three zero-shot paradigms are firstly proposed and evaluated across five diverse datasets: direct inference on the entire summary or each summary window; entity verification through question generation and answering. Our experiments suggest that LLM itself is capable to resolve this task directly under the correct paradigm design, which surpasses the baselines by up to 4.7% on average. To further promote efficiency in practice, we then propose training strategies to distill smaller open-source LLM that learns to score the entire summary at once with high accuracy, which outperforms the zero-shot approaches by much larger LLM, serving as an effective ready-to-use scorer.

Paper Type: long

Research Area: Summarization

Contribution Types: NLP engineering experiment

Languages Studied: English