On the Intractability to Synthesize Factual Inconsistencies in Summarization

Anonymous

Abstract:

Factual consistency detection has gotten raised attention in the task of abstractive summarization.
Many of existing works relies on synthetic training data, which may not accurately reflect or match the inconsistencies produced by summarization models.
In this paper, we first systematically analyze the shortcomings of the current methods in synthesizing inconsistent summaries.
Current synthesis methods may fail to produce inconsistencies of coreference errors and discourse errors, per our quantitative and qualitative study.
Then, employing the parameter-efficient finetuning (PEFT) technique, we discover that a competitive factual consistency detector can be achieved using thousands of real model-generated summaries with human annotations.
Our study demonstrates the importance of human annotation in NLG evaluation as our model outperforms the SOTA by 8, 4.5, and 2.36 percentage points on the datasets CoGenSumm, Frank, and SummEval, respectively.

Paper Type: long

Research Area: Summarization

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Publicly available software and/or pre-trained models, Data analysis

Languages Studied: English

Consent To Share Submission Details: On behalf of all authors, we agree to the terms above to share our submission details.