Revisiting and Extending Similarity-based Metrics in Summary Factual Consistency Detection

Yuxuan Ye, Edwin Simpson, Raul Santos-Rodriguez

Keywords: Factual Consistency, Summarisation

Abstract:

Cutting-edge abstractive summarisers generate fluent summaries, but the factuality of the generated text is not guaranteed.
Early summary factuality evaluation metrics are usually based on n-gram overlap and embedding similarity, but are reported fail to align with human annotations.
Therefore, many techniques for detecting factual inconsistencies build pipelines around natural language inference (NLI) or question-answering (QA) models with additional supervised learning steps.
In this paper, we revisit similarity-based metrics,
showing that this failure stems from the use of reference texts for comparison and the granularity of the comparison.
We propose a new zero-shot factuality evaluation metric,
Sentence-BERT Score (SBERTScore), which compares sentences between the summary and the source document.
It outperforms widely-used word-word metrics including BERTScore and can compete with existing NLI and QA-based factuality metrics on the benchmark without needing any fine-tuning.
Our experiments indicate that each technique has different strengths, with SBERTScore particularly effective at identifying correct summaries.
Additionally, we demonstrate how a combination of techniques is more effective at detecting various types of error.

Primary Area: other topics in machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 11792