QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization

Anonymous

Paper Link: https://openreview.net/forum?id=l1rk8Or3_k

Paper Type: Long paper (up to eight pages of content + unlimited references and appendices)

Abstract: Factual consistency is an essential quality of text summarization models in practical settings. Existing work in evaluating this dimension can be broadly categorized into two lines of research, entailment-based and question answering (QA)-based metrics, and different experimental setups often lead to contrasting conclusions as to which paradigm performs the best. In this work, we conduct an extensive comparison of entailment and QA-based metrics, demonstrating that carefully choosing the components of a QA-based metric, especially question generation and answerability classification, is critical to performance. Building on those insights, we propose an optimized metric, which we call QAFactEval, that leads to a 14% average improvement over previous QA-based metrics on the SummaC factual consistency benchmark, and also outperforms the best-performing entailment-based metric. Moreover, we find that QA-based and entailment-based metrics can offer complementary signals and be combined into a single metric for a further performance boost.

Presentation Mode: This paper will be presented virtually

Virtual Presentation Timezone: UTC-4

Copyright Consent Signature (type Name Or NA If Not Transferrable): Alexander Fabbri

Copyright Consent Name And Address: Alexander Fabbri