AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web

NeurIPS 2023 Track Datasets and Benchmarks Submission468 Authors

Published: 26 Sept 2023, Last Modified: 02 Feb 2024NeurIPS 2023 Datasets and Benchmarks PosterEveryoneRevisionsBibTeX
Keywords: fact-checking, real-world fact-checking, fact extraction and verification, natural language processing, information retrieval, question generation, question generation and answering
TL;DR: The paper introduces a novel dataset for real-world fact-checking that provides both question-answer decomposition and justifications, and avoids issues of context dependence, evidence insufficiency, and temporal leakage.
Abstract: Existing datasets for automated fact-checking have substantial limitations, such as relying on artificial claims, lacking annotations for evidence and intermediate reasoning, or including evidence published after the claim. In this paper we introduce AVeriTeC, a new dataset of 4,568 real-world claims covering fact-checks by 50 different organizations. Each claim is annotated with question-answer pairs supported by evidence available online, as well as textual justifications explaining how the evidence combines to produce a verdict. Through a multi-round annotation process, we avoid common pitfalls including context dependence, evidence insufficiency, and temporal leakage, and reach a substantial inter-annotator agreement of $\kappa=0.619$ on verdicts. We develop a baseline as well as an evaluation scheme for verifying claims through question-answering against the open web.
Supplementary Material: pdf
Submission Number: 468