NUMTEMP: A real-world benchmark to verify claims with statistical and temporal expressions

Venktesh V, Abhijit Anand, Avishek Anand, Vinay Setty

Abstract:

Automated fact checking has gained immense
interest to tackle the growing misinformation
in the digital era. Existing systems primarily
focus on synthetic claims on Wikipedia, and
noteworthy progress has also been made on
real-world claims. In this work, we release
NUMTEMP, a diverse, multi-domain dataset
focused exclusively on numerical claims, encompassing temporal, statistical and diverse
aspects with fine-grained metadata and an evidence collection without leakage. This addresses the challenge of verifying real-world
numerical claims, which are complex and often lack precise information, not addressed
by existing works that mainly focus on synthetic claims. We evaluate and quantify the
limitations of existing solutions for the task of
verifying numerical claims. We also evaluate
claim decomposition based methods, numerical understanding based models and our best
baselines achieves a macro-F1 of 58.32. This
demonstrates that NUMTEMP serves as a challenging evaluation set for numerical claim verification