CaseFacts: A Benchmark for Legal Fact-Checking and Precedent Retrieval

ACL ARR 2026 January Submission8661 Authors

Keywords: fact checking, legal fact checking, court cases, benchmark

Abstract: Automated Fact-Checking has largely focused on verifying general knowledge against static corpora, overlooking high-stakes domains like law where truth is evolving and technically complex. We introduce CaseFacts, a benchmark for verifying colloquial legal claims against U.S. Supreme Court precedents. Unlike existing resources that map formal texts to formal texts, CaseFacts challenges systems to bridge the semantic gap between layperson assertions and technical jurisprudence while accounting for temporal validity. The dataset consists of 6,294 claims categorized as Supported, Refuted, or Overruled. We construct this benchmark using a multi-stage pipeline that leverages Large Language Models (LLMs) to synthesize claims from expert case summaries, employing a novel semantic similarity heuristic to efficiently identify and verify complex legal overrulings. Experiments with state-of-the-art LLMs reveal that the task remains challenging; notably, augmenting models with unrestricted web search degrades performance compared to closed-book baselines due to the retrieval of noisy, non-authoritative precedents. We release CaseFacts to spur research into legal fact verification systems.

Paper Type: Long

Research Area: Resources and Evaluation

Research Area Keywords: Legal NLP, Fact checking, Benchmarking, NLP datasets, Evaluation methodologies, Passage retrieval, Dense retrieval

Contribution Types: Data resources

Languages Studied: English

Submission Number: 8661