FactReasoner: A Probabilistic Approach to Long-Form Factuality Assessment for Large Language Models

ACL ARR 2025 February Submission2568 Authors

Abstract: Large langauge models (LLMs) have demonstrated vast capabilities on generative tasks in recent years, yet they struggle with guaranteeing the factual correctness of the generated content. This makes these models unreliable in realistic situations where factually accurate responses are expected. In this paper, we propose FactReasoner, a new factuality assessor that relies on probabilistic reasoning to assess the factuality of a long-form generated response. Specifically, FactReasoner decomposes the response into atomic units, retrieves relevant contexts for them from an external knowledge source, and constructs a joint probability distribution over the atoms and contexts using probabilistic encodings of the logical relationships (entailment, contradiction) between the textual utterances corresponding to the atoms and contexts. FactReasoner then computes the posterior probability of whether atomic units in the response are supported by the retrieved contexts. Our experiments on labeled and unlabeled benchmark datasets demonstrate clearly that FactReasoner improves considerably over state-of-the-art prompt-based approaches in terms of both factual precision and recall.

Paper Type: Long

Research Area: Resources and Evaluation

Research Area Keywords: LLM factuality evaluation, long-form answer evaluation, graphical models, probabilistic reasoning

Languages Studied: English

Submission Number: 2568