FG-PRM: Fine-grained Hallucination Detection and Mitigation in Language Model Mathematical Reasoning

ACL ARR 2025 February Submission2814 Authors

Abstract:

Hallucinations in large language models (LLMs) pose significant challenges in tasks requiring complex multi-step reasoning, such as
mathematical problem-solving. Existing approaches primarily detect the presence of hallucinations but lack a nuanced understanding of
their types and manifestations. In this paper, we first introduce a comprehensive taxonomy that categorizes the common hallucinations in mathematical reasoning tasks into six types. We then propose FG-PRM (Fine-Grained Process Reward Model), an augmented model designed to detect and mitigate hallucinations in a fine-grained, step-level manner. To address the limitations of manually labeling training data, we propose an automated method for generating fine-grained hallucination data using LLMs. Our FG-PRM demonstrates superior performance across two key tasks: 1) Fine-grained hallucination detection: classifying hallucination types for each reasoning step; and 2) Verification: ranking multiple LLM-generated outputs to select the most accurate solution. Our experiments show that FG-PRM outperforms ChatGPT-3.5 and Claude-3 on fine-grained hallucination detection and substantially boosts the performance of LLMs on GSM8K and
MATH benchmarks.

Paper Type: Long

Research Area: Question Answering

Research Area Keywords: math QA, logical reasoning, reasoning

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: English

Submission Number: 2814