The Lessons of Developing Process Reward Models in Mathematical Reasoning

ACL ARR 2025 February Submission6716 Authors

Abstract:

Process Reward Models (PRMs) aim to identify and mitigate intermediate errors in the reasoning processes in mathematical reasoning of Large Language Models (LLMs).
However, the development of effective PRMs faces significant challenges, particularly in data annotation and evaluation methodologies.
In this paper, through extensive experiments, we demonstrate that commonly used Monte Carlo (MC) estimation-based data synthesis for PRMs typically yields inferior performance and generalization compared to LLM-as-a-judge and human annotation methods.
Furthermore, we identify potential biases in conventional Best-of-N (BoN) evaluation strategies for PRMs.
To address these challenges, we develop a consensus filtering mechanism that effectively integrates MC estimation with LLM-as-a-judge and advocates a more comprehensive evaluation framework that combines response-level and step-level metrics. Based on the mechanisms, we significantly improve both model performance and data efficiency in the BoN evaluation and the step-wise error identification task.
Finally, we release a new state-of-the-art PRM that outperforms existing open-source alternatives and provides practical guidelines for future research.

Paper Type: Long

Research Area: Language Modeling

Research Area Keywords: reward model, reasoning, process reward model

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Reproduction study, Publicly available software and/or pre-trained models, Data analysis

Languages Studied: English

Submission Number: 6716