Fixing the Broken Compass: Diagnosing and Improving Inference-Time Reward Modeling

ICLR 2026 Conference Submission16548 Authors

Keywords: Model analysis & interpretability, Reasoning, Inference-time Scaling

TL;DR: This paper develops CRISP, an inference-time method that clusters reasoning paths by answers and adaptively aggregates rewards, yielding accuracy improvements of up to 10% for LLM reasoning tasks.

Abstract: Inference-time scaling techniques have shown promise in enhancing the reasoning capabilities of large language models (LLMs). While recent research has primarily focused on training-time optimization, our work highlights inference-time reward model (RM)-based reasoning as a critical yet overlooked avenue. In this paper, we conduct a systematic analysis of RM behavior across downstream reasoning tasks, revealing three key limitations: (1) RM can impair performance on simple questions, (2) its discriminative ability declines with increased sampling, and (3) high search diversity undermines RM performance. To address these issues, we propose CRISP (Clustered Reward Integration with Stepwise Prefixing), a novel inference-time algorithm that clusters generated reasoning paths by final answers, aggregates reward signals at the cluster level, and adaptively updates prefix prompts to guide generation. Experimental results demonstrate that CRISP significantly enhances LLM reasoning performance, achieving up to 5% accuracy improvement over other RM-based inference methods and an average of 10% gain over advanced reasoning models.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 16548