Don’t "Overthink'" Pointwise Reranking: Is Reasoning Truly Necessary?

ICLR 2026 Conference Submission16133 Authors

Keywords: passage retrieval, reranking, reasoning, large language models

TL;DR: In this work, we study whether the generation of a reasoning chain prior to making a relevance prediction actually improves the accuracy of pointwise rerankers.

Abstract: With the growing success of reasoning models across complex natural language tasks, researchers in the Information Retrieval (IR) community have begun exploring how similar reasoning capabilities can be integrated into passage rerankers built on Large Language Models (LLMs). These methods typically employ an LLM to produce an explicit, step-by-step reasoning process before arriving at a final relevance prediction. But, *does reasoning actually improve pointwise reranking accuracy?* In this paper, we dive deeper into this question, studying the impact of the reasoning process by comparing reasoning-based pointwise rerankers (Rank1) to standard, non-reasoning pointwise rerankers (StandardRanker) under identical training conditions, and observe that StandardRanker generally outperforms Rank1. Building on this observation, we then study the importance of reasoning to Rank1 by disabling its reasoning process (Rank1-NoReason), and find that Rank1-NoReason is surprisingly more effective than Rank1. Examining the cause of this result, our findings reveal that pointwise reasoning rerankers are bottlenecked by the LLM's reasoning process, which pushes it toward polarized relevance scores and thus fails to consider the *partial* relevance of passages, a key factor for the accuracy of pointwise rerankers. The source code is in the supplementary materials and will be made public upon acceptance.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 16133