Reinforcing Agentic Search Via Reward Density Optimization

ACL ARR 2026 January Submission3446 Authors

Keywords: deep search, tool-integrated reasoning, search agent, large language models

Abstract:

Reinforcement Learning with Verifiable Rewards (RLVR) is a promising approach for enhancing agentic search.
However, its performance is often hindered by reward sparsity, whereby agents receive very limited positive feedback despite incurring significant exploration costs.
In this paper, we formalize this challenge as a new research problem termed **Reward Density Optimization**, which aims to improve the reward obtained per unit of exploration cost.
To address this problem, we introduce InfoFlow, a systematic framework that operates along three complementary dimensions: 1) **Sub-goal Scaffolding**: which decomposes long-horizon tasks into intermediate objectives and assigns process-level rewards to provide denser learning signals; 2) **Pathfinding Hints**: which injects corrective guidance into stalled trajectories to increase the ratio of successful trials; and 3) **Dual-agent Refinement**: which employs a dual-agent architecture to offload the cognitive burden of deep exploration.
We evaluate InfoFlow on several popular agentic search benchmarks, where it significantly outperforms strong baselines and enables lightweight LLMs to achieve performance comparable to that of advanced proprietary models.

Paper Type: Long

Research Area: Question Answering

Research Area Keywords: deep search,tool-integrated reasoning,search agent,chain-of-thought, LLM/AI agents

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: English

Submission Number: 3446