HRM: Enhancing Agentic Retrieval Augmented Generation With Hierarchical Reward Modeling

ACL ARR 2025 February Submission2318 Authors

Abstract: Retrieval-Augmented Generation (RAG) systems have emerged as a pivotal methodology for enhancing Large Language Models (LLMs) through the dynamic integration of external knowledge. To further improve RAG's flexibility, Agentic RAG introduces autonomous agents into the workflow. However, Agentic RAG faces several challenges: (1) the success of each step depends on both high-quality planning and accurate searching, (2) the lack of supervision for intermediate reasoning steps, and (3) the exponentially large candidate space for planning and searching. To address these challenges, we propose HRM, a novel framework that decouples planning and search processes using dual value models, enabling independent optimization of planning reasoning and search grounding. Our approach constructs a reasoning tree, where each node represents planning and searching steps. We leverage Monte Carlo Tree Search to efficiently assess the quality of each step. During inference, Hierarchical Beam Search iteratively refines plan and search candidates through reward-guided optimization. Extensive experiments on five datasets, across policy models of varying parameter sizes, demonstrate the effectiveness of our method.

Paper Type: Long

Research Area: Information Retrieval and Text Mining

Research Area Keywords: Agentic RAG, Process Reward Model

Contribution Types: NLP engineering experiment

Languages Studied: English

Submission Number: 2318