R²AG: Learning to Reason, Retrieve, and Self-Evolve through a Multi-Branch Retrieval Tree

Yeming Jian, Yonggang Zhang, Xueqiong Li, Tongliang Liu, Long Lan, Bo Han

Keywords: Retrieval-Augmented Generation (RAG), Retieval-augmented Retrieval Tree (RRT), Large Language Model, Reinforcement-Learning

Abstract:

Retrieval-Augmented Generation (RAG) boosts large language models (LLMs) with external retrieval. Despite its effectiveness, RAG struggles on complex multi-hop tasks due to the retriever’s inability to recover sufficient evidence chains based on previously retrieved context in single-step search.
To mitigate this, we adopt a step-wise retrieval paradigm that retrieve evidence chains through successive retrievals conditioned on previously retrieved context. We conceptualize this process as constructing a multi-branch retrieval tree, rooted at the original query, where each branch represents a context-dependent retrieval chain. This reformulation motivates a reasoning-driven strategy for tree expansion. Thus, we propose Reasoning-Augmented RAG (R²AG), which post-trains a retrieval steering model via reinforcement learning to drive the reasoning-augmented expansion of a multi-hop retrieval tree (i.e., Reasoning-Augmented Retrieval Tree, RRT), grounded in multi-hop context comprehension.
Yet a new challenge arises: as branches explode and distracting retrievals accumulates, expanding along precise chains in subsequent steps becomes intractable. To address this, we propose Top-Survivor, a method that selects the accurate branches in RRT for further expansion under strict gold-hit based constraints.
Moreover, R²AG adopts a progressively iterative training design, enabling the model to incrementally self-evolve its reasoning capability by constructing and refining RRT through learning from accurate chains.
Extensive experiments demonstrate that our method substantially improves the retrieval quality of RAG. Compared to naive RAG, R²AG yields 24.1\%/20.4\% gains in recall/accuracy, without incurring significant latency.

Primary Area: generative models

Submission Number: 2359