Search or Think? Rethinking Iterative RAG from An Entropy Perspective

Yuxin Wang, Fang Shicheng, Bo Wang, Qi Luo, Xuanjing Huang, Yining Zheng, Xipeng Qiu

Keywords: RAG; Reinforcement Learning

Abstract: Retrieval-Augmented Generation (RAG) has emerged as a powerful paradigm for Large Language Models (LLMs) to address knowledge-intensive queries requiring domain-specific or up-to-date information. To handle complex multi-hop questions that are challenging for single-step retrieval, iterative RAG approaches incorporating reinforcement learning have been proposed. However, existing iterative RAG systems typically \textit{think first} to decompose questions without leveraging information about the available retrieval corpus, leading to inefficient retrieval and reasoning chains that cascade into suboptimal performance. In this paper, we introduce Search-Initialized Thinking (SIT), a novel framework that \textit{searches first} before think in iterative RAG systems with contextually relevant retrieved knowledge. From an entropy perspective, we demonstrate that incorporating initial knowledge with search reduces unnecessary exploration during the reasoning process, enabling the model to focus more effectively on relevant information subsets. Extensive experiments on six standard RAG datasets demonstrate that by establishing a stronger reasoning foundation, SIT significantly improves retrieval precision, reduces cascading errors, and enhances both performance and efficiency. Moreover, SIT proves effective as a versatile, training-free inference strategy that scales seamlessly to large models.Generalization tests across diverse datasets and retrieval corpora confirm the robustness of our approach. Overall, SIT advances the state-of-the-art in iterative RAG systems while illuminating the critical interplay between structured reasoning and efficient exploration in reinforcement learning-augmented frameworks.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 522