Democratizing Agentic RAG: Distillation-Guided Policy Optimization for Compact Language Models

Rikuto Kotoge, Mai Nishimura, Jiaxin Ma

Keywords: agent, LLM, reinforcement learning, distillation, retrieval-augmented generation, open-domain QA

TL;DR: We propose DGPO, a distillation-guided RL method that enables compact models to perform agentic RAG, and ARC, a fine-grained evaluation framework.

Abstract: Reinforcement Learning has emerged as a post-training approach to elicit agentic RAG behaviors such as search and planning from language models. However, compact language models (e.g., 0.5B parameters) struggle due to poor reasoning ability, resulting in sparse rewards and unstable training. To overcome these difficulties, we propose Distillation-Guided Policy Optimization (DGPO), which addresses the challenges through cold-start initialization from teacher demonstrations and continuous teacher guidance during policy optimization. To systematically evaluate our approach, we introduce Agentic RAG Capabilities (ARC), a fine-grained metric analyzing reasoning, search coordination, and response synthesis. Comprehensive experiments demonstrate that DGPO enables compact models to achieve sophisticated agentic search behaviors, even surpassing the larger teacher model in some cases. DGPO significantly makes agentic RAG feasible in computing resource-constrained environments.

Submission Type: Research Paper (4-9 Pages)

Submission Number: 32