Can Compact Language Models Search Like Agents? Distillation-Guided Policy Optimization for Preserving Agentic RAG Capabilities

ACL ARR 2026 January Submission8687 Authors

Keywords: distillation, agentic retrieval-augmented generation, reinforcement learning

Abstract:

Reinforcement Learning has emerged as a dominant post-training approach to elicit agentic RAG behaviors such as search and planning from language models.
Despite its success with larger models, applying RL to compact models (e.g., 0.5-1B parameters) presents unique challenges.
The compact models exhibit poor initial performance, resulting in sparse rewards and unstable training.
To overcome these difficulties, we propose Distillation-Guided Policy Optimization (DGPO), which employs cold-start initialization from teacher demonstrations and continuous teacher guidance during policy optimization.
To understand how compact models preserve agentic behavior, we introduce Agentic RAG Capabilities (ARC), a fine-grained metric analyzing reasoning, search coordination, and response synthesis.
Comprehensive experiments demonstrate that DGPO enables compact models to achieve sophisticated agentic search behaviors, even outperforming the larger teacher model in some cases. DGPO makes agentic RAG feasible in computing resource-constrained environments.

Paper Type: Long

Research Area: LLM Efficiency

Research Area Keywords: distillation, retrieval-augmented generation, reinforcement learning, evaluation, open-domain QA

Contribution Types: Model analysis & interpretability, Approaches low compute settings-efficiency

Languages Studied: English

Submission Number: 8687