PilotRL: Training Language Model Agents via Global Planning-Guided Progressive Reinforcement Learning

Keer Lu, Chong Chen, Leng Huang, Bin CUI, Wentao Zhang

Keywords: Large Language Models, Agent Paradigm, Reinforcement Learning

TL;DR: We propose PilotRL, a global planning-guided training framework for LLM agents driven by progressive reinforcement learning.

Abstract: Large Language Models (LLMs) have shown remarkable advances in the ability to tackle agent-oriented tasks. Despite their potential, existing work faces challenges when deploying LLMs in agent-based environments. The widely adopted agent paradigm ReAct centers on the integration of single-step reasoning with immediate action execution, which limits its effectiveness in complex tasks that require long-term strategic planning. Furthermore, the coordination between the planner and executor during problem-solving is also a critical factor to consider in agent design. Additionally, current approaches predominantly rely on supervised fine-tuning, which often leads models to memorize established task completion trajectories, thereby restricting their generalization ability when confronted with novel problem contexts. To address these challenges, we introduce an adaptive global plan-based agent paradigm **AdaPlan**, aiming to synergize high-level explicit guidance with execution to support effective long-horizon decision-making. Based on the proposed paradigm, we further put forward **PilotRL**, a global planning-guided training framework for LLM agents driven by progressive reinforcement learning. We first develop the model’s ability to follow explicit guidance from global plans when addressing agent tasks. Subsequently, on the basis of this foundation, we focus on optimizing the quality of generated plans. Finally, we conduct joint optimization of the model's planning and execution coordination. Experiments indicate that **PilotRL** could achieve state-of-the-art performances, with LLaMA3.1-8B-Instruct + PilotRL surpassing closed-sourced GPT-4o by 3.60%, while showing a more substantial gain of 55.78% comparing to GPT-4o-mini at a comparable parameter scale.

Primary Area: reinforcement learning

Submission Number: 6639