Multi-Agent Tool-Integrated Policy Optimization

AAAI 2026 Workshop TrustAgent Submission4 Authors

Keywords: LLM Agent; Multi-Agent; Tool-Integrated LLM;LLM Post-training

TL;DR: An LLM RL post-training algorithm that enables training multiple agent roles within a single LLM.

Abstract:

Large language models (LLMs) increasingly rely on multi-turn tool-integrated planning for knowledge-intensive and complex reasoning tasks.
Existing implementations typically rely on a single agent, but they suffer from limited context length and noisy tool responses.
A natural solution is to adopt a multi-agent framework with planner- and worker-agents to manage context.
However, no existing methods support effective reinforcement learning post-training of tool-integrated multi-agent frameworks.
To address this gap, we propose Multi-Agent Tool-Integrated Policy Optimization (**MATPO**), which enables distinct roles (planner and worker) to be trained within a single LLM instance using role-specific prompts via reinforcement learning.
MATPO is derived from a principled credit assignment mechanism across planner and worker rollouts. This design eliminates the need to deploy multiple LLMs, which would be memory-intensive, while preserving the benefits of specialization.
Experiments on GAIA-text, WebWalkerQA, and FRAMES show that MATPO consistently outperforms single-agent baselines by an average of 18.38 \% relative improvement in performance and exhibits greater robustness to noisy tool responses. Our findings highlight the effectiveness of unifying multiple agent roles within a single LLM and provide practical insights for stable and efficient multi-agent RL training. Our code is available at https://github.com/mzf666/MATPO.

Submission Number: 4