MAPPA: Scaling Multiagent Systems with Process Rewards

Ed Li, Junyu Ren, Cat Yan

Keywords: Multiagent Systems, Agentic Systems, Post-training, Reinforcement Learning, Large Language Models, LLM Agents, Process Reward, LLM-as-a-Judge, Distributed Training

TL;DR: We show that dense, context-aware process rewards from AI feedback enable end-to-end finetuning of multiagent systems, opening a new frontier for scaling towards larger systems for complex, long-horizon tasks.

Abstract: While multiagent systems have shown promise for tackling complex tasks via specialization, finetuning multiple agents simultaneously faces two key challenges: (1) credit assignment across agents, and (2) sample efficiency of expensive multiagent rollouts. In this work, we propose finetuning multiagent systems with per-action process rewards from AI feedback (MAPPA) to address both. Through assigning credit to individual agent actions rather than only at task completion, MAPPA enables fine-grained supervision without ground truth labels while extracting maximal training signal from each rollout. We demonstrate our approach on competition math problems and tool-augmented data analysis tasks. On unseen math problems, MAPPA achieves +5.0--17.5pp on AIME and +7.8--17.2pp on AMC. For data analysis tasks, our method improves success rate by +16.7pp while quality metrics improve by up to 47%, validating that per-action supervision can lead to improvements across different multiagent systems on various domains. By addressing these challenges, our work takes a first step toward scaling multiagent systems for complex, long-horizon tasks with minimal human supervision.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 91