DISTRIBUTED MULTI-AGENT DEEP REINFORCEMENT LEARNING

ICLR 2026 Conference Submission20284 Authors

Keywords: Multi-Agent Reinforcement Learning, Graph Attention Networks, Multi-Agent Systems

TL;DR: We propose a Distributed Multi-Agent Reinforcement Learning algorithm based on Distributed Graph Attention Networks that enable scalable coordination among heterogeneous agents without any centralized training or privileged information.

Abstract: Centralized training with decentralized execution (CTDE) has been the dominant paradigm in multi-agent reinforcement learning (MARL), but its reliance on global state information during training introduces scalability, robustness, and generalization bottlenecks. Moreover, in practical scenarios such as adding/dropping teammates or facing environment dynamics that differ from the training, CTDE methods can be brittle and costly to retrain, whereas distributed approaches allow agents to adapt using only local information and peer-to-peer communication. We present a distributed MARL framework that removes the need for centralized critics or global information. Firstly, we develop a novel Distributed Graph Attention Network (D-GAT) that performs global state inference through multi-hop communication, where agents integrate neighbor features via input-dependent attention weights in a fully distributed manner. Leveraging D-GAT, we develop the distributed graph-attention MAPPO (DG-MAPPO) -- a distributed MARL framework where agents optimize local policies and value functions using local observations, multi-hop communication, and shared/averaged rewards. Empirical evaluation on the StarCraftII Multi-Agent Challenge (SMAC) demonstrates that our method consistently outperforms strong CTDE baselines, achieving superior coordination across a wide range of cooperative tasks with both homogeneous and heterogeneous teams. Our distributed MARL framework offers a principled and scalable solution for robust collaboration without requiring centralized training or global observability. To the best of our knowledge, DG-MAPPO appears to be the first to fully eliminate reliance on privileged centralized information, enabling agents to learn and act solely through peer-to-peer communication.

Primary Area: reinforcement learning

Submission Number: 20284