Scalable Constrained Multi-Agent Reinforcement Learning via State Augmentation and Consensus for Separable Dynamics

ICLR 2026 Conference Submission19390 Authors

Keywords: Constrained Multi-Agent Reinforcement Learning, Decentralized Algorithms, Consensus, Distributed Optimization, Scalable Multi-Agent Coordination, Reinforcement Learning with Constraints, Smart Grid Management

TL;DR: We propose a decentralized multi-agent reinforcement learning algorithm that uses a consensus mechanism to satisfy global constraints while each agent independently optimizes local rewards, and demonstrate its scalability in a smart grid application.

Abstract: We present a distributed approach for constrained Multi Agent Reinforcement Learning (MARL) which combines learning of policies with augmented state and distributed coordination of dual variables through consensus. Our method addresses a specific class of problems in which the agents have separable dynamics and local observations, but need to collectively satisfy constraints on global resources. The main technical contribution of the paper consists of the integration of constrained single agent RL (with state augmentation) in a multi-agent environment, through a distributed consensus over the Lagrange multipliers. This enables independent training of policies while maintaining coordination during execution. Unlike other centralized training with decentralized execution (CTDE) approaches that scale sub optimally with the number of agents, our method achieves a linear scaling both in training and execution by exploiting the separable structure of the problem. Each agent trains an augmented policy with local estimates of the global dual variables, and then coordinates through neighbor to neighbor communication on an undirected graph to reach consensus on constraint satisfaction. We show that, under mild connectivity assumptions, the agents obtain a bounded consensus error, ensuring a collective near-optimal behaviour. Experiments on demand response in smart grids show that our consensus mechanism is critical for feasibility: without it, the agents postpone demand indefinitely despite meeting consumption constraints.

Primary Area: reinforcement learning

Submission Number: 19390