Breaking the Curse of Multiagency in Robust Multi-Agent Reinforcement Learning

Laixi Shi, Jingchu Gai, Eric Mazumdar, Yuejie Chi, Adam Wierman

TL;DR: We propose a new class of robust Markov games and propose the first provable algorithm to break the curse of multiagency of the sample complexity for solving robust Markov games, regardless of the uncertainty set formulation.

Abstract: Standard multi-agent reinforcement learning (MARL) algorithms are vulnerable to sim-to-real gaps. To address this, distributionally robust Markov games (RMGs) have been proposed to enhance robustness in MARL by optimizing the worst-case performance when game dynamics shift within a prescribed uncertainty set. RMGs remains under-explored, from reasonable problem formulation to the development of sample-efficient algorithms. Two notorious and open challenges are the formulation of the uncertainty set and whether the corresponding RMGs can overcome the curse of multiagency, where the sample complexity scales exponentially with the number of agents. In this work, we propose a natural class of RMGs inspired by behavioral economics, where each agent's uncertainty set is shaped by both the environment and the integrated behavior of other agents. We first establish the well-posedness of this class of RMGs by proving the existence of game-theoretic solutions such as robust Nash equilibria and coarse correlated equilibria (CCE). Assuming access to a generative model, we then introduce a sample-efficient algorithm for learning the CCE whose sample complexity scales polynomially with all relevant parameters. To the best of our knowledge, this is the first algorithm to break the curse of multiagency for RMGs, regardless of the uncertainty set formulation.

Lay Summary: Multi-agent reinforcement learning (MARL) involves training multiple decision-makers (agents), such as robots or self-driving cars, to work together or compete in dynamic environments. While these systems work well in simulations, they often fail when deployed in the real world due to uncertainty in the environment and the behaviors of other agents. To address this, distributionally robust Markov games (RMGs) have been proposed to enhance robustness in MARL by optimizing the worst-case performance when game dynamics shift within a prescribed uncertainty set. RMGs remain under-explored, from reasonable problem formulation to the development of sample-efficient algorithms. The authors propose a new, more realistic formulation of uncertainty inspired by behavioral economics and develop a learning algorithm, Robust-Q-FTRL, that can efficiently train agents even as their number increases. This work makes it possible to train robust multi-agent systems that scale efficiently and are more practical for real-world utilization.

Primary Area: Reinforcement Learning->Multi-agent

Keywords: multi-agent reinforcement learning, robust Markov games, game theory, distribution shift, behavioral economics

Submission Number: 5644