Mean-Field Sampling for Cooperative Multi-Agent Reinforcement Learning

Emile Timothy Anand, Ishani Karmarkar, Guannan Qu

Keywords: multi-agent reinforcement learning, mean-field RL, online decision making, sampling theory, large-scale systems

TL;DR: We develop and analyze a scalable algorithm for multi-agent RL by sampling from the mean-field distribution of the agents to overcome the curse of dimensionality.

Abstract: Designing efficient algorithms for multi-agent reinforcement learning (MARL) is fundamentally challenging because the size of the joint state and action spaces grows exponentially in the number of agents. These difficulties are exacerbated when balancing sequential global decision-making with local agent interactions. In this work, we propose a new algorithm $\texttt{SUBSAMPLE-MFQ}$ ($\textbf{Subsample}$-$\textbf{M}$ean-$\textbf{F}$ield-$\textbf{Q}$-learning) and a decentralized randomized policy for a system with $n$ agents. For any $k\leq n$, our algorithm learns a policy for the system in time polynomial in $k$. We prove that this learned policy converges to the optimal policy on the order of $\tilde{O}(1/\sqrt{k})$ as the number of subsampled agents $k$ increases. In particular, this bound is independent of the number of agents $n$.

Primary Area: Theory (e.g., control theory, learning theory, algorithmic game theory)

Submission Number: 6109