Revisiting Neighbourhoods in Mean Field Reinforcement Learning

Sriram Ganapathi Subramanian, Matthew E. Taylor, Kate Larson, Mark Crowley, Pascal Poupart

Keywords: Mean Field Reinforcement Learning, Multi-agent Reinforcement Learning, Reinforcement Learning, Neighbourhood, Attention Mechanism

TL;DR: This paper relaxes the neighbourhood assumption in Mean Field Reinforcement Learning, providing practical algorithms that can be used in real-world environments without strict notion of neighbourhoods.

Abstract: Many multi-agent reinforcement learning (MARL) algorithms do not scale well as the number of agents increases due to an exponential time and space complexity dependency on the number of agents in the environment. Mean field theory has been used to address this problem by approximating the effect of neighbourhoods of agents by a single representative agent. While this approximation allows MARL algorithms to scale to environments with many agents, approaches typically assumed that agents 1) inside a neighbourhood are homogeneous, and 2) outside a neighbourhood have no influence (and can therefore be ignored). This paper relaxes these assumptions and proposes a novel framework, mean field attention (MFA), which uses an attention mechanism for local responses and the mean field approximation for global responses. We implement MFA with two new algorithms leveraging Q-learning and actor-critic. These novel MFA algorithms consistently outperform other MARL algorithms, including prior mean field-based algorithms, across multiple metrics and benchmarks.

Area: Learning and Adaptation (LEARN)

Generative A I: I acknowledge that I have read and will follow this policy.

Submission Number: 863