Learning Mean Field Control on Sparse Graphs

Christian Fabian, Kai Cui, Heinz Koeppl

Abstract: Large agent networks are abundant in applications and nature and pose difficult challenges in the field of multi-agent reinforcement learning (MARL) due to their computational and theoretical complexity. While graphon mean field games and their extensions provide efficient learning algorithms for dense and moderately sparse agent networks, the case of realistic sparser graphs remains largely unsolved. Thus, we propose a novel mean field control model inspired by local weak convergence to include sparse graphs such as power law networks with coefficients above two. Besides a theoretical analysis, we design scalable learning algorithms which apply to the challenging class of graph sequences with finite first moment. We compare our model and algorithms for various examples on synthetic and real world networks with mean field algorithms based on Lp graphons and graphexes. As it turns out, our approach outperforms existing methods in many examples and on various networks due to the special design aiming at an important, but so far hard to solve class of MARL problems.

Lay Summary:

Networks are the building blocks of countless systems in nature and of human-made structures such as the human brain or social networks. Despite their high relevance in real world applications, it is hard to learn optimal behavior in large networks because one has to consider millions or even billions of agents.

The concept of graphon mean field games (GMFGs) and corresponding extensions provide tools to understand and learn in large networks by reducing the large number of individuals to a few representative ones. While GMFGs and their extensions give valuable scientific insights, their assumptions on the average number of connections of an individual are often far from what is observed in real world networks.

In this paper, we overcome these sometimes unrealistic assumptions with our local weak mean field control (LWMFC) model. The LWMFC approach divides all individuals according to their number of connections. Then, we can efficiently learn behavior in large networks and also provide mathematical insights. We illustrate the advantages of LWMFC with different examples, such as two epidemic models and the spreading of rumors in a social network.

Primary Area: Reinforcement Learning->Multi-agent

Keywords: Mean Field Control, Large Graphs, Sparse Networks, Multi-Agent Reinforcement Learning

Submission Number: 10114