Networked Communication for Decentralised Agents in Mean-Field Games

TMLR Paper5527 Authors

Abstract: We introduce networked communication to the mean-field game framework, in particular to oracle-free settings where $N$ decentralised agents learn along a single, non-episodic run of the empirical system. We prove that our architecture has sample guarantees bounded between those of the centralised- and independent-learning cases. We provide the order of the difference in these bounds in terms of network structure and number of communication rounds, and also contribute a policy-update stability guarantee. We discuss how the sample guarantees of the three theoretical algorithms do not actually result in practical convergence times. We thus contribute practical enhancements to all three algorithms allowing us to present their first empirical demonstrations, where we do not need to enforce several of the theoretically required assumptions. We then show that in practical settings where the theoretical hyperparameters are not observed (leading to poor estimation of the Q-function), our communication scheme considerably accelerates learning over the independent case, which hardly seems to learn at all. Indeed networked agents often perform similarly to the centralised case, while removing the restrictive assumption of the latter. We provide ablations and additional studies showing that our networked approach also has advantages over both alternatives in terms of robustness to update failures and to changes in population size.

Submission Length: Long submission (more than 12 pages of main content)

Assigned Action Editor: ~Alec_Koppel1

Submission Number: 5527