Cyclicity-Regularized Coordination Graphs

Oliver Järnefelt, Mahdi Kallel, Carlo D'Eramo

Keywords: Multi-agent reinforcement learning, Sparse coordination graphs, Graph cyclicity

TL;DR: A novel sparsification scheme for learning sparse coordination graphs in Multi-Agent Reinforcement Learning which demonstrates the importance of topology awareness during sparsification.

Abstract: In parallel with the rise of the successful value function factorization approach, numerous recent studies on Cooperative Multi-Agent Reinforcement Learning (MARL) have explored the application of Coordination Graphs (CG) to model the communication requirements among the agent population. These coordination problems often exhibit structural sparsity, which facilitates accurate joint value function learning with CGs. Value-based methods necessitate the computation of argmaxes over the exponentially large joint action space, leading to the adoption of the max-sum method from the distributed constraint optimization (DCOP) literature. However, it has been empirically observed that the performance of max-sum deteriorates with an increase in the number of agents, attributed to the increased cyclicity of the graph. While previous works have tackled this issue by sparsifying the graph based on a metric of edge importance, thereby demonstrating improved performance, we argue that neglecting topological considerations during the sparsification procedure can adversely affect action selection. Consequently, we advocate for the explicit consideration of graph cyclicity alongside edge importances. We demonstrate that this approach results in superior performance across various challenging coordination problems.

Already Accepted Paper At Another Venue: already accepted somewhere else

Submission Number: 46