Graph Convolutional Value Decomposition in Multi-Agent Reinforcement Learning

Navid Naderializadeh, Fan H. Hung, Sean Soleyman, Deepak Khosla

Keywords: multi-agent deep reinforcement learning, graph neural networks, value function factorization, attention mechanisms

Abstract: We propose a novel framework for value function factorization in multi-agent deep reinforcement learning using graph neural networks (GNNs). In particular, we consider the team of agents as the set of nodes of a complete directed graph, whose edge weights are governed by an attention mechanism. Building upon this underlying graph, we introduce a mixing GNN module, which is responsible for two tasks: i) factorizing the team state-action value function into individual per-agent observation-action value functions, and ii) explicit credit assignment to each agent in terms of fractions of the global team reward. Our approach, which we call GraphMIX, follows the centralized training and decentralized execution paradigm, enabling the agents to make their decisions independently once training is completed. Experimental results on the StarCraft II multi-agent challenge (SMAC) environment demonstrate the superiority of our proposed approach as compared to the state-of-the-art.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: We use a graph neural network for value function factorization and explicit credit assignment in multi-agent deep reinforcement learning.

Reviewed Version (pdf): https://openreview.net/references/pdf?id=huuUs7f1t