Learning Attentive Cooperation in Multi-agent Reinforcement Learning with Graph Convolutional Network

Yuqi Zhu, Yuntao Liu, Xinhai Xu, Yuan Li

Abstract: Cooperative multi-agent reinforcement learning (MARL) is a key tool for addressing many real-world problems and is becoming a growing concern due to communication constraints during execution and partial observation. In prior works, the popular paradigm of centralized training with decentralized execution (CTDE) is used to mitigate this problem. However, the intensity of the cooperative relationship is not paid too much attention. In this paper, we propose an Attentive Cooperative MARL framework based on the Neighborhood Graph Convolutional Network (AttCoop-Q) to help agents communicate with each other and generate cooperative features for attentive cooperative policy learning. AttCoop-Q consists of the neighborhood graph convolutional network (NGCN) module and the attentive cooperation policy learning module. NGCN encodes the current situation, constructs a neighboring agent graph and uses the architecture of Neighborhood Graph Convolutional Network (GCN) to extracts cooperative features. The attentive cooperation policy learning module generates weight vectors using cooperative features to generate adapted Q-values which are further used to learn the total Q value. Experimental results on challenging multi-agent StarCraft benchmark tasks show that our proposed method greatly boosts the performance compared with other popular MARL approaches.