A Variational Approach to Mutual Information-Based Coordination for Multi-Agent Reinforcement Learning

Woojun Kim, Whiyoung Jung, Myungsik Cho, Youngchul Sung

Keywords: Multi-Agent Reinforcement Learning, Variational Approach, Mutual Information, Coordination

TL;DR: This paper proposes a new mutual information framework for multi-agent reinforcement learning to coordinate simultaneous actions among multiple agents.

Abstract: In this paper, we propose a new mutual information (MMI) framework for multi-agent reinforcement learning (MARL) to enable multiple agents to learn coordinated behaviors by regularizing the accumulated return with the mutual information between multi-agent actions. By introducing a latent variable to induce nonzero mutual information between multi-agent actions and applying a variational bound, we derive a tractable lower bound on the considered MMI-regularized objective function. Applying policy iteration to maximize the derived lower bound, we propose a practical algorithm named variational maximum mutual information multi-agent actor-critic (VM3-AC). We evaluated VM3-AC for several games requiring coordination, and numerical results show that VM3-AC outperforms other MARL algorithms in multi-agent tasks requiring coordination.