Learning to Communicate using Contrastive Learning

Yat Long Lo, Biswa Sengupta, Jakob Nicolaus Foerster, Michael Noukhovitch

Keywords: Reinforcement Learning, Multi-Agent Reinforcement Learning, Multi-Agent Communication

TL;DR: A novel approach and perspective to decentralized communication learning in MARL based on contrastive learning with a suite of evaluation methods (e.g. protocol symmetry, representation probing and zero-shot communication) to analyze protocols.

Abstract: Communication is a powerful tool for coordination in multi-agent RL. Inducing an effective, common language has been a difficult challenge, particularly in the decentralized setting. In this work, we introduce an alternative perspective where communicative messages sent between agents are considered as different incomplete views of the environment state. Based on this perspective, we propose to learn to communicate using contrastive learning by maximizing the mutual information between messages of a given trajectory. In communication-essential environments, our method outperforms previous work in both performance and learning speed. Using qualitative metrics and representation probing, we show that our method induces more symmetric communication and captures task-relevant information from the environment. Finally, we demonstrate promising results on zero-shot communication, a first for MARL. Overall, we show the power of contrastive learning, and self-supervised learning in general, as a method for learning to communicate.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Reinforcement Learning (eg, decision and control, planning, hierarchical RL, robotics)

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2307.01403/code)