Generalising Multi-Agent Cooperation through Task-Agnostic Communication

Dulhan Jayalath, Steven Morad, Amanda Prorok

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: multi-agent reinforcement learning, multi-agent communication

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We propose a communication strategy for MARL that can be applied to any task within a specific environment, avoiding the need to relearn communication.

Abstract: In cooperative multi-agent reinforcement learning (MARL), existing communication methods are almost exclusively task-specific, necessitating the training of new communication strategies for each unique task. This paper addresses this inherent inefficiency by introducing a task-agnostic, environment-specific communication strategy applicable to any task within a given environment. We pre-train the communication strategy without task-specific reward guidance in a self-supervised manner using a set autoencoder. Our objective is to learn a latent Markov state from a set of local observations, coming from a variable number of agents. Under mild assumptions, we prove that policies using our latent representations are guaranteed to converge, and upper bound the value error introduced by our Markov state approximation. Our method enables seamless adaptation to novel tasks without relearning or fine-tuning the communication strategy, gracefully supports scaling to more agents than present during training, and detects out-of-distribution events in an environment. Empirical results on diverse MARL scenarios validate the effectiveness of our approach, surpassing task-specific communication strategies in unseen tasks.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 2747