Value Factorization for Asynchronous Multi-Agent Reinforcement Learning

Enrico Marchesini, Yuchen Xiao, Christopher Amato

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Reinforcement Learning, Asynchronous Multi-Agent Reinforcement Learning, Macro-Actions

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: Our work introduces value factorization in the asynchronous framework to generalize the synchronous case and solve challenging coordination tasks where synchronous methods fail.

Abstract:

Value factorization has become widely used to design high-quality and scalable multi-agent reinforcement learning algorithms. However, existing methods assume agents execute synchronously, which does not align with the asynchronous nature of real-world multi-agent systems. In these systems, agents often make decisions at different times, executing asynchronous (macro-)actions characterized by varying and unknown duration. Our work introduces value factorization to the asynchronous framework. To this end, we formalize the consistency requirement between joint and individual macro-action selection, proving it generalizes the synchronous case. We then propose approaches that use asynchronous centralized information to enable factorization architectures to support macro-actions. We evaluate the resultant asynchronous value factorization algorithms across increasingly complex domains that are standard benchmarks in the macro-action literature. Crucially, the proposed methods scale well in these challenging coordination tasks where their synchronous counterparts fail.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5344