IGTO: Individual Global Transform Optimization for Multi-Agent Reinforcement Learning

Xiaoliang Hu, Zhen Cui

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Individual-Global Normalized Transformation, Reinforcement Learning, Multi-agent Cooperative Learning

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract: The rigorous equivalency of individual-global actions is accustomedly assumed for Centralized Training with Decentralized Execution (CTDE) in Multi-Agent Reinforcement Learning (MARL), wherever Individual-Global-Max (IGM) or Individual-Global-Optimal (IGO) it is. To release the restriction, in this work, we pose an individual-global action-transformed condition, named individual-global-Transform-Optimal (IGTO), to permit inconsistent individual-global actions while guaranteeing the equivalency of their policy distributions. Conditioned by IGTO, accordingly, we design a Individual-Global Normalized Transformation (IGNT) rule, which could be seamlessly implanted into many existing CTDE-based algorithms. Theoretically, we prove that individual-global policies can converge to the optimum under this rule. Empirically, we integrate IGNT into Multi-agent Actor-Critic (named IGNT-MAC) as well as various MARL algorithms, then test on StarCraft Multi-Agent Challenge (SMAC) and Multi-Agent Particle Environment (MPE). Extensive experiments demonstrate that our method can achieve remarkable improvement over the existing MARL baselines.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3268