Learning to Coordinate in Multi-Agent Systems: A Coordinated Actor-Critic Algorithm and Finite-Time Guarantees

Siliang Zeng, Tianyi Chen, Alfredo Garcia, Mingyi Hong

Keywords: Reinforcement Learning, Multi-Agent System, Optimization

Abstract: Multi-agent reinforcement learning (MARL) has attracted much research attention recently. However, unlike its single-agent counterpart, many theoretical and algorithmic aspects of MARL have not been well-understood. In this paper, we study the emergence of coordinated behavior by autonomous agents using an actor-critic (AC) algorithm. Specifically, we propose and analyze a class of coordinated actor-critic algorithms (CAC) in which individually parametrized policies have a shared part (which is jointly optimized among all agents) and a personalized part (which is only locally optimized). Such kind of partially personalized policy allows agents to learn to coordinate by leveraging peers’ past experience and adapt to individual tasks. The flexibility in our design allows the proposed MARL-CAC algorithm to be used in a fully decentralized setting, where the agents can only communicate with their neighbors, as well as a federated setting, where the agents occasionally communicate with a server while optimizing their (partially personalized) local models. Theoretically, we show that under some standard regularity assumptions, the proposed MARL-CAC algorithm requires $\mathcal{O}(\epsilon^{-\frac{5}{2}})$ samples to achieve an$\epsilon$-stationary solution (defined as the solution whose squared norm of the gradient of the objective function is less than$\epsilon$). To the best of our knowledge, this work provides the first finite-sample guarantee for decentralized AC algorithm with partially personalized policies.

One-sentence Summary: This paper develops a novel collaboration mechanism for designing robust MARL systems with theoretical guarantees.