Fully Decentralized Model-based Policy Optimization with Networked Agents

Yuchen Liu, Yali Du, Runji Lin, Hangrui Bi, Mingdong Wu, Jun Wang, Hao Dong

Keywords: Reinforcement learning, model-based, multi-agent, deep learning, networked system control.

Abstract: Model-based RL is an effective approach for reducing sample complexity. However, when it comes to multi-agent setting where the number of agent is large, the model estimation can be problematic due to the exponential increased interactions. In this paper, we propose a decentralized model-based reinforcement learning algorithm for networked multi-agent systems, where agents are cooperative and communicate locally with their neighbors. We analyze our algorithm theoretically and derive an upper bound of performance discrepancy caused by model usage, and provide a sufficient condition of monotonic policy improvement. In our experiments, we compare our algorithm against other strong multi-agent baselines and demonstrate that our algorithm not only matches the asymptotic performance of model-free methods but also largely increases its sample efficiency.

One-sentence Summary: A decentralized multi-agent model-based RL algorithm and a theoretical analysis on its performance.