Towards Understanding Linear Value Decomposition in Cooperative Multi-Agent Q-Learning

Jianhao Wang, Zhizhou Ren, Beining Han, Jianing Ye, Chongjie Zhang

Keywords: Multi-agent reinforcement learning, Fitted Q-iteration, Value factorization, Credit assignment

Abstract: Value decomposition is a popular and promising approach to scaling up multi-agent reinforcement learning in cooperative settings. However, the theoretical understanding of such methods is limited. In this paper, we introduce a variant of the fitted Q-iteration framework for analyzing multi-agent Q-learning with value decomposition. Based on this framework, we derive a closed-form solution to the empirical Bellman error minimization with linear value decomposition. With this novel solution, we further reveal two interesting insights: 1) linear value decomposition implicitly implements a classical multi-agent credit assignment called counterfactual difference rewards; and 2) On-policy data distribution or richer Q function classes can improve the training stability of multi-agent Q-learning. In the empirical study, our experiments demonstrate the realizability of our theoretical closed-form formulation and implications in the didactic examples and a broad set of StarCraft II unit micromanagement tasks, respectively.

One-sentence Summary: A theoretical and empirical understanding of cooperative multi-agent Q-learning with linear value decomposition.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=biJe1D7Q6