Finite-Sample Analysis of Distributed Q-learning for Multi-Agent Networks

Paulo Heredia, Hasan Ghadialy, Shaoshuai Mou

Abstract: We will provide a finite-sample analysis to the convergence of a distributed algorithm for Q-learning in multi-agent systems, in which each agent is assigned a private reward function and can only communicate with its nearby neighbors under time-varying networks. We will provide an upper bound of the average squared error, which takes into account the time evolution of a consensus error and does not rely on the assumption of independent and identically distributed (i.i.d) samples. In addition, our results indicate that the error of each agent converges to zero asymptotically, supporting empirical results in the literature.