Provable distributed adaptive temporal-difference learning over time-varying networks

Zhu Junlong, Bing Li, lin Wang, Mingchuan Zhang, Ling Xing, Jiangtao Xi, Qingtao Wu

Abstract: Multi-agent reinforcement learning (MARL) has been successfully applied in many fields. In MARL, the policy evaluation problem is one of crucial problems. In order to solve this problem, distributed Temporal-Difference (TD) learning algorithm is one of the most popular methods in a cooperative manner. Despite its empirical success, however, the theory of the adaptive variant of distributed TD learning still remain limited. To fill this gap, we propose an adaptive distributed temporal-difference algorithm (referred to as $MS-ADTD$) under Markovian sampling over time-varying networks. Furthermore, we rigorously analyze the convergence of $MS-ADTD$, the theoretical results show that the local estimation can converge linearly to the optimal neighborhood. Meanwhile, the theoretical results are verified by simulation experiments.