Improving Stability of Parameter Sharing in Cooperative Multi-agent Reinforcement Learning

Yurui Li, Li Zhang, Shijian Li, Gang Pan

Abstract: Parameter sharing is widely employed in policy networks within multi-agent reinforcement learning (MARL) methods, enhancing efficiency and performance in specific tasks. However, this approach also introduces significant challenges, particularly poor stability. In this study, we use QMIX [10] as a case study to investigate the stability issue inherent in parameter sharing. Our analysis reveals that this issue stem from the inability of parameter sharing networks to learn multiple distinct policies simultaneously. To validate this hypothesis, we introduce a specialized task designed to assess the limitations of parameter sharing in learning diverse policies. Building on these efforts, we propose a novel method to enhance the stability of parameter sharing and implement it with QMIX. Experimental results demonstrate that our approach not only improves stability but also enables QMIX to achieve superior performance across tasks. The source code and the videos of the diversity policies is available on https://github.com/Yurui-Li/ImS.

External IDs:dblp:conf/icann/LiZLP25