A Single Global Merging Suffices: Recovering Centralized Learning Performance in Decentralized Learning

Tongtian Zhu, Tianyu Zhang, Mingze Wang, Zhanpeng Zhou, Can Wang

Published: 05 Mar 2025, Last Modified: 05 Mar 2025ICLR 2025 Workshop Weight Space Learning PosterEveryoneRevisionsBibTeXCC BY 4.0
Track: long paper (up to 8 pages)
Keywords: Model Merging, Decentralized Learning
TL;DR: We discover and theoretically reveal why and when a single global parameter merging at the end of decentralized training can recover the performance of centralized training, even under heterogeneous and communication-constrained settings.
Submission Number: 5