Communication-efficient Random-Walk Optimizer for Decentralized Learning

Hansi Yang, James Kwok

Primary Area: optimization

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: decentralized optimization, random walk

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract:

Decentralized learning has gained popularity due to its flexibility and the ability to operate without a central server. A popular family of decentralized learning methods is based on random-walk optimization, which is easy to implement and has a low computation cost.
However, random-walk optimization with adaptive optimizers can suffer from high communication cost. In this paper, we propose to address this problem from three directions. First, we eliminate the communication of auxiliary parameters, such as momentum and preconditioner, in adaptive optimizers. We also perform multiple model updates on the same client before sending the model to next client. Additionally, we extend sharpness-aware minimization (SAM) to random-walk optimization to avoid overfitting on local training data. Our theoretical analysis demonstrates that the proposed method can converge faster than existing approaches with the same communication cost. Empirical results on various datasets, communication networks, and network sizes show that the proposed method outperforms existing approaches while significantly reducing communication costs.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3292