LoDAdaC: a unified local training-based decentralized framework with Adam-type updates and compressed communication

TMLR Paper6265 Authors

Abstract:

In the decentralized distributed learning, achieving fast convergence and low communication cost is essential for scalability and high efficiency. Despite extensive research, existing decentralized methods can either have fast convergence or enjoy low communication cost but cannot achieve both goals simultaneously. This disadvantage causes significant inefficiency (either in computation or communication) in solving large-scale decentralized learning problems,
e.g., in large language model training. To address this limitation, we propose LoDAdaC, a unified multiple \textbf{Lo}cal Training (MLT) \textbf{D}ecentralized framework with \textbf{Ada}m-type updates and \textbf{C}ompressed communication (CC). LoDAdaC accommodates a broad class of optimizers for its local adaptive updates, including AMSGrad, Adam, and AdaGrad; it is compatible with standard (possibly biased) compressors such as low-bit quantization and sparsification. MLT and CC enable LoDAdaC to achieve multiplied reduction of communication cost, while the technique of adaptive updates enables fast convergence. We rigorously prove the combined advantage through complexity analysis. In addition, experiments on image classification and large language model training validate our theoretical findings and show that LoDAdaC significantly outperforms existing decentralized algorithms in terms of convergence speed and communication efficiency.

Submission Length: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Franck_Iutzeler1

Submission Number: 6265