Decentralized Deep Learning with Arbitrary Communication Compression

Anastasia Koloskova*, Tao Lin*, Sebastian U Stich, Martin Jaggi

TL;DR: We propose Choco-SGD---decentralized SGD with compressed communication---for non-convex objectives and show its strong performance in various deep learning applications (on-device learning, datacenter case).

Abstract: Decentralized training of deep learning models is a key element for enabling data privacy and on-device learning over networks, as well as for efficient scaling to large compute clusters. As current approaches are limited by network bandwidth, we propose the use of communication compression in the decentralized training context. We show that Choco-SGD achieves linear speedup in the number of workers for arbitrary high compression ratios on general non-convex functions, and non-IID training data. We demonstrate the practical performance of the algorithm in two key scenarios: the training of deep learning models (i) over decentralized user devices, connected by a peer-to-peer network and (ii) in a datacenter.

Code: https://github.com/epfml/ChocoSGD

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:1907.09356/code)