Detached Error Feedback for Distributed SGD with Random Sparsification

An Xu, Heng Huang

Abstract: The communication bottleneck has been a critical problem in large-scale distributed deep learning. In this work, we study distributed SGD with random block-wise sparsification as the gradient compr...