DRAGONN: Distributed Randomized Approximate Gradients of Neural Networks

Zhuang Wang, Zhaozhuo Xu, Xinyu Crystal Wu, Anshumali Shrivastava, T. S. Eugene Ng

Abstract: Data-parallel distributed training (DDT) has become the de-facto standard for accelerating the training of most deep learning tasks on massively parallel hardware. In the DDT paradigm, the communic...