Faster Distributed Synchronous SGD with Weak Synchronization

Feb 15, 2018 (modified: Oct 27, 2017) ICLR 2018 Conference Blind Submission readers: everyone Show Bibtex
  • Abstract: Distributed training of deep learning is widely conducted with large neural networks and large datasets. Besides asynchronous stochastic gradient descent~(SGD), synchronous SGD is a reasonable alternative with better convergence guarantees. However, synchronous SGD suffers from stragglers. To make things worse, although there are some strategies dealing with slow workers, the issue of slow servers is commonly ignored. In this paper, we propose a new parameter server~(PS) framework dealing with not only slow workers, but also slow servers by weakening the synchronization criterion. The empirical results show good performance when there are stragglers.
  • Keywords: distributed, deep learning, straggler
0 Replies