Extrapolation for Large-batch Training in Deep Learning

Tao Lin, Lingjing Kong, Sebastian U. Stich, Martin Jaggi

Abstract: Deep learning networks are typically trained by Stochastic Gradient Descent (SGD) methods that iteratively improve the model parameters by estimating a gradient on a very small fraction of the trai...