Special Properties of Gradient Descent with Large Learning Rates

Amirkeivan Mohtashami, Martin Jaggi, Sebastian U. Stich

Abstract: When training neural networks, it has been widely observed that a large step size is essential in stochastic gradient descent (SGD) for obtaining superior models. However, the effect of large step ...