AdaGrad stepsizes: sharp convergence over nonconvex landscapes

Rachel Ward, Xiaoxia Wu, Léon Bottou

Abstract: Adaptive gradient methods such as AdaGrad and its variants update the stepsize in stochastic gradient descent on the fly according to the gradients received along the way; such methods have gained ...