Scheduled Restart Momentum for Accelerated Stochastic Gradient Descent

Bao Wang, Tan Minh Nguyen, Tao Sun, Andrea Bertozzi, Richard Baraniuk, Stanley Osher

Keywords: Nesterov Accelerated Gradient, Deep Learning, Image Classification

Abstract: Stochastic gradient descent (SGD) algorithms, with constant momentum and its variants such as Adam, are the optimization methods of choice for training deep neural networks (DNNs). There is great interest in speeding up the convergence of these methods due to their high computational expense. Nesterov accelerated gradient (NAG) with a time-varying momentum, denoted as NAG below, improves the convergence rate of gradient descent (GD) for convex optimization using a specially designed momentum; however, it accumulates error when an inexact gradient is used (such as in SGD), slowing convergence at best and diverging at worst. In this paper, we propose scheduled restart SGD (SRSGD), a new NAG-style scheme for training DNNs. SRSGD replaces the constant momentum in SGD by the increasing momentum in NAG but stabilizes the iterations by resetting the momentum to zero according to a schedule. Using a variety of models and benchmarks for image classification, we demonstrate that, in training DNNs, SRSGD significantly improves convergence and generalization; for instance, in training ResNet-200 for ImageNet classification, SRSGD achieves an error rate of 20.93% vs. the benchmark of 22.13%. These improvements become more significant as the network grows deeper. Furthermore, on both CIFAR and ImageNet, SRSGD reaches similar or even better error rates with significantly fewer training epochs compared to the SGD baseline.

One-sentence Summary: We propose a Nesterov Accelerated Gradient-style momentum to improve training deep neural networks.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Community Implementations: [ 5 code implementations](https://www.catalyzex.com/paper/arxiv:2002.10583/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=_TqZsXVORTy