Training Deep Networks with Stochastic Gradient Normalized by Layerwise Adaptive Second MomentsDownload PDF

25 Sep 2019 (modified: 24 Dec 2019)ICLR 2020 Conference Blind SubmissionReaders: Everyone
  • Original Pdf: pdf
  • Abstract: We propose NovoGrad, an adaptive stochastic gradient descent method with layer-wise gradient normalization and decoupled weight decay. In our experiments on neural networks for image classification, speech recognition, machine translation, and language modeling, it performs on par or better than well tuned SGD with momentum and Adam/AdamW. Additionally, NovoGrad (1) is robust to the choice of learning rate and weight initialization, (2) works well in a large batch setting, and (3) has two times smaller memory footprint than Adam.
  • Keywords: deep learning, optimization, SGD, Adam, NovoGrad, large batch training
  • TL;DR: NovoGrad - an adaptive SGD method with layer-wise gradient normalization and decoupled weight decay.
7 Replies

Loading