Normalized Direction-preserving Adam

Zijun Zhang, Lin Ma, Zongpeng Li, Chuan Wu

Abstract: Optimization algorithms for training deep models not only affects the convergence rate and stability of the training process, but are also highly related to the generalization performance of trained models. While adaptive algorithms, such as Adam and RMSprop, have shown better optimization performance than stochastic gradient descent (SGD) in many scenarios, they often lead to worse generalization performance than SGD, when used for training deep neural networks (DNNs). In this work, we identify two problems regarding the direction and step size for updating the weight vectors of hidden units, which may degrade the generalization performance of Adam. As a solution, we propose the normalized direction-preserving Adam (ND-Adam) algorithm, which controls the update direction and step size more precisely, and thus bridges the generalization gap between Adam and SGD. Following a similar rationale, we further improve the generalization performance in classification tasks by regularizing the softmax logits. By bridging the gap between SGD and Adam, we also shed some light on why certain optimization algorithms generalize better than others.

TL;DR: A tailored version of Adam for training DNNs, which bridges the generalization gap between Adam and SGD.

Keywords: optimization, generalization, Adam, SGD

Code: [ zj10/ND-Adam](https://github.com/zj10/ND-Adam)

Data: [CIFAR-10](https://paperswithcode.com/dataset/cifar-10), [CIFAR-100](https://paperswithcode.com/dataset/cifar-100)

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/normalized-direction-preserving-adam/code)