SING: A Plug-and-Play DNN Learning Technique

Adrien Courtois, Damien Scieur, Jean-michel Morel, Pablo Arias, Thomas Eboli

Keywords: optimization, normalization, standardization, centralization, gradient descent, adam, stabilization, plug-and-play

TL;DR: A technique that can be used to any optimizer based on gradient standardization

Abstract: We propose SING (StabIlized and Normalized Gradient), a plug-and-play technique that improves the stability and generalization of the Adam(W) optimizer. SING is straightforward to implement and has minimal computational overhead, requiring only a layer-wise standardization of the gradients fed to Adam(W) without introducing additional hyper-parameters. We support the effectiveness and practicality of the proposed approach by showing improved results on a wide range of architectures, problems (such as image classification, depth estimation, and natural language processing), and in combination with other optimizers. We provide a theoretical analysis of the convergence of the method, and we show that by virtue of the standardization, SING can escape local minima narrower than a threshold that is inversely proportional to the network's depth.

Submission Number: 5033