A spherical analysis of Adam with Batch Normalization

Simon Roburin, Yann Dubois de Mont-Marin, Andrei Bursuc, Renaud Marlet, Patrick Perez, Mathieu Aubry

Keywords: Deep Learning, Machine Learning, Adam, Batch Normalization

Abstract: Batch Normalization (BN) is a prominent deep learning technique. In spite of its apparent simplicity, its implications over optimization are yet to be fully understood. While previous studies mostly focus on the interaction between BN and stochastic gradient descent (SGD), we develop a geometric perspective which allows us to precisely characterize the relation between BN and Adam. More precisely, we leverage the radial invariance of groups of parameters, such as filters for convolutional neural networks, to translate the optimization steps on the $L_2$ unit hypersphere. This formulation and the associated geometric interpretation shed new light on the training dynamics. Firstly, we use it to derive the first effective learning rate expression of Adam. Then we show that, in the presence of BN layers, performing SGD alone is actually equivalent to a variant of Adam constrained to the unit hypersphere. Finally, our analysis outlines phenomena that previous variants of Adam act on and we experimentally validate their importance in the optimization process.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

One-sentence Summary: Analysis of the interaction between BN and Adam during the training process.

Reviewed Version (pdf): https://openreview.net/references/pdf?id=gBXijSxThp