The Noise Geometry of Stochastic Gradient Descent: A Quantitative and Analytical Characterization
Keywords: Stochastic gradient descent, Noise geometry, Loss landscape, Fisher matrix
TL;DR: This paper investigates the noise geometry of the Stochastic Gradient Descent theoretically, analyzing its alignment with the loss landscape and its implications for escaping from minima.
Abstract: Empirical studies have demonstrated that the noise in stochastic gradient descent (SGD) aligns favorably with the local geometry of loss landscape. However, theoretical and quantitative explanations for this phenomenon remain sparse. In this paper, we offer a comprehensive theoretical investigation into the aforementioned {\em noise geometry} for over-parameterized linear (OLMs) models and two-layer neural networks. We scrutinize both average and directional alignments, paying special attention to how factors like sample size and input data degeneracy affect the alignment strength. As a specific application, we leverage our noise geometry characterizations to study how SGD escapes from sharp minima, revealing that the escape direction has significant components along flat directions. This is in stark contrast to GD, which escapes only along the sharpest directions. To substantiate our theoretical findings, both synthetic and real-world experiments are provided.
Submission Number: 21
Loading