Revisiting the Noise Model of Stochastic Gradient Descent
Abstract: The stochastic gradient noise (SGN) is a significant factor in the success of stochastic gradient descent (SGD).
Following the central limit theorem, SGN was initially modeled as Gaussian, and lately, it has been suggested that stochastic gradient noise is better characterized using $S\alpha S$ Lévy distribution.
This claim was allegedly refuted and rebounded to the previously suggested Gaussian noise model.
This paper presents solid, detailed empirical evidence that SGN is heavy-tailed and better depicted by the $S\alpha S$ distribution. Furthermore, we argue that different parameters in a deep neural network (DNN) hold distinct SGN characteristics throughout training. To more accurately approximate the dynamics of SGD near a local minimum, we construct a novel framework in $\mathbb{R}^N$, based on Lévy-driven stochastic differential equation (SDE), where one-dimensional Lévy processes model each parameter in the DNN. Next, we study the effect of learning rate decay (LRdecay) on the training process. We demonstrate theoretically and empirically that its main optimization advantage stems from the reduction of the SGN. Based on our analysis, we examine the mean escape time, trapping probability, and more properties of DNNs near local minima. Finally, we prove that the training process will likely exit from the basin in the direction of parameters with heavier tail SGN. We will share our code for reproducibility.
Submission Length: Regular submission (no more than 12 pages of main content)
Changes Since Last Submission: Major changes and new results are highlighted in red in the manuscript.
Assigned Action Editor: ~Changyou_Chen1
Submission Number: 292
Loading