A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks

Umut Simsekli, Levent Sagun, Mert Gürbüzbalaban

Abstract: The gradient noise (GN) in the stochastic gradient descent (SGD) algorithm is often considered to be Gaussian in the large data regime by assuming that the classical central limit theorem (CLT) kic...