Can SGD Handle Heavy-Tailed Noise?

Ilyas Fatkhullin, Florian Hübler, Guanghui Lan

Keywords: SGD, heavy-tailed noise, infinite variance, lower bounds, in probability

TL;DR: We prove vanilla SGD achieves minimax-optimal convergence under heavy-tailed noise across convex, strongly convex, and non-convex settings.

Abstract: Stochastic Gradient Descent (SGD) is a cornerstone of large-scale optimization, yet its theoretical behavior under heavy-tailed noise---common in modern machine learning and reinforcement learning---remains poorly understood. In this work, we rigorously investigate whether vanilla SGD, devoid of any adaptive modifications, can provably succeed under such adverse stochastic conditions. Assuming only that stochastic gradients have bounded $p$-th moments for some $p \in (1, 2]$, we establish sharp convergence guarantees for (projected) SGD across convex, strongly convex, and non-convex problem classes. In particular, we show that SGD achieves minimax optimal sample complexity under minimal assumptions in the convex and strongly convex regimes: $\mathcal{O}(\varepsilon^{-\frac{p}{p-1}})$ and $\mathcal{O}(\varepsilon^{-\frac{p}{2(p-1)}})$, respectively. For non-convex objectives under Hölder smoothness, we prove convergence to a stationary point with rate $\mathcal{O}(\varepsilon^{-\frac{2p}{p-1}})$, and complement this with a matching lower bound specific to SGD.

Submission Number: 87