Cyclic and Randomized Stepsizes Invoke Heavier Tails in SGD than Constant Stepsize

Mert Gurbuzbalaban, Yuanhan Hu, Umut Simsekli, Lingjiong Zhu

Abstract: Cyclic and randomized stepsizes are widely used in the deep learning practice and can often outperform standard stepsize choices such as constant stepsize in SGD. Despite their empirical success, not much is currently known about when and why they can theoretically improve the generalization performance. We consider a general class of Markovian stepsizes for learning, which contain i.i.d. random stepsize, cyclic stepsize as well as the constant stepsize as special cases, and motivated by the literature which shows that heaviness of the tails (measured by the so-called ``tail-index”) in the SGD iterates is correlated with generalization, we study tail-index and provide a number of theoretical results that demonstrate how the tail-index varies on the stepsize scheduling. Our results bring a new understanding of the benefits of cyclic and randomized stepsizes compared to constant stepsize in terms of the tail behavior. We illustrate our theory on linear regression experiments and show through deep learning experiments that Markovian stepsizes can achieve even a heavier tail and be a viable alternative to cyclic and i.i.d. randomized stepsize rules.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

We prepared a camera-ready version with the following changes:

- We removed the colored text (which was introduced during the previous revision to highlight the changes with respect to the previous version).
- We added a link to our code.
- We added acknowledgments.

Code: https://github.com/YhHu96/Cyclic-and-Randomized-Stepsizes-Invoke-Heavier-Tails-in-SGD-than-Constant-Stepsize

Assigned Action Editor: ~Sebastian_U_Stich1

License: Creative Commons Attribution 4.0 International (CC BY 4.0)

Submission Number: 1260