High probability error bounds of SGD in unbounded domain

Huang Fang

Abstract: This paper studies the high probability convergence behaviour of the stochastic gradient descent (SGD) method applied to convex problems. The existing tail-bound analysis of SGD relies crucially on assuming the domain of the problem to be bounded. In this work, we show that the bounded domain assumption can be removed for free. That is, we prove SGD in an unbounded domain enjoys the same high probability error bound as the bound established in the bounded domain; SGD converges with rate $O(\log(1/\delta)/\epsilon^2)$ no matter the problem domain is bounded or not. As a by-product, we also prove that the trajectory of SGD is guaranteed to stay in a neighbourhood of the initialization with almost bounded diameter. As simple extensions of our analysis, we further establish the high probability error bounds of the last iterate of SGD and SGD with momentum, respectively.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Optimization (eg, convex and non-convex optimization)