Optimizing $(L_0, L_1)$-Smooth Functions by Gradient Methods

Daniil Vankov, Anton Rodomanov, Angelia Nedich, Lalitha Sankar, Sebastian U Stich

Keywords: $(L_0, L_1)$-smoothness, gradient methods, convex optimization, worst-case complexity bounds, acceleration, Polyak stepsizes, nonconvex optimization

Abstract:

We study gradient methods for optimizing $(L_0, L_1)$-smooth functions, a
class that generalizes Lipschitz-smooth functions and has gained attention for
its relevance in machine learning.
We provide new insights into the structure of this function class and develop
a principled framework for analyzing optimization methods in this setting.
While our convergence rate estimates recover existing results for minimizing
the gradient norm in nonconvex problems, our approach significantly improves
the best-known complexity bounds for convex objectives.
Moreover, we show that the gradient method with Polyak stepsizes and the
normalized gradient method achieve nearly the same complexity guarantees as
methods that rely on explicit knowledge of $(L_0, L_1)$.
Finally, we demonstrate that a carefully designed accelerated gradient
method can be applied to $(L_0, L_1)$-smooth functions, further improving all
previous results.

Primary Area: optimization

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 13268