Optimizing $(L_0, L_1)$-Smooth Functions by Gradient Methods

ICLR 2025 Conference Submission13268 Authors

Keywords: $(L_0, L_1)$-smoothness, gradient methods, convex optimization, worst-case complexity bounds, acceleration, Polyak stepsizes, nonconvex optimization

Abstract:

We study gradient methods for solving an optimization problem with an $(L_0, L_1)$-smooth objective function. This problem class generalizes that of Lipschitz-smooth problems and has gained interest recently, as it captures a broader range of machine learning applications.
We provide novel insights on the properties of this function class and develop a general framework for analyzing optimization methods
for $(L_0, L_1)$-smooth function in a principled manner.
While our convergence rate estimates recover existing results for minimizing the gradient norm for nonconvex problems,
our approach allows us to significantly improve the current state-of-the-art complexity results in the case of convex problems.
We show that both the gradient method with Polyak stepsizes and the normalized gradient method, without any knowledge of the parameters $L_0$ and $L_1$, achieve the same complexity bounds as the method with the knowledge of these constants.
In addition to that, we show that a carefully chosen accelerated gradient method can be applied to $(L_0, L_1)$-smooth functions, further improving previously known results.
In all cases, the efficiency bounds we establish do not have an exponential dependency on $L_0$ or $L_1$, and do not depend on the initial gradient norm.

Primary Area: optimization

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 13268