Variance-Reduced Normalized Zeroth Order Method for Generalized-Smooth Non-Convex Optimization

Yuxing Peng, Yuanyuan Liu, Fanhua Shang, Hongying Liu, Zhouchen Lin

Keywords: Non-convex Optimization, generalized smooth, zero-order, gradient-free, $(L_{0}, L_{1})$-smooth

TL;DR: We first present an analysis of the zeroth-order method under the $(L_{0}, L_{1})$-smooth condition。

Abstract:

The generalized smooth condition, $(L_{0}, L_{1})$-smoothness, has triggered people’s interest since it is more realistic in many optimization problems shown by both empirical and theoretical evidence. To solve the generalized smooth optimization, gradient clipping methods are often employed, and have theoretically been shown to be as effective as the traditional gradient-based methods\citep{Chen_2023, xie2024}. However, whether these methods can be safely extended to zeroth-order case is still unstudied. To answer this important question, we propose a zeroth-order normalized gradient method(ZONSPIDER) for both finite sum and general expectation case, and we prove that we can find $\epsilon$- stationary point of $f(x)$ with optimal decency on $d$ and $\epsilon$, specifically, the complexes are $\mathcal{O}(d\epsilon^{-2}\sqrt{n}\max\{L_{0}, L_{1}\})$ in the finite sum case and $\mathcal{O}(d\epsilon^{-3}\max\{\sigma_{1}^{2}, \sigma_{0}^{2}\}\max\{L_{0}, L_{1}\})$ in the general expectation case.
To the best of our knowledge, this is the first time that sample complexity bounds are established for a zeroth-order method under generalized smoothness.

Primary Area: optimization

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 10327