Parameter-Free Variance Reduced Zeroth-Order Optimization for Non-Convex Problems

Yuxing Peng, Yuanyuan Liu, Fanhua Shang, Hongying Liu

Keywords: zeroth-order method, parameter-free, non-convex optimization

TL;DR: The first parameter-free zeroth-order optimization method with variance reduction for non-convex problems without tuning

Abstract:

eroth-order optimization has become a vital tool for solving black-box learning problems where explicit gradients are unavailable. However, standard zeroth-order methods typically require careful tuning of algorithmic parameters such as the smoothing parameter and step size, which limits their practicality. In this paper, we propose PF-VRZO(Parameter free variance reduced zeroth-order methods), a novel parameter-free variance-reduced zeroth-order optimization framework for nonconvex finite-sum problems. Our method only requires minimal input information—problem dimension $d$ and sample size $n$—and adaptively adjusts the smoothing and step size parameters during the optimization process. We develop two algorithmic variants based on coordinate-wise and random-direction gradient estimators, respectively. We establish non-asymptotic convergence guarantees showing that PF-VRZO achieves function query complexity of $\widetilde{\mathcal{O}}(d\sqrt{n}\epsilon^{-2})$ for finding stationary points. Additionally, we conduct experiments on non-convex phase retrieval and distributional robust optimization to validate the effectiveness of our method.
To the best of our knowledge, PF-VRZO is the first parameter-free zeroth-order algorithm that incorporates variance reduction techniques tailored specifically for nonconvex optimization problems.

Primary Area: optimization

Submission Number: 10092