Provable Faster Zeroth-order Method for Bilevel Optimization with Optimal Dependency on Error and Dimension
Keywords: Stochastic bilevel optimization, Hessian-free algorithms, near-optimal complexity
TL;DR: We improve the best-known complexity of fully zeroth-order bilevel optimization.
Abstract: In this paper, we study and analyze zeroth-order stochastic approximation algorithms for solving black-box bilevel optimization problems, where only the upper and lower function values can be obtained. \citep{Saeed2024} proposed the first full zeroth-order bilevel method that utilizes Gaussian smoothing to estimate the first- and second-order partial derivatives of functions with two independent blocks of variables. However, this method suffers from a high dimensional dependency of $\mathcal{O}((d_{1}+d_{2})^{4})$, where $d_{1}$ and $d_{2}$ are the dimensions of the outer and inner problems, respectively. They left an open question: can this dimension dependency be improved? To answer this question, we propose a single-loop accelerated zeroth-order bilevel algorithm, which achieves a dimension dependency of $\mathcal{O}(d_{1}+d_{2})$ by incorporating coordinate-wise smoothing gradient estimators (coord).
We develop a new theoretical analysis for the proposed algorithm, which converges to a stationary point of $\Phi(x)$ with a complexity of $\mathcal{O}((d_{1}+d_{2})\epsilon^{-3})$ in expectation settings and $\mathcal{O}((d_{1}+d_{2})\sqrt{n}\epsilon^{-2})$ in finite sum settings. These complexities are both best-known with respect to dimension and error $\epsilon$. We also provide experiment to validate the effectiveness of the proposed algorithm.
Primary Area: optimization
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 8398
Loading