Double Momentum Method for Lower-Level Constrained Bilevel Optimization

Wanli Shi, Yi Chang, Bin Gu

Primary Area: optimization

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: bilevel optimization, constrained optimization, nonsmooth

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We propose a double momentum method for lower-level constrained bilevel optimization with convergence analysis

Abstract:

Bilevel optimization (BO) has recently gained prominence in many machine learning applications due to its ability to capture the nested structure inherent in these problems.
Recently, many gradient-based methods have been proposed as effective solutions for solving large-scale problems.
However, current methods for the lower-level constrained bilevel optimization (LCBO) problems lack a solid analysis of convergence rate, primarily because of the non-smooth nature of the solutions to the lower-level problem. What's worse, existing methods require either double-loop updates, which are sometimes
less efficient.
To solve this problem, in this paper, we propose a novel \textit{single-loop single-timescale} method with theoretical guarantees for LCBO problems.
Specifically, we leverage the Gaussian smoothing to design an approximation of the hypergradient.
Then, using this hypergradient, we propose a \textit{single-loop single-timescale} algorithm based on the double-momentum method and adaptive step size method.
Theoretically, we demonstrate that our methods can return a stationary point with $\tilde{\mathcal{O}}(\dfrac{\sqrt{d_2}}{\delta \epsilon^{4}})$ iterations. In addition, experiments on two applications also demonstrate the superiority of our proposed method.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3454