An Inexact Conditional Gradient Method for Constrained Bilevel Optimization

Nazanin Abolfazli, Ruichen Jiang, Aryan Mokhtari, Erfan Yazdandoost Hamedani

Primary Area: optimization

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Bilevel Optimization, Conditional Gradient Method

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract:

Bilevel optimization is an important class of optimization problems where one optimization problem is nested within another. This framework is widely used in machine learning problems, including meta-learning, data hyper-cleaning, and matrix completion with denoising. In this paper, we focus on a bilevel optimization problem with a strongly convex lower-level problem and a smooth upper-level objective function over a compact and convex constraint set.
Several methods have been developed for tackling unconstrained bilevel optimization problems, but there is limited work on methods for the constrained setting. In fact, for those methods that can handle constrained problems, either the convergence rate is slow or the computational cost per iteration is expensive. To address this issue, in this paper, we introduce a novel single-loop projection-free method using a nested approximation technique.
Our proposed method has an improved per-iteration complexity, surpassing existing methods, and achieves optimal convergence rate guarantees matching the best-known complexity of projection-free algorithms for solving convex constrained single-level optimization problems. In particular, when the upper-level objective function is convex, our method requires $\tilde{\mathcal{O}}(\epsilon^{-1})$ iterations to find an $\epsilon$-optimal solution. Moreover, when the upper-level objective function is non-convex the complexity of our method is $\mathcal{O}(\epsilon^{-2})$ to find an $\epsilon$-stationary point. We also present numerical experiments to showcase the superior performance of our method compared with state-of-the-art methods.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8319