Bilevel Optimization without Lower-Level Strong Convexity from the Hyper-Objective Perspective

Lesi Chen, Jing Xu, Jingzhao Zhang

Primary Area: learning theory

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Bilevel Optimization

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We provide hardness results, key regular conditions and non-asymptotic analysis for bilevel optimization without lower-level strong convexity.

Abstract:

Bilevel optimization reveals the inner structure of otherwise oblique optimization problems, such as hyperparameter tuning, neural architecture search, and meta-learning. A common goal in bilevel optimization is to find stationary points of the hyper-objective function.
Although this hyper-objective approach is widely used, its theoretical properties have not been thoroughly investigated in cases where the lower-level functions lack strong convexity.
This work takes a step forward when the typical lower-level strong convexity assumption is absent.
Our hardness results show that bilevel optimization for general convex lower-level functions is intractable to solve.
We then identify several regularity conditions of the lower-level
problems that can provably confer tractability.
Under these conditions, we propose the Inexact Gradient-Free Method (IGFM), which uses the Switching Gradient Method (SGM) as an efficient sub-routine, to find an approximate stationary point of the hyper-objective in polynomial time.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4955