AdaProx: A Novel Method for Bilevel Optimization under Pessimistic Framework

Ziwei Guan, Daouda Sow, Sen Lin, Yingbin Liang

Keywords: pessimistic bilevel optimization, convergence analysis, nonconvex, gradient-based method

TL;DR: We propose a novel algorithm for pessimistic bilevel optimization and characterize its convergence rate guarantee.

Abstract: As a powerful framework for various machine learning problems, bilevel optimization has attracted significant attention recently. While many modern gradient-based algorithms have been devised for optimistic bilevel optimization (OBO), pessimistic bilevel optimization (PBO) is much less explored and there is almost no formally designed algorithms for nonlinear PBO with provable convergence guarantee. To fill this gap, we investigate PBO with nonlinear inner- and outer-level objective functions in this work. By leveraging an existing reformulation of PBO into a single-level constrained optimization problem, we propose an Adaptive Proximal (AdaProx) method which features novel designs of adaptive constraint relaxation and accuracy level in order to guarantee an efficient and provable convergence. We further show that AdaProx converges sublinearly to an $\epsilon$-KKT point, and characterize the corresponding computational complexity. Our experiments on an illustrative example and the robust hyper-representation learning problem validate our algorithmic design and theoretical analysis. To the best of our knowledge, this is the first work that develops principled gradient-based algorithms and characterizes the convergence rate for PBO under nonlinear settings.

Submission Number: 51