On iteratively regularized first-order methods for simple bilevel optimization

Sepideh Samadi, Daniel Burbano, Farzad Yousefian

Abstract:

We consider simple bilevel optimization problems in which the goal is to compute, among the optimal solutions of a composite convex lower-level problem, one that minimizes a secondary upper-level objective. We develop a class of iteratively regularized first-order methods for addressing such problems under different settings.
First, for composite strongly convex upper-level objectives, we propose an iteratively regularized proximal gradient method and establish asymptotic convergence to the unique optimal solution, along with simultaneous sublinear convergence rates for infeasibility and suboptimality. Under a weak sharp minimality condition, linear convergence rates are obtained.
Second, we introduce a regularized accelerated proximal gradient method achieving improved sublinear rates, including quadratically decaying convergence, and linear rates under weak sharp minimality with improved dependence on the condition number.
Third, for smooth nonconvex upper-level objectives, we propose an inexactly projected iteratively regularized method and derive new convergence guarantees for computing stationary points. Numerical experiments on ill-posed inverse problems demonstrate the effectiveness of the proposed methods.