STOCHASTIC BILEVEL PROJECTION-FREE OPTIMIZATION

Zeeshan Akhtar, Amrit Bedi, Ketan Rajawat

Keywords: Stochastic gradient, bilevel optimization, Frank-Wolfe

TL;DR: STOCHASTIC BILEVEL PROJECTION-FREE OPTIMIZATION

Abstract:

Bi-level optimization is a powerful framework to solve a
rich class of problems such as hyper-parameter optimization,
model-agnostic meta-learning, data distillation, and matrix
completion. The existing first-order solutions to bi-level
problems exhibit scalability limitations (for example, in matrix completion) because of the requirement of projecting
solutions onto the feasible set. In this work, we propose a
novel Stochastic Bi-level Frank-Wolfe (SBFW) algorithm
to solve the stochastic bi-level optimization problems in a
projection-free manner. We utilize a momentum-based gradient tracker that results in a sample complexity of O(ϵ−3) for
convex outer objectives with strongly convex inner objectives.
We formulate the matrix completion problem with denoising
as a stochastic bilevel problem and show that SBFW outperforms the state-of-the-art methods for the problem of matrix
completion with denoising and achieves improvements of up
to 82% in terms of the wall-clock time required to achieve the
same level of accuracy