Efficient Fully Single-Loop Variance Reduced Methods for Stochastic Bilevel Optimization

Kai Yi, Yaoliang Yu

Primary Area: optimization

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: stochastic bilevel optimization, single-loop methods, variance reduction

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract:

Stochastic Bilevel Optimization (StocBO) has gained traction given its unique nested structure, which is increasingly popular in machine learning areas like meta-learning and hyperparameter optimization. A recent innovation by Dagreou et al. provided a unified single-loop framework for finite-sum StocBO. This presented the SABA method, a SAGA-type approach, achieving an iteration complexity of $\mathcal{O}({(m+n)^{3/2}}/{T})$ and a memory cost of $\mathcal{O}((m+n)(d+p))$. In this context, $m$ and $n$ symbolize the finite sum counts for the outer and inner-level tasks, while $d$ and $p$ describe their parameter dimensions. However, a drawback surfaces with memory consumption, especially with significantly large values of $m$ or $n$.
In response to this, we present the SBO-LSVRG, an adept solution inspired by Loopless-SVRG (LSVRG). This avant-garde method not only achieves the desired iteration complexity but also substantially trims the memory cost to a leaner $\mathcal{O}(d+p)$. To our awareness, this paper pioneers in illustrating, from a theoretical lens, the application of LSVRG to bilevel optimization, particularly in non-convex realms. Furthermore, our variance-reduced method, SBO-LSVRG, excels with an optimal convergence speed. Comprehensive experiments validate the efficiency of our proposed approach.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 110