Fully First-order Methods for Contextual Stochastic Bilevel Optimization

ICLR 2026 Conference Submission22615 Authors

Keywords: Stochastic Bilevel Optimization, Contextual Optimization, Multilevel Monte-Carlo, Fully First-Order Algorithm

TL;DR: We propose the first fully first-order method for solving contextual stochastic bilevel optimization problems and accelerate it using random truncated multilevel Monte-Carlo techniques, with performance confirmed by real-world experiments.

Abstract: Contextual stochastic bilevel optimization (CSBO) is a new paradigm for decision making under uncertainty that generalizes stochastic bilevel optimization (SBO) by integrating contextual information in the lower level optimization problem and thus offers a stronger modeling capability. Nevertheless, owing to its semi-infinite nature, CSBO is extremely challenging from a computational perspective, hindering its real-world applications. Indeed, many algorithms designed for SBO are not applicable to CSBO. In this paper, we devise a double-loop fully first-order algorithm for solving CSBO and prove that both sample and gradient complexities of the algorithm are $\widetilde{\mathcal{O}}(\epsilon^{-8})$. To tackle the increasing number of inner loop iterations, we further develop an accelerated version of our algorithm using the random truncated multilevel Monte Carlo technique. The accelerated algorithm enjoys the improved complexities of $\widetilde{\mathcal{O}}(\epsilon^{-6})$. Our algorithms are fully first-order in the sense that they do not rely on second-order information, and hence these complexities cannot be directly compared with those of Hessian-based methods. Numerical experiments on meta-learning with real datasets demonstrate the superiority of the proposed algorithms, especially the accelerated version, over existing Hessian-based method in terms of both speed and accuracy.

Primary Area: optimization

Submission Number: 22615