On the Bias-Variance-Cost Tradeoff of Stochastic OptimizationDownload PDF

May 21, 2021 (edited Jan 22, 2022)NeurIPS 2021 PosterReaders: Everyone
  • Keywords: stochastic optimization, multilevel Monte Carlo, bias variance and cost tradeoff, biased first-order methods
  • TL;DR: This paper provide a systematic study of the bias-variance-cost tradeoff of a naïve biased gradient method and several multilevel Monte Carlo gradient methods under a generic biased oracle model for stochastic optimization.
  • Abstract: We consider stochastic optimization when one only has access to biased stochastic oracles of the objective, and obtaining stochastic gradients with low biases comes at high costs. This setting captures a variety of optimization paradigms widely used in machine learning, such as conditional stochastic optimization, bilevel optimization, and distributionally robust optimization. We examine a family of multi-level Monte Carlo (MLMC) gradient methods that exploit a delicate trade-off among the bias, the variance, and the oracle cost. We provide a systematic study of their convergences and total computation complexities for strongly convex, convex, and nonconvex objectives, and demonstrate their superiority over the naive biased stochastic gradient method. Moreover, when applied to conditional stochastic optimization, the MLMC gradient methods significantly improve the best-known sample complexity in the literature.
  • Supplementary Material: pdf
  • Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.
11 Replies