On the Bias-Variance-Cost Tradeoff of Stochastic Optimization

Yifan Hu, Xin Chen, Niao He

Keywords: stochastic optimization, multilevel Monte Carlo, bias variance and cost tradeoff, biased first-order methods

TL;DR: This paper provide a systematic study of the bias-variance-cost tradeoff of a naïve biased gradient method and several multilevel Monte Carlo gradient methods under a generic biased oracle model for stochastic optimization.

Abstract: We consider stochastic optimization when one only has access to biased stochastic oracles of the objective, and obtaining stochastic gradients with low biases comes at high costs. This setting captures a variety of optimization paradigms widely used in machine learning, such as conditional stochastic optimization, bilevel optimization, and distributionally robust optimization. We examine a family of multi-level Monte Carlo (MLMC) gradient methods that exploit a delicate trade-off among the bias, the variance, and the oracle cost. We provide a systematic study of their convergences and total computation complexities for strongly convex, convex, and nonconvex objectives, and demonstrate their superiority over the naive biased stochastic gradient method. Moreover, when applied to conditional stochastic optimization, the MLMC gradient methods significantly improve the best-known sample complexity in the literature.

Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.