An Even More Optimal Stochastic Optimization Algorithm: Minibatching and Interpolation LearningDownload PDF

21 May 2021, 20:51 (modified: 26 Oct 2021, 07:06)NeurIPS 2021 PosterReaders: Everyone
Keywords: Stochastic, Optimization, Learning, Minibatch, Optimistic, Interpolation
TL;DR: We present optimal minibatch stochastic optimization algorithm for interpolation learning.
Abstract: We present and analyze an algorithm for optimizing smooth and convex or strongly convex objectives using minibatch stochastic gradient estimates. The algorithm is optimal with respect to its dependence on both the minibatch size and minimum expected loss simultaneously. This improves over the optimal method of Lan, which is insensitive to the minimum expected loss; over the optimistic acceleration of Cotter et al., which has suboptimal dependence on the minibatch size; and over the algorithm of Liu and Belkin, which is limited to least squares problems and is also similarly suboptimal. Applied to interpolation learning, the improvement over Cotter et al.~and Liu and Belkin translates to a linear, rather than square-root, parallelization speedup.
Supplementary Material: pdf
Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.
10 Replies