Optimal Rates for Random Order Online OptimizationDownload PDF

21 May 2021, 20:46 (modified: 22 Jan 2022, 17:00)NeurIPS 2021 OralReaders: Everyone
Keywords: Online Learning, Convex Optimization, Algorithmic Stability, Stochastic Gradient Descent
TL;DR: Average stability of SGD under without-replacement sampling leads to optimal regret upper bounds for random order online optimization.
Abstract: We study online convex optimization in the random order model, recently proposed by Garber et al. (2020), where the loss functions may be chosen by an adversary, but are then presented to the online algorithm in a uniformly random order. Focusing on the scenario where the cumulative loss function is (strongly) convex, yet individual loss functions are smooth but might be non-convex, we give algorithms that achieve the optimal bounds and significantly outperform the results of Garber et al. (2020), completely removing the dimension dependence and improve their scaling with respect to the strong convexity parameter. Our analysis relies on novel connections between algorithmic stability and generalization for sampling without-replacement analogous to those studied in the with-replacement i.i.d. setting, as well as on a refined average stability analysis of stochastic gradient descent.
Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.
11 Replies