Drago: Primal-Dual Coupled Variance Reduction for Faster Distributionally Robust Optimization

Ronak Mehta, Jelena Diakonikolas, Zaid Harchaoui

Keywords: Distributionally Robust Optimization, Stochastic Optimization, Convex Optimization, Saddle Point

TL;DR: A stochastic primal-dual algorithm for solving distributionally robust optimization problems. It achieves a state-of-the-art linear convergence rate and combines randomized and cyclic components.

Abstract: We consider the penalized distributionally robust optimization (DRO) problem with a closed, convex uncertainty set, a setting that encompasses learning using $f$-DRO and spectral/$L$-risk minimization. We present Drago, a stochastic primal-dual algorithm which combines cyclic and randomized components with a carefully regularized primal update to achieve dual variance reduction. Owing to its design, Drago enjoys a state-of-the-art linear convergence rate on strongly convex-strongly concave DRO problems witha fine-grained dependency on primal and dual condition numbers. The theoretical results are supported with numerical benchmarks on regression and classification tasks.

Primary Area: Optimization (convex and non-convex, discrete, stochastic, robust)

Submission Number: 8200