Stochastic Projective Splitting: Solving Saddle-Point Problems with Multiple Regularizers

Patrick R. Johnstone, Jonathan Eckstein, Thomas Flynn, Shinjae Yoo

Keywords: convex optimization, min-max games, saddle-point problems, first-order stochastic methods, proximal methods, operator splitting

Abstract: We present a new, stochastic variant of the projective splitting (PS) family of algorithms for monotone inclusion problems. It can solve min-max and noncooperative game formulations arising in applications such as robust ML without the convergence issues associated with gradient descent-ascent, the current de facto standard approach in ML applications. Our proposal is the first version of PS able to use stochastic gradient oracles. It can solve min-max games while handling multiple constraints and nonsmooth regularizers via projection and proximal operators. Unlike other stochastic splitting methods that can solve such problems, our method does not rely on a product-space reformulation of the original problem. We prove almost-sure convergence of the iterates to the solution and a convergence rate for the expected residual. By working with monotone inclusions rather than variational inequalities, our analysis avoids the drawbacks of measuring convergence through the restricted gap function. We close with numerical experiments on a distributionally robust sparse logistic regression problem.

One-sentence Summary: We develop a stochastic splitting method that can easily handle min-max problems with multiple regularizers and constraints