A Stochastic Variance Reduced Extragradient Method for Sparse Machine Learning Problems

Lin Kong, Xiaying Bai, Yang Hu, Fanhua Shang, Yuanyuan Liu, Hongying Liu

Abstract: This paper mainly considers the optimization problem of minimizing the average of a number of smooth convex functions and a general convex function capable of proximal mapping. In particular, we consider non-smooth functions, and assume that the smooth convex function is strongly convex, which often occur in data mining and deep learning. For non-smooth large-scale high-dimensional machine learning problems, this paper introduces both the variance reduction technique in SVRG and extragradient descent, and proposes an efficient proximal stochastic variance reduced extragradient algorithm, called EProx-SVRG. From our theoretical results, we know that in the expected form, EProx-SVRG converges to the optimum at a linear convergence rate for strongly convex problems, and the convergence guarantee for non-strongly convex problems is also provided. Moreover, EProx-SVRG achieves the same oracle complexity as Prox-SVRG. Finally, our experimental results show that our algorithm can obtain an improved performance compared with Prox-SVRG and Katyusha.