Revisiting EXTRA for Smooth Distributed Optimization

Huan Li, Zhouchen Lin

Abstract: EXTRA is a popular method for dencentralized distributed optimization and has broad applications. This paper revisits EXTRA. First, we give a sharp complexity analysis for EXTRA with the improved $O\big(\big(\frac{L}{\mu}+\frac{1}{1-\sigma_2({W})}\big)\log\frac{1}{\epsilon(1-\sigma_2({W}))}\big)$ communication and computation complexities for $\mu$-strongly convex and $L$-smooth problems, where $\sigma_2({W})$ is the second largest singular value of the weight matrix ${W}$. When the strong convexity is absent, we prove the $O\big(\big(\frac{L}{\epsilon}+\frac{1}{1-\sigma_2({W})}\big)\log\frac{1}{1-\sigma_2({W})}\big)$ complexities. Then, we use the Catalyst framework to accelerate EXTRA and obtain the $O\big(\sqrt{\frac{L}{\mu(1-\sigma_2({W}))}}\log\frac{ L}{\mu(1-\sigma_2({W}))}\log\frac{1}{\epsilon}\big)$ communication and computation complexities for strongly convex and smooth problems and the $O\big(\sqrt{\frac{L}{\epsilon(1-\sigma_2({W}))}}\log\frac{1}{\epsilon(1-\sigma_2(\mathbf{W}))}\big)$ complexities for nonstrongly convex ones. Our communication complexities of the accelerated EXTRA are only worse by the factors of $\big(\log\frac{L}{\mu(1-\sigma_2(\mathbf{W}))}\big)$ and $\big(\log\frac{1}{\epsilon(1-\sigma_2({W}))}\big)$ from the lower complexity bounds for strongly convex and nonstrongly convex problems, respectively.