Convergence Rate Analysis for Distributed Optimization with Localization

Hsu Kao, Vijay G. Subramanian

Abstract: We study the effect of the localization scheme on the convergence rate in the setting of distributed gradient descent for smooth optimization. Localization is a technique that exploits the partial dependency structure in the objective functions to reduce the memory storage and communication involved in distributed optimization algorithms. We find that the localization scheme could reduce the convergence rate by a constant factor, which depends on the sizes of the local networks and the spectral radii of the associated optimal doubly stochastic matrices for averaging. The choice of local networks directly affects the memory cost, the communication cost, and the convergence speed. In order to characterize the relation, we explore how the optimal spectral radius depends on the topology of a graph, i.e., local network in the context. We provide a numerical example that shows the saving in terms of memory, communication, as well as iterations required could be tremendous with the localization schemes.