Refined Analysis of Constant Step Size Federated Averaging and Federated Richardson-Romberg Extrapolation

Paul Mangold, Alain Oliviero Durmus, Aymeric Dieuleveut, Sergey Samsonov, Eric Moulines

TL;DR: We conduct an analysis of FedAvg and give first-order expansion of the bias, then propose a new method to reduce the bias.

Abstract:

In this paper, we present a novel analysis of $\texttt{FedAvg}$ with constant step size, relying on the Markov property of the underlying process. We demonstrate that the global iterates of the algorithm converge to a stationary distribution and analyze its resulting bias and variance relative to the problem's solution.
We provide a first-order bias expansion in both homogeneous and heterogeneous settings. Interestingly, this bias decomposes into two distinct components: one that depends solely on stochastic gradient noise and another on client heterogeneity.
Finally, we introduce a new algorithm based on the Richardson-Romberg extrapolation technique to mitigate this bias.

Submission Number: 1993