Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning

Sai Praneeth Karimireddy, Martin Jaggi, Satyen Kale, Mehryar Mohri, Sashank J. Reddi, Sebastian U Stich, Ananda Theertha Suresh

Keywords: Federated learning, Federated optimization, Adaptive optimization, Adam, Variance Reduction, Distributed optimization, Decentralized optimization

Abstract:

Federated learning is a challenging optimization problem due to the heterogeneity of the data across different clients. Such heterogeneity has been observed to induce \emph{client drift} and significantly degrade the performance of algorithms designed for this setting.
In contrast, centralized learning with centrally collected data does not experience such drift, and has seen great empirical and theoretical progress with innovations such as momentum, adaptivity, etc.
In this work, we propose a general framework {\sc Mime} which mitigates client-drift and adapts arbitrary centralized optimization algorithms (e.g. SGD, Adam, etc.) to federated learning.
{\sc Mime} uses a combination of \emph{control-variates} and \emph{server-level statistics} (e.g. momentum) at every client-update step to ensure that each local update mimics that of the centralized method. Our thorough theoretical and empirical analyses strongly establish \mime's superiority over other baselines.

One-sentence Summary: Global momentum can be used during local client updates to reduce the effect of non-iid data in cross-device federated learning.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Reviewed Version (pdf): https://openreview.net/references/pdf?id=QgIPIotxVg