Personalization Mitigates the Perils of Local SGD for Heterogeneous Distributed Learning

Kumar Kshitij Patel, Nidham Gazagnadou, Lingxiao Wang, Lingjuan Lyu

Keywords: Federated Learning, Optimization, Data Heterogenenity, Local SGD, Federated Averaging, Personalization, Convergence Analysis, Privacy Preserving Machine Learning

TL;DR: Improved rates for personalized local SGD dominating all existing baselines without data-heterogeneity assumptions.

Abstract: This paper investigates a personalized version of Local Stochastic Gradient Descent (Local SGD). We establish improved convergence guarantees for this personalized approach, eliminating the need for extra assumptions about data or gradient heterogeneity. Our theoretical analysis reveals that personalized Local SGD outperforms both pure local training and federated learning algorithms that produce a consensus model for all devices. This performance gain is primarily due to over-parameterization, which allows for reducing the consensus error between clients with more communication—something that is not observed in non-personalized approaches. We illustrate our observations using experiments on synthetic convex and smooth objectives.

Primary Area: optimization

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 2054