Understanding the Theoretical Generalization Performance of Federated Learning

Peizhong Ju, Haibo Yang, Jia Liu, Yingbin Liang, Ness Shroff

Primary Area: learning theory

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Federated Learning, generalization performance, double descent, overfitting

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract:

Federated Learning (FL) has become widely popular because of its applicability in training ML on different sites without data sharing.
However, the generalization performance of FL has remained relatively under-explored, primarily due to the intricate interplay between data heterogeneity and the local update procedures intrinsic to FL.
This motivates us to answer a fundamental question in FL: How can we precisely quantify the impact of data heterogeneity and the local update process on the generalization performance for FL as the learning process evolves?
To this end, we conduct a comprehensive theoretical study of FL's generalization performance using a linear model as the first step, where the data heterogeneity is considered for both the stationary and online/non-stationary cases.
By providing closed-form expressions of the model error, we rigorously quantify the impact of local update steps (denoted as $K$) under three distinct settings ($K=1$, $K<\infty$, and $K=\infty$) and how the generalization performance evolves with the round number $t$.
Our investigation also provides a comprehensive understanding of how different configurations (including the number of model parameters $p$ and the number of training samples $n$) contribute to the overall generalization performance, thus shedding new insights (such as benign overfitting) for the practical implementation of FL.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8031