Personalized Federated Learning with Clustered Generalization

Xueyang Tang, Song Guo, Jingcai Guo

Keywords: Federated Learning, Personalization, Generalization, Clustering, Convergence

Abstract: The prevalent personalized federated learning (PFL) usually pursues a trade-off between personalization and generalization by maintaining a shared global model to guide the training process of local models. However, the sole global model may easily transfer deviated knowledge (e.g., biased updates) to some local models when rich statistical diversity exists across the local datasets. Thus, we argue it is of crucial importance to maintain the diversity of generalization to provide each client with fine-grained common knowledge that can better fit the local data distributions and facilitate faster model convergence. In this paper, we propose a novel concept called clustered generalization (CG) to handle the challenge of statistical heterogeneity, and properly design a CG-based framework of PFL, dubbed CGPFL. Concretely, we maintain K global (i.e., generalized) models in the server and each local model is dynamically associated with the nearest global model to conduct ‘push’ and ‘pull’ operations during the iterative algorithm. We conduct detailed theoretical analysis, in which the convergence guarantee is presented and $\mathcal{O}(\sqrt{K})$ speedup over most existing methods is granted. To quantitatively study the generalization-personalization trade-off, we introduce the ‘generalization error’ measure and prove that the proposed CGPFL can achieve a better trade-off than existing solutions. Moreover, our theoretical analysis further inspires a heuristic algorithm to find a near-optimal trade-off in CGPFL. Experimental results on multiple real-world datasets show that our approach surpasses the state-of-the-art methods on test accuracy by a significant margin.

One-sentence Summary: A novel clustered generalization based personalized federated learning framework and provide complete theoretical and experimental proofs.