Robust Clustered Federated Learning

Tiandi Ye, Senhui Wei, Jamie Cui, Cen Chen, Yingnan Fu, Ming Gao

Abstract: Federated learning (FL) is a special distributed machine learning paradigm, where decentralized clients collaboratively train a model under the orchestration of a global server while protecting users’ data privacy. Concept shift across clients as a specific type of data heterogeneity challenges the generic federated learning methods, which output the same model for all clients. Clustered federated learning is a natural choice for addressing concept shift. However, we empirically show that existing state-of-the-art clustered federated learning methods cannot match some personalized learning methods. We attribute it to the fact they group clients based on their entangled signals, which results in poor clustering. To tackle the problem, in this paper, we devise a lightweight disentanglement mechanism, which explicitly captures client-invariant and client-specific patterns. Incorporating the disentanglement mechanism into clients’ local training, we propose a robust clustered federated learning framework (RCFL), which groups the clients based on their client-specific signals. We conduct extensive experiments on three popular benchmark datasets to show the superiority of RCFL over the competitive baselines, including personalized federated learning methods and clustered federated learning methods. Additional experiments demonstrate its robustness against several sensitive factors. The ablation study verifies the effectiveness of our introduced components in RCFL.