pFedKT: Personalized Federated Learning via Knowledge Transfer

Liping Yi, Xiaorong Shi, Gang Wang, xiaoguang Liu

Keywords: Personalized Federated Learning, Knowledge Transfer, Local Hypernetwork, Contrastive Learning

Abstract: Federated learning (FL) has been widely studied as a new paradigm to achieve multi-party collaborative modelling on decentralized data with privacy protection. Unfortunately, traditional FL suffers from Non-IID data distribution, where clients' private models after FL are even inferior to models trained standalone. Existing approaches to tackle this challenge fall into two directions: a) pursuing a better global model through mitigating biases of private models, and b) improving personalized private models by personalized federated learning (PFL). Still, both of them have limited accuracy improvements in private models. To this end, \textit{we design pFedKT, a novel personalized federated learning framework with knowledge transfer, towards boosting the performances of personalized private models on Non-IID data}. It involves two types of knowledge transfer: a) transferring \textit{historical private knowledge} to new private models by local hypernetworks; b) transferring \textit{the global model's knowledge} to private models through contrastive learning. After absorbing the historical private knowledge and the latest global knowledge, the personalization and generalization of private models are both enhanced. Besides, we derive pFedKT's generalization and prove its convergence theoretically. Extensive experiments verify that pFedKT presents $0.31\%-3.46\%$ accuracy improvements of private models than the state-of-the-art baseline.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Social Aspects of Machine Learning (eg, AI safety, fairness, privacy, interpretability, human-AI interaction, ethics)