All for One and One for All: A Collaborative FL Framework for Generic Federated Learning with Personalized Plug-ins

Lei Shen, Zhenheng Tang, Lijun Wu, Yonggang Zhang, Xiaowen Chu, Tao Qin, Bo Han

Primary Area: general machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Federated Learning, Deep Learning

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract: Personalized federated learning (PFL) mitigates the notorious data heterogeneity issue in generic federated learning (GFL) by assuming that client models only need to fit on local datasets individually. However, real-world FL clients may meet with test data from other distributions. To endow clients with the ability to handle other datasets, we theoretically formulate a new problem named as Selective FL (SFL), bridging the GFL and PFL together. To practically solve SFL, we design a general effective framework named as Hot-Pluggable Federated Learning (HPFL). In HPFL, clients firstly learn a global shared feature extractor. Next, with the frozen feature extractor, multiple personalized plug-in modules are individually learned based on the local data and saved in a modular store on the server. In inference stage, an accurate selection algorithm allows clients to choose and download suitable plug-in modules from the modular store to achieve the high generalization performance on target data distribution. We conduct comprehensive experiments and ablation studies following common FL settings including four datasets and three neural networks, showing that HPFL significantly outperforms advanced FL algorithms. Additionally, we empirically show the remarkable potential of HPFL to resolve other practical FL problems like continual federated learning and discuss its possible applications in one-shot FL, anarchic FL and an FL plug-in market.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 2503