FedGSE:Gradient-based Sub-model Extraction for Resource-constrained Federated Learning

yabo jia, Haozhao Wang, Genlang Chen, Tianwei Zhang, Chaoyi Pang

Keywords: Federated Learning, Submodel selection, Resource-constrained

TL;DR: A gradient-based submodel selection strategy for federated learning of large models on resource-constrained devices.

Abstract: Federated Learning with Model Heterogeneity has emerged as an important domain, especially with the increasing number of devices that possess diverse resources. However, many clients with valuable data are unable to contribute to training the global model due to the limitations of their resource-constrained devices. One method to overcome this challenge is to extract sub-models from the global model specifically for these resource-limited clients. Unfortunately, existing methods for sub-model extraction rely on predetermined rules, which fail to consider the relationship between the update gradients of the global and client models. In this paper, we propose a novel method called FedGSE, which selects neurons within each layer that exhibit large gradients generated by training the global model on public dataset on the server side, and the selected neurons are used to form sub-models for training on the client side using local dataset. This ensure the gradient updates produced by the sub-model closely resemble the gradient updates that would be produced when training the client data on the global model. As a result, the performance of the sub-model becomes more aligned with that of the global model. Experimental results demonstrate that our method achieves state-of-the-art performance on multiple datasets when compared to other baseline methods.

Primary Area: general machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3570