From Random to Relevant: Harnessing Salient Masks in Non-IID Federated Learning

Riyasat Ohib, Bishal Thapaliya, Jingyu Liu, Vince Calhoun, Sergey Plis

Primary Area: general machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Sparsity, Pruning, Federated Learning, Sparse Federated Learning, Communication efficiency, Efficient FL, Pruning at Initialization

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We propose a parameter saliency based sparse decentralized federated learning method that finds salient masks based on local data in the non-IID setting resulting in a performant sparse FL method.

Abstract:

Federated learning (FL) offers the ability to train models using decentralized data at client sites, ensuring data privacy by eliminating the need for data centralization. A predominant challenge with FL is the constrained computation and narrow communication bandwidth, particularly evident in resource-restricted edge client nodes. Various solutions, such as transmitting sparse models and iterative pruning have been suggested to tackle this. However, many existing methods necessitate the transmission of full model weights throughout the training, rely heavily on arbitrary or random pruning criteria or costly iterative pruning schedules.

In this work, we propose SSFL, a streamlined approach for sparse decentralized FL training and communication. SSFL identifies a subnetwork prior to training, leveraging parameter saliency scores keeping in mind the distribution of local client data in non-IID scenarios. Distinctively, only the sparse model weights are communicated in each round between client models in a decentralized manner, sidestepping the conventional need of transferring the complete dense model at any phase of training. We validate SSFL's effectiveness using standard non-IID benchmarks, noting marked improvements in the sparsity-accuracy trade-offs. Finally, we deploy our method in a real-world federated learning framework and report improvement in communication time.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 9040