SpaFL: Communication-Efficient Federated Learning with Sparse Models and Low Computational Overhead

Minsu Kim, Walid Saad, Merouane Abdelkader DEBBAH, Choong Seon Hong

Primary Area: general machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Federated Learning, Communication efficiency, Sparse training, Computational overhead

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We develop a communication efficient FL framework with low computational costs by enabling clients to learn how to prune through threshold sharing.

Abstract: The large communication and computation overhead of federated learning (FL) is one of the main challenges facing its practical deployment over resource-constrained clients and systems. In this work, SpaFL: a communication-efficient FL framework is proposed to optimize both personalized model parameters and sparse model structures with low computational overhead. In SpaFL, a trainable threshold is defined for each neuron/filter to prune its connected parameters. Both model parameters and thresholds are jointly optimized to enable the automatic sparsification of the models while recovering prematurely pruned parameters during training. To reduce communication costs, only thresholds are communicated between a server and clients instead of parameters, thereby enabling the clients to learn how to prune. Further, global thresholds are used to update model parameters by extracting aggregated parameter importance. The convergence of SpaFL is analyzed, and the results provide new insights into the tradeoff between computation overhead and learning performance. Experimental results show that SpaFL improves accuracy while requiring much less communication and computing resources compared to both dense and sparse personalized baselines.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6217