On the Effectiveness of One-Shot Federated Ensembles in Heterogeneous Cross-Silo Settings

Youssef Allouah, Akash Dhasade, Rachid Guerraoui, Nirupam Gupta, Anne-marie Kermarrec, Rafael Pinot, Rafael Pires, Rishi Sharma

Primary Area: general machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Federated Learning, One-Shot, Communication Efficiency, Ensembles

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract: FL is a popular approach for training machine learning models on decentralized data. For communication efficiency, one-shot FL trades the iterative exchange of models between clients and the FL server for one single round of communication. However, one-shot FL does not perform as well as iterative FL, and struggles under high data heterogeneity. While ensembles have repeatedly appeared as strong contenders in one-shot FL literature, their full potential is still under-explored. In this work, we extensively examine federated ensembles across the heterogeneity spectrum, in conjunction with various aggregation functions from the ensemble literature, with a specific focus on cross-silo settings. Our experiments reveal that an aggregator based on a shallow neural network can significantly boost the performance of ensembles under high data heterogeneity. Through comprehensive evaluations on the CIFAR-10, SVHN and the cross-silo healthcare FLamby benchmark, we show that federated ensembles not only achieve up to 26% higher accuracy over current one-shot methods but can also match the performance of iterative FL under high data heterogeneity, all while being up to 9.1x more efficient in terms of communication due to their one-shot nature.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6345