FedORION: Aggregation-Assisted Proxyless Distillation for Heterogeneous Federated Learning

Nirbhay sharma, Mayank Raj, Deepak Mishra

Keywords: Federated Learning, Knowledge Distillation, Aggregation, Deep Mutual Learning, Proxyless Distillation

TL;DR: A novel federated learning framework designed for heterogeneous clients that enables server side global aggregation which currently is infeasible and reduces the dependency on knowledge distillation.

Abstract: System heterogeneity in Federated Learning (FL) is commonly dealt with knowledge distillation by combining the clients' knowledge via distillation into a global model. However, such knowledge transfer to the global model is often limited by distillation efficiency and unavailability of the client data. Most of the existing approaches require proxy data on the server side for distillation, which becomes a bottleneck. To circumevent these limitations, we propose a novel FL framework, FedORION (Aggregation-Assisted Proxyless Distillation for Heterogeneous Federated Learning) that comprises of deep mutual learning (DML) at client end, and global aggregation followed by noise engineered data-free distillation at the server end. DML enables server side global aggregation which otherwise is infeasible due to different client model architectures. The aggregation results in knowledge integration which is further boosted by the subsequent distillation. This, however, also increases the burden on clients, especially with low computational budget. We, therefore, further introduce the idea of selective mutual learning where only those clients perform DML that are not limited by computational capacity. This reduces the overall computational burden without any compromise in the performance. We conduct rigorous experiments on various publicly available datasets and observe a remarkable improvement in the performance over the existing heterogeneous FL methods. For example, for TinyImagenet dataset, FedORION shows almost three times better performance as compared to the best baseline. The results provide evidence for the utility and effectiveness of our approach and open up a new direction for heterogeneous FL.

Primary Area: general machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5453