Federated Learning with a Single Shared Image

Sunny Soni, Aaqib Saeed, Yuki M Asano

Keywords: Single Datum, Federated Learning, Knowledge Distillation, Single Image

TL;DR: Federated learning using knowledge distillation with a single shared image using patch selection

Abstract:

Federated Learning (FL) enables multiple machines to collaboratively train a machine learning model without sharing of private training data.
Yet, especially for heterogeneous models, a key bottleneck remains the transfer of knowledge gained from each client model with the server.
One popular method, FedDF, uses distillation to tackle this task with the use of a common, shared dataset on which predictions are exchanged.
However, in many contexts such a dataset might be difficult to acquire due to privacy and the clients might not allow for storage of a large shared dataset.
To this end, in this paper, we introduce a new method that improves this knowledge distillation method to only rely on a \textit{single} shared image between clients and server.
In particular, we propose a novel adaptive dataset pruning algorithm that selects the most informative crops generated from only a single image.
With this, we show that federated learning with distillation under a limited shared dataset budget works better by using a single image compared to multiple individual ones.
Finally, we extend our approach to allow for training heterogeneous client architectures by incorporating a non-uniform distillation schedule and client-model-mirroring on the server-side.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 2895