One-shot-but-not-degraded Federated Learning

Hui Zeng, Minrui Xu, Tongqing Zhou, Xinyi Wu, Jiawen Kang, Zhiping Cai, Dusit Niyato

Abstract: Transforming the multi-round vanilla Federated Learning (FL) into one-shot FL (OFL) significantly reduces the communication burden and makes a big leap toward practical deployment. However, we note that existing OFL methods all build on model lossy reconstruction (i.e., aggregating while partially discarding local knowledge in clients’ models), which attains one-shot at the cost of degraded inference performance. By identifying the root cause of stressing too much on finding a one-fit-all model, this work proposes a novel one-shot FL framework by embodying each local model as an independent expert and leveraging a Mixture-of-Experts network to maintain all local knowledge intact. A dedicated self-supervised training process is designed to tune the network, where the sample generation is guided by approximating underlying distributions of local data and making distinct predictions among experts. Notably, the framework also fuels FL with flexible, data-free aggregation and heterogeneity tolerance. Experiments on 4 datasets show that the proposed framework maintains the one-shot efficiency, facilitates superior performance compared with 8 OFL baselines (+5.54% on CIFAR-10), and even attains over $\times$4 performance gain compared with 3 multi-round FL methods, while only requiring less than 85% trainable parameters.

Primary Subject Area: [Systems] Systems and Middleware

Secondary Subject Area: [Systems] Data Systems Management and Indexing, [Generation] Multimedia Foundation Models, [Content] Vision and Language

Relevance To Conference:

We focus on one-shot federated learning (OFL), with the objective of fusing multimodal/multimedia data distributed across various parties while ensuring privacy.
This approach significantly contributes to the advancement of multimedia knowledge fusing, facilitating the development of more powerful multimedia models.
Furthermore, OFL finds widespread application in the model market, where users can exchange multimedia knowledge through model transactions and seamlessly integrate it using OFL aggregation algorithms. We introduces a novel one-shot-but-not-degraded federated learning framework that efficiently and effectively leverages knowledge from diverse multimedia/multi-modal sources, marking a significant step towards enhancing multimedia knowledge fusion.

Submission Number: 1634