FedDEO: Description-Enhanced One-Shot Federated Learning with Diffusion Models

Mingzhao Yang, Shangchao Su, Bin Li, Xiangyang Xue

Abstract: In recent years, the attention towards One-Shot Federated Learning (OSFL) has been driven by its capacity to minimize communication. With the development of the diffusion model (DM), several methods employ the DM for OSFL, utilizing model parameters, image features, or textual prompts as mediums to transfer the local client knowledge to the server. However, these mediums often require public datasets or the uniform feature extractor, significantly limiting their practicality. In this paper, we propose FedDEO, a Description-Enhanced One-Shot Federated Learning Method with DMs, offering a novel exploration of utilizing the DM in OSFL. The core idea of our method involves training local descriptions on the clients, serving as the medium to transfer the knowledge of the distributed clients to the server. Firstly, we train local descriptions on the client data to capture the characteristics of client distributions, which are then uploaded to the server. On the server, the descriptions are used as conditions to guide the DM in generating synthetic datasets that comply with the distributions of various clients, enabling the training of the aggregated model. Theoretical analyses and sufficient quantitation and visualization experiments on three large-scale real-world datasets demonstrate that through the training of local descriptions, the server is capable of generating synthetic datasets with high quality and diversity. Consequently, with advantages in communication and privacy protection, the aggregated model outperforms compared FL or diffusion-based OSFL methods and, on some clients, outperforms the performance ceiling of centralized training.

Primary Subject Area: [Generation] Social Aspects of Generative AI

Secondary Subject Area: [Generation] Generative Multimedia

Relevance To Conference: Federated Learning, as an important paradigm for leveraging multimedia data in practical machine learning scenarios, has always been a hot topic. In recent years, the development of foundational models such as diffusion models has brought new possibilities to federated learning. Motivated by this, this paper proposes FedDEO, a Descriptor-Enhanced One-shot Federated learning method with diffusion models. Based on descriptors trained on the clients, FedDEO can aggregate knowledge from various clients within a single-round communication and obtain an aggregated model which has the potenial to outperform the performance ceiling of centralized training. Through theoretical analyses and extensive quantitation and visualization experiments conducted on multiple real-world high-resolution image datasets, FedDEO demonstrates the vast potential of diffusion models in federated learning. Taking image data as a starting point, the framework of FedDEO also holds significant inspirations for the federated learning of other modal data, such as audio, video, text, etc.

Submission Number: 4284