Personalized Federated Training of Latent Diffusion Models with Privacy Guarantees

A F M Mahfuzul Kabir, Lingxiao Wang

Keywords: Federated Learning, Latent Diffusion Models, Differential Privacy

TL;DR: We propose PF-LDM, enabling privacy-preserving federated image generation by privatizing latent representations instead of pixels under formal DP guarantees.

Abstract: We study the problem of federated training of diffusion models (DMs) with privacy guarantees. For high-dimensional data, existing private federated DMs often exhibit a poor privacy-utility tradeoff, since the privacy noise introduced to protect client data becomes increasingly damaging as the dimension of the image representation grows. To address this challenge, we propose Personalized Federated Training of Latent Diffusion Models (PF-LDM), a framework that performs private federated diffusion training in latent space. The key idea is to identify latent representations whose distributional structure is more favorable for private learning. In particular, we show that latent spaces with more discriminative feature representations can better preserve utility under privacy constraints. PF-LDM further combines a shared server-side diffusion model with personalized client-side refinement models: the server captures cross-client generative structure from privatized latent data, while client-specific models refine samples to recover fine-grained local details. Experiments on the CelebAHQ dataset demonstrate that our method enables high-dimensional image generation, improves performance on underrepresented classes across clients, and maintains strong privacy protection.

Submission Number: 134