Exploring Hypernetwork to Enhance Model Heterogeneous Personalized Federated Learning with Data Distillation

ICLR 2026 Conference Submission15193 Authors

Keywords: Hypernetwork, Personalized Federated Learning, Data Distillation

TL;DR: We are the first to propose a data-driven perspective for model heterogeneity in pFL. Our MH-pFedHNDD is the first to integrate data distillation into the pFL hypernetwork as well as better balance personalization and generalization.

Abstract:

Personalized federated learning (pFL) aims to provide each client with a customized model based on global knowledge. However, in highly heterogeneous scenarios, pFL often struggles to obtain effective global information and faces a trade-off between personalization and generalization, which can degrade overall generalization performance.
To address this issue, we propose a **M**odel-**H**eterogeneous **p**ersonalized **Fed**erated learning framework based on **H**yper**N**etworks with **D**ata **D**istillation, **MH-pFedHNDD**, which, for the first time, incorporates data distillation into a hypernetwork-based federated learning framework, introducing a data-driven perspective to tackle this problem.
We design two effective regularization terms:
(1) Contrastive Condensation Loss, which encourages the latent embeddings of synthetic data to be more compact and closely aligned with the local data of clients used as anchors;
(2) Reg Loss, which integrates the latent embeddings of all clients’ synthetic data as anchors to guide the optimization direction for generalization, thereby enhancing each client’s personalized optimization performance on its local data along with the use of universum negatives.
By leveraging synthetic data distilled with more robust global information, our method enhances local training on clients, is the first to alleviate the imbalance between commonality and personalization for hypernetworks, and improves the performance and generalization of the hypernetwork.
Extensive experiments under various settings demonstrate the effectiveness of our MH-pFedHNDD in personalized federated learning. Our code is available at \url{https://anonymous.4open.science/r/MH-pFedHNDD}.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 15193