FedHyMoe: Hypernetwork-Driven Mixture-of-Experts for Federated Domain Generalization

ICLR 2026 Conference Submission24915 Authors

Keywords: Federated Learning, Domain Generalization, Hypernetworks, Mixture-of-Experts, Privacy-Preserving Learning, Cross-Domain Adaptation

TL;DR: TL;DR: **FedHyMoe uses hypernetworks with client embeddings to synthesize Mixture-of-Experts adapters, enabling robust, efficient, and privacy-preserving domain generalization in federated learning under heterogeneity and partial participation.**

Abstract: Federated Learning (FL) enables collaborative model training without sharing raw data, but most existing solutions implicitly assume that each client’s data originate from a single homogeneous domain. In practice, domain shift is pervasive: clients gather data from diverse sources, domains are heterogeneously distributed across clients, and only a subset of clients participate in each round. These factors cause substantial degradation on unseen target domains. Prior Federated Domain Generalization (FedDG) methods often assume complete single-domain datasets per client and sometimes rely on sharing domain-level information, raising privacy concerns and limiting applicability in real-world federations. In this paper, we introduce FedHyMoe, a Hypernetwork-Driven Mixture-of-Experts framework that addresses these challenges by shifting from parameter-space fusion to embedding-space parameter synthesis. Each client is represented by a compact domain embedding, and a shared hypernetwork generates its Mixture-of-Experts (MoE) adapter parameters. At test time, unseen domains are handled by attending over source client embeddings to form a test-domain embedding, which the hypernetwork uses to synthesize a specialized adapter. This enables non-linear interpolation and extrapolation beyond convex averages of stored parameters, while reducing communication and storage overhead and mitigating privacy risks by exchanging only low-dimensional embeddings. FedHyMoe consistently achieves higher generalization accuracy and improved calibration compared to baselines under domain heterogeneity and partial participation highlighting embedding-driven hypernetwork synthesis as a powerful inductive bias for robust, efficient, and privacy-conscious Federated Domain Generalization.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 24915