Go to ICLR 2026 Conference homepageFedDUET: Bridging Modality Gaps with Decoupled Uncertainty-Enhanced Training
Keywords: federated learning, multimodal AI, health sensing
TL;DR: We define and tackle the dual challenges of inter-client (device) and intra-client (sensor dropout) modality heterogeneity in multimodal federated health sensing, introducing FedDUET to set a new state-of-the-art in robust performance.
Abstract: Federated learning enables collaborative model training for multimodal health sensing while preserving data privacy. A critical challenge, however, is modality heterogeneity, which manifests along two axes: intra-client instability, caused by per-sample sensor dropouts, and inter-client heterogeneity, driven by differences in clients' sensor suites. Existing federated methods often rely on oversimplified assumptions about missing data and fail to capture these complex dynamics. We address this gap by introducing a realistic problem formulation and a principled simulation framework. Building on this foundation, we propose FedDUET (Decoupled Uncertainty-Enhanced Training), an approach designed to handle both axes of modality heterogeneity. To mitigate intra-client instability, FedDUET leverages an Uncertainty-as-Temperature (UT) loss to dynamically calibrate predictions based on data uncertainty. To manage inter-client heterogeneity, it employs a Decoupled Training (DT) strategy that specializes a private model head for each client's unique sensor suite while isolating the shared representation to preserve its generalizability. Across four real-world multimodal sensing datasets and diverse heterogeneity regimes, FedDUET achieves state-of-the-art performance. Our results highlight that explicitly modeling uncertainty and decoupling generalization from personalization are essential principles for making multimodal federated learning robust in real-world settings.
Primary Area: learning on time series and dynamical systems
Submission Number: 23485
Loading