Stabilizing Heterogeneous Federated Learning via Feature Decorrelation and Bidirectional Alignment

ICLR 2026 Conference Submission16506 Authors

Keywords: federated learning, data heterogeneity, dimensional collapse, feature alignment

TL;DR: This paper proposes FedBlade, a federated learning framework with bidirectional alignment and feature decorrelation.

Abstract: Data heterogeneity poses a major challenge in federated learning, leading to significant degradation in global model performance. Prior studies have shown that heterogeneity induces dimensional collapse and biased classifiers, which hinder the learning of both feature extractors and classifiers. To tackle these issues, existing approaches apply feature decorrelation to mitigate dimensional collapse and adopt a synthetic classifier with a projector to reduce classifier bias. However, these decorrelation methods fail to prevent small singular values from collapsing to zero, slowing the mitigation of dimensional collapse. Besides, the synergy among the feature extractor, projector and synthetic classifier is overlooked, leading to divergent optimization across clients. To overcome these limitations, we propose FedBlade, a federated learning framework with bidirectional alignment and feature decorrelation. Our feature decorrelation method accelerates the mitigation of dimensional collapse by yielding exponential gradients, while the bidirectional alignment method enhances synergy among model modules and ensures consistency across clients. Extensive experimental results demonstrate that FedBlade outperforms relevant baselines and achieves faster convergence of the global model.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 16506