Understanding the Robustness of Distributed Self-Supervised Learning Frameworks Against Non-IID Data

ICLR 2026 Conference Submission14970 Authors

Keywords: Distributed Learning, Heterogeneous and Unlabeled Data, Self-Supervised Learning, Federated Learning, Decentralized Learning

TL;DR: We theoretically characterize the robustness of different distributed self-supervised learning frameworks under non-IID unlabeled data, and validate the derived insights through extensive experiments.

Abstract: Recent research has introduced distributed self-supervised learning (D-SSL) approaches to leverage vast amounts of unlabeled decentralized data. However, D-SSL faces the critical challenge of data heterogeneity, and there is limited theoretical understanding of how different D-SSL frameworks respond to this challenge. To fill this gap, we present a rigorous theoretical analysis of the robustness of D-SSL frameworks under non-IID (non-independent and identically distributed) settings. Our results show that pre-training with Masked Image Modeling (MIM) is inherently more robust to heterogeneous data than Contrastive Learning (CL), and that the robustness of decentralized SSL increases with average network connectivity, implying that federated learning (FL) is no less robust than decentralized learning (DecL). These findings provide a solid theoretical foundation for guiding the design of future D-SSL algorithms. To further illustrate the practical implications of our theory, we introduce MAR loss, a refinement of the MIM objective with local-to-global alignment regularization. Extensive experiments across model architectures and distributed settings validate our theoretical insights, and additionally confirm the effectiveness of MAR loss as an application of our analysis.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 14970