Mitigating Spurious Features in Contrastive Learning with Spectral Regularization

Naghmeh Ghanooni, Waleed Mustafa, Dennis Wagner, Sophie Fellenz, Anthony Widjaja Lin, Marius Kloft

Keywords: Contrastive Learning, Spurious Correlation, Self-supervised Learning, Representation Learning

TL;DR: We introduce a regularizer that promotes diverse, task-relevant features over spurious ones in contrastive learning.

Abstract:

Neural networks generally prefer simple and easy-to-learn features. When these features are spuriously correlated with the labels, the network's performance can suffer, particularly for underrepresented classes or concepts. Self-supervised representation learning methods, such as contrastive learning, are especially prone to this issue, often resulting in worse performance on downstream tasks.
We identify a key spectral signature of this failure: early reliance on dominant singular modes of the learned feature matrix. To mitigate this, we propose a novel framework that promotes a uniform eigenspectrum of the feature covariance matrix, encouraging diverse and semantically rich representations. Our method operates in a fully self-supervised setting, without relying on ground-truth labels or any additional information. Empirical results on SimCLR and SimSiam demonstrate consistent gains in robustness and transfer performance, suggesting broad applicability across self-supervised learning paradigms. Code: https://github.com/NaghmehGh/SpuriousCorrelation_SSRL

Primary Area: General machine learning (supervised, unsupervised, online, active, etc.)

Submission Number: 28422