Stepwise Feature Learning in Self-Supervised Learning

ICLR 2026 Conference Submission17357 Authors

Keywords: shortcut learning, self-supervised learning, stepwise learning, feature learning, learning dynamics

Abstract: Recent advances in self-supervised learning (SSL) have shown remarkable progress in representation learning. However, SSL models often exhibit shortcut learning phenomenon, where they exploit dataset-specific biases rather than learning generalizable features, sometimes leading to severe over-optimization on particular datasets. We present a theoretical framework that analyzes this shortcut learning phenomenon through the lens of $\textit{extent bias}$ and $\textit{amplitude bias}$. By investigating the relations among extent bias, amplitude bias, and learning priorities in SSL, we demonstrate that learning dynamics is fundamentally governed by the dimensional properties and amplitude of features rather than their semantic importance. Our analysis reveals how the eigenvalues of the feature cross-correlation matrix influence which features are learned earlier, providing insights into why models preferentially learn shortcut features over more generalizable features.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 17357