PCA++: How Uniformity Induces Robustness to Background Noise in Contrastive Learning

Mingqi Wu, Qiang Sun, Archer Y. Yang

Keywords: Contrastive Learning, Principal Component Analysis, Uniformity Regularization, Self-Supervised Learning, Representation Learning

TL;DR: This paper proposes PCA++, a uniformity-constrained contrastive PCA for robust signal recovery under strong background noise.

Abstract: High-dimensional data often conceal low-dimensional signals beneath structured background noise, limiting standard PCA. Motivated by contrastive learning, we address the problem of recovering shared signal subspaces from positive pairs--paired observations sharing the same signal but differing in background. Our baseline, PCA+, uses alignment-only contrastive learning and succeeds when background variation is mild, but fails under strong noise or high-dimensional regimes. To address this, we introduce PCA++, a hard uniformity-constrained contrastive PCA that enforces identity covariance on projected features. PCA++ has a closed-form solution via a generalized eigenproblem, remains stable in high dimensions, and provably regularizes against background interference. We provide exact high-dimensional asymptotics in both fixed-aspect-ratio and growing-spike regimes, showing uniformity’s role in robust signal recovery. Empirically, PCA++ outperforms standard PCA and alignment-only PCA+ on simulations, corrupted-MNIST, and single-cell transcriptomics, reliably recovering condition-invariant structure. More broadly, we clarify uniformity’s role in contrastive learning—showing that explicit feature dispersion defends against structured noise and enhances robustness.

Primary Area: General machine learning (supervised, unsupervised, online, active, etc.)

Submission Number: 26142