Theoretical Guarantees and Training Dynamics of Contrastive Learning: How Misaligned Data Influence Feature Purity

Jiawei Sun, Shuai Zhang, Hongkang Li, Meng Wang

Keywords: Contrastive Learning, Training Dynamics, Feature Learning, Generalization Guarantees, Deep Learning Theory

Abstract:

Contrastive learning is a powerful framework for learning discriminative representations from image-text pairs.
Despite its success, its theoretical foundations, especially when the image-text pair exhibits misalignment, remain underexplored.
This paper provides the first theoretical analysis of contrastive learning under data misalignment, proving how the ground-truth modality-paired features are amplified while spurious features are suppressed through the training dynamics analysis. Specifically, we study two nonlinear encoders trained jointly with a contrastive loss and demonstrate that noisy (or misaligned) data pairs
result in mixed representations and degrade the model's generalization ability.
In contrast, recaptioning and filtering improve the data alignment, which in turn purifies the features learned by neurons and subsequently enhances generalization. Our analysis identifies feature purity as a key factor in the success of contrastive learning and offers insights into how data quality and training procedures impact representation learning and downstream generalization. Theoretical insights are supported by experiments on standard benchmarks.

Student Paper: Yes

Submission Number: 88