Multi-modal contrastive learning adapts to intrinsic dimensions of shared latent variables

Yu Gui, Cong Ma, Zongming Ma

Keywords: contrastive learning; multi-modal learning; machine learning theory;

Abstract:

Multi-modal contrastive learning as a self-supervised representation learning technique has achieved great success in foundation model training, such as CLIP~\citep{radford2021learning}.
In this paper, we study the theoretical properties of the learned representations from multi-modal contrastive learning beyond linear representations and specific data distributions.
Our analysis reveals that, enabled by temperature optimization, multi-modal contrastive learning not only maximizes mutual information between modalities but also adapts to intrinsic dimensions of data, which can be much lower than user-specified dimensions for representation vectors.
Experiments on both synthetic and real-world datasets demonstrate the ability of contrastive learning to learn low-dimensional and informative representations, bridging theoretical insights and practical performance.

Primary Area: Theory (e.g., control theory, learning theory, algorithmic game theory)

Submission Number: 18132