The Power of Contrast for Feature Learning: A Theoretical Analysis

Wenlong Ji, Zhun Deng, Ryumei Nakada, James Zou, Linjun Zhang

Keywords: contrastive learning, self-supervised learning, transfer learning

Abstract: Contrastive learning has achieved state-of-the-art performance in various self-supervised learning tasks and even outperforms its supervised counterpart. Despite its empirical success, the theoretical understanding of why contrastive learning works is still limited. In this paper, (i) we provably show that contrastive learning outperforms autoencoder, a classical unsupervised learning method, on both feature recovery and downstream tasks; (ii) we also illustrate the role of labeled data in supervised contrastive learning. This provides theoretical support for recent findings that contrastive learning with labels improves the performance of learned representations in the in-domain downstream task, but it can harm the performance in transfer learning. We verify our theory with numerical experiments.

One-sentence Summary: This paper provides a theoretical understanding about the power of contrastive learning on representation learning, downstream task performance, and transferability.