What Makes for Good Representations for Contrastive Learning

Haoqing Wang, Xun Guo, Zhi-Hong Deng, Yan Lu

Keywords: Unsupervised learning, Self-supervised learning, Contrastive learning, Minimal sufficient representation

Abstract: Contrastive learning between different views of the data achieves outstanding success in the field of self-supervised representation learning and the learned representations are useful in various downstream tasks. Since all supervision information for one view comes from the other view, contrastive learning tends to obtain the minimal sufficient representation which contains the shared information and eliminates the non-shared information between views. Considering the diversity of the downstream tasks, it can not be guaranteed that all task-relevant information is shared between views. Therefore, we assume the task-relevant information that is not shared between views can not be ignored and theoretically prove that the minimal sufficient representation in contrastive learning is not sufficient for the downstream tasks, which causes performance degradation. This reveals a new problem that the contrastive learning models have the risk of over-fitting to the shared information between views. To alleviate this problem, we propose to increase the mutual information between the representation and input as regularization to approximately introduce more task-relevant information since we can not utilize any downstream task information during training. Extensive experiments verify the rationality of our analysis and the effectiveness of our method. It significantly improves the performance of several classic contrastive learning models in downstream tasks.

One-sentence Summary: This paper theoretically and empirically verifies that the minimal sufficient representation in contrastive learning is not sufficient for downstream tasks and contrastive learning has the risk of over-fitting to the shared information between views.