An Augmentation-Aware Theory for Self-Supervised Contrastive Learning

Jingyi Cui, Hongwei Wen, Yisen Wang

TL;DR: We for the first time propose an augmentation-aware error bound for self-supervised contrastive learning, enabling discussions on the effect of specific data augmentations.

Abstract: Self-supervised contrastive learning has emerged as a powerful tool in machine learning and computer vision to learn meaningful representations from unlabeled data. Meanwhile, its empirical success has encouraged many theoretical studies to reveal the learning mechanisms. However, in the existing theoretical research, the role of data augmentation is still under-exploited, especially the effects of specific augmentation types. To fill in the blank, we for the first time propose an augmentation-aware error bound for self-supervised contrastive learning, showing that the supervised risk is bounded not only by the unsupervised risk, but also explicitly by a trade-off induced by data augmentation. Then, under a novel semantic label assumption, we discuss how certain augmentation methods affect the error bound. Lastly, we conduct both pixel- and representation-level experiments to verify our proposed theoretical results.

Lay Summary: Machine learning models often need large amounts of labeled data, but labeling data can be costly and time-consuming. A promising solution is self-supervised learning, where models learn patterns from unlabeled data by creating their own learning signals. One popular method is contrastive learning, which teaches models to recognize similarities and differences between data points. In this work, we look at an important, but underexplored, part of contrastive learning: data augmentation. Data augmentation involves slightly changing data samples (like cropping or color distortion) to help the model learn more robustly. We develop a new theoretical framework that connects how well a contrastive learning model performs to the types of data augmentation used. We introduce an augmentation-aware error bound, showing that the choice of augmentation affects the model’s overall accuracy and generalization. This result enables discussions on the impact of specific types of data augmentation. We also back up our theory with experiments on real datasets, showing how different augmentation choices influence the model’s learning.

Primary Area: Deep Learning->Self-Supervised Learning

Keywords: theoretical understanding, contrastive learning, self-supervised learning

Submission Number: 5548