Unsupervised Transfer Learning via Adversarial Contrastive Training

Chenguang Duan, Yuling Jiao, Huazhen Lin, Wensen Ma, Jerry Zhijian Yang

Abstract:

Learning transferable data representations from abundant unlabeled data remains a critical yet challenging task in machine learning. While numerous self-supervised contrastive learning methods have emerged to address this challenge, a notable class of these approaches focuses on aligning the covariance or correlation matrix with the identity matrix. Despite their impressive performance across various downstream tasks,
these methods often suffer from biased sample risk. This bias not only leads to significant optimization offsets, especially in mini-batch scenarios, but also complicates the development of theoretical frameworks.
In this paper, we introduce Adversarial Contrastive Training (ACT), a novel unbiased self-supervised transfer learning approach. This method allows us to develop a comprehensive end-to-end theoretical analysis for self-supervised contrastive learning. Our theoretical results reveal that minimaxing the loss function of ACT can lead to the downstream data distribution being clustered in the representation space, provided that the upstream unlabeled sample size is sufficient. As a result, even with a few downstream samples, ACT can achieve outstanding classification performance,
offering valuable insights for few-shot learning. Furthermore, ACT demonstrates state-of-the-art classification performance across multiple benchmark datasets.

Primary Area: Theory->Domain Adaptation and Transfer Learning

Keywords: contrastive learning, transfer learning, error analysis

Submission Number: 9105