Deep Contrastive Learning Approximates Ensembles of One-Class SVMs with Neural Tangent Kernels

Yun Yue, Ziming Zhang

Keywords: contrastive learning, one-class SVM, neural tangent kernel, sequential convex programming

Abstract: To demystify the (self-supervised) contrastive learning in representation learning, in the paper we show that a model learned by deep contrastive learning with a family of loss functions such as InfoNCE essentially approximates an ensemble of one-class support vector machines (SVMs) with neural tangent kernels (NTKs). This result comes from the fact that each gradient for network weight update in contrastive learning can be interpreted approximately as the primal solution for a one-class SVM with contrastive gradients as input. From the dual perspective, the Lagrange multipliers provide unique insights into the importance of the anchor-positive-negative triplet samples. In this way, we further propose a novel sequential convex programming (SCP) algorithm for contrastive learning, where each sub-problem is a one-class SVM. Empirically we demonstrate that our approach can learn better gradients than conventional contrastive learning approaches that significantly improve performance.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Deep Learning and representational learning