Rethinking Self-Supervise Learning: An Instance-wise Similarity Perspective

Xianghong Fang, Ronghe Qiu, Zenglin Xu, Lei Ma, Qiang Sun, Jie Fu

Keywords: Self-Supervised Learning, Instance-wise Similarity, Sparse structure

TL;DR: We propose a novel self-supervised learning approach, to learn an appropriately sparse IwS matrix in the representation space

Abstract: This paper studies self-supervised learning from the perspective of instance-wise similarity (IwS), characterized by the pairwise similarity matrix among all instances. Ideally, the IwS matrix in the representation space should closely mirror that in the input space so that the learned representations retain their discriminative power and account for semantic similarities. This perspective not only allows us to understand diverse existing self-supervised learning methodologies better but also uncovers a notable limitation within current approaches: the discrepancy between IwS matrices in the input and representation spaces. Indeed, many established methods, including SimCLR and MoCo v3, implicitly assume that the IwS matrix within the representation space is an identity matrix, even when the IwS matrix in the input space may deviate from this form. Inspired by this observation, we introduce sparse contrastive learning, a new approach that learns an appropriately sparse IwS matrix within the representation space instead of presuming an identity IwS matrix. Our comprehensive experiments conducted on ImageNet and CIFAR datasets substantiate the superior performance of our method in comparison to other state-of-the-art methods.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5519