Towards Demystifying Representation Learning with Non-contrastive Self-supervision

Xiang Wang, Xinlei Chen, Simon Shaolei Du, Yuandong Tian

Keywords: self-supervised learning, representation learning, non-contrastive methods, DirectPred, theoretical analysis

Abstract: Non-contrastive methods of self-supervised learning (such as BYOL and SimSiam) learn representations by minimizing the distance between two views of the same image. These approaches have achieved remarkable performance in practice, but it is not well understood 1) why these methods do not collapse to the trivial solutions and 2) how the representation is learned. Tian et al made an initial attempt on the first question and proposed DirectPred that sets the predictor directly. In our work, we analyze a generalized version of DirectPred, called DirectSet($\alpha$). We show that in a simple linear network, DirectSet($\alpha$) provably learns a desirable projection matrix and also reduces the sample complexity on downstream tasks. Our analysis suggests that weight decay acts as an implicit threshold that discard the features with high variance under augmentation, and keep the features with low variance. Inspired by our theory, we simplify DirectPred by removing the expensive eigen-decomposition step. On CIFAR-10, CIFAR-100, STL-10 and ImageNet, DirectCopy, our simpler and more computationally efficient algorithm rivals or even outperforms DirectPred.

One-sentence Summary: We analyzed the representation learning process of non-contrastive self-supervised learning in linear networks, and also proposed a new algorithm DirectCopy that achieves good performance in various datasets.