A Framework For Contrastive Self-Supervised Learning And Designing A New Approach

William Alejandro Falcon, Kyunghyun Cho

Keywords: self-supervised learning, contrastive learning

Abstract: Contrastive self-supervised learning (CSL) is an approach to learn useful representations by solving a pretext task which selects and compares anchor, negative and positive (APN) features from an unlabeled dataset. We present a conceptual framework which characterizes CSL approaches in five aspects (1) data augmentation pipeline, (2) encoder selection, (3) representation extraction, (4) similarity measure, and (5) loss function. We analyze three leading CSL approaches--AMDIM, CPC and SimCLR--, and show that despite different motivations, they are special cases under this framework. We show the utility of our framework by designing Yet Another DIM (YaDIM) which achieves competitive results on CIFAR-10, STL-10 and ImageNet, and is more robust to the choice of encoder and the representation extraction strategy. To support ongoing CSL research, we release the PyTorch implementation of this conceptual framework along with standardized implementations of AMDIM, CPC (V2), SimCLR, BYOL, Moco (V2) and YADIM.

One-sentence Summary: We formulate a conceptual framework to characterize contrastive learning methods, evaluate 3 state-of-the-art methods, found what drives a lot of their results and created a new approach using our framework to show its usefulness.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2009.00104/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=QdCyL7fpa9