Self-Contrastive Learning

Sangmin Bae, Sungnyun Kim, Jongwoo Ko, Gihun Lee, SeungJong Noh, Se-Young Yun

Keywords: contrastive learning, representation learning, image classification, mutual information

Abstract: This paper proposes a novel contrastive learning framework, called Self-Contrastive (SelfCon) Learning, that self-contrasts within multiple outputs from the different levels of a multi-exit network. SelfCon learning does not require additional augmented samples, which resolves the concerns of multi-viewed batch (e.g., high computational cost and generalization error). Furthermore, we prove that SelfCon loss guarantees the lower bound of label-conditional mutual information between the intermediate and the last feature. In our experiments including ImageNet-100, SelfCon surpasses cross-entropy and Supervised Contrastive (SupCon) learning without the need for a multi-viewed batch. We demonstrate that the success of SelfCon learning is related to the regularization effect associated with the single-view and sub-network.

One-sentence Summary: This paper proposes a novel contrastive framework, called Self-Contrastive (SelfCon) Learning, that self-contrasts within multiple outputs from the different levels of a multi-exit network.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2106.15499/code)