Self-labelling via simultaneous clustering and representation learningDownload PDF

Published: 20 Dec 2019, Last Modified: 03 Apr 2024ICLR 2020 Conference Blind SubmissionReaders: Everyone
TL;DR: We propose a self-supervised learning formulation that simultaneously learns feature representations and useful dataset labels by optimizing the common cross-entropy loss for features _and_ labels, while maximizing information.
Abstract: Combining clustering and representation learning is one of the most promising approaches for unsupervised learning of deep neural networks. However, doing so naively leads to ill posed learning problems with degenerate solutions. In this paper, we propose a novel and principled learning formulation that addresses these issues. The method is obtained by maximizing the information between labels and input data indices. We show that this criterion extends standard cross-entropy minimization to an optimal transport problem, which we solve efficiently for millions of input images and thousands of labels using a fast variant of the Sinkhorn-Knopp algorithm. The resulting method is able to self-label visual data so as to train highly competitive image representations without manual labels. Our method achieves state of the art representation learning performance for AlexNet and ResNet-50 on SVHN, CIFAR-10, CIFAR-100 and ImageNet and yields the first self-supervised AlexNet that outperforms the supervised Pascal VOC detection baseline.
Keywords: self-supervision, feature representation learning, clustering
Code: [![github](/images/github_icon.svg) yukimasano/self-label]( + [![Papers with Code](/images/pwc_icon.svg) 3 community implementations](
Data: [CIFAR-10](, [CIFAR-100](, [ImageNet](, [ImageNet-1K](, [Places205](, [SVHN](
Community Implementations: [![CatalyzeX](/images/catalyzex_icon.svg) 4 code implementations](
Original Pdf: pdf
18 Replies