Unsupervised Visual Representation Learning via Mutual Information Regularized Assignment

Dong Hoon Lee, Sungik Choi, Hyunwoo Kim, Sae-Young Chung

Keywords: unsupervised representation learning, pseudo-labeling, mutual information maximization, self-supervised learning

Abstract: This paper proposes Mutual Information Regularized Assignment (MIRA), a pseudo-labeling algorithm for unsupervised representation learning inspired by information maximization. We formulate online pseudo-labeling as an optimization problem to find pseudo-labels that maximize the mutual information between the label and data while being close to a given model probability. We derive a fixed-point iteration method and prove its convergence to the optimal solution. In contrast to baselines, MIRA combined with pseudo-label prediction enables a simple yet effective clustering-based representation learning without incorporating extra training techniques or artificial constraints such as sampling strategy, equipartition constraints, etc. With relatively small training epochs, representation learned by MIRA achieves state-of-the-art performance on various downstream tasks, including the linear/${\it k}$-NN evaluation and transfer learning. Especially, with only 400 epochs, our method applied to ImageNet dataset with ResNet-50 architecture achieves 75.6% linear evaluation accuracy.

TL;DR: For unsupervised representation learning, we apply information maximization to pseudo-labeling in a principled way.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2211.02284/code)