Multi-label Cluster Discrimination for Visual Representation Learning

Xiang An, Kaicheng Yang, Jing Yang, Ziyong Feng, Jia Guo, Tongliang Liu, Jiankang Deng

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: multi-label classification, feature representation learning

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: multi-label cluster discrimination, vision-language learning

Abstract:

Contrastive Language-Image Pre-training (CLIP) has recently demonstrated success across various tasks due to superior feature representation empowered by image-text contrastive learning. However, the instance discrimination method used by CLIP can hardly encode the semantic structure of training data. To handle this limitation, cluster discrimination has been proposed through iterative classification and cluster assignment. Nevertheless, most cluster discrimination approaches only define a single pseudo-label for each image, neglecting multi-label signals in the image.
In this paper, we propose a novel multi-label cluster discrimination method to enhance representation learning. In the clustering step, we first cluster the large-scale LAION 400M dataset into one million centers based on off-the-shelf embedding features. Considering that natural images frequently contain multiple visual targets, we select the multiple closest centers as additional class labels.
In the discrimination step, we design an efficient multi-label classification loss, which elegantly separates losses from positive classes and negative classes and facilitates distributed training on large-scale data. We validate the proposed multi-label cluster discrimination method with experiments on different scales of models and pre-training datasets. Experimental results show that our method achieves state-of-the-art performance on multiple downstream tasks including linear probe, zero-shot classification, and image-text retrieval.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3251