A Unified Framework of Theoretically Robust Contrastive Loss against Label Noise

Jingyi Cui, Yi-Ge Zhang, Hengyu Liu, Yisen Wang

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: contrastive learning, learning from label noise

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We for the first time establish a unified theoretical framework for robust contrastive losses and propose a theory-guided robust version of SimCLR.

Abstract: Learning from noisy labels is a critical challenge in machine learning, with vast implications for numerous real-world scenarios. While supervised contrastive learning has recently emerged as a powerful tool for navigating label noise, many existing solutions remain heuristic, often devoid of a systematic theoretical foundation for crafting robust supervised contrastive losses. To address the gap, in this paper, we propose a unified theoretical framework for robust losses under the pairwise contrastive paradigm. In particular, we for the first time derive a general robust condition for arbitrary contrastive losses, which serves as a criterion to verify the theoretical robustness of a supervised contrastive loss against label noise. This framework is not only holistic -- encompassing prior techniques such as nearest-neighbor (NN) sample selection and robust contrastive loss -- but also instrumental, guiding us to develop a robust version of the popular InfoNCE loss, termed Symmetric InfoNCE (SymNCE). Extensive experiments on benchmark datasets demonstrate the superiority of SymNCE against label noise.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6681