A Simple Framework for Uncertainty in Contrastive Learning

Mike Wu, Noah Goodman

Keywords: uncertainty, contrastive learning, unsupervised learning, anomaly detection, out of distribution, corruption

Abstract: Contrastive approaches to representation learning have recently shown great promise. In contrast to generative approaches, these contrastive models learn a deterministic encoder with no notion of uncertainty or confidence. In this paper, we introduce a simple approach based on "contrasting distributions" that learns to assign uncertainty for pretrained contrastive representations. In particular, we train a deep network from a representation to a distribution in representation space, whose variance can be used as a measure of confidence. In our experiments, we show that this deep uncertainty model can be used (1) to visually interpret model behavior, (2) to detect new noise in the input to deployed models, (3) to detect anomalies, where we outperform 10 baseline methods across 11 tasks with improvements of up to 14% absolute, and (4) to classify out-of-distribution examples where our fully unsupervised model is competitive with supervised methods.

One-sentence Summary: A method to capture uncertainty in contrastive learning with applications to interpretability, corruption detection, anomaly detection, and out-of-distribution classification.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Community Implementations: [ 7 code implementations](https://www.catalyzex.com/paper/arxiv:2010.02038/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=6kvi2BNRpQ