Hybrid Discriminative-Generative Training via Contrastive Learning

Hao Liu, Pieter Abbeel

Keywords: Hybrid Models, Contrastive Learning, Energy-Based Models, Discriminative-Generative Models

Abstract: Contrastive learning and supervised learning have both seen significant progress and success. However, thus far they have largely been treated as two separate objectives, brought together only by having a shared neural network. In this paper we show that through the perspective of hybrid discriminative-generative training of energy-based models we can make a direct connection between contrastive learning and supervised learning. Beyond presenting this unified view, we show our specific choice of approximation of the energy-based loss significantly improves energy-based models and contrastive learning based methods in confidence-calibration, out-of-distribution detection, adversarial robustness, generative modeling, and image classification tasks. In addition to significantly improved performance, our method also gets rid of SGLD training and does not suffer from training instability. Our evaluations also demonstrate that our method performs better than or on par with state-of-the-art hand-tailored methods in each task.

One-sentence Summary: We propose a hybrid discriminative-generative model based on contrastive loss and energy-based models, which significantly improves state-of-the-art energy-based models and contrastive learning methods in multiple tasks.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Community Implementations: [ 5 code implementations](https://www.catalyzex.com/paper/arxiv:2007.09070/code)

Reviewed Version (pdf): https://openreview.net/references/pdf?id=-PGu4-vUwI