Hybrid Mutual Information Lower-bound Estimators for Representation LearningDownload PDF

Mar 04, 2021 (edited Apr 01, 2021)Neural Compression Workshop @ ICLR 2021Readers: Everyone
  • Keywords: generative models, contrastive learning
  • TL;DR: We propose a hybrid approach where we combine generative models and contrastive learning
  • Abstract: Self-supervised representation learning methods based on the principle of maximizing mutual information have been successful in unsupervised learning of visual representations. These approaches are low-variance mutual information lower bound estimators, yet the lack of distributional assumptions prevent them from learning certain important information such as texture. Estimators that are based on distributional assumptions bypass this issue with autoencoders but they tend to have worse performance on downstream classification. To this end, we consider a hybrid approach that incorporates both the distribution-free contrastive lower bound and the distribution-based autoencoder lower bound. We illustrate that with one set of representations, the hybrid approach is able to achieve good performance on multiple downstream tasks such as classification, reconstruction, and generation.
1 Reply