Contrastive MIM: A Contrastive Mutual Information Framework for Unified Generative and Discriminative Representation Learning

Micha Livne

Keywords: representation learning, contrastive learning, latent variable models, probabilistic models

TL;DR: We introduce cMIM for life-sciences tasks, a contrastive extension of the probabilistic auto-encoder MIM, that unifies generative and discriminative representation learning without contrastive data augmentation and is robust to small batch sizes.

Abstract:

We present Contrastive Mutual Information Machine (cMIM), a probabilistic framework that
adds a contrastive objective to the Mutual Information Machine (MIM) ,
yielding representations that are effective for both discriminative and generative tasks.
Unlike conventional contrastive learning,
cMIM does not require positive data augmentations and
exhibits reduced sensitivity to batch size.
We further introduce informative embeddings, a generic training-free method to extract enriched features from
decoder hidden states of encoder--decoder models.

We evaluate cMIM on life-science tasks, including molecular property prediction on ZINC-15
(ESOL, FreeSolv, Lipophilicity) and biomedical image classification (MedMNIST).
cMIM consistently improves downstream accuracy over MIM and InfoNCE baselines while
maintaining comparable reconstruction quality.
These results indicate cMIM is a promising foundation-style representation learner for
biomolecular and biomedical applications and is readily extendable to multi-modal settings
(e.g., molecules + omics + imaging).

Submission Number: 70