On Discriminative Probabilistic Modeling for Self-Supervised Representation Learning

Bokun Wang, Yunwen Lei, Yiming Ying, Tianbao Yang

Keywords: Discriminative Probabilistic Modeling; Self-Supervised Representation Learning; Multiple Importance Sampling

Abstract:

We study the discriminative probabilistic modeling problem over a continuous domain for (multimodal) self-supervised representation learning. To address the challenge of computing the integral in the partition function for each anchor data, we leverage the multiple importance sampling (MIS) technique for robust Monte Carlo integration, which can recover the InfoNCE-based contrastive loss as a special case. Within this probabilistic modeling framework, we reveal the limitation of current InfoNCE-based contrastive loss for self-supervised representation learning and derive insights for developing better approaches by reducing the error of Monte Carlo integration. To this end, we propose a novel non-parametric method for approximating the sum of conditional densities required by MIS through optimization, yielding a new contrastive objective for self-supervised representation learning.
Moreover, we design an efficient algorithm for solving the proposed objective. Experimental results on bimodal contrastive representation learning demonstrate the overall superior performance of our approach on downstream tasks.

Submission Number: 5