Novel Losses for Contrastive Learning using Siamese Energy-Based Models

TMLR Paper8470 Authors

Abstract: Learning useful representations without labels is central to modern machine learning, especially when annotation is costly, motivating the development of self-supervised learning. To that end, contrastive learning methods, such as SimCLR, aim to discover representations that are invariant to user-defined augmentations. Recent work has shown that these methods can be reinterpreted as energy-based models (EBMs) that learn to “de-augment” data. Building on this perspective, we propose a principled EBM formulation of contrastive representation learning. Through this formulation, we are able to offer new objectives to train this model. Particularly, we propose a Fisher-Hyvärinen divergence loss, which leverages score matching to bypass the need for negative samples. Our framework bridges contrastive learning with EBM and posterior estimation, offering a new foundation for unsupervised representation learning.

Submission Type: Long submission (more than 12 pages of main content)

Assigned Action Editor: ~Dmitry_Kobak2

Submission Number: 8470