TeMo: Temperature Modulation for Multimodal Contrastive Learning

Dhimitrios Duka, Bernt Schiele, Hilde Kuehne, Anna Kukleva

Keywords: Contrastive Learning, Self-Supervised Learning, Temperature

Abstract: Contrastive learning approaches achieve strong performance by training models to bring similar samples closer while pushing dissimilar samples apart. A crucial component of contrastive learning is the temperature hyperparameter $\tau$, which controls the penalty strength applied to negative samples. However, most existing methods either fix this hyperparameter or learn a global value during training. In this paper, we introduce TeMo, $\underline{Te}$mperature $\underline{Mo}$dulation framework, a similarity-based modulation approach that adaptively adjusts the temperature for each positive-negative pair according to their similarity, enabling more fine-grained multimodal contrastive learning. Our approach seamlessly integrates temperature-modulated multimodal and unimodal losses with the standard multimodal contrastive loss by gradually transitioning between them. This design allows the model to capture both coarse- and fine-grained semantics at different training stages. Extensive experiments demonstrate that each component of TeMo consistently enhances performance across diverse zero-shot retrieval and classification tasks, establishing new state-of-the-art results.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 9080