Closing the Modality Gap Aligns Group-Wise Semantics

ICLR 2026 Conference Submission16765 Authors

Keywords: Multimodal Learning, Representation Alignment, Modality Gap

TL;DR: We prove that closing the modality gap, while irrelevant for instance-wise tasks, significantly enhances performance in group-wise tasks and we propose a combination of novel losses to do so.

Abstract: In multimodal learning, CLIP has been recognized as the \textit{de facto} method for learning a shared latent space across multiple modalities, placing similar representations close to each other and moving them away from dissimilar ones. Although CLIP-based losses effectively align modalities at the semantic level, the resulting latent spaces often remain only partially shared, revealing a structural mismatch known as the modality gap. While the necessity of addressing this phenomenon remains debated, particularly given its limited impact on instance-wise tasks (e.g., retrieval), we prove that its influence is more pronounced in group-level tasks (e.g., clustering). To support this claim, we introduce a novel method designed to consistently reduce this discrepancy in two-modal settings, with a straightforward extension to the general $n$-modal case. Through our extensive evaluation, we prove our novel insight: while reducing the gap provides only marginal or inconsistent improvements in traditional instance-wise tasks, it significantly enhances group-wise tasks. These findings may reshape our understanding of the modality gap, highlighting its key role in improving performance on tasks requiring semantic grouping.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 16765