Towards Uniformity and Alignment for Multimodal Representation Learning

Wenzhe Yin, Pan Zhou, Zehao Xiao, Jie Liu, Shujian Yu, Jan-Jakob Sonke, Stratis Gavves

Abstract: Multimodal representation learning aims to construct a shared embedding space in which heterogeneous modalities are semantically aligned. Despite strong empirical results, InfoNCE-based objectives introduce inherent conflicts that yield distribution gaps across modalities. In this work, we identify two conflicts in the multimodal regime, both exacerbated as the number of modalities increases: (i) an alignment–uniformity conflict, whereby the repulsion of uniformity undermines pairwise alignment, and (ii) an intra-alignment conflict, where aligning multiple modalities induces competing alignment directions. To address these issues, we propose a principled decoupling of alignment and uniformity for multimodal representations, providing a conflict-free recipe for multimodal learning that simultaneously supports discriminative and generative use cases without task-specific modules. We then provide a theoretical guarantee that our method acts as an efficient proxy for a global Hölder divergence over multiple modality distributions, and thus reduces the distribution gap among modalities. Extensive experiments on retrieval and UnCLIP-style generation demonstrate consistent gains.

Lay Summary: This paper studies how AI systems can better connect different modalities, such as text, images, audio, and video, in a shared representation space. We show that common contrastive training objectives create conflicts between making representations discriminative and aligning different modalities, especially as the number of modalities increases. To address this, we propose UniAlign, which separates within-modality diversity from cross-modality alignment. This reduces the gap between modality distributions and improves both cross-modal retrieval and generation, such as retrieving videos from text and generating images from text or audio inputs.

Primary Area: General Machine Learning->Representation Learning

Keywords: Multimodal Representation Learning, CLIP

Submission Number: 9511