Beyond Cross-Modal Alignment: Measuring and Leveraging Modality Gap in Vision-Language Models

ACL ARR 2026 January Submission2109 Authors

Keywords: multimodal, interpretability, monosemanticity

Abstract: The success of vision-language models is primarily attributed to effective cross-modal alignment between vision and language. However, modality gaps persist even in well-aligned models and may be necessary for human perception, as evidenced by modality-specific phenomena such as visual texture and linguistic tone. These observations motivate us to computationally measure and leverage modality gaps to explore their utility in downstream applications. In this paper, we introduce the \textbf{M}odality \textbf{D}ominance \textbf{S}core (\textbf{MDS}), which attributes multimodal features to specific modalities by categorizing them as vision-dominant, language-dominant, or cross-modal. We then propose automatic interpretability metrics to evaluate these modality-specific features in a scalable manner. Finally, we demonstrate how the identified modality-specific features enable training-free probing and editing methods for understanding model perception across genders, generating adversarial examples, and controlling text-to-image generation. Combined with task-agnostic interpretability tools, our work provides a systematic framework for analyzing and efficiently controlling multimodal models.

Paper Type: Long

Research Area: Multimodality and Language Grounding to Vision, Robotics and Beyond

Research Area Keywords: interpretability

Contribution Types: Model analysis & interpretability

Languages Studied: English

Submission Number: 2109