Evaluating and Steering Modality Preference in Multimodal Large Language Model

ICLR 2026 Conference Submission16105 Authors

Keywords: Modality Preference, MLLM, Representation Engineering

TL;DR: In this paper, we evaluate and induce the modality preference. Based on the findings, the proposed method demonstrates excellent performance in hallucination mitigation and multimodal machine translation.

Abstract:

Multimodal large language models (MLLMs) have achieved remarkable performance on complex multimodal tasks.
However, it remains insufficiently explored whether they exhibit \textit{modality preference}, a tendency to favor one modality over another when processing multimodal contexts.
To study this question, we introduce $MC^2$ benchmark, which constructs controlled evidence-conflict scenarios to systematically evaluate modality preference in decision-making.
Extensive experiments reveals that all 20 tested MLLMs generally demonstrate clear modality preferences, and such preferences can serve as a useful indictor of downstream task performances of MLLMs.
Further analysis shows that modality preference can be controlled by instruction guidance and be captured within the latent representations of MLLMs.
Built on these insights, we propose a probing and steering method based on representation engineering to explicitly control modality preference without requiring additional fine-tuning.
This method effectively amplifies modality preference toward a desired direction and demonstrates promising improvements across multiple downstream applications, including multimodal visual understanding and multimodal machine translation.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 16105