Unraveling Cross-Modality Knowledge Conflicts in Large Vision-Language Models
Keywords: Knowledge Conflict, Vision and Language
TL;DR: This paper investigates cross-modality parametric knowledge conflicts in LVLMs by systematically detecting, interpreting, and mitigating these conflicts.
Abstract: Large Vision-Language Models (LVLMs) have demonstrated impressive capabilities for capturing and reasoning over multimodal inputs.
However, these models are prone to parametric knowledge conflicts, which arise from inconsistencies of represented knowledge between their vision and language components.
In this paper, we formally define the problem of **cross-modality parametric knowledge conflict** and present a systematic approach to detect, interpret, and mitigate them.
We introduce a pipeline that identifies conflicts between visual and textual answers, showing a persistently high conflict rate across modalities in recent LVLMs regardless of the model size.
We further investigate how these conflicts interfere with the inference process and propose a contrastive metric to discern the conflicting samples from the others.
Building on these insights, we develop a novel dynamic contrastive decoding method that removes undesirable logits inferred from the less confident modality components based on answer confidence.
For models that do not provide logits, we also introduce two prompt-based strategies to mitigate the conflicts.
Our methods achieve promising improvements in accuracy on both the ViQuAE and InfoSeek datasets.
Specifically, using LLaVA-34B, our proposed dynamic contrastive decoding improves an average accuracy of 2.24\%.
Supplementary Material: zip
Primary Area: alignment, fairness, safety, privacy, and societal considerations
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 5767
Loading