Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusions
Keywords: Multimodal Large Language Model, Model Integration
Abstract: Multimodal LLMs (MLLMs) equip language models with visual capabilities by aligning vision encoders with language models.
Existing methods to enhance the visual perception of MLLMs often involve designing more powerful vision encoders, which requires re-aligning these vision modules with the language model, leading to expensive and time-consuming training processes.
In this paper, we introduce VisionFuse, a novel integration framework that efficiently utilizes multiple vision encoders from off-the-shelf MLLMs to enhance visual perception without requiring additional training.
Our approach is motivated by the observation that different MLLMs tend to focus on distinct regions of the same query and image. Moreover, we find that the feature distributions of vision encoders within an MLLM family, a group of MLLMs sharing the same pretrained LLM, are highly aligned.
Building on these insights, VisionFuse enriches the visual context by concatenating the tokens generated by the vision encoders of selected MLLMs within a family. By merging the parameters of language models from different MLLMs, VisionFuse allows a single language model to align with various vision encoders, significantly reducing deployment overhead.
We conduct comprehensive evaluations across multiple multimodal benchmarks using various MLLM combinations,
demonstrating substantial improvements
in multimodal tasks. Notably, when integrating MiniGemini-8B and SLIME-8B, VisionFuse achieves an average performance increase of over 4\%.
Primary Area: foundation or frontier models, including LLMs
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 9452
Loading