eRAM-V: From Interaction to Integration in Efficient Multimodal Large Language Models
Keywords: Multimodal Large Language Model; Interpretability of MLLM
Abstract: Multimodal large language models (MLLMs) have made significant progress in recent years, yet the interaction between vision and language representations remains underexplored. Prior work has primarily relied on empirical heuristics to guide architecture design. While effective, this approach can lead to sub-optimal designs and computational redundancy. In this work, we examine the fusion process between visual and textual data. Our findings indicate that in auto-regressive MLLMs, fine-grained interactions between visual and text tokens primarily occur in the middle layers. This leads to redundancy in the shallow and deep layers, where modeling only selected visual representations is sufficient. Based on these insights, we introduce eRAM-V, an MLLM that balances computational efficiency and performance. eRAM-V models selected visual features across all layers and integrates fine-grained visual features at specific layers, as needed. Extensive experiments show that eRAM-V outperforms baseline models with equivalent computational budgets, achieving superior results across various benchmarks.
Primary Area: applications to computer vision, audio, language, and other modalities
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 4209
Loading