Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings

Qiong Wu, Wenhao Lin, Yiyi Zhou, Weihao Ye, Zhanpeng Zeng, Xiaoshuai Sun, Rongrong Ji

Keywords: Multimodal Large Language Modal, Token Prune

Abstract:

In this paper, we study the visual redundancy problem of multimodal large language models (MLLMs) from the perspective of attention behaviors.
Via extensive empirical experiments, we observe and conclude three main inference stages of MLLMs:
(i) Early fusion between tokens is first accomplished quickly.
(ii) Intra-modality modeling then comes to play.
(iii) Multimodal reasoning resumes and lasts until the end of inference.
In particular, we reveal that visual tokens will stop contributing to reasoning when the text tokens receive enough image information.
Based on this observation, we propose an effective method to improve the efficiency of MLLMs, termed dynamic visual-token exit (DyVTE), which is orthogonal but collaborative to previous token-wise visual compression methods.
To validate the efficacy of DyVTE, we apply it to a set of MLLMs, including LLaVA, VILA, EAGLE and InternVL.
The experimental results not only show the effectiveness of our DyVTE in improving MLLMs' efficiency, e.g., DyVTE reduces the computation overhead of LLaVA-1.5 by up to 45.7% without performance drop, but also reveal a general pattern across multiple MLLMs, well facilitating the in-depth analysis of MLLMs.
Our code is anonymously released at https://anonymous.4open.science/r/AnonymousDyVTE-26AB/.

Primary Area: Applications (e.g., vision, language, speech and audio, Creative AI)

Submission Number: 19549