Adaptive Vision Token Selection for Multimodal Inference

ICLR 2026 Conference Submission22106 Authors

Keywords: Multimodality, Vision encoders, Reconstruction, Feature selection

TL;DR: Training-free token selection for vision encoders that prunes up to 50% of visual context in VLMs with near-parity accuracy

Abstract: Vision encoders typically generate a large number of visual tokens, providing information-rich representations but significantly increasing computational demands. This raises the question of whether all generated tokens are equally valuable or if some of them can be discarded to reduce computational costs without compromising quality. In this paper, we introduce a new method for determining feature utility based on the idea that less valuable features can be reconstructed from more valuable ones. We implement this concept by integrating an autoencoder with a Gumbel-Softmax selection mechanism, that allows identifying and retaining only the most informative visual tokens. Experiments show that the sampler can reduce effective tokens and inference FLOPs by up to 50% while retaining 99-100% of the original performance on average. On challenging OCR-centric benchmarks, it also surpasses prior SOTA. The sampler transfers to the video setting as well: despite minor drops, zero-shot results remain strong without video-specific training. Our results highlight a promising direction towards adaptive and efficient multimodal pruning that facilitates scalable and low-overhead inference without compromising performance.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 22106