Hidden in plain sight: VLMs overlook their visual representations

Stephanie Fu, tyler bonnen, Devin Guillory, Trevor Darrell

Keywords: vision, language, representation, benchmark, encoder, vlm

TL;DR: VLMs perform worse on vision-centric tasks than their underlying vision models, relying on their language priors instead. Improving their integration of visual data—not just adding stronger vision backbones—is key to unlocking their full potential.

Abstract:

Language provides a natural interface to specify and evaluate performance on visual tasks.
To realize this possibility, vision language models (VLMs) must successfully integrate visual and linguistic information.
Our work compares VLMs to a direct readout of their visual encoders to understand their ability to integrate across these modalities. Across a series of vision-centric benchmarks (e.g., depth estimation, correspondence), we find that VLMs perform substantially worse than their visual encoders, dropping to near-chance performance. We investigate these results through a series of analyses across the entire VLM: namely 1) the degradation of vision representations, 2) brittleness to task prompt, and 3) the language model's role in solving the task. We find that the bottleneck in performing these vision-centric tasks lies in this third category; VLMs are not effectively using visual information easily accessible throughout the \textit{entire} model, and they inherit their language biases.
Our work helps diagnose the failure modes of open-source VLMs, and presents a series of evaluations useful for future investigations into visual understanding within VLMs.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the COLM Code of Ethics on https://colmweb.org/CoE.html

Author Guide: I certify that this submission complies with the submission instructions as described on https://colmweb.org/AuthorGuide.html

Award Nomination: true

Submission Number: 31