Uncovering Object Localization Mechanisms in VLMs

Timothy Schaumlöffel, Martina G. Vilas, Gemma Roig

Keywords: Vision transformers, Other

Other Keywords: VLM,Localization

TL;DR: We show that VLMs localize objects by reconstructing spatial layout from token order, defining boundaries through object-token containerization, and processing localization in early–middle layers.

Abstract: Visually-grounded language models (VLMs) are highly effective in linking visual and textual information, yet they often struggle with basic classification and localization tasks. While classification mechanisms have been studied more extensively, the processes that support object detection remain less clear. In this work, we analyze foundational VLMs and show that image tokens corresponding to the object directly contain the information required for localization. We find that the model applies a containerization mechanism: it uses object-related tokens to define spatial boundaries, while largely discarding semantic context. Our analysis further reveals that this information is processed in the early to middle layers of the language model and that classification and detection rely on shared mechanisms. Finally, we demonstrate that spatial grounding does not come solely from positional encodings in the visual backbone, but rather from residual positional signals combined with the language model’s ability to infer spatial order from token sequences.

Submission Number: 76