Interpreting Vision Grounding in Vision-Language Models: A Case Study in Coordinate Prediction

Clement Neo, Yongsen Zheng, Kwok-Yan Lam, Luke Ong

Keywords: Probing, Other, Applications of interpretability

Other Keywords: multimodal, computer-use

TL;DR: We interpret how computer-use models predict screenshot coordinates to carry out actions

Abstract: Vision-language models increasingly power autonomous agents that require precise spatial actions, from computer-use agents clicking interface elements to robots grasping objects. We present the first mechanistic analysis of computer-use models, using UI-TARS 1.5 on a controlled task where models must click colored squares in grid images. We discover a systematic failure mode where the model misclicks approximately 50\% of the time, often targeting locations exactly one patch below the correct target despite high confidence. Through activation patching, layer-wise analysis, and coordinate probing, we reveal that failures stem from biased late-layer selection rather than visual misunderstanding. The model simultaneously maintains accurate representations of both correct and incorrect locations yet systematically outputs wrong coordinates. Our analysis identifies strong patching effects at specific token positions in the final layers, with probes successfully detecting the systematic downward bias. Our work establishes coordinate prediction as a tractable testbed for multimodal interpretability and provides insights for improving spatial grounding reliability in deployed vision-language agents.

Submission Number: 311