Visual symbolic mechanisms: Emergent symbol processing in Vision Language Models

Rim Assouel, Declan Iain Campbell, Yoshua Bengio, Taylor Whittington Webb

Keywords: visual object binding, vision-langue model, symbolic reasoning, interpretability

TL;DR: We describe a set of symbolic-like mechanisms that VLMs use to bind to visual entities in context

Abstract: To accurately process a visual scene, observers must bind features together to represent individual objects. This capacity is necessary, for instance, to distinguish an image containing a red square and a blue circle from an image containing a blue square and a red circle. Recent work has found that language models solve this ‘binding problem’ via a set of symbol-like, content-independent indices, but it is unclear whether similar mechanisms are employed by Vision Language Models (VLM). This question is especially relevant, given the persistent failures of VLMs on tasks that require binding. Here, we identify a previously unknown set of emergent symbolic mechanisms that support binding specifically in VLMs, via a content-independent, spatial indexing scheme. Moreover, we find that binding errors, when they occur, can be traced directly to failures in these mechanisms. Taken together, these results shed light on the mechanisms that support symbol-like processing in VLMs, and suggest possible avenues for reducing the number of binding failures exhibited by these models.

Primary Area: interpretability and explainable AI

Submission Number: 23139