Textural or Textual: How Visual Models Understand Texts in Images

Hanzhang Wang, Qingyuan Ma

Keywords: Typographic attack, Vision-Language Pre-taining, Intrinsic Dimension, CLIP

TL;DR: This study explores how multimodal visual models represent textual semantics and the mechanisms through which text disrupts visual understanding.

Abstract: It is widely assumed that typographic attacks succeed because multimodal pre-trained visual models can recognize the semantics of text within images, allowing text to interfere with image understanding. However, the assumption that these models truly comprehend textual semantics remains unclear and underexplored. We investigate how the CLIP encoder represents textual semantics and identify the mechanisms through which text disrupts visual semantic understanding. To facilitate this analysis, we propose a novel ToT (Texture or Textual) dataset, which includes a subset that disentangles orthographic forms (i.e., the visual shape of words) from their semantics. Using Intrinsic Dimension (ID) to assess layer-wise representation complexity, we examine whether the representations are built on texture or textual information under typographic manipulations. Contrary to the common belief that semantics are progressively built across layers, we find that texture and semantics compete in the early layers. In the later layers, while semantic accuracy improves, this gain primarily stems from texture learning that aids orthographic recognition. Only in the final block does the visual model construct a semantic-focused representation.

Primary Area: interpretability and explainable AI

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3793