Modal Aphasia: Can Unified Multimodal Models Describe Images From Memory?

ICLR 2026 Conference Submission22011 Authors

Keywords: multi-modal language models, memorization, safety, unified representations

TL;DR: We introduce modal aphasia, a surprising and systematic dissociation in which unified multimodal models demonstrate strong capabilities for generating visual content while simultaneously failing to access that same knowledge through text queries.

Abstract:

We present *modal aphasia*, a systematic dissociation in which current unified multimodal models
accurately memorize concepts visually but fail to articulate them in writing, despite being trained on images and text simultaneously.
For one, we show that leading frontier models can generate near-perfect reproductions of iconic movie artwork, but confuse crucial details when asked for textual descriptions.
We corroborate those findings through controlled experiments on synthetic datasets in multiple architectures.
Our experiments confirm that modal aphasia reliably emerges as a fundamental property of current unified multimodal models, not just as a training artifact.
In practice, modal aphasia can introduce vulnerabilities in AI safety frameworks, as safeguards applied to one modality may leave harmful concepts accessible in other modalities.
We demonstrate this risk by showing how a model aligned solely on text
remains capable of generating harmful images.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 22011