Explaining How Visual, Textual and Multimodal Encoders Share Concepts

ICLR 2026 Conference Submission14010 Authors

Keywords: Explainability, Sparse Autoencoders, Multimodality

TL;DR: We conduct a large scale study on how visual, textual and multimodal models share concepts across modalities and introduce two new dedicated indicators.

Abstract: Sparse autoencoders (SAEs) have emerged as a powerful technique for extracting human-interpretable features from neural networks activations. Previous works compared different models based on SAE-derived features but those comparisons have been restricted to models within the same modality. We propose a novel indicator allowing quantitative comparison of models across SAE features, and use it to conduct a comparative study of visual, textual and multimodal encoders. We also propose to quantify the *Comparative Sharedness* of individual features between different classes of models. With these two new tools, we conduct several studies on 21 encoders of the three types, with two significantly different sizes, and considering generalist and domain specific datasets. The results allow to revisit previous studies at the light of encoders trained in a multimodal context and to quantify to which extent all these models share some representations or features. They also suggest that visual features that are specific to VLMs among vision encoders are shared with text encoders, highlighting the impact of text pretraining.

Primary Area: interpretability and explainable AI

Submission Number: 14010