CELF: A Self-Supervised Multimodal Framework for Concept-Based Interpretability

ICLR 2026 Conference Submission13361 Authors

Keywords: Explainable Artificial Intelligence, Concept-based XAI, Large Language Models, Concept Extraction, Vision-Language Models, Self-Supervised Learning, Multimodal Representation Learning

TL;DR: Conceptual-MNIST, a vision-language dataset generator with controllable ground truth concepts, is proposed to support the evaluation of CELF, a novel framework that extracts concepts through self-supervised multimodal learning.

Abstract: Traditional XAI techniques in computer vision, such as heatmaps and saliency maps, highlight input regions that influence model predictions. However, they often lack precision and may introduce bias. Concept-based XAI approaches, such as concept bottleneck models or textual explanations of latent neurons, aim to provide more interpretable representations but typically rely on human-annotated concept sets, which are scarce in specialized domains. Moreover, Large Language Models (LLMs) used for automatic concept generation can hallucinate, reducing reliability and trust. To address these challenges, we propose the Concept Extraction and Learning Framework (CELF), a self-supervised multimodal method for extracting and retrieving human-interpretable concepts from vision-language data without manual annotations. CELF integrates attention-guided keyphrase extraction with contrastive learning, and applies a graph-guided refinement stage to promote semantic consistency. For controlled evaluation, we introduce C-MNIST, a configurable dataset generator with ground truth concepts. Experiments on C-MNIST, Visual Genome, and CUB demonstrate that CELF outperforms prior baselines in concept extraction and improves multi-label classification performance on C-MNIST.

Primary Area: interpretability and explainable AI

Submission Number: 13361