Explaining is Harder Than Predicting Alone: Evaluating Concept-based Explanations of MLLMs as ICL Visual Classifiers

Carmen Quiles Ramírez, Leticia Lorena Rodriguez, Nicolas Martorell, Natalia Díaz-Rodríguez

Keywords: In-Context Learning, Few-Shot Learning, Multimodal LLMs, Explainable AI, Semantic Explanations, Description Logics, Compositional Reasoning, LLM-as-a-Judge, Visual Classification

Abstract:

In-context learning (ICL) enables multimodal large language models (MLLMs) to classify images from a few labeled examples. However, how these models leverage the provided context remains poorly understood. While Chain-of-Thought prompting is widely adopted, recent work suggests it may not faithfully reflect the models’ internal reasoning processes. In this paper, we systematically evaluate the concept-based explainability of frozen MLLMs under few-shot ICL across five conditions of increasing formal rigor, ranging from baseline classification to Description Logics (DL) axiom generation.

We assess four state-of-the-art MLLMs using an independent LLM-as-a-judge evaluation pipeline and show that explaining is genuinely harder than predicting. Notably, enforcing formally structured, concept-based explanations leads to a monotonic decrease in predictive accuracy (from 93.8% to 90.1%), challenging the common assumption that explicit reasoning universally improves performance. At the same time, when models successfully identify class-discriminative visual features, explanation quality strongly correlates with correct predictions.

Overall, our results indicate that while MLLMs are highly effective at visual classification, they lack the instruction tuning necessary to produce formal, machine-verifiable explanations.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 87