Are Large Language Models Good XAI Annotators?

Ao Sun, Yuanyuan Yuan, Pingchuan Ma, Shuai Wang

Keywords: XAI, LLM, Concept-based Model

Abstract: Explainable AI (XAI) methods for deep neural networks (DNNs) typically rely on costly annotations to supervise concept–class relationships. To alleviate this burden, recent studies have leveraged large language models (LLMs) and vision–language models (VLMs) to automatically generate these annotations. However, the sufficiency of such automated annotations — whether the generated concepts sufficiently characterize their corresponding classes — remains underexplored. In this paper, we propose the *Fast and Slow Effect* (FSE), a unified evaluation framework designed to assess annotation sufficiency without human supervision. FSE first guides the LLMs to progressively annotate concept–class test cases along a continuum, ranging from a *fast mode*, involving opaque visual labeling without any conceptual reasoning, to a *slow mode*, employing a multi–step, conceptual coarse–to–fine annotation strategy. Then, to systematically validate the sufficiency at each step, our framework leverages the models to self–evaluate annotations using the *Class Representation Index* (CRI), a metric designed to measure how sufficiently annotated concepts represent the target classes against semantically similar alternatives. Our experiments reveal that the current annotation methods fail to provide sufficient semantic coverage for accurate concept–class mapping, especially in fine–grained datasets. Specifically, a significant performance gap is observed between fast and slow modes, with the CRI dropping by over **25%** on average in slow mode, indicating that while the annotators’ intrinsic knowledge enables rapid inference, it remains challenging for them to conceptualize this knowledge in the slow mode, making such expertise difficult to access and interpret. These findings underscore the need for more transparent frameworks to enable reliable, concept–aware annotation in XAI.

Primary Area: interpretability and explainable AI

Submission Number: 10159