Decision Rules are in the Pixels: Towards Pixel-level Evaluation of Saliency-based XAI Models

Kanglong FAN, Chen Ma, YIFAN PENG, Yuming Fang, Kede Ma

Keywords: post-hoc XAI, evaluation, pixel attributions, shapley value, feature deletion

Abstract: The intricate and opaque nature of deep neural networks (DNNs) makes it difficult to decipher how they make decisions. Explainable artificial intelligence (XAI) has emerged as a promising remedy to this conundrum. However, verifying the correctness of XAI methods remains challenging, due to the absence of universally accepted ground-truth explanations. In this study, we focus on assessing the correctness of saliency-based XAI models applied to DNN-based image classifiers at the pixel level. The proposed evaluation protocol departs significantly from previous human-centric correctness assessment at the semantically meaningful object part level, which may not correspond to the actual decision rules derived by classifiers. A crucial step in our approach involves introducing a spatially localized shortcut, a form of decision rule that DNN-based classifiers tend to adopt preferentially, without disrupting original image patterns and decision rules therein. After verifying the shortcut as the dominant decision rule, we estimate the Shapley value for each pixel within the shortcut area to generate the ground-truth explanation map, assuming that pixels outside this area have null contributions. We quantitatively evaluate fourteen saliency-based XAI methods for classifiers utilizing convolutional neural networks and vision Transformers, trained on perturbed CIFAR-10, CIFAR-100, and ImageNet datasets, respectively. Comprehensive experimental results show that existing saliency-based XAI models struggle to offer accurate pixel-level attributions, casting doubt on the recent progress in saliency-based XAI.

Primary Area: interpretability and explainable AI

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 1262