Go to ICLR 2025 Conference homepageDecision Rules are in the Pixels: Towards Pixel-level Evaluation of Saliency-based XAI Models
The intricate and opaque nature of deep neural networks (DNNs) makes it difficult to decipher how they make decisions. Explainable artificial intelligence (XAI) has emerged as a promising remedy to this conundrum. However, verifying the correctness of XAI methods remains challenging, due to the absence of universally accepted ground-truth explanations. In this study, we focus on assessing the correctness of saliency-based XAI models applied to DNN-based image classifiers at the pixel level. The proposed evaluation protocol departs significantly from previous human-centric correctness assessment at the semantically meaningful object part level, which may not correspond to the actual decision rules derived by classifiers. A crucial step in our approach involves introducing a spatially localized shortcut, a form of decision rule that DNN-based classifiers tend to adopt preferentially, without disrupting original image patterns and decision rules therein. After verifying the shortcut as the dominant decision rule, we estimate the Shapley value for each pixel within the shortcut area to generate the ground-truth explanation map, assuming that pixels outside this area have null contributions. We quantitatively evaluate fourteen saliency-based XAI methods for classifiers utilizing convolutional neural networks and vision Transformers, trained on perturbed CIFAR-10, CIFAR-100, and ImageNet datasets, respectively. Comprehensive experimental results show that existing saliency-based XAI models struggle to offer accurate pixel-level attributions, casting doubt on the recent progress in saliency-based XAI.