Why Sanity Check for Saliency Metrics Fails?

Revoti Prasad Bora, Kiran Raja, Philipp Terhörst, Raymond Veldhuis, Raghavendra Ramachandra

Primary Area: societal considerations including fairness, safety, privacy

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Saliency Maps, Saliency Metrics, Fidelity Metrics, XAI, Perturbation

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: Study to understand why Saliency Metrics Fail Sanity Checks

Abstract: Saliency maps are widely leveraged as a post-ad-hoc approach to explain the decision-making process of Deep Learning-based image classification models. However, despite their popularity, ensuring the fidelity of saliency maps remains a complex problem. Researchers have, therefore, introduced saliency metrics to evaluate the fidelity of saliency maps. However, previous studies observed several statistical inconsistencies in the existing saliency metrics without investigating the reason behind the inconsistencies. In this study, we investigate the reason behind the observed statistical inconsistencies. We analyze the inconsistencies by studying the variation in pixel importance ranks, specifically by choosing a case study of varying levels of Gaussian blur (with different σ values for the width of the Gaussian Kernel) as the perturbation mechanism. Our findings indicate that the effect of perturbations on prediction probability and pixel importance ranks varies widely across different levels of Gaussian Blur. Consequently, the existing saliency metrics that rely on pixel importance become unreliable for measuring the fidelity of saliency maps. This insight necessitates careful use of saliency metrics and the perturbation technique used while assessing the fidelity of saliency maps in eXplainable AI (XAI). We used Gaussian Blur as our perturbation mechanism, but our approach applies to any perturbation.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4212