Fooling SHAP with Stealthily Biased Sampling

gabriel laberge, Ulrich Aïvodji, Satoshi Hara, Mario Marchand, Foutse Khomh

Keywords: Explainability, Robustness, SHAP, Stealthily Sampling

TL;DR: We show that Shapley-based explanation techniques commonly used in ML can be manipulated to show false compliance (e.g., during an algorithmic fairness audit) and that this type of attack can be hard to detect.

Abstract:

SHAP explanations aim at identifying which features contribute the most to the difference in model prediction at a specific input versus
a background distribution. Recent studies have shown that they can be manipulated by malicious adversaries to produce arbitrary desired
explanations. However, existing attacks focus solely on altering the black-box model itself. In this paper, we propose a complementary family
of attacks that leave the model intact and manipulate SHAP explanations using stealthily biased sampling of the data points used to approximate expectations w.r.t the background distribution. In the context of fairness audit, we show that our attack can reduce the importance of a sensitive feature when explaining the difference in outcomes between groups while remaining undetected. More precisely, experiments performed on real-world datasets showed that our attack could yield up to a 90\% relative decrease in amplitude of the sensitive feature attribution. These results highlight the manipulability of SHAP explanations and encourage auditors to treat them with skepticism.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Social Aspects of Machine Learning (eg, AI safety, fairness, privacy, interpretability, human-AI interaction, ethics)

Community Implementations: [ 3 code implementations](https://www.catalyzex.com/paper/arxiv:2205.15419/code)