The Manifold Hypothesis for Gradient-Based Explanations

Sebastian Bordt, Uddeshya Upadhyay, Zeynep Akata, Ulrike von Luxburg

Keywords: Interpretability, Explainability

Abstract: When are gradient-based explanations meaningful? We propose a necessary criterion: explanations need to be aligned with the tangent space of the data manifold. To test this hypothesis, we employ autoencoders to estimate and generate data manifolds. Across a range of different datasets -- MNIST, EMNIST, CIFAR10, X-ray pneumonia and Diabetic Retinopathy detection -- we demonstrate empirically that the more an explanation is aligned with the tangent space of the data, the more interpretable it tends to be. In particular, popular post-hoc explanation methods such as Integrated Gradients and SmoothGrad tend to align their results with the data manifold. The same is true for the outcome of adversarial training, which has been claimed to lead to more interpretable explanations. Empirically, alignment with the data manifold happens early during training, and to some degree even when training with random labels. However, we theoretically prove that good generalization of neural networks does not imply good or bad alignment of model gradients with the data manifold. This leads to a number of interesting follow-up questions regarding gradient-based explanations.

One-sentence Summary: We explore a simple criterion for gradient-based explanations: That they lie in the tangent space of the data manifold.

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2206.07387/code)