Saliency is a Possible Red Herring When Diagnosing Poor Generalization

Joseph D Viviano, Becks Simpson, Francis Dutil, Yoshua Bengio, Joseph Paul Cohen

Keywords: Feature Attribution, Generalization, Saliency

Abstract: Poor generalization is one symptom of models that learn to predict target variables using spuriously-correlated image features present only in the training distribution instead of the true image features that denote a class. It is often thought that this can be diagnosed visually using attribution (aka saliency) maps. We study if this assumption is correct. In some prediction tasks, such as for medical images, one may have some images with masks drawn by a human expert, indicating a region of the image containing relevant information to make the prediction. We study multiple methods that take advantage of such auxiliary labels, by training networks to ignore distracting features which may be found outside of the region of interest. This mask information is only used during training and has an impact on generalization accuracy depending on the severity of the shift between the training and test distributions. Surprisingly, while these methods improve generalization performance in the presence of a covariate shift, there is no strong correspondence between the correction of attribution towards the features a human expert have labelled as important and generalization performance. These results suggest that the root cause of poor generalization may not always be spatially defined, and raise questions about the utility of masks as 'attribution priors' as well as saliency maps for explainable predictions.

One-sentence Summary: We controlled feature construction on images using masks to help models generalize to test distributions with covariate shift and noticed that it didn't affect the saliency maps in the way one would expect even though it improved generalization.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Data: [ChestX-ray14](https://paperswithcode.com/dataset/chestx-ray14), [PadChest](https://paperswithcode.com/dataset/padchest)

Code: [ josephdviviano/saliency-red-herring](https://github.com/josephdviviano/saliency-red-herring)