### Where do Models go Wrong? Parameter-Space Saliency Maps for Explainability
This repository contains the implementation of parameter-saliency methods from our paper Where do Models go Wrong? Parameter-Space Saliency Maps for Explainability.


Getting started
---------------
This repo uses <a href = "https://www.python.org/downloads/">Python 3</a>. To install the requirements, run
```bash
pip install -r requirements.txt
```

Basic Use
---------
The script input_saliency.py computes both the parameter-saliency profile of an image which allows to find misbehaving filters in a neural network responsible for misclassification of a given image. In addition, the script computes the image-space saliency which highlights pixels which drive the high filter saliency values.

To compute the parameter saliency profile for a given image, the script accepts 
* either path to the raw image + image target label
```bash
python3 parameter_and_input_saliency.py --model resnet50 --image_path raw_images/great_white_shark_mispred_as_killer_whale.jpeg --image_target_label 2
```
* or reference_id -- the index of the given image in ImageNet validation set (path to ImageNet would need to be specified in the code).
```bash
python3 parameter_and_input_saliency.py --reference_id 107 --k_salient 10
```

here --reference_id specifies the image id from ImageNet validation set

--k_salient specifies the number of top salient filters to use for the input-space visualization

The resulting plots (input space colormap and filter saliency plot) will be saved to /figures

Demo
-----
The demo raw image is in /raw_images. The results are in /figures.

Project Organization
------------
    ├── README.md
    ├── requirements.txt 
    ├── utils.py  <- helper functions       
    ├── parameter_and_input_saliency.py  <- main script which computes both input saliency and parameter saliency
    │
    ├── figures <- folder for resulting figures
    ├── helper_objects <- precomputed objects to speed up computation (inference results on ImageNet valset and parameter saliency mean and std for standardization)
    │   └─ resnet50   
    ├── raw_images <- images to use for parameter space saliency computation and for input space saliency visualization
    └── parameter_saliency
        └── saliency_model_backprop  <- script with SaliencyModel class, parameter saliency implementation 
