Reproducibility Study: Label-Free Explainability for Unsupervised Models

Sławomir Garcarz, Andreas Giorkatzi, Ana Ivășchescu, Theodora-Mara Pîslar

Keywords: Label-Free Explainability, Explainability, Interpretability, Unsupervised models

TL;DR: Paper is a reproducibility study of "Label-Free Explainability for Unsupervised Models".

Abstract:

Reproducibility Summary

Scope of Reproducibility — This work studies the reproducibility of the paper ”Label‐Free
Explainability for Unsupervised Models” by Crabbé and van der Schaar to validate their
main claims. These state that: (1) their extension of linear feature importance methods
to the label‐free setting is able to extract the key attributes of the data, (2) the adaptation
of example importance methods to the unsupervised setting succeeds in highlighting
the most influential examples, (3) different pretext tasks do not produce interchangeable
representations and (4) the interpretability of saliency maps is uncorrelated to the level
of disentanglement between individual latent units.

Methodology — The authors provided the code written in PyTorch needed to reproduce all
the experiments. Some parts of the code were modified in order to extend the original
experiments. The total computation time required to perform the original and extended
versions of the experiments is 103 GPU hours. Most of the experiments were performed
on NVIDIA TITAN RTX GPU.

Results — The plots supporting the label‐free feature and example importance match the
ones from the paper, except for the label‐free feature importance experiment for CIFAR‐
10. Similarly, the Pearson correlation results were successfully reproduced. Due to the
nature of the autoencoders used for evaluation, we could not obtain the exact numerical
results. However, we visually and numerically compare the trends, and in most cases,
we observe that our results are similar to the ones in the paper.

What was easy — The paper comes with publicly available code and an extensive appendix
containing the setup for all experiments. With that, we were able to reproduce all the
experiments with only minor changes to the code.

What was difficult — Despite the fact that running the original experiments was straight‐
forward, extending them to new datasets or models was more challenging. Moreover,
some of the experiments are more resource‐consuming and require more time to run.

Communication with original authors — We contacted the authors to resolve our concerns
regarding some of the results. They were very helpful and answered all of our questions.
Moreover, they provided us with a pre‐trained SimCLR model. We used this model to
validate our results.

Paper Url: https://proceedings.mlr.press/v162/crabbe22a/crabbe22a.pdf

Paper Venue: ICML 2022

Confirmation: The report pdf is generated from the provided camera ready Google Colab script, The report metadata is verified from the camera ready Google Colab script, The report contains correct author information., The report contains link to code and SWH metadata., The report follows the ReScience latex style guides as in the Reproducibility Report Template (https://paperswithcode.com/rc2022/registration)., The report contains the Reproducibility Summary in the first page., The latex .zip file is verified from the camera ready Google Colab script

Journal: ReScience Volume 9 Issue 2 Article 16

Doi: https://www.doi.org/10.5281/zenodo.8173688

Code: https://archive.softwareheritage.org/swh:1:dir:e76ce9ca64bef8b8ab34ef48336017ade33d40b9