Keywords: Label-Free Explainability, Explainability, Interpretability, Unsupervised models
TL;DR: Paper is a reproducibility study of "Label-Free Explainability for Unsupervised Models".
Abstract: Reproducibility Summary Scope of Reproducibility — This work studies the reproducibility of the paper ”Label‐Free Explainability for Unsupervised Models” by Crabbé and van der Schaar to validate their main claims. These state that: (1) their extension of linear feature importance methods to the label‐free setting is able to extract the key attributes of the data, (2) the adaptation of example importance methods to the unsupervised setting succeeds in highlighting the most influential examples, (3) different pretext tasks do not produce interchangeable representations and (4) the interpretability of saliency maps is uncorrelated to the level of disentanglement between individual latent units. Methodology — The authors provided the code written in PyTorch needed to reproduce all the experiments. Some parts of the code were modified in order to extend the original experiments. The total computation time required to perform the original and extended versions of the experiments is 103 GPU hours. Most of the experiments were performed on NVIDIA TITAN RTX GPU. Results — The plots supporting the label‐free feature and example importance match the ones from the paper, except for the label‐free feature importance experiment for CIFAR‐ 10. Similarly, the Pearson correlation results were successfully reproduced. Due to the nature of the autoencoders used for evaluation, we could not obtain the exact numerical results. However, we visually and numerically compare the trends, and in most cases, we observe that our results are similar to the ones in the paper. What was easy — The paper comes with publicly available code and an extensive appendix containing the setup for all experiments. With that, we were able to reproduce all the experiments with only minor changes to the code. What was difficult — Despite the fact that running the original experiments was straight‐ forward, extending them to new datasets or models was more challenging. Moreover, some of the experiments are more resource‐consuming and require more time to run. Communication with original authors — We contacted the authors to resolve our concerns regarding some of the results. They were very helpful and answered all of our questions. Moreover, they provided us with a pre‐trained SimCLR model. We used this model to validate our results.
Paper Url: https://proceedings.mlr.press/v162/crabbe22a/crabbe22a.pdf
Paper Venue: ICML 2022
Confirmation: The report pdf is generated from the provided camera ready Google Colab script, The report metadata is verified from the camera ready Google Colab script, The report contains correct author information., The report contains link to code and SWH metadata., The report follows the ReScience latex style guides as in the Reproducibility Report Template (https://paperswithcode.com/rc2022/registration)., The report contains the Reproducibility Summary in the first page., The latex .zip file is verified from the camera ready Google Colab script
Journal: ReScience Volume 9 Issue 2 Article 16