[Re] On the Reproducibility of CartoonX

Robin Sasse, Aniek Eijpe, Jona Ruthardt, Elias Dubbeldam

Keywords: Machine Learning, Artificial Intelligence, Deep Learning, Reproducibility, Explainable AI, Explanation Methods, Image Classifiers, CartoonX, Vision Transformer, runtime efficiency, Python, rescience c, Explainable AI, XAI, Computer Vision, Reproducibility, Transparency

TL;DR: Reproducing and extending the paper "Cartoon Explanations of Image Classifiers"

Abstract:

Scope of Reproducibility — CartoonX [1] is a novel explanation method for image classifiers. In this reproducibility study, we examine the claims of the original authors of CartoonX that it: (i) extracts relevant piece‐wise smooth parts of the image, resulting in explanations which are more straightforward to interpret for humans; (ii) achieves lower distortion in the model output, using fewer coefficients than other state-of‐the‐art methods; (iii) is model‐agnostic. Finally, we examine how to reduce the runtime.

Methodology — The original authors’ open‐sourced implementation has been used to examine (i). We implemented the code to examine (ii), as there was no public code available for this. We tested claim (iii) by performing the same experiments with a Vision Transformer instead of a CNN. To reduce the runtime, we extended the existing implementation with multiple enhanced initialization techniques. All experiments took approximately 38.4 hours on a single NVIDIA Titan RTX.

Results — Our results support the claims made by the original authors. (i) We observe that CartoonX produces piece‐wise smooth explanations. Most of the explanations give valuable insights. (ii) Most experiments, that show how CartoonX achieves lower distortion outputs compared to other methods, have been reproduced. In the cases where exact reproducibility has not been achieved, claim (ii) of the author still holds. (iii) The model‐agnosticism claim still holds as the overall quality of the ViT‐based explanations almost matches that of the CNN‐based explanations. Finally, simple heuristical initializations did not improve the runtime.

What was easy — The mathematical background and intuition of CartoonX were clearly explained by the original authors. Moreover, the original author’s code was well structured and documented, which made it straightforward to run and extend.

What was difficult — Some hyperparameter settings and implementation details needed to reproduce the experiments were not clear or transparent from the original paper or code. This made it difficult to implement and reproduce these experiments.

Communication with original authors — We have been in brief communication with the original authors. They were able to address most of our points, providing us with some additional clarifications about the exact implementation and hyperparameter settings.

Paper Url: https://doi.org/10.1007/978-3-031-19775-8_26

Paper Review Url: https://openreview.net/forum?id=RYTBAtyXqJ

Paper Venue: ECCV 2022

Confirmation: The report follows the ReScience latex style guides as in the Reproducibility Report Template (https://paperswithcode.com/rc2022/registration)., The report contains the Reproducibility Summary in the first page.

Journal: ReScience Volume 9 Issue 2 Article 10

Doi: https://www.doi.org/10.5281/zenodo.8173672

Code: https://archive.softwareheritage.org/swh:1:dir:691b7400b38017cfe651a59735dc715755388c4d