Reproducibility study of 'Proto2Proto: Can you recognise the car, the way I do?'

Gerson de Kleuver, David Matthias Bikker, Wenhua Hu, Bram Veenman

Keywords: Reproducibility, Interpretability, Proto2Proto, ProtoPNet, Prototypes, Knowledge distillation

TL;DR: This paper analyses the reproducibility of the study "Proto2Proto: Can you recognize the car, the way I do?" and attempts to confirm the claims made in the paper.

Abstract:

Scope of Reproducibility —

This paper analyses the reproducibility of the study Proto2Proto: Can you recognize the car, the way I do? The main contributions and claims of the study are: 1) Using Proto2Proto, a shallower student model is more faithful to the teacher in terms of interpretability than a baseline student model while also showing the same or better accuracy; 2) Global Explanation loss forces student prototypes to be close to teacher prototypes; 3) Patch‐Prototype Correspondence loss enforces the local representations of the student to be similar to those of the teacher; 4) The proposed evaluation metrics determine the faithfulness of the student to the teacher in terms of interpretability.

Methodology —

A public code repository was available for the paper, which provided a working but incomplete and minimally documented codebase. With some modifications we were able to carry out the experiments that were best supported by the codebase. We spent a total of 60 computational GPU hours on reproduction.

Results —

The results we were able to produce support claim 1, albeit weakly. Further results are in line with the paper, but we found them to go against claim 3. In addition, we carried out a theoretical analysis which provides support for claim 4. Finally, we were unable to carry out our intended experiment to verify claim 2.

What was easy —

The original paper was clearly structured and understandable. The experiments for which configurations were provided were simple to conduct.

What was difficult —

The public codebase contained minimal documentation. Moreover, the use of variable names did not correspond between the code and the paper. Furthermore, the codebase lacked elements vital to reproducing some experiments. Another significant constraint were the computational requirements needed to reproduce the original experiments. Finally, the code required to reproduce one of the visualizations was not provided.

Communication with original authors —

We contacted the authors to ask for trained model weights and missing hyperparameters for several experiments. We did not receive a
response.

Paper Url: https://arxiv.org/abs/2204.11830

Paper Venue: CVPR 2022

Confirmation: The report pdf is generated from the provided camera ready Google Colab script, The report metadata is verified from the camera ready Google Colab script, The report contains correct author information., The report contains link to code and SWH metadata., The report follows the ReScience latex style guides as in the Reproducibility Report Template (https://paperswithcode.com/rc2022/registration)., The report contains the Reproducibility Summary in the first page., The latex .zip file is verified from the camera ready Google Colab script

Journal: ReScience Volume 9 Issue 2 Article 22

Doi: https://www.doi.org/10.5281/zenodo.8173705

Code: https://archive.softwareheritage.org/swh:1:dir:07b0f527f206c44a96373df590b251c5c3bf669d