[Re] Reproducibility study of “Explaining Deep Convolutional Neural Networks via Latent Visual-Semantic Filter Attention”

Erik Buis, Sebastiaan Dijkstra, Bram Heijermans

Keywords: Reproduce, Interpretability, Convolutional Neural Networks, Post-hoc global method, Latent Visual Semantic Filter Attention, Python, PyTorch, Common Objects in Context, machine learning, rescience c

TL;DR: Successful reproduction of “Explaining Deep Convolutional Neural Networks via Latent Visual-Semantic Filter Attention”.

Abstract:

Scope of Reproducibility
In this work, we aim to reproduce the findings of the paper Explaining Deep Convolutional Neural Networks via Latent Visual-Semantic Filter Attention (LaViSE). This paper presents a global post-hoc explanation framework for deep learning models that generates semantic explanations for CNN filters. To assess the reproducibility of this work, we verify the main claims made in the paper. More specifically, we evaluate whether the framework creates an accurate mapping to the semantic space, generates words which were not seen in the training data, and is able to generalize to any pre-trained CNN.

Methodology
To reproduce the experiments detailed in the original paper, we first obtained the author's code. However, we had to modify the code for the experiments to be executable, adding missing code, debugging, and making the code more maintainable. Additionally, we evaluated the model's generalizability to other CNNs. The project required a total of 62 GPU hours.

Results
Our recall scores and qualitative experiments validate all claims of the authors: the framework creates an accurate mapping between the visual and semantic space, can analyze any trained CNN regardless of original training data availability, and is able to generate novel out-of-dataset descriptions for filters.

What was easy
The paper was well-written and easy to understand, with helpful figures illustrating the LaViSE framework that aided in the implementation process.

What was difficult
The implementation of the methodology outlined in the paper was particularly challenging due to limited documentation and insufficient details about parts that were not implemented in the existing codebase. Additionally, some experiments could not be recreated because they would require a significant amount of resources to verify.

Communication with original authors
We contacted the authors to clarify missing information and aspects that were not functioning as expected. However, we did not receive a response to our questions.

Paper Url: https://ieeexplore.ieee.org/document/9878481

Paper Review Url: https://openreview.net/forum?id=DOeoQiw89rzy

Paper Venue: CVPR 2022

Confirmation: The report pdf is generated from the provided camera ready Google Colab script, The report metadata is verified from the camera ready Google Colab script, The report contains correct author information., The report contains link to code and SWH metadata., The report follows the ReScience latex style guides as in the Reproducibility Report Template (https://paperswithcode.com/rc2022/registration)., The report contains the Reproducibility Summary in the first page., The latex .zip file is verified from the camera ready Google Colab script

Journal: ReScience Volume 9 Issue 2 Article 25

Doi: https://www.doi.org/10.5281/zenodo.8173713

Code: https://archive.softwareheritage.org/swh:1:dir:3b9cb41cd6d8680801f3b80b29410b641408348e