Reproducibility Study of "Languange-Image COnsistency"

TMLR Paper2247 Authors

Abstract:

This report aims to verify the findings and expand upon the evaluation and training methods from the paper LICO: Explainable Models with Language-Image COnsistency. The
main claims are that LICO (i) enhances interpretability by producing more explainable
saliency maps in conjunction with a post-hoc explainability method and (ii) improves image classification performance without computational overhead during inference. We have
reproduced the key experiments conducted by Lei et al.; however, the obtained results
do not support the original claims. Additionally, we identify a vulnerability in the paper’s main evaluation method that favors non-robust models, and propose robust experimental setups for quantitative analysis. Furthermore, we undertake additional studies
on LICO’s training methodology to enhance its interpretability. Our code is available at
https://anonymous.4open.science/r/lico-reproduction-7FEB.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission:

We have incorporated some of the changes requested by the reviewers. In particular:
- we expand on our explanation of the original LICO method to clarify certain aspects of it
- we provide justification for our choice of datasets with a reasoning why they should be enough to observe trends reported by the original authors
- we clarify aspects of the Insertion/Deletion and MOSU metrics calculation; additionally, we add a new figure providing visual intuition on how the metrics ought to be computed
- we add the original images to Figure 2 (now Figure 3) due to the fact that the overlaid salience was indeed to strong and made it difficult to examine the test images

In addition to the above main point, we improved wording and fixed minor mistakes throughout our work. In accordance with the reviewers suggestions, we have tried to soften the wording at some parts of our study.

Assigned Action Editor: ~Krzysztof_Jerzy_Geras1

Submission Number: 2247