Reproducibility report for ML Reproducibility Challenge 2022

Anonymous

Keywords: Reproducibility, GNN, Robustness, Counterfactuality, Fidelity, Pandas, AI, UvA

TL;DR: Some findings of original paper were reproducible, but not everything was able to be confirmed.

Abstract:

The goal of this replication study is to find out to what extend the results of the paper "Robust Counterfactual
Explanations on Graph Neural Networks" [1] are reproducable. We investigate this matter by evaluating the following
claims:
• Using the trained RCExplainer and baseline RCExplainer-NoLDB, does it deliver the same performance on
fidelity as stated in the paper?
• Using the trained RCExplainer and baseline RCExplainer-NoLDB, does it deliver the same performance on
robustness as stated in the paper?

In order to reproduce the results as stated in the original paper of Bajaj et al. [2021], we used the originally provided
source code. However, contrary to our expectations, the provided source code on its own was not enough to redo the
experiments. Therefore, the main approach of this reproducibility paper was to adjust the provided source code in order
to execute some of the the conducted experiments. The source code and our adjustments are written in Python and used
the PyTorch library.

The first claim in the scope of reproducibility was not accepted in terms of this paper. The RCExplainer was close to
the results found by manual training but outside of the range provided by the standard deviation of manual training.
The performance of the RCExp-NoLDB model showed a major difference from the results reported in the paper.
The robustness results were hard to compare due to lack of actual numbers, but qualitative analysis found it to be
reproducible, as values lied within margin of the standard deviation of manually trained models.

Paper Url: https://arxiv.org/abs/2107.04086

Paper Venue: NeurIPS 2021