[Re] FOCUS: Flexible Optimizable Counterfactual Explanations for Tree Ensembles

Kyosuke Morita

Keywords: Counterfactual explanation

Abstract:

Scope of Reproducibility

This study aims to reproduce the results of the paper 'FOCUS: Flexible Optimizable Counterfactual Explanations for Tree Ensembles' by Lucic et al.
The main claims of the original paper are that FOCUS is able to (i) generate counterfactual explanations for all the instances in a dataset;
and (ii) find counterfactual explanations that are closer to the original input for tree-based algorithms than existing methods.

Methodology
This study replicates the original experiments using the code, data, and models provided by the authors. Additionally, this study re-implements code and retrains the models to evaluate the robustness and generality of FOCUS.
All the experiments were conducted on a personal laptop with a quad-core CPU with 8GB of RAM and it approximately took 33 hours in total.

Results
This study was able to replicate the results of the original paper in terms of finding counterfactual explanations for all instances in datasets. Additional experiments were conducted to validate the robustness and generality of the conclusion. While there were slight deviations in terms of generating smaller mean distances, half of the models still outperformed the results of the existing method.

What was easy
The implementation of the original paper is publicly available on GitHub. The repository contains the models and data used in the original experiments. Also, the authors provided a technical appendix, which includes all the hyperparameters that were used for the experiments for reproduction upon request.

What was difficult
Although the implementation code was available, it employs outdated packages and the code structure is complex. Also, the comments in the functions and the documentation of the code are sparse or nonexistent, which made it difficult to follow the code.

Communication with original authors
I reached out to the authors to obtain the hyperparameters used in the experiments. The authors responded promptly with a detailed technical appendix of the original paper.

Paper Url: https://arxiv.org/abs/1911.12199

Paper Venue: AAAI 2022

Confirmation: The report pdf is generated from the provided camera ready Google Colab script, The report metadata is verified from the camera ready Google Colab script, The report contains correct author information., The report contains link to code and SWH metadata., The report follows the ReScience latex style guides as in the Reproducibility Report Template (https://paperswithcode.com/rc2022/registration)., The report contains the Reproducibility Summary in the first page., The latex .zip file is verified from the camera ready Google Colab script

Journal: ReScience Volume 9 Issue 2 Article 12

Doi: https://www.doi.org/10.5281/zenodo.8173678

Code: https://archive.softwareheritage.org/swh:1:dir:e096a518285f9ee2f9ee2c5943293ba30f7e17b0