CARLA: A Python Library to Benchmark Algorithmic Recourse and Counterfactual Explanation AlgorithmsDownload PDF

Published: 29 Jul 2021, Last Modified: 08 Sept 2024NeurIPS 2021 Datasets and Benchmarks Track (Round 1)Readers: Everyone
TL;DR: We present CARLA, a python library for benchmarking counterfactual explanation methods across both different data sets and different machine learning models.
Abstract: Counterfactual explanations provide means for prescriptive model explanations by suggesting actionable feature changes (e.g., increase income) that allow individuals to achieve favourable outcomes in the future (e.g., insurance approval). Choosing an appropriate method is a crucial aspect for meaningful counterfactual explanations. As documented in recent reviews, there exists a quickly growing literature with available methods. Yet, in the absence of widely available open--source implementations, the decision in favour of certain models is primarily based on what is readily available. Going forward -- to guarantee meaningful comparisons across explanation methods -- we present \texttt{CARLA} (\textbf{C}ounterfactual \textbf{A}nd \textbf{R}ecourse \textbf{L}ibr\textbf{A}ry), a python library for benchmarking counterfactual explanation methods across both different data sets and different machine learning models. In summary, our work provides the following contributions: (i) an extensive benchmark of 11 popular counterfactual explanation methods, (ii) a benchmarking framework for research on future counterfactual explanation methods, and (iii) a standardized set of integrated evaluation measures and data sets for transparent and extensive comparisons of these methods. We have open sourced \texttt{CARLA} and our experimental results on \href{https://github.com/indyfree/CARLA}{Github}, making them available as competitive baselines. We welcome contributions from other research groups and practitioners.
Supplementary Material: zip
URL: https://github.com/indyfree/CARLA
Community Implementations: [![CatalyzeX](/images/catalyzex_icon.svg) 1 code implementation](https://www.catalyzex.com/paper/carla-a-python-library-to-benchmark/code)
7 Replies

Loading