Conceptual Graph Counterfactuals

Angeliki Dimitriou, Maria Lymperaiou, Georgios Filandrianos, Konstantinos Thomas, Giorgos Stamou

Primary Area: societal considerations including fairness, safety, privacy

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Graph Neural Networks, Graph Matching, Graph Similarity, Scene Graphs, Counterfactual Explanations, Conceptual Counterfactuals

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: Model-agnostic retrieval of counterfactual images based on scene graph representations obtained via graph neural networks.

Abstract:

Conceptual counterfactuals refer to hypothetical scenarios involving changes in a
high-level conceptual representation. In the realm of XAI, conceptual Counterfac
tual Explanations (CEs) allow for more meaningful and interpretable modifications.
For instance, instead of explaining image predictions through superficial pixel-level
changes, the focus shifts to alterations in the underlying semantics. In this work,
we propose representing input data as semantic graphs to achieve more descriptive,
accurate, and human-aligned explanations. Furthermore, we introduce a model-
agnostic GNN-powered method to efficiently compute counterfactuals. We begin
by representing images as scene graphs and obtain appropriate representations
through GNNs to bypass solving the NP-hard graph similarity problem for all input
pairs, an integral part of the CE computation process. We apply our method to
widely-used datasets and compare our CEs with previous state-of-the-art explana
tion models based on semantics, including both white and black-box approaches.
We outperform both approaches quantitatively and qualitatively, as validated by
human subjects, specifically when the graphs contain numerous edges, highlighting
the significance of capturing intricate relationships. Given the model-agnostic
nature of our approach and the generalizability of the graph representation, this
method is successfully extended to diverse modalities and classifiers, including
non-neural models. Additionally, it is proven consistent across generated anno
tations, at least in the case of scene graph generation. Our approach is, to our
knowledge, the first to emphasize semantic graphs as a vehicle for CEs, allowing
the transition from low-level features to concepts. It uniquely leverages graph
matching GNNs as a XAI tool achieving efficient approximation and significant
acceleration in comparison to the exact Graph Edit Distance (GED) algorithm. It is
widely applicable and easily extensible, producing actionable explanations.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4995