Reliability, Faithfulness, and the Limits of Post-hoc Explanations of Opaque Scientific Models

Nick Oh, Helen Jin

Keywords: explainable AI, post-hoc explanation, scientific machine learning, opacity, scientific understanding

TL;DR: A reliable model and a faithful explanation of it do not, by composition alone, license a justified claim about the structure of the phenomenon the model was trained on.

Abstract: Post-hoc explanation methods are routinely used to interpret scientific machine learning models, with the deliverable understood to be insight into the phenomenon the model has been trained on. The transition may be taken to be secured once the model is reliable enough and the explanation faithful enough. We argue it is not. The two standards yield verdicts of different epistemic kinds, and their composition does not deliver a justified claim about the phenomenon's structure. The chain can support candidate hypotheses under external corroboration, but it cannot, on its own, support claims about how the phenomenon is in fact structured.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 120