Do Explanations Help or Hurt? Saliency Maps vs Natural Language Explanations in a Clinical Decision-Support Setting
Abstract: As AI models are becoming more powerful, their adoption is becoming more widespread, including in safety-critical domains. Explainable AI (XAI) has the aim of making these models safer to use, for instance by making their decision-making process more transparent. However, current explainability methods are seldom evaluated in the way they are intended to be used: by real-world end users. To address this, we conducted a large-scale user study with 85 clinicians in the context of human-AI collaborative chest X-ray analysis. We evaluated three types of explanations: saliency maps, natural language explanations, and their combination. We specifically examine how different explanation types influence users depending on whether the AI is correct. We find that the quality of explanations, i.e., how much correct information they entail, and how much this aligns with AI correctness, significantly impacts the usefulness of the different explanation types.
Paper Type: Long
Research Area: Interpretability and Analysis of Models for NLP
Research Area Keywords: Human-Centered NLP, Interpretability and Analysis of Models for NLP, Multimodality and Language Grounding to Vision, Robotics and Beyond, NLP Applications
Contribution Types: Model analysis & interpretability, Data analysis
Languages Studied: English
Submission Number: 5622
Loading