"Why Is There a Tumor?": Tell Me the Reason, Show Me the Evidence

Mengmeng Ma, Tang Li, Yunxiang Peng, Lu Lin, Volkan Beylergil, Binsheng Zhao, Oguz Akin, Xi Peng

Abstract: Medical AI models excel at tumor detection and segmentation. However, their latent representations often lack explicit ties to clinical semantics, producing outputs less trusted in clinical practice. Most of the existing models generate either segmentation masks/labels (localizing where without why) or textual justifications (explaining why without where), failing to ground clinical concepts in spatially localized evidence. To bridge this gap, we propose to develop models that can justify the segmentation or detection using clinically relevant terms and point to visual evidence. We address two core challenges: First, we curate a rationale dataset to tackle the lack of paired images, annotations, and textual rationales for training. The dataset includes 180K image-mask-rationale triples with quality evaluated by expert radiologists. Second, we design rationale-informed optimization that disentangles and localizes fine-grained clinical concepts in a self-supervised manner without requiring pixel-level concept annotations. Experiments across medical benchmarks show our model demonstrates superior performance in segmentation, detection, and beyond. The anonymous link to our code.

Lay Summary: Artificial intelligence (AI) can identify tumors in medical scans like MRIs, but doctors often hesitate to trust these results because the AI can’t explain why it thinks a region is cancerous. Currently, AI either highlights tumor locations or gives text explanations, but rarely both at once. To solve this, we developed an AI system that can justify its cancer predictions using terms familiar to doctors and point to visual evidence. We trained it on 180,000 expert-reviewed medical scans paired with radiologists’ notes, teaching the AI to tie its explanations to specific image regions. Tests show our AI outperforms existing tools in both accuracy and interpretability.

Link To Code: https://github.com/deep-real/MedRationale

Primary Area: Applications->Health / Medicine

Keywords: Trustworthy Medical Image Analysis

Submission Number: 548