Towards Multi-Label Concept Bottleneck Models in Medical Imaging: An Exploratory Survey

Berthine Nyunga Mpinda, Mehran Hosseinzadeh, Valay Bundele, Hendrik Lensch

Keywords: Concept bottleneck models, Multi-label classification, vision-language models, interpretability

Abstract: Deep neural networks achieve strong performance in medical image classification, but their lack of interpretability hinders clinical adoption. Concept Bottleneck Models (CBMs) address this by predicting human-understandable intermediate concepts, yet prior work has focused almost exclusively on single-label tasks and has not examined CBMs under the multi-label conditions typical of medical imaging. Because multiple concepts may appear in an image regardless of the final task, CBMs are highly sensitive to class imbalance, co-occurring pathologies, and concept noise. We present the first systematic study of label-free CBMs for multi-label chest X-ray classification, using LLMs to generate concepts and VLMs to align them with images. We evaluate performance, robustness, and interpretability under realistic clinical conditions, analyzing long-tail label distributions, label co-occurrence, and the impact of VLM choice on concept alignment and downstream prediction. Experiments show that label-free CBMs achieve competitive AUROC but reduced precision on minority classes, with performance strongly influenced by backbone selection and class imbalance. Medical-domain VLMs (e.g., BiomedCLIP, BioViL, RAD-DINO) provide consistent gains, and balanced losses improve minority-class sensitivity. Concept-level analysis indicates that pseudo-concepts are reliable for common conditions but unstable for rare ones, affecting multi-label prediction quality. Overall, this work offers the first comprehensive evaluation of label-free CBMs in multi-label medical imaging and practical guidelines for building interpretable, clinically meaningful models.

Primary Subject Area: Interpretability and Explainable AI

Secondary Subject Area: Foundation Models

Registration Requirement: Yes

Visa & Travel: Yes

Read CFP & Author Instructions: Yes

Originality Policy: Yes

Single-blind & Not Under Review Elsewhere: Yes

LLM Policy: Yes

Submission Number: 46