DCBM: Data-Efficient Visual Concept Bottleneck Models

Katharina Prasse, Patrick Knab, Sascha Marton, Christian Bartelt, Margret Keuper

TL;DR: Data-efficient CBMs (DCBMs) generate concepts from image regions detected by segmentation or detection models.

Abstract:

Concept Bottleneck Models (CBMs) enhance the interpretability of neural networks by basing predictions on human-understandable concepts. However, current CBMs typically rely on concept sets extracted from large language models or extensive image corpora, limiting their effectiveness in data-sparse scenarios. We propose Data-efficient CBMs (DCBMs), which reduce the need for large sample sizes during concept generation while preserving interpretability. DCBMs define concepts as image regions detected by segmentation or detection foundation models, allowing each image to generate multiple concepts across different granularities. Exclusively containing dataset-specific concepts, DCBMs are well suited for fine-grained classification and out-of-distribution tasks. Attribution analysis using Grad-CAM demonstrates that DCBMs deliver visual concepts that can be localized in test images. By leveraging dataset-specific concepts instead
of predefined or general ones, DCBMs enhance adaptability to new domains. The code is available at: https://github.com/KathPra/DCBM.

Lay Summary:

AI models are often powerful but difficult to understand. In high-stakes areas like healthcare or science, it’s not enough for a model to be correct — we also need to know why it made a certain prediction. We investigate the task of predicting what is the content of an image. One way to improve transparency is to base model decisions on human-understandable concepts. For example, predict that it is an image of a "car" because it has concept "wheel" and concept "steering wheel". This type of models are called concept bottleneck models (CBMs). CBMs differ in the way they select the concepts for the explanations.

We introduce Data-efficient Concept Bottleneck Models (DCBMs) — a new method that works even when we have few images. Instead of relying on predefined concepts, DCBMs use modern AI tools to detect meaningful regions of each image, automatically creating visual concepts that are specific to the dataset. This allows each image to contribute multiple, interpretable pieces of information at different levels of detail.

Since DCBMs concept are derived from a single dataset instead of a large general dataset, they perform well for fine-grained tasks such as predicting what breed of bird is in an image. Additionally, it can be used for many different datasets, as we have showed its performance does not depend on the dataset. Even when changing the style of the image, for example a photo of a strawberry to a painting of a strawberry, the model can explain its decisions well. By focusing on what can be seen in the images — and not on what we expect to see — DCBMs make machine learning more transparent, accessible, and trustworthy in real-world applications.

Application-Driven Machine Learning: This submission is on Application-Driven Machine Learning.

Link To Code: https://github.com/KathPra/DCBM

Primary Area: Applications->Computer Vision

Keywords: Explainable AI, concept bottleneck models, foundation models

Submission Number: 13036