Concepts in Motion: Temporal Concept Bottleneck Model for Interpretable Video Classification

Patrick Knab, Sascha Marton, Philipp Johannes Schubert, Drago Andres Guggiana Nilo, Christian Bartelt

Keywords: Concept Bottleneck Model, Video Classification

TL;DR: MoTIF extends concept bottleneck models to video using per-channel temporal attention and agentic concept discovery, enabling faithful and interpretable classification of arbitrary-length clips.

Abstract: Concept Bottleneck Models (CBMs) enable interpretable image classification by structuring predictions around human-understandable concepts, but extending this paradigm to video remains challenging due to the difficulty of extracting concepts and modeling them over time. In this paper, we introduce MoTIF (Moving Temporal Interpretable Framework), a transformer-based concept architecture that operates on sequences of temporally grounded concept activations, by employing per-concept temporal self-attention to model when individual concepts recur and how their temporal patterns contribute to predictions. Central to the framework is a class-conditioned VLM-based concept discovery module that extracts object- and action-centric textual concepts from training videos, yielding temporally expressive concept sets without manual concept annotation. Across multiple video benchmarks, this combination improves over global concept bottlenecks and remains competitive within the interpretable concept-bottleneck setting, while narrowing the gap to strong black-box video baselines that we report as contextual references.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 48