Action Without Interaction: Probing the Physical Foundations of Video LMMs via Contact-Release Detection

Daniel Harari, Michael Sidorov, Chen Shterental, Liel David, Abrham Kahsay Gebreselasie, Muhammad Haris Khan

Track: Track 1: Papers with IEEE/CVF Workshop Proceedings

Keywords: Robustness and Generalization, Dual-process Reasoning, Compositional and Structured Reasoning, foundational grounding limitations, interaction understanding, contact\release detection, video understanding

TL;DR: Action Without Interaction: Probing LMMs via Contact-Release Detection

Abstract:

Large multi-modal models (LMMs) show increasing performance in realistic visual tasks for images and, more recently, for videos. For example, given a video sequence, such models are able to describe in detail objects, the surroundings and dynamic actions. In this study, we explored the extent to which these models ground their semantic understanding in the actual visual input. Specifically, given sequences of hands interacting with objects, we asked models when and where the interaction begins or ends. For this purpose, we introduce a first of its kind, large-scale dataset with more than 20K annotated interactions on videos from the Something-Something-V2 dataset. 250 AMTurk human annotators labeled core interaction events, particularly when and where objects and agents become attached ('contact') or detached ('release'). We asked SoTA LMMs, including GPT, Gemini and Qwen to locate these events in short videos, each with a single event. The results show that while models reliably name target objects and identify actions, they exhibit a form of 'shortcut learning' where semantic success masks a failure in physical grounding.
Specifically, they consistently fail to identify the frame where the interaction begins or ends and poorly localize the physical event within the scene. This disconnect suggests that while LMMs excel at System 1 intuitive pattern recognition (naming the action and objects), they lack the System 2 cognitive foundations required to reason about physical primitives like 'contact' and 'release.', hence truly ground dynamic scenes in physical reality.

Submission Number: 13