Are we using Motion in Referring Segmentation? A Motion-Centric Evaluation

Mennatullah Siam

Keywords: referring video segmentation, multi-modal large language models

Abstract: Multi-modal large language models (MLLMs) have shown impressive generalization across tasks using images and text modalities. While their extension to video modality has enabled tasks such as video question answering and video captioning, their dense spatiotemporal understanding particularly in referring video segmentation is less studied. In this work, we raise the pertinent question of whether motion is used in referring segmentation and whether video MLLMs designed for this task truly leverage motion cues when segmenting objects based on natural language expressions. We identify critical shortcomings in the current benchmarks, where we show a single frame can often suffice for capturing the motion referring expression without any temporal reasoning. To address this, we introduce a motion-centric probing and evaluation framework that automatically selects key-frames within videos designed to mislead models with apparent motion lacking true spatiotemporal change, to assess whether models rely on genuine motion cues or merely static visual features. Our empirical analysis reveals that existing video MLLMs underutilize motion information in this dense prediction task, it also shows the kind of properties existent in referring expressions that makes it more motion oriented than others. We further establish strong baselines using MLLMs that outperform prior methods, offering new insights into the interplay between spatial and temporal information in dense video-language understanding tasks. Our motion centric evaluation and findings challenge future models to improve dense spatiotemporal grounding and pixel-level understanding within videos.

Primary Area: Deep learning (e.g., architectures, generative models, optimization for deep networks, foundation models, LLMs)

Submission Number: 5674