Go to DMLR homepageCycliST: A Video Language Model Benchmark for Reasoning on Cyclical State Transitions
Abstract: We present CycliST, a novel benchmark dataset designed to evaluate Video Language Models (VLM) on their reasoning over cyclical state transitions. CycliST captures fundamental aspects of real-world processes in synthetic, richly structured video sequences featuring periodic visual patterns. Furthermore, CycliST offers a tiered evaluation, providing increasing difficulty levels by varying the number of cyclic objects, scene clutter, and lighting conditions. We conduct extensive experiments with current open-source and proprietary state-of-the-art VLMs and reveal their limitations in generalizing to cyclical dynamics, such as linear and orbital motion, as well as to time-dependent changes in visual attributes like color and scale. Our results demonstrate that present-day VLMs struggle to reliably detect and exploit cyclic patterns, lack a notion of temporal understanding, and are unable to extract quantitative insights from scenes, such as the number of objects in motion, highlighting a significant technical gap that needs to be addressed. More specifically, we find no single model consistently outperforms others: neither size nor architecture correlates strongly with outcomes, and no model performs equally well across all tasks. By providing a targeted challenge and a comprehensive evaluation framework, CycliST paves the way for visual reasoning models that surpass the state-of-the-art in understanding periodic patterns.
Keywords: Video Question Answering, Scene Understanding, Spatio-Temporal Reasoning, Benchmark Dataset
Previous DMLR Submission Url: https://openreview.net/forum?id=95jFeCBepf¬eId=95jFeCBepf
Changes Since Last Submission: We have corrected the font to comply with the official DMLR format.
Code: https://github.com/simon-kohaut/CycliST
Assigned Action Editor: ~Lijie_Hu1
Submission Number: 185
Loading