VLV-Bench: A Comprehensive benchmark for very long-form videos understanding
Abstract: Understanding long videos, ranging from tens of minutes to several hours, presents unique challenges in video comprehension. Despite the increasing importance of long-form video content, existing benchmarks primarily focus on shorter clips. To address this gap, we introduce a comprehensive benchmark for Very Long Videos understanding (VLV-Bench), which presents 1) The longest video duration, averaging 76.34 minutes; 2) The largest number of question-answer pairs, 108.2K; 3) Diversity in questions that examine nine different skills and include both multiple-choice questions and open-ended questions; 4) Human-centric, as the video sources come from movies and daily TV shows, with specific human-level question designs such as Movie Spoiler Questions that require critical thinking and comprehensive understanding. Using VLV-Bench, we comprehensively evaluate existing Large Multi-Modality Models (LMMs) on each skill, including the commercial model Gemini 1.5 Flash and the open-source models.
The evaluation shows significant challenges in our benchmark.
Our results show that the best AI models such Gemini struggles to perform well with 42.72 % average accuracy and 2.71 out of 5 average score.
We hope this benchmark will stimulate the LMMs community towards long video and human-level understanding. Our benchmark can be accessed at (https://vlv-bench.github.io/VLV-website/) and will be made publicly available.
Paper Type: Long
Research Area: Multimodality and Language Grounding to Vision, Robotics and Beyond
Research Area Keywords: video understanding, benchmark, long video benchmark, long video understanding
Contribution Types: Data resources, Data analysis
Languages Studied: English
Submission Number: 5822
Loading