VidText: Towards Comprehensive Evaluation for Video Text Understanding

ICLR 2026 Conference Submission13661 Authors

Keywords: Video text understanding, Benchmarks

Abstract: Visual texts embedded in videos carry rich semantic information, which is crucial for both holistic video understanding and fine-grained reasoning about local human actions. However, existing video understanding benchmarks largely overlook textual information, while OCR-specific benchmarks are constrained to static images, limiting their ability to capture the interaction between text and dynamic visual contexts. To address this gap, we propose VidText, a new benchmark designed for comprehensive and in-depth evaluation of video text understanding. VidText offers the following key features: 1) It covers a wide range of real-world scenarios and supports multilingual content, encompassing diverse settings where video text naturally appears. 2) It introduces a hierarchical evaluation framework with video-level, clip-level, and instance-level tasks, enabling assessment of both global summarization and local retrieval capabilities. 3) The benchmark also introduces a set of paired tasks for perception and reasoning, ranging from visual text perception to cross-modal reasoning between textual and visual information. Extensive experiments on 18 state-of-the-art Large Multimodal Models (LMMs) reveal that current models struggle across most tasks, with significant room for improvement. Further analysis validates the effectiveness of VidText—spanning joint video-text and multimodal reasoning, multi-granularity task structure, and temporal modeling. It also reveals substantial effects from model-intrinsic factors (input resolution, OCR capability) and external factors (auxiliary context and video-text-centric Chain-of-Thought strategies). We hope VidText will fill the current gap in video understanding benchmarks and serve as a foundation for future research on multimodal reasoning with video text in dynamic environments.

Primary Area: datasets and benchmarks

Submission Number: 13661