VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling

Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang, Yu Qiao, Yali Wang, Limin Wang

Keywords: long video understanding, video language model

TL;DR: Comprehensive Solution for Performance-Leading and Highly Efficient Long-Video Understanding Models

Abstract:

Long-context video modeling is critical for multimodal large language models
(MLLMs), enabling them to process movies, online video streams, and so on.
Despite its advances, handling long videos remains challenging due to the difficulty
in efficiently understanding the extremely long video context. This paper aims to
address this issue from aspects of the model architecture, training data, training
strategy, and evaluation benchmark. First, we propose a novel Hierarchical video
token Compression (HiCo) method, which leverages visual redundancy in long
videos to compress long video context from Clip-level to Video-level, reducing the
computation significantly while preserving essential details, achieving an extreme
compression ratio of approximately 1/50 with almost no performance loss. Second,
we introduce a multi-stage short-to-long learning scheme, a large-scale dataset of
real-world long videos named LongVid, and a challenging “Multi-Hop Needle-In-
A-Video-Haystack” benchmark. Finally, we build a powerful video MLLM named
VideoChat-Flash, which shows a leading performance on both mainstream long
and short video benchmarks at the 2B and 7B model scales. It first gets 99.1%
accuracy over 10,000 frames in NIAH among open-source models.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 5388