Video-ToC: Video Tree-of-Cue Reasoning

Qizhong Tan, Zhuotao Tian, Guangming Lu, Jun Yu, Wenjie Pei

Keywords: video understanding, multimodal large language model reasoning

TL;DR: This paper proposes Video-ToC, a video reasoning framework that employs a tree-of-cue reasoning pattern and a reasoning-demand reward mechanism to enhance Video LLMs, achieving superior performance on video understanding and hallucination benchmarks.

Abstract: Existing Video Large Language Models (Video LLMs) struggle with complex video understanding, exhibiting limited reasoning capabilities and potential hallucinations. In particular, these methods tend to perform reasoning solely relying on the pretrained inherent reasoning rationales whilst lacking perception-aware adaptation to the input video content. To address this, we propose \textbf{Video-ToC}, a novel video reasoning framework that enhances video understanding through tree-of-cue reasoning. Specifically, our approach introduces three key innovations: (1) A tree-guided visual cue localization mechanism, which endows the model with enhanced fine-grained perceptual capabilities through structured reasoning patterns; (2) A reasoning-demand reward mechanism, which dynamically adjusts the reward value for reinforcement learning (RL) based on the estimation of reasoning demands, enabling on-demand incentives for more effective reasoning strategies; and (3) An automated annotation pipeline that constructs the Video-ToC-SFT-1k and Video-ToC-RL-2k datasets for supervised fine-tuning (SFT) and RL training, respectively. Extensive evaluations on six video understanding benchmarks and a video hallucination benchmark demonstrate the superiority of Video-ToC over baselines and recent methods. All code, models, and datasets will be released.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 7483