Interpreting Any Condition to Caption for Controllable Video Generation

Shengqiong Wu, Weicai Ye, Jiahao Wang, Quande Liu, Xintao Wang, Pengfei Wan, Di ZHANG, Kun Gai, Shuicheng YAN, Hao Fei, Tat-Seng Chua

Keywords: Mulit-Modal Understanding, Video Caption, Controllable Video Generation

TL;DR: we present Any2Caption, a novel framework for controllable video generation from any condition.

Abstract: To address the bottleneck of accurately interpreting user intent within the current video generation community, we present Any2Caption, a novel framework for controllable video generation from any condition. The key idea is decoupling various condition interpretation steps from the video synthesis step. By leveraging modern multimodal large language models (MLLMs), Any2Caption interprets diverse inputs—text, images, videos, and specialized cues such as region, motion, and camera poses—into dense, structured captions that offer backbone video generators with better guidance. We also introduce Any2CapIns, a large-scale dataset with 337K instances and 407K annotations for any-condition-to-caption instruction tuning. Comprehensive evaluations demonstrate significant improvements of our system in controllability and video quality across various aspects of existing video generation models.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 10483