DynamicEval: Rethinking Evaluation for Dynamic Text-to-Video Synthesis

Nithin C. Babu, Aniruddha Mahapatra, Harsh Rangwani, Rajiv Soundararajan, Kuldeep Kulkarni

Keywords: Text-to-video generation, generative models, evaluation metrics, quality assessment, human annotations, video benchmark, video dataset

TL;DR: DynamicEval is a benchmark for camera-motion video generation, introducing interpretable background and foreground consistency metrics that align better with human preferences than existing deep-feature methods.

Abstract:

Existing text-to-video (T2V) evaluation benchmarks, such as VBench and EvalCrafter, suffer from two main limitations. (i) While the emphasis is on subject-centric prompts or static camera scenes, camera motion which is essential for producing cinematic shots and the behavior of existing metrics under dynamic motion are largely unexplored. (ii) These benchmarks typically aggregate video-level scores into a single model-level score for ranking generative models. Such aggregation, however, overlook video-level evaluation, which is vital to selecting the better video among the candidate videos generated for a given prompt.
To address these gaps, we introduce DynamicEval, a benchmark consisting of systematically curated prompts emphasizing dynamic camera motion, paired with 45k human annotations on video pairs from 3k videos generated by ten T2V models. DynamicEval evaluates two key dimensions of video quality: background scene consistency and foreground object consistency.
For background scene consistency, we obtain the interpretable error maps based on the Vbench motion smoothness metric. Our key observation based on the error maps is that while the Vbench motion smoothness metric shows promising alignment with human judgments, it fails in two cases, namely, occlusions/disocclusions arising from camera and foreground object movements. Building on this, we propose a new background consistency metric that leverages object error maps to correct two major failure cases in a principled manner. Our second innovation is the introduction of a foreground consistency metric that tracks points and their neighbors within each object instance to better assess object fidelity.
Extensive experiments demonstrate that our proposed metrics achieve stronger correlations with human preferences at both the video level and the model level (an improvement of more than $2$\% points), establishing DynamicEval as a more comprehensive benchmark for evaluating T2V models under dynamic camera motion.

Primary Area: datasets and benchmarks

Submission Number: 23607