VEBench: Towards Comprehensive and Automatic Evaluation for Text-guided Video Editing

bingshuai liu, Ante Wang, Zijun Min, Chenyang Lyu, Longyue Wang, Jinsong Su

Keywords: Video Editing, MLLMs, Meta Evaluation

TL;DR: We introduce VEBench, a large-scale benchmark for evaluating video editing metrics, and VEScore, which leverages MLLMs for automatic evaluation, reducing reliance on labor-intensive human annotations.

Abstract: Video editing task has gained widespread attention in recent years due to their practical applications and rapid advancements, driven by the emergence of diffusion techniques and Multi-modal Large Language Models (MLLMs). However, current automatic evaluation metrics for video editing are mostly unreliable and poorly aligned with human judgments. As a result, researchers heavily rely on human annotation for evaluation, which is not only time-consuming and labor-intensive but also difficult to ensure consistency and objectivity. To address this issue, we introduce **VEBench**, the largest-ever video editing meta-evaluation benchmark to evaluate the reliability of automatic metrics. It includes 152 video clips and 962 text prompts, from which 160 instances are sampled to generate 1,280 edited videos using 8 open-source video editing models, accompanied by human annotations. Especially, the text prompts are first crafted using GPT-4, followed by manual review and careful categorization based on editing types for a systematic evaluation. Our human annotations cover 3 criteria: *Textual Faithfulness*, *Frame Consistency*, and *Video Fidelity*, ensuring the comprehensiveness of evaluation. Since human evaluation is costly, we also propose **VEScore**, employing MLLMs as evaluators to assess edited videos from the criteria above. Experiments show that the best-performing video editing model only reaches an average score of 3.18 (out of a perfect 5), highlighting the challenge of VEBench. Besides, results from more than 10 MLLMs demonstrate the great potential of utilizing VEScore for automatic evaluation. Notably, for Textual Faithfulness, VEScore equipped with LLaVA-OneVision-7B achieves a Pearson Correlation score of 0.48, significantly outperforming previous methods based on CLIP with the highest score of 0.21. The dataset and code will be released upon acceptance.

Primary Area: datasets and benchmarks

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 9221