AlignMMBench: Evaluating Chinese Multimodal Alignment in Large Vision-Language Models

ACL ARR 2025 February Submission3054 Authors

Abstract: Evaluating the alignment capabilities of large Vision-Language Models (VLMs) is essential for determining their effectiveness as helpful assistants. However, existing benchmarks primarily focus on basic abilities using nonverbal methods, such as yes-no and multiple-choice questions. In this paper, we address this gap by introducing AlignMMBench, which provides more nuanced evaluations of alignment capabilities and is the first benchmark specifically designed for Chinese visual contexts. This benchmark is meticulously curated from real-world scenarios and internet sources, encompassing thirteen specific tasks across three categories, and includes both single-turn and multi-turn dialogue scenarios. Incorporating a prompt rewrite strategy, AlignMMBench encompasses 1,054 images and 4,978 question-answer pairs. To facilitate the evaluation pipeline, we propose CritiqueVLM, a rule-calibrated evaluator that exceeds GPT-4's evaluation ability. Additionally, we propose the “alignment score”, a quantitative metric designed to assess the robustness and stability of models across diverse prompts. Finally, we report the performance of representative VLMs on AlignMMBench, offering insights into the capabilities and limitations of different VLM architectures. All evaluation codes and data are available on github: https://anonymous.4open.science/r/AlignMMBench-B689.

Paper Type: Long

Research Area: Resources and Evaluation

Research Area Keywords: benchmarking, automatic evaluation of datasets, evaluation

Contribution Types: Data resources

Languages Studied: Chinese

Submission Number: 3054