Evian: Towards Explainable Visual Instruction-tuning Data Auditing

Zimu Jia, Mingjie Xu, Andrew Estornell, Jiaheng Wei

Keywords: Multimodal Models, Data Quality, Instruction Tuning, Data Auditing, Automated Evaluation

TL;DR: We propose EVIAN (Explainable Visual Instruction-tuning Data AuditiNg), an effective pipeline for evaluating the quality of visual instruction tuning data.

Abstract:

The efficacy of Large Vision-Language Models (LVLMs) is critically dependent on the quality of their training data, requiring a precise balance between visual fidelity and instruction-following capability. Existing datasets, however, are plagued by inconsistent quality, and current data filtering methods rely on coarse-grained scores that lack the granularity to identify nuanced semantic flaws like logical fallacies or factual errors. This creates a fundamental bottleneck in developing more reliable models.
To address this, we make three core contributions. First, we construct a large-scale, 300K-sample benchmark by systematically injecting diverse, subtle defects to provide a challenging testbed for data auditing. Second, we introduce a novel "Decomposition-then-Evaluation" paradigm that breaks model responses into constituent cognitive components: visual description, subjective inference, and factual claim, enabling targeted analysis. Third, we instantiate this paradigm via EVIAN (Explainable Visual Instruction-tuning Data AuditiNg), an automated framework that evaluates these components along the orthogonal axes of Image-Text Consistency, Logical Coherence, and Factual Accuracy. Our empirical findings challenge the prevailing scale-centric paradigm: a model fine-tuned on a compact, high-quality subset curated by EVIAN consistently surpassed models trained on orders-of-magnitude larger datasets. We also reveal that dividing complex auditing into verifiable subtasks enables robust curation, and that Logical Coherence is the
most critical factor in data quality evaluation.

Primary Area: alignment, fairness, safety, privacy, and societal considerations

Submission Number: 10385