MMCR: Advancing Visual Language Model in Multimodal Multi-Turn Contextual Reasoning

Dawei Yan, Yang Li, Qing-Guo Chen, Weihua Luo, PENG WANG, Wei Dong, Qingsen Yan, Haokui Zhang, Chunhua Shen

Keywords: Multimodal Datasets, Instruction Tuning, Contextual Reasoning, Visual Language Model

Abstract: Compared to single-turn dialogue, multi-turn dialogue with contextual coherence better aligns with the needs of real-world human-AI interactions. Additionally, as training data, it provides richer contextual reasoning information, thereby guiding the model to achieve better performance. However, existing vision-language models (VLMs) primarily rely on single-turn dialogue training and evaluation benchmarks. In this paper, following the characteristics of human dialogue, such as focused topics and concise, clear content, we present MMCR (Multimodal Multi-turn Contextual Reasoning), a novel dataset comprising: (1) MMCR-310k, the largest multi-turn instruction tuning dataset with 310K contextual dialogues, each covering 1-4 images and 4 or 8 dialogue turns; and (2) MMCR-Bench, a diagnostic benchmark featuring dialogues, spanning 8 domains (Humanities, Natural, Science, Education, etc.) and 40 sub-topics. Extensive evaluations demonstrate that models fine-tuned with MMCR-310k achieve 5.2% higher contextual accuracy on MMCR-Bench, while showing consistent improvements on existing benchmarks (+1.1% on AI2D, +1.2% on MMMU and MMVet). MMCR and prompt engineering will be released publicly.

Primary Area: datasets and benchmarks

Submission Number: 6591