3MDBench: Medical Multimodal Multi-agent Dialogue Benchmark

ACL ARR 2025 May Submission3215 Authors

Abstract: Though Large Vision-Language Models (LVLMs) are being actively explored in medicine, their ability to conduct telemedicine consultations combining accurate diagnosis with professional dialogue remains underexplored. In this paper, we present $\textbf{3MDBench}$ ($\textbf{M}\text{edical}$ $\textbf{M}\text{ultimodal}$ $\textbf{M}\text{ulti-agent}$ $\textbf{D}\text{ialogue}$ $\textbf{Bench}\text{mark})$, an open-source framework for simulating and evaluating LVLM-driven telemedical consultations. 3MDBench simulates patient variability through four temperament-based Patient Agents and an Assessor Agent that jointly evaluate diagnostic accuracy and dialogue quality. It includes 3013 cases across 34 diagnoses drawn from real-world telemedicine interactions, combining textual and image-based data. The experimental study compares diagnostic strategies for popular LVLMs, including GPT-4o-mini, LLaVA-3.2-11B-Vision-Instruct, and Qwen2-VL-7B-Instruct. We demonstrate that multimodal dialogue with internal reasoning improves F1 score by 6.5\% over non-dialogue settings, highlighting the importance of context-aware, information-seeking questioning. Moreover, injecting predictions from a diagnostic convolutional network into the LVLM's context boosts F1 by up to 20%. Source code is available at https://anonymous.4open.science/r/3mdbench_acl-0511.

Paper Type: Long

Research Area: NLP Applications

Research Area Keywords: multi-modal dialogue systems, LLM/AI agents, human-centered evaluation, healthcare applications, clinical NLP

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Publicly available software and/or pre-trained models, Data resources, Data analysis

Languages Studied: english

Submission Number: 3215