Inflated Excellence or True Performance? Rethinking Medical Diagnostic Benchmarks with Dynamic Evaluation

ACL ARR 2026 January Submission2673 Authors

Keywords: medical diagnostic, benchmark, dynamic evalution, large language model

Abstract:

Medical diagnostics is a high-stakes and complex domain that is critical to patient care.
However, current evaluations of large language models (LLMs) are fundamentally misaligned with real-world clinical practice.
Most rely on benchmarks derived from public exams, raising contamination bias that can inflate performance, and they overlook the confounded nature of real consultations beyond textbook cases.
Recent dynamic evaluations offer a promising alternative, but often remain insufficient for realistic diagnostic benchmarking, with limited coverage of clinically grounded confounders and trustworthiness beyond accuracy.
To address these gaps, we propose DyReMe, a dynamic benchmark for medical diagnostics that better reflects real clinical practice. Unlike static exam-style questions, DyReMe generates fresh, consultation-like cases that introduce distractors such as differential diagnoses and common misdiagnosis factors. It also varies expression styles to mimic diverse real-world query habits. Beyond accuracy, DyReMe evaluates LLMs on three additional clinically relevant dimensions: veracity, helpfulness, and consistency. Our experiments demonstrate that this dynamic approach yields more challenging and realistic assessments, revealing significant misalignments between the performance of state-of-the-art LLMs and real clinical practice. These findings highlight the urgent need for evaluation frameworks that better reflect the demands of trustworthy medical diagnostics.

Paper Type: Long

Research Area: Resources and Evaluation

Research Area Keywords: benchmarking,evaluation methodologies,automatic evaluation of datasets,evaluation

Contribution Types: Publicly available software and/or pre-trained models, Data resources, Data analysis

Languages Studied: Chinese

Submission Number: 2673