Strong Reasoning Isn’t Enough: Evaluating Evidence Elicitation in Interactive Diagnosis

ACL ARR 2026 January Submission10837 Authors

Keywords: Interactive Agent; Medical Agents;

Abstract:

Interactive medical consultation requires an agent to proactively elicit missing clinical evidence under uncertainty.
Yet existing evaluations largely remain static or outcome-centric, neglecting the evidence-gathering process.
In this work, we propose an interactive evaluation framework that explicitly models the consultation process using a simulated patient and a measurement module grounded in atomic evidences.
Based on this representation, we introduce Information Coverage Rate (ICR) to quantify how completely an agent uncovers necessary evidence during interaction.
To support systematic study, we build EviMed, an evidence-based benchmark spanning diverse conditions from common complaints to rare diseases, and evaluate 10 models with varying reasoning abilities.
We find that strong diagnostic reasoning does not guarantee effective information collection, and this insufficiency acts as a primary bottleneck limiting performance in interactive settings.
To address this, we propose REFINE, a strategy that leverages diagnostic verification to guide the agent in proactively resolving uncertainties.
Extensive experiments demonstrate that REFINE consistently outperforms baseline methods across diverse models and datasets, achieving superior information coverage and diagnostic accuracy.

Paper Type: Long

Research Area: Clinical and Biomedical Applications

Research Area Keywords: clinical decision support;clinical dialogue systems

Contribution Types: Model analysis & interpretability

Languages Studied: English

Submission Number: 10837