R2MED: A Benchmark for Reasoning-Driven Medical Retrieval

ACL ARR 2026 May Submission14436 Authors

Keywords: Medical Information Retrieval, Reasoning, Retrieval benchmark, LLMs

Abstract: Current medical retrieval benchmarks primarily emphasize lexical or shallow semantic similarity, overlooking the reasoning-intensive demands that are central to clinical decision-making. In practice, physicians often retrieve authoritative medical evidence to support diagnostic hypotheses. Such evidence typically aligns with an inferred diagnosis rather than the surface form of a patient's symptoms, leading to low lexical or semantic overlap between queries and relevant documents. To address this gap, we introduce R2MED, the first benchmark explicitly designed for reasoning-driven medical retrieval. It comprises 876 queries spanning three tasks: Q\&A reference retrieval, clinical evidence retrieval, and clinical case retrieval. These tasks are drawn from five representative medical scenarios and twelve body systems, capturing the complexity and diversity of real-world medical information needs. We evaluate 15 widely-used retrieval systems on R2MED and find that even the best model achieves only 31.4 nDCG@10, demonstrating the benchmark’s difficulty. Although reasoning-based enhancements improve performance, a large gap remains. These findings underscore a substantial gap between current retrieval techniques and the reasoning demands of real clinical tasks. We release R2MED as a challenging benchmark to foster the development of next-generation medical retrieval systems with enhanced reasoning capabilities.

Paper Type: Long

Research Area: Resources and Evaluation

Research Area Keywords: Clinical and Biomedical Applications, Information Retrieval and Text Mining

Contribution Types: Data resources

Languages Studied: English

EMNLP 2026 AI Reviewing Experiment: yes

Submission Number: 14436