Go to ACL ARR 2025 May homepageReasonMed: A 370K Multi-Agent Generated Dataset for Advancing Medical Reasoning
Abstract: Though reasoning-based large language models (LLMs) have excelled in mathematics and programming, their capabilities in knowledge-intensive medical question answering remain underexplored. To address this, we introduce ReasonMed, the largest medical reasoning dataset, comprising 370k high-quality examples distilled from 1.7 million initial reasoning paths generated by various LLMs.
ReasonMed is constructed through a *multi-agent verification and refinement process*, where we design an *Error Refiner* to enhance the reasoning paths by identifying and correcting error-prone steps flagged by a verifier.
Leveraging ReasonMed, we systematically investigate best practices for training medical reasoning models and find that combining detailed Chain-of-Thought (CoT) reasoning with concise answer summaries yields the most effective fine-tuning strategy.
Based on this strategy, we train ReasonMed-7B, which sets a new benchmark for sub-10B models, outperforming the prior best by 4.17\% and even exceeding LLaMA3.1-70B on PubMedQA by 4.60\%.
Paper Type: Long
Research Area: Resources and Evaluation
Research Area Keywords: corpus creation, automatic creation and evaluation of language resources, language resources
Contribution Types: NLP engineering experiment, Publicly available software and/or pre-trained models, Data resources
Languages Studied: English
Submission Number: 5220
Loading