MedRAX: Medical Reasoning Agent for Chest X-ray

Adibvafa Fallahpour, Jun Ma, Alif Munim, Hongwei Lyu, BO WANG

TL;DR: We introduce MedRAX, a state-of-the-art AI agent for chest X-ray interpretation.

Abstract: Chest X-rays (CXRs) play an integral role in driving critical decisions in disease management and patient care. While recent innovations have led to specialized models for various CXR interpretation tasks, these solutions often operate in isolation, limiting their practical utility in clinical practice. We present MedRAX, the first versatile AI agent that seamlessly integrates state-of-the-art CXR analysis tools and multimodal large language models into a unified framework. MedRAX dynamically leverages these models to address complex medical queries without requiring additional training. To rigorously evaluate its capabilities, we introduce ChestAgentBench, a comprehensive benchmark containing 2,500 complex medical queries across 7 diverse categories. Our experiments demonstrate that MedRAX achieves state-of-the-art performance compared to both open-source and proprietary models, representing a significant step toward the practical deployment of automated CXR interpretation systems. Data and code have been publicly available at https://github.com/bowang-lab/MedRAX

Lay Summary:

Chest X-ray interpretation is a critical but labor-intensive task in medicine. Existing artificial intelligence (AI) tools often function as standalone applications, which restricts their integration into comprehensive clinical workflows. Moreover, current general-purpose AI models, despite their advancements, may not consistently provide the multi-step analytical capabilities or the transparent decision-making processes required in medical diagnostics.

We have developed MedRAX, an AI framework designed to overcome these limitations in chest X-ray analysis. MedRAX operates by coordinating a suite of specialized AI tools, each proficient in specific tasks such as disease detection, identifying and outlining anatomical structures, or answering detailed image-based questions. The system dynamically selects and sequences these tools, integrating their outputs to address complex medical queries without requiring retraining of the core framework when tools are added or modified.

This approach enables MedRAX to offer more accurate, detailed, and interpretable analyses of chest X-rays compared to existing methods, representing a significant advancement towards the practical application of AI in radiology. The system aims to improve diagnostic efficiency, reduce potential for error, and increase the clarity of AI-driven insights, thereby supporting medical professionals and potentially enhancing patient care through more robust AI assistance.

Application-Driven Machine Learning: This submission is on Application-Driven Machine Learning.

Link To Code: https://github.com/bowang-lab/MedRAX

Primary Area: Applications->Health / Medicine

Keywords: healthcare, agent, multimodal, chest X-ray, benchmark

Flagged For Ethics Review: true

Submission Number: 14282