PRS-MED: POSITION REASONING SEGMENTATION IN MEDICAL IMAGING

Quoc-Huy Trinh, Minh-Van Nguyen, Jun Zeng, Gorkem Durak, Halil Ertugrul Aktas, Ulas Bagci, Debesh Jha

Keywords: reasoning segmentation, position reasoning, multimodal-llm, medical image segmentation

TL;DR: Focus on the dataset creation and the method to solve the challenge of reasoning segmentation on medical imaging

Abstract: Recent advances in prompt-based medical image segmentation have enabled clinicians to identify tumors using simple input like bounding boxes or text prompts. However, existing methods face challenges when doctors need to interact through natural language or when position reasoning is required, which involves understanding the spatial relationships between anatomical structures and pathologies. We present PRS-Med, a framework that integrates vision-language models with segmentation capabilities to generate both accurate segmentation masks and corresponding spatial reasoning outputs. Additionally, we introduce the Medical Position Reasoning Segmentation (MedPos) dataset, which provides diverse, spatially-grounded question-answer pairs to address the lack of position reasoning data in medical imaging. PRS-Med demonstrates superior performance across six imaging modalities (CT, MRI, X-ray, ultrasound, endoscopy, skin), significantly outperforming state-of-the-art methods in both segmentation accuracy and position reasoning. Our approach enables intuitive doctor-system interaction through natural language, facilitating more efficient diagnoses. Our dataset pipeline, model, and codebase will be released to foster further research in spatially-aware multimodal reasoning for medical applications.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 4899