PRS-MED: Position Reasoning Segmentation in Medical Imaging

Quoc-Huy Trinh, Minh-Van Nguyen, Jun Zeng, Gorkem Durak, Ulas Bagci, Debesh Jha

Keywords: Multimodal-LLM, Position Reasoning, Medical Image Segmentation

Abstract:

Recent advances in prompt-based medical image segmentation have enabled clinicians to
identify tumors using simple input like bounding boxes or text prompts. However, exist-
ing methods face challenges when doctors need to interact through natural language or
when position reasoning is required, which involves understanding the spatial relationships
between anatomical structures and pathologies. We present PRS-Med, a framework that
integrates vision-language models with segmentation capabilities to generate both accu-
rate segmentation masks and corresponding spatial reasoning outputs. Additionally, we
introduce the Medical Position Reasoning Segmentation (MedPos) dataset, which provides
diverse, spatially-grounded question-answer pairs to address the lack of position reasoning
data in medical imaging. PRS-Med demonstrates superior performance across six imag-
ing modalities (CT, MRI, X-ray, ultrasound, endoscopy, skin), significantly outperforming
state-of-the-art methods in both segmentation accuracy and position reasoning. Our ap-
proach enables intuitive doctor-system interaction through natural language, facilitating
more efficient diagnoses. Our dataset pipeline, model, and codebase will be released to
foster further research in spatially-aware multimodal reasoning for medical applications.
(github available after blind review process).

Primary Subject Area: Segmentation

Secondary Subject Area: Generative Models

Registration Requirement: Yes

Visa & Travel: Yes

Read CFP & Author Instructions: Yes

Originality Policy: Yes

Single-blind & Not Under Review Elsewhere: Yes

LLM Policy: Yes

Submission Number: 93