All You Need Is A Reference: Cross-modality Referring Segmentation for Abdominal MRI

Sumin Kim, Jun Ma, BO WANG

Keywords: promptable segmentation model, cross-modality, referring segmentation

Abstract:

Multi-modality MRI scans can provide comprehensive diagnoses of abdominal disease but this also introduces new segmentation burdens to derive quantitative imaging biomarkers. In this work, we propose a referring segmentation task where users only need to draw simple scribbles on one modality, called reference modality, to guide the segmentation of both the unseen target modalities and the reference modality. To benchmark the multi-modality segmentation task, we provide a new dataset with 3,277 organs from 534 MRI scans, covering five commonly used MRI modalities. Furthermore, we present a referring segmentation model, CrossMR, to simultaneously segment multiple modalities based on scribbles on reference modality. Experiments demonstrate that our method can achieve comparable performance to the state of the art on one in-distribution reference modality and significantly better generalization ability on four out-of-distribution modalities. This opens a door for efficiently segmenting targets across multiple modalities. The new dataset, code, and trained model weights will be publicly available at https://ref-seg-mr.github.io/.

Primary Area: applications to computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 12337