Fine-tuning Vision Foundation Models for Multi-Modal Prostate MR Sequence Classification

Stefan Denner, Balint Kovacs, David Zimmerer, Deepa Krishnaswamy, Dimitrios Bounias, Raphael Stock, Markus Ralf Bujotzek, Fergus Imrie, Andriy Fedorov, Klaus Maier-Hein

Keywords: MRI, Prostate Cancer, Sequence Classification, Vision Foundation Models

TL;DR: Fine-tuned vision foundation models integrating image and metadata achieve state-of-the-art sequence classification in prostate MRI.

Abstract: Assigning MRI sequence types is essential yet remains a tedious, manual step in prostate imaging workflows. Current automated approaches relying solely on images or DICOM metadata often struggle with protocol variability and metadata inaccuracies, limiting their generalizability. We propose fine-tuning vision foundation models within different fusion strategies integrating image and metadata. We achieve state-of-the-art F1-score of 1.00 and 0.98 on internal and external test sets, respectively, demonstrating robust generalization.

Submission Number: 80