RETQuerySeg: Very-low-parameter adaptation of retinal foundation models for segmentation with query vectors

Justin Engelmann, Lucas Gago

Keywords: Medical Segmentation, Foundation Models, Vector Embeddings

Abstract: Timely segmentation of retinal structures and lesions is critical for screening diseases such as diabetic retinopathy (DR) and retinopathy of prematurity (ROP), yet conventional models remain data- and compute-intensive. We in-troduce RETQuerySeg, a simple approach for adapting retinal foundation models for image segmentations by learning a single query vector. Concrete-ly, we take the final vector embeddings for each patch, take the dot product with our learnable query and use the result as our predicted segmentation. We apply our approach to two openly available datasets, IDRiD and HVDROPDB, on three segmentation tasks using RETFound-Green as the foundation model. RETQuerySeg achieved strong performance for optic disc segmentation (AUC: 0.9995, Dice: 0.9072) and reasonable performance for ROP ridge segmentation (AUC: 0.9847, Dice: 0.5699), demonstrating gener-alizability across adult and neonatal retinal images. Performance was limited for small diabetic lesions (AUC: 0.9159, Dice: 0.1173), reflecting the coarse spatial resolution constraint of the patch-based approach. While not achiev-ing pixel-perfect segmentation, RETQuerySeg offers exceptional parameter efficiency, modularity, and computational advantages. Multiple segmenta-tion tasks can be performed simultaneously with minimal additional com-pute cost, and segmentations can be obtained virtually for free when compu-ting image-level embeddings for classification tasks, making it valuable for resource-constrained settings and explainable AI applications in retinal im-age analysis.

Submission Number: 14