Exploring Segment Anything Foundation Models for Out of Domain Crevasse Drone Image Segmentation

Steven Wallace, Aiden Durrant, William David Harcourt, Richard Hann, Georgios Leontidis

Keywords: Deep Learning, Foundation Models, Image Segmentation, Climate Science, Remote Sensing

Abstract: In this paper, we explore the application of Segment Anything (SAM) foundation models for segmenting crevasses in Uncrewed Aerial Vehicle (UAV) images of glaciers. We evaluate the performance of the SAM and SAM 2 models on ten high-resolution UAV images from Svalbard, Norway. Each SAM model has been evaluated in inference mode without additional fine-tuning. Using both automated and manual prompting methods, we compare the segmentation quantitatively using Dice Score Coefficient (DSC) and Intersection over Union (IoU) metrics. Results show that the SAM 2 Hiera-L model outperforms other variants, achieving average DSC and IoU scores of 0.43 and 0.28 respectively with automated prompting. However, the overall off-the-shelf performance suggests that further improvements are still required to enable glaciologists to examine crevasse patterns and associated physical processes (e.g. iceberg calving), indicating the need for further fine-tuning to address domain shift challenges. Our results highlight the potential of segmentation foundation models for specialised remote sensing applications while also identifying limitations in applying them to high-resolution UAV images, as well as ways to enhance further model performance on out-of-domain glacier imagery, such as few-shot and weakly supervised learning techniques.

Submission Number: 25