GazeSAM: Interactive Image Segmentation with Eye Gaze and Segment Anything Model

Bin Wang, Armstrong Aboah, Zheyuan Zhang, Hongyi Pan, Ulas Bagci

Submission Type: Full Paper

Keywords: Eye Gaze, Segment Anything Model, Interactive Image Segmentation, Eye Tracking

TL;DR: We propose GazeSAM, a novel interactive image segmentation system that utilizes eye gaze as the interactive prompt instead of the mouse-based interaction. It enhances the annotation workflow efficiency by nearly 50%.

Abstract: Interactive image segmentation aims to assist users in efficiently generating high-quality data annotations through user-friendly interactions such as clicking, scribbling, and bounding boxes. However, mouse-based interaction methods can induce user fatigue during large-scale dataset annotation and are not entirely suitable for some domains, such as radiology. This study introduces eye gaze as a novel interactive prompt for image segmentation, different than previous model-based applications. Specifically, leveraging the real-time interactive prompting feature of the recently proposed Segment Anything Model (SAM), we present the GazeSAM system to enable users to collect target segmentation masks by simply looking at the region of interest. GazeSAM tracks users' eye gaze and utilizes it as the input prompt for SAM, generating target segmentation masks in real time. To our best knowledge, GazeSAM is the first work to combine eye gaze and SAM for interactive image segmentation. Experimental results demonstrate that GazeSAM can improve nearly 50\% efficiency in 2D natural image and 3D medical image segmentation tasks. The code is available in https://github.com/ukaukaaaa/GazeSAM.

Submission Number: 1