Exploring Context-Switching in Medical Image Retrieval Using Segmentation Models

Sai Susmitha Arvapalli, Vinay P. Namboodiri

Abstract: Medical Image Retrieval (MIR) is essential for clinical workflows, enabling accurate diagnosis and advancing medical research. In this work, we aim to better understand the use of context for medical image retrieval. To this end, we propose a novel context-switching-based medical image retrieval framework. Our approach uses the MedSAM2 foundation model to extract segmentations and generate three distinct input versions of each image: the original image, the region-of-interest (ROI) image, and a bounding box around the region of interest that captures additional contextual information. We generate the corresponding embeddings using a contrastive loss-based metric learning approach. A selective backpropagation mechanism enables the model to dynamically identify and utilize the most informative feature embedding for retrieval. Our results suggest that the general context is often more beneficial than a specific context for accurate image retrieval. Our proposed method is evaluated on the ISIC17 and COVID-QU-Ex chest X-ray datasets and demonstrates superior performance compared to strongly related baselines.

External IDs:doi:10.1007/978-3-031-98694-9_8