Enhancing skin lesion classification using segmentation-based global-local feature fusion

Runxuan Yu, Xing Yao, Yuankai Huo, Catie Chang

Abstract: The classification of skin lesions using dermatoscopic images is vital for early diagnosis and management. However, accurate skin lesion classification remains a challenging task. It has been proposed that image segmentation techniques may help to mitigate these challenges by extracting regions of interest (ROIs) and removing the background, generating segmented images as the input for skin-lesion classification. However, focusing solely on ROIs may ignore valuable global context. Here, we propose and evaluate a global-to-local feature fusion (GLFF) framework that combines global and local features from original and segmented images, respectively. The proposed segmentation-based GLFF framework (segGLFF) extracts features from the original and segmented images using two separate networks, which are then fused for multi-class classification. Evaluation on the HAM10000 dataset indicates that segGLFF outperforms baselines that use only the segmented image, or only the full original image, for classification. A segGLFF variant using human-labeled ground truth masks (GT-segGLFF) achieves the highest accuracy at 83.02%. We also explore U-Net and Segment Anything Model (SAM) methods for generating segmentation masks within the segGLFF framework. The UNet-segGLFF achieves an accuracy of 82.67%, while the SAM-segGLFF, despite having slightly lower accuracy (81.72%), offers significant operational efficiency by generating masks from rough bounding boxes without extensive manual annotation. These findings underscore the importance of integrating global and local features for improved skin lesion classification and highlight the potential for further optimizing segmentation mask quality in future work.