Crafting Good Views of Medical Images for Contrastive Learning via Expert-level Visual Attention

Sheng Wang, Zihao Zhao, Lichi Zhang, Dinggang Shen, Qian Wang

Submission Type: Full Paper

Keywords: Eye-tracking, Contrastive Learning, Medical Image, Human Visual Attention

Abstract:

Recent advancements in contrastive learning methods have shown significant improvements, which focus on minimizing the distances between different views of the same image.
These methods typically craft two randomly augmented views of the same image as a positive pair, expecting the model to capture the inherent representation of the image.
However, random data augmentation might not fully preserve image semantic information and can lead to a decline in the quality of the augmented views, thereby affecting the effectiveness of contrastive learning. This issue is particularly pronounced in the domain of medical images, where lesion areas can be subtle and are susceptible to distortion or removal.
To address this issue, we leverage insights from radiologists' expertise in diagnosing medical images and propose Gaze-Conditioned Augmentation (GCA) to craft high-quality contrastive views of medical images given the radiologist's visual attention.
Specifically, we track the gaze movements of radiologists and model their visual attention when reading to diagnose X-ray images.
The learned model can predict visual attention of the radiologist when presented with a new X-ray image, and further guide the attention-aware augmentation, ensuring that it pays special attention to preserving disease-related abnormalities.
Our proposed GCA can significantly improve the performance of contrastive learning methods on knee X-ray images, revealing its potential in medical applications.

Submission Number: 2