GazeDiff: A radiologist visual attention guided diffusion model for zero-shot disease classification

Moinak Bhattacharya, Prateek Prasanna

Keywords: Eye-gaze, diffusion, chest x-rays, disease classification, zero-shot

Abstract: We present GazeDiff, a novel architecture that leverages radiologists' eye gaze patterns as controls to text-to-image diffusion models for zero-shot classification. Eye-gaze patterns provide important cues during the visual exploration process; existing diffusion-based models do not harness the valuable insights derived from these patterns during image interpretation. GazeDiff utilizes a novel expert visual attention-conditioned diffusion model to generate robust medical images. This model offers more than just image generation capabilities; the density estimates derived from the gaze-guided diffusion model can effectively improve zero-shot classification performance. We show the zero-shot classification efficacy of GazeDiff on four publicly available datasets for two common pulmonary disease types, namely pneumonia, and tuberculosis.

Submission Number: 73