Eyes on the Image: Gaze Supervised Multimodal Learning for Chest X-ray Diagnosis and Report Generation

ICLR 2026 Conference Submission19751 Authors

Keywords: Multimodal Learning, Medical Imaging, Gaze Supervision, Large Language Models, Contrastive Learning

TL;DR: Two-stage multimodal chest X-ray model; gaze-guided contrastive learning aligns attention with radiologist fixations, then a keyword-region-LLM chain crafts grounded reports, improving saliency and interpretability.

Abstract: Medical vision-language models still struggle to match radiologists’ attention and to verbalize findings with explicit spatial grounding. We address this gap with a two-stage multimodal framework for chest X-ray interpretation built on the MIMIC-Eye dataset. In the first stage introduces a gaze-token classifier that fuses image patches, bounding-box masks, transcription embeddings, and radiologist fixations. A curriculum-scheduled, trust-calibrated composite loss supervises the gaze token, boosting both accuracy and spatial alignment. Adding fixation supervision raises AUC 4.4% and F1 13.3%, and Pearson correlation rises to 0.306, confirming clinically relevant focus. In stage 2, classifier predictions are translated into region-specific diagnostic sentences. Confidence-weighted keywords are extracted, mapped to 17 thoracic regions through an expert dictionary, and expanded with a prompted large language model, boosting clinical-term BERTScore and ROUGE scores over keyword baselines. All components are toggle-able for ablation, and the full pipeline is reproducible, offering a new benchmark for interpretable, gaze-aware chest-X-ray analysis. Integrating eye-tracking signals demonstrably enhances both diagnostic accuracy and the transparency of generated reports.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 19751