Molecularly informed analysis of histopathology images using natural language

ICML 2025 Workshop FM4LS Submission40 Authors

Keywords: Computational Pathology, Histopathology Image Analysis, Multimodal Learning, Interpretable Representations, Machine Learning in Medicine

TL;DR: We developed SpotWhisperer, a multimodal AI that integrates histopathological H&E images with inferred molecular profiles and natural-language analysis, complementing existing pathology VLMs with enhanced detection of fine-grained structures

Abstract:

Histopathology refers to the microscopic examination of diseased tissues and routinely guides treatment decisions for cancer and other diseases. Currently, this analysis focuses on morphological features but rarely considers gene expression information, which can add an important molecular dimension.
Here, we introduce SpotWhisperer, an AI method that links histopathological images to spatial gene expression profiles and their text annotations, enabling molecularly grounded histopathology analysis through natural language.
Our method outperforms pathology vision-language models on a newly curated benchmark dataset, dedicated to spatially resolved H&E annotation.
Integrated into a web interface, SpotWhisperer enables interactive exploration of cell types and disease mechanisms using free-text queries with access to inferred spatial gene expression profiles.
In summary, SpotWhisperer analyzes cost-effective pathology images with spatial gene expression and natural-language AI, demonstrating a path for routine integration of microscopic molecular information into histopathology.

Submission Number: 40