Histopathobiome – integrating histopathology and microbiome data via multimodal deep learning

Agata Polejowska, Annemarie Boleij, Francesco Ciompi

Keywords: Computational pathology, microbiome, multimodality, fusion, deep learning, Inflammatory Bowel Disease, Ulcerative Colitis

TL;DR: Multimodal deep learning models can learn cross-modal interpretable tissue-microbe interactions delivering improved results compared to single-modality performance in classifying IBD patients.

Abstract: We introduce Histopathobiome, a term representing the integration of histopathology and microbiome data to explore tissue-microbe interactions. Using a dataset of colon biopsy whole-slide images paired with microbiota composition samples, we assess the benefits of combining these modalities. Initially, we evaluate the unimodal performance of state-of-the-art algorithms using vectors representing bacterial species abundances or histopathology slide-level embeddings. We compare single-modality models with bimodal networks that use various fusion strategies. Our results prove that histopathology and microbiome data are complementary. By yielding improved performance over single-modality approaches, bimodal deep learning models can learn cross-modal interpretable tissue-microbe patterns.

Submission Number: 21