Histopathobiome – integrating histopathology and microbiome data via multimodal deep learning

Agata Polejowska, Annemarie Boleij, Francesco Ciompi

Keywords: Computational pathology, microbiome, multimodality, fusion, deep learning, inflammatory bowel disease, ulcerative colitis

TL;DR: Multimodal deep learning models can learn cross-modal interpretable tissue-microbe interactions delivering improved results compared to single-modality performance in classifying patients with ulcerative colitis.

Abstract:

We introduce Histopathobiome, a term representing the integration of histopathology and microbiome data to explore tissue-microbe interactions. Using a dataset of colon biopsy whole-slide images paired with microbiota composition samples, we assess the benefits of
combining these modalities to distinguish patients with inflammatory bowel disease (IBD) subtype – ulcerative colitis (UC) from non-IBD controls. Initially, we evaluate the unimodal performance of state-of-the-art algorithms using vectors representing bacterial species abundances or histopathology slide-level embeddings. We compare single-modality models with bimodal networks with various fusion strategies. Our results prove that histopathology and microbiome data are complementary in UC classification. By demonstrating improved
performance over single-modality approaches, we prove that bimodal deep learning models can be used to learn meaningful and interpretable cross-modal tissue-microbe patterns.

Submission Number: 21