ADAPT: Multimodal Learning for Detecting Physiological Changes under Missing Modalities

Julie Mordacq, Leo Milecki, Maria Vakalopoulou, Steve Oudot, Vicky Kalogeiton

Keywords: Multimodality, Missing Modalities, Contrastive Learning, Biomedical signals

Abstract:

Multimodality has recently gained attention in the medical domain, where imaging or video
modalities may be integrated with biomedical signals or health records. Yet, two challenges
remain: balancing the contributions of modalities, especially in cases with a limited amount
of data available, and tackling missing modalities. To address both issues, in this paper, we
introduce the AnchoreD multimodAl Physiological Transformer (ADAPT), a multimodal,
scalable framework with two key components: (i) aligning all modalities in the space of the
strongest, richest modality (called anchor ) for learning a joint embedding space, and (ii)
a masked multimodal transformer, leveraging both inter- and intra-modality correlations
while handling missing modalities. We focus on detecting alterations of physiological re-
sponses in two real-life scenarios, namely stress in individuals induced by specific triggers
and fighter pilots’ loss of consciousness induced by g-forces. We validate the generalizability
of ADAPT through extensive experiments on two datasets for these tasks, where we set the
new state of the art while demonstrating its robustness across various modality scenarios
and its high potential for real-life applications.

Submission Number: 249