Evidential Remote Physiological Measurement via Uncertainty-aware Fusion of Video and RF

Jieyi Ge, Zhaodong Sun, Wei Peng, Chenhang Ying, Yuwei Chen, Kui Ren, Xiaobai Li

Abstract: Remote physiological measurement enables the capture of vital signals in a non-contact way, which offers significant potential for various applications. Monitoring these signals is achieved through video cameras or radio frequency (RF) sensors, with recent few methods attempting to fuse both sources to leverage complementary patterns for enhanced accuracy. However, these two modalities operate on distinct principles, where video-based methods detect subtle facial color changes from blood volume variations, while RF-based methods capture subtle body vibration due to heartbeats. In practical applications, they may encounter interference at different occasions. Treating these modalities as equally reliable in all situations can lead to suboptimal fusion. To address this issue, we propose an evidential video-RF fusion framework for robust remote physiological signal measurement. We design an uncertainty regression head for each uni-modality, which estimates uncertainty features together with the corresponding physiological signal in each branch. Then an evidential multi-modal fusion module is employed to dynamically fuse the two modalities according to their uncertainty. Extensive experiments carried on public and self-collected datasets show that the proposed method not only achieves superior fusion performance on easy data collected under well-controlled environment, it also generalizes well to unseen data which represents challenging practical conditions that one or both sensors are disturbed.

External IDs:doi:10.1145/3746027.3754594