InfoMAE: Pairing-Efficient Cross-Modal Alignment with Informational Masked Autoencoders for IoT Signals

Tomoyoshi Kimura, Xinlin Li, Osama Hanna, Yatong Chen, Yizhuo Chen, Denizhan Kara, Tianshi Wang, Jinyang Li, Xiaomin OUYANG, Shengzhong Liu, Mani Srivastava, Suhas Diggavi, Tarek F. Abdelzaher

Track: Systems and infrastructure for Web, mobile, and WoT

Keywords: Internet of Things, Self-Supervised Learning, Multimodal Learning

Abstract:

Standard multimodal self-supervised learning (SSL) algorithms regard cross-modal synchronization as implicit supervisory labels
during pretraining, thus posing high requirements on the scale
and quality of multimodal samples. These constraints significantly
limit the performance of sensing intelligence in IoT applications,
where the heterogeneity and the non-interpretability of time-series
signals result in abundant unimodal data but scarce high-quality
multimodal pairs. This paper proposes InfoMAE, a cross-modal
alignment framework that tackles the challenge of multimodal
pair efficiency under the SSL setting by facilitating efficient cross-
modal alignment of pretrained unimodal representations. InfoMAE
achieves efficient cross-modal alignment with limited data pairs
through a novel information theory-inspired formulation that simultaneously addresses distribution-level and instance-level align-
ment. Extensive experiments on two real-world IoT applications
are performed to evaluate InfoMAE’s pairing efficiency to bridge
pretrained unimodal models into a cohesive joint multimodal model.
InfoMAE enhances downstream multimodal tasks by over 60% with
significantly improved multimodal pairing efficiency. It also improves unimodal task accuracy by an average of 22%

Submission Number: 393