ExGAT: Build Explicit Dependencies for Incomplete Multi-Modal Learning via Graph Attention Network

Binyu Zhao, Wei Zhang, Zhaonian Zou

Abstract: Recent research in multi-modal applications has highlighted the challenge of handling missing modalities. Most existing methods either overlook the dependencies between modalities or rely on deep learning techniques to learn implicit dependencies, often tailored to specific tasks. In this paper, we propose ExGAT, a novel graph attention network designed to explicitly model dependencies for incomplete multi-modal learning. ExGAT introduces Modal Dependency Learning (MDL) to construct a graph that captures inter-modality dependencies and aids in reconstructing missing modality features, and Modal Importance Learning (MIL) to create a graph with a pseudo missing modality, enabling the exploration of the importance of each modality by reconstructing the pseudo modality. Additionally, we incorporate nested sampling and an auxiliary completion task to further enhance the reconstruction process. Extensive evaluations on multiple tasks demonstrate the effectiveness of ExGAT, highlighting its potential to address incomplete multi-modal learning challenges across diverse domains. Code is available at https://github.com/byzhaoAI/ExGAT.

External IDs:dblp:conf/icmcs/ZhaoZZ25