Knowledge Graphs for Multi-modal Learning: Survey and Perspective

Zhuo Chen, Yichi Zhang, Yin Fang, Yuxia Geng, Lingbing Guo, Jiaoyan Chen, Xiaoze Liu, Jeff Z. Pan, Ningyu Zhang, Huajun Chen, Wen Zhang

Abstract: Integrated with multi-modal learning, knowledge graphs (KGs) as structured knowledge repositories, can enhance AI for processing and understanding complex, real-world data. This paper provides a comprehensive survey of cutting-edge research on KG-aware multi-modal learning. For these core areas, we provide task definitions, evaluation benchmarks, and comprehensive insights into key breakthroughs, offering detailed explanations critical for conducting related research. Furthermore, we also discuss current challenges, highlighting emerging trends and future research directions. The repository for this paper can be found at https://github.com/zjukg/KG-MM-Survey.