Cross-Modal Knowledge Discovery, Inference, and Challenges

Meng Wang, Ningyu Zhang

Abstract: In recent years, multimodal knowledge has become a popular research topic in many fields, such as knowledge graphs and natural language processing. Multimodal knowledge involves multimodal knowledge graphs, multimodal pre-trained language models, multimodal knowledge inference, etc.; from online shopping to medical care, whether it is theoretical research or engineering application, the knowledge representation, discovery, and inference of multimodal knowledge have become the core technologies of the academic and industrial concern. This tutorial focuses on the state of the art of cross-modal knowledge discovery and inference and presents future research opportunities and challenges.