Empowering multimodal analysis with visualization: A survey

Jiachen Wang, Zikun Deng, Dazhen Deng, Xingbo Wang, Rui Sheng, Yi Cai, Huamin Qu

Abstract: Multimodal data, which encompasses text, audio, image, and other modalities, is a popular research target in the field of visualization research. Existing visualization techniques for multimodal data are scattered and categorized by application domains, such as multimodal model analysis or online education. It lacks a comprehensive review from the perspective of data that summarizes the methodologies, research gaps, and future trends for researchers and practitioners. In this study, we delve into existing visualization research, identifying their data modalities, applications, strengths, and limitations. Furthermore, we shed light on the potential challenges and opportunities for further research in this domain to advance intelligent visualizations for multimodal data.