SMSMO: Learning to generate multimodal summary for scientific papers

Xinyi Zhong, Zusheng Tan, Shen Gao, Jing Li, Jiaxing Shen, Jing-Yu Ji, Jeff K. T. Tang, Billy Chiu

Abstract: Highlights•We study Scientific Multimodal Summarization with Multimodal Output (SMSMO).•We propose a cross-modality model (with multimodal objectives) to perform SMSMO.•We construct two datasets to assess the performance of SMSMO models.