Design of a Multimodal Short Video Classification Model

Xinyan Cao, He Yan, Changjin Li, Yongjian Zhao, Jinming Che, Wei Ren, Jinlong Lin, Jian Cao

Abstract: With the development of mobile Internet, a large amount of short video data is generated on the Internet. The urgent problem of short video classification is how to better fuse the information of different multimodal information. This paper proposes a short video multimodal fusion (SV-MF) scheme based on deep learning combined with pre-trained models to complete the classification task of short video. The main innovations of the SV-MF scheme are as follows: (1) We find that text modalities contain higher-order information and tend to perform better than audio and visual modalities, and with the use of pre-trained language models, text modalities have been further improved in multimodal video classification. (2) Due to the strong semantic representation ability of text. The SVMF scheme proposes a local fusion method based on Transformer for low-order visual and audio modal information to alleviate the information deviation caused by multi-mode fusion. (3) The SV-MF scheme proposes a post processing strategy based on keywords to further improve the classification accuracy of the model. Experimental results based on a multimodal short video classification dataset derived from social networks show that the performance of the SV-MF scheme is better than the previous video fusion scheme.