A unified multimodal classification framework based on deep metric learning

Liwen Peng, Songlei Jian, Minne Li, Zhigang Kan, Linbo Qiao, Dongsheng Li

Abstract: Highlights•A unified multimodal classification framework that can handle various multimodal classification tasks.•Flexibly process data from multiple modalities, including images, texts, audio, and videos.•Metric-based triplet learning to extract intra-modal relationships in every modality.•Contrastive pairwise learning to capture inter-modal relationships across multiple modalities.