Diversity-oriented Deep Multi-modal Clustering

Yanzheng Wang, Xin Yang, Yujun Wang, Shizhe Hu, Mingliang Xu

Keywords: Multi-modal clustering, diversity learning, contrastive learning

TL;DR: We propose a diversity-oriented deep multimodal clustering method.

Abstract: Deep multi-modal clustering (DMC) aims to explore the correlated information from different modalities to improve the clustering performance. Most existing DMCs attempt to investigate the consistency or/and complementarity information by fusing all modalities, but this will lead to the following challenges: 1) Information conflicts between modalities emerge. 2) Information-rich modalities may be weakened. To address the above challenges, we propose a diversity-oriented deep multi-modal clustering (DDMC) method, where the core is dominant modality enhancement instead of multi-modal fusion. Specifically, we select the modality with the highest average silhouette coefficient as the dominant modality, then learn the diversity information between the dominant madality and the remaining ones with diversity learning, and finally enhance the dominant modality for clustering. Extensive experiments show the superiority of the proposed method over several compared DMC methods. To our knowledge, this is the first work to perform multi-modal clustering by enhancing the dominant modality instead of fusion.

Primary Area: General machine learning (supervised, unsupervised, online, active, etc.)

Submission Number: 4310