Sequence-agnostic Continual Multi-modal Clustering

Guoliang Zou, Yangdong Ye, Tongji Chen, Sijia Li, Jinlan Wang, Shizhe Hu

Keywords: Multi-modal Clustering, Continual Learning

TL;DR: We propose a novel sequence-agnostic continual multi-modal clustering method.

Abstract:

Continual multi-modal clustering (CMC) aims to address the challenges posed by the continuous arrival of multi-modal data streams, enabling models to progressively update cluster assignments while avoiding catastrophic forgetting.
CMC closely aligns with the requirements of real-world scenarios and has attracted significant attention from researchers.
However, existing CMC methods face two limitations.
(1) They fail to reliably model the relationship between historical and new information, leading to redundancy in the shared representation and weakened discriminative power of clustering.
(2) They are highly sensitive to modality sequence, as early high-quality modalities are gradually forgotten, making the results dependent on the input order.
To address these limitations, we propose a novel Sequence-agnostic Continual Multi-modal Clustering (SCMC) method that achieves reliable continual learning and is insensitive to the modality arrival sequence. Specifically, SCMC employs a residual fusion network to suppress the update bias introduced by the newly arrived modalities. It then leverages a cross-temporal knowledge collaboration mechanism to bidirectionally filter information between the historical information and the new modalities, thereby maximizing the preservation of task-relevant information and ensuring reliable continual learning.
To eliminate the high sequence sensitivity, we design a sequence-agnostic anti-forgetting strategy, which aligns the current features and cluster distribution with the previous step through cross-temporal consistency transfer, and then prioritizes retaining high-value modality information based on modality importance scores.
Extensive experiments demonstrate that SCMC outperforms existing SOTA methods, exhibiting sequence insensitivity and strong anti-forgetting capabilities. To the best of our knowledge, SCMC is the first approach to explicitly address the sequence sensitivity problem in CMC.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 4298