Dynamic Alignment for Multimodal Continual Learning

Jiayao Tan, Fan Lyu, Tianle Liu, Fuyuan Hu, Wei Feng

Keywords: Life-Long and Continual Learning, Multimodal Learning

Abstract: Multimodal Continual Learning (MMCL) aims to enable models to continuously accumulate knowledge across multiple tasks and modalities without forgetting previously learned information. Compared to single modal continual learning, MMCL presents greater challenges, as success heavily depends on effective cooperation and complementarity between modalities. Existing methods typically treat modality alignment as a static process, assuming that alignment, once established at a certain layer, remains constant. In this work, we highlight that modality alignment is, in fact, a dynamic process that evolves with task learning and feature propagation through the network. To address this, we propose a novel approach that explicitly models the evolving alignment through Dynamic Alignment Graph Regularization (DAGR), capturing the dynamic changes in modality alignment across layers. Our method incorporates multi-level graph regularization to stabilize this dynamic alignment process, effectively mitigating catastrophic forgetting. Extensive experiments on challenging benchmarks demonstrate that our method outperforms static alignment-based approaches and other continual learning methods, achieving superior stability.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 10981