Reversible Primitive–Composition Alignment for Continual Vision–Language Learning

Canran Xiao, Tianxiang Xu, siyuanma, Yiyang Jiang, Haoyu Gao, Yuhan Wu

Keywords: continual learning, vision-language models, catastrophic forgetting

Abstract: Vision-language (VL) models are increasingly deployed in non-stationary settings, yet under sequential adaptation they often preserve primitive recognition while losing compositional structure, especially with tight rehearsal budgets and no task IDs. We address this gap by asking how a continual VL system can maintain structurally dependable behaviour while safeguarding zero-shot performance. We introduce Compo-ReAlign, a structure-first recipe built around three components: a reversible composer that maps primitive embeddings to compositions by design, a multi-positive InfoNCE that jointly aligns textual and composed views of the same target, and a spectral trust region that clips updates when alignment sensitivity inflates. Across compositional DIL and multi-domain MTIL retrieval, Compo-ReAlign sets a new state of the art, improves over the strongest prior by +2.4 R@1, and reduces forgetting by 40%. We provide a compact, reversible alignment head with geometry-aware training for compositionally robust VL continual learning.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 9572