SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning

Jinpeng Chen, Runmin Cong, Yuzhi Zhao, Hongzheng Yang, Guangneng Hu, Horace Ip, Sam Kwong

Abstract: Multimodal Continual Instruction Tuning (MCIT) aims to enable Multimodal Large Language Models (MLLMs) to incrementally learn new tasks without catastrophic forgetting, thus adapting to evolving requirements. In this paper, we explore the forgetting caused by such incremental training, categorizing it into superficial forgetting and essential forgetting. Superficial forgetting refers to cases where the model’s knowledge may not be genuinely lost, but its responses to previous tasks deviate from expected formats due to the influence of subsequent tasks’ answer styles, making the results unusable. On the other hand, essential forgetting refers to situations where the model provides correctly formatted but factually inaccurate answers, indicating a true loss of knowledge. Assessing essential forgetting necessitates addressing superficial forgetting first, as severe superficial forgetting can conceal the model’s knowledge state. Hence, we first introduce the Answer Style Diversification (ASD) paradigm, which defines a standardized process for data style transformations across different tasks, unifying their training sets into similarly diversified styles to prevent superficial forgetting caused by style shifts. Building on this, we propose RegLoRA to mitigate essential forgetting. RegLoRA stabilizes key parameters where prior knowledge is primarily stored by applying regularization to LoRA’s weight update matrices, enabling the model to retain existing competencies while remaining adaptable to new tasks. Experimental results demonstrate that our overall method, SEFE, achieves state-of-the-art performance.

Lay Summary: Multimodal large language models often suffer from forgetting previously acquired capabilities when adapted to new tasks. We categorize this forgetting into two distinct types: superficial forgetting, which reflects a loss of correct response style, and essential forgetting, which corresponds to the actual degradation of learned knowledge. To mitigate both types, we propose SEFE, a simple yet effective method that promotes consistent response styles across tasks to reduce superficial forgetting, while constraining updates to critical parameters to mitigate essential forgetting. Experimental results demonstrate that SEFE effectively alleviates forgetting and improves overall performance.

Link To Code: https://github.com/jinpeng0528/SEFE

Primary Area: Deep Learning

Keywords: Multimodal Continual Instruction Tuning, Multimodal Large Language Model, Continual Learning

Submission Number: 13874