Subnetwork Knowledge Injection and Transferable Parameter Updating Strategy for Continual Learning of Vision-and-Language Tasks

Jiong Wang, Qing Zhang, Jie Liu, Kaifeng Nie, Ze Zhang, Linqi Song, Yinqiao Li

Abstract: The external environment of the real world is often filled with various types of multimodal information that are highly dynamic and unpredictable. Therefore, intelligent systems need to continuously learn and retain knowledge. Continual learning (CL) provides a foundation for vision-and-language (VaL) models to adapt to the changes in the real world. Existing research on CL for VaL models mainly focuses on preserving memory stability to overcome catastrophic forgetting. However, it is difficult for existing methods to flexibly adapt to the dynamic growth of external information. In this article, we propose a generic approach that appropriately attenuates and employs old memories in parameter distributions to improve learning plasticity. We propose a new parameter-sharing CL method that combines adaptive parameter update strategies with network pruning to enhance the plasticity of VaL models during the CL process. When learning a new task, we utilize the pruned model for learning new tasks. Then, we employ the subnetwork parameter initialization updating strategy to transfer the most important knowledge from learned tasks. Subsequently, to strengthen the stability of the model, we further update the model parameters, adjusting old memories to better adapt to new task learning. Experiments on a series of VaL tasks have shown that our proposed method outperforms the compared methods in improving model plasticity and is more stable than existing parameter sharing methods.

External IDs:doi:10.1109/tai.2025.3564915