Gated Integration of Low-Rank Adaptation for Continual Learning of Language Models

Yan-Shuo Liang, Wu-Jun Li

TL;DR: We propose a new method based on LoRA for continual learning of language models.

Abstract: Continual learning, which requires the model to learn multiple tasks sequentially, is crucial for language models (LMs). Recently, low-rank adaptation (LoRA), one of the most representative parameter-efficient fine-tuning (PEFT) methods, has gained increasing attention in continual learning of LMs. However, most existing continual learning methods based on LoRA typically expand a new LoRA branch to learn each new task and force the new and old LoRA branches to contribute equally to old tasks, potentially leading to forgetting. In this work, we propose a new method, called gated integration of low-rank adaptation (GainLoRA), for continual learning of LMs. GainLoRA expands a new LoRA branch for each new task and introduces gating modules to integrate the new and old LoRA branches. Furthermore, GainLoRA leverages the new gating module to minimize the contribution from the new LoRA branch to old tasks, effectively mitigating forgetting and improving the model's overall performance. Experimental results on continual learning benchmarks demonstrate that GainLoRA outperforms existing state-of-the-art methods.

Primary Area: General Machine Learning

Keywords: continual learning, catastrophic forgetting

Submission Number: 6694