ASO-LoRA: Attribution Scores-based Soft Orthogonality Low-Rank Adaptation for Large Language Model Continual Learning

Yitong Wang, Xue Han, wenchun.gao, Qian Hu, Jiahui Wang, Chao Deng, Junlan Feng

Keywords: Parameter-efficient-training, LLM Efficiency, Low-rank adaptation, Soft Orthogonality

TL;DR: Attribution Scores-based Soft Orthogonality Low-Rank Adaptation (ASO-LoRA), an effective and efficient framework for Continual Learning

Abstract: Continual learning (CL) remains a critical challenge when applying large language models (LLMs) to real-world situations. On the one hand, billions of parameters for LLMs add a huge computing overhead to CL. Existing techniques, on the other hand, solely address catastrophic forgetting while ignoring the possibility of knowledge transfer between tasks. Facing these challenges, we propose Attribution Scores-based Soft Orthogonality Low-Rank Adaptation (ASO-LoRA), an effective and efficient framework that simultaneously facilitates knowledge transfer while mitigating catastrophic forgetting. Specifically, ASO-LoRA initially assigns task-specific parameter subspaces for new tasks utilizing multi-LoRA modules, enabling for efficient training and inference without relying on task labels. Then, ASO-LoRA leverages attribution scores to evaluate task similarity and suggests gradient steps in a soft orthogonal direction between task-specific subspaces, achieving a balance between knowledge transfer and preservation. Experiments are carried out on both the T5-large and the LLaMA2-7B, showing ASO-LoRA's suitability as a plug-in CL solution for general Transformer-based LLMs. Experimental results on CL benchmarks demonstrate that ASO-LoRA outperforms other strong baselines.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 8813