SplitLoRA: Balancing Stability and Plasticity in Continual Learning Through Gradient Space Splitting

ICLR 2026 Conference Submission17141 Authors

Keywords: Continual Learning, Low-Rank Adaptation, Gradient orthogonal projection

TL;DR: This paper presents SplitLoRA, a method for continual learning that combines orthogonal projection with LoRA. It improves the balance between plasticity and stability by effectively mitigating interference between new and old tasks.

Abstract: Continual Learning (CL) requires a model to learn multiple tasks in sequence while maintaining both stability—preserving knowledge from previously learned tasks, and plasticity—effectively learning new tasks. Orthogonal projection has emerged as an effective and popular paradigm in CL, where it partitions the gradient space of previously learned tasks into two orthogonal subspaces: a primary subspace and a minor subspace. New tasks are learned effectively within the minor subspace, thereby reducing interference with previously acquired knowledge. However, existing orthogonal projection methods struggle to achieve an optimal balance between plasticity and stability, as it is hard to appropriately partition the gradient space. In this work, we consider a continual learning paradigm based on Low-Rank Adaptation (LoRA), which has gained considerable attention due to its efficiency and wide applicability, and propose a novel approach for continual learning, called SplitLoRA. We first provide a theoretical analysis of how subspace partitioning affects model stability and plasticity. Informed by this analysis, we then introduce an effective method that derives the optimal partition of the gradient space for previously learned tasks. This approach effectively balances stability and plasticity in continual learning. Experimental results on multiple datasets demonstrate that the proposed method achieves state-of-the-art performance. The code is available at https://anonymous.4open.science/r/SplitLoRA-FB45.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 17141