When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning

Keywords: Continual Learning, Parameter-efficient Fine-tuning, Low-Rank Adaptation, Sparsity

Abstract: Low-Rank Adaptation (LoRA) has become the standard tool for parameter-efficient fine-tuning of large pretrained models. When applied sequentially across tasks in Continual Learning (CL), the standard assumption is that each new task requires a dedicated low-rank adapter. In this work, we challenge this assumption empirically and structurally. We show that task-specific LoRA adapters in CL exhibit significant $\textit{low-rank redundancy}$: the subspaces spanned by adapters trained on different tasks substantially overlap, and in many cases earlier adapters can faithfully represent later tasks. Building on this observation, we propose $\textbf{LiteLoRA}$, a $\textit{plug-and-play}$ gating mechanism that learns at train time whether to recruit a new adapter or reuse existing low-rank representations. Our method reduces the number of active adapters by $20–70$\% while matching or exceeding state-of-the-art performance on standard CL benchmarks, revealing that structural redundancy is pervasive and that selective learning is sufficient to achieve stability without sacrificing plasticity.

Submission Number: 39