One LoRA for All: Unlocking Latent Capacity of LoRA for Continual Adaptation

ACL ARR 2026 January Submission5231 Authors

Keywords: Continual Learning; Large Language Models; Parameter-Efficient Fine-Tuning; Low-Rank Adaptation

Abstract:

Continual Learning (CL) enables Large Language Models (LLMs) to incrementally acquire new knowledge without costly retraining.
However, existing CL methods exhibit a fundamental trade-off. Regularization-based approaches maintain a fixed parameter budget but often suffer from degraded performance over long task sequences. Expansion-based methods, in contrast, preserve performance by dynamically extending the model architecture. However, this comes at the cost of a growing memory footprint.
To resolve this trade-off, we propose Continual Adaptation via Subspace Trimming (CAST), a parameter-efficient framework that systematically allocates and organizes task-specific subspaces within a single LoRA module, maintaining strong performance over long task sequences without increasing the parameter budget.
CAST leverages the intrinsic redundancy of LoRA to allocate sparse, task-specific subspaces and utilizes a lightweight semantic routing mechanism to facilitate task-agnostic inference.
Experiments on LLaMA 3.1-8b and Qwen 3-8b show that CAST outperforms related continual learning methods while maintaining a nearly constant training memory footprint and preserving over 25% idle capacity even after 15 sequential tasks.

Paper Type: Long

Research Area: Machine Learning for NLP

Research Area Keywords: continual learning;

Contribution Types: Approaches low compute settings-efficiency

Languages Studied: English

Submission Number: 5231