Continual Gradient Low-Rank Projection Fine-Tuning for LLMs

ACL ARR 2025 February Submission3761 Authors

Abstract: Large Language Models (LLMs) exhibit remarkable performance across diverse domains, and the advent of Low-Rank Adaptation (LoRA) has broadened the possibilities for continual fine-tuning. Furthermore, its explicit parameter constraints do not dynamically adapt to evolving gradient spaces across tasks, hindering effective knowledge transfer. In this paper, we propose GORP ($\underline{\textbf{G}}$radient L$\underline{\textbf{O}}$w $\underline{\textbf{R}}$ank $\underline{\textbf{P}}$rojection ) for continual learning, a novel training strategy that integrates full-rank and low-rank parameters with low-rank gradient updates. Specifically, GORP jointly trains both full-rank and low-rank parameters, with the former updated in a low-rank fashion. Both parameter sets are then projected into a unified low-rank gradient space to mitigate catastrophic forgetting. Extensive experiments on multiple continual learning benchmarks demonstrate that our method consistently outperforms existing state-of-the-art approaches.

Paper Type: Long

Research Area: NLP Applications

Research Area Keywords: LLMs, continual fine-tuning, low-rank projection

Contribution Types: Approaches to low-resource settings, Approaches low compute settings-efficiency

Languages Studied: English

Submission Number: 3761