DropLoRA: Sparse Low-Rank Adaptation for Parameter-Efffcient Fine-Tuning

ACL ARR 2025 May Submission389 Authors

Abstract: LoRA-based large model parameter-efficient fine-tuning (PEFT) methods use low-rank decomposition to approximate updates to model parameters. However, compared to full-parameter fine-tuning, low-rank updates often lead to a performance gap in downstream tasks. To address this, we introduce DropLoRA, a novel pruning-based approach that focuses on pruning the rank dimension. Unlike conventional methods that attempt to overcome the low-rank bottleneck, DropLoRA innovatively integrates a pruning module between the two low-rank matrices in LoRA to simulate dynamic subspace learning. This dynamic low-rank subspace learning allows DropLoRA to overcome the limitations of traditional LoRA, which operates within a static subspace. By continuously adapting the learning subspace, DropLoRA significantly boosts performance without incurring additional training or inference costs. Our experimental results demonstrate that DropLoRA consistently outperforms LoRA in fine-tuning the LLaMA series across a wide range of large language model generation tasks, including commonsense reasoning, mathematical reasoning, code generation, and instruction-following. Our code will be available after the anonymous review.

Paper Type: Long

Research Area: Efficient/Low-Resource Methods for NLP

Research Area Keywords: Efficient/Low-Resource Methods for NLP

Contribution Types: NLP engineering experiment, Approaches to low-resource settings

Languages Studied: English

Keywords: Pruning, LoRA, Subspace Learning

Submission Number: 389