Go to ACL ARR 2025 February homepageEnhancing PEFT Efficiency by Adaptively Reducing Low-Rank Modules
As a prominent Parameter-Efficient Fine-Tuning (PEFT) method, LoRA is widely used for efficiently fine-tuning large language models (LLMs). However, LoRA’s uniform insertion of trainable modules to target modules across all layers often results in redundancy in the number of trainable modules, and we contend that reducing the number of these modules can further enhance the efficiency of PEFT. To address this issue, we propose Gradient-Guided Redundancy Reduction ($\mathcal{G}^2\mathcal{R}^2$), a novel module-level approach that adaptively prunes redundant LoRA modules, which boosts fine-tuning efficiency while preserving or even improving performance. Specifically, $\mathcal{G}^2\mathcal{R}^2$ evaluates the contribution and redundancy of trainable modules using a Gradient-Based Redundancy Evaluation score, which leverages gradient information to achieve this. Based on this score, $\mathcal{G}^2\mathcal{R}^2$ progressively eliminates redundant LoRA modules through a Three-Stage Redundancy Reduction Strategy. Extensive experiments demonstrate that $\mathcal{G}^2\mathcal{R}^2$ not only boosts fine-tuning efficiency but also maintains or even surpasses state-of-the-art methods across commonsense reasoning and natural language understanding tasks.