PiLoRA: Gradient-Informed Parameter-Importance-Aware Low-Rank Adaptation

Huiyi Wang, Lina Yao, Dong Gong

Keywords: Low-rank adaptation, Large language models, Fine-tuning

Abstract: Parameter-Efficient Fine-Tuning methods, such as Low-Rank Adaptation (LoRA), are widely used to adapt pre-trained models (PTMs) due to its computational efficiency. However, LoRA and existing LoRA-based methods treat weight modules in the PTMs as indivisible units and apply a uniform adaptation capacity. This coarse, module-level approach overlooks the varying importance of parameters within a module and fails to exploit the inherent structured sparsity of foundation models. To address these limitations, we propose Parameter-Importance-Aware Low-Rank Adaptation (PiLoRA), a novel PEFT method that allocates different adaptation capacity based on parameter importance. Our approach uses efficient low-rank gradients to approximate parameter importance in the full weight parameter space, enabling partition of neurons into distinct groups. Important neurons receive a higher adaptation capacity for targeted training, while others are tuned with lower capacity for efficiency. Comprehensive experiments on both the language reasoning and image classification settings demonstrate that PiLoRA consistently outperforms vanilla LoRA with targeted fine-tuning using fewer parameters, striking a balance between effective adaptation and efficiency. Our work shifts the focus of adaptation from the module level to a more granular neuron level and unlocks a more powerful and efficient approach to PEFT.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 1898