Go to ICLR 2025 Conference homepageShareLoRA: Less Tuning, More Performance for LoRA Fine-tuning of LLMs
Fine-tuning large language models (LLMs) is prohibitively expensive, prompting the development of various parameter-efficient fine-tuning (PEFT) methods. These methods primarily focus on fine-tuning small, additional modules known as adapters, which account for only a small fraction of the total LLM parameters. One such method, low-rank adaptation (LoRA), has shown notable parameter efficiency while maintaining performance comparable to full fine-tuning. However, classical LoRA may still involve tuning more parameters than necessary given the intrinsic rank of pre-trained weights, as highlighted by prior work. In this work, we introduce ShareLoRA, a novel approach that further enhances parameter efficiency during LLM fine-tuning by leveraging redundancies in pre-trained model weights to share LoRA modules, thereby significantly reducing the number of trainable parameters. Specifically, ShareLoRA automatically identifies redundancies in the pre-trained weights and determines which LoRA adapters can share parameters. This is achieved by measuring the similarity between representations to assess information redundancy and using a greedy algorithm to maximize parameter sharing. We conducted extensive evaluations on the LLMs of the LLaMA family across benchmark tasks. Notably, ShareLoRA achieves better parameter efficiency, with up to a 23% reduction in the number of fine-tuned parameters while delivering performance comparable to or better than existing PEFT methods.