Unlocking the Global Synergies in Low-Rank Adapters
Abstract: Low-rank adaption (LoRA) has been the de-facto parameter-efficient fine-tuning technique for large language models. We present HeteroLoRA, a light-weight search algorithm that leverages zero-cost proxies to allocate the limited LoRA trainable parameters across the model for better fine-tuned performance. In addition to the allocation for the standard LoRA-adapted models, we also demonstrate the efficacy of HeteroLoRA by performing the allocation in a more challenging search space that includes LoRA modules and LoRA-adapted shortcut connections. Experiments show that HeteroLoRA enables improvements in model performance given the same parameter budge. For example, on RTE, we see an improvement of 6.7% in accuracy with a similar training parameter budget compared to a variety of state-of-the-art methods. We will open-source our algorithm once the paper is accepted.
Paper Type: Long
Research Area: Efficient/Low-Resource Methods for NLP
Research Area Keywords: parameter-efficient-training
Contribution Types: Approaches to low-resource settings
Languages Studied: English
Submission Number: 5178
Loading