NLoRA: Nyström-Initiated Low-Rank Adaptation for Large Language Models

ACL ARR 2025 February Submission6354 Authors

Abstract: Parameter-efficient fine-tuning (PEFT) is essential for adapting large language models (LLMs), with low rank adaptation (LoRA) being the most popular approach. However, LoRA suffers from slow convergence, and some recent LoRA variants, such as PiSSA, primarily rely on Singular Value Decomposition (SVD) for initialization, leading to expensive computation. To mitigate these problems, we resort to Nyström method, which follows a three-matrix manipulation. Therefore, we first introduce $\textbf{S}$tructured$\textbf{LoRA}$ (SLoRA), investigating to introduce a small intermediate matrix between the low-rank matrices A and B. Secondly, we propose $\textbf{N}$yström$\textbf{LoRA}$ (NLoRA), which leverages Nyström-based initialization for SLoRA to improve its effectiveness and efficiency. Finally, we propose $\textbf{Int}$ermediate$\textbf{Tune}$ (IntTune) to explore fine-tuning exclusively the intermediate matrix of NLoRA to furthermore boost LLMs' efficiency. We evaluate our methods on 5 natural language generation (NLG) tasks and 8 natural language understanding (NLU) tasks. On GSM8K, SLoRA and NLoRA achieve accuracies of 56.48\% and 57.70\%, surpassing LoRA by 33.52\% and 36.41\% with only 3.67M additional trainable parameters. IntTune boosts average NLG performance over LoRA by 7.45\% while using only 1.25\% of its parameters. These results demonstrate the efficiency and effectiveness of our approach in enhancing model performance with minimal parameter overhead.

Paper Type: Long

Research Area: Efficient/Low-Resource Methods for NLP

Research Area Keywords: parameter-efficient-training

Contribution Types: Approaches low compute settings-efficiency

Languages Studied: English

Submission Number: 6354