ProRA: Projection Aware Low-Rank Adaptation for Parameter Efficient Fine-Tuning

Ritik Mishra, M. Tanveer

Keywords: PEFT, LoRA, LLM, Finetune, Projection.

TL;DR: We propose ProRA, a novel PEFT method for LLMs that initializes adapters via orthonormal projections of pre-trained weights, preserving geometric structure, accelerating convergence, and improving performance over standard low-rank adaptation.

Abstract: Despite the remarkable success of large language models (LLMs) across diverse tasks, the computational cost of fine-tuning them remains high. Low-Rank Adaptation (LoRA) addresses this by updating through the product of two low rank matrices. LoRA initializes low-rank matrices using random Gaussian noise and zeros, while keeping the pretrained weights frozen. However, such random and zero initialization leads to slow convergence and limits expressiveness. To overcome these limitations, we propose Projection Aware Low-Rank Adaptation (ProRA). ProRA initializes adapter matrices by projecting the original weight matrix into its orthonormal subspace and keeps the residual weight matrix frozen. ProRA leverages the orthonormal projection to ensure that updates preserve the geometric structure of pretrained models and are aligned with orthogonal subspaces, leading to faster convergence and improved performance. Furthermore, we interpret ProRA through the lens of geometric complexity. ProRA lowers geometric complexity in the frozen weights, which facilitates more efficient fine-tuning. Our proposed ProRA demonstrates empirical superiority over LoRA across diverse tasks. On the GSM8K benchmark dataset, ProRA achieves 78.11\% accuracy with GEMMA-7B, outperforming LoRA’s 74.53\% by 3.58\%. Comparative evaluations across various model architectures consistently show that ProRA outperforms LoRA, highlighting its robustness and effective fine-tuning capability.

Primary Area: optimization

Submission Number: 23786