OP-LoRA: The Blessing of Dimensionality with Overparameterized Low-Rank Adaptation

Piotr Teterwak, Kate Saenko, Bryan A. Plummer, Ser-Nam Lim

Keywords: PeFT, LoRA, Finetuning

TL;DR: LoRA is difficult to optimize; we fix that by re-parameterization.

Abstract: Low-rank adapters (LoRA) enable finetuning of large models with only a small number of parameters, reducing storage costs and minimizing the risk of catastrophic forgetting. However, they often suffer from an ill-conditioned loss landscape, leading to difficult optimization. Prior work addresses these challenges by aligning adapter updates with full finetuning gradients via custom optimizers, but these methods lack the flexibility to accommodate new adapter architectures and are computationally expensive. We instead introduce OP-LoRA, a novel method which replaces each LoRA adapter with weights predicted by an extra MLP, which is discarded after training. This temporarily allows additional parameters during training to improve optimization, yet requires less wall time than custom optimizers and zero extra cost at inference time because the MLP is discarded. Crucially, extending OP-LoRA to other adapters is as simple as modifying the size of the prediction head for each new adapter type. Since the additional parameters are used only during training and thrown away before inference, there is no risk of overfitting due to increased representational capacity, unlike simply raising the LoRA rank. Instead, we show that this approach allows the optimization to adaptively increase or decrease step size, improving performance and decreasing sensitivity to learning rate. On both small and large-scale LoRA tuning tasks, we observe consistent performance gains of OP-LoRA relative to LoRA and its variants. We achieve especially notable improvements in image generation, with OP-LoRA CMMD scores improving by up to 15 points relative to LoRA.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 15099