Orthonormal Regularization in Low-Rank Adaptation

Yang Cao, Zhao Song

Keywords: Low-rank Adaptation, parameter efficient fine-tuning, large language models

Abstract:

Performance degradation on tasks outside the fine-tuning domain is often observed while performing parameter-efficient fine-tuning (PEFT) on neural networks with limited data. For example, fine-tuning on mathematical datasets may impair the large language model’s coding ability. We analyze this issue and identify the condition number of weight matrices as a key factor contributing to such degradation. To address this, we propose Singular Values and Orthonormal Regularized Singular Vectors Adaptation, or SORSA,
a novel PEFT method that explicitly improves the conditioning of the adapted model parameters, thereby mitigating degradation and preserving broader capabilities. Empirically, we demonstrate that SORSA outperforms full fine-tuning, LoRA, PiSSA and AdaLoRA.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 2891