Balanced Low-Rank Adaptation: Removing Invariance for Fast and Stable Fine-Tuning

Valérie Castin, Kimia Nadjahi, Pierre Ablin, Gabriel Peyré

Keywords: Fine-tuning, LoRA, Transformers, balanced manifold, invariance

TL;DR: BaLoRA accelerates convergence and improves robustness to hyperparameter tuning by projecting LoRA iterates onto a balanced manifold.

Abstract:

Low-Rank Adaptation (LoRA) is the most widely used method for fine-tuning large language models.
LoRA is overparameterized: multiple pairs of low-rank factors correspond to the same adapted weight matrix.
We observe both theoretically and numerically that these pairs can have significantly different condition numbers: converging to different minimizers of the loss affects the convergence rate of LoRA.
Building on this remark, we introduce Balanced Low-Rank Adaptation (BaLoRA), a variant of LoRA that projects iterates onto a balanced manifold where the conditioning of the loss is improved while keeping the same adapted matrix.
This projection step is computationally inexpensive and integrates seamlessly with existing fine-tuning pipelines.
Empirically, BaLoRA converges faster than standard LoRA and exhibits greater robustness to hyperparameter choices across a range of fine-tuning tasks.

Primary Area: optimization

Submission Number: 11305