Global Factorized Adaptation: Parameter-Efficient Fine-Tuning via Cross-Module Low-Rank Sharing

Keywords: low-rank adaptation, parameter-efficient fine-tuning, LoRA, DoRA, foundation models

Abstract:

We introduce *Global Factorized Adaptation* (GFA), a parameter-efficient fine-tuning primitive that replaces LoRA's $N$ per-module low-rank factorizations with a single global factorization over the concatenated *weight update*.
The substitution targets a previously underexplored operating point: LoRA-like per-step compute, competitive accuracy against a strong per-module baseline, and adapter budgets below the full-precision local rank-$1$ floor.
Across $17$ (model, task) cells covering $11$ unique tasks on RoBERTa-large, Qwen-2.5-3B, and LLaMA-3.2-3B, GFA is competitive with DoRA at a matched number of parameters. At the same time, GFA matches LoRA's median tokens-per-second and VRAM, whereas DoRA pays a $24~74\%$ wallclock penalty.
Under an equal-module approximation, GFA uses $O(r\sqrt d)$ trainable parameters rather than LoRA's $O(r\sqrt{dN})$, yielding a $\sqrt N$ parameter-count advantage. This enables a sub-LoRA-$r=1$ budget anchor that uses roughly $0.3\times$ the parameters of LoRA $r=1$ while retaining at least $97\%$ of rank-$16$ accuracy on every evaluated decoder task.

Submission Number: 127