D2-LoRA: A Synergistic Approach to Differential and Directional Low-Rank Adaptation

Nozomu Fujisawa

Keywords: parameter-efficient fine-tuning, low-rank adapters, directional normalization, mergeability, large language models

Abstract: We present a systematic exploration of the parameter-efficient fine-tuning design space under practical constraints, yielding D$^{2}$-LoRA—a method that reaches 76.4% average accuracy on eight QA/RC benchmarks using only 5k training samples per task and two epochs, while retaining algebraic mergeability at inference with near-exact numerical equivalence. D$^{2}$-LoRA combines a differential signed low-rank residual with a directional per-column normalization applied only during training. Specifically, given a frozen $W_0$, we learn two rank-$r$ components forming an update $\Delta W=\tfrac{\alpha}{r}(A_+B_+-\tau A_-B_-)$. This update is then projected onto the original column norms of $W_0$ to yield $W^\star$, thereby allowing optimization to adjust directional components while preserving the original magnitude. At inference time, we merge $W^\star$ and $\Delta W$ into $\widehat{W}$, which incurs no additional latency. Compared to baselines, D$^{2}$-LoRA achieves a +2.2pp macro improvement over LoRA (74.2%), and matches or exceeds DoRA. At matched parameter counts (LoRA at rank $2r$ vs. D$^{2}$-LoRA at rank $r$), the improvement is +1.6pp, confirming that gains stem from architectural innovations rather than increased parameterization. Beyond QA/RC, D$^{2}$-LoRA improves generative tasks (+1.2pp ROUGE-L, +1.1% win rate) and exhibits 36% lower training volatility. It also preserves numerical equivalence after merging (mean gap $\approx 0.03$pp; worst $0.7$pp), while restoring $\sim 1.91\times$ evaluation throughput. Training overhead is 19%—comparable to DoRA—and decreases with longer input sequences. A geometric analysis explains why projection stabilizes low-rank training, and ablation studies isolate the effects of the negative branch, rank, target modules, scoring function, and fixed $\tau$.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 21196