Exponential Low-Rank Adapters

Shayan Mohajer Hamidi, Mert Pilanci

Keywords: LoRA, LLM fine-tuning

Abstract: Low rank adaptation (LoRA) is a standard parameter efficient fine tuning method, but its updates are rank limited and act as local additive perturbations of the weights. We introduce exponential low rank adapters (ELRA), which replace LoRA’s additive update $\Delta W=AB$ with a multiplicative transformation $W_{\mathrm{new}}=\exp(\eta AB)\,W_0$, where $A\in\mathbb{R}^{d\times r}$ and $B\in\mathbb{R}^{r\times d}$ define a low rank generator and $W_0$ is the frozen pretrained weight. The matrix exponential lifts the generator to a full rank, invertible map that acts coherently on $W_0$. Geometrically, ELRA traces curves on $GL(d)$; when the generator is normal, the path $t\mapsto \exp(tG)W_0$ is a constant speed, locally energy minimizing geodesic under the left invariant trace metric. This motivates ELRA\text{-}PSD, which constrains $G$ to be symmetric positive semidefinite, stabilizing training by enforcing geodesic flows with an energy control on path length. To couple stability with expressivity, we propose ELRA-Hyb, which interpolates between PSD and general generators, starting stable and progressively unlocking capacity. Experiments on diverse language and vision benchmarks show that ELRA-Hyb consistently outperforms state of the art LoRA variants under matched parameter budgets.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 1062