ODELoRA: Training Low-Rank Adaptation by Solving Ordinary Differential Equations

Keywords: low-rank adaptation, ordinary differential equations, Runge--Kutta methods, balanced factorization

Abstract:

Low-rank adaptation (LoRA) has emerged as a widely adopted parameter-efficient fine-tuning method in deep transfer learning, due to its reduced number of trainable parameters and lower memory requirements enabled by Burer–Monteiro factorization on adaptation matrices.
However, classical LoRA training methods treat the low-rank factor matrices individually and optimize them using standard gradient-based algorithms. Such decoupled optimization schemes are theoretically and empirically suboptimal, as they fail to fully exploit the intrinsic structure of the LoRA parameterization. In this work, we propose a novel continuous-time optimization dynamic for LoRA factor matrices in the form of an ordinary differential equation (ODE) that emulates the gradient flow of full fine-tuning on the balanced manifold.
We term this approach ODELoRA. To faithfully track the trajectories of ODELoRA, we adopt well-established and theoretically grounded time-discretization schemes, including Euler and Runge--Kutta methods. Our framework provides a unified ODE-based perspective for understanding and designing LoRA training algorithms. Empirical results on training physics-informed neural networks demonstrate the superiority of ODELoRA over existing baselines, especially in the training stability.

Submission Number: 72