DynamicRank LoRA: Real-Time Adaptive Fine-Tuning \\ for Code Models via Token-Level Importance and Loss Landscape Awareness

ICLR 2026 Conference Submission25435 Authors

Keywords: Real-Time Adaptive Fine-Tuning

Abstract:

\begin{abstract}
We propose \textbf{DynamicRank LoRA}, a novel fine-tuning mechanism for code models that dynamically adjusts the rank of low-rank adaptation (LoRA) matrices in real-time, addressing the limitations of static rank configurations in conventional LoRA. The proposed approach combines two fundamental ingredients: token level importance scoring: the structural importance of their input tokens and loss landscape aware rank adaptation: rank modulation, which can be adjusted with information about gradient dynamics and curvature. High importance tokens, namely syntax keywords or variable names, will result in rank increases to get finer grain patterns, and flat loss regions, to reduce rank for faster convergence. The mechanism is tightly coupled with transformer architectures, and makes use of attention weights and gradient norms to "plasma" LoRA matrices through truncated SVD through training. We apply DynamicRank LoRA in the framework of a GPT-3.5-turbo where dense layers in the feed-forward blocks are replaced with those of adaptive-rank LoRA pairs modulated by a lightweight MLP. This design allows the model to very well balance the speed and precision of adaptation between the various combinations of input complexity, e.g. verbose or terse code, and task requirements, i.e. bug fixing, code generation, etc. Experimental results show that DynamicRank LoRA is more efficient and accurate for fine-tuning compared to fixed-rank baselines, especially under the need of fast adaptation to inhomogeneous code structures. The two-fold rank modulation technology and the transformer-specific integration of the methodology distinguishes it from previous works to provide a scalable solution for real time code model customization without compromising the latency.
\end{abstract}

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 25435