Randomized Asymmetric Chain of LoRA: The First Optimization-Theoretic Framework for Low-Rank Adaptation

Abstract: Fine-tuning is a common approach to adapt large models to specific tasks. Low-Rank Adaptation (LoRA) is a widely used method for parameter-efficient fine-tuning, where updates are represented as products of low-rank matrices. Although LoRA performs well in practice, it often gives weaker results than full-parameter fine-tuning (FPFT), and its theoretical understanding is still limited. In this work, we show that LoRA and its extensions, such as Asymmetric LoRA and Chain of LoRA, can have convergence problems. To solve this, we introduce Randomized Asymmetric Chain of LoRA (RAC-LoRA), a general optimization framework that provides theoretical guarantees of convergence for LoRA-based methods. Our approach keeps the practical advantages of LoRA but adds important algorithmic changes to ensure convergence. We prove that RAC-LoRA can reach the same solution as FPFT, and we analyze its convergence rate. Our results cover smooth, non-convex objectives and include gradient descent, stochastic gradient descent, and federated learning setups. Experiments support our theory.

Submission Type: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Binhang_Yuan1

Submission Number: 8053