LoRa-Over: A Matrix Decomposition-Based Over-Parameterization for Efficient LLM Fine-Tuning

ICLR 2026 Conference Submission20270 Authors

Keywords: LoRA; large language models; matrix decomposition

Abstract: The rise of large language models (LLMs) has revolutionized machine learning, yielding state-of-the-art results across various tasks through extensive pre-training. While full-parameter fine-tuning is the standard for adapting LLMs, its high storage and computational costs are major drawbacks. Parameter-efficient methods, such as LoRA, mitigate these issues by updating a small subset of parameters, but often sacrifice generalization performance. In this work, we introduce LoRa-Over (Over-Parameterization for Low-Rank Adaptation), a novel approach that enhances generalization by strategically over-parameterizing low-rank matrices during fine-tuning. Using matrix decomposition, LoRa-Over achieves near-lossless reconstruction and maintains inference efficiency. It employs static and dynamic strategies to selectively over-parameterize critical matrices, balancing computational cost and performance. LoRa-Over is validated on tasks such as natural language understanding (GLUE with T5-Base), dialogue generation (MT-Bench), mathematical reasoning (GSM8K), and code generation (HumanEval) using Llama 2-7B and Llama 3.1-8B models. Results show significant performance improvements over vanilla LoRA, demonstrating its potential as a scalable, efficient fine-tuning framework for diverse downstream applications. All the experimental codes will be released after the review period.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 20270