FlexLoRA: Entropy-Guided Flexible Low-Rank Adaptation

Muqing Liu, Chongjie Si, Yuheng Jia

Keywords: PEFT; Dynamic Rank; LoRA

Abstract:

Large pre-trained models achieve remarkable success across diverse domains, yet fully fine-tuning incurs prohibitive computational and memory costs.
Parameter-efficient fine-tuning (PEFT) has thus become a mainstream paradigm.
Among them, Low-Rank Adaptation (LoRA) introduces trainable low-rank matrices and shows strong performance, nevertheless, its fixed-rank design limits flexibility.
Dynamic rank allocation methods mitigate this issue by pruning redundant directions; however, they often rely on heuristic, element-level metrics that globally sort rank directions without matrix-wise distinction, and they lack mechanisms to expand capacity in layers requiring additional adaptation.
To overcome these limitations, we propose FlexLoRA, an entropy-guided flexible low-rank adaptation framework that (i) evaluates matrix importance via spectral energy entropy, (ii) supports rank pruning and expansion under a global budget, and (iii) employs zero-impact initialization for newly added singular directions to ensure stability.
By addressing granularity, flexibility, and stability limitations, FlexLoRA provides a more principled solution for PEFT.
Extensive experiments show that FlexLoRA consistently outperforms state-of-the-art baselines across benchmarks.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 1418