The Scaling Law of LoRA Based on Mutual Information Upper Bound

ACL ARR 2025 February Submission7989 Authors

Abstract: LoRA (Low-Rank Adaptation) is a widely used LLM fine-tuning method. During the fine-tuning process, the Scaling Law can guide the selection of the optimal model scale and data complexity to balance model performance and fine-tuning costs. Although existing methods frequently rely on external metrics (e.g., cross-entropy or perplexity) to evaluate model performance, the scaling law may exhibit instability during testing, which is largely attributed to the generalization gap between training and testing. To address this issue, we propose the Mutual Information Upper Bound (MIUB) metric between base modules and LoRA modules, to investigate the Scaling Law in the large-scale LoRA fine-tuning context. The metric gauges the dependency between the general knowledge obtained during pre-training and the task-specific knowledge acquired through LoRA adaptation. In doing so, the metric pays more attention to the distribution changes within the LoRA architecture, so as to evaluate the Scaling Law more robustly. In our experiments, we validated this approach on benchmark datasets, using the Llama3-8B and Phi3-3B models. The results show that the proposed MIUB metric aligns more accurately and stably with the scaling law of LoRA fine-tuning compared to cross-entropy, perplexity and more metrics.

Paper Type: Long

Research Area: Language Modeling

Research Area Keywords: Language Modeling, Scaling, Fine-tuning

Languages Studied: English

Submission Number: 7989