Kron-LoRA: Hybrid Kronecker-LoRA Adapters for Scalable, Sustainable Fine-tuning

Yixin Shen

Keywords: Kron-LoRA, Kronecker factorization, LoRA, parameter-efficient fine-tuning, quantization-friendly adapters, multi-task adaptation, continual learning

TL;DR: Kron-LoRA integrates Kronecker structure with low-rank LoRA, yielding up to 4× parameter reduction while preserving accuracy and enabling efficient multi-task and sequential fine-tuning of LLMs.

Abstract: Fine-tuning massive pre-trained language models across many tasks demands adapters that are both parameter-efficient and expressive. We introduce Kron-LoRA, a hybrid adapter that combines Kronecker-structured factorization with low-rank LoRA compression—an integration that, to our knowledge, has not been explored in the literature on parameter-efficient fine-tuning or matrix approximation. Kron-LoRA achieves up to 4× fewer parameters than standard LoRA while retaining similar expressivity. Experiments on DistilBERT, Mistral-7B, LLaMA-2-7B, and LLaMA-3-8B across eight benchmarks show that Kron-LoRA matches or exceeds LoRA baselines with modest memory savings and only a 5–8% speed overhead. In sequential fine-tuning, it also delivers competitive cross-task transfer despite using only one-quarter of the adapter parameters. Kron-LoRA thus offers a scalable, sustainable solution for multi-task adaptation of large language models.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 7199