EigenLoRAx: Efficient Low Rank Adaptation Using Recycled Principal Subspaces

ICLR 2026 Conference Submission12710 Authors

Keywords: model recycling, efficiency, carbon footprint

TL;DR: This paper discovers the existence of shared low rank subspace in LoRAs which allow for very efficient finetuning.

Abstract: The rapid growth of large models has raised concerns about their environmental impact and equity in accessibility due to significant computational costs. Low-Rank Adapters (LoRA) offer a lightweight solution for finetuning large models, resulting in an abundance of publicly available adapters tailored to diverse domains. We ask: Can these pretrained adapters be leveraged to further streamline adaptation to new tasks while addressing these challenges? We introduce EigenLoRAx, a parameter-efficient finetuning method that recycles existing adapters to create a principal subspace aligned with their shared domain knowledge which can be further augmented with orthogonal basis vectors in low-resource scenarios. This enables rapid adaptation to new tasks by learning only lightweight coefficients on the principal components of the subspace, eliminating the need to finetune entire adapters. EigenLoRAx requires significantly fewer parameters and memory, improving efficiency for both training and inference. Our method demonstrates strong performance across diverse domains and tasks, offering a scalable solution for edge-based applications and equitable deployment of large models in resource-constrained environments.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 12710