Enhancing LoRA with Shared Random-Span Augmentation for Parameter-Efficient Visual Adaptation

ICLR 2026 Conference Submission17457 Authors

Keywords: Parameter-efficient Fine-tuning; Low-rank Adaptation; Random Matrix; Parameter Sharing

TL;DR: We propose a noval PEFT method that enhances the expressive capability of LoRA with a rank adaptation mechanism.

Abstract: Low-rank Adaptation (LoRA) efficiently adapts large pre-trained models to downstream tasks by learning low-rank adapters, significantly reducing computational and memory costs without sacrificing performance. Recent studies highlight the promise of rank adaptation methods in improving the flexibility and performance of LoRA. Grounded in Singular Value Decomposition (SVD) theory, these methods decompose the weight update into parameterized unitary matrices and learnable scaling coefficients, thereby allowing dynamic rank allocation of adapters based on coefficients. However, the parameterized construction of unitary matrices presents a significant computational bottleneck. To address this limitation, we propose Shared Random-Span Augmentation (SRSA), a novel Parameter-Efficient Fine-Tuning (PEFT) method that replaces the learnable unitary matrices with fixed, layer-shared random matrices. Our method facilitates flexible rank adaptation by learning scaling vectors within the shared random space, while maintaining parameter and memory efficiency. We provide both empirical and theoretical evidence to demonstrate the feasibility of substituting the unitary matrices with a shared random matrix. To evaluate the representational ability of our method, we conduct extensive experiments on various visual tasks. The results demonstrate that our method achieves compelling adaptation performance.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 17457