One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning
Abstract: We present Generalized LoRA (GLoRA), a flexible approach for universal parameter-efficient fine-tuning tasks. Enhancing Low-Rank Adaptation (LoRA), GLoRA employs a generalized prompt module to optimize pre-trained model weights and adjust intermediate activations, providing more flexibility and capability across diverse tasks and datasets. Moreover, GLoRA facilitates efficient parameter adaptation by employing a scalable, modular, and layer-wise structure search that learns the individual adapter of each layer. Originating from a unified mathematical formulation, GLoRA exhibits strong transfer learning, few-shot learning and domain generalization abilities, as it adapts to new tasks through not only weights but also additional dimensions like activations. Comprehensive experiments demonstrate that GLoRA outperforms all previous methods in natural, specialized, and structured benchmarks in the field of vision, achieving superior accuracy with fewer parameters and computations. To demonstrate the applicability in the language domain, we perform GLoRA on LLaMA-1/2 models, which also achieve considerable enhancements compared to the original LoRA. Furthermore, our structural re-parameterization design ensures that GLoRA incurs no extra inference cost, rendering it a practical solution for resource-limited applications.
Submission Length: Regular submission (no more than 12 pages of main content)
Assigned Action Editor: ~Vincent_Dumoulin1
Submission Number: 2077
Loading