ULoRA: Universal Low-Rank Adaptation of Diverse Deep Learning Architectures
Keywords: LoRA, Universal Adaptation, Deep Learning, Parameter Fine Tuning, Transformer, Language Models
Abstract: To train Large Language Models (LLMs) having a large number of parameters, the Parameter-Efficient Fine Tuning (PEFT) method based on LoRA, which allows fine-tuning with fewer parameters, is widely employed. However, these methods are primarily designed for application to Transformer architectures, which presents challenges when attempting to apply them to models such as Mamba. To address this limitation, this work proposes Universal LoRA (ULoRA), which applies a Low-Rank Adapter to all deep learning models at the level of universally common blocks. ULoRA achieves generalizability by applying Low-Rank Adapters to blocks, making it applicable to models that do not utilize Transformer architectures. Furthermore, by grouping multiple blocks and applying a single Low-Rank Adapter, ULoRA provides structural flexibility that allows a further reduction in the number of parameters. This significantly reduces resource usage and inference time, making it well-suited for on-device environments with limited resources, while only incurring a slight performance loss. Additionally, if all blocks are grouped to use a single Low-Rank Adapter, task switching during inference is enabled by computing only the adapter. Experimental results show that, for LLaMA-3-8B, ULoRA achieves comparable performance to LoRA with only about 60% of the parameters, while delivering up to 8% higher throughput. For Mamba-2.8B, ULoRA outperforms LoRA with only about 20% of the parameters. In scenarios with limited available resources, ULoRA can be applied using just 4% of the parameters of LoRA, with only a 10% reduction in performance.
Primary Area: transfer learning, meta learning, and lifelong learning
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 11065
Loading