ModuLoRA: Finetuning 2-Bit LLMs on Consumer GPUs by Integrating with Modular Quantizers

Junjie Yin, Jiahao Dong, Yingheng Wang, Christopher De Sa, Volodymyr Kuleshov

Event Certifications: iclr.cc/ICLR/2024/Journal_Track

Abstract: We propose a memory-efficient finetuning algorithm for large language models (LLMs) that supports finetuning LLMs with 65B parameters in 2/3/4-bit precision on as little as one 24GB GPU. Our method, modular low-rank adaptation (ModuLoRA), integrates any user-specified weight quantizer with finetuning via low-rank adapters (LoRAs). Our approach relies on a simple quantization-agnostic backward pass that adaptively materializes low-precision LLM weights from a custom black-box quantization module. This approach enables finetuning 2-bit and 3-bit LLMs for the first time---leveraging state-of-the-art 2-bit QuIP# quantization and 3-bit OPTQ quantization---outperforming finetuning that relies on less sophisticated 4-bit and 8-bit methods. In our experiments, ModuLoRA attains competitive performance on text classification, natural language infernece, and instruction following tasks using significantly less memory than existing approaches, and we also surpass the state-of-the-art ROUGE score on a popular summarization task. We release ModuLoRA together with a series of low-precision models as part of LLMTOOLS, a user-friendly library for quantizing, running, and finetuning LLMs on consumer GPUs.

Submission Length: Regular submission (no more than 12 pages of main content)

Changes Since Last Submission: We have updated the paper throughout to reflect our integration of 2-bit QuIP quantization (please see revised Title, Abstract, Introduction, and other sections).

Certifications: Featured Certification

Assigned Action Editor: ~Shixiang_Gu1

License: Creative Commons Attribution 4.0 International (CC BY 4.0)

Submission Number: 1596