FedQLoRA: Federated Quantization-Aware LoRA for Large Language Models

Zhiwei Hu, Liang Zhang, Shan Dai, Shihua Gong, Qingjiang Shi

Keywords: Quantization, LoRA, Large Language Models, Federated Learning

Abstract: Large language models (LLMs) with billions of parameters have achieved remarkable success across various applications, but they require substantial computational resources and large datasets. While parameter-efficient fine-tuning methods like LoRA and QLoRA have significantly reduced computational costs and memory usage, robustly training LLMs for individual clients with datasets distributed on isolated devices remains challenging. To address this, recent work has explored the use of federated learning (FL) to collaboratively train LLM adapters on distributed private data, thereby avoiding the high computational and communication costs. In these approaches, the LLMs are frozen, and the adapters are collaboratively trained through adapter-sharing and aggregation methods. However, in this paper, we identify a significant issue: these approaches may suffer from quantization bias when clients operate with different levels of quantization on LLMs. To resolve this, we propose a novel framework called Federated Quantization-Aware LoRA (FedQLoRA), which estimates the quantization error and separates it from the LoRA adapter trained on local data via a quantization-aware adapter. Additionally, we address the heterogeneity bias problem that arises from severe data heterogeneity among clients, such as in non-IID settings. We propose an iterative version of the framework that improves both the dynamic quantization-aware adapter and the LoRA adapter alternately within the FL framework. We conduct extensive experiments to validate the performance of our proposed framework.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6966