CE-FFT: Communication-Efficient Federated Fine-Tuning for Large Language Models via Quantization and In-Context Learning

Pengyu Zhang, Yingjie Liu, Yingbo Zhou, Ming Hu, Xian Wei, Mingsong Chen

Abstract: Although Federated Fine-Tuning (FFT) facilitates the fine-tuning of Large Language Models (LLMs) across data owners without compromising their privacy, it suffers from severe communication overheads caused by numerous parameters of LLMs even with Parameter-Efficient Fine-Tuning (PEFT) methods. To address this, we propose a novel communication-efficient FFT framework by reducing per-round transmission costs and simultaneously diminishing the required convergence rounds. Specifically, we freeze the parameter-intensive backbone of LLMs, activate low-rank adapters for fine-tuning, and design a parameter quantization mechanism to reduce per-round communication costs. Additionally, inspired by in-context learning, we align the distributions of local data by identifying and rewriting unimportant samples using the global model, retaining important local information, and guiding local fine-tuning towards a uniform trajectory, thereby accelerating convergence and reducing transmission burdens. Experiments on two instruction datasets show that our method can reduce communication costs by up to 7.76× without compromising performance.

External IDs:dblp:conf/icassp/ZhangLZ0W025