Sparsity for Communication-Efficient LoRA

Kevin Kuo, Arian Raje, Kousik Rajesh, Virginia Smith

Keywords: federated learning, efficient finetuning, sparsity, pruning

TL;DR: We propose a simple FL baseline which combines LoRA with a constant sparsity during communication only.

Abstract: Recently, several works have used unstructured pruning to augment adapter methods. However, these ``sparse adapter'' methods have limited communication benefits in federated learning. In this work, we propose a simple baseline which combines LoRA with a constant sparsity during communication only. On three FL image and text tasks, our method reduces communication costs by up to $10\times$ over vanilla (dense) LoRA and up to $5\times$ over more complex sparse LoRA baselines. Our work highlights the importance of considering system-specific constraints when developing efficient fine-tuning approaches, and serves as a competitive baseline for future work in federated fine-tuning.

Submission Number: 80