On the Convergence of LoRA-Based Federated Learning: A Unified Analysis of Aggregation-Broadcast Operators

Xin Chen, Shuaijun Chen, Omid Tavallaie, Nguyen H. Tran, Shuhuang Xiang, Albert Zomaya

Keywords: federated learning, low rank adaptation, convergence analysis, fine turning

TL;DR: Theoretical convergence analysis for lora enabled distributed fine-tuning.

Abstract:

Federated Learning (FL) enables collaborative model training across decentralized data sources while preserving data privacy. However, the increasing scale of Machine Learning (ML) models poses significant communication and computation challenges in FL. Low-Rank Adaptation (LoRA) has recently been integrated into FL as a Parameter-Efficient Fine-Tuning (PEFT) strategy, substantially lowering communication costs by transmitting only a small set of trainable parameters. Nevertheless, how to aggregate LoRA-updated local models on the server remains a critical and understudied problem. This paper presents a comprehensive theoretical analysis of LoRA-based FL frameworks. We first classify existing aggregation schemes into two main categories: Sum-Product (SP) and Product-Sum (PS). We then introduce the Aggregation-Broadcast Operator (ABO) as a general class encompassing all aggregation-broadcast methods. Any method in this class ensures local or global convergence as long as the corresponding Weak or Strong Convergence Condition is satisfied. In particular, we prove that the SP and PS aggregation methods satisfy the weak and strong convergence conditions, respectively, but differ in their ability to achieve the optimal convergence rate. Moreover, we conducted extensive experiments on standard open datasets to verify our theoretical findings.

AI Acknowledgment: We acknowledge that AI tools were employed to assist in paper writing and polishing the text to improve readability.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 24862