LoRA Fails under Non-IID Conditions: Rethinking Federated Low-Rank Adaptation

Hongyi Peng, Han Yu

Keywords: Federated Learning, Non-IID Data, Low-Rank Method, LoRA

TL;DR: LoRA fails under non-IID in federated learning; FedLoRe uses GaLore-style compression with randomized SVD and correction to improve memory, convergence, and robustness.

Abstract:

Low-Rank Adaptation (LoRA) has become a popular technique for memory-efficient fine-tuning of large models and has recently been adopted in federated learning (FL) due to its reduced parameter footprint. However, we show that LoRA significantly underperforms full-parameter fine-tuning (FFT) in FL, especially under non-IID client distributions. Our neural tangent kernel (NTK) analysis points to a simple cause: non-IID shifts diversify and misalign client gradients, increasing the effective rank (spectral energy) of the NTK / gradient-Gram matrix. Because LoRA commits to a fixed low-rank subspace, it cannot capture this additional structure; the induced kernel deviates and its spectral floor drops, leading to slower convergence and weaker generalization. Based on this finding, we argue that low-rank compression methods—such as GaLore—are inherently better suited for FL than low-rank reparameterization.
Motivated by this insight, we propose FedLore On the client side, FedLore uses a GaLore-style optimizer while replacing SVD with randomized SVD to reduce computational overhead. On the server side, FedLore estimates a shared low-rank gradient from client updates and broadcasts it to configure each client’s GaLore projector, aligning update subspaces and mitigating drift under heterogeneity. Across NLU, vision, FedLore consistently achieves higher accuracy and robustness under non-IID conditions than LoRA-based strategies, while using comparable or less memory.

Primary Area: other topics in machine learning (i.e., none of the above)

Submission Number: 25234