FedSAGD: federated learning with stable and accelerated client gradient descent

Tianxiang Chen, Wenjie Hou, Feng Wang, Tiantong Wang, Shaoting Tang, Zhiming Zheng, Wei Yang Bryan Lim

Keywords: federated learning;momentum-based acceleration;stability;Nesterov accelerated gradient descent;generalization performance

Abstract:

Federated Learning (FL) has become a promising paradigm for distributed machine learning. However, FL often suffers from degraded generalization performance due to the inconsistency between local and global optimization objectives and client-side overfitting. In this paper, we introduce global-update stability as an analytical tool to study generalization error and derive the stability bounds of mainstream FL optimization algorithms under non-convex settings. Our analyses reveal how the number of global update steps, data heterogeneity, and update rules influence their stability. We observe that momentum-based FL acceleration methods do not improve stability.
To address this issue, we propose FedSAGD, a new FL algorithm that leverages the global momentum acceleration mechanism and a hybrid proximal term to enhance stability. This design ensures updates follow a globally consistent descent direction while retaining the benefits of acceleration.
Theoretical analysis shows that FedSAGD achieves an advanced stability upper bound of $\mathcal{O}(1-(1-\Gamma)^T) (0 < \Gamma < 1)$ and attains a convergence rate of $\mathcal{O}(\frac{1}{\sqrt{sKT}})$ on non-i.i.d. datasets in the non-convex settings. Extensive experiments on real-world datasets demonstrate that FedSAGD significantly outperforms multiple baseline methods under standard FL settings, achieving faster convergence and state-of-the-art performance.

Primary Area: optimization

Submission Number: 16476