FedSU: Communication-efficient Federated Learning with Speculative Updating

Wei Yu, Chen Chen, Qinbin Li, Jieru Zhao, Shixuan Sun, Bo Li, Minyi Guo

Abstract: Federated learning enables mobile devices to collaboratively learn a global model in iterative communication rounds. Many sparsification methods have been proposed for communication compression of FL, working by not synchronizing insignificant updates. However, we find there still exist unexploited sparsification opportunities: given the update similarity across different rounds, parameters often exhibit a linear updating pattern; motivated by speculative execution in computer architecture domain, it is promising to use the predicted gradients to refine the linearly-updating parameters without synchronization. To that end, we propose Federated Learning with Speculative Updating, or FedSU, to attain larger sparsification ratio without compromising model accuracy. In particular, to identify the linearly-updating parameters efficiently at runtime, we devise a regression-free method that diagnoses parameter linearity based on whether the second-order parameter difference is oscillating around 0. Meanwhile, to ensure convergence validity, FedSU leverages the prediction error as a feedback signal—so as to timely return to regular updating if the parameter no longer follows the linear pattern in reality. We have implemented FedSU as a Python module, and large-scale experiments in an emulated FL setup confirm that FedSU can remarkably improve the communication efficiency of FL, with a convergence speedup of over 40%.

External IDs:dblp:conf/icdcs/Yu0LZS0G25