Local Reactivation for Communication Efficient Federated Learning Based on Sparse Gradient Deviation

Chang Mu, Ziyang Zhang, Xiang Tian, Kailing Guo, Xiangmin Xu

Abstract: Communication volume is a critical bottleneck restricting federated learning efficiency. Pruning is commonly employed to address this problem. To enhance the expressive ability of the pruned models, parameter reactivation which restores the training parameters of the pruned model is proposed. Compared to global server reactivation methods, local client parameter reactivation shows great potential for improvement in highly heterogeneous scenarios. However, current methods employ the same reactivation ratio across different clients, resulting in excessive reactivation in some clients and insufficient reactivation in others. We propose a local reactivation method based on sparse gradients according to the remaining parameters after pruning, using a randomly sampled mini-batch of data. The reactivation ratio for each client is determined by the deviation degree between the sparse gradients and the complete gradients. Assigning a greater reactivation ratio to a client with a greater deviation degree allow the pruned model to better approximate the complete model. The experimental results on image recognition datasets MNIST, CIFAR-10, and CIFAR-100 show our superior performance compared to the baseline method under the same uploading bandwidth constraints.