Adaptive Regularization and Resilient Estimation in Federated Learning

Md. Palash Uddin, Yong Xiang, Yao Zhao, Mumtaz Ali, Yushu Zhang, Longxiang Gao

Abstract: Federated Learning (FL) is an emerging research area that produces a globally trained model using numerous local users’ data and maintains their privacy. Heterogeneous or non-Independent and Identically Distributed (non-IID) data affect the global model's convergence and, therefore, cause high communication costs. These are because traditional FL approaches often disregard an adaptive regularized objective for the user-side training and utilize conventional arithmetic mean on the locally trained models for the server-side aggregation. To alleviate these issues, we propose a novel FL scheme in this paper. In particular, we propose an adaptive regularization approach to add to the classical objective function of the users’ local models during training and a resilient estimation approach to the locally trained models during aggregation. The adaptive regularization approach is derived using the users’ local and global performance diversification while the resilient estimation scheme uses a modified geometric mean aggregation over the local models’ parameters. We provide consolidated theoretical results and perform extensive experiments on the IID and non-IID settings of MNIST, CIFAR-10, and Shakespeare datasets with various deep networks. The results manifest that our FL scheme outperforms the state-of-the-art approaches in terms of communication speedup, test-set performance, training convergence stability, and resiliency against attacks.