Robust Federated Learning Under Realistic Corruption: An Iterative Filtering Approach

Jiafei Wu, Puning Zhao, Chong Wang, Zhe Liu

Abstract: Robustness is one of the critical concerns in federated learning. Existing research focuses primarily on the worst case, typically modeled as the Byzantine attack, which alters the gradients in an optimal way. However, in practice, the corruption usually happens randomly, and is much weaker than the Byzantine attack. Therefore, existing methods overestimate the power of corruption, resulting in unnecessary sacrifice of performance. In this article, we build practical algorithms that can withstand realistic corruption, which is weaker than the Byzantine attack, in a better way. Toward this goal, we propose a new iterative filtering approach. In each iteration, it calculates the geometric median of all gradient vectors uploaded from clients and remove the gradients that are far away from the geometric median. A theoretical analysis is then provided, showing that under suitable parameter regimes, gradient vectors from corrupted clients are filtered if the noise is large, while those from benign clients are never filtered throughout the training process. For realistic gradient noise, our approach significantly outperforms existing methods, while the performance under the worst-case attack (i.e., the Byzantine attack) remains nearly the same. Experiments on both synthesized and real data validate our theoretical results, as well as the practical performance of our approach. In particular, we have achieved 3%–10% increase in MNIST and CIFAR10 datasets.