Robust and Personalized Federated Learning with Spurious Features: an Adversarial Approach

Xiaoyang Wang, Han Zhao, Klara Nahrstedt, Oluwasanmi O Koyejo

Keywords: federated learning, personalization, spurious features

Abstract: A common approach for personalized federated learning is fine-tuning the global machine learning model to each local client. While this addresses some issues of statistical heterogeneity, we find that such personalization methods are often vulnerable to spurious features, leading to bias and diminished generalization performance. However, debiasing the personalized models under spurious features is difficult. To this end, we propose a strategy to mitigate the effect of spurious features based on our observation that the global model in the federated learning step has a low accuracy disparity due to statistical heterogeneity. Then, we estimate and mitigate the accuracy disparity of personalized models using the global model and adversarial transferability in the personalization step. Empirical results on MNIST, CelebA, and Coil20 datasets show that our method reduces the accuracy disparity of the personalized model on the bias-conflicting data samples from 15.12% to 2.15%, compared to existing personalization approaches, while preserving the benefit of enhanced average accuracy from fine-tuning.

One-sentence Summary: We propose a strategy to preventing personalized federated learning models from entangling spurious features based on the adversarial transferability between the global and personalized models.