FedDebias: Reducing the Local Learning Bias Improves Federated Learning on Heterogeneous Data

Yongxin Guo, Xiaoying Tang, Tao Lin

Keywords: Federated Learning

TL;DR: We propose a unified method to reduce the local feature and classifier bias in Federated Learning.

Abstract:

Federated Learning (FL) is a machine learning paradigm that learns from data kept locally to safeguard the privacy of clients, whereas local SGD is typically employed on the clients' devices to improve communication efficiency. However, such a scheme is currently constrained by the slow and unstable convergence induced by clients' heterogeneous data.
In this work, we identify three under-explored phenomena of the biased local learning that may explain these challenges caused by local updates in supervised FL.
As a remedy, we propose FedDebias, a novel unified algorithm that reduces the local learning bias on features and classifiers to tackle these challenges.
FedDebias consists of two components:
The first component alleviates the bias in the local classifiers by balancing the output distribution of models.
The second component learns client invariant features that are close to global features but considerably distinct from those learned from other input distributions.
In a series of experiments, we show that FedDebias consistently outperforms other SOTA FL and domain generalization (DG) baselines, in which both two components have individual performance gains.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Deep Learning and representational learning