Easing Non-IID Pain with Dual Relaxations in Federated Learning: SimFAFL redeems an enhanced efficacy
Keywords: Federated learning, Non-iid, feature alignment
TL;DR: Using fixed common feature centroid and global task parameters to constrain the center and shape of feature distribution in local training process, the effect of federated learning under Non-iid condition is improved.
Abstract: Federated learning enables collaborative training across clients without sharing local data, making it well-suited for privacy preservation. However, statistical heterogeneity in local datasets, known as non-identically distributed (non-iid) problem, leads to client drift and poor convergence for global model. Feature alignment federated learning (FAFL) methods have emerged to tackle this problem by constraining local feature distribution with global per-class representations and achieved remarkable performance. However, issues persist around 1) lacking expandability and extensibility (i.e., tight coupling with classification tasks), 2) requiring additional communication and computational cost and 3) expecting rigorous theoretical analysis. To address these issues, this paper presents a simpler version of FAFL - SimFAFL, which decouples the constantly updated FAFL constraints from explicit categorical dependencies using two modular constraints. Specifically, the proposed constraints are i) a fixed global reference distribution and ii) globally shared task parameters. These act as centroid and shape regularizers to restrict drift in local feature distributions without requiring explicit categorization. We provide theoretical analysis proving the constraints reduce the deviation upper bound of the objective function, demonstrating efficacy of SimFAFL in mitigating harmful drift. Extensive experiments demonstrate SimFAFL's state-of-the-art performance compared to prevalent methods. Moreover, the modular design also expands model flexibility and benefits generalization without imposing communication/computation costs.
Primary Area: general machine learning (i.e., none of the above)
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 1697
Loading