Overcoming Data and Model heterogeneities in Decentralized Federated Learning via Synthetic Anchors

Chun-Yin Huang, Kartik Srinivas, Xin Zhang, Xiaoxiao Li

Primary Area: general machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Federated Learning, Data Heterogeneity, Model Heterogeneity

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract:

Conventional Federated Learning (FL) involves collaborative training of a global model by multiple client local models. In this emerging paradigm, the central server assumes a critical role in aggregating local models and maintaining the global model. However, it encounters various challenges, including scalability, management, and inefficiencies arising from idle client devices.
Recently, studies on serverless decentralized FL have shown advantages in overcoming these challenges, enabling clients to own different local models and separately optimize local data. Despite the promising advancements in decentralized FL, it is crucial to thoroughly investigate the implications of data and model heterogeneity, which pose unique challenges that must be overcome. Therefore, the research question to be answered in this study is: How can every client's local model learn generalizable representation?
To address this question, we propose a novel Decentralized FL technique by introducing Synthetic Anchors, dubbed as DeSA. Inspired by the theory of domain adaptation and Knowledge distillation (KD), we leverage the synthetic anchors to design two effective regularization terms for local training: 1) anchor loss that matches the distribution of the client's latent embedding with an anchor and 2) KD loss that enables clients learning from others.
In contrast to previous KD-based heterogeneous FL methods, we don’t presume access to real public or a global data generator.
DeSA enables each client's model to become robust to distribution shift across different client-domains. Through extensive experiments on diverse client data distributions, we showcase the effectiveness of \ours{} in enhancing both inter and intra-domain accuracy of each client.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 545