Are We in (A)Sync?: Guidance for Efficient Federated Learning
Primary Area: general machine learning (i.e., none of the above)
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: federated learning, synchronous federated learning, asynchronous federated learning, time-to-accuracy, resource efficiency
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
TL;DR: The paper introduces a formulation of time and resource usage of synchronous and asynchronous Federated Learning (FL), allowing practitioners with limited resource budget to make informed decisions before running FL.
Abstract: Federated Learning (FL) methods have widely adopted synchronous FL (syncFL), where a server distributes and aggregates the model weights with clients in coordinated rounds. As syncFL suffers from low resource utilization on clients with heterogeneous computing power, asynchronous FL (asyncFL), which allows the server to exchange models with available clients continuously, has been proposed. Despite numerous studies on syncFL and asyncFL, how they differ in training time and resource efficiency is still unclear. Given the training and communication speed of participating clients, we present a formulation of time and resource usage on syncFL and asyncFL. Our formulation weights asyncFL against its inefficiencies stemming from stale model updates, enabling more accurate comparison to syncFL in achieving the same objectives. Unlike previous findings, the formulation reveals that no single approach always works better than the other regarding time and resource usage. Our experiments across five datasets show that the formulation predicts relative time and resource usage of syncFL and asyncFL with up to 5.5$\times$ smaller root-mean-square error (RMSE) compared to the baseline methods. We envision our formulation to guide FL practitioners in making informed decisions between syncFL and asyncFL, depending on their resource constraints.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 3178
Loading