FedorAS: Federated Architecture Search under system heterogeneity
Abstract: Federated learning (FL) has gained considerable attention due to its ability to learn on decentralised data while preserving client privacy.
However, it also poses additional challenges related to the heterogeneity of the participating devices, both in terms of their computational capabilities and contributed data. Meanwhile, Neural Architecture Search (NAS) has been successfully used with centralised datasets, producing state-of-the-art results in constrained or unconstrained settings. Such centralised datasets may not be always available for training, though. Most recent work at the intersection of NAS and FL attempts to alleviate this issue in a cross-silo federated setting, which assumes homogeneous compute environments with datacenter-grade hardware. In this paper we explore the question of whether we can design architectures of different footprints in a cross-device federated setting, where the device landscape, availability and scale are very different. To this end, we design our system, FedorAS, to discover and train promising architectures in a resource-aware manner when dealing with devices of varying capabilities holding non-IID distributed data. We present empirical evidence of its effectiveness across different settings, spanning across three different modalities (vision, speech, text), and showcase its superior
performance compared to state-of-the-art federated solutions, while maintaining resource efficiency.
Submission Length: Regular submission (no more than 12 pages of main content)
Changes Since Last Submission: n/a
Assigned Action Editor: ~Zachary_B._Charles1
Submission Number: 1978
Loading