Distributed Architecture Search Over Heterogeneous Distributions

Erum Mushtaq, Chaoyang He, Jie Ding, Salman Avestimehr

Abstract: Federated learning (FL) is an efficient learning framework that assists distributed machine learning when data cannot be shared with a centralized server. Recent advancements in FL use predefined architecture-based learning for all clients. However, given that clients' data are invisible to the server and data distributions are non-identical across clients, a predefined architecture discovered in a centralized setting may not be an optimal solution for all the clients in FL. Motivated by this challenge, we introduce SPIDER, an algorithmic framework that aims to Search PersonalIzed neural architecture for feDERated learning. SPIDER is designed based on two unique features: (1) alternately optimizing one architecture-homogeneous global model in a generic FL manner and architecture-heterogeneous local models that are connected to the global model by weight-sharing-based regularization, (2) achieving architecture-heterogeneous local models by a perturbation-based neural architecture search method. Experimental results demonstrate superior prediction performance compared with other state-of-the-art personalization methods.

Submission Length: Regular submission (no more than 12 pages of main content)

Code: https://github.com/ErumMushtaq/SPIDER.git

Assigned Action Editor: ~Sebastian_U_Stich1

License: Creative Commons Attribution 4.0 International (CC BY 4.0)

Submission Number: 1431