Efficient and Light-Weight Federated Learning via Asynchronous Distributed Dropout

Chen Dun, Mirian Del Carmen Hipolito Garcia, Christopher Jermaine, Dimitrios Dimitriadis, Anastasios Kyrillidis, Anastasios Kyrillidis

Keywords: Efficient large scale neural network training, federated learning

TL;DR: We propose AsyncDrop, a novel asynchronous FL framework that utilizes dropout regularization to handle device heterogeneity in distributed settings.

Abstract: Asynchronous learning protocols have regained attention lately, especially in the Federated Learning (FL) setup, where slower clients can severely impede the learning process. Herein, we propose \texttt{AsyncDrop}, a novel asynchronous FL framework that utilizes dropout regularization to handle device heterogeneity in distributed settings. Overall, \texttt{AsyncDrop} achieves better performance compared to state of the art asynchronous methodologies, while resulting in less communication and training time overheads. We implement our approach and compare it against other asynchronous baselines, both by design and by adapting existing synchronous FL algorithms to asynchronous scenarios. Empirically, \texttt{AsyncDrop} reduces the communication cost and training time, while matching or improving the final test accuracy in diverse non-i.i.d. FL scenarios.