Motley: Benchmarking Heterogeneity and Personalization in Federated LearningDownload PDF

Published: 21 Oct 2022, Last Modified: 22 Oct 2023FL-NeurIPS 2022 PosterReaders: Everyone
TL;DR: We explore baselines for personalized federated learning in both cross-device and cross-silo settings.
Abstract: Personalized federated learning considers learning models unique to each client in a heterogeneous network. The resulting client-specific models have been purported to improve metrics such as accuracy, fairness, and robustness in federated networks. However, despite a plethora of work in this area, it remains unclear: (1) which personalization techniques are most effective in various settings, and (2) how important personalization truly is for realistic federated applications. To better answer these questions, we propose Motley, a benchmark for personalized federated learning. Motley consists of a suite of cross-device and cross-silo federated datasets from varied problem domains, as well as thorough evaluation metrics for better understanding the possible impacts of personalization. We establish baselines on the benchmark by comparing a number of representative personalized federated learning methods. These initial results highlight strengths and weaknesses of existing approaches, and raise several open questions for the community. Motley aims to provide a reproducible means with which to advance developments in personalized and heterogeneity-aware federated learning, as well as the related areas of transfer learning, meta-learning, and multi-task learning. Code for the benchmark is open-source and available at:
Is Student: No
Community Implementations: [![CatalyzeX](/images/catalyzex_icon.svg) 1 code implementation](
4 Replies