Motley: Benchmarking Heterogeneity and Personalization in Federated Learning

Shanshan Wu, Tian Li, Zachary Charles, Yu Xiao, Ken Liu, Zheng Xu, Virginia Smith

TL;DR: We explore baselines for personalized federated learning in both cross-device and cross-silo settings.

Abstract: Personalized federated learning considers learning models unique to each client in a heterogeneous network. The resulting client-specific models have been purported to improve metrics such as accuracy, fairness, and robustness in federated networks. However, despite a plethora of work in this area, it remains unclear: (1) which personalization techniques are most effective in various settings, and (2) how important personalization truly is for realistic federated applications. To better answer these questions, we propose Motley, a benchmark for personalized federated learning. Motley consists of a suite of cross-device and cross-silo federated datasets from varied problem domains, as well as thorough evaluation metrics for better understanding the possible impacts of personalization. We establish baselines on the benchmark by comparing a number of representative personalized federated learning methods. These initial results highlight strengths and weaknesses of existing approaches, and raise several open questions for the community. Motley aims to provide a reproducible means with which to advance developments in personalized and heterogeneity-aware federated learning, as well as the related areas of transfer learning, meta-learning, and multi-task learning. Code for the benchmark is open-source and available at: https://github.com/google-research/federated/tree/master/personalization_benchmark

Is Student: No

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2206.09262/code)