A New Class of Benchmarks for Federated Multi-Objective Learning

Maria Hartmann, Grégoire Danoy, Pascal Bouvry

Keywords: Federated Learning, Federated Multi-objective learning

TL;DR: We introduce a new challenging class of benchmarking problem for federated multi-objective learning, arguing that currently used benchmarks do not adequately represent the problem.

Abstract: Federated Learning allows effective machine learning in distribution without exposing the underlying training data. An emerging direction of research is the combination of Federated Learning with multi-objective methods, capturing the complexities of real-world problems by parameterizing training across multiple metrics of success even where such metrics conflict. The evaluation of novel methods requires suitable benchmarks. In Federated Learning, benchmarks are commonly transferred from centralised settings without modification. In this work, we show that this practice is not always sufficient: in one natural setting, where federated clients have heterogeneous preferences over multiple objectives, the most commonly used class of benchmarks can be solved easily even by baseline algorithms, in apparent contrast to the difficulty of the problem in the non-federated setting. Following this insight, we introduce a different, more challenging class of benchmarking problem, derived from the field of fair machine learning. These benchmarks are adaptable, easy to implement, permit different model architectures and different (numbers of) objectives, include a range of different well-established datasets and do not require special adaptation of the federated algorithm. Finally, we demonstrate the versatility and applicability of the proposed benchmarks in a range of configurations on state-of-the-art algorithms, showing to a range of common Federated Learning scenarios.

Primary Area: datasets and benchmarks

Submission Number: 9300