Personalized One-Shot Collaborative Learning

Marie Garin, Antoine de Mathelin, Mathilde Mougeot, Nicolas Vayatis

Abstract: We consider the problem of collaborative learning with non-identical inter-node distributions and unbalanced training sample sizes. This problem arises in many real-world machine-learning scenarios and is particularly difficult to address due to the specificity of data architecture. Moreover, global models produced by traditional collaborative learning approaches, as federated learning methods, often fail to provide accurate models for each node due to distribution shifts. In this work, we propose to address this problem through a personalized collaborative algorithm returning a weighted average of the other nodes’ models to each node. The personalization consists in deriving a local weighting based on the optimization of an estimate of the weighted average model risk. We provide a theoretical framework of the approach by showing that the proposed optimization leads to a water-filling optimization where the optimal weighting solves a trade-off between integrating nearby nodes and minimizing the local variance. We derive theoretical insights based on the role of local bias and local variance of each model. We test our algorithm on five datasets, including four real-world ones, and show significant improvements in terms of individual node accuracy.