CoBo: Collaborative Learning via Bilevel Optimization

Diba Hashemi, Lie He, Martin Jaggi

Keywords: collaborative learning, personalized federated learning, bilevel optimization, distributed learning

TL;DR: Collaborative learning through solving a bilevel optimization problem.

Abstract:

Collaborative learning is an important tool to train multiple clients more effectively by enabling communication among clients. Identifying helpful clients, however, presents challenging and often introduces significant overhead. In this paper, we model **client-selection** and **model-training** as two interconnected optimization problems, proposing a novel bilevel optimization problem for collaborative learning.
We introduce **CoBo**, a *scalable* and *elastic*, SGD-type alternating optimization algorithm that efficiently addresses these problem with theoretical convergence guarantees. Empirically, **CoBo** achieves superior performance, surpassing popular personalization algorithms by 9.3% in accuracy on a task with high heterogeneity, involving datasets distributed among 80 clients.

Primary Area: Optimization (convex and non-convex, discrete, stochastic, robust)

Submission Number: 20477