Distributed Event-Based Learning via ADMM

Guner Dilsad ER, Sebastian Trimpe, Michael Muehlebach

TL;DR: Our distributed learning method reduces communication cost, handles non-i.i.d. data distributions, ensures fast convergence, and is robust to failures, outperforming standard baselines.

Abstract: We consider a distributed learning problem, where agents minimize a global objective function by exchanging information over a network. Our approach has two distinct features: (i) It substantially reduces communication by triggering communication only when necessary, and (ii) it is agnostic to the data-distribution among the different agents. We can therefore guarantee convergence even if the local data-distributions of the agents are arbitrarily distinct. We analyze the convergence rate of the algorithm both in convex and nonconvex settings and derive accelerated convergence rates in a convex setting. We also characterize the effect of communication failures and demonstrate that our algorithm is robust to communication failures. The article concludes by presenting numerical results from distributed learning tasks on the MNIST and CIFAR-10 datasets. The experiments underline communication savings of 35\% or more due to the event-based communication strategy, show resilience towards heterogeneous data-distributions, and highlight that our approach outperforms common baselines such as FedAvg, FedProx, SCAFFOLD and FedADMM.

Lay Summary:

In our increasingly connected world, many smart devices like phones, sensors, and computers have data spread across different locations and aim to train the best global model. However, gathering all the data into one central server is not always possible due to privacy concerns or memory limitations.

A common strategy is to let each device train its own local model and periodically share updates to form a shared global model (often to a central server). But constantly exchanging these updates across the network can be costly.

This paper introduces a more efficient alternative where devices communicate only when significant changes occur. This technique, known as event-based communication, allows devices to skip unnecessary updates while still maintaining learning accuracy. We, then, integrate this idea into a widely used optimization method, ADMM, which helps devices converge to the best model, even when each has different data.

Experiments on popular image datasets (MNIST and CIFAR-10) show that our method reduces communication cost by 35\% without losing accuracy and is robust to communication failures.

This makes our technique well-suited for large-scale, resource-constrained systems, such as networks of sensors, wearable technology, or smart city infrastructure where saving bandwidth and energy is critical.

Link To Code: https://github.com/guner-dilsad-er/Distributed-Event-Based-ADMM

Primary Area: Optimization

Keywords: distributed learning, event-based optimization, dynamical systems, communication efficiency, heterogeneous data-distribution

Submission Number: 1817