Tiered Gossip Learning: Communication-Frugal and Scalable Collaborative Learning

ICLR 2026 Conference Submission21461 Authors

Keywords: collaborative machine learning, scalable, communication efficient, fault-tolerant, hierarchical, federated learning, peer-to-peer learning, tiered gossip learning, decentralized learning, row stochastic mixing matrix

TL;DR: Tiered Gossip Learning uses a two-layer push–gossip–pull scheme that blends federated and peer-to-peer training, delivering both fault tolerance and low communication cost.

Abstract: Modern edge deployments require collaborative training schemes that avoid both the single-server bottleneck of federated learning and the high communication burden of peer-to-peer (P2P) systems. We propose Tiered Gossip Learning (TGL), a two-layer push–gossip–pull protocol that combines the fault tolerance of P2P training with the efficiency of hierarchical aggregation. In each round, device-level leaves push their models to a randomly selected set of relays; relays gossip among themselves; and each leaf then pulls and averages models from another random subset of relays. Unlike other hierarchical schemes, TGL is fully coordinator-free, with communication and aggregation decentralized across nodes. It matches baseline accuracy with up to two-thirds fewer model exchanges, and surpasses it when exchanges are equal, across diverse datasets including CIFAR-10, FEMNIST, and AG-News. We provide convergence guarantees for TGL under standard smoothness, bounded variance and heterogeneity assumptions, and show how its layered structure enables explicit control of consensus-distance bounds. Thus, TGL brings together the strengths of FL and P2P design, enabling robust, low-cost mixing enabling large scale collaborative learning.

Primary Area: infrastructure, software libraries, hardware, systems, etc.

Submission Number: 21461