A Rate--Distortion View on Model Updates

Nicole Elyse Mitchell, Johannes Ballé, Zachary Charles, Jakub Konečný

Keywords: Federated Learning, Information Theory, Compression

TL;DR: We frame communication of model updates in federated learning as a rate-distortion optimization problem, and present an empirically justified compression algorithm with superior rate-distortion performance compared to existing approaches.

Abstract: Compressing model updates is critical for reducing communication costs in federated learning. We examine the problem using rate--distortion theory to present a compression method that is near-optimal in many use cases. We empirically show that common transforms applied to model updates in standard compression algorithms, normalization in QSGD and random rotation in DRIVE, yield sub-optimal compressed representations in practice.