Dynamic Compression in Distributed Communications for Reduction of Transmission Losses

Alexander Shestakov, Ivan Yanko, Ksenya Kurinova, Aleksandr Beznosikov

Keywords: optimization, distributed optimization, communication compression

Abstract: Recent trends in machine learning demonstrate an increasing demand for computational resources, compelling the ML community to leverage multiple devices for training. This concept is realized through distributed and federated learning approaches. However, despite the numerous advantages offered by these paradigms, they suffer from a significant limitation: the necessity of frequent information exchange between devices. A common solution to this issue involves compression. Existing operator definitions account only for second-moment deviation, which does not fully reveal the changes. Also, it is not assumed to vary during the learning process, but the forwarded information may have patterns, for example decreasing from iteration number. To address this limitation, we propose several novel classes of compression operators. Additionally, we introduce dynamic data types that adapt to the nature of the transmitted data. Our comprehensive theoretical analysis demonstrates the efficiency of this approach when applied to state-of-the-art algorithms, such as EF21, DIANA and DASHA. Experimental results further validate that the reduction in compression error and accelerates the convergence.

Primary Area: optimization

Submission Number: 7960