A Computation and Communication Efficient Method for Distributed Nonconvex Problems in the Partial Participation Setting

Published: 21 Sept 2023, Last Modified: 26 Dec 2023NeurIPS 2023 posterEveryoneRevisionsBibTeX
Keywords: Nonconvex Optimization, Partial Participation, Variance Reduction, Compressed Communication, Distributed Optimization
Abstract: We present a new method that includes three key components of distributed optimization and federated learning: variance reduction of stochastic gradients, partial participation, and compressed communication. We prove that the new method has optimal oracle complexity and state-of-the-art communication complexity in the partial participation setting. Regardless of the communication compression feature, our method successfully combines variance reduction and partial participation: we get the optimal oracle complexity, never need the participation of all nodes, and do not require the bounded gradients (dissimilarity) assumption.
Supplementary Material: pdf
Submission Number: 1013