MindFlayer SGD: Efficient Parallel SGD in the Presence of Heterogeneous and Random Worker Compute Times

Arto Maranjyan, Omar Shaikh Omar, Peter Richtárik

Keywords: asynchronous optimization, parallel optimization, federated learning, distributed learning, nonconvex optimization, asynchronous methods, Random Compute Heterogeneity, time complexity, heterogeneous clients, stragglers

TL;DR: MindFlayer SGD is a novel parallel SGD method that efficiently handles heterogeneous and random worker compute times by discarding slow computations and restarting them, outperforming existing methods especially under heavy-tailed noise.

Abstract: We investigate the problem of minimizing the expectation of smooth nonconvex functions in a distributed setting with multiple parallel workers that are able to compute stochastic gradients. A significant challenge in this context is the presence of arbitrarily heterogeneous and stochastic compute times among workers, which can severely degrade the performance of existing parallel stochastic gradient descent (SGD) methods. While some parallel SGD algorithms achieve optimal performance under deterministic but heterogeneous delays, their effectiveness diminishes when compute times are random—a scenario not explicitly addressed in their design. To bridge this gap, we introduce MindFlayer SGD, a novel parallel SGD method specifically designed to handle stochastic and heterogeneous compute times. Through theoretical analysis and empirical evaluation, we demonstrate that MindFlayer SGD consistently outperforms existing baselines, particularly in environments with heavy-tailed noise. Our results highlight its robustness and scalability, making it a compelling choice for large-scale distributed learning tasks.

Readers: auai.org/UAI/2025/Conference, auai.org/UAI/2025/Conference/Area_Chairs, auai.org/UAI/2025/Conference/Reviewers, auai.org/UAI/2025/Conference/Submission807/Authors, auai.org/UAI/2025/Conference/Submission807/Reproducibility_Reviewers

Submission Number: 807