Eliminating Steady-State Oscillations in Distributed Optimization and Learning via Adaptive Stepsize

ICLR 2026 Conference Submission15329 Authors

Keywords: Distributed learning, linear convergence, adaptive stepsize

TL;DR: We propose an adaptive stepsize approach that eliminates steady-state oscillations and ensures fast convergence in distributed learning and optimization.

Abstract: Distributed stochastic optimization and learning is gaining increasing traction due to its ability to enable large-scale data processing and model training across multiple agents without the need for centralized coordination. However, existing distributed stochastic optimization and learning approaches, such as distributed SGD and their variants, generally face a dilemma in stepsize selection: a small stepsize leads to low convergence speed, whereas a large stepsize often incurs pronounced steady-state oscillations, which prevents the algorithm from achieving stable convergence accuracy. In this paper, we propose an adaptive stepsize approach for distributed stochastic optimization and learning that can eliminate steady-state oscillations and ensure fast convergence. Such guarantees are unattained by existing adaptive stepsize approaches, even in centralized optimization and learning. We prove that our proposed algorithm achieves linear convergence with respect to the iteration number, and that the convergence error decays sublinearly with the batch size of sampled data points. In the specific case in terms of deterministic distributed optimization with exact gradients accessible to agents, we prove that our proposed algorithm linearly converges to an exact optimal solution. Moreover, we quantify that the computational complexity of the proposed algorithm is on the order of $\mathcal{O}(\log(\epsilon^{-1}))$, which matches the existing results on adaptive stepsize approaches for centralized optimization and learning. Experimental results on machine learning benchmarks confirm the effectiveness of our proposed approach.

Primary Area: learning on graphs and other geometries & topologies

Submission Number: 15329