Distributed Retraction-Free and Communication-Efficient Optimization on the Stiefel Manifold

Yilong Song, Peijin Li, Bin Gao, Kun Yuan

Abstract: Optimization problems on the Stiefel manifold, ranging from principal component analysis to enhancing neural network robustness, are ubiquitous in machine learning. The Landing algorithm avoids computationally expensive retraction operations on manifolds, making it highly competitive for large-scale problems. This paper extends this method to distributed settings, introducing *EF-Landing*, the first retraction-free and communication-efficient algorithm for distributed stochastic optimization on the Stiefel manifold. By incorporating communication compression and error feedback, EF-Landing ensures convergence and constraint feasibility while significantly reducing communication overhead. We provide sharp convergence guarantees, demonstrating that EF-Landing achieves the same asymptotic linear speedup convergence rate as existing methods without communication compression. Furthermore, our analysis is highly versatile, applying to both deterministic and stochastic settings and encompassing algorithms based on gradient descent or momentum-based gradient descent. We also generalize EF-Landing to operate on block-wise Stiefel manifolds, enabling greater flexibility for structured constraints. Extensive numerical experiments validate our theoretical results.

Lay Summary:

Many machine learning tasks involve solving optimization problems on the Stiefel manifold, which can be regarded as an orthogonal constraint for matrices. Problems like principal component analysis need this constraint, and adding extra orthogonal constraint for training neural networks can also improve their robustness. Traditional Riemannian methods for solving these problems often require costly computations to retract the iteration back onto the manifold. The Landing algorithm avoids these expensive steps, making it efficient for large-scale problems.

In this paper, we introduce *EF-Landing*, the first distributed optimization method on the Stiefel manifold that eliminates costly computations while also reducing communication between machines. With error correcting technique, EF-Landing maintains the same accuracy while significantly cutting down communication costs—a crucial advantage for large-scale distributed learning.

Our convergence analysis shows that EF-Landing converges as accurately as existing methods, even with compressed communication, and it works well in both deterministic and stochastic settings. Additionally, we extend EF-Landing to handle block-wise constraints, making it more flexible for structured problems. Experiments confirm that our method is both practical and effective.

Primary Area: Optimization->Large Scale, Parallel and Distributed

Keywords: Stiefel manifold, distributed optimization, error feedback, retraction-free

Submission Number: 14625