A Split-Client Approach to Second-Order Optimization

El Mahdi Chayti, Martin Jaggi

Keywords: Second-order optimization; Delayed Hessians; Split client

TL;DR: \textbf{TL;DR:} We propose an asynchronous split-client framework for cubic Newton methods that overlaps gradient and Hessian computations, yielding a $\sqrt{\tau}$ wall-clock speedup and outperforming Lazy Hessian baselines in practice.

Abstract: Second-order methods promise faster convergence but are rarely used in practice because Hessian computations and decompositions are significantly more expensive than gradient computations. We propose a \emph{split-client} framework where gradients and curvature are computed asynchronously by separate clients. This abstraction captures realistic delays and inexact Hessian updates while avoiding the manual tuning required by Lazy Hessian methods. Focusing on cubic regularization, we show that our approach retains strong convergence guarantees and achieves a provable wall-clock speedup of order $\sqrt{\tau}$, where $\tau$ is the relative time needed to compute and decompose the Hessian compared to a gradient step. Since $\tau$ can be orders of magnitude larger than one in high-dimensional problems, this improvement is practically significant. Empirical results on convex and nonconvex problems confirm that the asynchronous method achieves faster wall-clock convergence while naturally satisfying the bounded-delay and inexactness assumptions.

Primary Area: optimization

Submission Number: 11459