Acceleration By Stepsize Hedging: Silver Stepsize Schedule For Smooth Convex Optimization

Jason Altschuler

Abstract: We provide a concise, self-contained proof that the Silver Stepsize Schedule proposed in our companion paper directly applies to smooth (non-strongly) convex optimization. Specifically, we show that with these stepsizes, gradient descent computes an $\eps$-minimizer in $O(\eps^{-\log_{\rho} 2}) = O(\eps^{-0.7864})$ iterations, where $\rho = 1+\sqrt{2}$ is the silver ratio. This is intermediate between the textbook unaccelerated rate $O(\eps^{-1})$ and the accelerated rate $O(\eps^{-1/2})$ due to Nesterov in 1983. The Silver Stepsize Schedule is a simple explicit fractal: the $i$-th stepsize is $1 + \rho^{\nu(i)-1}$ where $\nu(i)$ is the $2$-adic valuation of~$i$. The design and analysis are conceptually identical to the strongly convex setting in our companion paper, but simplify remarkably in this specific setting.