Analysis of Schedule Free Non-Convex Optimization

Connor Brown, Ahmed Khaled, Chi Jin

Keywords: schedule free learning, gradient descent, lyapunov, convergence analysis, nonconvex analysis

TL;DR: A novel non-convex analysis of a schedule-free gradient descent algorithm showing near optimal rates without assuming fixed time horizon.

Abstract: The Schedule-Free (SF) method promises optimal performance with hyper-parameters that are independent of a known training time horizon $T$. Nevertheless, non-convex analysis of SF has been limited or reliant on strong global assumptions. Under minimal assumptions of smoothness lower-boundedness, and bounded variance assumptions, we introduce a robust Lyapunov framework for analyzing SF in the non-convex setting, yielding a family of horizon-agnostic $\mathcal{O}(1/\log T)$, $\mathcal{O}(1/T)$, and $\mathcal{O} \bigl(T^{-(1-\alpha)}\bigr)$ rates under a variety of conditions. Our results --- complemented by Performance Estimation Problem (PEP) experiments --- extend SF’s horizon-free guarantees to smooth non-convex optimization and charts future directions for optimal non-convex rates.

Submission Number: 49