Silver Stepsize for Faster Zeroth-Order Optimization

Ahmed El Bajdali, Hualin Zhang, Peilin Zhao, Bin Gu, zhiqiang xu

Keywords: Zeroth-Order Optimization, Silver Stepsize, Gradient-Free

Abstract: We study gradient-free minimization of smooth convex functions via Silver stepsizes, a non‑monotone 2‑adic schedule that accelerates gradient descent, composed with two-point zeroth-order (ZO) estimators on a smoothed objective. We show that the multi-step Lyapunov Silver analysis carries over when exact gradients are replaced by conditionally unbiased two‑point estimators, with a stochastic tax that reduces to a quadratic variance term. We control this term under a fixed query budget by an orthogonal-on-spikes batching policy $B_t\\propto\\alpha_t$, which is budget-optimal. Empirically, we validate our approach on numerical quadratics across different conditioning regimes and MeZO-style forward-only fine‑tuning of RoBERTa‑large on GLUE tasks (SST‑2, RTE), ZO-Silver reduces evaluation loss faster than tuned constant‑LR MeZO under the same query budget.

Primary Area: optimization

Submission Number: 25582