Stochastic Adaptive Gradient Descent Without Descent

Jean-François Aujol, Jérémie Bigot, Camille Castera

Keywords: Stochastic Optimization, Convex Optimization, Automatic Hyperparameter tuning, Convergence guarantees

Abstract: We introduce a new adaptive step-size strategy for convex optimization with stochastic gradient that exploits the local geometry of the objective function only by means of a first-order stochastic oracle and without any hyper-parameter tuning. The method comes from a theoretically-grounded adaptation of the Adaptive Gradient Descent Without Descent method to the stochastic setting. We prove the convergence of stochastic gradient descent with our step-size under various assumptions, and we show that it empirically competes against tuned baselines.

Primary Area: optimization

Submission Number: 3438