Accelerated Zeroth-order Method for Non-Smooth Stochastic Convex Optimization Problem with Infinite Variance

Published: 21 Sept 2023, Last Modified: 23 Jan 2024NeurIPS 2023 posterEveryoneRevisionsBibTeX
Keywords: stochastic optimization, gradient-free optimization, zero-order oracle, gradient clipping, infinite variance
TL;DR: We propose a gradient-free method built upon an accelerated gradient method (Sadiev et al., 2023) with batching technique which is optimal with respect to iteration complexity and the maximal noise level with infinite variance.
Abstract: In this paper, we consider non-smooth stochastic convex optimization with two function evaluations per round under infinite noise variance. In the classical setting when noise has finite variance, an optimal algorithm, built upon the batched accelerated gradient method, was proposed in (Gasnikov et. al., 2022). This optimality is defined in terms of iteration and oracle complexity, as well as the maximal admissible level of adversarial noise. However, the assumption of finite variance is burdensome and it might not hold in many practical scenarios. To address this, we demonstrate how to adapt a refined clipped version of the accelerated gradient (Stochastic Similar Triangles) method from (Sadiev et al., 2023) for a two-point zero-order oracle. This adaptation entails extending the batching technique to accommodate infinite variance — a non-trivial task that stands as a distinct contribution of this paper.
Supplementary Material: pdf
Submission Number: 6976