Maximum Noise Level as Third Optimality Criterion in Black-box Optimization Problem

Aleksandr Lobanov

Abstract: This paper is devoted to the study (common in many applications) of the black-box optimization problem, where the black-box represents a gradient-free oracle $\tilde{f}_p = f(x) + \xi_p$ providing the objective function value with some stochastic noise. Assuming that the objective function is $\mu$-strongly convex, and also not just $L$-smooth, but has a higher order of smoothness ($\beta \geq 2$) we provide a novel optimization method: _Zero-Order Accelerated Batched Stochastic Gradient Descent_, whose theoretical analysis closes the question regarding the iteration complexity, _achieving optimal estimates_. Moreover, we provide a thorough analysis of the maximum noise level, and show under which condition the maximum noise level will take into account information about batch size $B$ as well as information about the smoothness order of the function $\beta$. Finally, we show the importance of considering the maximum noise level $\Delta$ as a third optimality criterion along with the standard two on the example of a numerical experiment of interest to the machine learning community, where we compare with state-of-the-art gradient-free algorithms.

Primary Area: Optimization->Zero-order and Black-box Optimization

Keywords: Black-box optimization, Higher order smoothness function, Strongly convex optimization, Maximum noise level

Submission Number: 129