New Aspects of Black Box Conditional Gradient: Variance Reduction and One Point Feedback

Andrey Veprikov, Alexander Bogdanov, Vladislav Mikhailovich Minashkin, Aleksandr Beznosikov

Keywords: Optimization, zero-order optimization, Frank-Wolfe methods

Abstract:

This paper deals with the black-box optimization problem. In this setup, we do not have access to the gradient of the objective function, therefore, we need to estimate it somehow.
We propose a new type of approximation $\texttt{JAGUAR}$, that memorizes information from previous iterations and requires $\mathcal{O}(1)$ oracle calls.
We implement this approximation in the Frank-Wolfe and Gradient Descent algorithms and prove the convergence of these methods with different types of zero-order oracle. Our theoretical analysis covers scenarios of non-convex, convex and PL-condition cases. Also in this paper, we consider the stochastic minimization problem on the set $Q$ with noise in the zero-order oracle; this setup is quite unpopular in the literature, but we prove that the $\texttt{JAGUAR}$ approximation is robust not only in deterministic minimization problems, but also in the stochastic case. We perform experiments to compare our gradient estimator with those already known in the literature and confirm the dominance of our methods.

Submission Number: 63