High-Probability Bounds for Robust Stochastic Frank-Wolfe Algorithm

Tongyi Tang, Krishna Balasubramanian, Thomas Chun Man Lee

Keywords: Stochastic Optimization, Frank Wolfe, High-Probability bound

TL;DR: We provide high-probability bounds for Stochastic Frank Wolfe Algorithm under various tail assumptions on the stochastic gradient.

Abstract: We develop and analyze robust Stochastic Frank-Wolfe type algorithms for projection-free stochastic convex optimization problems with heavy-tailed stochastic gradients. Existing works on the oracle complexity of such algorithms require a uniformly bounded variance assumption, and hold only in expectation. We develop tight high-probability bounds for robust versions of Stochastic Frank-Wolfe type algorithm under heavy-tailed assumptions, including infinite variance, on the stochastic gradient. Our methodological construction of the robust Stochastic Frank-Wolfe type algorithms leverage techniques from the robust statistic literature. Our theoretical analysis highlights the need to utilize robust versions of Stochastic Frank-Wolfe type algorithm for dealing with heavy-tailed data arising in practice.