More efficient sparsity-inducing algorithms using inexact gradient

Alain Rakotomamonjy, Sokol Koço, Liva Ralaivola

Abstract: In this paper, we tackle the problem of adapting a set of classic sparsity-inducing methods to cases when the gradient of the objective function is either difficult or very expensive to compute. Our contributions are two-fold: first, we propose methodologies for computing fair estimations of inexact gradients, second we propose novel stopping criteria for computing these gradients. For each contribution we provide theoretical backgrounds and justifications. In the experimental part, we study the impact of the proposed methods for two well-known algorithms, Frank-Wolfe and Orthogonal Matching Pursuit. Results on toy datasets show that inexact gradients can be as useful as exact ones provided the appropriate stopping criterion is used.