Parallel Stochastic Successive Convex Approximation Method for Large-Scale Dictionary Learning

Alec Koppel, Aryan Mokhtari, Alejandro Ribeiro

Abstract: We consider the problem of dictionary learning over training sets whose sample size and parameter dimension are large-scale, which is formulated as a non-convex stochastic program where the objective decomposes into a smooth non-convex part and a convex sparsity-promoting penalty. We propose a Doubly Stochastic Successive Convex approximation scheme (DSSC) as a new numerical tool to address this task which operates by decomposing the dictionary and sparse codes into blocks and operates on random subsets of blocks at each step. The algorithm belongs to the family of successive convex approximation methods since we replace the original non-convex stochastic objective by a strongly convex sample surrogate function, and solve the resulting convex program, for each randomly selected block in parallel. The method operates on subsets of features (block coordinate methods) and training examples (stochas-tic approximation) at each step. In contrast to many training schemes for dictionary learning, DSSC attains almost sure convergence to a stationary solution of the problem. We observe the practical benefits of this approach for stable learning and computational speedup when applied to streaming visual data gathered by a field robot.