Solving Structured Sparsity Regularization with Proximal Methods

Sofia Mosci, Lorenzo Rosasco, Matteo Santoro, Alessandro Verri, Silvia Villa

Abstract: Proximal methods have recently been shown to provide effective optimization procedures to solve the variational problems defining the ℓ1 regularization algorithms. The goal of the paper is twofold. First we discuss how proximal methods can be applied to solve a large class of machine learning algorithms which can be seen as extensions of ℓ1 regularization, namely structured sparsity regularization. For all these algorithms, it is possible to derive an optimization procedure which corresponds to an iterative projection algorithm. Second, we discuss the effect of a preconditioning of the optimization procedure achieved by adding a strictly convex functional to the objective function. Structured sparsity algorithms are usually based on minimizing a convex (not strictly convex) objective function and this might lead to undesired unstable behavior. We show that by perturbing the objective function by a small strictly convex term we often reduce substantially the number of required computations without affecting the prediction performance of the obtained solution.