FASt global convergence of gradient methods for solving regularized M-estimation

Alekh Agarwal, Sahand N. Negahban, Martin J. Wainwright

Abstract: We analyze the convergence rates of composite gradient methods for solving problems based on regularized M-estimators, working within a high-dimensional framework that allows the data dimension d to grow with (and possibly exceed) the sample size n. This high-dimensional structure precludes the usual global assumptions-namely, strong convexity and smoothness conditions-that underlie much of classical optimization analysis. We define appropriately restricted versions of these conditions, and show that they are satisfied with high probability for various statistical models. Under these conditions, our theory guarantees that composite gradient descent has a globally geometric rate of convergence up to the statistical precision of the model, meaning the typical distance between the true unknown parameter θ* and an optimal solution θ̂. This result is substantially sharper than previous results, which yielded sublinear convergence or linear convergence up to the noise level, and builds on our earlier work for constrained estimation problems. Our analysis applies to a wide range of M-estimators and statistical models, including sparse linear regression using Lasso (ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> -regularized regression); group Lasso for block sparsity; log-linear models with regularization; low-rank matrix recovery using nuclear norm regularization; and matrix decomposition. Overall, our analysis reveals interesting connections between statistical precision and computational efficiency in high-dimensional estimation.