Path following algorithms for $\ell_2$-regularized $M$-estimation with approximation guarantee

Yunzhang Zhu, Renxiong Liu

Keywords: regularization, optimization, tuning parameter selection

TL;DR: We propose a novel grid point selection scheme and an adaptive stopping criterion for $\ell_2$ penalized $M$-estimation problem using any optimization algorithm that produces an approximated solution path with approximation error guarantee.

Abstract: Many modern machine learning algorithms are formulated as regularized M-estimation problems, in which a regularization (tuning) parameter controls a trade-off between model fit to the training data and model complexity. To select the ``best'' tuning parameter value that achieves a good trade-off, an approximated solution path needs to be computed. In practice, this is often done through selecting a grid of tuning parameter values and solving the regularized problem at the selected grid points. However, given any desired level of accuracy, it is often not clear how to choose the grid points and also how accurately one should solve the regularized problems at the selected gird points, both of which can greatly impact the overall amount of computation. In the context of $\ell_2$-regularized $M$-estimation problem, we propose a novel grid point selection scheme and an adaptive stopping criterion for any given optimization algorithm that produces an approximated solution path with approximation error guarantee. Theoretically, we prove that the proposed solution path can approximate the exact solution path to arbitrary level of accuracy, while saving the overall computation as much as possible. Numerical results also corroborate with our theoretical analysis.

Submission Number: 9422