Feature Selection in Generalized Linear models via the Lasso: To Scale or Not to Scale?

Anant Mathur, Sarat Babu Moka, Zdravko Botev

Keywords: Lasso, standardization, feature scaling, Logistic regression

Abstract:

The Lasso regression is a popular regularization method for feature selection in statistics.
Prior to computing the Lasso estimator in both linear and generalized linear models, it is common to conduct a preliminary rescaling of the feature matrix to ensure that all the features are standardized. Without this standardization, it is argued, the Lasso estimate will unfortunately depend on the units used to measure the features. We propose a new type of iterative rescaling of the features in the context of generalized linear models. Whilst existing Lasso algorithms perform a single scaling as a preprocessing step, the proposed rescaling is applied iteratively throughout the Lasso computation until convergence. We provide numerical examples, with both real and simulated data, illustrating that the proposed iterative rescaling can significantly improve the statistical performance of the Lasso estimator without incurring any significant additional computational cost.

Submission Number: 103