A Comparison of Hamming Errors of Representative Variable Selection Methods

Tracy Ke, Longlin Wang

Keywords: Lasso, Hamming error, phase diagram, rare and weak signals, elastic net, SCAD, thresholded Lasso, forward selection, forward backward selection

Abstract: Lasso is a celebrated method for variable selection in linear models, but it faces challenges when the covariates are moderately or strongly correlated. This motivates alternative approaches such as using a non-convex penalty, adding a ridge regularization, or conducting a post-Lasso thresholding. In this paper, we compare Lasso with 5 other methods: Elastic net, SCAD, forward selection, thresholded Lasso, and forward backward selection. We measure their performances theoretically by the expected Hamming error, assuming that the regression coefficients are ${\it iid}$ drawn from a two-point mixture and that the Gram matrix is block-wise diagonal. By deriving the rates of convergence of Hamming errors and the phase diagrams, we obtain useful conclusions about the pros and cons of different methods.

One-sentence Summary: A theoretical comparison of the Hamming errors for 6 different variable selection methods