A Comparison of Hamming Errors of Representative Variable Selection MethodsDownload PDF

29 Sept 2021, 00:35 (edited 24 Mar 2022)ICLR 2022 PosterReaders: Everyone
  • Keywords: Lasso, Hamming error, phase diagram, rare and weak signals, elastic net, SCAD, thresholded Lasso, forward selection, forward backward selection
  • Abstract: Lasso is a celebrated method for variable selection in linear models, but it faces challenges when the covariates are moderately or strongly correlated. This motivates alternative approaches such as using a non-convex penalty, adding a ridge regularization, or conducting a post-Lasso thresholding. In this paper, we compare Lasso with 5 other methods: Elastic net, SCAD, forward selection, thresholded Lasso, and forward backward selection. We measure their performances theoretically by the expected Hamming error, assuming that the regression coefficients are ${\it iid}$ drawn from a two-point mixture and that the Gram matrix is block-wise diagonal. By deriving the rates of convergence of Hamming errors and the phase diagrams, we obtain useful conclusions about the pros and cons of different methods.
  • One-sentence Summary: A theoretical comparison of the Hamming errors for 6 different variable selection methods
  • Supplementary Material: zip
5 Replies