A stable model for maximizing the number of significant features

Eunkyung Park, Raymond K. Wong, Junbum Kwon, Victor W. Chu

Abstract: Identifying significant features (SFs) is important because they are driving factors of a target outcome. However, it is difficult when they have much more features than observations. The problem becomes more challenging when there are multicollinearity and infrequent common features. In such case, standard explainable methods such as OLS and Lasso often fail to identify many SF. To tackle these problems, we propose a stable model to maximize the number of SFs using selective inference called SFLasso-SI. First, in each point in the regularization path, SFLasso-SI conducts selective inference for conservative significance test. Then, it chooses the optimum value of regularization that maximizes the number of SFs. Our extensive experiments across different types of data - text, image, and video show that our SFLasso-SI can find the biggest number of SFs while maintaining similar prediction accuracy as the benchmarking methods.