A Weighted Nearest Mean Classifier for Sparse Subspaces

Cor J. Veenman, David M. J. Tax

Abstract: In this paper we focus on high dimensional data sets for which the number of dimensions is an order of magnitude higher than the number of objects. From a classifier design standpoint, such small sample size problems have some interesting challenges. First, in any subspace with as many dimensions as objects the data set can be separated with an almost arbitrary linear hyperplane. Second, another important issue is to determine which features are responsible for the phenomenon under consideration. This problem comes down to finding as few features as possible that still can discriminate the classes involved. To attack these problems, we propose the LESS (lowest error in a sparse subspace) classifier. The LESS classifier is a weighted nearest mean classifier that efficiently finds linear discriminants in sparse subspaces, where the subspace is found automatically. In the experiments we compare LESS to related state-of-the-art classifiers like among others linear ridge regression with the LASSO and the support vector machine. It turns out that LESS performs competitively while it uses the fewest features.