Filter-Based Information-Theoretic Feature Selection

Farinaz Pisheh, Ricardo Vilalta

Abstract: Feature subset selection methods aim at identifying the smallest subset of features that maximize generalization performance, while preserving the true nature of the joint data distribution. In classification tasks, this is tantamount to finding an optimal subset of features relevant to the target class. A distinctive family of feature selection methods use a distance metric to identify relevant features, even under high feature interaction, by looking at the local class distribution. In this study we present EBFS: a new algorithm that is inspired by Relieff and uses an entropy-based metric to discover relevant features. Results on UCI data-sets show the effectiveness of our approach when compared to other filter-based feature selection methods.