On the Connection between Fisher's Criterion and Shannon's Capacity: Theoretical Concepts and Implementation

Ibrahim Alsolami, Tomoki Fukai

Keywords: Feature selection, Fisher's Criterion, Shannon's Capacity, Neural Networks.

TL;DR: A feature selection scheme is developed by relating Fisher's criterion to Shannon's channel capacity.

Abstract: Fisher's criterion is arguably among the most widely used tools in machine learning for feature selection. The higher the value of Fisher's criterion, the more favorable a feature is. A rather different but nevertheless very important tool is Shannon's channel capacity. With Shannon’s capacity, one can determine the maximum rate at which information can flow across a channel. Fisher's criterion and Shannon’s capacity appear to be unrelated, yet both capture in their unique way the separation between probability distributions. In this study, we investigate whether Fisher's class separation criterion and Shannon’s capacity can be related to each other. We formulate our research problem as a binary classification task and derive analytic expressions to determine if there is a potential link between Fisher's criterion and Shannon's capacity. It turns out that Fisher's class separation criterion and Shannon’s channel capacity are intimately connected through two principal assumptions. Using this result, we develop a divergence measure for feature selection. Additionally, we show how our results can be used to solve classification problems and conduct a proof-of-concept experiment to demonstrate the viability of our approach.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: General Machine Learning (ie none of the above)