Nonparametric Direct Entropy Difference Estimation

Puning Zhao, Lifeng Lai

Abstract: We propose a nonparametric method to directly estimate the difference of Shannon entropy between two continuous random variables using finite number of samples. This method is based on a k-nearest-neighbor approach. We provide a finite sample analysis of the bias and variance of our proposed estimator. Numerical experiments show that our method performs better than estimating the entropy of two random variables separately. As an application of our estimator, we show that it can be used to significantly improve the performance of mutual information estimation using k-nearest-neighbor method for strongly dependent variables.