A Novel Entropy and Mutual Information Measure for High Dimensional Data and Deep Neural Networks

Chen Liu

Abstract:

This is neither a research article nor a review article. This is a casual explanation of our paper that introduced the concept of Diffusion Spectral Entropy (DSE) and Diffusion Spectral Mutual Information (DSMI). Treat this as a blog. You are welcome to cite the papers listed on the next page if you find this blog helpful.

TLDR: DSE and DSMI are not intended to be unbiased estimators of the Shannon entropy and mutual information, unlike earlier works such as Mutual Information Neural Estimation (MINE) or Smoothed Mutual Information “Lower-bound” Estimator (SMILE). Instead, they serve as alternative measures based on the von Neumann formulation, which we found to be well-suited for high dimensional data and particularly descriptive for deep neural networks.