MILE: Mutual Information LogDet Estimator

Chunlin Ji, Feng Chen, Fu Yuhao, Jun Liu

Keywords: Mutual Information, Entropy, LogDet Function, Lower and Upper bounds, Self-consistency, Self-supervised Learning

Abstract: Mutual information (MI) estimation plays an important role in representational learning. However, accurately estimating mutual information is challenging, especially for high-dimensional variables with limited batch data. In this work, we approach the mutual information estimation problem via the logdet function of data covariance. To extend the logdet function for entropy estimation of non-Gaussian variables, we assume that the data can be approximated well by a Gaussian mixture distribution and introduce a lower and upper bound for the entropy of such distributions. To deal with high dimensionality, we introduce ``ridge'' term in the logdet function to stabilize the estimation. Consequently, the mutual information can be estimated by the entropy decomposition. Our method MILE significant outperforms conventional neural network-based MI estimators in obtaining low bias and low variance MI estimation. Besides, it well pass the challenging self-consistency tests. Simulation studies also show that, beyond a better MI estimator, MILE can simultaneously gain competitive performance with SOTA MI based loss in self-supervised learning.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 2981