Formal Limitations on the Measurement of Mutual Information

David McAllester, Karl Stratos

Abstract: Motivated by applications to unsupervised learning, we consider the problem of measuring mutual information. Recent analysis has shown that naive kNN estimators of mutual information have serious statistical limitations motivating more refined methods. In this paper we prove that serious statistical limitations are inherent to any measurement method. More specifically, we show that any distribution-free high-confidence lower bound on mutual information cannot be larger than $O(\ln N)$ where $N$ is the size of the data sample. We also analyze the Donsker-Varadhan lower bound on KL divergence in particular and show that, when simple statistical considerations are taken into account, this bound can never produce a high-confidence value larger than $\ln N$. While large high-confidence lower bounds are impossible, in practice one can use estimators without formal guarantees. We suggest expressing mutual information as a difference of entropies and using cross entropy as an entropy estimator. We observe that, although cross entropy is only an upper bound on entropy, cross-entropy estimates converge to the true cross entropy at the rate of $1/\sqrt{N}$.

Keywords: mutual information, predictive coding, unsupervised learning, predictive learning, generalization bounds, MINE, DIM, contrastive predictive coding

TL;DR: We give a theoretical analysis of the measurement and optimization of mutual information.

Code: [ karlstratos/doe](https://github.com/karlstratos/doe) + [ 1 community implementation](https://paperswithcode.com/paper/?openreview=BkedwoC5t7)

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/formal-limitations-on-the-measurement-of/code)