On the Properties and Estimation of Pointwise Mutual Information Profiles

Paweł Czyż, Frederic Grabowski, Julia E Vogt, Niko Beerenwinkel, Alexander Marx

Keywords: mutual information, Bayesian inference, Monte Carlo, mixture models

Abstract: Mutual information quantifies the dependence between two random variables. In this work, we explore the pointwise mutual information profile, which is the distribution of the pointwise mutual information values. We analytically describe the profiles of multivariate normal distributions and introduce a novel family of distributions, Bend and Mix Models, for which the profile can be accurately estimated using Monte Carlo methods. We then show how Bend and Mix Models can be used to study the limitations of existing mutual information estimators and understand the effect of experimental outliers on mutual information estimation. Finally, we show how Bend and Mix Models can be used to obtain model-based Bayesian estimates of mutual information, suitable for problems with available domain expertise in which uncertainty quantification is necessary.

Submission Number: 4