Keywords: mutual information, Bayesian inference, Monte Carlo, mixture models
Abstract: Mutual information quantifies the dependence between two random variables. In this work, we explore the pointwise mutual information profile, which is the distribution of the pointwise mutual information values. We analytically describe the profiles of multivariate normal distributions and introduce a novel family of distributions, Bend and Mix Models, for which the profile can be accurately estimated using Monte Carlo methods. We then show how Bend and Mix Models can be used to study the limitations of existing mutual information estimators and understand the effect of experimental outliers on mutual information estimation. Finally, we show how Bend and Mix Models can be used to obtain model-based Bayesian estimates of mutual information, suitable for problems with available domain expertise in which uncertainty quantification is necessary.
Submission Number: 4
Loading