Mutual Information Estimation via Normalizing Flows

Ivan Butakov, Alexander Tolmachev, Sofia Malanchuk, Anna Neopryatnaya, Alexey Frolov

Keywords: Normalizing flows, information theory, mutual information

TL;DR: Normalizing flows are used to allow for explicit mutual information estimation via closed-form expressions

Abstract: We propose a novel approach to the problem of mutual information (MI) estimation via introducing a family of estimators based on normalizing flows. The estimator maps original data to the target distribution, for which MI is easier to estimate. We additionally explore the target distributions with known closed-form expressions for MI. Theoretical guarantees are provided to demonstrate that our approach yields MI estimates for the original data. Experiments with high-dimensional data are conducted to highlight the practical advantages of the proposed method.

Primary Area: Probabilistic methods (for example: variational inference, Gaussian processes)

Submission Number: 20910