KNIFE: Kernelized-Neural Differential Entropy Estimation

Georg Pichler, Pierre Colombo, Malik Boudiaf, Günther Koliander, Pablo Piantanida

Keywords: differential entropy estimation, differential entropy, mutual information, kernel estimation

Abstract: Estimation of (differential) entropy and the related mutual information has been pursued with significant efforts by the machine learning community. To address shortcomings in previously proposed estimators for differential entropy, here we introduce KNIFE, a fully parameterized, differentiable kernel-based estimator of differential entropy. The flexibility of our approach also allows us to construct KNIFE-based estimators for conditional (on either discrete or continuous variables) differential entropy, as well as mutual information. We empirically validate our method on high-dimensional synthetic data and further apply it to guide the training of neural networks for real-world tasks. Our experiments on a large variety of tasks, including visual domain adaptation, textual fair classification, and textual fine-tuning demonstrate the effectiveness of KNIFE-based estimation.

One-sentence Summary: We introduce and empirically evaluate KNIFE, a fully parameterized, differentiable kernel-based estimator of differential entropy.

Community Implementations: [ 6 code implementations](https://www.catalyzex.com/paper/arxiv:2202.06618/code)