Redundant Information Neural Estimation

Michael Kleinman, Alessandro Achille, Stefano Soatto, Jonathan Kao

Abstract: We introduce the Redundant Information Neural Estimator (RINE), a method that allows efficient estimation for the component of information about a target variable that is common to a set of sources, previously referred to as the “redundant information.” We show that existing definitions of the redundant information can be recast in terms of an optimization over a family of deterministic or stochastic functions. In contrast to previous information decompositions, which can only be evaluated for discrete variables over small alphabets, by optimizing over functions we show empirically that we can recover the redundant information on simple benchmark tasks and that we can approximate the redundant information for high-dimensional predictors on image classification tasks, paving the way for application in different domains.