Generalized Information Bottleneck for Gaussian Variables
Abstract: The information bottleneck (IB) method offers an attractive framework for understanding representation learning, however its applications are often limited by its computational intractability. Analytical characterization of the IB method is not only of practical interest, but it can also lead to new insights into learning phenomena. Here we consider two different generalizations of the IB problem, in which the mutual information is replaced by correlation measures based on Renyi and Jeffreys divergences, respectively. We derive an exact, analytical IB optimal linear Gaussian encoder for Gaussian correlated variables. Our analysis reveals a series of structural transitions, similar to those previously observed in the original IB case. We find further that although solving the original, Renyi and Jeffreys IB problems yields different representations in general, the structural transitions occur at the same critical tradeoff parameters, and the Renyi and Jeffreys IB solutions perform well under the original IB objective. Our results suggest that formulating the IB method with alternative correlation measures could offer a strategy for obtaining an approximate solution to the original IB problem.
Submission Length: Regular submission (no more than 12 pages of main content)
Assigned Action Editor: ~Blake_Aaron_Richards1
Submission Number: 1015
Loading