Geometric IB: Improving Information Bottleneck with Geometry-Aware Compression on Statistical Manifolds

ICLR 2026 Conference Submission18332 Authors

Keywords: Information bottleneck; information geometry

TL;DR: We revisit the Information Bottleneck (IB) through the lens of information geometry and propose a Geometric Information Bottleneck (G-IB) that dispenses with direct mutual information (MI) estimation.

Abstract: We revisit the Information Bottleneck (IB) through the lens of information geometry and propose a Geometric Information Bottleneck (G-IB) that dispenses with direct mutual information (MI) estimation. We show that mutual information $I(X;Z)$ and $I(Z;Y)$ admit exact projection forms as minimal Kullback–Leibler (KL) distances from the joint distributions to their respective independence manifolds. Guided by this view, G-IB controls information compression with two complementary terms: (i) a distribution-level Fisher–Rao (FR) discrepancy, which matches KL to second order and is reparameterization-invariant; and (ii) a geometry-level Jacobian–Frobenius (JF) term that provides a local capacity-type upper bound on $I_\phi(Z;X)$ by penalizing pullback volume expansion of the encoder. We further derive a natural-gradient optimizer consistent with the FR metric, proving the first-order equivalence between the geodesic update and the standard additive natural-gradient step. We conducted extensive experiments and observed that the G-IB achieves a better trade-off between prediction accuracy and compression ratio in the information plane than the mainstream IB baselines on popular datasets. G-IB offers a principled and scalable alternative that unifies distributional and geometric regularization under a single bottleneck multiplier, improving invariance and optimization stability. The source code of G-IB is released at \url{https://anonymous.4open.science/r/G-IB-0569}.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 18332