Geometric and Information Compression of Representations in Deep Learning

ICLR 2026 Conference Submission12628 Authors

Keywords: information theory, neural collapse, representation learning, generalization

TL;DR: Low MI between inputs and representations does not necessarily imply geometric compression. Their relationship is subtle, with generalization possibly acting as a confounder.

Abstract:

Deep neural networks transform input data into latent representations that support a
wide range of downstream tasks. These representations can be characterized along
information-theoretic and geometric dimensions, but their relationship remains
poorly understood. A central open question is whether low mutual information (MI)
between inputs and representations necessarily implies geometrically compressed
latent spaces and vice versa. We investigate this question using neural collapse
as a measure of geometric compression and theoretically sound MI estimation in
conditional entropy bottleneck (CEB) networks and continuous dropout networks.
We evaluate the interplay between MI, geometric compression, and generalization
on classification tasks under controlled noise injection schemes. Our findings show
that low MI does not reliably correspond to geometric compression, and that the
connection between the two is more nuanced than often assumed. We conjecture
that generalization acts as a potential confounder in this connection rather than
being a direct consequence.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Submission Number: 12628