Representation Gap: Explaining the Unreasonable Effectiveness of Neural Networks from a Geometric Perspective

ICLR 2026 Conference Submission21817 Authors

Keywords: manifold, equivariance, generalization bounds

TL;DR: We introduce the representation gap to study neural networks generalization through a geometric perspective, deriving asymptotic bounds in the context of generative modeling and supervised prediction.

Abstract: Understanding generalization is a central issue in machine learning. Recent work has identified two key mechanisms to explain it: the strong memorization capabilities of neural networks, and the task-aligned invariants imposed by their architecture as well as training procedure. Remarkably, it is possible to characterize the neural network behavior for some classes of invariants widely used in practice. Leveraging this characterization, we introduce the representation gap, a metric that generalizes empirical risk and enables asymptotic analysis across three common settings: (i) unconditional generative modeling, where we obtain a precise asymptotic equivalent; (ii) supervised prediction; and (iii) ambiguous prediction tasks. A central outcome is that generalization is governed by a single parameter -- the intrinsic dimension of the task -- which captures task difficulty. As a corollary, we prove that popular strategies such as equivariant architectures improve performance by explicitly reducing this intrinsic dimension.

Primary Area: learning theory

Submission Number: 21817