Non-Vacuous Generalization Bounds: Can Rescaling Invariances Help?

Damien Rouchouse, Antoine Gonon, Rémi Gribonval, Benjamin Guedj

Keywords: Generalization, Rescaling Invariance, ReLU, PAC-Bayes

TL;DR: We show how PAC-Bayes theory can be combined with rescaling-invariant lifted ReLU representations to derive tightened, rescaling-invariant generalization bounds for neural networks.

Abstract: A central challenge in understanding generalization is to obtain non-vacuous guarantees that go beyond worst-case complexity over data or weight space. Among existing approaches, PAC-Bayes bounds stand out as they can provide tight, data-dependent guarantees even for large networks. However, in ReLU networks, rescaling invariances mean that different weight distributions can represent the same function while leading to arbitrarily different PAC-Bayes complexities. We propose to study PAC-Bayes bounds in an invariant, lifted representation that resolves this discrepancy. This paper explores both the guarantees provided by this approach (invariance, tighter bounds via data processing) and the algorithmic aspects of KL-based rescaling-invariant PAC-Bayes bounds.

Primary Area: learning theory

Submission Number: 9481