Nonvacuous Generalization Bounds For Deep Networks With Improved Size Dependence

TMLR Paper5717 Authors

Abstract: Despite being massively overparameterized, deep neural networks exhibit a remarkable ability to generalize well to unseen data. Existing generalization bounds fail to explain this phenomenon, often becoming vacuous due to their strong dependence on network depth and width. To address this, we introduce novel nonvacuous generalization bounds for deep networks, offering tighter estimates of their Rademacher complexity by introducing a new analysis of covering number, which exhibits much milder depth dependence. Our bounds grow at a much slower rate of $O(\sqrt{Dpr})$, with network depth $D$, width $p$, and weights of rank $r$, compared to previous works that scale at a rate $O(\sqrt{D^3pr})$. Moreover, under certain plausible assumptions on the width of the network, we establish bounds that grow at a sub-logarithmic rate of $O(\sqrt{log \ D})$ with depth. This novel bound is much tighter and represents a substantial improvement over prior bounds that scale at a polynomial rate with depth. We provide rigorous empirical validation, demonstrating that our bounds offer consistently tighter estimates compared to the state-of-the-art results. Thus, our bounds offer improved insight into the excellent generalization capabilities of deep overparameterized networks.

Submission Length: Long submission (more than 12 pages of main content)

Previous TMLR Submission Url: https://openreview.net/forum?id=QDdwsCWtOY

Changes Since Last Submission:

1) Changed and amended the previous assumption on the bound on the spectral norm of the layers $\|W_i\|<1$ to a strict equality $\|W_i\|=1$ to correctly derive the growth rate of the bounds based on network dimensions.
2) Eliminated the assumption that the radius of the function class is larger than that of the cover element $r>\epsilon$ in Lemma A2, which might not hold true. Updated the Lemma accordingly after removing the assumption. Derived a new, refined proof for the bound described in Theorem 3 based on these changes.

Assigned Action Editor: ~Antoine_Patrick_Isabelle_Eric_Ledent1

Submission Number: 5717