Role of Over-Parameterization in Generalization of 3-layer ReLU Networks

Simranjit Singh, Aditya Golatkar, Avijit Verma

Keywords: rademacher complexity, generalization, overparameterization

TL;DR: We derive a rademacher complexity bound for 3-layer ReLU networks

Abstract:

Over-parameterized neural networks defy conventional wisdom by generalizing effectively; however, standard complexity metrics like norms and margins fail to account for this. A recent work introduced a novel measure considering unit-wise capacities and provided a better explanation and tighter generalization bounds but was confined to two-layer networks. This paper extends that framework to three-layer ReLU networks. We empirically confirm the applicability of these measures and introduce a corresponding theoretical Rademacher complexity bound.

Submission Number: 169