Reproducing Kernel Banach Space Models for Neural Networks with Application to Rademacher Complexity Analysis

Alistair Shilton, Sunil Gupta, Santu Rana, Svetha Venkatesh

Keywords: uniform convergence, rademacher complexity, resnet, transformer

TL;DR: We construct an exact RKBS model for neural networks with arbitrary width, depth and topology and use this model to derive tight bounds on Rademacher complexity.

Abstract: This paper explores the use of Hermite transform based reproducing kernel Banach space methods to construct exact or un-approximated models of feedforward neural networks of arbitrary width, depth and topology, including ResNet and Transformers networks, assuming only a feedforward topology, finite energy activations and finite (spectral-) norm weights and biases. Using this model, two straightforward but surprisingly tight bounds on Rademacher complexity are derived, precisely (1) a general bound that is width-independent and scales exponentially with depth; and (2) a width- and depth-independent bound for networks with appropriately constrained (below threshold) weights and biases.

Primary Area: Theory (e.g., control theory, learning theory, algorithmic game theory)

Submission Number: 16263