Barron Spaces and the Compositional Function Spaces for Neural Network Models

Weinan E, Chao Ma, Lei Wu

Abstract:

One of the key issues in the analysis of machine learning models is to identify the appropriate function space for the model. This is the space of functions that the particular
machine learning model can approximate with good accuracy, endowed with a natural norm
associated with the approximation process. In this paper, we address this issue for two representative neural network models: the two-layer networks and the residual neural networks.
We define Barron space and show that it is the right space for two-layer neural network
models in the sense that optimal direct and inverse approximation theorems hold for functions in the Barron space. For residual neural network models, we construct the so-called
compositional function space, and prove direct and inverse approximation theorems for this
space. In addition, we show that the Rademacher complexity has the optimal upper bounds
for these spaces.