Deep Learning meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive?

Kaiqi Zhang, Yu-Xiang Wang

Keywords: neural network, nonparametric regression, local adaptivity

TL;DR: We proved that parallel neural networks are locally adaptive with only weight decay.

Abstract:

We study the theory of neural network (NN) from the lens of classical nonparametric regression problems with a focus on NN's ability to \emph{adaptively} estimate functions with \emph{heterogeneous smoothness} --- a property of functions in Besov or Bounded Variation (BV) classes.
Existing work on this problem requires tuning the NN architecture based on the function spaces and sample sizes.
We consider a ``Parallel NN'' variant of deep ReLU networks and show that the standard weight decay is equivalent to promoting the $\ell_p$-sparsity ($0<p<1$) of the coefficient vector of an end-to-end learned function bases, i.e., a dictionary.
Using this equivalence, we further establish that by tuning only the weight decay, such Parallel NN achieves an estimation error arbitrarily close to the minimax rates for both the Besov and BV classes.
Notably, it gets exponentially closer to minimax optimal as the NN gets deeper. Our research sheds new lights on why depth matters and how NNs are more powerful than kernel methods.