On the Asymptotic Learning Curves of Kernel Ridge Regression under Power-law Decay

Yicheng Li, Haobo Zhang, Qian Lin

Keywords: generalization, reproducing kernel Hilbert space, bias-variance trade-off

Abstract:

The widely observed 'benign overfitting phenomenon' in the neural network literature raises the challenge to the `bias-variance trade-off' doctrine in the statistical learning theory.
Since the generalization ability of the 'lazy trained' over-parametrized neural network can be well approximated by that of the neural tangent kernel regression,
the curve of the excess risk (namely, the learning curve) of kernel ridge regression attracts increasing attention recently.
However, most recent arguments on the learning curve are heuristic and are based on the 'Gaussian design' assumption.
In this paper, under mild and more realistic assumptions, we rigorously provide a full characterization of the learning curve in the asymptotic sense
under a power-law decay condition of the eigenvalues of the kernel and also the target function.
The learning curve elaborates the effect and the interplay of the choice of the regularization parameter, the source condition and the noise.
In particular, our results suggest that the 'benign overfitting phenomenon' exists in over-parametrized neural networks only when the noise level is small.

Submission Number: 6258