Target alignment in truncated kernel ridge regression

Arash A Amini, richard baumgartner, Dai Feng

Keywords: kernel methods, target alignment, minimax rates, double descent

TL;DR: Truncated kernel ridge regression can improve the statistical rate of convergence when the target is strongly aligned with the kernel, beyond what is achievable by ordinary kernel ridge regression.

Abstract: Kernel ridge regression (KRR) has recently attracted renewed interest due to its potential for explaining the transient effects, such as double descent, that emerge during neural network training. In this work, we study how the alignment between the target function and the kernel affects the performance of the KRR. We focus on the truncated KRR (TKRR) which utilizes an additional parameter that controls the spectral truncation of the kernel matrix. We show that for polynomial alignment, there is an over-aligned regime, in which TKRR can achieve a faster rate than what is achievable by full KRR. The rate of TKRR can improve all the way to the parametric rate, while that of full KRR is capped at a sub-optimal value. This shows that target alignemnt can be better leveraged by utilizing spectral truncation in kernel methods. We also consider the bandlimited alignment setting and show that the regularization surface of TKRR can exhibit transient effects including multiple descent and non-monotonic behavior. Our results show that there is a strong and quantifable relation between the shape of the alignment spectrum and the generalization performance of kernel methods, both in terms of rates and in finite samples.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2206.14255/code)