Random Projections for Spectral Algorithms in Mis-specified Setting: Sobolev Norm Learning Rates and Minimax Optimality

Caixing Wang

Keywords: Random projections; Kernel methods; Mis-specified learning; Minimax optimality; Interpolation space

Abstract: Random projections (RP) offer an effective approach to reducing computational and storage costs while preserving the geometric structure of the data. However, existing studies primarily focus on the optimal generalization performance of specific kernel-regularized algorithms with RP in the well-specified setting under restrictive conditions. In this paper, we provide a comprehensive and improved analysis of the generalization performance of RP-based spectral algorithms under general conditions, without increasing computational complexity. By leveraging the embedding property of the RKHS and a refined analysis of the operator similarity, we establish optimal learning rates in Sobolev norms that match the minimax lower bounds up to logarithmic factors. For both randomized sketches and Nystr\"{o}m sub-sampling (uniform or leverage-based), we show that the projection dimension needed for optimality is proportional to the average or maximal effective dimension, yielding a significant reduction in computational cost while maintaining the statistical efficiency. Our results do not rely on the uniform boundedness assumption on the target function and hold for a broad range of source conditions, i.e., $s\geq \alpha-1/\beta$, where $s,\beta$, and $\alpha$ denote the smoothness index, capacity index, and the embedding index, respectively. In the benign case when $\alpha=1/\beta$, the optimality holds for all $s\in (0,2\tau]$ with $\tau$ denoting the quantification index. Experimental results confirm our theoretical findings and demonstrate the practical effectiveness of RP.

Primary Area: learning theory

Submission Number: 11812