Regularized least squares learning with heavy-tailed noise is minimax optimal

Mattes Mollenhauer, Nicole Mücke, Dimitri Meunier, Arthur Gretton

Keywords: kernel methods, nonparametric regression, kernel ridge regression

Abstract:

This paper examines the performance of ridge regression in reproducing kernel Hilbert spaces
in the presence of noise that exhibits a finite number of higher moments. We establish excess
risk bounds consisting of subgaussian and polynomial terms based on the well known integral
operator framework. The dominant subgaussian component allows to achieve convergence rates
that have previously only been derived under subexponential noise—a prevalent assumption in
related work from the last two decades. These rates are optimal under standard eigenvalue decay
conditions, demonstrating the asymptotic robustness of regularized least squares against heavy-
tailed noise. Our derivations are based on a Fuk–Nagaev inequality for Hilbert-space valued
random variables.

Primary Area: Theory (e.g., control theory, learning theory, algorithmic game theory)

Submission Number: 13430