The Price of Linear Time: Error Analysis of Structured Kernel Interpolation

Alexander Moreno, Justin Xiao, Jonathan Mei

TL;DR: We establish the first rigorous error bounds for Structured Kernel Interpolation (SKI), revealing dimension-dependent trade-offs between computational complexity and approximation accuracy.

Abstract: Structured Kernel Interpolation (SKI) scales Gaussian Processes (GPs) by approximating the kernel matrix via inducing point interpolation, achieving linear computational complexity. However, it lacks rigorous theoretical error analysis. This paper bridges this gap by proving error bounds for the SKI Gram matrix and examining their effect on hyperparameter estimation and posterior inference. We further provide a practical guide to selecting the number of inducing points under convolutional cubic interpolation: they should grow as \(n^{d/3}\) for error control. Crucially, we identify two dimensionality regimes for the SKI Gram matrix spectral norm error vs. complexity trade-off. For \(d<3\), \textit{any} error tolerance can achieve linear time for sufficiently large sample size. For \(d\geq 3\), the error must \textit{increase} with sample size for our guarantees to hold. Our analysis provides key insights into SKI's scalability-accuracy trade-offs, establishing precise conditions for achieving linear-time GP inference with controlled error.

Lay Summary: We use powerful computational methods called Gaussian Processes (GPs) for tasks like prediction, but these can be slow with large datasets. A technique called Structured Kernel Interpolation (SKI) offers a way to speed them up significantly. Our research fills a crucial gap by carefully analyzing the potential errors introduced by this SKI speed-up. We provide practical guidelines on how to use SKI effectively, revealing how the amount of data and its underlying complexity (or 'dimensionality') influence the trade-off between calculation speed and prediction accuracy. For simpler data, our work shows it's possible to achieve both fast computations and high accuracy. However, for more complex, high-dimensional data, our findings indicate that to maintain speed with very large datasets, a slight increase in potential error might be an unavoidable consequence. This analysis helps researchers make more informed decisions when using SKI, allowing them to harness its speed benefits while understanding and managing the accuracy implications.

Link To Code: https://github.com/onenoc/priceoflineartime

Primary Area: Theory->Everything Else

Keywords: Gaussian processes, kernel methods

Submission Number: 3030