Scalable Gaussian Process via Hilbert-Schmidt Singular Value Decomposition

Kevin Wang, Geoffery Wang, Didong Li

Keywords: Scalability, Gaussian process regression, Hilbert Schmidt singular value decomposition, compact Mat\'ern

Abstract: Gaussian process regression is widely used for its flexible mean predictions and inherent uncertainty quantification. However, its scalability is limited by cubic time complexity, $O(n^3)$, and quadratic space complexity, $O(n^2)$, making it infeasible for large-scale datasets. Although recent advances have introduced approximate methods with time complexity $O(nm^2)$, where $m\ll n$ is a tuning parameter, these methods each have their own bottlenecks, such as requiring a relatively large $m$ or involving expensive preprocessing steps. Moreover, for extremely large datasets with millions of samples, the space complexity $O(n^2)$ becomes another significant bottleneck. In this paper, we present a novel method based on the Hilbert-Schmidt singular value decomposition that obtains a low-rank decomposition ``for free", reducing both time complexity to $O(nm^2)$ and space complexity to $O(nm)$, with no preprocessing overhead. We used simulated large-scale datasets to demonstrate the performance of our method compared to state-of-the-art approaches.

Primary Area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8262