A Dynamic Fast Gaussian Transform

Baihe Huang, Zhao Song, Omri Weinstein, Hengjie Zhang, Ruizhe Zhang

Abstract: The Fast Gaussian Transform (FGT) enables subquadratic-time multiplication of an $n\times n$ Gaussian kernel matrix $\mathsf{K}_{i,j}= \exp ( - \| x_i - x_j \|_2^2 ) $ with an arbitrary vector $h \in \mathbb{R}^n$, where $x_1,\dots, x_n \in \mathbb{R}^d$ are a set of fixed source points. This kernel plays a central role in machine learning and random feature maps. Nevertheless, in most modern ML and data analysis applications, datasets are dynamically changing, and recomputing the FGT from scratch in (kernel-based) algorithms, incurs a major computational overhead ($\gtrsim n$ time for a single source update $\in \mathbb{R}^d$). These applications motivate the development of a dynamic FGT algorithm, which maintains a dynamic set of sources under kernel-density estimation (KDE) queries in sublinear time, while retaining Mat-Vec multiplication accuracy and speed. Our main result is an efficient dynamic FGT algorithm, supporting the following operations in $\log^{O(d)}(n/\epsilon)$ time: (1) Adding or deleting a source point, and (2) Estimating the "kernel-density" of a query point with respect to sources with $\epsilon$ additive accuracy. The core of the algorithm is a dynamic data structure for maintaining the "interaction rank" between source and target boxes, which we decouple into finite truncation of Taylor series and Hermite expansions.