Turbo Scan: Fast Sequential Nearest Neighbor Search in High Dimensions

Edgar Chávez, Eric Sadit Tellez

Abstract: This paper introduces Turbo Scan (TS), a novel k-nearest neighbor search solution tailored for high-dimensional data and specific workloads where indexing can’t be efficiently amortized over time. There exist situations where the overhead of index construction isn’t warranted given the few queries executed on the dataset. Rooted in the Johnson-Lindenstrauss (JL) lemma, our approach sidesteps the need for random rotations. To validate TS’s superiority, we offer in-depth algorithmic and experimental evaluations. Our findings highlight TS’s unique attributes and confirm its performance, surpassing sequential scans by 1.7x at perfect recall and a remarkable 2.5x at 97% recall.