Probabilistic Optimization of Top-k Aggregation in Distributed Environments

Dong Hoang Van, Md Monjurul Karim, Qiang Qu

Abstract: Top-k aggregation queries are vital in distributed systems for efficiently extracting relevant information from large datasets across domains such as e-healthcare, log management, and edge computing. Although existing methods have improved query efficiency through conditional filters and reduced network traffic, they still face significant challenges in meeting the demands of modern distributed systems. These include high communication overhead and computational inefficiency. To overcome these limitations, this paper proposes the Value-Poisson Filter Top-k (VPF), a novel probabilistic filtering method that integrates Poisson sampling with a lightweight value-based filter. VPF minimizes the transmission of low-quality data, optimizing both bandwidth usage and computational costs. This approach significantly enhances resource utilization and enables efficient processing under high-load conditions. Experimental evaluations on public datasets demonstrate that VPF reduces bandwidth consumption by over 50% and achieves twice the processing speed of state-of-the-art methods while maintaining high accuracy.

External IDs:doi:10.1007/978-981-95-5716-5_35