CompressStreamDB: Fine-Grained Adaptive Stream Processing without Decompression

Yu Zhang, Feng Zhang, Hourun Li, Shuhao Zhang, Xiaoyong Du

Abstract: Stream processing prevails and SQL query on streams has become one of the most popular application scenarios. For example, in 2021, the global number of active IoT endpoints reaches 12.3 billion. Unfortunately, the increasing scale of data and strict user requests place much pressure on existing stream processing systems, requiring high processing throughput with low latency. To further improve the performance of current stream processing systems, we propose a compression-based stream processing engine, called CompressStreamDB, which enables adaptive fine-grained stream processing directly on compressed streams, without decompression. Particularly, CompressStreamDB involves eight compression methods targeting various data types in streams, and it also provides a cost model for dynamically selecting the appropriate compression methods. By exploring data redundancy among streams, CompressStreamDB not only saves space in data transmission between client and server, but also achieves high throughput with low latency in SQL query on stream processing. Our experimental results show that compared to the state-of-the-art stream processing system on uncompressed streams, CompressStreamDB achieves 3.24× throughput improvement and 66.0% lower latency on average. Besides, CompressStreamDB saves 66.8% space.