Proactive event matching with predictive analysis in content-based publish/subscribe systems

Yongpeng Dong, Shiyou Qian, Tianchen Ding, Jian Cao, Guangtao Xue, Minglu Li

Abstract: The real-time efficacy of content-based publish/subscribe systems is largely dependent on the efficiency of matching algorithms. Current methodologies mainly focus on overall matching performance, often ignoring the dynamic nature and evolving trends of hot events. This paper introduces a novel, learning-driven approach – the proactive adjustment framework (PAF) – tailored to dynamically adapt to hot event trends. By strategically prioritizing subscriptions in alignment with the changing dynamics of hot events, PAF enhances the efficiency of matching algorithms and optimize the system real-time performance. One challenge of PAF is the trade-off that needs to be made between the gains of improving real-time performance by identifying matching subscriptions earlier and the cost of increasing matching time due to subscription classification and adjustment. We design a concise scheme to classify subscriptions, establish a lightweight adjustment mechanism to handle dynamics, and propose an efficient greedy algorithm to compute adjustment plans. This approach helps to mitigate the impact of PAF on matching performance. The experiment results show that the 95th percentile of the determining time of matching subscriptions is improved by about 50.5% and the throughput is also increased by 13%, compared to the baseline SCSL. Furthermore, we integrate PAF into Apache Kafka to augment it as a content-based publish/subscribe system. We test the effectiveness of PAF using two real-world datasets. Compared with two baselines, SCSL and REIN, PAF achieves an improvement of 22.5% and 51.8% respectively in average event transfer latency.