Elastic Resource Allocation Based on Dynamic Perception of Operator Influence Domain in Distributed Stream Processing

Fan Liu, Weilin Zhu, Weimin Mu, Yun Zhang, Mingyang Li, Ziyuan Zhu, Weiping Wang

Abstract: With the development of distributed stream processing systems, elastic resource allocation has become a powerful means to deal with the fluctuating data stream. The existing methods either focus on a single operator or only consider the static correlation between operators to perform elastic scaling. However, they ignore the dynamic correlation between operators in data stream processing applications, which leads to lagging and inaccuracy resource allocation, increasing processing latency. To address these issues, we propose an elastic resource allocation method, which is based on the dynamic perception of operator influence domain, to perform resource allocation dynamically and in advance. The experimental results show that compared with the existing methods, our method not only guarantees that the end-to-end latency meets QoS requirements but also reduces resource utilization.