Online Runtime Environment Prediction for Complex Colocation Interference in Distributed Streaming Processing

Fan Liu, Weilin Zhu, Weimin Mu, Yun Zhang, Mingyang Li, Can Ma, Weiping Wang

Abstract: To improve system resource utilization, multiple operators are co-located in the distributed stream processing systems. In the colocation scenarios, the node runtime environment and co-located operators affect each other. The existing methods mainly study the impact of the runtime environment on operator performance. However, there is still a lack of in-depth research on the interference of operator colocation to the runtime environment. It will lead to inaccurate prediction of the performance of the co-located operators, and further affect the effect of operator placement. To solve these problems, we propose an online runtime environment prediction method based on the operator portraits for complex colocation interference. The experimental results show that compared with the existing works, our method can not only accurately predict the runtime environment online, but also has strong scalability and continuous learning ability. It is worth noting that our method exhibits excellent online prediction performance for runtime environments in large-scale colocation scenarios.