OMOPredictor: An Online Multi-Step Operator Performance Prediction Framework in Distributed Streaming Processing

Weimin Mu, Zongze Jin, Fan Liu, Weilin Zhu, Weiping Wang

Abstract: Recently, with the development of distributed stream processing systems, the elastic resource scaling technique has been significantly improved. Many researchers focus on leveraging the approaches based on predicting the trend of data load to implement the elastic scaling. However, the existing predicting methods cannot track and predict the fluctuation of performance online accurately, and they need to utilize more dimensions of the raw data and resources to enhance the performance of prediction. To address these issues, we propose a framework named OMOPredictor to make an accurate prediction of operator performance online. The experimental results show that OMOPredictor can enhance the prediction of the operator performance on three real-world datasets.