SP-Partitioner: A novel partition method to handle intermediate data skew in spark streaming

Guipeng Liu, Xiaomin Zhu, Ji Wang, Deke Guo, Weidong Bao, Hui Guo

Abstract: Highlights•We proposed a new architecture to address the data skew problem in Spark Streaming.•A novel Partitioner named SP-Partitioner was devised based on our proposed architecture.•Our study decreases the impacts of partitioning skew on system performance.•Extensive experimental results validated the good performance of our proposed SP-Partitioner.