Two-Stage Data Distribution for Distributed Surveillance Video Processing with Hybrid Storage Architecture

Yangyang Gao, Haitao Zhang, Bingchang Tang, Yanpei Zhu, Huadong Ma

Abstract: Distributed computing framework such asHadoop is now a prominent solution for the efficient big dataprocessing. However, the current data placement strategiesin these frameworks lack in consideration of the types ofstorage devices and the features of surveillance video tasks.So using the current distributed computing framework forlarge-scale video processing in the cloud computing clusterwith hybrid storage architecture will lead to the low utilizationof the high-performance devices and the skew of the videotask completion time. In this paper, we propose a novel Two-Stage Data Distribution Approach (TSDDA) for optimizingthe data placement in the cloud computing platform. Ourapproach adopts an accurate Processing Time Prediction Model(PTPM) to estimate the processing time of the video task byincorporating the several important video task features, theheterogeneous I/O capabilities and processing abilities in thehybrid cluster. In our approach, an initial data placementalgorithm is used to place the data needed by the videoprocessing jobs on the appropriate processing nodes in theprocess of the initial data loading, and then a dynamic datamigration algorithm is further used to migrate the video datafrom HDD to SSD for further improving the SSD’s utilizationand the processing efficiency during the process of the jobexecution. Finally, we conduct the extensive experiments, andthe experimental results verify the accuracy of PTPM and showthat the proposed TSDDA outperforms the current method