Scheduling algorithm based on prefetching in MapReduce clusters

Mingming Sun, Hang Zhuang, Changlong Li, Kun Lu, Xuehai Zhou

Abstract: Highlights•We explain in detail the architecture of prefetching module in Section 4.4.•We detail the framework of HPSO by example in Section 4.1.•We modify the scheduling algorithm based on prefetching to fully exploit the potential map tasks with data locality in Section 4.3.1. This method has the advantages of reducing network transmission. Furthermore, we consider part of nodes, whose remaining time is less then threshold Tunder to avoid invalid data prefetching.•We conduct a serial of experiments to evaluate performance of the proposed system using different 5 applications (Section 5).•A survey on the state-of-the-art method for improving data locality is conducted in Section 6.