Maximizing link utilization with coflow-aware scheduling in datacenter networks

Jingjie Jiang, Shiyao Ma, Bo Li, Baochun Li, Jiangchuan Liu

Abstract: Link utilization has received extensive attention since datacenters become the most prevalent platform for data-parallel computing applications. A specific job of such applications involves communication among multiple machines. The coflow abstraction depicts such communication and captures application performance through corresponding network requirements. Existing techniques to improve link utilization, however, either restrict themselves to work conservation, or merely focus on flow-level metrics and ignore coflow-level performance. In this paper, we address the coflow-aware scheduling problem with the objective of maximizing link utilization. Through theoretic analyses, we formulate the coflow-aware scheduling problem as a NP-hard open shop scheduling problem with heterogeneous concurrency. Despite the hardness of this problem, we design Maluca, a hierarchical scheduling framework to conduct both inter- and intra-link scheduling. Maluca's algorithm is not only starvation-free and work-conserving, but also 2-approximate in terms of link utilization. Extensive simulation results demonstrate that Maluca outperforms both per-flow and coflow schemes in terms of link utilization, and achieves similar coflow performance in comparison with the state-of-art coflow scheduling schemes.