Minimum Makespan Workflow Scheduling for Malleable Jobs with Precedence Constraints and Lifetime Resource Demands

Chen Chen, Xiaodi Ke, Timothy Zeyl, Kaixiang Du, Sam Sanjabi, Shane Bergsma, Reza Pournaghi, Chong Chen

Abstract: Scheduling complex workflows for big data systems is both fundamentally challenging and of great practical importance. Some state-of-the-art schedulers ignore important real-world considerations for the sake of algorithmic tractability, while others are tailored for specific workloads. We consider the preemption version of the Minimum Makespan Workflow Scheduling of Malleable Jobs with Precedence Constraints (MMWS-MP) problem [24], [27] and generalize it to be applicable to a broader range of real-world big data applications. In particular, we formulate MMWSMPL by introducing an additional constraint on the lifetime resource demand, which models constant resource consumption throughout the lifetime of a set of jobs. Practical examples include ApplicationMasters in YARN, port reservations, software licenses, and GPU cycles. We devise two scheduling strategies for MMWS-MPL: (1) LPSched, which takes a linear programming approach, and (2) BoltSched, a mostly greedy heuristic. We prove that LPSched achieves a constant approximation ratio of (2+ε) for any ε>0 and thus serves as a theoretically sound comparison baseline. We empirically evaluate both strategies on synthetic benchmarks and show that BoltSched produces schedules that are nearly as good as LPSched at a fraction (around 10%) of the computational cost.