Optimal Resource Provisioning Approach based on Cost Modeling for Spark Applications in Public Clouds

Jianfei Ruan, Qinghua Zheng, Bo Dong

Abstract: Efficient resource provisioning is required when running Spark applications in public clouds. However, how to optimize resource provisioning to minimize the time and/or monetary cost for a specific application remains an intractable problem since resource provisioning may differ from application to application and even be affected by the amount of input data. Existing resource settings heavily rely on random selection or previous deployer experience, frequently leading to low-quality resource provisioning. Therefore, there is an urgent need to propose an approach towards optimal resource provisioning for Spark applications in public clouds. This is a PhD proposal, where an approach based on time and monetary cost modeling is presented for cloud resource provisioning optimization under two typical constrained scenarios. The approach systematically drives resource provisioning for a specific Spark application, which may save a significant amount of time and money, compared to randomly selected settings.