StarShip: Mitigating I/O Bottlenecks in Serverless Computing for Scientific Workflows

Rohan Basu Roy, Devesh Tiwari

Abstract: This work highlights the significance of I/O bottlenecks that data-intensive HPC workflows face in serverless environments - an issue that has been largely overlooked by prior works. We propose StarShip, a framework that leverages different storage options and multi-tier functions to reduce I/O overhead by co-optimizing for service time and service cost. StarShip leverages the Levenberg-Marquardt optimization to find an effective solution in a large, complex search space. It outperforms existing methods with a 45% improvement in service time and a 37.6% reduction in service cost.