SwitchFlow: Optimizing HPC Workflow Performance with Heterogeneous Serverless Frameworks

Hao Chen, Yucong Dong, Xin Wen, Zichen Xu

Abstract: High-performance serverless computing has garnered significant attention. Researchers have developed numerous optimization strategies for serverless frameworks to fully leverage the benefits of serverless computing. However, they often overlook the architectural heterogeneity of various serverless frameworks, leading to performance discrepancies. We designs SwitchFlow by analyzing and characterizing the performance of different serverless frameworks. The core idea is to utilize the time series characteristics of scientific workflows to predict upcoming functions, thereby switching them to the optimal framework for execution. Furthermore, SwitchFlow’s unique design concept enables integration with existing optimization strategies. Experimental results demonstrate that SwitchFlow can effectively reduce overall workflow execution time, optimize resource efficiency, and enhance service availability compared to running on a single serverless framework.