Flux: Decoupled Auto-Scaling for Heterogeneous Query Workload in Alibaba AnalyticDB

Wei Li, Jiachi Zhang, Ye Yin, Yan Li, Zhanyang Zhu, Wenchao Zhou, Liang Lin, Feifei Li

Abstract: Modern cloud data warehouses are integral to processing heterogeneous query workloads, which range from quick online transactions to intensive ad-hoc queries and extract, transform, load (ETL) processes. The synchronization of heterogeneous workloads, particularly the blend of short and long-running queries, often degrades performance due to intricate concurrency controls and cooperative multi-tasking execution models. Additionally, the auto-scaling mechanisms for mixed workloads can lead to spikes in demand and underutilized resources, impacting both performance and cost-efficiency. This paper introduces the Flux, a cloud-native workload auto-scaling platform designed for Alibaba AnalyticDB, which implements a pioneering decoupled auto-scaling architecture. By separating the scaling mechanisms for short and long-running queries, Flux not only resolves performance bottlenecks but also harnesses the elasticity of serverless container instances for on-demand resource provisioning. Our extensive evaluations demonstrate Flux's superiority over traditional scaling methods, with up to a 75% reduction in query response time (RT), a 19.0% increase in resource utilization ratio, and a 77.8% decrease in cost overhead.