Reasoning Over Streaming System Performance Using Response Surface Methodology

Stuart Jamieson, Matthew Forshaw

Abstract: Developing models of streaming system performance is of critical operational importance. Achieving representative coverage of potential operational conditions commonly requires large-scale experimentation which can be time-consuming and resource-intensive. Existing approaches to experimental design found in the literature typically scale poorly to the range of operating conditions required to provide robust experimental findings. This paper explores the potential of a class of approaches used commonly in other disciplines, namely Response Surface Methodology. The work presents an empirical analysis of the effectiveness of three RSM approaches, 2k Factorial, Central Composite and Box-Behnken Designs. We further evaluate the effectiveness of a suite of goodness-of-fit methods to quantify performance degradation in streaming performance, in order to understand the interplay between experimental design and choice of metric. We provide insights into combinations of experimental design and evaluation metric which we believe to be instructive to system operators. We make our implementation available to support replication efforts and support the uptake of these methods more broadly in the community.

External IDs:dblp:conf/bigdataconf/JamiesonF24