A framework for evaluating meta-models for simulation-based optimisation

Pepijn Van Heiningen, Bas van Stein, Thomas Bäck

Abstract: In order to evaluate complex and computationally expensive experiments, data-driven meta-models are used to replace costly experiments and approximate the real experiments' outcome. In this study, an evaluation framework is proposed for measuring the performance of these models. Explored is how the performance is related to the difference between the benchmark optimum and the model optimum, and a novel method to measure the performance of this `optima-fit' is proposed. The evaluation framework is presented using an experimental setup of four meta-modeling techniques (Decision Tree, Random Forests, Support Vector Regression and Kriging), which are systematically compared to each other. The techniques are fitted to eleven benchmarks, with dimensionality ranging from 2 to 32. The meta-models are trained with varying input sample sizes and sampling strategies. In addition, the relations between the performance to the various sampling strategies and sizes on these benchmarks are explored. Our novel designed metric can provide additional insight in the performance of a specific range of meta-models.