Model Selection Methods for Model-Bridge Simulation Calibration

Bojan Batalo, Lincon S. Souza, Keisuke Yamazaki

Abstract: Computer simulations are ubiquitous in many scientific communities analyzing complex phenomena, such as physics, material science, medicine, and others. For simulations to yield credible insight, they must accurately represent key aspects of their real-world counterparts, making the calibration of simulation parameters crucial. Often, manual calibration is time-consuming, error-prone, and dependent on expert knowledge. Therefore, many algorithmic approaches have been explored, from heuristic-based and Bayesian methods to search and genetic algorithms. Such attempts often obtain good parameters, though at a high computational cost, as they require running the simulation many times to explore the parameter space. In contrast, the recently proposed model-bridge framework significantly speeds up this process by machine learning on a set of previous observations. In model-bridge framework, a complex simulation is represented by a simpler, uninterpretable surrogate; then, using past simulation data, a bridge model is trained to map predictions of the uninterpretable surrogate model to calibrated, interpretable simulation parameters. However, this approach introduces another problem: designing surrogate models and choosing their parameters. In this paper, we evaluate cross-validation and information theory-based model selection strategies for choosing the optimal surrogate models. Through experiments on synthetic signal and fluid dynamics simulations based on the finite element method, we show that model selection and the choice of the surrogate are essential to enabling high model-bridge performance, comparable to and surpassing established calibration approaches, at a fraction of the computational cost and time. Further, our experiments show that information theory-based methods such as Akaike information criterion (AIC) can obtain close to optimal models several orders of magnitude faster than cross-validation strategies. Finally, we discuss theoretical requirements which a surrogate model should satisfy to allow the use of information theory-based methods.