Validation of Composite Systems by Discrepancy Propagation

David Reeb, Kanil Patel, Karim Said Barsim, Martin Schiegg, Sebastian Gerwinn

Keywords: validation, covariate shift, model error, maximum mean discrepancy, convex optimization, semidefinite programming, failure probability, simulation-based validation

TL;DR: By convex relaxations of distributional discrepancy optimizations, our method quantifies how close a chain of models approximates a real-world system.

Abstract: Assessing the validity of a real-world system with respect to given quality criteria is a common yet costly task in industrial applications due to the vast number of required real-world tests. Validating such systems by means of simulation offers a promising and less expensive alternative, but requires an assessment of the simulation accuracy and therefore end-to-end measurements. Additionally, covariate shifts between simulations and actual usage can cause difficulties for estimating the reliability of such systems. In this work, we present a validation method that propagates bounds on distributional discrepancy measures through a composite system, thereby allowing us to derive an upper bound on the failure probability of the real system from potentially inaccurate simulations. Each propagation step entails an optimization problem, where -- for measures such as maximum mean discrepancy (MMD) -- we develop tight convex relaxations based on semidefinite programs. We demonstrate that our propagation method yields valid and useful bounds for composite systems exhibiting a variety of realistic effects. In particular, we show that the proposed method can successfully account for data shifts within the experimental design as well as model inaccuracies within the simulation.