ASTRA: Understanding the Practical Impact of Robustness for Probabilistic Programs

Zixin Huang, Saikat Dutta, Sasa Misailovic

Keywords: Probabilistic Programming, Robustness, Bayesian Inference

Abstract:

We present the first systematic study of effectiveness of robustness transformations on a diverse set
of 24 probabilistic programs representing generalized linear models, mixture models, and time-series models. We evaluate five robustness transformations from literature on each model. We quantify and present insights on (1) the improvement of the posterior prediction accuracy and (2) the execution time overhead of the robustified programs, in the presence of three input noise models.

To automate the evaluation of various robustness transformations, we developed ASTRA – a novel framework for quantifying the robustness of probabilistic programs and exploring the trade-offs between robustness and execution time. Our experimental results indicate that the existing transformations are often suitable only for specific noise models, can significantly increase execution time, and have non-trivial interaction with the inference algorithms.