One-Shot Optimal Design for Gaussian Process Analysis of Randomized Experiments

Jelena Markovic-Voronov, Qing Feng, Eytan Bakshy

Keywords: Bayesian optimization, Randomized experiments, Gaussian processes, Meta-Learning

TL;DR: We perform meta-analysis of real randomized experiments conducted at Meta to build a model and a policy for finding the optimal experiment design for resource-efficient online randomized experiments that use Bayesian optimization..

Abstract: Bayesian optimization provides a sample-efficient approach to optimize systems that are evaluated with randomized experiments, such as Internet experiments (A/B tests) and clinical trials. Such evaluations are often resource- and time-consuming in order to measure noisy and long-term outcomes. Thus, the initial randomized design, i.e., determining the number of test groups and their sample sizes, plays a critical role in building an accurate Gaussian Process (GP) model to optimize efficiently and decreasing experimentation cost. We develop a simulation-based method with meta-learned priors to decide the optimal design for the initial batch of GP-modeled randomized experiments. The meta-learning is performed on a large corpus of randomized experiments conducted at Meta, obtaining sensible GP priors for simulating across different designs. The one-shot optimal design policy is derived by training a machine learning model with simulation data to map experiment characteristics to an optimal design. Our evaluations show that our proposed optimal design significantly improves resource-efficiency while achieving a target GP model accuracy.