Improving precision of A/B experiments using trigger intensity

TANMOY DAS, Dohyeon Lee, Arnab Sinha

Keywords: A/B experiment, Randomized Controlled Trial, Variance Reduction, Sensitivity, Precision, Trigger Intensity

TL;DR: We developed a more efficient way to measure the impact of A/B experiments by examining just a tiny fraction of user interactions, achieving better results than existing methods that require analyzing all data.

Abstract: Online randomized controlled experiments (A/B tests) measure causal changes in industry. While these experiments use incremental changes to minimize disruption, they often yield statistically insignificant results due to low signal-to-noise ratios. Precision improvement (or reducing standard error) traditionally focuses on trigger observations - where treatment and control outputs differ. Though effective, detecting all triggers (full knowledge) is prohibitively expensive. We propose a sampling-based approach (partial knowledge) where the bias in the evaluation outcome decreases inversely with sample size. Simulations (See Appendix C) show bias approaches zero with just ≤ 0.1% of observations sampled. Empirical testing demonstrates a 38% variance reduction compared to CUPED methods [1].

Submission Number: 12