Conservative Inference in Switchback Experiments

Jose Blanchet, Peter Glynn, Ramesh Johari, Linjia Wu, Wenqian Xing

Abstract: Switchback experiments are widely used in dynamic systems---such as ridesharing platforms and online marketplaces---to evaluate interventions under interference. However, the standard pipeline of estimating the average treatment effect (ATE) with a difference-in-means (DM) estimator can exhibit systematic bias in dynamic settings with evolving system state, due to intertemporal dependence (``carryover effects"). In this paper, we study this bias in a continuous-time Markov chain model of switchback experiments with stochastically monotone dynamics and state-monotone rewards; these are reasonable representations of mean-reverting and auto-regressive systems. We show the DM estimator systematically underestimates the true ATE, because it targets an average of transient treatment effects rather than the ATE itself. Using the Ornstein-Uhlenbeck process as a tractable example, we derive closed-form expressions for bias and variance; this analysis shows that standard approaches overestimate the true variance. Taken together, these effects mean that standard switchback experiment analysis yields overly conservative inference. We validate our theory using a ride-sharing simulation with real-world calibration.

Submission Number: 1065