Efficient Policy Evaluation Across Multiple Different Experimental Datasets

Yonghan Jung, Alexis Bellot

Keywords: dynamic treatment regime, data fusion, off-policy evaluation, transportability

TL;DR: We provide estimators for evaluating the effect of policies leveraging data from related domains.

Abstract: Artificial intelligence systems are trained combining various observational and experimental datasets from different source sites, and are increasingly used to reason about the effectiveness of candidate policies. One common assumption in this context is that the data in source and target sites (where the candidate policy is due to be deployed) come from the same distribution. This assumption is often violated in practice, causing challenges for generalization, transportability, or external validity. Despite recent advances for determining the identifiability of the effectiveness of policies in a target domain, there are still challenges for the accurate estimation of effects from finite samples. In this paper, we develop novel graphical criteria and estimators for evaluating the effectiveness of policies (e.g., conditional, stochastic) by combining data from multiple experimental studies. Asymptotic error analysis of our estimators provides fast convergence guarantee. We empirically verified the robustness of estimators through simulations.

Primary Area: Causal inference

Submission Number: 16243