Batch-Adaptive Annotations for Causal Inference with Complex-Embedded Outcomes

Ezinne Nwankwo, Lauri Goldkind, Angela Zhou

Keywords: causal inference, measurement error, adaptive data annotation, homelessness services

TL;DR: We derive closed-form optimal annotation probability minimizing ATE estimator variance with missing outcomes. We propose batch-adaptive procedure under budget constraints that improves ATE estimation. Validated on two real datasets.

Abstract: Estimating the causal effects of an intervention on outcomes is crucial to policy and decision-making. But often, information about outcomes can be missing or subject to non-standard measurement error. It may be possible to reveal ground-truth outcome information at a cost, for example, via data annotation or follow-up; but budget constraints entail that only a fraction of the dataset can be labeled. In this setting, we optimize \textit{which data points should be sampled for outcome information} and therefore efficient average treatment effect estimation with missing data. We do so by allocating data annotation in batches. We extend to settings where outcomes may be recorded in unstructured data that can be annotated at a cost, such as text or images, for example, in healthcare or social services. Our motivating application is a collaboration with a street outreach provider with millions of case notes, where it is possible to label some, but not all, ground-truth outcomes. We demonstrate how expert labels and noisy imputed labels can be combined into a doubly robust causal estimator. We run experiments on simulated data and two real-world datasets, including one on street outreach interventions in homelessness services, to show the versatility of our proposed method.

Submission Number: 217