Interactive identification of individuals with positive treatment effect while controlling false discoveries

Boyan Duan, Larry Wasserman, Aaditya Ramdas

Abstract:

Out of the participants in a randomized experiment with anticipated heterogeneous treatment
effects, is it possible to identify which ones have a positive treatment effect, even though each
has only taken either treatment or control but not both? While subgroup analysis has received
attention, claims about individual participants are more challenging. We frame the problem in
terms of multiple hypothesis testing: we think of each individual as a null hypothesis (the potential
outcomes are equal, for example) and aim to identify individuals for whom the null is false (the
treatment potential outcome stochastically dominates the control, for example). We develop a
novel algorithm that identifies such a subset, with nonasymptotic control of the false discovery
rate (FDR). Our algorithm allows for interaction — a human data scientist (or a computer program
acting on the human’s behalf) may adaptively guide the algorithm in a data-dependent manner
to gain high identification power. We also propose several extensions: (a) relaxing the null to
nonpositive effects, (b) moving from unpaired to paired samples, and (c) subgroup identification.
We demonstrate via numerical experiments and theoretical analysis that the proposed method has
valid FDR control in finite samples and reasonably high identification power.