Data-driven Design of Randomized Control Trials with Guaranteed Treatment Effects

Santiago Cortes-Gomez, Naveen Janaki Raman, Aarti Singh, Bryan Wilder

Abstract:

Randomized controlled trials (RCTs) generate guarantees for treatment effects. However, RCTs often spend unnecessary resources exploring sub-optimal treatments, which can reduce the power of treatment guarantees. To address this, we propose a two-stage RCT design. In the first stage, a data-driven screening procedure prunes low-impact treatments, while the second stage focuses on developing high-probability lower bounds for the best-performing treatment effect.
Unlike existing adaptive RCT frameworks, our method is simple enough to be implemented in scenarios with limited adaptivity.
We derive optimal designs for two-stage RCTs and demonstrate how such designs can be implemented through sample splitting.
Empirically, we demonstrate that two-stage designs improve upon single-stage approaches, especially for scenarios where domain knowledge is available through a prior. Our work is thus, a simple yet effective design for RCTs, optimizing for the ability to certify with high probability the largest possible treatment effect for at least one of the arms studied.

Lay Summary:

Randomized controlled trials (RCTs) are used to figure out which treatments or interventions actually work. But often, these trials waste time and resources testing options that turn out to be ineffective — which makes it harder to confidently prove when something does work.

In this paper, we introduce a new, simple two-step approach to improve RCTs. First, we run a screening step that filters out the least promising treatments using the data we’ve gathered so far. Then, we focus all remaining resources on carefully testing the most promising treatments, aiming to confidently guarantee at least one strong result.

What makes our approach special is that, unlike many existing methods, it’s practical to use even in settings where only limited adjustments can be made as the trial unfolds. We also provide mathematical tools to help design these kinds of trials in the best possible way.

When we tested our approach, we found it consistently outperformed traditional one-stage trials, especially when experts’ prior knowledge about the treatments was available. In short, our method offers a practical and powerful way to run RCTs that makes the best use of limited data and resources.

Primary Area: Social Aspects

Keywords: Experimental design, RCT, adaptive algorithms, certificate

Submission Number: 7881