Unit Selection: Case Study and Comparison with A/B Test Heuristic

Anonymous

Keywords: Unit selection, A/B test, RCT, Counterfactual

TL;DR: This study explains the relationships between the benefit function in the unit selection problem defined by Li and Pearl and the A/B test heuristics.

Abstract: The unit selection problem aims to identify a set of individuals who are most likely to exhibit a desired mode of behavior, for example, selecting individuals who would respond one way if encouraged and a different way if not encouraged. Using a combination of experimental and observational data, Li and Pearl derived tight bounds on the ``benefit function'', which is the payoff/cost associated with selecting an individual with given characteristics. In this study, we explain the relationships between the benefit function and the A/B test heuristics by analyzing the essence of the latter one. We further show the A/B test heuristics failed in some common real-world scenarios using simulated case studies.