A/B testing under Identity Fragmentation

Shiv Shankar, Ritwik Sinha, Saayan Mitra, Madalina Fiterau

Primary Area: causal reasoning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: online experimentation, causal inference, A/B testing, privacy

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: We propose a method to estimate treatment effects in A/B testing when user behaviour is split across multiple identities

Abstract: Randomized online experimentation is a key cornerstone of the online world. The infrastructure enabling such methodologies is critically dependent on user identification. However, nowadays consumers routinely interact with online businesses across multiple devices which are often recorded with different identifiers for the same consumer. The inability to match different device identities across consumers leads to an incorrect estimation of various causal effects. Moreover, without strong assumptions about the device-user graph, the causal effects are not identifiable. In this paper, we consider the task of estimating global treatment effects (GATE) from a fragmented view of exposures and outcomes. Our experiments validate our theoretical analysis, and estimators obtained through our procedure are shown be superior to standard estimators, with a lower bias and increased robustness.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 518