Adjustment for Confounding using Pre-Trained Representations

Rickmer Schulte, David Rügamer, Thomas Nagler

Abstract: There is growing interest in extending average treatment effect (ATE) estimation to incorporate non-tabular data, such as images and text, which may act as sources of confounding. Neglecting these effects risks biased results and flawed scientific conclusions. However, incorporating non-tabular data necessitates sophisticated feature extractors, often in combination with ideas of transfer learning. In this work, we investigate how latent features from pre-trained neural networks can be leveraged to adjust for sources of confounding. We formalize conditions under which these latent features enable valid adjustment and statistical inference in ATE estimation, demonstrating results along the example of double machine learning. We discuss critical challenges inherent to latent feature learning and downstream parameter estimation arising from the high dimensionality and non-identifiability of representations. Common structural assumptions for obtaining fast convergence rates with additive or sparse linear models are shown to be unrealistic for latent features. We argue, however, that neural networks are largely insensitive to these issues. In particular, we show that neural networks can achieve fast convergence rates by adapting to intrinsic notions of sparsity and dimension of the learning problem.

Lay Summary:

Scientists often want to understand if a treatment or intervention genuinely works. But when analyzing data, factors not initially obvious, like images or text, could secretly affect the results. Ignoring these hidden factors, known as “confounders”, can lead to incorrect conclusions about a treatment’s real effect.

This study investigates a novel way to handle these hidden confounders using advanced methods from deep learning. It explores how features extracted from pre-trained deep learning models, such as those trained to recognize images or interpret text, can help identify and adjust for confounding factors. The paper examines under which conditions these deep learning-derived features allow for accurate estimation of treatment effects.

While the study highlights challenges that traditional statistical methods face in this context, such as the high dimensionality of deep learning-generated features, it investigates the use of neural networks, which are particularly well suited in this case. Neural networks are shown to effectively handle these complexities, producing more reliable and accurate estimates of treatment effects.

Link To Code: https://github.com/rickmer-schulte/Pretrained-Causal-Adjust

Primary Area: General Machine Learning->Causality

Keywords: ATE, Pre-trained Models, Doubly Robust Estimation, Intrinsic Dimensions

Submission Number: 6599