Causal Information Splitting: Engineering Proxy Features for Robustness to Distribution Shifts

Bijan Mazaheri, Atalanti A. Mastakouri, Dominik Janzing, Michaela Hardt

Keywords: Transportability, Causal inference, Dataset shift, domain generalization, node splitting, causal information splitting, auxiliary tasks, robustness, sampling bias, proxy

TL;DR: Causal perspectives to distribution shift robustness break down when we only have proxies for the causal model. Training auxiliary tasks can help separate universal relationships from domain-specific ones.

Abstract: Statistical prediction models are often trained on data that is drawn from different probability distributions than their eventual use cases. One approach to proactively prepare for these shifts harnesses the intuition that causal mechanisms should remain invariant between environments. Here we focus on a challenging setting in which the causal and anticausal variables of the target are unobserved. Leaning on information theory, we develop feature selection and engineering techniques for the observed downstream variables that act as proxies. We identify proxies that help to build stable models and moreover utilize auxiliary training tasks to extract stability-enhancing information from proxies. We demonstrate the effectiveness of our techniques on synthetic and real data.