Towards Understanding Extrapolation: a Causal Lens

Lingjing Kong, Guangyi Chen, Petar Stojanov, Haoxuan Li, Eric P. Xing, Kun Zhang

Keywords: generalization, extrapolation, identification, adaptation

TL;DR: We investigate the theoretical aspects of extrapolation under a causal model formulation.

Abstract:

Canonical work handling distribution shifts typically necessitates an entire target distribution that lands inside the training distribution.
However, practical scenarios often involve only a handful target samples, potentially lying outside the training support, which requires the capability of extrapolation.
In this work, we aim to provide a theoretical understanding of when extrapolation is possible and offer principled methods to achieve it without requiring an on-support target distribution.
To this end, we formulate the extrapolation problem with a latent-variable model that embodies the minimal change principle in causal mechanisms.
Under this formulation, we cast the extrapolation problem into a latent-variable identification problem.
We provide realistic conditions on shift properties and the estimation objectives that lead to identification even when only one off-support target sample is available, tackling the most challenging scenarios.
Our theory reveals the intricate interplay between the underlying manifold's smoothness and the shift properties.
We showcase how our theoretical results inform the design of practical adaptation algorithms. Through experiments on both synthetic and real-world data, we validate our theoretical findings and their practical implications.

Primary Area: Causal inference

Submission Number: 7667