When Does Causal Regularization Help? A Systematic Study of Boundary Conditions in Spurious Correlation Learning

TMLR Paper6433 Authors

Abstract:

We challenge the conventional wisdom that explicit causal regularization is necessary for out-of-distribution generalization. Through systematic investigation on ColoredMNIST, we discover that reconstructive architectures like autoencoders provide a powerful implicit causal bias that largely obviates the need for explicit methods like IRM or HSIC. Autoen-coder baselines achieve 82-86% accuracy with 99% spurious correlation, with explicit causal losses adding only marginal (0-4pp) gains.
Using the Atlasing Pattern Space (APS) framework—a modular toolkit combining topology preservation (T), causal invariance (C), and energy shaping (E)—we establish clear bound-ary conditions for when explicit regularization helps. Our experiments across multiple do-mains reveal that: (1) explicit causal methods become critical only when architectural bias is absent or spurious correlations are pathologically strong; (2) topology preservation im-proves kNN fidelity in high-dimensional vision tasks but fails completely in low-dimensional synthetic settings; and (3) energy-based regularization effectively prevents overfitting while maintaining OOD accuracy.
Through controlled experiments including a systematic study of component domain-specificity, we demonstrate that regularization components are not universally beneficial but rather require careful domain-specific validation. Our results reframe causal learning as a hierarchical process: architectural choice is primary, with explicit regularizers serving as targeted, domain-specific corrections when architectural bias proves insufficient.

Submission Type: Long submission (more than 12 pages of main content)

Previous TMLR Submission Url: https://openreview.net/forum?id=v3qhzdwbMV

Changes Since Last Submission: Reformat to adhere to TMLR style.

Assigned Action Editor: ~Lechao_Xiao2

Submission Number: 6433