ECAM: Enhancing Causal Reasoning in Foundation Models with Endogenous Causal Attention Mechanism

ICLR 2026 Conference Submission13291 Authors

Keywords: Causal Reasoning, Foundation Models, Causal Attention, Structural Causal Models, Intervention & Counterfactual Inference

TL;DR: We propose ECAM, a causal attention mechanism that integrates structural causal models into foundation models, enabling intervention and counterfactual reasoning for improved causal understanding and task performance.

Abstract: Attention mechanisms in foundation models are powerful but often capture spurious correlations rather than true causal dependencies. We propose the Endogenous Causal Attention Mechanism (ECAM), a plug-and-play module that integrates structural causal modeling into transformer-based architectures. ECAM learns Local Causal Graphs (LCGs) from data or expert priors and leverages them to modulate attention scores, enabling interventional and counterfactual reasoning within the model. Unlike prior approaches such as CausalVAE or causal regularization in pretraining, ECAM directly embeds causal graphs into the attention computation, making it task-agnostic and adaptable to both NLP and vision tasks. We provide theoretical analysis of its structural guarantees and extensive empirical results on causal reasoning benchmarks (CLUTRR, causal VQA, and synthetic data), showing that ECAM improves robustness, interpretability, and generalization. Our work highlights a novel pathway to endow foundation models with causal-awareness, offering a reusable causal reasoning layer that can serve as a building block for future causal foundation models.

Primary Area: causal reasoning

Submission Number: 13291