The Challenging Growth: Evaluating the Scalability of Causal Models

Nicholas Tagliapietra, Juergen Luettin, Lavdim Halilaj, Moritz Willig, Tim Pychynski, Kristian Kersting

Keywords: Causality, benchmark, causal discovery, causal inference

TL;DR: We test multiple causal models on two large datasets generated from a realistic physics-based simulator, built and validated by domain experts, and highlight how distant current models are from tackling those real world scenarios.

Abstract:

One of the pillars of causality is the study of causal models and understanding under which hypotheses we can guarantee their ability to grasp causal information and to leverage it for making inferences.
Real causal phenomena, however, may involve drastically different settings such as high dimensionality, causal insufficiency, and nonlinearities, which can be in stark contrast with the initial assumptions made by most models.
Additionally, providing fair benchmarks under such conditions presents challenges due to the lack of realistic data where the true data generating process is known.
Consequently, most analyses converge towards either small and synthetic toy examples or theoretical analyses, while empirical evidence is limited.
In this work, we present in-depth experimental results on two large datasets modeling a real manufacturing scenario.
We show the nontrivial behavior of a well-understood manufacturing process, simulated using a physics-based simulator built and validated by domain experts.
We demonstrate the inadequacy of many state-of-the-art models and analyze the wide differences in their performance and tractability, both in terms of runtime and memory complexity.
We observe that a wide range of causal models are computationally prohibitive for certain tasks, whereas others lack in expressiveness.
We release all artefacts to serve as reference for future research on real world applications of causality, including a general web-page and a leader-board for benchmarking.

Primary Area: causal reasoning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7347