Since Faithfulness Fails: The Performance Limits of Neural Causal Discovery

Mateusz Olko, Mateusz Gajewski, Joanna Wojciechowska, Mikołaj Morzy, Piotr Sankowski, Piotr Miłoś

TL;DR: We show that the strong faithfulness violations limits the structure discovery accuracy of neural causal discovery method on standard benchmarks.

Abstract:

Neural causal discovery methods have recently improved in terms of scalability and computational efficiency. However, our systematic evaluation highlights significant room for improvement in their accuracy when uncovering causal structures. We identify a fundamental limitation:
\textit{unavoidable likelihood score estimation errors disallow distinguishing the true structure},
even for small graphs and relatively large sample sizes.
Furthermore, we identify the faithfulness property as a critical bottleneck: (i) it is likely to be violated across any reasonable dataset size range, and (ii) its violation directly undermines the performance of neural penalized-likelihood discovery methods. These findings lead us to conclude that progress within the current paradigm is fundamentally constrained, necessitating a paradigm shift in this domain.

Lay Summary:

Recent advances in neural causal discovery methods have improved their speed and ability to handle large datasets. However, a careful evaluation shows that these methods still struggle to identify true cause-and-effect relationships accurately. A key issue is that these approaches rely on estimating certain scores (likelihood scores), but even small errors in these estimates make it hard to recover the correct causal structure, even with relatively simple graphs and large amounts of data.

From the other perspective, the “faithfulness” assumption, underlying most of causal discovery methods, might be the bottleneck. It expects that the data clearly reflect the underlying causal relationships. In practice, this assumption often doesn’t hold, and when it’s violated, the performance of these methods drops significantly.

These issues suggest that current approaches may be fundamentally limited, and that real progress in this area may require a new way of thinking about the problem.

Primary Area: General Machine Learning->Causality

Keywords: causal discovery, faithfulness assumption, neural networks

Submission Number: 12736