CausalGraphBench: a Benchmark for Evaluating Language Models capabilities of Causal Graph discovery

Nikolay Babakov, Ehud Reiter, Alberto Bugarín-Diz

Keywords: bayesian networks, causal graph discovery, large language models, causality, benchmark

TL;DR: Benchmark for Evaluating Language Models capabilities of Causal Graph discovery

Abstract:

This paper introduces CausalGraphBench, a benchmark designed to evaluate the ability of large language models (LLMs) to construct Causal Graphs (CGs), a critical component of reasoning models like Bayesian Networks. The benchmark comprises 35 CGs sourced from publicly available repositories and academic papers, each enriched with detailed metadata to facilitate systematic and consistent evaluation. We explore various LLM-driven methods for CG discovery, analyzing their performance across different graph sizes and complexity levels. Additionally, we examine the effects of data contamination on the quality of the generated CGs.

Our findings reveal that methods relying on approaches with a limited number of queries to LLM, particularly those leveraging the full graph context, consistently outperform query-intensive and exhaustive approaches, which tend to overemphasize local relationships. Across all methods, performance declines as graph size increases.

Archival Status: Archival

Paper Length: Long Paper (up to 8 pages of content)

Submission Number: 60