On Incorporating Prior Knowledge Extracted from Pre-trained Language Models into Causal Discovery

Chanhui Lee, Juhyeon Kim, YongJun Jeong, Yoonseok Yeom, Juhyun Lyu, Jung-Hee Kim, Sangmin Lee, Sangjun Han, Hyeokjun Choe, Soyeon Park, Woohyung Lim, Kyunghoon Bae, Sungbin Lim, Sanghack Lee

Keywords: Causal discovery; Pre-trained language model; Time-series

TL;DR: We propose a new framework for causal discovery on static and time-series dataset, by integrating PLMs-based causal reasoning into data-driven causal discovery.

Abstract:

Pre-trained Language Models (PLMs) can reason about causality by leveraging vast pre-trained knowledge and text descriptions of datasets, proving their effectiveness even when data is scarce.
However, there are crucial limitations in current PLM-based causal reasoning methods: i) PLM cannot utilize large datasets in prompt due to the limits of context length, and ii) the methods are not adept at comprehending the whole interconnected causal structures.
On the other hand, data-driven causal discovery can discover the causal structure as a whole, although it works well only when the number of data observations is sufficiently large enough.
To overcome each other approaches' limitations, we propose a new framework that integrates PLMs-based causal reasoning into data-driven causal discovery, resulting in improved and robust performance.
Furthermore, our framework extends to the time-series data and exhibits superior performance.

Submission Number: 33