Improving Causal Event Attribution in LLMs using Cross-Questions to Validate Causal Inference Assumptions
Abstract: In this paper, we address the challenge of identifying real-world events that could have caused observed anomalies in time-series data of public indicators. Previously, this was a daunting task in a data analysis pipeline due to the open-ended nature of the answer domain. However, with the advent of modern large language models (LLMs), this task appears within reach. Our experiments on three diverse public time-series datasets shows that while LLMs can retrieve meaningful events with a single prompt, they often struggle with establishing the causal validity of these events.
To enhance causal validity, we design a set of carefully crafted cross-questions that check adherence to fundamental assumptions of causal inference in a temporal setting. The responses when combined through a simple feature-based classifier, improve the accuracy of causal event attributation from average of 65\% to 90\%. Our approach, including the questions, features, and classifier, generalizes across different datasets, serving as a meta-layer for temporal causal reasoning on event-anomaly pairs.
We release our code and three datasets, which include time-series data with tagged anomalies and corresponding real-world events.
Paper Type: Long
Research Area: NLP Applications
Research Area Keywords: commonsense reasoning, event extraction, zero/few-shot extraction, prompting, structured prediction, fact checking, financial/business NLP
Contribution Types: Model analysis & interpretability, Data resources, Data analysis
Languages Studied: English
Submission Number: 2982
Loading