MultiLogicNMR(er): A Benchmark and Neural-Symbolic Framework for Non-monotonic Reasoning with Multiple Extensions
Abstract: Non-monotonic reasoning (NMR) refers to the fact that conclusions may be invalidated by new information. It is widely used in daily life and legal reasoning. In recent years, some work has been done exploring the NMR abilities of LLMs. An essential concept in NMR is an extension, which can be interpreted as a set of plausible conclusions. A default theory may have multiple extensions, and there are two reasoning modes -- skeptical and credulous reasoning, depending on whether to believe facts in all extensions or any one extension. However, the multi-extension NMR capabilities of LLMs have not received adequate attention. In this paper, we synthesize a multi-extension NMR dataset MultiLogicNMR. To evaluate the generalization and robustness of the models, we construct two variants of the dataset with more extensions or text diversity. In the experimental part, we systematically evaluate the multi-extension NMR abilities of the LLMs, and the results show that LLMs still face significant challenges in such abilities. Finally, we propose a neural-symbolic framework MultiLogicNMRer for multi-extension NMR. Experimental results reveal the effectiveness of the framework, with an average accuracy gain of about 15\% compared to prompt-based models, and even outperforming some fine-tuned models. All the code and data are publicly available (https://anonymous.4open.science/r/NMRer-C5B3 ).
Paper Type: Long
Research Area: Resources and Evaluation
Research Area Keywords: benchmarking, evaluation methodologies, automatic creation and evaluation of language resources
Contribution Types: Data resources
Languages Studied: English
Submission Number: 2013
Loading