How LLMs Address Relational Knowledge: An Empirical Evaluation

ACL ARR 2025 May Submission5404 Authors

Abstract:

Alongside the need for advanced reasoning capabilities, there is growing interest in augmenting LLMs with knowledge.
The standard approach is supervised fine-tuning; however, studies have identified the ``reversal curse'', where models trained on texts with ``A=B'' fail to infer ``B=A''.
In this study, we focus on broader cases and conduct a comprehensive evaluation of LLMs' ability to learn and generalize relational knowledge --- particularly knowledge with symmetric, antisymmetric, one-to-many, and transitive properties.
We observe a significant gap between supervised fine-tuning and in-context learning paradigms, and to address these limitations, we further propose a method that incorporates transformation noise and logical rules into the training process.
Through extensive experiments, we show that our method significantly improves the model's generalization and reasoning capabilities over such relations.
With these insights, we hope our seminal work sheds lights on the understanding of LLMs' behavior in knowledge learning and provides practical solutions to enhance their performance in real-world applications.Our code and data will be available at http.

Paper Type: Long

Research Area: Information Extraction

Research Area Keywords: Information Extraction

Contribution Types: Model analysis & interpretability, Reproduction study

Languages Studied: English

Submission Number: 5404