Temporal Knowledge Graph Forecasting Without Knowledge Using In-Context Learning
Submission Type: Regular Long Paper
Submission Track: NLP Applications
Submission Track 2: Theme Track: Large Language Models and the Future of NLP
Keywords: Temporal Knowledge Graph, In-context Learning, Large Language Model
Abstract: Temporal knowledge graph (TKG) forecasting benchmarks challenge models to predict future facts using knowledge of past facts.
In this paper, we develop an approach to use in-context learning (ICL) with large language models (LLMs) for TKG forecasting.
Our extensive evaluation compares diverse baselines, including both simple heuristics and state-of-the-art (SOTA) supervised models, against pre-trained LLMs across several popular benchmarks and experimental settings.
We observe that naive LLMs perform on par with SOTA models, which employ carefully designed architectures and supervised training for the forecasting task, falling within the (-3.6\%, +1.5\%) Hits@1 margin relative to the median performance.
To better understand the strengths of LLMs for forecasting, we explore different approaches for selecting historical facts, constructing prompts, controlling information propagation, and parsing outputs into a probability distribution.
A surprising finding from our experiments is that LLM performance endures ($\pm$0.4\% Hit@1) even when semantic information is removed by mapping entities/relations to arbitrary numbers, suggesting that prior semantic knowledge is unnecessary; rather, LLMs can leverage the symbolic patterns in the context to achieve such a strong performance.
Our analysis also reveals that ICL enables LLMs to learn irregular patterns from the historical context, going beyond frequency and recency biases
Submission Number: 4049
Loading