Can Large Language Models Reason on Dynamic Graphs?

Cazamere Comrie, Jon Kleinberg

Keywords: Graph Reasoning, Graph Modification, Benchmark Dataset, Large Language Models

TL;DR: Benchmarking LLMs on Fully Dynamic Graphs

Abstract: Graphs are essential tools for modeling complex relationships, and recent work has shown that large language models (LLMs) have grown increasingly powerful at reasoning over graph-structured data. This existing work has focused primarily on static graphs that do not change over time. In any many applications, however, the underlying graph is dynamic, in that it changes over time. In this work, we address the capabilities of LLMs for reasoning on dynamic graphs, focusing on a number of challenging aspects of the problem: the fully dynamic case in which both nodes and edges can be added or deleted, and in multiple settings where the graph may be implicitly described in natural-language text or may be represented as structured data. To explore these dimensions of the problem, we introduce DyGraphQA, a new benchmark dataset for dynamic graph reasoning by LLMs. The benchmark contains prompts specifying graphs both in natural language (DyGraphQA-Real) and as structured data (DyGraphQA-Synth). We find that current LLMs struggle with dynamic graph data in both these forms, and analyze how graph structure, size, edge density, and prompting strategies impact performance, finding that each factor significantly shapes model accuracy and reasoning behavior. Our findings highlight a critical gap in current LLM capabilities regarding dynamic graph reasoning tasks and underscore the potential of techniques like MaP to mitigate these challenges.

Submission Number: 2