Lost-in-distance: Impact of Contextual Proximity on LLM Performance in Graph Tasks

Hamed Firooz, Maziar Sanjabi, Wenlong Jiang, Xiaoling Zhai

Track: long paper (up to 4 pages)

Keywords: Large Language Models, Graph Tasks, Long Context

TL;DR: LLMs struggle with graph tasks when relevant information is far apart in the input context

Abstract: Large Language Models (LLMs) exhibit blind spots that impair their ability to retrieve and process relevant contextual data effectively. We demonstrate that LLM performance in graph tasks with complexities beyond the "needle-in-a-haystack" scenario—where solving the problem requires cross-referencing and reasoning across multiple subproblems *jointly*—is influenced by the proximity of relevant information within the context, a phenomenon we term "lost-in-distance". We examine two fundamental graph tasks: identifying common connections between two nodes and assessing similarity among three nodes, and show that the model's performance in these tasks significantly depends on the relative positioning of common edges. We evaluate three publicly available LLMs using various graph encoding techniques that represent graph structures for LLM input. Results indicate that model accuracy can decline by up to 6x as the distance between node connections increases, independent of graph encoding and model size.

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Submission Number: 22