Parameter Efficient Graph Encoding for Large Language Models

Bryan Perozzi, Bahare Fatemi, Dustin Zelle, Syed A Rizvi, Anton Tsitsulin, Sami Abu-El-Haija, Hasan Awais, Mehran Kazemi, David van Dijk, Rami Al-Rfou', Jonathan Halcrow

Keywords: large language models, structured data, graph data, graph neural networks, gnns, llms, graphtoken

TL;DR: When GNNs process graph data for LLMs, it can significantly boost graph reasoning capabilities.

Abstract: How can we best encode structured data into sequential form for use in large language models (LLMs)? In this work, we introduce a parameter-efficient method to explicitly represent structured data for LLMs. Our method, GraphToken, learns an encoding function to extend prompts with explicit structured information. The encoding function in GraphToken uses graph neural networks to effectively transfer the relational inductive biases in the structured data to a LLM. Unlike other work which focuses on limited domains (e.g., knowledge graph representation), our work is the first effort focused on the general encoding of structured data to be used for various reasoning tasks. We show that explicitly representing the graph structure allows significant improvements to graph reasoning tasks. Specifically, we see across the board improvements - up to 73% points - on a wide variety of node, edge and, graph-level tasks on benchmarks for graph reasoning (GraphQA) and molecular property prediction (ChemLLMBench).

Primary Area: foundation or frontier models, including LLMs

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 11564