Formulating and Unveiling In-Context Learning over Graphs

ACL ARR 2025 February Submission4358 Authors

Abstract: In-context learning (ICL) is a fascinating capability of large language models (LLMs), which can adapt to queries through demonstrations without optimizing model parameters. Although LLMs have demonstrated the ability of ICL in graph tasks, the graph in-context learning (GraphICL) mechanism is still a black box. In this paper, we introduce a novel framework for understanding and analyzing in-context learning over graphs, focusing on graph tasks, with thorough formulations, innovative mechanisms, and comprehensive benchmarks.We are the first to systematically and rigorously formalize GraphICL by explicitly defining task categories, the number of demonstrations, and graph structures in graph reasoning tasks. We reveal the mechanism of GraphICL, where the LLMs generate more accurate answers by weighting and aggregating the query representations and demonstrations representations. However, existing benchmarks lack datas with the same graph structure, which is crucial for analyzing the impact of graph structure on the GraphICL ability. We introduce two new datasets, comprising a total of 17,155 graph questions across graphs of varying sizes and multiple task categories. With these datasets, our experiments comprehensively explore for the first time how to activate GraphICL's capabilities from the perspectives of the number of demonstrations, graph structures, task categories, etc., and verify our proposed formulation and mechanism. The benchmarks and codes are available at: https://github.com/Graph-ICL/GraphICL.

Paper Type: Long

Research Area: Question Answering

Research Area Keywords: few-shot QA; generalization; reasoning; conversational QA;

Contribution Types: Data resources, Data analysis, Theory

Languages Studied: English

Submission Number: 4358