Learnable Spatial-Temporal Positional Encoding for Link Prediction

Katherine Tieu, Dongqi Fu, Zihao Li, Ross Maciejewski, Jingrui He

Abstract:

Accurate predictions rely on the expressiveness power of graph deep learning frameworks like graph neural networks and graph transformers, where a positional encoding mechanism has become much more indispensable in recent state-of-the-art (SOTA) works to record the canonical position information. However, the current positional encoding limits in three aspects, at least: (1) most positional encodings are pre-defined, and fixed functions, which are inadequate to adapt to the complex attributed graphs; (2) a few pioneering works propose the learnable positional encoding but still limited to the structural information, leaving the real-world time-evolving topological and feature information untouched; (3) most positional encodings should be equipped with transformer's attention mechanism to fully release the power, where the dense or relational attention is often unaffordable on large-scale structured data.
Hence, we study the possibility of Learnable Spatial-Temporal Positional Encoding in an effective and efficient manner and then propose a simple temporal link prediction model named L-STEP. Briefly, for L-STEP, we (1) prove the proposed positional learning scheme can preserve the graph property from the spatial-temporal spectral viewpoint, (2) verify that MLPs can fully exploit the expressiveness and reach Transformers' performance on that encoding, (3) change different initial positional encoding inputs to show robustness, (4) analyze the theoretical complexity and obtain less empirical running time than SOTA, and (5) demonstrate its temporal link prediction out-performance on 13 classic datasets and with 10 algorithms in both transductive and inductive settings using 3 different sampling strategies. Also, L-STEP obtains the leading performance in the newest large-scale TGB benchmark.

Lay Summary:

Accurate predictions on complex data structures like graphs, representing relationships in networks such as social platforms, or biological processes, depend on how well models can understand the relative positions and connections between nodes. Modern graph learning frameworks rely on positional encoding to capture this structural information. However, current methods fall short: they often rely on fixed formulas, overlook how graphs change over time, and are computationally expensive, especially for large datasets.

To address these limitations, we developed L-STEP, a simple yet effective model for temporal link prediction. L-STEP introduces a learnable way to encode structural and time-evolving information. We show that this method preserves key graph properties from a spectral perspective, and surprisingly, simple Multi-Layer Perceptrons (MLPs) can match the performance of more complex Transformer models when using our encoding.

Our work bridges the gap between accuracy and scalability in graph learning. By providing a learnable and efficient way to understand how graph structures evolve over time, L-STEP can lead to more reliable predictions in real-world scenarios, whether forecasting traffic patterns, or understanding evolving relationships in real-world data. Notably, our model achieves leading results on the different kinds of temporal networks, demonstrating its practical impact and potential in large-scale applications.

Link To Code: https://github.com/kthrn22/L-STEP

Primary Area: Deep Learning->Graph Neural Networks

Keywords: Positional Encoding, Link Prediction, Transformer

Submission Number: 8546