What Do GNNs Actually Learn? Towards Understanding their Representations
Keywords: Graph neural networks, graph representation learning
Abstract: Although prior work has shed light on the expressiveness of Graph Neural Networks (GNNs) (\ie whether they can distinguish pairs of non-isomorphic graphs), it is still not clear what structural information is encoded into the node representations that are learned by those models.
In this paper, we address this gap by studying the node representations learned by four standard GNN models.
We find that some models produce identical representations for all nodes, while the representations learned by other models are linked to some notion of walks of specific length that start from the nodes.
We establish Lipschitz bounds for these models with respect to the number of (normalized) walks.
Additionally, we investigate the influence of node features on the learned representations.
We find that the representations learned at the $k$-the layer of the models are related to the initial features of nodes that can be reached in exactly $k$ steps.
We bound the Lipschitz constant of these models with respect to an optimization problem matching nodes' sets of walks.
Our theoretical analysis is validated through experiments on synthetic and real-world datasets.
We also apply our findings to understand the phenomenon of oversquashing that occurs in GNNs.
Primary Area: learning on graphs and other geometries & topologies
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 5105
Loading