Revisiting Embeddings for Graph Neural Networks

Skye Purchase, Yiren Zhao, Robert D. Mullins

Keywords: Graph Neural Networks, Embeddings, Graph Attention, Large Pretrained Models, Transfer Learning

TL;DR: We question current graph neural network embedding quality and present new GNN techniques to use large models (pre-trained or trained from scratch) to work directly on graph-connected data

Abstract:

Current graph representation learning techniques use Graph Neural Networks
(GNNs) to extract features from dataset embeddings. In this work, we examine
the quality of these embeddings and assess how changing them can affect the ac-
curacy of GNNs. We explore different embedding extraction techniques for both
images and texts; and find that the choice of embedding biases the performance
of different GNN architectures and thus the choice of embedding influences the
selection of GNNs regardless of the underlying dataset. In addition, we only see
an improvement in accuracy from some GNN models compared to the accuracy of
models trained from scratch or fine-tuned on the underlying data without utilising
the graph connections. As an alternative, we propose Graph-connected Network
(GraNet) layers to better leverage existing unconnected models within a GNN.
Existing language and vision models are thus improved by allowing neighbour-
hood aggregation. This gives a chance for the model to use pre-trained weights, if
possible, and we demonstrate that this approach improves the accuracy compared
to traditional GNNs: on Flickr v2, GraNet beats GAT2 and GraphSAGE by 7.7%
and 1.7% respectively.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Deep Learning and representational learning